From david.holmes at oracle.com Wed Nov 1 01:42:35 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 1 Nov 2017 11:42:35 +1000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> Message-ID: <60aa3ea7-ae40-cf29-c3d2-494092a1356c@oracle.com> On 1/11/2017 4:39 AM, coleen.phillimore at oracle.com wrote: > On 10/31/17 12:39 PM, John Rose wrote: >> On Oct 30, 2017, at 8:50 PM, David Holmes > > wrote: >>> >>> The flag should not be experimental - enabling resizable tables is >>> not an experiment that the user is controlling, it is the default >>> behaviour that we intend to occur. The flag is there to turn it off >>> if for some reason this doesn't work >> >> This describes a diagnostic flag, not a product flag. ?We mask those >> flags to avoid polluting the list of true product flags with >> diagnostic hooks. I can live with it being a diagnostic flag ... >>> - the flag should be a product flag. >> >> Hence, it should be a diagnostic flag, if we think there is a long-term >> diagnostic benefit to turning it off. ?Otherwise it should be removed as >> Gerard suggests. ?Diagnosis could be for either bugs or performance. >> >> If there is a non-diagnostic reason, where we expect a user to flip it >> more or less permanently for their use case, it could be product. That's really the unknown part. As usual we think a change will be to the benefit of all, but we can't be certain of that. > I agree this should be a diagnostic flag.?? Technically we shouldn't > need it because this feature has been well developed and tested, but > we've been erring on the side of caution lately. > > Do we still need a CSR for the diagnostic flag? All flag additions/removals/modification require a CSR request. >> >> Experimental flags should be short-lived. // experimental flags are in support of features that are not // part of the officially supported product, but are available // for experimenting with. They could, for example, be performance // features that may not have undergone full or rigorous QA, but which may // help performance in some cases and released for experimentation // by the community of users and developers. This flag also allows one to // be able to build a fully supported product that nonetheless also // ships with some unsupported, lightly tested, experimental features. Many of our experimental flags are long-lived. Perhaps it is time for a cull? Thanks, David >> > > Hopefully the diagnostic flag should be short-lived as well. > > Thanks, > Coleen >> ? John > From ioi.lam at oracle.com Wed Nov 1 02:43:56 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 31 Oct 2017 19:43:56 -0700 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> Message-ID: <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> Hi, Here's the new webrev for both the AppCDS implementation and tests. During internal review of the JEP, we have decided to integrate both implementation and tests at the same time. http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ As mentioned before, most of the "diffs" shown in this webrev are the result of copying the closed source files on top of files of the same name in the open repo. So in reviewing, instead of focusing on what's "changed", please focus on the entire content of the new version of each file. Testing: I did an OpenJDK linux-x64 build (without Oracle closed sources) and all the new appcds tests passed. Thanks - Ioi On 10/30/17 8:52 AM, Ioi Lam wrote: > Hi Dmitry, > > In the latest JDK 10 repo, is_jrt has been renamed to > is_modules_image. Please change the code accordingly. > > I will post my latest diff soon, with some test cases as well. > > Thanks > > - Ioi > > > On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >> Ioi, >> >> I'd tried to apply your patch to latest open JDK10 and >> the compilation fails with: >> >> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >> >> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >> >> Did I miss something? >> >> -Dmitry >> >> On 13.10.2017 02:48, Ioi Lam wrote: >>> Hi, >>> >>> Please review this change set. >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>> ???? https://bugs.openjdk.java.net/browse/JDK-8188791 >>> >>> This is the first step of implementing the following JEP, which moves >>> AppCDS from >>> closed repos into the openjdk repo: >>> >>> ???? https://bugs.openjdk.java.net/browse/JDK-8185996 >>> >>> In JDK 9, significant portion of AppCDS code resided in the closed >>> repo. >>> As part >>> of the open-sourcing effort of JDK 18.3, we will move the source code >>> into the >>> open repo. >>> >>> In this changeset, the code is moved verbatim as much as possible. The >>> intention is >>> only to relocate the sources, not to changing existing behaviors, >>> and not >>> to do any sort of refactoring. >>> >>> Most of the "diffs" shown in this webrev are the result of copying the >>> closed source >>> files on top of files of the same name in the open repo. So in >>> reviewing, instead of >>> focusing on what's "changed", it's better to focus on the entire >>> content >>> of the new >>> version of each file. >>> >>> The only functional change in this task is that the UseAppCDS flag is >>> changed from >>> a "commercial" flag to a regular "product" flag. This is because >>> "commercial" >>> flags are not supported by the OpenJDK build. >>> >>> Source code refactoring may be desirable, because the old open/closed >>> source >>> code structure had introduced some intermediary APIs to connect code >>> between >>> the two repos. Such API should be removed in a separate RFE. >>> >>> Also, some AppCDS tests are currently in the closed repo. These tests >>> will be >>> moved in a separate task. See JDK-8188792 for details. >>> >>> All the AppCDS tests (currently still in closed sources) passed with >>> both Oracle JDK >>> and OpenJDK. >>> >>> Thanks >>> - Ioi >> > From dean.long at oracle.com Wed Nov 1 03:03:09 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Tue, 31 Oct 2017 20:03:09 -0700 Subject: [10] JBS: 8167408: Invalid critical JNI function lookup In-Reply-To: References: <34e7e1e6-09bd-a0c3-d3de-23a825474dbb@oracle.com> <2e195876-97cf-bc1f-5e3b-e19c5c5f240d@oracle.com> Message-ID: Much better.? But why does the test need -Xbatch -Xcomp?? To me this doesn't look like a compiler-only issue.? The test should give the same result with -Xint right?? Adding runtime alias for their input... dl On 10/31/17 12:37 PM, jamsheed wrote: > Hi Dean, > > Thank you for the review, > > tested with a test case, previously it was not working for > windows-x86, now it works. > > revised webrev with test > case:http://cr.openjdk.java.net/~jcm/8167408/webrev.01/ > > Best regards, > > Jamsheed > > > On Tuesday 31 October 2017 02:18 AM, dean.long at oracle.com wrote: >> I think you need a native test for Windows x86 that defines >> JavaCritical methods with various signatures (especially arrays) to >> make sure this is working correctly. >> >> dl >> >> >> On 10/30/17 9:45 AM, jamsheed wrote: >>> Hi, >>> >>> request for review, >>> >>> jbs : https://bugs.openjdk.java.net/browse/JDK-8167408 >>> >>> webrev: http://cr.openjdk.java.net/~jcm/8167408/webrev.00/ >>> >>> (contributed by Ioannis Tsakpinis) >>> >>> desc: >>> >>> -- it starts with JavaCritical_ instead of Java_; >>> -- it does not have extra JNIEnv* and jclass arguments; >>> -- Java arrays are passed in two arguments: the first is an array >>> length, and the second is a pointer to raw array data. That is, no >>> need to call GetArrayElements and friends, you can instantly use a >>> direct array pointer. >>> >>> updated arg_size calculation wrt above points. >>> >>> Best regards, >>> >>> Jamsheed >>> >> > From david.holmes at oracle.com Wed Nov 1 03:11:44 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 1 Nov 2017 13:11:44 +1000 Subject: [10] JBS: 8167408: Invalid critical JNI function lookup In-Reply-To: References: <34e7e1e6-09bd-a0c3-d3de-23a825474dbb@oracle.com> <2e195876-97cf-bc1f-5e3b-e19c5c5f240d@oracle.com> Message-ID: <10969e74-47a1-5457-fcb2-75c3d1271243@oracle.com> On 1/11/2017 1:03 PM, dean.long at oracle.com wrote: > Much better.? But why does the test need -Xbatch -Xcomp?? To me this > doesn't look like a compiler-only issue.? The test should give the same > result with -Xint right?? Adding runtime alias for their input... AFAICS this should be a completely compiler and platform agnostic issue. Good to add tests but they shouldn't be restricted to only 32-bit Windows (especially as we no longer support it or do promoted builds for it, or even build in Mach5!). Thanks, David > dl > > > On 10/31/17 12:37 PM, jamsheed wrote: >> Hi Dean, >> >> Thank you for the review, >> >> tested with a test case, previously it was not working for >> windows-x86, now it works. >> >> revised webrev with test >> case:http://cr.openjdk.java.net/~jcm/8167408/webrev.01/ >> >> Best regards, >> >> Jamsheed >> >> >> On Tuesday 31 October 2017 02:18 AM, dean.long at oracle.com wrote: >>> I think you need a native test for Windows x86 that defines >>> JavaCritical methods with various signatures (especially arrays) to >>> make sure this is working correctly. >>> >>> dl >>> >>> >>> On 10/30/17 9:45 AM, jamsheed wrote: >>>> Hi, >>>> >>>> request for review, >>>> >>>> jbs : https://bugs.openjdk.java.net/browse/JDK-8167408 >>>> >>>> webrev: http://cr.openjdk.java.net/~jcm/8167408/webrev.00/ >>>> >>>> (contributed by Ioannis Tsakpinis) >>>> >>>> desc: >>>> >>>> -- it starts with JavaCritical_ instead of Java_; >>>> -- it does not have extra JNIEnv* and jclass arguments; >>>> -- Java arrays are passed in two arguments: the first is an array >>>> length, and the second is a pointer to raw array data. That is, no >>>> need to call GetArrayElements and friends, you can instantly use a >>>> direct array pointer. >>>> >>>> updated arg_size calculation wrt above points. >>>> >>>> Best regards, >>>> >>>> Jamsheed >>>> >>> >> > From david.holmes at oracle.com Wed Nov 1 04:46:53 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 1 Nov 2017 14:46:53 +1000 Subject: [10] JBS: 8167408: Invalid critical JNI function lookup In-Reply-To: <10969e74-47a1-5457-fcb2-75c3d1271243@oracle.com> References: <34e7e1e6-09bd-a0c3-d3de-23a825474dbb@oracle.com> <2e195876-97cf-bc1f-5e3b-e19c5c5f240d@oracle.com> <10969e74-47a1-5457-fcb2-75c3d1271243@oracle.com> Message-ID: <87e06194-5c4a-650c-8029-fe778182030a@oracle.com> On 1/11/2017 1:11 PM, David Holmes wrote: > On 1/11/2017 1:03 PM, dean.long at oracle.com wrote: >> Much better.? But why does the test need -Xbatch -Xcomp?? To me this >> doesn't look like a compiler-only issue.? The test should give the >> same result with -Xint right?? Adding runtime alias for their input... > > AFAICS this should be a completely compiler and platform agnostic issue. Platform agnostic yes, but the test fails without -Xcomp (doesn't seem to need -Xbatch). So there's something about this critical function mechanism that I don't understand. That said the test needs more logging so that if it does fail you can see what, if anything got executed. So the non-critical versions of the methods should print that they were called, and for good measure also the critical versions. Thanks, David From thomas.stuefe at gmail.com Wed Nov 1 07:15:56 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 1 Nov 2017 08:15:56 +0100 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: Hi Coleen, thanks a lot! On Tue, Oct 31, 2017 at 8:18 PM, wrote: > > This change looks good. > Coleen > > > On 10/30/17 8:49 AM, Lindenmaier, Goetz wrote: > >> Hi Thomas, >> >> the change looks good and I know we have made good use of this >> already. >> >> Can you look into the chunks? It could be useful to know that, >> for example, a medium chunk is used by a class loader, but >> still mostly empty (i.e., empty but not free). Well, there is no >> third variant after lower and upper case, but maybe empty >> parts could be indicated in the line with the dots by 'e's (obviously >> at the granularity of small chunks). >> >> Thanks, >> Goetz. >> >> -----Original Message----- >>> From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- >>> bounces at openjdk.java.net] On Behalf Of Thomas St?fe >>> Sent: Wednesday, October 25, 2017 6:52 AM >>> To: hotspot-runtime-dev at openjdk.java.net >>> Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace >>> fragmentation >>> >>> Hi all, >>> >>> could I please have your thoughts and reviews for this enhancement: >>> >>> Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 >>> Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- >>> metaspace-map/webrev.00/webrev/ >>> >>> At SAP, we added a something we call a metaspace map to the metaspace >>> analysis functions. This is a very simple ASCII print of the metaspace >>> layout. It shows the composition of the VirtualSpaceNodes on a chunk >>> basis. >>> It facilitates examining fragmentation and has been proven useful when >>> experimenting with the metaspace allocator. >>> >>> This change adds the map printing code and invokes it if a Metaspace OOM >>> occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, we >>> already print out a bunch of things). We also may consider adding this to >>> NMT or as a jcmd addition, but I did not want to overload the patch. >>> >>> Example output looks like this (mind that this will only make sense with >>> a >>> monospaced font). Dots indicate starts of chunks. Letters indicate chunk >>> type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case >>> letters indicating >>> a free chunk, upper case letters indicating a chunk in use. >>> >>> 0x0000000100000000: ...... >>> xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> 0x0000000100020000: >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> 0x0000000100040000: >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> HHHHHHHHH >>> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> 0x0000000100060000: . . . . . . . . . . . . . . >>> SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> M >>> 0x0000000100080000: . . . . >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >>> mmmmmmmmmmmmmmmmMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> M >>> 0x00000001000a0000: . . . . >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> M >>> 0x00000001000c0000: . . . . >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >>> mmmmmmmmmmmmmmmmMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> M >>> 0x00000001000e0000: . . . . >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> M >>> 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . >>> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM >>> MMMMMMMMMMMMMMMMMMMSS >>> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . >>> . . >>> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . >>> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> >>> >>> Thank you! >>> >>> Thomas >>> >> > From thomas.stuefe at gmail.com Wed Nov 1 07:20:24 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 1 Nov 2017 08:20:24 +0100 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: <667f07ea2fb64e97b8a149310961693f@sap.com> References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: Hi Goetz, On Mon, Oct 30, 2017 at 1:49 PM, Lindenmaier, Goetz < goetz.lindenmaier at sap.com> wrote: > Hi Thomas, > > the change looks good and I know we have made good use of this > already. > > Can you look into the chunks? It could be useful to know that, > for example, a medium chunk is used by a class loader, but > still mostly empty (i.e., empty but not free). Well, there is no > third variant after lower and upper case, but maybe empty > parts could be indicated in the line with the dots by 'e's (obviously > at the granularity of small chunks). > > Thanks, > Goetz. > > thank you for the review! Indicating which chunks are filled to what degree can certainly be done. Question is how useful this is (see Zhengyu's work for https://bugs.openjdk.java.net/browse/JDK-8189688), because within chunks there is no fragmentation, so the in-chunk map could look rather unexciting. I will take a look when I am back next week. Kind Regards, Thomas > -----Original Message----- > > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > > bounces at openjdk.java.net] On Behalf Of Thomas St?fe > > Sent: Wednesday, October 25, 2017 6:52 AM > > To: hotspot-runtime-dev at openjdk.java.net > > Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace > > fragmentation > > > > Hi all, > > > > could I please have your thoughts and reviews for this enhancement: > > > > Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 > > Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- > > metaspace-map/webrev.00/webrev/ > > > > At SAP, we added a something we call a metaspace map to the metaspace > > analysis functions. This is a very simple ASCII print of the metaspace > > layout. It shows the composition of the VirtualSpaceNodes on a chunk > basis. > > It facilitates examining fragmentation and has been proven useful when > > experimenting with the metaspace allocator. > > > > This change adds the map printing code and invokes it if a Metaspace OOM > > occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, we > > already print out a bunch of things). We also may consider adding this to > > NMT or as a jcmd addition, but I did not want to overload the patch. > > > > Example output looks like this (mind that this will only make sense with > a > > monospaced font). Dots indicate starts of chunks. Letters indicate chunk > > type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case > > letters indicating > > a free chunk, upper case letters indicating a chunk in use. > > > > 0x0000000100000000: ...... > > xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100020000: > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100040000: > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100060000: . . . . . . . . . . . . . . > > SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x0000000100080000: . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > > mmmmmmmmmmmmmmmmMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000a0000: . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000c0000: . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > > mmmmmmmmmmmmmmmmMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000e0000: . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . > . . > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > > > > > Thank you! > > > > Thomas > From zgu at redhat.com Wed Nov 1 12:37:44 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 1 Nov 2017 08:37:44 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <85d1d9de-69fa-0c30-1c26-763528ffa245@oracle.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> <85d1d9de-69fa-0c30-1c26-763528ffa245@oracle.com> Message-ID: Hi David, > > Yes there is. See the comments in Threads::destroy_vm and > VMThread::wait_for_vm_thread_exit. > > JNI_DestroyJavaVM -> > Threads::destroy_vm() -> > VMThread::wait_for_vm_thread_exit(); > _should_terminate = true; > > VMThread::run() > this->loop(); // returns when _should_terminate is true > ... > // 4526887 let VM thread exit at Safepoint > _no_op_reason = "Halt"; > SafepointSynchronize::begin(); > > The VMThread always takes the VM to a safepoint on a non-aborting exit. I see. These safepoints are bit too late, cause print_statistics() calls usually come before them. There are two solutions I can think of: 1) Printing NMT final report after VMThread takes the VM to the safepoint. 2) Leave NMT final reporting at where it is now, but execute a safepoint to report metadata stats. I don't see any issue for requesting a safepoint, do I miss anything? Thanks, -Zhengyu From vladimir.x.ivanov at oracle.com Wed Nov 1 12:46:53 2017 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Wed, 1 Nov 2017 15:46:53 +0300 Subject: [10] JBS: 8167408: Invalid critical JNI function lookup In-Reply-To: <87e06194-5c4a-650c-8029-fe778182030a@oracle.com> References: <34e7e1e6-09bd-a0c3-d3de-23a825474dbb@oracle.com> <2e195876-97cf-bc1f-5e3b-e19c5c5f240d@oracle.com> <10969e74-47a1-5457-fcb2-75c3d1271243@oracle.com> <87e06194-5c4a-650c-8029-fe778182030a@oracle.com> Message-ID: <4c6c9f5a-8680-9116-edef-990a9abccdeb@oracle.com> No magic here: the test isn't designed to be executed in interpreter and always expect LookUp::test() to be compiled first. Normal JNI entries are empty, but the test checks that the first element is written to on the first invocation. I agree that the bug is specific to JIT-compilers, since critical entry points (JavaCritical_) are called only from compiled code. Interpreter always goes through ordinary native counterparts (Java_). IMO using -Xcomp is fine to force the methods to be compiled. -Xbatch is redundant in that case. Best regards, Vladimir Ivanov On 11/1/17 7:46 AM, David Holmes wrote: > On 1/11/2017 1:11 PM, David Holmes wrote: >> On 1/11/2017 1:03 PM, dean.long at oracle.com wrote: >>> Much better.? But why does the test need -Xbatch -Xcomp?? To me this >>> doesn't look like a compiler-only issue.? The test should give the >>> same result with -Xint right?? Adding runtime alias for their input... >> >> AFAICS this should be a completely compiler and platform agnostic issue. > > Platform agnostic yes, but the test fails without -Xcomp (doesn't seem > to need -Xbatch). So there's something about this critical function > mechanism that I don't understand. > > That said the test needs more logging so that if it does fail you can > see what, if anything got executed. So the non-critical versions of the > methods should print that they were called, and for good measure also > the critical versions. > > Thanks, > David > From robbin.ehn at oracle.com Wed Nov 1 15:45:05 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 1 Nov 2017 16:45:05 +0100 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: <1d8cc30a-72a9-a32a-899a-496f829801f5@oracle.com> Hi Gerard, I looked at your pending v4 version. As mentioned in other forums the jtreg test should preferably be named TestXXX while the helper not starting with Test. I don't need to see another webrev. Looks good, thanks for fixing! /Robbin On 10/10/2017 10:40 PM, Gerard Ziemski wrote: > hi all, > > Please review this change that adds dynamic resizing of system dictionaries. > > The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock > > A few notes: > > - dynamic resizing is off when either dumping or using shared spaces > - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) > - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) > > bug: https://bugs.openjdk.java.net/browse/JDK-8184765 > webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 > > > cheers > From dean.long at oracle.com Wed Nov 1 18:08:58 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 1 Nov 2017 11:08:58 -0700 Subject: [10] JBS: 8167408: Invalid critical JNI function lookup In-Reply-To: <4c6c9f5a-8680-9116-edef-990a9abccdeb@oracle.com> References: <34e7e1e6-09bd-a0c3-d3de-23a825474dbb@oracle.com> <2e195876-97cf-bc1f-5e3b-e19c5c5f240d@oracle.com> <10969e74-47a1-5457-fcb2-75c3d1271243@oracle.com> <87e06194-5c4a-650c-8029-fe778182030a@oracle.com> <4c6c9f5a-8680-9116-edef-990a9abccdeb@oracle.com> Message-ID: OK, somehow I missed the part about JavaCritical methods only being called from compiled code.? So -Xcomp makes sense. dl On 11/1/17 5:46 AM, Vladimir Ivanov wrote: > No magic here: the test isn't designed to be executed in interpreter > and always expect LookUp::test() to be compiled first. > > Normal JNI entries are empty, but the test checks that the first > element is written to on the first invocation. > > I agree that the bug is specific to JIT-compilers, since critical > entry points (JavaCritical_) are called only from compiled code. > Interpreter always goes through ordinary native counterparts (Java_). > > IMO using -Xcomp is fine to force the methods to be compiled. -Xbatch > is redundant in that case. > > Best regards, > Vladimir Ivanov > > On 11/1/17 7:46 AM, David Holmes wrote: >> On 1/11/2017 1:11 PM, David Holmes wrote: >>> On 1/11/2017 1:03 PM, dean.long at oracle.com wrote: >>>> Much better.? But why does the test need -Xbatch -Xcomp?? To me >>>> this doesn't look like a compiler-only issue.? The test should give >>>> the same result with -Xint right?? Adding runtime alias for their >>>> input... >>> >>> AFAICS this should be a completely compiler and platform agnostic >>> issue. >> >> Platform agnostic yes, but the test fails without -Xcomp (doesn't >> seem to need -Xbatch). So there's something about this critical >> function mechanism that I don't understand. >> >> That said the test needs more logging so that if it does fail you can >> see what, if anything got executed. So the non-critical versions of >> the methods should print that they were called, and for good measure >> also the critical versions. >> >> Thanks, >> David >> From calvin.cheung at oracle.com Wed Nov 1 19:07:26 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 01 Nov 2017 12:07:26 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> Message-ID: <59FA1B6E.4090000@oracle.com> On 10/27/17, 12:55 AM, Thomas St?fe wrote: > Hi Calvin, > > On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung > > wrote: > > Hi Thomas, > > On 10/25/17, 11:54 PM, Thomas St?fe wrote: > > Hi Calvin, > > this is a worthwhile addition, thank you for your work! > > Thanks for your review. > > > Thanks for your work :) > > First off, there is another issue in file_attribute_data_to_stat(). We > actually had this issue before, but your work uncovered it: > > According to POSIX > (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html) > and every unix manpage I looked at (solaris, linux, aix..), st_ctime > is not the file creation time but the last time file status was > changed. Only Microsoft gets this wrong in their posix layer, its > stat() function returns > (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx) creation > time in st_ctime. > > But as os::stat() is a platform independent layer, I'd say we should > behave the same on all platforms, and that would be the Posix way. > > I did not find any code calling os::stat() and using st_ctime, so this > is still save to change for windows. (Note that I found that > perfMemory_xxx.cpp uses plain OS ::stat and misuses ctime as "creation > time" - I opened a bug for that > (https://bugs.openjdk.java.net/browse/JDK-8190260 - but that does not > affect us, as they do not call os::stat()). > > There is one more complication: in os::stat() we use plain ::stat() in > the single byte case.: > > *+ if (strlen(path) < MAX_PATH) {* > *+ ret = ::stat(pathbuf, sbuf);* > *+ } else {* > * > * > But ::stat() on Windows is broken, as I wrote above. We should not use > it, especially not if we change the meaing of st_ctime in the double > byte path. > > My suggestion would be to just always call GetFileAttributes(), also > for the single byte path. A very simple solution would be to just > always go the double byte path with UNC translation, regardless of the > path length. Or, if you are worried about performance, for paths > shorter than MAX_PATH we use GetFileAttributesA(), for longer paths > GetFileAttributesW with UNC translation. In both cases you get > a WIN32_FILE_ATTRIBUTE_DATA which you can feed tp your > file_attribute_data_to_stat() routine and have the struct stat you > return be always consistent. I ran into an issue with the dwFileAttributes value for a jar file. On Windows Server O/S, the value is set to 0x2020 which is (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) but on a desktop O/S like Windows 7, it is set to 0x0020 which is just FILE_ATTRIBUTE_ARCHIVE. I've fixed it in file_attribute_data_to_stat(). I've also used GetTickCounts() to measure the performance of ::stat() vs GetFileAttributesA() plus file_attribute_data_to_stat(). There's no difference at the ms resolution. Using the high-resolution timestamp (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx), it doesn't show much difference. > > > But onward: > > > > ========================= > > src/hotspot/os/windows/os_windows.cpp > > Could you please use wchar_t instead of WCHAR? > > Fixed. > > > --------------- > in os::stat(): > > This cumbersome malloc/strcpy sequence: > > ! size_t path_len = strlen(path) + 1; > + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), > mtInternal); > + if (pathbuf == NULL) { > + errno = ENOMEM; > return -1; > } > os::native_path(strcpy(pathbuf, path)); > > can be reduced to a simple strdup: > > char* pathbuf = os::strdup(path, mtInternal); > if (pathbuf == NULL) { > errno = ENOMEM; > return -1; > } > os::native_path(pathbuf); > > This also would separate strcpy() from os::native_path() which > is a bit unreadable. > > I've made the change. > > > -------------------- > > The single-byte path (strdup, ::stat()), together with its > free(), can be moved inside the (strlen(path) < MAX_PATH) > condition. os::native_path will not change the path length (it > better not, as it operates on the input buffer). That avoids > having two allocations on the doublebyte path. > > > os::native_path() converts a path like C:\\\\somedir to C:\\somedir > So I'll need the converted path in the wchar_t case too. The > freeing of the pathbuf is being done at the end of the function or > in the middle of the wchar_t case if there's an error. > > > Oh, you are right,of course. I missed that it this is needed for both > paths. > > > > ----------------------- > > But seeing that translation to UNC paths is something we might > want more often, I would combine allocation and UNC prefix > adding to one function like this, to avoid the memove and > increase reusability: > > // Creates an unc path from a single byte path. Return buffer > is allocated in C heap and needs to be freed by caller. > Returns NULL on error. > static whchar_t* create_unc_path(const char* s) { > - does s start with \\?\ ? > - yes: > - os::malloc(strlen(s) + 1) and mbstowcs_s > - no: > - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to > fourth position in string, prefix with L"\\?\" > } > > > I also include the case for adding L"\\\\?\\UNC\0" at the > beginning to be consistent with libjava/canonicalize_md.c. > > > We also need error checking to mbstowcs_s. > > I've added assert like the following after the call: > > assert(converted_chars == path_len, "sanity"); > > > > create_unc_path() : > > - could you convert the /**/ to // comments, please? Fixed. > > - not sure about the assert after mbstowcs_s: if we happen to > encounter an invalid multibyte character, function will fail and > return an error: > > https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx > "If mbstowcs_s encounters an invalid multibyte character, it puts 0 in > *``pReturnValue, sets the destination buffer to an empty string, sets > errno to EILSEQ, and returns EILSEQ." I've changed create_unc_path() so that the caller will get the errno and removed the assert. static wchar_t* create_unc_path(const char* path, errno_t &err) > > As this is dependent on user data, we should not assert, but handle > the return code of mbstowcs_s gracefully. Same goes for > libjava/canonicalize_md.c. > > - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + 7, path, > path_len); > third parameter is wrong, as we hand in an offset into the buffer, we > must decrement the buffer size by this offset, so correct would be > path_len +7 - 7 or just path_len. > > - Same error below: + ::mbstowcs_s(&converted_chars, &wpath[4], > path_len + 4, path, path_len); Fixed in both places. > > > > Just for cleanliness, I would then wrap deallocation into an > own function. > > static viud destroy_unc_path(whchar_t* s) { os::free(s); } > > I've added the destroy function. > > > These two functions could be candidates of putting into > os::win32 namespace, as this form of ANSI->UNC path > translation is quite common - whoever wants to use the xxxxW() > functions must do this. > > I'm leaving them in os_windows.cpp since they're being used only > within that file. > > > Fine by me. > > > > ----------------------- > > FindFirstFileW: > > I am pretty sure that you can achieve the same result with > GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. > > It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very similar > to WIN32_FIND_DATAW. > > > It is way more straightforward to use than FindFirstFileW, as > it does not require you to write a callback hook. > > I've switched to using GetFileAttributesExW(). > > > Thank you, this is way more readable. > Another issue: do we still need the fix for 6539723 if we switch from > stat() to GetFileAttributesExW()? This fix looks important, but the > comment > indicates that it could break things if the original bug is not present. > > Btw, this is another strong argument for scrapping ::stat() altogether > on all code paths, not only for long input paths, because stat() and > GetFileAttributesExW() may behave > differently. So it would be good to use the same API on all code > paths, in order to get the best test coverage. For this round of change, I'm using GetFileAttributesEx[A|W]() for both code paths. webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ thanks, Calvin > > > > ------- > > eval_find_data(): This is more of a generic helper function, > could you rename this to something clearer, e.g. > make_double_word() ? > > Ok. I've renamed it. > > > Also, setup_stat(), could this be renamed more clearly to > something like WIN32_FIND_DATA_to_stat? or lowercase if this > bothers you :) > > I'm naming the function as file_attribute_data_to_stat() to match > with the data structure name. > > > Thanks for taking my suggestions. > > > > ================================== > src/hotspot/share/classfile/classLoader.cpp > > In ClassPathDirEntry::open_stream(), I would feel better if we > were asserting _dir and name to be != NULL before feeding it > to strlen. > > I've added an assert statement. > > > =================== > > Here's an updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ > > > > Thanks! > > Thomas > > thanks, > Calvin > > > Thanks! > > Thomas > > > > > > > On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung > > >> wrote: > > I've reworked this fix by using the Unicode version of open > (wopen) to handle path name longer than max path with the path > prefixed to indicate an extended length path as described > in [1]. > > The Unicode version of stat (wstat) doesn't work well with > long > path [2]. So FindFirstFileW will be used.The data in > WIN32_FIND_DATA returned from FindFirstFileW needs to be > transferred to the stat structure since the caller expects a > return stat structure and other platforms return a stat > structure. > > In classLoader.cpp, calculate the size of buffer required > instead > of limiting it to JVM_MAXPATHLEN. > In os_windows.cpp, dynamically allocate buffers in > os::open and > os::stat. > > updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ > > > > > It passed hs-tier2 testing using mach5. > > thanks, > Calvin > > [1] > https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH > > > > > [2] > https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral > > > > > > > On 10/16/17, 3:15 PM, Calvin Cheung wrote: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 > > > > > Adding a warning message if the full path or the directory > length exceeds MAX_PATH on windows. > > Example warning messages. > > 1) The full path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: construct > full path > name failed: total length 270 exceeds max length of 260 > dir > > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > length 259 > name Hello.class length 11 > > 2) The directory path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: os::stat > failed: > path length 265 exceeds max length 260 > path > > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx > > webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ > > > > > Testing: > JPRT (including the new test) > > thanks, > Calvin > > > From jiangli.zhou at Oracle.COM Wed Nov 1 19:28:00 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Wed, 1 Nov 2017 12:28:00 -0700 Subject: RFR: 8184206: Resolve all string constants in shared classes at CDS dump time Message-ID: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> Hi, Please review the following change for 8184206. webrev: http://cr.openjdk.java.net/~jiangli/8184206/webrev.00/ RFE: https://bugs.openjdk.java.net/browse/JDK-8184206?filter=14921 Before the change, constant pool string entries in shared classes were only resolved to existing interned strings at CDS dump time. The string constants remained as unresolved if the strings were not yet interned. The change in the above webrev resolves all string constants (excluding pseudo strings) for shared classes. For any strings that are not yet interned, java.lang.String instances are created and added to the string tabled when the string constants are resolved. All string instances referenced from the string table are archived. Resolving all string constants provides better support for AOT and CDS integration. It also improves CDS/AppCDS usability by removing the needs for profiling target applications to create a string list for archiving. The change results more string objects being created and archived at CDS dump time. Measurement on linux-x64 using the default class list shows there is about 1.7% size increase for the CDS archive with all strings resolved. The increase of the generated archive size is minimal comparing to the benefits listed above. Tested with tier1, tier2, and tier3 tests. Thanks, Jiangli From gerard.ziemski at oracle.com Wed Nov 1 19:40:21 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Wed, 1 Nov 2017 14:40:21 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <1d8cc30a-72a9-a32a-899a-496f829801f5@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <1d8cc30a-72a9-a32a-899a-496f829801f5@oracle.com> Message-ID: <18AAC38F-5258-4A19-92AB-8380CE199118@oracle.com> hi Robin, Thank you for the review. Just for completeness, the final rev4, updated with your feedback is here: http://cr.openjdk.java.net/~gziemski/8184765_rev4 cheers > On Nov 1, 2017, at 10:45 AM, Robbin Ehn wrote: > > Hi Gerard, > > I looked at your pending v4 version. As mentioned in other forums the jtreg test should preferably be named TestXXX while the helper not starting with Test. > > I don't need to see another webrev. > > Looks good, thanks for fixing! > > /Robbin > > On 10/10/2017 10:40 PM, Gerard Ziemski wrote: >> hi all, >> Please review this change that adds dynamic resizing of system dictionaries. >> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >> A few notes: >> - dynamic resizing is off when either dumping or using shared spaces >> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >> cheers From coleen.phillimore at oracle.com Wed Nov 1 20:29:04 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 1 Nov 2017 16:29:04 -0400 Subject: RFR (xs) 8190491: SA tests failed after 8189610 changes Message-ID: Summary: Changed a type in hotspot not reflected in SA code. open webrev at http://cr.openjdk.java.net/~coleenp/8190491.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8190491 Ran test/hotspot/serviceability/sa and gc/metaspace tests locally with fix, tier1 testing locally in progress.? I'll run more tests if we wish to wait. Thanks, Coleen From harold.seigel at oracle.com Wed Nov 1 20:46:12 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 1 Nov 2017 16:46:12 -0400 Subject: RFR (xs) 8190491: SA tests failed after 8189610 changes In-Reply-To: References: Message-ID: <2910d241-52e8-50c1-d924-33cc039f3704@oracle.com> Hi Coleen, The fix looks good. Harold On 11/1/2017 4:29 PM, coleen.phillimore at oracle.com wrote: > Summary: Changed a type in hotspot not reflected in SA code. > > open webrev at http://cr.openjdk.java.net/~coleenp/8190491.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8190491 > > Ran test/hotspot/serviceability/sa and gc/metaspace tests locally with > fix, tier1 testing locally in progress.? I'll run more tests if we > wish to wait. > > Thanks, > Coleen From jiangli.zhou at oracle.com Wed Nov 1 22:57:39 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Wed, 1 Nov 2017 15:57:39 -0700 Subject: RFR (xs) 8190491: SA tests failed after 8189610 changes In-Reply-To: References: Message-ID: Looks good. Thanks, Jiangli > On Nov 1, 2017, at 1:29 PM, coleen.phillimore at oracle.com wrote: > > Summary: Changed a type in hotspot not reflected in SA code. > > open webrev at http://cr.openjdk.java.net/~coleenp/8190491.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8190491 > > Ran test/hotspot/serviceability/sa and gc/metaspace tests locally with fix, tier1 testing locally in progress. I'll run more tests if we wish to wait. > > Thanks, > Coleen From harold.seigel at oracle.com Wed Nov 1 23:44:38 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 1 Nov 2017 19:44:38 -0400 Subject: RFR (xs) 8190491: SA tests failed after 8189610 changes In-Reply-To: References: Message-ID: <04eb376c-6678-831f-e53b-9d753f63ee3f@oracle.com> Thanks Jiangli! I went ahead and pushed the change for Coleen. Harold On 11/1/2017 6:57 PM, Jiangli Zhou wrote: > Looks good. > > Thanks, > Jiangli > >> On Nov 1, 2017, at 1:29 PM, coleen.phillimore at oracle.com wrote: >> >> Summary: Changed a type in hotspot not reflected in SA code. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8190491.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8190491 >> >> Ran test/hotspot/serviceability/sa and gc/metaspace tests locally with fix, tier1 testing locally in progress. I'll run more tests if we wish to wait. >> >> Thanks, >> Coleen From coleen.phillimore at oracle.com Thu Nov 2 02:16:46 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 1 Nov 2017 22:16:46 -0400 Subject: RFR (xs) 8190491: SA tests failed after 8189610 changes In-Reply-To: <04eb376c-6678-831f-e53b-9d753f63ee3f@oracle.com> References: <04eb376c-6678-831f-e53b-9d753f63ee3f@oracle.com> Message-ID: <7bde7233-f453-d1e2-9083-8a4f734f637e@oracle.com> Thanks for the review and pushing the change! Coleen On 11/1/17 7:44 PM, harold seigel wrote: > Thanks Jiangli! > > I went ahead and pushed the change for Coleen. > > Harold > > > On 11/1/2017 6:57 PM, Jiangli Zhou wrote: >> Looks good. >> >> Thanks, >> Jiangli >> >>> On Nov 1, 2017, at 1:29 PM, coleen.phillimore at oracle.com wrote: >>> >>> Summary: Changed a type in hotspot not reflected in SA code. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/8190491.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8190491 >>> >>> Ran test/hotspot/serviceability/sa and gc/metaspace tests locally >>> with fix, tier1 testing locally in progress.? I'll run more tests if >>> we wish to wait. >>> >>> Thanks, >>> Coleen > From david.holmes at oracle.com Thu Nov 2 02:23:12 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 2 Nov 2017 12:23:12 +1000 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> <85d1d9de-69fa-0c30-1c26-763528ffa245@oracle.com> Message-ID: <1dfb4131-262d-2fbd-c77a-6447327d2627@oracle.com> On 1/11/2017 10:37 PM, Zhengyu Gu wrote: > Hi David, >> >> Yes there is. See the comments in Threads::destroy_vm and >> VMThread::wait_for_vm_thread_exit. >> >> JNI_DestroyJavaVM -> >> ?? Threads::destroy_vm() -> >> ???? VMThread::wait_for_vm_thread_exit(); >> ?????? _should_terminate = true; >> >> VMThread::run() >> ?? this->loop(); // returns when _should_terminate is true >> ?? ... >> ?? // 4526887 let VM thread exit at Safepoint >> ?? _no_op_reason = "Halt"; >> ?? SafepointSynchronize::begin(); >> >> The VMThread always takes the VM to a safepoint on a non-aborting exit. > > > I see. > > These safepoints are bit too late, cause print_statistics() calls > usually come before them. Ok. It was just a thought. > There are two solutions I can think of: > > 1) Printing NMT final report after VMThread takes the VM to the safepoint. > > 2) Leave NMT final reporting at where it is now, but execute a safepoint > to report metadata stats. I don't see any issue for requesting a > safepoint, do I miss anything? Sorry I haven't followed this too closely. You'd need to be careful about requesting a synchronous safepoint during termination. David > > Thanks, > > -Zhengyu > > From mikhailo.seledtsov at oracle.com Thu Nov 2 03:11:26 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 1 Nov 2017 20:11:26 -0700 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration Message-ID: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> Please review these tests that were developed to test JVM's container awareness in Docker environment. ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 ??? Webrev: ????? Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ ????? WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ ??? Testing: ??????? 1. Locally: Linux-x64, docker engine version: 17.06.2-ce ?????????? Ran the developed tests via jtreg ?????????? Pass ??????? 2. Automated testing system - run these tests ?????????? In progress Thank you, Misha From jini.george at oracle.com Thu Nov 2 04:54:03 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 2 Nov 2017 10:24:03 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> From sangheon.kim at oracle.com Thu Nov 2 05:53:12 2017 From: sangheon.kim at oracle.com (sangheon.kim) Date: Wed, 1 Nov 2017 22:53:12 -0700 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: Message-ID: Hi Kishor, On 10/31/2017 04:53 PM, Kharbas, Kishor wrote: > > Greetings, > > I am continuing the earlier discussion [1] with a revised issue number > representing the implementation of the JEP [2]. > > I have addressed the remaining unaddressed comments (copied below) in > these webrevs - > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11/ > > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11_to_10/ > > > Also, I would appreciate a review of the CSR for the new flag at > https://bugs.openjdk.java.net/browse/JDK-8190309. > CSR: Reviewed. > > > ?- in that same thread there has also been the question about the > > > > need > > > > for blocking the signals for the process that has gone unanswered. > > > > > > Removed the blocking of signals. > > > > ?- Arguments::finalize_vm_init_args: maybe at the place where the > > > > change adds the warning/info message about NUMA support being > > > > specific > > > > to the file system; also add the same warning about UseLargePages > > > > that > > > > is located elsewhere. > > > > > > > > Like "UseXXXX may not be supported in some specific file system > > > > implementations." - or just ignore as Andrew said in the other > > > > email thread. > > > > > > > > Note that I am not sure that the selected place is the correct > > > > place > > > > for such warning about incompatible flags (and maybe > > > > UseNUMA/UseLargePages should be forcibly disabled here? But then > > > > again, it does not hurt just having it enabled?). > > > > > > > > I will let the runtime team comment as a lot of that argument > > > > processing changed in jdk9. > > > > > > > > Maybe Arguments::check_vm_args_consistency() is better. > > > > > > > > There is similar code in Arguments::adjust_after_os()... > > > > > > 1. Moved the check from finalize_vm_init_args() to > check_vm_args_consistency() > > 2. When UseNUMA flag is true, adaptive logical group chunk resizing is > used for Numa aware collectors such as ParallelOld. Just like UseNUMA > is disabled in os::inti_2() in Linux when UseLargePages is set, it > will be a good idea to disable UseNUMA when the new option is used. > > 3. I think its ok to leave UseNUMAInterleaving ON as it can be used > for other memory areas. > > 4. For the same reason I also do not disable UseLargePages. > > 5. About handling UseLargePages, I thought of handling it the same way > as when reserve_memory_special() fails to allocate large pages, i.e. > generating a log_debug message. > > /// failed; try to reserve regular memory below/ > > /? if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) ||/ > > /!FLAG_IS_DEFAULT(LargePageSizeInBytes))) {/ > > /?log_debug(gc, heap, coops)("Reserve regular memory without large > pages");/ > > > > ?- here I may probably be speaking wrongly on behalf of the > > > > runtime > > > > team, but os.hpp, as an abstraction of the OS should probably not > > > > have > > > > platform specific ifdefs in it. > > > > > > and > > > > > - You removed os::attempt_reserve_memory_at() from os.cpp and > > > > > split > > > > > into os_posix.cpp and os_windows.cpp. > > > > > ? But I think you should remain os::attempt_reserve_memory_at() > > > > > at > > > > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > > > > for > > > > > pd_xxx. Of cource move MemTracker function call as well. > > In the updated webrev, I move the implementation from os_posix.cpp and > os_window.cpp to respective pd_xxx functions. No AIX specific ifdef is > required now. > Thank you for all changes. I have minor nits now: ============================================== - os***.cpp has some function names which include *dax*. I would prefer not to include it. As you know, Thomas also pointed it. ============================================== src/hotspot/share/runtime/arguments.cpp 2537???? if (!FLAG_IS_DEFAULT(UseNUMAInterleaving) || !FLAG_IS_DEFAULT(UseNUMA)) { - Don't we need to check these 2 flags' value to be true? i.e. if user sets to false, below message will be printed. ============================================== test/hotspot/jtreg/gc/TestAllocateHeapAt.java - On other discussion, I mentioned to test only for Windows and Linux as the JEP described only about those 2. But without *dax* function names, it seems like not filtering OS seems okay too. 2? * Copyright (c) 2017, xxx Oracle and/or its affiliates. All rights reserved. - Please remove 'xxx '. 47???? Collections.addAll(vmOpts, new String[] {"-XX:AllocateHeapAt="+test_dir, - Add spaces. '+test_dir' -> ' + test_dir' 49????????????????????????????????????????????? "-Xmx50m", 50????????????????????????????????????????????? "-Xms50m", - You said there were no special reason for 200m(heap size of webrev.10) on other discussion. I would recommend 32m. FYI, I ran hs-tier1 and hs-tier2 with your patch and the result is good. i.e. no new failures. Thanks, Sangheon > Thank you and looking forward for your feedback. > > - Kishor > > [1] > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-October/020682.html > > [2] https://bugs.openjdk.java.net/browse/JDK-8171181 > > From jini.george at oracle.com Thu Nov 2 04:54:03 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 2 Nov 2017 10:24:03 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> X-sender: X-Receiver: X-CreatedBy: MSExchange15 X-HeloDomain: mx.expurgate.net X-ExtendedProps: BQBjAAoAqbJ+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAGcrAAB2AAAABQBkAA8ABAAAAEVkZ2U= X-Source: SMTP:External Receive Connector X-SourceIPAddress: 195.190.135.10 X-EndOfInjectedXHeaders: 7733 Received: from mx.expurgate.net (195.190.135.10) by smtpgw03.sap-ag.de (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov 2017 05:54:39 +0100 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1eA7WZ-000EZw-D4 for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:39 +0100 Received: from [156.151.31.69] (helo=ucsinet41.oracle.com) by mxtls.expurgate.net with ESMTPS (eXpurgate 4.2.1-3) (envelope-from ) id 59faa50b-7c01-c0a8033409dd-9c971f454783-3 for ; Thu, 02 Nov 2017 05:54:38 +0100 Received: from aojmv0009 (unknown [137.254.59.6]) by ucsinet41.oracle.com with smtp id 4eb7_00c7_3ecf1c93_28fd_4a9f_97f0_f31de9ddb295; Thu, 02 Nov 2017 04:54:17 +0000 Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) by aojmv0009 (Postfix) with ESMTP id 15A99229CEC; Thu, 2 Nov 2017 04:57:26 +0000 (UTC) X-Original-To: serviceability-dev at openjdk.java.net Delivered-To: serviceability-dev at openjdk.java.net Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; Thu, 02 Nov 2017 04:54:08 +0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA24s8KW028461 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 From: Jini George To: "serguei.spitsyn at oracle.com" , "serviceability-dev at openjdk.java.net" , "hotspot-runtime-dev at openjdk.java.net" , References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> Organization: Oracle Corporation Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> Date: Thu, 2 Nov 2017 10:24:03 +0530 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Source-IP: userv0022.oracle.com [156.151.31.74] X-BeenThere: serviceability-dev at openjdk.java.net X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Technical discussion about the development of serviceability technologies \(debugging, profiling, monitoring, and management\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: serviceability-dev-bounces at openjdk.java.net Sender: serviceability-dev X-purgate-ID: tlsNG-9b91ac/1509598479-00007C01-219CB02A/0/0 X-purgate-type: clean X-purgate-size: 2000 X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de Comment: eleven transport token MTU2LjE1MS4zMS42OQBzZXJ2aWNlYWJpbGl0eS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADgwYzQ2ZTI0N2Q2MTBlYjNiNTQ5MmE5YWE2NzJjNzJjN2NiNjcwNDQ= X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-SAP-SPAM-Status: clean Return-Path: serviceability-dev-bounces at openjdk.java.net X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:39.6785 (UTC) X-MS-Exchange-Organization-OriginalClientIPAddress: 195.190.135.10 X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-Network-Message-Id: 8fba1fcd-01cd-4a67-1ee6-08d521add313 X-MS-Exchange-Organization-OriginalSize: 7400 X-MS-Exchange-Organization-HygienePolicy: Standard X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11997.291|UTH=0.001|RST=11997.244|QS=0.029|CATCM-McAfeeTxRoutingAgent=0.011|CATCM=0.011|CAT=0.012;2017-11-02T08:14:36.969Z Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> X-sender: X-Receiver: X-CreatedBy: MSExchange15 X-HeloDomain: mx.expurgate.net X-ExtendedProps: BQBjAAoAALN+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAIMrAAB2AAAABQBkAA8ABAAAAEVkZ2U= X-Source: SMTP:External Receive Connector X-SourceIPAddress: 194.145.224.110 X-EndOfInjectedXHeaders: 7704 Received: from mx.expurgate.net (194.145.224.110) by smtpgw03.sap-ag.de (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov 2017 05:54:46 +0100 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1eA7Wg-0004R0-FP for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:46 +0100 Received: from [141.146.126.229] (helo=acsinet41.oracle.com) by mxtls.expurgate.net with ESMTPS (eXpurgate 4.3.1) (envelope-from ) id 59faa50f-65eb-c0a8029b09dd-8d927ee53db9-3 for ; Thu, 02 Nov 2017 05:54:45 +0100 Received: from aojmv0009 (aojmv0009.oracle.com [137.254.59.6]) by acsinet41.oracle.com with smtp id 6461_15f0_154b24ba_5aa0_408c_88bd_1cdb37c9c8f4; Thu, 02 Nov 2017 04:54:31 +0000 Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) by aojmv0009 (Postfix) with ESMTP id 3B280229CF4; Thu, 2 Nov 2017 04:57:26 +0000 (UTC) X-Original-To: hotspot-runtime-dev at openjdk.java.net Delivered-To: hotspot-runtime-dev at openjdk.java.net Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; Thu, 02 Nov 2017 04:54:08 +0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA24s8KW028461 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 From: Jini George To: "serguei.spitsyn at oracle.com" , "serviceability-dev at openjdk.java.net" , "hotspot-runtime-dev at openjdk.java.net" , References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> Organization: Oracle Corporation Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> Date: Thu, 2 Nov 2017 10:24:03 +0530 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Source-IP: userv0022.oracle.com [156.151.31.74] X-BeenThere: hotspot-runtime-dev at openjdk.java.net X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical discussion about the development of the HotSpot runtime subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: hotspot-runtime-dev-bounces at openjdk.java.net Sender: hotspot-runtime-dev X-purgate-ID: tlsNG-81024b/1509598486-000065EB-186BA960/0/0 X-purgate-type: clean X-purgate-size: 2000 X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de Comment: eleven transport token MTQxLjE0Ni4xMjYuMjI5AGhvdHNwb3QtcnVudGltZS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADE0ZjI2NjVmYTY4ZGRiOWMwNGJiMGU0MmE2Njk2M2ZiNWQxNzAyOGI= X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-SAP-SPAM-Status: clean Return-Path: hotspot-runtime-dev-bounces at openjdk.java.net X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:46.7566 (UTC) X-MS-Exchange-Organization-OriginalClientIPAddress: 194.145.224.110 X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-Network-Message-Id: 9365fa06-08de-47df-2865-08d521add74b X-MS-Exchange-Organization-OriginalSize: 7380 X-MS-Exchange-Organization-HygienePolicy: Standard X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11990.260|UTH=0.001|RST=11990.260|CATCM-McAfeeTxRoutingAgent=0.004|CATCM=0.005|CAT=0.006;2017-11-02T08:14:37.016Z Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> From jini.george at oracle.com Thu Nov 2 04:54:03 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 2 Nov 2017 10:24:03 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> X-sender: X-Receiver: X-CreatedBy: MSExchange15 X-HeloDomain: mx.expurgate.net X-ExtendedProps: BQBjAAoAtfd+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAN8MAAB3AAAABQBkAA8ABAAAAEVkZ2U= X-Source: SMTP:External Receive Connector X-SourceIPAddress: 194.145.224.110 X-EndOfInjectedXHeaders: 7619 Received: from mx.expurgate.net (194.145.224.110) by smtpgw03.sap-ag.de (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov 2017 06:15:06 +0100 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1eA7qL-0003Wx-9M for axel.siebenborn at sap.com; Thu, 02 Nov 2017 06:15:05 +0100 Received: from [156.151.31.69] (helo=ucsinet41.oracle.com) by mxtls.expurgate.net with ESMTPS (eXpurgate 4.3.1) (envelope-from ) id 59faa520-6357-c0a8029709dd-9c971f45d7b5-3 for ; Thu, 02 Nov 2017 05:55:02 +0100 Received: from aojmv0009 (aojmv0009.oracle.com [137.254.59.6]) by ucsinet41.oracle.com with smtp id 3f0c_0176_ccf6c206_c66b_4d53_aac8_13819c89af9f; Thu, 02 Nov 2017 04:54:29 +0000 Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) by aojmv0009 (Postfix) with ESMTP id 2FB68229CF1; Thu, 2 Nov 2017 04:57:26 +0000 (UTC) X-Original-To: hotspot-dev at openjdk.java.net Delivered-To: hotspot-dev at openjdk.java.net Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; Thu, 02 Nov 2017 04:54:08 +0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA24s8KW028461 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 From: Jini George To: "serguei.spitsyn at oracle.com" , "serviceability-dev at openjdk.java.net" , "hotspot-runtime-dev at openjdk.java.net" , References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> Organization: Oracle Corporation Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> Date: Thu, 2 Nov 2017 10:24:03 +0530 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Source-IP: userv0022.oracle.com [156.151.31.74] X-BeenThere: hotspot-dev at openjdk.java.net X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical discussion about the development of the HotSpot virtual machine that's not specific to any particular component List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: hotspot-dev-bounces at openjdk.java.net Sender: hotspot-dev X-purgate-ID: tlsNG-38ea93/1509598502-00006357-E5788386/2/24257462887 X-purgate-type: clean X-purgate-size: 2000 X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de Comment: eleven transport token MTU2LjE1MS4zMS42OQBob3RzcG90LWRldi1ib3VuY2VzQG9wZW5qZGsuamF2YS5uZXQAYXhlbC5zaWViZW5ib3JuQHNhcC5jb20AZDg1NTEwM2IxOTM3MzJhYTJiNzFhMmY0MTMxYjAyMWRmYzllZTI4Ng== X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-SAP-SPAM-Status: clean Return-Path: hotspot-dev-bounces at openjdk.java.net X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 05:15:06.2733 (UTC) X-MS-Exchange-Organization-OriginalClientIPAddress: 194.145.224.110 X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-Network-Message-Id: 72539627-be68-44eb-0652-08d521b0ae31 X-MS-Exchange-Organization-OriginalSize: 7295 X-MS-Exchange-Organization-HygienePolicy: Standard X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=10779.618|UTH=0.001|RST=10779.603|CATCM-McAfeeTxRoutingAgent=0.001|CATCM=0.001|CAT=0.002;2017-11-02T08:14:45.891Z Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the changes >>>> are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> From mikhailo.seledtsov at oracle.com Thu Nov 2 18:49:24 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Thu, 2 Nov 2017 11:49:24 -0700 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> Message-ID: <1b768e77-d8d6-4cc9-c74c-d2c59bc30ce2@oracle.com> Test changes look good, Misha On 10/31/2017 07:43 PM, Ioi Lam wrote: > Hi, > > Here's the new webrev for both the AppCDS implementation and tests. > During internal review of the JEP, we have decided to integrate both > implementation and tests at the same time. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ > > As mentioned before, most of the "diffs" shown in this webrev are the > result of copying the closed source files on top of files of the same > name in the open repo. So in reviewing, instead of focusing on what's > "changed", please focus on the entire content of the new version of > each file. > > Testing: I did an OpenJDK linux-x64 build (without Oracle closed > sources) and all the new appcds tests passed. > > Thanks > > - Ioi > > > On 10/30/17 8:52 AM, Ioi Lam wrote: >> Hi Dmitry, >> >> In the latest JDK 10 repo, is_jrt has been renamed to >> is_modules_image. Please change the code accordingly. >> >> I will post my latest diff soon, with some test cases as well. >> >> Thanks >> >> - Ioi >> >> >> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>> Ioi, >>> >>> I'd tried to apply your patch to latest open JDK10 and >>> the compilation fails with: >>> >>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>> >>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>> >>> Did I miss something? >>> >>> -Dmitry >>> >>> On 13.10.2017 02:48, Ioi Lam wrote: >>>> Hi, >>>> >>>> Please review this change set. >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>> ???? https://bugs.openjdk.java.net/browse/JDK-8188791 >>>> >>>> This is the first step of implementing the following JEP, which moves >>>> AppCDS from >>>> closed repos into the openjdk repo: >>>> >>>> ???? https://bugs.openjdk.java.net/browse/JDK-8185996 >>>> >>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>> repo. >>>> As part >>>> of the open-sourcing effort of JDK 18.3, we will move the source code >>>> into the >>>> open repo. >>>> >>>> In this changeset, the code is moved verbatim as much as possible. The >>>> intention is >>>> only to relocate the sources, not to changing existing behaviors, >>>> and not >>>> to do any sort of refactoring. >>>> >>>> Most of the "diffs" shown in this webrev are the result of copying the >>>> closed source >>>> files on top of files of the same name in the open repo. So in >>>> reviewing, instead of >>>> focusing on what's "changed", it's better to focus on the entire >>>> content >>>> of the new >>>> version of each file. >>>> >>>> The only functional change in this task is that the UseAppCDS flag is >>>> changed from >>>> a "commercial" flag to a regular "product" flag. This is because >>>> "commercial" >>>> flags are not supported by the OpenJDK build. >>>> >>>> Source code refactoring may be desirable, because the old open/closed >>>> source >>>> code structure had introduced some intermediary APIs to connect code >>>> between >>>> the two repos. Such API should be removed in a separate RFE. >>>> >>>> Also, some AppCDS tests are currently in the closed repo. These tests >>>> will be >>>> moved in a separate task. See JDK-8188792 for details. >>>> >>>> All the AppCDS tests (currently still in closed sources) passed with >>>> both Oracle JDK >>>> and OpenJDK. >>>> >>>> Thanks >>>> - Ioi >>> >> > From mikhailo.seledtsov at oracle.com Thu Nov 2 21:03:22 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Thu, 2 Nov 2017 14:03:22 -0700 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message Message-ID: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> Please review this simple change that makes the error message in jtreg-ext/requires/VMProps.java conditional on java property "vmprops.trace.verbose". W/o this condition an error message is printed each time any jtreg test is ran, including non-docker tests. ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 ??? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ ======= Background and details: VMProps.java is called for each test run of jtreg (be it a run containing a singe test or multiple tests). VMProps evaluates the @requires properties. Therefore, in case of this bug, any time a user runs any jtreg VM test on a machine w/o an operational docker engine, this error will pop up. The fix makes printing of this error message conditional, off by default. The message can be turend on via a property vmprops.trace.verbose when detailed diagnostic info is desired. ======== Testing: Local testing: ? W/o docker present: ??? jtreg ????? No error message ??? jtreg -Dvmprops.trace.verbose=true ????? Detailed error message ??? jtreg DockerBasicTest.java ????? Test skipped ? With docker installed but no permissions: ??? jtreg ????? No error message ??? jtreg -Dvmprops.trace.verbose=true ????? Detailed error message ??? jtreg DockerBasicTest.java ????? Test skipped ? With docker installed and operational: ??? jtreg ????? No error message ??? jtreg -Dvmprops.trace.verbose=true ????? No error message ??? jtreg DockerBasicTest.java ????? Test passed Testing via automated distributed test system: ? - tier1 ? - docker tests ? In progress Thank you, Misha From david.holmes at oracle.com Thu Nov 2 23:47:47 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 3 Nov 2017 09:47:47 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> Message-ID: Hi Misha, I don't see anything else in VMProps that writes anything out so to me the fix is to delete the System.err.println completely. That said I did not look at this in the past but I feel the whole docker test here is not really appropriate/suitable to always be run on linux-x64. It has to exec another process and wait up to 10 seconds to get a result! Can't this actually be done via a WhiteBox check once Bob's container support updates are in place? Thanks, David On 3/11/2017 7:03 AM, mikhailo wrote: > Please review this simple change that makes the error message in > jtreg-ext/requires/VMProps.java conditional > on java property "vmprops.trace.verbose". W/o this condition an error > message is printed each time any jtreg > test is ran, including non-docker tests. > > ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 > ??? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ > > > > ======= Background and details: > VMProps.java is called for each test run of jtreg (be it a run > containing a singe test or multiple tests). > VMProps evaluates the @requires properties. Therefore, in case of this > bug, any time a user runs any jtreg VM test > on a machine w/o an operational docker engine, this error will pop up. > The fix makes printing of this error message conditional, off by > default. The message can be turend on > via a property vmprops.trace.verbose when detailed diagnostic info is > desired. > > > > ======== Testing: > Local testing: > ? W/o docker present: > ??? jtreg > ????? No error message > ??? jtreg -Dvmprops.trace.verbose=true > ????? Detailed error message > ??? jtreg DockerBasicTest.java > ????? Test skipped > > ? With docker installed but no permissions: > ??? jtreg > ????? No error message > ??? jtreg -Dvmprops.trace.verbose=true > ????? Detailed error message > ??? jtreg DockerBasicTest.java > ????? Test skipped > > ? With docker installed and operational: > ??? jtreg > ????? No error message > ??? jtreg -Dvmprops.trace.verbose=true > ????? No error message > ??? jtreg DockerBasicTest.java > ????? Test passed > > Testing via automated distributed test system: > ? - tier1 > ? - docker tests > ? In progress > > > Thank you, > Misha > From calvin.cheung at oracle.com Thu Nov 2 23:57:54 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 02 Nov 2017 16:57:54 -0700 Subject: RFR(XS): 8187347: Do not abort CDS archive creation when some classes are unverifiable Message-ID: <59FBB102.2040403@oracle.com> This change is for deprecating the IgnoreUnverifiableClassesDuringDump vm option in JDK10 and set its default value to true. JBS: https://bugs.openjdk.java.net/browse/JDK-8187347 webrev: http://cr.openjdk.java.net/~ccheung/8187347/webrev.00/ Approved CSR: https://bugs.openjdk.java.net/browse/JDK-8187348 Testing: hs-tier1, hs-tier2 on linux-x64, macosx, sparcv9, windows-x64. thanks, Calvin From coleen.phillimore at oracle.com Fri Nov 3 00:10:50 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 2 Nov 2017 20:10:50 -0400 Subject: RFR (xs) 8190617: test/jdk/sun/tools/jhsdb/BasicLauncherTest.java fails Message-ID: Summary: change type expected by SA for PerfMemory::_initialize to int. open webrev at http://cr.openjdk.java.net/~coleenp/8190617.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8190617 Tested with failing test case and mach5 tier1, tier2, all platforms in progress. Thanks, Coleen From david.holmes at oracle.com Fri Nov 3 00:13:00 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 3 Nov 2017 10:13:00 +1000 Subject: RFR(XS): 8187347: Do not abort CDS archive creation when some classes are unverifiable In-Reply-To: <59FBB102.2040403@oracle.com> References: <59FBB102.2040403@oracle.com> Message-ID: <5fd7fa56-b37c-44a5-3d94-434071a19f18@oracle.com> Looks good. Thanks, David On 3/11/2017 9:57 AM, Calvin Cheung wrote: > This change is for deprecating the IgnoreUnverifiableClassesDuringDump > vm option in JDK10 and set its default value to true. > > JBS: > ??? https://bugs.openjdk.java.net/browse/JDK-8187347 > > webrev: > ??? http://cr.openjdk.java.net/~ccheung/8187347/webrev.00/ > > Approved CSR: > ??? https://bugs.openjdk.java.net/browse/JDK-8187348 > > Testing: > ??? hs-tier1, hs-tier2 on linux-x64, macosx, sparcv9, windows-x64. > > thanks, > Calvin From david.holmes at oracle.com Fri Nov 3 00:14:36 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 3 Nov 2017 10:14:36 +1000 Subject: RFR (xs) 8190617: test/jdk/sun/tools/jhsdb/BasicLauncherTest.java fails In-Reply-To: References: Message-ID: Looks good. Thanks, David On 3/11/2017 10:10 AM, coleen.phillimore at oracle.com wrote: > Summary: change type expected by SA for PerfMemory::_initialize to int. > > open webrev at http://cr.openjdk.java.net/~coleenp/8190617.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8190617 > > Tested with failing test case and mach5 tier1, tier2, all platforms in > progress. > > Thanks, > Coleen From mikhailo.seledtsov at oracle.com Fri Nov 3 00:22:11 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Thu, 2 Nov 2017 17:22:11 -0700 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> Message-ID: <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> Hi David, ? Thank you for taking a look at this change. On 11/02/2017 04:47 PM, David Holmes wrote: > Hi Misha, > > I don't see anything else in VMProps that writes anything out so to me > the fix is to delete the System.err.println completely. Removal is an option, and seems a simpler one. However, we will loose the diagnostic information. On the other hand, if an issue arises on a specific host or system, I can patch this file with extra trace statements, and diagnose the failure that way. If you think removing the print statement is a better option, and I do not hear objections or other opinions, I can just remove the print statement. > That said I did not look at this in the past but I feel the whole > docker test here is not really appropriate/suitable to always be run > on linux-x64. It has to exec another process and wait up to 10 seconds > to get a result! Can't this actually be done via a WhiteBox check once > Bob's container support updates are in place? Actually, WhiteBox is not necessary for such check. The ability to run docker on a given host/node can be determined in a body of a jtreg test, via a test utility. In fact, that was my original intent. When this change was presented for a discussion, this aspect was debated and reviewers have reached a consensus to do such check in VMProps.java. We mainly discussed within VM SQE team. The main argument was that using @requires is a more proper way of "skipping" the test(s) on unsupported environments, rather then checking this in the body of the test and then returning. When using @requires, the test execution system will report the test as "not run". When skipping the test using "return" from the body of a test, the execution system will report test as "ran" and passed; jtreg does not yet support the "skipped" state. Unfortunately, the way our "@requires" mechanism works is that it runs each time prior to starting a test run, even if the test being run is not concerned with a specific property. It was my concern that this could be a burden on the test run startup, but I eventually agreed to the opinion of other reviewers. If you have a strong opinion or concern about this, let me know. However, I am not sure what is the best process to use to overturn a decision on a prior review or design discussion. I guess one option is to file a bug or an RFE, and bring it up for a discussion. For this change under review, if I do not hear opinions from other reviewers, I can simply delete the print statement. Thank you, Misha > > Thanks, > David > > On 3/11/2017 7:03 AM, mikhailo wrote: >> Please review this simple change that makes the error message in >> jtreg-ext/requires/VMProps.java conditional >> on java property "vmprops.trace.verbose". W/o this condition an error >> message is printed each time any jtreg >> test is ran, including non-docker tests. >> >> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >> >> >> >> ======= Background and details: >> VMProps.java is called for each test run of jtreg (be it a run >> containing a singe test or multiple tests). >> VMProps evaluates the @requires properties. Therefore, in case of >> this bug, any time a user runs any jtreg VM test >> on a machine w/o an operational docker engine, this error will pop up. >> The fix makes printing of this error message conditional, off by >> default. The message can be turend on >> via a property vmprops.trace.verbose when detailed diagnostic info is >> desired. >> >> >> >> ======== Testing: >> Local testing: >> ?? W/o docker present: >> ???? jtreg >> ?????? No error message >> ???? jtreg -Dvmprops.trace.verbose=true >> ?????? Detailed error message >> ???? jtreg DockerBasicTest.java >> ?????? Test skipped >> >> ?? With docker installed but no permissions: >> ???? jtreg >> ?????? No error message >> ???? jtreg -Dvmprops.trace.verbose=true >> ?????? Detailed error message >> ???? jtreg DockerBasicTest.java >> ?????? Test skipped >> >> ?? With docker installed and operational: >> ???? jtreg >> ?????? No error message >> ???? jtreg -Dvmprops.trace.verbose=true >> ?????? No error message >> ???? jtreg DockerBasicTest.java >> ?????? Test passed >> >> Testing via automated distributed test system: >> ?? - tier1 >> ?? - docker tests >> ?? In progress >> >> >> Thank you, >> Misha >> From jiangli.zhou at oracle.com Fri Nov 3 00:20:35 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 2 Nov 2017 17:20:35 -0700 Subject: RFR(XS): 8187347: Do not abort CDS archive creation when some classes are unverifiable In-Reply-To: <5fd7fa56-b37c-44a5-3d94-434071a19f18@oracle.com> References: <59FBB102.2040403@oracle.com> <5fd7fa56-b37c-44a5-3d94-434071a19f18@oracle.com> Message-ID: <2DD5254D-1BB7-418D-9329-BB567227F130@oracle.com> +1. Thanks, Jiangli > On Nov 2, 2017, at 5:13 PM, David Holmes wrote: > > Looks good. > > Thanks, > David > > On 3/11/2017 9:57 AM, Calvin Cheung wrote: >> This change is for deprecating the IgnoreUnverifiableClassesDuringDump vm option in JDK10 and set its default value to true. >> JBS: >> https://bugs.openjdk.java.net/browse/JDK-8187347 >> webrev: >> http://cr.openjdk.java.net/~ccheung/8187347/webrev.00/ >> Approved CSR: >> https://bugs.openjdk.java.net/browse/JDK-8187348 >> Testing: >> hs-tier1, hs-tier2 on linux-x64, macosx, sparcv9, windows-x64. >> thanks, >> Calvin From david.holmes at oracle.com Fri Nov 3 01:02:12 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 3 Nov 2017 11:02:12 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> Message-ID: Hi Misha, On 3/11/2017 10:22 AM, mikhailo wrote: > Hi David, > > ? Thank you for taking a look at this change. > > > On 11/02/2017 04:47 PM, David Holmes wrote: >> Hi Misha, >> >> I don't see anything else in VMProps that writes anything out so to me >> the fix is to delete the System.err.println completely. > Removal is an option, and seems a simpler one. However, we will loose > the diagnostic information. > On the other hand, if an issue arises on a specific host or system, I > can patch this file with extra trace statements, and diagnose the > failure that way. > If you think removing the print statement is a better option, and I do > not hear objections or other opinions, I can just remove the print > statement. Delete please :) >> That said I did not look at this in the past but I feel the whole >> docker test here is not really appropriate/suitable to always be run >> on linux-x64. It has to exec another process and wait up to 10 seconds >> to get a result! Can't this actually be done via a WhiteBox check once >> Bob's container support updates are in place? > Actually, WhiteBox is not necessary for such check. The ability to run > docker on a given host/node can be determined in a body of a jtreg test, > via a test utility. In fact, that was my original intent. When this > change was presented for a discussion, this aspect was debated and > reviewers have reached a consensus to do such check in VMProps.java. We > mainly discussed within VM SQE team. The main argument was that using > @requires is a more proper way of "skipping" the test(s) on unsupported > environments, rather then checking this in the body of the test and then > returning. When using @requires, the test execution system will report > the test as "not run". When skipping the test using "return" from the > body of a test, the execution system will report test as "ran" and > passed; jtreg does not yet support the "skipped" state. > > Unfortunately, the way our "@requires" mechanism works is that it runs > each time prior to starting a test run, even if the test being run is > not concerned with a specific property. It was my concern that this > could be a burden on the test run startup, but I eventually agreed to > the opinion of other reviewers. > > If you have a strong opinion or concern about this, let me know. > However, I am not sure what is the best process to use to overturn a > decision on a prior review or design discussion. I guess one option is > to file a bug or an RFE, and bring it up for a discussion. Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using WhiteBox connected to Bob's new container support code is better than exec'ing a separate process (which I'm not even sure what it does). > > For this change under review, if I do not hear opinions from other > reviewers, I can simply delete the print statement. Sounds good to me. Thanks, David > > Thank you, > Misha >> >> Thanks, >> David >> >> On 3/11/2017 7:03 AM, mikhailo wrote: >>> Please review this simple change that makes the error message in >>> jtreg-ext/requires/VMProps.java conditional >>> on java property "vmprops.trace.verbose". W/o this condition an error >>> message is printed each time any jtreg >>> test is ran, including non-docker tests. >>> >>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>> >>> >>> >>> ======= Background and details: >>> VMProps.java is called for each test run of jtreg (be it a run >>> containing a singe test or multiple tests). >>> VMProps evaluates the @requires properties. Therefore, in case of >>> this bug, any time a user runs any jtreg VM test >>> on a machine w/o an operational docker engine, this error will pop up. >>> The fix makes printing of this error message conditional, off by >>> default. The message can be turend on >>> via a property vmprops.trace.verbose when detailed diagnostic info is >>> desired. >>> >>> >>> >>> ======== Testing: >>> Local testing: >>> ?? W/o docker present: >>> ???? jtreg >>> ?????? No error message >>> ???? jtreg -Dvmprops.trace.verbose=true >>> ?????? Detailed error message >>> ???? jtreg DockerBasicTest.java >>> ?????? Test skipped >>> >>> ?? With docker installed but no permissions: >>> ???? jtreg >>> ?????? No error message >>> ???? jtreg -Dvmprops.trace.verbose=true >>> ?????? Detailed error message >>> ???? jtreg DockerBasicTest.java >>> ?????? Test skipped >>> >>> ?? With docker installed and operational: >>> ???? jtreg >>> ?????? No error message >>> ???? jtreg -Dvmprops.trace.verbose=true >>> ?????? No error message >>> ???? jtreg DockerBasicTest.java >>> ?????? Test passed >>> >>> Testing via automated distributed test system: >>> ?? - tier1 >>> ?? - docker tests >>> ?? In progress >>> >>> >>> Thank you, >>> Misha >>> > From calvin.cheung at oracle.com Fri Nov 3 01:27:36 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 02 Nov 2017 18:27:36 -0700 Subject: RFR(XS): 8187347: Do not abort CDS archive creation when some classes are unverifiable In-Reply-To: <2DD5254D-1BB7-418D-9329-BB567227F130@oracle.com> References: <59FBB102.2040403@oracle.com> <5fd7fa56-b37c-44a5-3d94-434071a19f18@oracle.com> <2DD5254D-1BB7-418D-9329-BB567227F130@oracle.com> Message-ID: <59FBC608.5050506@oracle.com> David, Jiangli, Thanks for your quick review. Calvin On 11/2/17, 5:20 PM, Jiangli Zhou wrote: > +1. > > Thanks, > Jiangli > >> On Nov 2, 2017, at 5:13 PM, David Holmes wrote: >> >> Looks good. >> >> Thanks, >> David >> >> On 3/11/2017 9:57 AM, Calvin Cheung wrote: >>> This change is for deprecating the IgnoreUnverifiableClassesDuringDump vm option in JDK10 and set its default value to true. >>> JBS: >>> https://bugs.openjdk.java.net/browse/JDK-8187347 >>> webrev: >>> http://cr.openjdk.java.net/~ccheung/8187347/webrev.00/ >>> Approved CSR: >>> https://bugs.openjdk.java.net/browse/JDK-8187348 >>> Testing: >>> hs-tier1, hs-tier2 on linux-x64, macosx, sparcv9, windows-x64. >>> thanks, >>> Calvin From dean.long at oracle.com Fri Nov 3 02:02:59 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Thu, 2 Nov 2017 19:02:59 -0700 Subject: 8u144 hotspot fails to reach safepoint due to compiler thread - VM frozen In-Reply-To: References: Message-ID: <5f39dfcf-64f9-7a9f-1ed3-c810eb6f1985@oracle.com> From a quick search, this sounds similar to 8085965.? I wonder if another circularity in the siblings list has been created somehow.? Can you check for that with gdb? dl On 10/31/17 11:08 AM, Vitaly Davidovich wrote: > Hi guys, > > I have some colleagues who appear to be running into > https://bugs.openjdk.java.net/browse/JDK-8059128 on Oracle JDK 8u144 > (Linux, x86-64).? Naturally, there's no reproducer but they've seen > this happen several times in the last couple of months. > > The symptom is the JVM becomes unresponsive - the application is not > servicing any traffic, and jstack doesn't work without the force > option. ?jstack output (with native frames) captured some time apart > shows the compiler thread either in Parse::do_all_blocks -> > do_one_block -> do_one_bytecode -> ... > InstanceKlass::has_finalizable_subclass -> > Dependencies::find_finalizable_subclass or > ... Dependencies::has_finalizable_subclass() -> Klass::next_sibling() > > I see that 8059128 was closed as Incomplete, but it does look like > there's a real issue here.? Has anyone looked into this further or has > any new thoughts/ideas? > > My understanding is the working theory is it's related to some data > race between class unloading and the compiler thread observing an > inconsistent (corrupt?) type hierarchy.? I see > https://bugs.openjdk.java.net/browse/JDK-8114823 is also noted as > possibly related - the app we're having trouble with is using G1, but > class unloading isn't disabled of course.? Is there some work around > to reduce the likelihood of having the compiler thread and GC cross > paths like this? > > Let me know if you need more info. > > Thanks From serguei.spitsyn at oracle.com Fri Nov 3 03:21:10 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 2 Nov 2017 20:21:10 -0700 Subject: RFR (xs) 8190617: test/jdk/sun/tools/jhsdb/BasicLauncherTest.java fails In-Reply-To: References: Message-ID: <220d13aa-abb9-07d3-03b1-64e289b060ea@oracle.com> Looks good ++ Thanks, Serguei On 11/2/17 17:14, David Holmes wrote: > Looks good. > > Thanks, > David > > On 3/11/2017 10:10 AM, coleen.phillimore at oracle.com wrote: >> Summary: change type expected by SA for PerfMemory::_initialize to int. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8190617.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8190617 >> >> Tested with failing test case and mach5 tier1, tier2, all platforms >> in progress. >> >> Thanks, >> Coleen From kishor.kharbas at intel.com Fri Nov 3 08:55:04 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Fri, 3 Nov 2017 08:55:04 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: Message-ID: Hi Sangheon, Thanks for the review and comments. Here is an updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 In addition to your suggested corrections, I added code to set Linux core dump filter ensuring Heap is dumped correctly when this feature is used. This is follow-up to Jini George's comment (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap-allocation-on-alternative-memory-devices-td300109.html#a300450). (some reply is inline) Thanks Kishor From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Wednesday, November 1, 2017 10:53 PM To: Kharbas, Kishor ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Cc: Thomas Schatzl Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 10/31/2017 04:53 PM, Kharbas, Kishor wrote: Greetings, I am continuing the earlier discussion [1] with a revised issue number representing the implementation of the JEP [2]. I have addressed the remaining unaddressed comments (copied below) in these webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11_to_10/ Also, I would appreciate a review of the CSR for the new flag at https://bugs.openjdk.java.net/browse/JDK-8190309. CSR: Reviewed. > > - in that same thread there has also been the question about the > > need > > for blocking the signals for the process that has gone unanswered. > > Removed the blocking of signals. > > - Arguments::finalize_vm_init_args: maybe at the place where the > > change adds the warning/info message about NUMA support being > > specific > > to the file system; also add the same warning about UseLargePages > > that > > is located elsewhere. > > > > Like "UseXXXX may not be supported in some specific file system > > implementations." - or just ignore as Andrew said in the other > > email thread. > > > > Note that I am not sure that the selected place is the correct > > place > > for such warning about incompatible flags (and maybe > > UseNUMA/UseLargePages should be forcibly disabled here? But then > > again, it does not hurt just having it enabled?). > > > > I will let the runtime team comment as a lot of that argument > > processing changed in jdk9. > > > > Maybe Arguments::check_vm_args_consistency() is better. > > > > There is similar code in Arguments::adjust_after_os()... > > 1. Moved the check from finalize_vm_init_args() to check_vm_args_consistency() 2. When UseNUMA flag is true, adaptive logical group chunk resizing is used for Numa aware collectors such as ParallelOld. Just like UseNUMA is disabled in os::inti_2() in Linux when UseLargePages is set, it will be a good idea to disable UseNUMA when the new option is used. 3. I think its ok to leave UseNUMAInterleaving ON as it can be used for other memory areas. 4. For the same reason I also do not disable UseLargePages. 5. About handling UseLargePages, I thought of handling it the same way as when reserve_memory_special() fails to allocate large pages, i.e. generating a log_debug message. // failed; try to reserve regular memory below if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) || !FLAG_IS_DEFAULT(LargePageSizeInBytes))) { log_debug(gc, heap, coops)("Reserve regular memory without large pages"); > > - here I may probably be speaking wrongly on behalf of the > > runtime > > team, but os.hpp, as an abstraction of the OS should probably not > > have > > platform specific ifdefs in it. > > and > > > - You removed os::attempt_reserve_memory_at() from os.cpp and > > > split > > > into os_posix.cpp and os_windows.cpp. > > > But I think you should remain os::attempt_reserve_memory_at() > > > at > > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > > for > > > pd_xxx. Of cource move MemTracker function call as well. In the updated webrev, I move the implementation from os_posix.cpp and os_window.cpp to respective pd_xxx functions. No AIX specific ifdef is required now. Thank you for all changes. I have minor nits now: ============================================== - os***.cpp has some function names which include *dax*. I would prefer not to include it. As you know, Thomas also pointed it. [Kharbas, Kishor] Done. ============================================== src/hotspot/share/runtime/arguments.cpp 2537 if (!FLAG_IS_DEFAULT(UseNUMAInterleaving) || !FLAG_IS_DEFAULT(UseNUMA)) { - Don't we need to check these 2 flags' value to be true? i.e. if user sets to false, below message will be printed. [Kharbas, Kishor] That's correct. I changed it such that you check the flag is true and it's not default. In future if these flags become 'true' by default, we may not want to print warning. ============================================== test/hotspot/jtreg/gc/TestAllocateHeapAt.java - On other discussion, I mentioned to test only for Windows and Linux as the JEP described only about those 2. But without *dax* function names, it seems like not filtering OS seems okay too. 2 * Copyright (c) 2017, xxx Oracle and/or its affiliates. All rights reserved. - Please remove 'xxx '. 47 Collections.addAll(vmOpts, new String[] {"-XX:AllocateHeapAt="+test_dir, - Add spaces. '+test_dir' -> ' + test_dir' 49 "-Xmx50m", 50 "-Xms50m", - You said there were no special reason for 200m(heap size of webrev.10) on other discussion. I would recommend 32m. [Kharbas, Kishor] All done. FYI, I ran hs-tier1 and hs-tier2 with your patch and the result is good. i.e. no new failures. [Kharbas, Kishor] That's great. Thanks! Thanks, Sangheon Thank you and looking forward for your feedback. - Kishor [1] http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-October/020682.html [2] https://bugs.openjdk.java.net/browse/JDK-8171181 From sharath.ballal at oracle.com Fri Nov 3 10:14:50 2017 From: sharath.ballal at oracle.com (Sharath Ballal) Date: Fri, 3 Nov 2017 03:14:50 -0700 (PDT) Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> Message-ID: Hi Jini, You have appended 'Field' for most of the SA variables. Example: private static CIntegerField pcOffsetField; pcOffsetField = type.getCIntegerField("_pc_offset"); However that is not the case in private static long MinChunkSizeInBytes; MinChunkSizeInBytes = (type.getCIntegerField("_min_chunk_size_in_bytes")).getValue(); You may want to follow the same convention here. Rest of the changes look ok. Thanks, Sharath (not a reviewer) -----Original Message----- From: Jini George Sent: Thursday, November 02, 2017 10:24 AM To: Serguei Spitsyn; serviceability-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-dev at openjdk.java.net Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the >>>> changes are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> X-sender: X-Receiver: X-CreatedBy: MSExchange15 X-HeloDomain: mx.expurgate.net X-ExtendedProps: BQBjAAoAqbJ+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAGcrAAB2AAAABQBkAA8ABAAAAEVkZ2U= X-Source: SMTP:External Receive Connector X-SourceIPAddress: 195.190.135.10 X-EndOfInjectedXHeaders: 7733 Received: from mx.expurgate.net (195.190.135.10) by smtpgw03.sap-ag.de (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov 2017 05:54:39 +0100 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1eA7WZ-000EZw-D4 for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:39 +0100 Received: from [156.151.31.69] (helo=ucsinet41.oracle.com) by mxtls.expurgate.net with ESMTPS (eXpurgate 4.2.1-3) (envelope-from ) id 59faa50b-7c01-c0a8033409dd-9c971f454783-3 for ; Thu, 02 Nov 2017 05:54:38 +0100 Received: from aojmv0009 (unknown [137.254.59.6]) by ucsinet41.oracle.com with smtp id 4eb7_00c7_3ecf1c93_28fd_4a9f_97f0_f31de9ddb295; Thu, 02 Nov 2017 04:54:17 +0000 Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) by aojmv0009 (Postfix) with ESMTP id 15A99229CEC; Thu, 2 Nov 2017 04:57:26 +0000 (UTC) X-Original-To: serviceability-dev at openjdk.java.net Delivered-To: serviceability-dev at openjdk.java.net Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; Thu, 02 Nov 2017 04:54:08 +0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA24s8KW028461 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 From: Jini George To: "serguei.spitsyn at oracle.com" , "serviceability-dev at openjdk.java.net" , "hotspot-runtime-dev at openjdk.java.net" , References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> Organization: Oracle Corporation Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> Date: Thu, 2 Nov 2017 10:24:03 +0530 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Source-IP: userv0022.oracle.com [156.151.31.74] X-BeenThere: serviceability-dev at openjdk.java.net X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Technical discussion about the development of serviceability technologies \(debugging, profiling, monitoring, and management\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: serviceability-dev-bounces at openjdk.java.net Sender: serviceability-dev X-purgate-ID: tlsNG-9b91ac/1509598479-00007C01-219CB02A/0/0 X-purgate-type: clean X-purgate-size: 2000 X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de Comment: eleven transport token MTU2LjE1MS4zMS42OQBzZXJ2aWNlYWJpbGl0eS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADgwYzQ2ZTI0N2Q2MTBlYjNiNTQ5MmE5YWE2NzJjNzJjN2NiNjcwNDQ= X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-SAP-SPAM-Status: clean Return-Path: serviceability-dev-bounces at openjdk.java.net X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:39.6785 (UTC) X-MS-Exchange-Organization-OriginalClientIPAddress: 195.190.135.10 X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-Network-Message-Id: 8fba1fcd-01cd-4a67-1ee6-08d521add313 X-MS-Exchange-Organization-OriginalSize: 7400 X-MS-Exchange-Organization-HygienePolicy: Standard X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11997.291|UTH=0.001|RST=11997.244|QS=0.029|CATCM-McAfeeTxRoutingAgent=0.011|CATCM=0.011|CAT=0.012;2017-11-02T08:14:36.969Z Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the >>>> changes are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> X-sender: X-Receiver: X-CreatedBy: MSExchange15 X-HeloDomain: mx.expurgate.net X-ExtendedProps: BQBjAAoAALN+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAIMrAAB2AAAABQBkAA8ABAAAAEVkZ2U= X-Source: SMTP:External Receive Connector X-SourceIPAddress: 194.145.224.110 X-EndOfInjectedXHeaders: 7704 Received: from mx.expurgate.net (194.145.224.110) by smtpgw03.sap-ag.de (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov 2017 05:54:46 +0100 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1eA7Wg-0004R0-FP for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:46 +0100 Received: from [141.146.126.229] (helo=acsinet41.oracle.com) by mxtls.expurgate.net with ESMTPS (eXpurgate 4.3.1) (envelope-from ) id 59faa50f-65eb-c0a8029b09dd-8d927ee53db9-3 for ; Thu, 02 Nov 2017 05:54:45 +0100 Received: from aojmv0009 (aojmv0009.oracle.com [137.254.59.6]) by acsinet41.oracle.com with smtp id 6461_15f0_154b24ba_5aa0_408c_88bd_1cdb37c9c8f4; Thu, 02 Nov 2017 04:54:31 +0000 Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) by aojmv0009 (Postfix) with ESMTP id 3B280229CF4; Thu, 2 Nov 2017 04:57:26 +0000 (UTC) X-Original-To: hotspot-runtime-dev at openjdk.java.net Delivered-To: hotspot-runtime-dev at openjdk.java.net Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; Thu, 02 Nov 2017 04:54:08 +0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA24s8KW028461 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 From: Jini George To: "serguei.spitsyn at oracle.com" , "serviceability-dev at openjdk.java.net" , "hotspot-runtime-dev at openjdk.java.net" , References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> Organization: Oracle Corporation Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> Date: Thu, 2 Nov 2017 10:24:03 +0530 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Source-IP: userv0022.oracle.com [156.151.31.74] X-BeenThere: hotspot-runtime-dev at openjdk.java.net X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical discussion about the development of the HotSpot runtime subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: hotspot-runtime-dev-bounces at openjdk.java.net Sender: hotspot-runtime-dev X-purgate-ID: tlsNG-81024b/1509598486-000065EB-186BA960/0/0 X-purgate-type: clean X-purgate-size: 2000 X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de Comment: eleven transport token MTQxLjE0Ni4xMjYuMjI5AGhvdHNwb3QtcnVudGltZS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADE0ZjI2NjVmYTY4ZGRiOWMwNGJiMGU0MmE2Njk2M2ZiNWQxNzAyOGI= X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-SAP-SPAM-Status: clean Return-Path: hotspot-runtime-dev-bounces at openjdk.java.net X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:46.7566 (UTC) X-MS-Exchange-Organization-OriginalClientIPAddress: 194.145.224.110 X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp X-MS-Exchange-Organization-Network-Message-Id: 9365fa06-08de-47df-2865-08d521add74b X-MS-Exchange-Organization-OriginalSize: 7380 X-MS-Exchange-Organization-HygienePolicy: Standard X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11990.260|UTH=0.001|RST=11990.260|CATCM-McAfeeTxRoutingAgent=0.004|CATCM=0.005|CAT=0.006;2017-11-02T08:14:37.016Z Could I please get one more review done for this ? Thanks, Jini. On 10/27/2017 9:19 PM, Jini George wrote: > Thank you very much, Serguei. > > -Jini. > > On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> The fix looks good to me. >> >> Thanks, >> Serguei >> >> >> On 10/24/17 00:31, Jini George wrote: >>> Adding hotspot-dev too. >>> >>> Thanks, >>> Jini. >>> >>> On 10/24/2017 12:05 PM, Jini George wrote: >>>> Hello, >>>> >>>> As a part of SA next, I am working on writing a test case which >>>> compares the fields and the types of the fields of the SA java >>>> classes with the corresponding entries in the vmStructs tables. >>>> This, to some extent, would help in preventing errors in SA due to >>>> the changes in hotspot. As a precursor to this, I am in the process >>>> of making some cleanup related changes (mostly in SA). I plan to >>>> have the changes done in parts. For this webrev, most of the >>>> changes are for: >>>> >>>> 1. Avoiding having some values being redefined in SA. Instead have >>>> those exported through vmStructs, and read it in SA. >>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>> CompactibleFreeListSpace::IndexSetSize) >>>> >>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>> value gets altered in hotspot and the corresponding modification >>>> gets missed out in SA. >>>> >>>> 2. To remove some unused code (JNIid.java). >>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>> names, so that the comparison of the fields become easier. Most of >>>> the changes belong to this group. >>>> >>>> Could I please get reviews done for these precursor changes ? >>>> >>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>> >>>> Thank you, >>>> Jini. >>>> >> From coleen.phillimore at oracle.com Fri Nov 3 11:19:40 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 3 Nov 2017 07:19:40 -0400 Subject: RFR (xs) 8190617: test/jdk/sun/tools/jhsdb/BasicLauncherTest.java fails In-Reply-To: <220d13aa-abb9-07d3-03b1-64e289b060ea@oracle.com> References: <220d13aa-abb9-07d3-03b1-64e289b060ea@oracle.com> Message-ID: Thanks Serguei and David. Coleen On 11/2/17 11:21 PM, serguei.spitsyn at oracle.com wrote: > Looks good ++ > > Thanks, > Serguei > > On 11/2/17 17:14, David Holmes wrote: >> Looks good. >> >> Thanks, >> David >> >> On 3/11/2017 10:10 AM, coleen.phillimore at oracle.com wrote: >>> Summary: change type expected by SA for PerfMemory::_initialize to int. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/8190617.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8190617 >>> >>> Tested with failing test case and mach5 tier1, tier2, all platforms >>> in progress. >>> >>> Thanks, >>> Coleen > From vitalyd at gmail.com Fri Nov 3 13:05:04 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Fri, 03 Nov 2017 13:05:04 +0000 Subject: 8u144 hotspot fails to reach safepoint due to compiler thread - VM frozen In-Reply-To: <5f39dfcf-64f9-7a9f-1ed3-c810eb6f1985@oracle.com> References: <5f39dfcf-64f9-7a9f-1ed3-c810eb6f1985@oracle.com> Message-ID: On Thu, Nov 2, 2017 at 10:03 PM wrote: > From a quick search, this sounds similar to 8085965. I wonder if > another circularity in the siblings list has been created somehow. Can > you check for that with gdb? Hi Dean, Yes it does sound like that bug, although that?s marked as fixed in 8u72. I guess the implication is that it?s not actually fixed or this is a different codepath/circumstance that ends up in the same blackhole. I?ll see if my colleagues can dig around in the core file. It?s a product build so not sure how easy it will be to make sense of it without debug symbols. Thanks > > > dl > > > On 10/31/17 11:08 AM, Vitaly Davidovich wrote: > > Hi guys, > > > > I have some colleagues who appear to be running into > > https://bugs.openjdk.java.net/browse/JDK-8059128 on Oracle JDK 8u144 > > (Linux, x86-64). Naturally, there's no reproducer but they've seen > > this happen several times in the last couple of months. > > > > The symptom is the JVM becomes unresponsive - the application is not > > servicing any traffic, and jstack doesn't work without the force > > option. jstack output (with native frames) captured some time apart > > shows the compiler thread either in Parse::do_all_blocks -> > > do_one_block -> do_one_bytecode -> ... > > InstanceKlass::has_finalizable_subclass -> > > Dependencies::find_finalizable_subclass or > > ... Dependencies::has_finalizable_subclass() -> Klass::next_sibling() > > > > I see that 8059128 was closed as Incomplete, but it does look like > > there's a real issue here. Has anyone looked into this further or has > > any new thoughts/ideas? > > > > My understanding is the working theory is it's related to some data > > race between class unloading and the compiler thread observing an > > inconsistent (corrupt?) type hierarchy. I see > > https://bugs.openjdk.java.net/browse/JDK-8114823 is also noted as > > possibly related - the app we're having trouble with is using G1, but > > class unloading isn't disabled of course. Is there some work around > > to reduce the likelihood of having the compiler thread and GC cross > > paths like this? > > > > Let me know if you need more info. > > > > Thanks > > -- Sent from my phone From thomas.schatzl at oracle.com Fri Nov 3 14:31:17 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 03 Nov 2017 15:31:17 +0100 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: Message-ID: <1509719477.3207.10.camel@oracle.com> Hi, On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: > Hi Sangheon, > ? > Thanks for the review and comments. Here is an updated webrev- > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 > ? > In addition to your suggested corrections, I added code to set Linux > core dump filter ensuring Heap is dumped correctly when this feature > is used. This is follow-up to Jini George?s comment > (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- > allocation-on-alternative-memory-devices-td300109.html#a300450). Some minor nits: - os_posix.cpp:300: please move the else next to the brace - arguments.cpp:4624: please add a space between "if" and the bracket I do not need to see a new webrev for these changes. Looks good. Thanks, Thomas From dean.long at oracle.com Fri Nov 3 16:56:42 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Fri, 3 Nov 2017 09:56:42 -0700 Subject: 8u144 hotspot fails to reach safepoint due to compiler thread - VM frozen In-Reply-To: References: <5f39dfcf-64f9-7a9f-1ed3-c810eb6f1985@oracle.com> Message-ID: <6e0b34c7-177c-4148-20b8-d717a1bab55d@oracle.com> On 11/3/17 6:05 AM, Vitaly Davidovich wrote: > > On Thu, Nov 2, 2017 at 10:03 PM > wrote: > > ?From a quick search, this sounds similar to 8085965.? I wonder if > another circularity in the siblings list has been created > somehow.? Can > you check for that with gdb? > > Hi Dean, > > Yes it does sound like that bug, although that?s marked as fixed in > 8u72.? I guess the implication is that it?s not actually fixed or this > is a different codepath/circumstance that ends up in the same blackhole. > > I?ll see if my colleagues can dig around in the core file.? It?s a > product build so not sure how easy it will be to make sense of it > without debug symbols. > I'm not an expert with jhsdb/clhsdb/hsdb, but I believe it can attach to a core file and give you some information even with a product build. dl > Thanks > > > > dl > > > On 10/31/17 11:08 AM, Vitaly Davidovich wrote: > > Hi guys, > > > > I have some colleagues who appear to be running into > > https://bugs.openjdk.java.net/browse/JDK-8059128 on Oracle JDK 8u144 > > (Linux, x86-64).? Naturally, there's no reproducer but they've seen > > this happen several times in the last couple of months. > > > > The symptom is the JVM becomes unresponsive - the application is not > > servicing any traffic, and jstack doesn't work without the force > > option. ?jstack output (with native frames) captured some time apart > > shows the compiler thread either in Parse::do_all_blocks -> > > do_one_block -> do_one_bytecode -> ... > > InstanceKlass::has_finalizable_subclass -> > > Dependencies::find_finalizable_subclass or one> > > ... Dependencies::has_finalizable_subclass() -> > Klass::next_sibling() > > > > I see that 8059128 was closed as Incomplete, but it does look like > > there's a real issue here.? Has anyone looked into this further > or has > > any new thoughts/ideas? > > > > My understanding is the working theory is it's related to some data > > race between class unloading and the compiler thread observing an > > inconsistent (corrupt?) type hierarchy.? I see > > https://bugs.openjdk.java.net/browse/JDK-8114823 is also noted as > > possibly related - the app we're having trouble with is using > G1, but > > class unloading isn't disabled of course.? Is there some work around > > to reduce the likelihood of having the compiler thread and GC cross > > paths like this? > > > > Let me know if you need more info. > > > > Thanks > > -- > Sent from my phone From ioi.lam at oracle.com Fri Nov 3 17:11:00 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 3 Nov 2017 10:11:00 -0700 Subject: RFR: 8184206: Resolve all string constants in shared classes at CDS dump time In-Reply-To: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> References: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> Message-ID: <0dbc712a-a4ca-86c0-03da-ff37230958c3@oracle.com> Hi Jiangli, This change looks good to me. Thanks! - Ioi On 11/1/17 12:28 PM, Jiangli Zhou wrote: > Hi, > > Please review the following change for 8184206. > > webrev: http://cr.openjdk.java.net/~jiangli/8184206/webrev.00/ > RFE: https://bugs.openjdk.java.net/browse/JDK-8184206?filter=14921 > > Before the change, constant pool string entries in shared classes were only resolved to existing interned strings at CDS dump time. The string constants remained as unresolved if the strings were not yet interned. > > The change in the above webrev resolves all string constants (excluding pseudo strings) for shared classes. For any strings that are not yet interned, java.lang.String instances are created and added to the string tabled when the string constants are resolved. All string instances referenced from the string table are archived. > > Resolving all string constants provides better support for AOT and CDS integration. It also improves CDS/AppCDS usability by removing the needs for profiling target applications to create a string list for archiving. > > The change results more string objects being created and archived at CDS dump time. Measurement on linux-x64 using the default class list shows there is about 1.7% size increase for the CDS archive with all strings resolved. The increase of the generated archive size is minimal comparing to the benefits listed above. > > Tested with tier1, tier2, and tier3 tests. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Fri Nov 3 17:34:07 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 3 Nov 2017 10:34:07 -0700 Subject: RFR: 8184206: Resolve all string constants in shared classes at CDS dump time In-Reply-To: <0dbc712a-a4ca-86c0-03da-ff37230958c3@oracle.com> References: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> <0dbc712a-a4ca-86c0-03da-ff37230958c3@oracle.com> Message-ID: <7CA2CCE0-8C2F-44A6-89C9-B3F366A98C33@oracle.com> Thanks, Ioi! Jiangli > On Nov 3, 2017, at 10:11 AM, Ioi Lam wrote: > > Hi Jiangli, > > This change looks good to me. Thanks! > > - Ioi > > > On 11/1/17 12:28 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review the following change for 8184206. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8184206/webrev.00/ >> RFE: https://bugs.openjdk.java.net/browse/JDK-8184206?filter=14921 >> >> Before the change, constant pool string entries in shared classes were only resolved to existing interned strings at CDS dump time. The string constants remained as unresolved if the strings were not yet interned. >> >> The change in the above webrev resolves all string constants (excluding pseudo strings) for shared classes. For any strings that are not yet interned, java.lang.String instances are created and added to the string tabled when the string constants are resolved. All string instances referenced from the string table are archived. >> >> Resolving all string constants provides better support for AOT and CDS integration. It also improves CDS/AppCDS usability by removing the needs for profiling target applications to create a string list for archiving. >> >> The change results more string objects being created and archived at CDS dump time. Measurement on linux-x64 using the default class list shows there is about 1.7% size increase for the CDS archive with all strings resolved. The increase of the generated archive size is minimal comparing to the benefits listed above. >> >> Tested with tier1, tier2, and tier3 tests. >> >> Thanks, >> Jiangli > From coleen.phillimore at oracle.com Fri Nov 3 17:34:30 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 3 Nov 2017 13:34:30 -0400 Subject: RFR: 8184206: Resolve all string constants in shared classes at CDS dump time In-Reply-To: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> References: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> Message-ID: Jiangli, This looks good!? It's very nice that it worked out to be so simple and consistent with what we'd expect. thanks, Coleen On 11/1/17 3:28 PM, Jiangli Zhou wrote: > Hi, > > Please review the following change for 8184206. > > webrev: http://cr.openjdk.java.net/~jiangli/8184206/webrev.00/ > RFE: https://bugs.openjdk.java.net/browse/JDK-8184206?filter=14921 > > Before the change, constant pool string entries in shared classes were only resolved to existing interned strings at CDS dump time. The string constants remained as unresolved if the strings were not yet interned. > > The change in the above webrev resolves all string constants (excluding pseudo strings) for shared classes. For any strings that are not yet interned, java.lang.String instances are created and added to the string tabled when the string constants are resolved. All string instances referenced from the string table are archived. > > Resolving all string constants provides better support for AOT and CDS integration. It also improves CDS/AppCDS usability by removing the needs for profiling target applications to create a string list for archiving. > > The change results more string objects being created and archived at CDS dump time. Measurement on linux-x64 using the default class list shows there is about 1.7% size increase for the CDS archive with all strings resolved. The increase of the generated archive size is minimal comparing to the benefits listed above. > > Tested with tier1, tier2, and tier3 tests. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Fri Nov 3 17:36:27 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 3 Nov 2017 10:36:27 -0700 Subject: RFR: 8184206: Resolve all string constants in shared classes at CDS dump time In-Reply-To: References: <390825B0-DCF2-46D5-BD2C-E5E1A17ED871@oracle.com> Message-ID: Thanks, Coleen! Jiangli > On Nov 3, 2017, at 10:34 AM, coleen.phillimore at oracle.com wrote: > > > Jiangli, > > This looks good! It's very nice that it worked out to be so simple and consistent with what we'd expect. > > thanks, > Coleen > > On 11/1/17 3:28 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review the following change for 8184206. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8184206/webrev.00/ >> RFE: https://bugs.openjdk.java.net/browse/JDK-8184206?filter=14921 >> >> Before the change, constant pool string entries in shared classes were only resolved to existing interned strings at CDS dump time. The string constants remained as unresolved if the strings were not yet interned. >> >> The change in the above webrev resolves all string constants (excluding pseudo strings) for shared classes. For any strings that are not yet interned, java.lang.String instances are created and added to the string tabled when the string constants are resolved. All string instances referenced from the string table are archived. >> >> Resolving all string constants provides better support for AOT and CDS integration. It also improves CDS/AppCDS usability by removing the needs for profiling target applications to create a string list for archiving. >> >> The change results more string objects being created and archived at CDS dump time. Measurement on linux-x64 using the default class list shows there is about 1.7% size increase for the CDS archive with all strings resolved. The increase of the generated archive size is minimal comparing to the benefits listed above. >> >> Tested with tier1, tier2, and tier3 tests. >> >> Thanks, >> Jiangli > From bob.vandette at oracle.com Fri Nov 3 18:04:27 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Fri, 3 Nov 2017 14:04:27 -0400 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> Message-ID: <30EFA226-1F76-4AF9-8838-6B0E2E39CF55@oracle.com> http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/CPUSetsReader.java.html Not sure this is a problem but If you specify --cpuset-cpus 2-3,1 in docker you end up with cpuset.cpus containing 1-3. Your cpusets test cases are all increasing in order so you won?t hit this issue. The read function in this file has a hard coded /sys/fs/cgroup/cpuset directory. This may not be where this ends up being mounted. Is this read function even used? http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/TestCPUAwareness.java.html Your test assumes that there are at least two physical processors on the host. You might want to check first. 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2) Everything else looks good, bob. > On Nov 1, 2017, at 11:11 PM, mikhailo wrote: > > Please review these tests that were developed to test JVM's container awareness in Docker environment. > JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 > Webrev: > Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ > WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ > Testing: > 1. Locally: Linux-x64, docker engine version: 17.06.2-ce > Ran the developed tests via jtreg > Pass > > 2. Automated testing system - run these tests > In progress > > Thank you, > Misha > From kishor.kharbas at intel.com Fri Nov 3 18:39:19 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Fri, 3 Nov 2017 18:39:19 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <1509719477.3207.10.camel@oracle.com> References: <1509719477.3207.10.camel@oracle.com> Message-ID: Thanks a lot! Link to updated webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ @Sangheon: Please let me know if you see any corrections needed. -Kishor > -----Original Message----- > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > Sent: Friday, November 3, 2017 7:31 AM > To: Kharbas, Kishor ; sangheon.kim > ; 'hotspot-gc-dev at openjdk.java.net' > ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative > memory devices and CSR review > > Hi, > > On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: > > Hi Sangheon, > > > > Thanks for the review and comments. Here is an updated webrev- > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 > > > > In addition to your suggested corrections, I added code to set Linux > > core dump filter ensuring Heap is dumped correctly when this feature > > is used. This is follow-up to Jini George?s comment > > (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- > > allocation-on-alternative-memory-devices-td300109.html#a300450). > > Some minor nits: > > - os_posix.cpp:300: please move the else next to the brace > - arguments.cpp:4624: please add a space between "if" and the bracket > > I do not need to see a new webrev for these changes. Looks good. > > Thanks, > Thomas From sangheon.kim at oracle.com Fri Nov 3 21:38:12 2017 From: sangheon.kim at oracle.com (sangheon.kim) Date: Fri, 3 Nov 2017 14:38:12 -0700 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: <1509719477.3207.10.camel@oracle.com> Message-ID: <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> Hi Kishor, On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: > Thanks a lot! > > Link to updated webrevs - > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ Thank you for fixing all. Looks good to me except below. Could you update the copyright format in TestAllocateHeapAt.java? 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. - Missing comma: * Copyright (c) 2017_*,*_ Oracle and/or its affiliates. All rights reserved. Thanks, Sangheon > > @Sangheon: Please let me know if you see any corrections needed. > > -Kishor > >> -----Original Message----- >> From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] >> Sent: Friday, November 3, 2017 7:31 AM >> To: Kharbas, Kishor ; sangheon.kim >> ; 'hotspot-gc-dev at openjdk.java.net' >> ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative >> memory devices and CSR review >> >> Hi, >> >> On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: >>> Hi Sangheon, >>> >>> Thanks for the review and comments. Here is an updated webrev- >>> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 >>> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 >>> >>> In addition to your suggested corrections, I added code to set Linux >>> core dump filter ensuring Heap is dumped correctly when this feature >>> is used. This is follow-up to Jini George?s comment >>> (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- >>> allocation-on-alternative-memory-devices-td300109.html#a300450). >> Some minor nits: >> >> - os_posix.cpp:300: please move the else next to the brace >> - arguments.cpp:4624: please add a space between "if" and the bracket >> >> I do not need to see a new webrev for these changes. Looks good. >> >> Thanks, >> Thomas From kishor.kharbas at intel.com Fri Nov 3 21:42:52 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Fri, 3 Nov 2017 21:42:52 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> Message-ID: Thanks Sangheon! I will make the change. -Kishor From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Friday, November 3, 2017 2:38 PM To: Kharbas, Kishor ; Thomas Schatzl ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: Thanks a lot! Link to updated webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ Thank you for fixing all. Looks good to me except below. Could you update the copyright format in TestAllocateHeapAt.java? 2 * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. Thanks, Sangheon @Sangheon: Please let me know if you see any corrections needed. -Kishor -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Friday, November 3, 2017 7:31 AM To: Kharbas, Kishor ; sangheon.kim ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime- dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi, On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: Hi Sangheon, Thanks for the review and comments. Here is an updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 In addition to your suggested corrections, I added code to set Linux core dump filter ensuring Heap is dumped correctly when this feature is used. This is follow-up to Jini George?s comment (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- allocation-on-alternative-memory-devices-td300109.html#a300450). Some minor nits: - os_posix.cpp:300: please move the else next to the brace - arguments.cpp:4624: please add a space between "if" and the bracket I do not need to see a new webrev for these changes. Looks good. Thanks, Thomas From kishor.kharbas at intel.com Fri Nov 3 21:59:59 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Fri, 3 Nov 2017 21:59:59 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> Message-ID: Hi Sangheon, Here is link to the updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ Thanks Kishor From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Friday, November 3, 2017 2:38 PM To: Kharbas, Kishor ; Thomas Schatzl ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: Thanks a lot! Link to updated webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ Thank you for fixing all. Looks good to me except below. Could you update the copyright format in TestAllocateHeapAt.java? 2 * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. Thanks, Sangheon @Sangheon: Please let me know if you see any corrections needed. -Kishor -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Friday, November 3, 2017 7:31 AM To: Kharbas, Kishor ; sangheon.kim ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime- dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi, On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: Hi Sangheon, Thanks for the review and comments. Here is an updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 In addition to your suggested corrections, I added code to set Linux core dump filter ensuring Heap is dumped correctly when this feature is used. This is follow-up to Jini George?s comment (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- allocation-on-alternative-memory-devices-td300109.html#a300450). Some minor nits: - os_posix.cpp:300: please move the else next to the brace - arguments.cpp:4624: please add a space between "if" and the bracket I do not need to see a new webrev for these changes. Looks good. Thanks, Thomas From sangheon.kim at oracle.com Fri Nov 3 23:07:16 2017 From: sangheon.kim at oracle.com (sangheon.kim) Date: Fri, 3 Nov 2017 16:07:16 -0700 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> Message-ID: <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> Hi Kishor, On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: > > Hi Sangheon, > > Here is link to the updated webrev- > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ > > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ > > Looks good to me. Thanks, Sangheon > Thanks > > Kishor > > *From:*sangheon.kim [mailto:sangheon.kim at oracle.com] > *Sent:* Friday, November 3, 2017 2:38 PM > *To:* Kharbas, Kishor ; Thomas Schatzl > ; 'hotspot-gc-dev at openjdk.java.net' > ; hotspot-runtime-dev at openjdk.java.net > *Subject:* Re: RFR(M): 8190308: Supporting heap allocation on > alternative memory devices and CSR review > > Hi Kishor, > > On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: > > Thanks a lot! > > Link to updated webrevs - > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ > > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ > > > Thank you for fixing all. > Looks good to me except below. > > Could you update the copyright format in TestAllocateHeapAt.java? > 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. > - Missing comma: * Copyright (c) 2017*_,_* Oracle and/or its > affiliates. All rights reserved. > > Thanks, > Sangheon > > > > @Sangheon: Please let me know if you see any corrections needed. > > -Kishor > > -----Original Message----- > > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > > Sent: Friday, November 3, 2017 7:31 AM > > To: Kharbas, Kishor ; sangheon.kim > > ; 'hotspot-gc-dev at openjdk.java.net > ' > > > ; hotspot-runtime- > > dev at openjdk.java.net > > Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative > > memory devices and CSR review > > Hi, > > On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: > > Hi Sangheon, > > Thanks for the review and comments. Here is an updated webrev- > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 > > > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 > > > In addition to your suggested corrections, I added code to set Linux > > core dump filter ensuring Heap is dumped correctly when this feature > > is used. This is follow-up to Jini George?s comment > > (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- > > allocation-on-alternative-memory-devices-td300109.html#a300450). > > Some minor nits: > > - os_posix.cpp:300: please move the else next to the brace > > - arguments.cpp:4624: please add a space between "if" and the bracket > > I do not need to see a new webrev for these changes. Looks good. > > Thanks, > > ? Thomas > From jiangli.zhou at Oracle.COM Fri Nov 3 23:46:25 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Fri, 3 Nov 2017 16:46:25 -0700 Subject: RFR(S) 8189840: CheckCachedResolvedReferencesApp has no cached resolved references Message-ID: Please review the fix for 8189840. CheckCachedResolvedReferencesApp currently is still a closed AppCDS test. The following webrev only contains the WhiteBox change with the new API (WhiteBox.openrchiveHeapObjectsMapped()) added. CheckCachedResolvedReferencesApp calls the new API to detect if the ?open archive? heap objects are mapped successfully in the current JVM execution. CheckCachedResolvedReferencesApp test change is not included in the webrev. webrev: http://cr.openjdk.java.net/~jiangli/8189840/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8189840?filter=14921 Tested on linux-x64. Also running hs-tier1, hs-tier2 tests. Thanks, Jiangli From mikhailo.seledtsov at oracle.com Sat Nov 4 00:21:26 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Fri, 03 Nov 2017 17:21:26 -0700 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> Message-ID: <59FD0806.5000200@oracle.com> Hi David, Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ Please see my comments below. On 11/2/17, 6:02 PM, David Holmes wrote: > Hi Misha, > > On 3/11/2017 10:22 AM, mikhailo wrote: >> Hi David, >> >> Thank you for taking a look at this change. >> >> >> On 11/02/2017 04:47 PM, David Holmes wrote: >>> Hi Misha, >>> >>> I don't see anything else in VMProps that writes anything out so to >>> me the fix is to delete the System.err.println completely. >> Removal is an option, and seems a simpler one. However, we will loose >> the diagnostic information. >> On the other hand, if an issue arises on a specific host or system, I >> can patch this file with extra trace statements, and diagnose the >> failure that way. >> If you think removing the print statement is a better option, and I >> do not hear objections or other opinions, I can just remove the print >> statement. > > Delete please :) > Done. >>> That said I did not look at this in the past but I feel the whole >>> docker test here is not really appropriate/suitable to always be run >>> on linux-x64. It has to exec another process and wait up to 10 >>> seconds to get a result! Can't this actually be done via a WhiteBox >>> check once Bob's container support updates are in place? >> Actually, WhiteBox is not necessary for such check. The ability to >> run docker on a given host/node can be determined in a body of a >> jtreg test, via a test utility. In fact, that was my original intent. >> When this change was presented for a discussion, this aspect was >> debated and reviewers have reached a consensus to do such check in >> VMProps.java. We mainly discussed within VM SQE team. The main >> argument was that using @requires is a more proper way of "skipping" >> the test(s) on unsupported environments, rather then checking this in >> the body of the test and then returning. When using @requires, the >> test execution system will report the test as "not run". When >> skipping the test using "return" from the body of a test, the >> execution system will report test as "ran" and passed; jtreg does not >> yet support the "skipped" state. >> >> Unfortunately, the way our "@requires" mechanism works is that it >> runs each time prior to starting a test run, even if the test being >> run is not concerned with a specific property. It was my concern that >> this could be a burden on the test run startup, but I eventually >> agreed to the opinion of other reviewers. >> >> If you have a strong opinion or concern about this, let me know. >> However, I am not sure what is the best process to use to overturn a >> decision on a prior review or design discussion. I guess one option >> is to file a bug or an RFE, and bring it up for a discussion. > > Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using > WhiteBox connected to Bob's new container support code is better than > exec'ing a separate process (which I'm not even sure what it does). I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy" Also, in current webrev, I though I could do something quick and easy to address your concerns, before we discuss and agree on a final solution. I have timed running "docker ps" on 2 different hosts, 30 times each. I saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec to 500ms. I thought 500 ms gives enough time cushion for slower test nodes/hosts. Let me know what you think. I can revert this part of the change or reduce the timeout value a bit more. Thank you, Misha > >> >> For this change under review, if I do not hear opinions from other >> reviewers, I can simply delete the print statement. > > Sounds good to me. > > Thanks, > David > >> >> Thank you, >> Misha >>> >>> Thanks, >>> David >>> >>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>> Please review this simple change that makes the error message in >>>> jtreg-ext/requires/VMProps.java conditional >>>> on java property "vmprops.trace.verbose". W/o this condition an >>>> error message is printed each time any jtreg >>>> test is ran, including non-docker tests. >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>> >>>> >>>> >>>> ======= Background and details: >>>> VMProps.java is called for each test run of jtreg (be it a run >>>> containing a singe test or multiple tests). >>>> VMProps evaluates the @requires properties. Therefore, in case of >>>> this bug, any time a user runs any jtreg VM test >>>> on a machine w/o an operational docker engine, this error will pop up. >>>> The fix makes printing of this error message conditional, off by >>>> default. The message can be turend on >>>> via a property vmprops.trace.verbose when detailed diagnostic info >>>> is desired. >>>> >>>> >>>> >>>> ======== Testing: >>>> Local testing: >>>> W/o docker present: >>>> jtreg >>>> No error message >>>> jtreg -Dvmprops.trace.verbose=true >>>> Detailed error message >>>> jtreg DockerBasicTest.java >>>> Test skipped >>>> >>>> With docker installed but no permissions: >>>> jtreg >>>> No error message >>>> jtreg -Dvmprops.trace.verbose=true >>>> Detailed error message >>>> jtreg DockerBasicTest.java >>>> Test skipped >>>> >>>> With docker installed and operational: >>>> jtreg >>>> No error message >>>> jtreg -Dvmprops.trace.verbose=true >>>> No error message >>>> jtreg DockerBasicTest.java >>>> Test passed >>>> >>>> Testing via automated distributed test system: >>>> - tier1 >>>> - docker tests >>>> In progress >>>> >>>> >>>> Thank you, >>>> Misha >>>> >> From david.holmes at oracle.com Mon Nov 6 03:28:28 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 6 Nov 2017 13:28:28 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <59FD0806.5000200@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> Message-ID: <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> Hi Misha, On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: > Hi David, > > Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ > > Please see my comments below. > > On 11/2/17, 6:02 PM, David Holmes wrote: >> Hi Misha, >> >> On 3/11/2017 10:22 AM, mikhailo wrote: >>> Hi David, >>> >>> ?? Thank you for taking a look at this change. >>> >>> >>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>> Hi Misha, >>>> >>>> I don't see anything else in VMProps that writes anything out so to >>>> me the fix is to delete the System.err.println completely. >>> Removal is an option, and seems a simpler one. However, we will loose >>> the diagnostic information. >>> On the other hand, if an issue arises on a specific host or system, I >>> can patch this file with extra trace statements, and diagnose the >>> failure that way. >>> If you think removing the print statement is a better option, and I >>> do not hear objections or other opinions, I can just remove the print >>> statement. >> >> Delete please :) >> > Done. >>>> That said I did not look at this in the past but I feel the whole >>>> docker test here is not really appropriate/suitable to always be run >>>> on linux-x64. It has to exec another process and wait up to 10 >>>> seconds to get a result! Can't this actually be done via a WhiteBox >>>> check once Bob's container support updates are in place? >>> Actually, WhiteBox is not necessary for such check. The ability to >>> run docker on a given host/node can be determined in a body of a >>> jtreg test, via a test utility. In fact, that was my original intent. >>> When this change was presented for a discussion, this aspect was >>> debated and reviewers have reached a consensus to do such check in >>> VMProps.java. We mainly discussed within VM SQE team. The main >>> argument was that using @requires is a more proper way of "skipping" >>> the test(s) on unsupported environments, rather then checking this in >>> the body of the test and then returning. When using @requires, the >>> test execution system will report the test as "not run". When >>> skipping the test using "return" from the body of a test, the >>> execution system will report test as "ran" and passed; jtreg does not >>> yet support the "skipped" state. >>> >>> Unfortunately, the way our "@requires" mechanism works is that it >>> runs each time prior to starting a test run, even if the test being >>> run is not concerned with a specific property. It was my concern that >>> this could be a burden on the test run startup, but I eventually >>> agreed to the opinion of other reviewers. >>> >>> If you have a strong opinion or concern about this, let me know. >>> However, I am not sure what is the best process to use to overturn a >>> decision on a prior review or design discussion. I guess one option >>> is to file a bug or an RFE, and bring it up for a discussion. >> >> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using >> WhiteBox connected to Bob's new container support code is better than >> exec'ing a separate process (which I'm not even sure what it does). > I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from > VMProps.java to determine presence of docker engine is deemed too heavy" > > Also, in current webrev, I though I could do something quick and easy to > address your concerns, before we discuss and agree on a final solution. > I have timed running "docker ps" on 2 different hosts, 30 times each. I > saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, > I reduced the timeout in the VMProps.java from 10 sec to 500ms. I > thought 500 ms gives enough time cushion for slower test nodes/hosts. I think a slowness would arise on systems that do not have docker installed. If I time "docker ps" on my system I get > time docker ps The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' real 0m4.103s user 0m0.044s sys 0m0.036s but the second call is much quicker: > time docker ps The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' real 0m0.078s user 0m0.052s sys 0m0.012s It may well depend on the user's path. If "docker" is present early in the path then it will be found and executed quickly, but if not present, or if further down the path it may take a while to occur. So I think the timeout change, as a short term proposal, may be more risky and so not worthwhile. I'd prefer to see the whole "docker ps" disappear altogether. Such an approach is not scalable if there may be different container systems. Thanks, David > Let me know what you think. I can revert this part of the change or > reduce the timeout value a bit more. > > Thank you, > Misha >> >>> >>> For this change under review, if I do not hear opinions from other >>> reviewers, I can simply delete the print statement. >> >> Sounds good to me. >> >> Thanks, >> David >> >>> >>> Thank you, >>> Misha >>>> >>>> Thanks, >>>> David >>>> >>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>> Please review this simple change that makes the error message in >>>>> jtreg-ext/requires/VMProps.java conditional >>>>> on java property "vmprops.trace.verbose". W/o this condition an >>>>> error message is printed each time any jtreg >>>>> test is ran, including non-docker tests. >>>>> >>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>> >>>>> >>>>> >>>>> ======= Background and details: >>>>> VMProps.java is called for each test run of jtreg (be it a run >>>>> containing a singe test or multiple tests). >>>>> VMProps evaluates the @requires properties. Therefore, in case of >>>>> this bug, any time a user runs any jtreg VM test >>>>> on a machine w/o an operational docker engine, this error will pop up. >>>>> The fix makes printing of this error message conditional, off by >>>>> default. The message can be turend on >>>>> via a property vmprops.trace.verbose when detailed diagnostic info >>>>> is desired. >>>>> >>>>> >>>>> >>>>> ======== Testing: >>>>> Local testing: >>>>> ?? W/o docker present: >>>>> ???? jtreg >>>>> ?????? No error message >>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>> ?????? Detailed error message >>>>> ???? jtreg DockerBasicTest.java >>>>> ?????? Test skipped >>>>> >>>>> ?? With docker installed but no permissions: >>>>> ???? jtreg >>>>> ?????? No error message >>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>> ?????? Detailed error message >>>>> ???? jtreg DockerBasicTest.java >>>>> ?????? Test skipped >>>>> >>>>> ?? With docker installed and operational: >>>>> ???? jtreg >>>>> ?????? No error message >>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>> ?????? No error message >>>>> ???? jtreg DockerBasicTest.java >>>>> ?????? Test passed >>>>> >>>>> Testing via automated distributed test system: >>>>> ?? - tier1 >>>>> ?? - docker tests >>>>> ?? In progress >>>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>> >>> From rkennke at redhat.com Mon Nov 6 11:13:01 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 6 Nov 2017 12:13:01 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> Message-ID: <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> Am 26.10.2017 um 11:43 schrieb Per Liden: > Hi, > > On 2017-10-25 18:11, Erik Osterlund wrote: > [...] >>> So let me just put your changes up for review (again), if you don't >>> mind: >>> >>> Full webrev: >>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>> > > I like this. Just two naming suggestions: > > src/hotspot/share/gc/shared/gcArguments.hpp: > > ? 39?? static jint create_instance(); > ? 40?? static bool is_initialized(); > ? 41?? static GCArguments* instance(); > > Could we call these: > > ? 39?? static jint initialize(); > ? 40?? static bool is_initialized(); > ? 41?? static GCArguments* arguments(); > > Reasoning: initialize() maps better to its companion is_initialized(). > GCArguments::arguments() maps better to the existing patterns we have > with CollectedHeap::heap(). Ok, that sounds good to me. Actually, it's almost full-circle back to my original proposal ;-) Differential: http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ Full: http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ Ok to push now? Roman From erik.osterlund at oracle.com Mon Nov 6 11:16:17 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 6 Nov 2017 12:16:17 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> Message-ID: <5A004481.3050001@oracle.com> Hi Roman, Looks good. Thanks, /Erik On 2017-11-06 12:13, Roman Kennke wrote: > Am 26.10.2017 um 11:43 schrieb Per Liden: >> Hi, >> >> On 2017-10-25 18:11, Erik Osterlund wrote: >> [...] >>>> So let me just put your changes up for review (again), if you don't >>>> mind: >>>> >>>> Full webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>> >> >> I like this. Just two naming suggestions: >> >> src/hotspot/share/gc/shared/gcArguments.hpp: >> >> 39 static jint create_instance(); >> 40 static bool is_initialized(); >> 41 static GCArguments* instance(); >> >> Could we call these: >> >> 39 static jint initialize(); >> 40 static bool is_initialized(); >> 41 static GCArguments* arguments(); >> >> Reasoning: initialize() maps better to its companion >> is_initialized(). GCArguments::arguments() maps better to the >> existing patterns we have with CollectedHeap::heap(). > Ok, that sounds good to me. Actually, it's almost full-circle back to > my original proposal ;-) > > Differential: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ > > Full: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ > > > Ok to push now? > > Roman From dmitry.samersoff at bell-sw.com Mon Nov 6 15:07:28 2017 From: dmitry.samersoff at bell-sw.com (Dmitry Samersoff) Date: Mon, 6 Nov 2017 18:07:28 +0300 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> Message-ID: <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> Ioi, I tested new patch on aarch64 and it works fine with the changes below[1]. Also tests doesn't work with exploded image - is it possible to check for this case and produce appropriate error message? 1. diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 09:45:23 2017 +0000 +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 15:02:51 2017 +0000 @@ -31,7 +31,6 @@ -#include "prims/jvm.h" diff -r dbf9cec6a568 src/hotspot/share/classfile/systemDictionaryShared.cpp --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov 06 09:45:23 2017 +0000 +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov 06 15:02:51 2017 +0000 @@ -518,7 +518,7 @@ { MutexLocker mu(SystemDictionary_lock, THREAD); - Klass* check = find_class(d_index, d_hash, name, dictionary); + Klass* check = dictionary->find_class(d_index, d_hash, name); if (check != NULL) { return InstanceKlass::cast(check); } -Dmitry On 11/01/2017 05:43 AM, Ioi Lam wrote: > Hi, > > Here's the new webrev for both the AppCDS implementation and tests. > During internal review of the JEP, we have decided to integrate both > implementation and tests at the same time. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ > > As mentioned before, most of the "diffs" shown in this webrev are the > result of copying the closed source files on top of files of the same > name in the open repo. So in reviewing, instead of focusing on what's > "changed", please focus on the entire content of the new version of each > file. > > Testing: I did an OpenJDK linux-x64 build (without Oracle closed > sources) and all the new appcds tests passed. > > Thanks > > - Ioi > > > On 10/30/17 8:52 AM, Ioi Lam wrote: >> Hi Dmitry, >> >> In the latest JDK 10 repo, is_jrt has been renamed to >> is_modules_image. Please change the code accordingly. >> >> I will post my latest diff soon, with some test cases as well. >> >> Thanks >> >> - Ioi >> >> >> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>> Ioi, >>> >>> I'd tried to apply your patch to latest open JDK10 and >>> the compilation fails with: >>> >>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>> >>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>> >>> Did I miss something? >>> >>> -Dmitry >>> >>> On 13.10.2017 02:48, Ioi Lam wrote: >>>> Hi, >>>> >>>> Please review this change set. >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>> ???? https://bugs.openjdk.java.net/browse/JDK-8188791 >>>> >>>> This is the first step of implementing the following JEP, which moves >>>> AppCDS from >>>> closed repos into the openjdk repo: >>>> >>>> ???? https://bugs.openjdk.java.net/browse/JDK-8185996 >>>> >>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>> repo. >>>> As part >>>> of the open-sourcing effort of JDK 18.3, we will move the source code >>>> into the >>>> open repo. >>>> >>>> In this changeset, the code is moved verbatim as much as possible. The >>>> intention is >>>> only to relocate the sources, not to changing existing behaviors, >>>> and not >>>> to do any sort of refactoring. >>>> >>>> Most of the "diffs" shown in this webrev are the result of copying the >>>> closed source >>>> files on top of files of the same name in the open repo. So in >>>> reviewing, instead of >>>> focusing on what's "changed", it's better to focus on the entire >>>> content >>>> of the new >>>> version of each file. >>>> >>>> The only functional change in this task is that the UseAppCDS flag is >>>> changed from >>>> a "commercial" flag to a regular "product" flag. This is because >>>> "commercial" >>>> flags are not supported by the OpenJDK build. >>>> >>>> Source code refactoring may be desirable, because the old open/closed >>>> source >>>> code structure had introduced some intermediary APIs to connect code >>>> between >>>> the two repos. Such API should be removed in a separate RFE. >>>> >>>> Also, some AppCDS tests are currently in the closed repo. These tests >>>> will be >>>> moved in a separate task. See JDK-8188792 for details. >>>> >>>> All the AppCDS tests (currently still in closed sources) passed with >>>> both Oracle JDK >>>> and OpenJDK. >>>> >>>> Thanks >>>> - Ioi >>> >> > From zgu at redhat.com Mon Nov 6 20:37:45 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 6 Nov 2017 15:37:45 -0500 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> Message-ID: Ping ... I need second reviewer and sponsor. Latest webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ Thanks, -Zhengyu On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >> >> >> Yes, this is not trivial. I hacked it in (because I wanted to see your >> output at the end of my tests) and had to disable the assert. I never >> ran into problems. Should java threads not all be stopped at that point? >> > Yes, it looks like that all Java threads are stopped at that point. > However, I am not so sure about workers. Could there be active workers > while JVM exiting? > > >> But this also can be done in a follow up issue. > > Yes, let's address it separately. > https://bugs.openjdk.java.net/browse/JDK-8190357 > > > Updated webrev based on your early comments: > > Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > Thanks, > > -Zhengyu > >> >> Thanks, Thomas >> >> >> Thanks! Thomas >> >> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >> >> > >> wrote: >> >> Hi Zhengyu, >> >> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > >> >> wrote: >> >> Hi Thomas, >> >> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >> >> Hi Zhengyu, >> >> - VM_PrintMetadata still has the _unit member, but >> I see it >> nowhere used. >> >> - I would have split the summary between class and >> non-class >> area, that may have been more useful. But this is a >> matter >> of taste, I leave it up to you. >> >> Agree, should go little further. >> >> Summary: >> Total class loaders= 754 capacity= 67528.00KB >> used= 61216.88KB free= 9453.11KB waste= 38.73KB >> Metadata capacity= 58780.00KB >> used= 54053.45KB free= 4726.55KB waste= 36.38KB >> Class data capacity= 8748.00KB >> used= 7163.43KB free= 1584.57KB waste= 2.35KB >> >> For anonymous classes= 580 capacity= 2732.00KB >> used= 1065.06KB free= 1666.94KB waste= 16.27KB >> Metadata capacity= 2152.00KB >> used= 738.46KB free= 1413.54KB waste= 16.27KB >> Class data capacity= 580.00KB >> used= 326.60KB free= 253.40KB waste= 0.00KB >> >> >> >> Nice, thanks :) >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> >> > > >> >> >> All is well. Note that you could reduce the footprint of >> your change >> somewhat by defining a structure like this: >> >> struct numbers_t { size_t cap; size_t used; size_t free; >> size_t waste; } >> >> and replacing the many members in >> PrintCLDMetaspaceInfoClosure with >> instances of this structure: >> >> numbers_t total; >> numbers_t total_class; >> numbers_t total_anon; >> numbers_t total_anon_class; >> Depending on how far you go, your code could deflate quite >> a bit. >> Give the structure a default ctor and your large >> initializer list >> goes away; give it a printing function and you reduce >> print-summary() to four function calls; with something >> like an >> numbers_t::add(size_t cap, size_t used, size_t free, size_t >> waste) >> you can reduce the additions in >> PrintCLDMetaspaceInfoClosure::print_metaspace() to four >> lines. >> >> Just a suggestion, I leave it up to you if you do this. >> >> Lines 3108 and 3129 miss each a space before BytesPerWord. >> Change >> looks fine otherwise. >> >> Thanks, Thomas >> >> >> Thanks, >> >> -Zhengyu >> >> >> All else looks fine to me now. I do not need >> another review. >> >> Thanks for your work, this will come in handy! >> >> ..Thomas >> >> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >> >> > >> >> >>> >> wrote: >> >> Hi Thomas, >> >> >> Not sure whether we misunderstood each >> other. I was >> thinking of >> something in the line of: >> >> <<<< >> .... >> ClassLoader: >> jdk/internal/reflect/DelegatingClassLoader >> Metadata: >> capacity: 5.00KB used: >> 3.32KB free: 1.68KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: >> 0.54KB free: 0.46KB >> waste: 0.00KB >> >> ClassLoader: for anonymous class >> Metadata: >> capacity: 1.00KB used: >> 1.00KB free: 0.00KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: >> 0.64KB free: 0.36KB >> waste: 0.00KB >> .... >> >> Summary: >> XX class loaders encountered, total >> capacity: xx, >> total waste: xx. >> >> Peak usage by class loader: xxxx >> >>>> >> >> Added summary lines: >> >> Total class loaders= 56 capacity= >> 6378.00KB >> used= 6205.08KB free= 172.92KB waste= >> 1.44KB >> For anonymous classes= 54 capacity= >> 212.00KB >> used= 96.95KB >> free= 115.05KB waste= 0.94KB >> >> >> >> >> These numbers would not be interpretation, >> just a >> convenience to >> the reader to get a clear picture. >> >> It even may be useful to separate the >> output >> between basic and >> detailed mode, and in basic mode omit >> all the >> single class >> loaders and just print the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> >> >> >> metaspace.hpp: >> >> I would have preferred to keep >> scale_unit file >> local static >> instead of exposing it via MetaspaceAux, >> because it >> does not >> really have anything to do with metaspace. >> >> Fixed >> >> >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - ChunkManager::print_statistics(): >> thanks for >> reusing this >> function! >> -> Scale can only be ever 1, K, M, >> G, yes? >> So, could you >> add an assert at the start of the >> function, and a >> comment at the >> prototype or function head? >> -> Also, now that >> ChunkManager::print_statistics() takes a >> scale, could you please change the >> printout to use >> floats like >> you did in >> PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> Done. >> >> >> Chunkmanager (non-class): >> 0 specialized (128 bytes) chunks, total >> 0.00KB >> 0 small (512 bytes) chunks, total 0.00KB >> 0 medium (8192 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> Chunkmanager (class): >> 0 specialized (128 bytes) chunks, total >> 0.00KB >> 0 small (256 bytes) chunks, total 0.00KB >> 0 medium (4096 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> >> >> - I am concerned about races in >> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >> Maybe my understanding is limited. We >> bail if >> cld->is_unloading. >> But nothing prevents a >> ClassLoaderDataGraph from >> starting to >> unload a ClassLoaderData and tearing >> down the >> Metaspace after we >> started printing it in another thread, >> right? Also, >> I do not see >> any fences. >> >> So, I wonder whether >> PrintCLDMetaspaceInfoClosure >> should run in >> lock protection. And then, I wonder if it >> would be >> good to >> separate data gathering and printing. We >> print to a >> (at least in >> principle) unknown output stream, which >> may be >> doing slow File >> IO or even Network IO. Doing this under >> lock >> protection seems >> not a good idea (but I see other places >> where this >> happens too). >> >> For an example, see >> ChunkManager::get_statistic() vs >> ChunkManager::print_statistic(). Could you >> do the >> same, have a >> data gathering step where you collect your >> quadrupel in a >> variable >> sized list of >> structures in memory, and print it out in a >> separate step? I >> realize though that the effort would be >> larger than >> for what we >> did in ChunkManager::get_statistic(), >> where we only >> fill a >> structure. >> >> >> Unlike other NMT queries, this query is >> executed at a >> safepoint by >> VMThread, so it should be okay. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> >> > > >> > >> > >> >> >> Thanks, >> >> -Zhengyu >> >> >> ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : do we still need >> _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 11:08 AM, Thomas St?fe >> wrote: >> >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, >> Zhengyu Gu >> >> > >> >> >> >> > >> > >> >> >>> >> > > > >> >> >> >> > >> > >> >> >>>>> >> wrote: >> >> >> >> Okay. So this is >> important for >> understanding cases >> where we have >> a large number of >> miniscule class >> loaders, >> each one >> only having >> a few numbers of chunks. >> In this >> case, would it be >> useful to >> have a running total of >> "free" >> and "waste", >> too? Or >> even better, >> also average and peak >> values of >> "free" and >> "waste" (to tell >> apart cases where you >> have one >> outlier vs >> cases where >> just all >> metaspaces waste a lot >> of memory). >> Just out of curiosity, I >> looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >> >>>> >> and >> it seems that "free" >> is the >> problem, not >> "waste"? So I >> guess >> what happens is that >> all the >> little classloaders >> allocate just >> enough memory to push >> them over >> "_small_chunk_limit=4", >> so they >> allocate the first >> medium chunk, >> from which >> then they >> take a >> small bite and leave the >> rest >> laying around? >> >> Yes. The free space of >> anonymous >> classes should be >> counted >> as waste, >> in my opinion. I am not >> sure if >> everyone agrees, >> so I took the >> summary line out of this >> patch. >> >> I would be more than happy >> to restore >> the summary >> line, if >> you find >> it is useful :-) >> >> >> I agree with you. Typically class >> loaders >> stop loading >> at some >> point, especially these tine >> ones, and >> often will not >> resume >> loading. So, effectivly, the >> space is >> wasted. It still >> helps to >> distinguish "free" in the current >> chunks >> from "free" in the >> other chunks to tell this >> situation apart >> from a >> situation where >> we have just a bug in metaspace >> chunk handling >> preventing us to >> use up our chunks. >> >> >> The latter >> would be an >> important >> addition, >> regardless >> if this is >> for customers >> or for us. >> Chunks idling in >> freelists can >> make a >> huge impact, and >> unfortunately this >> may happen >> in perfectly >> valid cases. >> See e.g. >> our JEP >> "https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>>>". We have >> already a >> printing >> routine for free >> chunks, in >> ChunkManager::print_all_chunkmanagers(). Could >> you add >> this to >> your output? >> >> >> Make sense! Could >> you >> recommend what data and >> format you >> would like >> to see? >> >> >> >> Would not >> ChunkManager::print_all_chunkmanagers() be >> sufficient? >> >> Okay, I might need to tweak >> output >> format. >> >> >> Fine by me. Nothing depends on it >> yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than >> above notes, >> the changes seem >> straighforward, I did >> not see >> anything wrong. >> Minor nits: >> >> - >> PrintCLDMetaspaceInfoClosure::_out, >> _scale >> and _unit >> could all >> be constant >> (with _out >> being a >> outputStream* >> const). >> - We also do >> not need to >> provide >> scale *and* >> unit as >> argument, >> one can be >> deduced from >> the other. >> This would >> also prevent >> mismatches.We >> do need >> scale here, >> since nmt >> command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing >> unit is >> ugly! I don't >> want to >> introduce >> inverse >> dependence >> (metaspace -> >> nmt), which is >> also ugly. >> >> >> Yes, that would be >> regrettable. >> >> Probably should >> move scale >> mapping code from >> NMTUtil to a >> common util? >> >> >> >> That would be possible, >> these >> functions >> (scale_from_name etc) >> are simple enough to be >> moved to >> a generic file. >> However, if you >> pass scale, deducing the >> string >> that goes with >> it is >> trivial and >> I would have no qualms >> doing this >> by hand in >> metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look >> closely >> at locking >> issues, I assume >> MetaspaceAux::print_metadata() is >> supposed to >> run only in >> Safepoints? >> >> >> No. >> sum_xxx_xxx_in_use >> queries are >> guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind >> Regards, >> Thomas >> >> >> >> >> >> >> On Fri, Oct 20, >> 2017 at >> 4:00 PM, >> Zhengyu Gu >> > >> > >> >> >> >> > > > >> >> >>> >> > >> > >> >> >> >> > > > >> >> >>>> >> >> > >> > > >> >> >> > >> > > >>> >> > >> > >> >> >> >> > > > >> >> >>>>> >> > >> > >> >> >> >> > > > >> >> >>> >> > >> > >> >> >> >> > > > >> >> >>>> >> >> >> > >> > > >> >> >> > >> > > >>> >> > >> > >> >> >> >> > > > >> >> >>>>>>> wrote: >> >> Up to now, >> there is >> little >> visibility >> into how >> metaspaces >> are used, >> and by >> whom. >> >> This >> proposed >> enhancement gives >> NMT the >> ability to >> report >> metaspace >> usages on >> per-classloader level, >> via a >> new NMT command >> "metadata" >> >> >> For >> example: >> >> 2496: >> Metaspaces: >> Metadata space: >> reserved= 8192KB >> committed= 5888KB >> Class >> space: >> reserved= 1048576KB >> committed= 640KB >> >> Per-classloader >> metadata: >> >> ClassLoader: for >> anonymous class >> Metadata capacity= 5.00KB >> used= 1.16KB >> free= >> 3.84KB >> waste= 0.05KB >> Class >> data >> capacity= 1.00KB >> used= 0.59KB >> free= >> 0.41KB >> waste= 0.00KB >> >> .... >> >> >> ClassLoader: >> >> Metadata capacity= 5640.00KB >> used= 5607.42KB >> free= >> 32.58KB >> waste= 0.46KB >> Class >> data >> capacity= 520.00KB >> used= 500.94KB >> free= >> 19.06KB >> waste= 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>>> >> > >> > > >> > >> >> From mikhailo.seledtsov at oracle.com Mon Nov 6 22:42:02 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Mon, 6 Nov 2017 14:42:02 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> Message-ID: <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> Hi David, On 11/05/2017 07:28 PM, David Holmes wrote: > Hi Misha, > > On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >> Hi David, >> >> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >> >> Please see my comments below. >> >> On 11/2/17, 6:02 PM, David Holmes wrote: >>> Hi Misha, >>> >>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>> Hi David, >>>> >>>> ?? Thank you for taking a look at this change. >>>> >>>> >>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>> Hi Misha, >>>>> >>>>> I don't see anything else in VMProps that writes anything out so >>>>> to me the fix is to delete the System.err.println completely. >>>> Removal is an option, and seems a simpler one. However, we will >>>> loose the diagnostic information. >>>> On the other hand, if an issue arises on a specific host or system, >>>> I can patch this file with extra trace statements, and diagnose the >>>> failure that way. >>>> If you think removing the print statement is a better option, and I >>>> do not hear objections or other opinions, I can just remove the >>>> print statement. >>> >>> Delete please :) >>> >> Done. >>>>> That said I did not look at this in the past but I feel the whole >>>>> docker test here is not really appropriate/suitable to always be >>>>> run on linux-x64. It has to exec another process and wait up to 10 >>>>> seconds to get a result! Can't this actually be done via a >>>>> WhiteBox check once Bob's container support updates are in place? >>>> Actually, WhiteBox is not necessary for such check. The ability to >>>> run docker on a given host/node can be determined in a body of a >>>> jtreg test, via a test utility. In fact, that was my original >>>> intent. When this change was presented for a discussion, this >>>> aspect was debated and reviewers have reached a consensus to do >>>> such check in VMProps.java. We mainly discussed within VM SQE team. >>>> The main argument was that using @requires is a more proper way of >>>> "skipping" the test(s) on unsupported environments, rather then >>>> checking this in the body of the test and then returning. When >>>> using @requires, the test execution system will report the test as >>>> "not run". When skipping the test using "return" from the body of a >>>> test, the execution system will report test as "ran" and passed; >>>> jtreg does not yet support the "skipped" state. >>>> >>>> Unfortunately, the way our "@requires" mechanism works is that it >>>> runs each time prior to starting a test run, even if the test being >>>> run is not concerned with a specific property. It was my concern >>>> that this could be a burden on the test run startup, but I >>>> eventually agreed to the opinion of other reviewers. >>>> >>>> If you have a strong opinion or concern about this, let me know. >>>> However, I am not sure what is the best process to use to overturn >>>> a decision on a prior review or design discussion. I guess one >>>> option is to file a bug or an RFE, and bring it up for a discussion. >>> >>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using >>> WhiteBox connected to Bob's new container support code is better >>> than exec'ing a separate process (which I'm not even sure what it >>> does). >> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from >> VMProps.java to determine presence of docker engine is deemed too heavy" >> >> Also, in current webrev, I though I could do something quick and easy >> to address your concerns, before we discuss and agree on a final >> solution. >> I have timed running "docker ps" on 2 different hosts, 30 times each. >> I saw a spread between 20ms to 150ms, mostly centered around 30ms. >> Hence, I reduced the timeout in the VMProps.java from 10 sec to >> 500ms. I thought 500 ms gives enough time cushion for slower test >> nodes/hosts. > > I think a slowness would arise on systems that do not have docker > installed. If I time "docker ps" on my system I get > > ?> time docker ps > The program 'docker' is currently not installed.? To run 'docker' > please ask your administrator to install the package 'docker' > > real??? 0m4.103s > user??? 0m0.044s > sys???? 0m0.036s > > but the second call is much quicker: > > ?> time docker ps > The program 'docker' is currently not installed.? To run 'docker' > please ask your administrator to install the package 'docker' > > real??? 0m0.078s > user??? 0m0.052s > sys???? 0m0.012s > > It may well depend on the user's path. If "docker" is present early in > the path then it will be found and executed quickly, but if not > present, or if further down the path it may take a while to occur. > > So I think the timeout change, as a short term proposal, may be more > risky and so not worthwhile. > > I'd prefer to see the whole "docker ps" disappear altogether. Such an > approach is not scalable if there may be different container systems. I have discussed this topic again with Igor and Leonid, and explained your concerns. We came up with alternative ideas: 1. VMProps.java will check the existence of docker on a given test machine. We will look at "/bin/docker", "/usr/bin/docker", "/usr/local/bin/docker" ??? For now we only support Linux x64. If any of these files exist, we return 'true' for a corresponding at-requires property, otherwise false. 2. In case someone wishes to specify a custom location for a docker binary, we thought we could use a property for that: ???? jdk.test.lib.docker.path="" ??? If this property is specified, VMProps.java will check the docker at that location. 3. No more "docker ps" execution in VMProps.java. Please let me know what you think. Thank you, Misha > > Thanks, > David > >> Let me know what you think. I can revert this part of the change or >> reduce the timeout value a bit more. >> >> Thank you, >> Misha >>> >>>> >>>> For this change under review, if I do not hear opinions from other >>>> reviewers, I can simply delete the print statement. >>> >>> Sounds good to me. >>> >>> Thanks, >>> David >>> >>>> >>>> Thank you, >>>> Misha >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>> Please review this simple change that makes the error message in >>>>>> jtreg-ext/requires/VMProps.java conditional >>>>>> on java property "vmprops.trace.verbose". W/o this condition an >>>>>> error message is printed each time any jtreg >>>>>> test is ran, including non-docker tests. >>>>>> >>>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>> >>>>>> >>>>>> >>>>>> ======= Background and details: >>>>>> VMProps.java is called for each test run of jtreg (be it a run >>>>>> containing a singe test or multiple tests). >>>>>> VMProps evaluates the @requires properties. Therefore, in case of >>>>>> this bug, any time a user runs any jtreg VM test >>>>>> on a machine w/o an operational docker engine, this error will >>>>>> pop up. >>>>>> The fix makes printing of this error message conditional, off by >>>>>> default. The message can be turend on >>>>>> via a property vmprops.trace.verbose when detailed diagnostic >>>>>> info is desired. >>>>>> >>>>>> >>>>>> >>>>>> ======== Testing: >>>>>> Local testing: >>>>>> ?? W/o docker present: >>>>>> ???? jtreg >>>>>> ?????? No error message >>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>> ?????? Detailed error message >>>>>> ???? jtreg DockerBasicTest.java >>>>>> ?????? Test skipped >>>>>> >>>>>> ?? With docker installed but no permissions: >>>>>> ???? jtreg >>>>>> ?????? No error message >>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>> ?????? Detailed error message >>>>>> ???? jtreg DockerBasicTest.java >>>>>> ?????? Test skipped >>>>>> >>>>>> ?? With docker installed and operational: >>>>>> ???? jtreg >>>>>> ?????? No error message >>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>> ?????? No error message >>>>>> ???? jtreg DockerBasicTest.java >>>>>> ?????? Test passed >>>>>> >>>>>> Testing via automated distributed test system: >>>>>> ?? - tier1 >>>>>> ?? - docker tests >>>>>> ?? In progress >>>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> >>>> From igor.ignatyev at oracle.com Tue Nov 7 00:31:18 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Mon, 6 Nov 2017 16:31:18 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> Message-ID: <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> Hi Misha, I have several comments: 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example > 73 Common.run( Common.newOpts(imageName) > 74 .addDockerOpts("--memory", valueToSet)) > 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); or > 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") > 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); > 98 opts.classParams.add("" + sizeToAllocInMb); 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 5. Test*: there is no need to have parentheses in '@requires (docker.support)' 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness Thanks, -- Igor > On Nov 1, 2017, at 8:11 PM, mikhailo wrote: > > Please review these tests that were developed to test JVM's container awareness in Docker environment. > JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 > Webrev: > Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ > WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ > Testing: > 1. Locally: Linux-x64, docker engine version: 17.06.2-ce > Ran the developed tests via jtreg > Pass > > 2. Automated testing system - run these tests > In progress > > Thank you, > Misha > From coleen.phillimore at oracle.com Tue Nov 7 01:00:54 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 6 Nov 2017 20:00:54 -0500 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> Message-ID: <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> Zhengyu,??? I've reviewed the code and this looks fine.? There is so much overlapping functionality for printing sizes of metaspaces in this file.? It would be nice to have just one way of printing the metaspaces, and have the ::dump functions go away, and have the "print" functions consistently print sizes and statistics.? There is also CLD walking and metaspace printing in ClassLoaderDataGraph::dump which is used for debugging.? I'd be happy if that was only printing statistics about the CLD and not details of the metaspace that they point to.?? This function was only for debugging though. I think your version may be the most currently useful so I'm fine with it.? I hope we can have a cleanup of the rest of the printing though. I will sponsor this. Thanks, Coleen On 11/6/17 3:37 PM, Zhengyu Gu wrote: > Ping ... > > I need second reviewer and sponsor. > > Latest webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > Thanks, > > -Zhengyu > > On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >>> >>> >>> Yes, this is not trivial. I hacked it in (because I wanted to see >>> your output at the end of my tests) and had to disable the assert. I >>> never ran into problems. Should java threads not all be stopped at >>> that point? >>> >> Yes, it looks like that all Java threads are stopped at that point. >> However, I am not so sure about workers. Could there be active >> workers while JVM exiting? >> >> >>> But this also can be done in a follow up issue. >> >> Yes, let's address it separately. >> https://bugs.openjdk.java.net/browse/JDK-8190357 >> >> >> Updated webrev based on your early comments: >> >> Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >> >> Thanks, >> >> -Zhengyu >> >>> >>> Thanks, Thomas >>> >>> >>> ??????? Thanks! Thomas >>> >>> ??????? On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>> ??????? >>> ??????? >> ??????? >> wrote: >>> >>> ???????????? Hi Zhengyu, >>> >>> ???????????? On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu >>> >> ??????? >>> ???????????? >> wrote: >>> >>> ???????????????? Hi Thomas, >>> >>> ???????????????? On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>> >>> ???????????????????? Hi Zhengyu, >>> >>> ???????????????????? - VM_PrintMetadata still has the _unit member, but >>> ??????? I see it >>> ???????????????????? nowhere used. >>> >>> ???????????????????? - I would have split the summary between class and >>> ??????? non-class >>> ???????????????????? area, that may have been more useful. But this >>> is a >>> ??????? matter >>> ???????????????????? of taste, I leave it up to you. >>> >>> ???????????????? Agree, should go little further. >>> >>> ???????????????? Summary: >>> ??????????????????? Total class loaders=?? 754 capacity= 67528.00KB >>> ??????? used=???????? 61216.88KB free=?? 9453.11KB waste= 38.73KB >>> ????????????????????????????????????? Metadata capacity= 58780.00KB >>> ??????? used=???????? 54053.45KB free=?? 4726.55KB waste= 36.38KB >>> ??????????????????????????????????? Class data capacity= 8748.00KB >>> ??????? used=?????????? 7163.43KB free=?? 1584.57KB waste=????? 2.35KB >>> >>> ???????????????? For anonymous classes=?? 580 capacity= 2732.00KB >>> ??????? used=?????????? 1065.06KB free=?? 1666.94KB waste= 16.27KB >>> ????????????????????????????????????? Metadata capacity= 2152.00KB >>> ??????? used=?????????? 738.46KB free=?? 1413.54KB waste= 16.27KB >>> ??????????????????????????????????? Class data capacity= 580.00KB >>> ??????? used=?????????? 326.60KB free=??? 253.40KB waste= 0.00KB >>> >>> >>> >>> ???????????? Nice, thanks :) >>> >>> ???????????????? Updated webrev: >>> ??????? http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>> >>> >> > >>> >>> >>> ???????????? All is well. Note that you could reduce the footprint of >>> ??????? your change >>> ???????????? somewhat by defining a structure like this: >>> >>> ???????????? struct numbers_t { size_t cap; size_t used; size_t free; >>> ??????? size_t waste; } >>> >>> ???????????? and replacing the many members in >>> ??????? PrintCLDMetaspaceInfoClosure with >>> ???????????? instances of this structure: >>> >>> ???????????? numbers_t total; >>> ???????????? numbers_t total_class; >>> ???????????? numbers_t total_anon; >>> ???????????? numbers_t total_anon_class; >>> ???????????? Depending on how far you go, your code could deflate quite >>> ??????? a bit. >>> ???????????? Give the structure a default ctor and your large >>> ??????? initializer list >>> ???????????? goes away; give it a printing function and you reduce >>> ???????????? print-summary() to four function calls; with something >>> like an >>> ???????????? numbers_t::add(size_t cap, size_t used, size_t free, >>> size_t >>> ??????? waste) >>> ???????????? you can reduce the additions in >>> ???????????? PrintCLDMetaspaceInfoClosure::print_metaspace() to four >>> lines. >>> >>> ???????????? Just a suggestion, I leave it up to you if you do this. >>> >>> ???????????? Lines 3108 and 3129 miss each a space before BytesPerWord. >>> ??????? Change >>> ???????????? looks fine otherwise. >>> >>> ???????????? Thanks, Thomas >>> >>> >>> ???????????????? Thanks, >>> >>> ???????????????? -Zhengyu >>> >>> >>> ???????????????????? All else looks fine to me now. I do not need >>> ??????? another review. >>> >>> ???????????????????? Thanks for your work, this will come in handy! >>> >>> ???????????????????? ..Thomas >>> >>> ???????????????????? On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >>> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >>> >>> ??????? wrote: >>> >>> ????????????????????????? Hi Thomas, >>> >>> >>> ????????????????????????????? Not sure whether we misunderstood each >>> ??????? other. I was >>> ???????????????????? thinking of >>> ????????????????????????????? something in the line of: >>> >>> ????????????????????????????? <<<< >>> ????????????????????????????? .... >>> ????????????????????????????? ClassLoader: >>> ??????? jdk/internal/reflect/DelegatingClassLoader >>> ????????????????????????????????? Metadata: >>> ???????????????????????????????????? capacity:???? 5.00KB used: >>> ????????? 3.32KB free:???????????????? 1.68KB >>> ????????????????????????????? waste: 0.00KB >>> ????????????????????????????????? Class data: >>> ???????????????????????????????????? capacity:???? 1.00KB used: >>> ????????? 0.54KB free:???????????????? 0.46KB >>> ????????????????????????????? waste: 0.00KB >>> >>> ????????????????????????????? ClassLoader: for anonymous class >>> ????????????????????????????????? Metadata: >>> ???????????????????????????????????? capacity:???? 1.00KB used: >>> ????????? 1.00KB free:???????????????? 0.00KB >>> ????????????????????????????? waste: 0.00KB >>> ????????????????????????????????? Class data: >>> ???????????????????????????????????? capacity:???? 1.00KB used: >>> ????????? 0.64KB free:???????????????? 0.36KB >>> ????????????????????????????? waste: 0.00KB >>> ????????????????????????????? .... >>> >>> ????????????????????????????? Summary: >>> ????????????????????????????? XX class loaders encountered, total >>> ??????? capacity: xx, >>> ???????????????????? total waste: xx. >>> >>> ????????????????????????????? Peak usage by class loader: xxxx >>> ??????????????????????????????? >>>> >>> >>> ????????????????????????? Added summary lines: >>> >>> ???????????????????????????? Total class loaders=??? 56 capacity= >>> ????????? 6378.00KB >>> ???????????????????? used=?????? 6205.08KB free=??? 172.92KB waste= >>> ??????? 1.44KB >>> ????????????????????????? For anonymous classes=??? 54 capacity=??? >>> 212.00KB >>> ???????????????????? used=???? 96.95KB >>> ????????????????????????? free=??? 115.05KB waste=????? 0.94KB >>> >>> >>> >>> >>> ????????????????????????????? These numbers would not be >>> interpretation, >>> ??????? just a >>> ???????????????????? convenience to >>> ????????????????????????????? the reader to get a clear picture. >>> >>> ????????????????????????????? It even may be useful to separate the >>> output >>> ???????????????????? between basic and >>> ????????????????????????????? detailed mode, and in basic mode omit >>> all the >>> ???????????????????? single class >>> ????????????????????????????? loaders and just print the summary line. >>> >>> ?????????????????????????????????? Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >>> >>> >>> ????????????????????????????? metaspace.hpp: >>> >>> ????????????????????????????? I would have preferred to keep >>> scale_unit file >>> ???????????????????? local static >>> ????????????????????????????? instead of exposing it via MetaspaceAux, >>> ??????? because it >>> ???????????????????? does not >>> ????????????????????????????? really have anything to do with >>> metaspace. >>> >>> ????????????????????????? Fixed >>> >>> >>> ????????????????????????????? Otherwise ok. >>> >>> ????????????????????????????? --- >>> >>> ????????????????????????????? metaspace.cpp >>> >>> ????????????????????????????? - ChunkManager::print_statistics(): >>> thanks for >>> ???????????????????? reusing this >>> ????????????????????????????? function! >>> ??????????????????????????????????? -> Scale can only be ever 1, K, M, >>> ??????? G, yes? >>> ???????????????????? So, could you >>> ????????????????????????????? add an assert at the start of the >>> ??????? function, and a >>> ???????????????????? comment at the >>> ????????????????????????????? prototype or function head? >>> ??????????????????????????????????? -> Also, now that >>> ???????????????????? ChunkManager::print_statistics() takes a >>> ????????????????????????????? scale, could you please change the >>> ??????? printout to use >>> ???????????????????? floats like >>> ????????????????????????????? you did in >>> PrintCLDMetaspaceInfoClosure::print_metaspace()? >>> >>> ????????????????????????? Done. >>> >>> >>> ????????????????????????? Chunkmanager (non-class): >>> ???????????????????????????? 0 specialized (128 bytes) chunks, total >>> 0.00KB >>> ???????????????????????????? 0 small (512 bytes) chunks, total 0.00KB >>> ???????????????????????????? 0 medium (8192 bytes) chunks, total 0.00KB >>> ???????????????????????????? 0 humongous chunks, total 0.00KB >>> ???????????????????????????? total size: 0.00KB. >>> ????????????????????????? Chunkmanager (class): >>> ???????????????????????????? 0 specialized (128 bytes) chunks, total >>> 0.00KB >>> ???????????????????????????? 0 small (256 bytes) chunks, total 0.00KB >>> ???????????????????????????? 0 medium (4096 bytes) chunks, total 0.00KB >>> ???????????????????????????? 0 humongous chunks, total 0.00KB >>> ???????????????????????????? total size: 0.00KB. >>> >>> >>> ????????????????????????????? - I am concerned about races in >>> ??????? PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>> ????????????????????????????? Maybe my understanding is limited. We >>> bail if >>> ???????????????????? cld->is_unloading. >>> ????????????????????????????? But nothing prevents a >>> ??????? ClassLoaderDataGraph from >>> ???????????????????? starting to >>> ????????????????????????????? unload a ClassLoaderData and tearing >>> down the >>> ???????????????????? Metaspace after we >>> ????????????????????????????? started printing it in another thread, >>> ??????? right? Also, >>> ???????????????????? I do not see >>> ????????????????????????????? any fences. >>> >>> ????????????????????????????? So, I wonder whether >>> ??????? PrintCLDMetaspaceInfoClosure >>> ???????????????????? should run in >>> ????????????????????????????? lock protection. And then, I wonder if it >>> ??????? would be >>> ???????????????????? good to >>> ????????????????????????????? separate data gathering and printing. We >>> ??????? print to a >>> ???????????????????? (at least in >>> ????????????????????????????? principle) unknown output stream, >>> which may be >>> ???????????????????? doing slow File >>> ????????????????????????????? IO or even Network IO. Doing this >>> under lock >>> ???????????????????? protection seems >>> ????????????????????????????? not a good idea (but I see other places >>> ??????? where this >>> ???????????????????? happens too). >>> >>> ????????????????????????????? For an example, see >>> ??????? ChunkManager::get_statistic() vs >>> ????????????????????????????? ChunkManager::print_statistic(). Could >>> you >>> ??????? do the >>> ???????????????????? same, have a >>> ????????????????????????????? data gathering step where you collect >>> your >>> ????????????????????????????? quadrupel in a >>> ??????? variable >>> ???????????????????? sized list of >>> ????????????????????????????? structures in memory, and print it out >>> in a >>> ???????????????????? separate step? I >>> ????????????????????????????? realize though that the effort would be >>> ??????? larger than >>> ???????????????????? for what we >>> ????????????????????????????? did in ChunkManager::get_statistic(), >>> ??????? where we only >>> ???????????????????? fill a >>> ????????????????????????????? structure. >>> >>> >>> ????????????????????????? Unlike other NMT queries, this query is >>> ??????? executed at a >>> ???????????????????? safepoint by >>> ????????????????????????? VMThread, so it should be okay. >>> >>> ????????????????????????? Updated webrev: >>> ??????? http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>> >>> >> > >>> >> >>> >> >> >>> >>> ????????????????????????? Thanks, >>> >>> ????????????????????????? -Zhengyu >>> >>> >>> ????????????????????????????? ------ >>> >>> ????????????????????????????? vm_operations.hpp >>> >>> ????????????????????????????? - VM_PrintMetadata : do we still need >>> _unit? >>> >>> >>> ?????????????????????????????????? Thanks, >>> >>> ?????????????????????????????????? -Zhengyu >>> >>> >>> >>> ????????????????????????????? Thanks! >>> >>> ????????????????????????????? Thomas >>> >>> >>> >>> ?????????????????????????????????? On 10/23/2017 11:08 AM, Thomas St?fe >>> ??????? wrote: >>> >>> >>> >>> ?????????????????????????????????????? On Mon, Oct 23, 2017 at 5:03 PM, >>> ??????? Zhengyu Gu >>> ????????????????????????????? >>> ??????? > >>> ???????????????????? >>> ??????? >> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >>> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>>>> >>> ??????? wrote: >>> >>> >>> >>> ??????????????????????????????????????????????? Okay. So this is >>> ??????? important for >>> ???????????????????? understanding cases >>> ?????????????????????????????????????? where we have >>> ??????????????????????????????????????????????? a large number of >>> ??????? miniscule class >>> ???????????????????? loaders, >>> ????????????????????????????? each one >>> ?????????????????????????????????????? only having >>> ??????????????????????????????????????????????? a few numbers of >>> chunks. >>> ??????? In this >>> ???????????????????? case, would it be >>> ?????????????????????????????????????? useful to >>> ??????????????????????????????????????????????? have a running total of >>> ??????? "free" >>> ???????????????????? and "waste", >>> ????????????????????????????? too? Or >>> ?????????????????????????????????????? even better, >>> ??????????????????????????????????????????????? also average and peak >>> ??????? values of >>> ???????????????????? "free" and >>> ????????????????????????????? "waste" (to tell >>> ??????????????????????????????????????????????? apart cases where you >>> ??????? have one >>> ???????????????????? outlier vs >>> ????????????????????????????? cases where >>> ?????????????????????????????????????? just all >>> ??????????????????????????????????????????????? metaspaces waste a lot >>> ??????? of memory). >>> ??????????????????????????????????????????????? Just out of >>> curiosity, I >>> ??????? looked at >>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> >>> ??????????????????? and >>> ??????????????????????????????????????????????? it seems that "free" >>> is the >>> ???????????????????? problem, not >>> ????????????????????????????? "waste"? So I >>> ?????????????????????????????????????? guess >>> ??????????????????????????????????????????????? what happens is that >>> all the >>> ???????????????????? little classloaders >>> ?????????????????????????????????????? allocate just >>> ??????????????????????????????????????????????? enough memory to push >>> ??????? them over >>> ????????????????????????????? "_small_chunk_limit=4", >>> ?????????????????????????????????????? so they >>> ??????????????????????????????????????????????? allocate the first >>> ??????? medium chunk, >>> ???????????????????? from which >>> ????????????????????????????? then they >>> ?????????????????????????????????????? take a >>> ??????????????????????????????????????????????? small bite and leave >>> the >>> ??????? rest >>> ???????????????????? laying around? >>> >>> ??????????????????????????????????????????? Yes. The free space of >>> anonymous >>> ???????????????????? classes should be >>> ????????????????????????????? counted >>> ?????????????????????????????????????? as waste, >>> ??????????????????????????????????????????? in my opinion. I am not >>> sure if >>> ???????????????????? everyone agrees, >>> ????????????????????????????? so I took the >>> ??????????????????????????????????????????? summary line out of this >>> patch. >>> >>> ??????????????????????????????????????????? I would be more than happy >>> ??????? to restore >>> ???????????????????? the summary >>> ????????????????????????????? line, if >>> ?????????????????????????????????????? you find >>> ??????????????????????????????????????????? it is useful :-) >>> >>> >>> ?????????????????????????????????????? I agree with you. Typically >>> class >>> ??????? loaders >>> ???????????????????? stop loading >>> ????????????????????????????? at some >>> ?????????????????????????????????????? point, especially these tine >>> ??????? ones, and >>> ???????????????????? often will not >>> ????????????????????????????? resume >>> ?????????????????????????????????????? loading. So, effectivly, the >>> space is >>> ???????????????????? wasted. It still >>> ????????????????????????????? helps to >>> ?????????????????????????????????????? distinguish "free" in the >>> current >>> ??????? chunks >>> ???????????????????? from "free" in the >>> ?????????????????????????????????????? other chunks to tell this >>> ??????? situation apart >>> ???????????????????? from a >>> ????????????????????????????? situation where >>> ?????????????????????????????????????? we have just a bug in metaspace >>> ??????? chunk handling >>> ????????????????????????????? preventing us to >>> ?????????????????????????????????????? use up our chunks. >>> >>> >>> ???????????????????????????????????????????????????????? The latter >>> ??????? would be an >>> ???????????????????? important >>> ????????????????????????????? addition, >>> ?????????????????????????????????????? regardless >>> ??????????????????????????????????????????????? if this is >>> ???????????????????????????????????????????????????????? for customers >>> ??????? or for us. >>> ???????????????????? Chunks idling in >>> ?????????????????????????????????????? freelists can >>> ??????????????????????????????????????????????? make a >>> ???????????????????????????????????????????????????????? huge >>> impact, and >>> ???????????????????? unfortunately this >>> ????????????????????????????? may happen >>> ?????????????????????????????????????? in perfectly >>> ???????????????????????????????????????????????????????? valid cases. >>> ??????? See e.g. >>> ???????????????????? our JEP >>> "https://bugs.openjdk.java.net/browse/JDK-8166690 >>> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>>> >>> >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>>>>". We have >>> already a printing >>> ???????????????????? routine for free >>> ????????????????????????????? chunks, in >>> ?????????? ChunkManager::print_all_chunkmanagers(). Could >>> ?????????????????????????????????????? you add >>> ??????????????????????????????????????????????? this to >>> ???????????????????????????????????????????????????????? your output? >>> >>> >>> ???????????????????????????????????????????????????? Make sense! >>> Could you >>> ???????????????????? recommend what data and >>> ?????????????????????????????????????? format you >>> ??????????????????????????????????????????????? would like >>> ???????????????????????????????????????????????????? to see? >>> >>> >>> >>> ??????????????????????????????????????????????? Would not >>> ChunkManager::print_all_chunkmanagers() be >>> ?????????????????????????????????????? sufficient? >>> >>> ??????????????????????????????????????????? Okay, I might need to tweak >>> ??????? output >>> ???????????????????? format. >>> >>> >>> ?????????????????????????????????????? Fine by me. Nothing depends >>> on it >>> ??????? yet. >>> >>> ?????????????????????????????????????? Thanks, Thomas >>> >>> ??????????????????????????????????????????? Thanks, >>> >>> ??????????????????????????????????????????? -Zhengyu >>> >>> >>> >>> ------------- >>> >>> ???????????????????????????????????????????????????????? Other than >>> ??????? above notes, >>> ???????????????????? the changes seem >>> straighforward, I did >>> ???????????????????????????????????????????????????????? not see >>> ??????? anything wrong. >>> ???????????????????? Minor nits: >>> >>> ???????????????????????????????????????????????????????? - >>> ???????????????????? PrintCLDMetaspaceInfoClosure::_out, >>> ????????????????????????????? _scale >>> ?????????????????????????????????????? and _unit >>> ??????????????????????????????????????????????? could all >>> ???????????????????????????????????????????????????????? be constant >>> ??????? (with _out >>> ???????????????????? being a >>> ????????????????????????????? outputStream* >>> ?????????????????????????????????????? const). >>> ???????????????????????????????????????????????????????? - We also do >>> ??????? not need to >>> ???????????????????? provide >>> ????????????????????????????? scale *and* >>> ?????????????????????????????????????? unit as >>> ??????????????????????????????????????????????? argument, >>> ???????????????????????????????????????????????????????? one can be >>> ??????? deduced from >>> ???????????????????? the other. >>> ????????????????????????????? This would >>> ?????????????????????????????????????? also prevent >>> mismatches.We >>> ??????? do need >>> ???????????????????? scale here, >>> ????????????????????????????? since nmt >>> ?????????????????????????????????????? command >>> ??????????????????????????????????????????????? line can set >>> ???????????????????????????????????????????????????????? scale and >>> we do >>> >>> ???????????????????????????????????????????????????? have to honor >>> that. >>> >>> ???????????????????????????????????????????????????? However, passing >>> ??????? unit is >>> ???????????????????? ugly! I don't >>> ????????????????????????????? want to >>> ?????????????????????????????????????? introduce >>> ??????????????????????????????????????????????? inverse >>> dependence >>> ??????? (metaspace -> >>> ???????????????????? nmt), which is >>> ????????????????????????????? also ugly. >>> >>> >>> ??????????????????????????????????????????????? Yes, that would be >>> ??????? regrettable. >>> >>> ???????????????????????????????????????????????????? Probably should >>> ??????? move scale >>> ???????????????????? mapping code from >>> ?????????????????????????????????????? NMTUtil to a >>> ??????????????????????????????????????????????? common util? >>> >>> >>> >>> ??????????????????????????????????????????????? That would be possible, >>> ??????? these >>> ???????????????????? functions >>> ?????????????????????????????????????? (scale_from_name etc) >>> ??????????????????????????????????????????????? are simple enough to be >>> ??????? moved to >>> ???????????????????? a generic file. >>> ?????????????????????????????????????? However, if you >>> ??????????????????????????????????????????????? pass scale, deducing >>> the >>> ??????? string >>> ???????????????????? that goes with >>> ????????????????????????????? it is >>> ?????????????????????????????????????? trivial and >>> ??????????????????????????????????????????????? I would have no qualms >>> ??????? doing this >>> ???????????????????? by hand in >>> ?????????????????????????????????????? metaspace.cpp. But >>> ??????????????????????????????????????????????? it is a matter of >>> taste. >>> >>> >>> ???????????????????????????????????????????????????????? I did not look >>> ??????? closely >>> ???????????????????? at locking >>> ????????????????????????????? issues, I assume >>> ??????? MetaspaceAux::print_metadata() is >>> ????????????????????????????? supposed to >>> ?????????????????????????????????????? run only in >>> Safepoints? >>> >>> >>> ???????????????????????????????????????????????????? No. >>> sum_xxx_xxx_in_use >>> ???????????????????? queries are >>> ????????????????????????????? guarded by locks. >>> >>> ???????????????????????????????????????????????????? Thanks, >>> >>> ???????????????????????????????????????????????????? -Zhengyu >>> >>> >>> ??????????????????????????????????????????????? Thanks, Thomas >>> >>> >>> >>> Thanks & Kind >>> ??????? Regards, >>> ???????????????????? Thomas >>> >>> >>> >>> >>> >>> >>> ???????????????????????????????????????????????????????? On Fri, Oct >>> 20, >>> ??????? 2017 at >>> ???????????????????? 4:00 PM, >>> ????????????????????????????? Zhengyu Gu >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>>> >>> ??????? >>> ???????????????????? > >>> ????????????????????????????? >> ??????? >> ??????? >> >>> ???????????????????? >>> ??????? > >>> ????????????????????????????? >> ??????? >> ??????? >>> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>>>> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>>> >>> >>> ??????? >>> ???????????????????? > >>> ????????????????????????????? >> ??????? >> ??????? >> >>> ???????????????????? >>> ??????? > >>> ????????????????????????????? >> ??????? >> ??????? >>> >>> >> ??????? >>> ???????????????????? > >>> ??????? >>> ???????????????????? >> >>> ????????????????????????????? >> ??????? >> ??????? > >>> ???????????????????? >>> ??????? >>>>>>> wrote: >>> >>> Up to now, >>> ??????? there is >>> ???????????????????? little >>> ????????????????????????????? visibility >>> ?????????????????????????????????????? into how >>> ??????????????????????????????????????????????? metaspaces >>> ???????????????????????????????????????????????????????? are used, >>> and by whom. >>> >>> This proposed >>> ???????????????????? enhancement gives >>> ????????????????????????????? NMT the >>> ?????????????????????????????????????? ability to >>> ??????????????????????????????????????????????? report >>> metaspace >>> usages on >>> ???????????????????? per-classloader level, >>> ????????????????????????????? via a >>> ?????????????????????????????????????? new NMT command >>> "metadata" >>> >>> >>> For example: >>> >>> 2496: >>> Metaspaces: >>> ??????? Metadata space: >>> ???????????????????? reserved=????????????? 8192KB >>> committed=????? 5888KB >>> ???????????????????????????????????????????????????????????????? >>> Class ?????????? space: >>> ???????????????????? reserved=?????????? 1048576KB >>> committed=?????? 640KB >>> >>> ????????? Per-classloader >>> ???????????????????? metadata: >>> >>> ????????? ClassLoader: for >>> ???????????????????? anonymous class >>> ??????? Metadata?????????????? capacity=????? 5.00KB >>> ?????????????????????????????????????? used=????? 1.16KB >>> ???????????????????????????????????????????????????????? free= >>> ????????? 3.84KB >>> ???????????????????? waste=????? 0.05KB >>> ???????????????????????????????????????????????????????????????? >>> Class data >>> ???????????????????? capacity=????? 1.00KB >>> ?????????????????????????????????????? used=????? 0.59KB >>> ???????????????????????????????????????????????????????? free= >>> ????????? 0.41KB >>> ???????????????????? waste=????? 0.00KB >>> >>> .... >>> >>> ClassLoader: >>> ???????????????????? >>> ??????? Metadata?????????????? capacity=?? 5640.00KB >>> ?????????????????????????????????????? used=?? 5607.42KB >>> ???????????????????????????????????????????????????????? free= >>> ????????? 32.58KB >>> ???????????????????? waste=????? 0.46KB >>> ???????????????????????????????????????????????????????????????? >>> Class data >>> ???????????????????? capacity=??? 520.00KB >>> ?????????????????????????????????????? used=??? 500.94KB >>> ???????????????????????????????????????????????????????? free= >>> ????????? 19.06KB >>> ???????????????????? waste=????? 0.02KB >>> >>> >>> Bug: >>> ??????? https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>>> >>> >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >> >>> ??????? >> >>> >> > >>> ????????? >> >>> >> >>>>> >>> >> >>> >> > >>> ????????? >> >>> >>> From david.holmes at oracle.com Tue Nov 7 01:34:19 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 7 Nov 2017 11:34:19 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> Message-ID: <82e41837-1674-ee35-3c3f-a11e7540eb89@oracle.com> Hi Misha, This seems reasonable. Thanks, David On 7/11/2017 8:42 AM, mikhailo wrote: > Hi David, > > > On 11/05/2017 07:28 PM, David Holmes wrote: >> Hi Misha, >> >> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>> Hi David, >>> >>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>> >>> Please see my comments below. >>> >>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>> Hi Misha, >>>> >>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>> Hi David, >>>>> >>>>> ?? Thank you for taking a look at this change. >>>>> >>>>> >>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>> Hi Misha, >>>>>> >>>>>> I don't see anything else in VMProps that writes anything out so >>>>>> to me the fix is to delete the System.err.println completely. >>>>> Removal is an option, and seems a simpler one. However, we will >>>>> loose the diagnostic information. >>>>> On the other hand, if an issue arises on a specific host or system, >>>>> I can patch this file with extra trace statements, and diagnose the >>>>> failure that way. >>>>> If you think removing the print statement is a better option, and I >>>>> do not hear objections or other opinions, I can just remove the >>>>> print statement. >>>> >>>> Delete please :) >>>> >>> Done. >>>>>> That said I did not look at this in the past but I feel the whole >>>>>> docker test here is not really appropriate/suitable to always be >>>>>> run on linux-x64. It has to exec another process and wait up to 10 >>>>>> seconds to get a result! Can't this actually be done via a >>>>>> WhiteBox check once Bob's container support updates are in place? >>>>> Actually, WhiteBox is not necessary for such check. The ability to >>>>> run docker on a given host/node can be determined in a body of a >>>>> jtreg test, via a test utility. In fact, that was my original >>>>> intent. When this change was presented for a discussion, this >>>>> aspect was debated and reviewers have reached a consensus to do >>>>> such check in VMProps.java. We mainly discussed within VM SQE team. >>>>> The main argument was that using @requires is a more proper way of >>>>> "skipping" the test(s) on unsupported environments, rather then >>>>> checking this in the body of the test and then returning. When >>>>> using @requires, the test execution system will report the test as >>>>> "not run". When skipping the test using "return" from the body of a >>>>> test, the execution system will report test as "ran" and passed; >>>>> jtreg does not yet support the "skipped" state. >>>>> >>>>> Unfortunately, the way our "@requires" mechanism works is that it >>>>> runs each time prior to starting a test run, even if the test being >>>>> run is not concerned with a specific property. It was my concern >>>>> that this could be a burden on the test run startup, but I >>>>> eventually agreed to the opinion of other reviewers. >>>>> >>>>> If you have a strong opinion or concern about this, let me know. >>>>> However, I am not sure what is the best process to use to overturn >>>>> a decision on a prior review or design discussion. I guess one >>>>> option is to file a bug or an RFE, and bring it up for a discussion. >>>> >>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using >>>> WhiteBox connected to Bob's new container support code is better >>>> than exec'ing a separate process (which I'm not even sure what it >>>> does). >>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from >>> VMProps.java to determine presence of docker engine is deemed too heavy" >>> >>> Also, in current webrev, I though I could do something quick and easy >>> to address your concerns, before we discuss and agree on a final >>> solution. >>> I have timed running "docker ps" on 2 different hosts, 30 times each. >>> I saw a spread between 20ms to 150ms, mostly centered around 30ms. >>> Hence, I reduced the timeout in the VMProps.java from 10 sec to >>> 500ms. I thought 500 ms gives enough time cushion for slower test >>> nodes/hosts. >> >> I think a slowness would arise on systems that do not have docker >> installed. If I time "docker ps" on my system I get >> >> ?> time docker ps >> The program 'docker' is currently not installed.? To run 'docker' >> please ask your administrator to install the package 'docker' >> >> real??? 0m4.103s >> user??? 0m0.044s >> sys???? 0m0.036s >> >> but the second call is much quicker: >> >> ?> time docker ps >> The program 'docker' is currently not installed.? To run 'docker' >> please ask your administrator to install the package 'docker' >> >> real??? 0m0.078s >> user??? 0m0.052s >> sys???? 0m0.012s >> >> It may well depend on the user's path. If "docker" is present early in >> the path then it will be found and executed quickly, but if not >> present, or if further down the path it may take a while to occur. >> >> So I think the timeout change, as a short term proposal, may be more >> risky and so not worthwhile. >> >> I'd prefer to see the whole "docker ps" disappear altogether. Such an >> approach is not scalable if there may be different container systems. > I have discussed this topic again with Igor and Leonid, and explained > your concerns. > > > We came up with alternative ideas: > > 1. VMProps.java will check the existence of docker on a given test > machine. We will look at "/bin/docker", "/usr/bin/docker", > "/usr/local/bin/docker" > ??? For now we only support Linux x64. If any of these files exist, we > return 'true' for a corresponding at-requires property, otherwise false. > > 2. In case someone wishes to specify a custom location for a docker > binary, we thought we could use a property for that: > ???? jdk.test.lib.docker.path="" > ??? If this property is specified, VMProps.java will check the docker > at that location. > > 3. No more "docker ps" execution in VMProps.java. > > Please let me know what you think. > > > > Thank you, > Misha > >> >> Thanks, >> David >> >>> Let me know what you think. I can revert this part of the change or >>> reduce the timeout value a bit more. >>> >>> Thank you, >>> Misha >>>> >>>>> >>>>> For this change under review, if I do not hear opinions from other >>>>> reviewers, I can simply delete the print statement. >>>> >>>> Sounds good to me. >>>> >>>> Thanks, >>>> David >>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>> Please review this simple change that makes the error message in >>>>>>> jtreg-ext/requires/VMProps.java conditional >>>>>>> on java property "vmprops.trace.verbose". W/o this condition an >>>>>>> error message is printed each time any jtreg >>>>>>> test is ran, including non-docker tests. >>>>>>> >>>>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> ======= Background and details: >>>>>>> VMProps.java is called for each test run of jtreg (be it a run >>>>>>> containing a singe test or multiple tests). >>>>>>> VMProps evaluates the @requires properties. Therefore, in case of >>>>>>> this bug, any time a user runs any jtreg VM test >>>>>>> on a machine w/o an operational docker engine, this error will >>>>>>> pop up. >>>>>>> The fix makes printing of this error message conditional, off by >>>>>>> default. The message can be turend on >>>>>>> via a property vmprops.trace.verbose when detailed diagnostic >>>>>>> info is desired. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ======== Testing: >>>>>>> Local testing: >>>>>>> ?? W/o docker present: >>>>>>> ???? jtreg >>>>>>> ?????? No error message >>>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>>> ?????? Detailed error message >>>>>>> ???? jtreg DockerBasicTest.java >>>>>>> ?????? Test skipped >>>>>>> >>>>>>> ?? With docker installed but no permissions: >>>>>>> ???? jtreg >>>>>>> ?????? No error message >>>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>>> ?????? Detailed error message >>>>>>> ???? jtreg DockerBasicTest.java >>>>>>> ?????? Test skipped >>>>>>> >>>>>>> ?? With docker installed and operational: >>>>>>> ???? jtreg >>>>>>> ?????? No error message >>>>>>> ???? jtreg -Dvmprops.trace.verbose=true >>>>>>> ?????? No error message >>>>>>> ???? jtreg DockerBasicTest.java >>>>>>> ?????? Test passed >>>>>>> >>>>>>> Testing via automated distributed test system: >>>>>>> ?? - tier1 >>>>>>> ?? - docker tests >>>>>>> ?? In progress >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>> > From ioi.lam at oracle.com Tue Nov 7 03:47:58 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 6 Nov 2017 19:47:58 -0800 Subject: RFR(S) 8189840: CheckCachedResolvedReferencesApp has no cached resolved references In-Reply-To: References: Message-ID: Hi Jiangli, Just one nit - I think the method openArchiveHeapObjectsMapped should be renamed to areOpenArchiveHeapObjectsMapped to be consistent with the other methods. ??? {CC"isShared",?????????? CC"(Ljava/lang/Object;)Z", (void*)&WB_IsShared }, ??? {CC"isSharedClass",????? CC"(Ljava/lang/Class;)Z", (void*)&WB_IsSharedClass }, ??? {CC"areSharedStringsIgnored",?????????? CC"()Z", (void*)&WB_AreSharedStringsIgnored }, ??? {CC"getResolvedReferences", CC"(Ljava/lang/Class;)Ljava/lang/Object;", (void*)&WB_GetResolvedReferences}, +?? {CC"openArchiveHeapObjectsMapped",????? CC"()Z", (void*)&WB_OpenArchiveHeapObjectsMapped}, Thanks - Ioi On 11/3/17 4:46 PM, Jiangli Zhou wrote: > Please review the fix for 8189840. CheckCachedResolvedReferencesApp currently is still a closed AppCDS test. The following webrev only contains the WhiteBox change with the new API (WhiteBox.openrchiveHeapObjectsMapped()) added. CheckCachedResolvedReferencesApp calls the new API to detect if the ?open archive? heap objects are mapped successfully in the current JVM execution. CheckCachedResolvedReferencesApp test change is not included in the webrev. > > webrev: http://cr.openjdk.java.net/~jiangli/8189840/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8189840?filter=14921 > > Tested on linux-x64. Also running hs-tier1, hs-tier2 tests. > > Thanks, > Jiangli > From mikhailo.seledtsov at oracle.com Tue Nov 7 04:05:44 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Mon, 06 Nov 2017 20:05:44 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <30EFA226-1F76-4AF9-8838-6B0E2E39CF55@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <30EFA226-1F76-4AF9-8838-6B0E2E39CF55@oracle.com> Message-ID: <5A013118.6030101@oracle.com> Hi Bob, Thank you for review. Please see my comments inline: On 11/3/17, 11:04 AM, Bob Vandette wrote: > http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/CPUSetsReader.java.html > > > Not sure this is a problem but If you specify --cpuset-cpus 2-3,1 in > docker you end up with cpuset.cpus containing 1-3. > Your cpusets test cases are all increasing in order so you won?t hit > this issue. > I did not do the exhaustive test of all combinations of cpu sets. I tried to cover more common cases while balancing vs complexity and execution time. In TestCPUSets.java I read all available cpu sets from the system, flattened them into a list/set, and then created a container with a subset of one, a full subset available on the system, and rounded half of the subset. If you wish I could file an RFE for 11 to add more corner test cases for this. Please let me know. > > The read function in this file has a hard coded /sys/fs/cgroup/cpuset > directory. This may not be where this > ends up being mounted. Is this read function even used? Yes. This function is used from " TestCPUSets.testTheSet()" to read the sets. I did design the method and test such that if the file can not be read, the test case for a given set will be skipped, to avoid false failures. I did not realize the location for this directory could vary. I can introduce a property jdk.test.docker.cpuset.location that test users or test system could specify to point to the location of cpuset directory; if not specify the default value would be as it is now, /sys/fs/cgroup/cpuset. Is this reasonable? > > > http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/TestCPUAwareness.java.html > > Your test assumes that there are at least two physical processors on > the host. You might want to check first. > > 55 testAPCCombo("*0,1",* 200*1000, 100*1000, 4*1024, 2); > 56 testAPCCombo("*0,1*", 200*1000, 100*1000, 1*1024, 2) Makes sense. Will do. > > Everything else looks good, > bob. Thank you, Misha > > >> On Nov 1, 2017, at 11:11 PM, mikhailo >> wrote: >> >> Please review these tests that were developed to test JVM's container >> awareness in Docker environment. >> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >> Webrev: >> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >> Testing: >> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >> Ran the developed tests via jtreg >> Pass >> >> 2. Automated testing system - run these tests >> In progress >> >> Thank you, >> Misha >> > From mikhailo.seledtsov at oracle.com Tue Nov 7 04:06:53 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Mon, 06 Nov 2017 20:06:53 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <82e41837-1674-ee35-3c3f-a11e7540eb89@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <82e41837-1674-ee35-3c3f-a11e7540eb89@oracle.com> Message-ID: <5A01315D.20705@oracle.com> Thank you David, I will make changes and post an updated webrev. Misha On 11/6/17, 5:34 PM, David Holmes wrote: > Hi Misha, > > This seems reasonable. > > Thanks, > David > > On 7/11/2017 8:42 AM, mikhailo wrote: >> Hi David, >> >> >> On 11/05/2017 07:28 PM, David Holmes wrote: >>> Hi Misha, >>> >>> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>>> Hi David, >>>> >>>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>>> >>>> Please see my comments below. >>>> >>>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>>> Hi Misha, >>>>> >>>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>>> Hi David, >>>>>> >>>>>> Thank you for taking a look at this change. >>>>>> >>>>>> >>>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>>> Hi Misha, >>>>>>> >>>>>>> I don't see anything else in VMProps that writes anything out so >>>>>>> to me the fix is to delete the System.err.println completely. >>>>>> Removal is an option, and seems a simpler one. However, we will >>>>>> loose the diagnostic information. >>>>>> On the other hand, if an issue arises on a specific host or >>>>>> system, I can patch this file with extra trace statements, and >>>>>> diagnose the failure that way. >>>>>> If you think removing the print statement is a better option, and >>>>>> I do not hear objections or other opinions, I can just remove the >>>>>> print statement. >>>>> >>>>> Delete please :) >>>>> >>>> Done. >>>>>>> That said I did not look at this in the past but I feel the >>>>>>> whole docker test here is not really appropriate/suitable to >>>>>>> always be run on linux-x64. It has to exec another process and >>>>>>> wait up to 10 seconds to get a result! Can't this actually be >>>>>>> done via a WhiteBox check once Bob's container support updates >>>>>>> are in place? >>>>>> Actually, WhiteBox is not necessary for such check. The ability >>>>>> to run docker on a given host/node can be determined in a body of >>>>>> a jtreg test, via a test utility. In fact, that was my original >>>>>> intent. When this change was presented for a discussion, this >>>>>> aspect was debated and reviewers have reached a consensus to do >>>>>> such check in VMProps.java. We mainly discussed within VM SQE >>>>>> team. The main argument was that using @requires is a more proper >>>>>> way of "skipping" the test(s) on unsupported environments, rather >>>>>> then checking this in the body of the test and then returning. >>>>>> When using @requires, the test execution system will report the >>>>>> test as "not run". When skipping the test using "return" from the >>>>>> body of a test, the execution system will report test as "ran" >>>>>> and passed; jtreg does not yet support the "skipped" state. >>>>>> >>>>>> Unfortunately, the way our "@requires" mechanism works is that it >>>>>> runs each time prior to starting a test run, even if the test >>>>>> being run is not concerned with a specific property. It was my >>>>>> concern that this could be a burden on the test run startup, but >>>>>> I eventually agreed to the opinion of other reviewers. >>>>>> >>>>>> If you have a strong opinion or concern about this, let me know. >>>>>> However, I am not sure what is the best process to use to >>>>>> overturn a decision on a prior review or design discussion. I >>>>>> guess one option is to file a bug or an RFE, and bring it up for >>>>>> a discussion. >>>>> >>>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think >>>>> using WhiteBox connected to Bob's new container support code is >>>>> better than exec'ing a separate process (which I'm not even sure >>>>> what it does). >>>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' >>>> from VMProps.java to determine presence of docker engine is deemed >>>> too heavy" >>>> >>>> Also, in current webrev, I though I could do something quick and >>>> easy to address your concerns, before we discuss and agree on a >>>> final solution. >>>> I have timed running "docker ps" on 2 different hosts, 30 times >>>> each. I saw a spread between 20ms to 150ms, mostly centered around >>>> 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec >>>> to 500ms. I thought 500 ms gives enough time cushion for slower >>>> test nodes/hosts. >>> >>> I think a slowness would arise on systems that do not have docker >>> installed. If I time "docker ps" on my system I get >>> >>> > time docker ps >>> The program 'docker' is currently not installed. To run 'docker' >>> please ask your administrator to install the package 'docker' >>> >>> real 0m4.103s >>> user 0m0.044s >>> sys 0m0.036s >>> >>> but the second call is much quicker: >>> >>> > time docker ps >>> The program 'docker' is currently not installed. To run 'docker' >>> please ask your administrator to install the package 'docker' >>> >>> real 0m0.078s >>> user 0m0.052s >>> sys 0m0.012s >>> >>> It may well depend on the user's path. If "docker" is present early >>> in the path then it will be found and executed quickly, but if not >>> present, or if further down the path it may take a while to occur. >>> >>> So I think the timeout change, as a short term proposal, may be more >>> risky and so not worthwhile. >>> >>> I'd prefer to see the whole "docker ps" disappear altogether. Such >>> an approach is not scalable if there may be different container >>> systems. >> I have discussed this topic again with Igor and Leonid, and explained >> your concerns. >> >> >> We came up with alternative ideas: >> >> 1. VMProps.java will check the existence of docker on a given test >> machine. We will look at "/bin/docker", "/usr/bin/docker", >> "/usr/local/bin/docker" >> For now we only support Linux x64. If any of these files exist, >> we return 'true' for a corresponding at-requires property, otherwise >> false. >> >> 2. In case someone wishes to specify a custom location for a docker >> binary, we thought we could use a property for that: >> jdk.test.lib.docker.path="" >> If this property is specified, VMProps.java will check the >> docker at that location. >> >> 3. No more "docker ps" execution in VMProps.java. >> >> Please let me know what you think. >> >> >> >> Thank you, >> Misha >> >>> >>> Thanks, >>> David >>> >>>> Let me know what you think. I can revert this part of the change or >>>> reduce the timeout value a bit more. >>>> >>>> Thank you, >>>> Misha >>>>> >>>>>> >>>>>> For this change under review, if I do not hear opinions from >>>>>> other reviewers, I can simply delete the print statement. >>>>> >>>>> Sounds good to me. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>> Please review this simple change that makes the error message >>>>>>>> in jtreg-ext/requires/VMProps.java conditional >>>>>>>> on java property "vmprops.trace.verbose". W/o this condition an >>>>>>>> error message is printed each time any jtreg >>>>>>>> test is ran, including non-docker tests. >>>>>>>> >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ======= Background and details: >>>>>>>> VMProps.java is called for each test run of jtreg (be it a run >>>>>>>> containing a singe test or multiple tests). >>>>>>>> VMProps evaluates the @requires properties. Therefore, in case >>>>>>>> of this bug, any time a user runs any jtreg VM test >>>>>>>> on a machine w/o an operational docker engine, this error will >>>>>>>> pop up. >>>>>>>> The fix makes printing of this error message conditional, off >>>>>>>> by default. The message can be turend on >>>>>>>> via a property vmprops.trace.verbose when detailed diagnostic >>>>>>>> info is desired. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ======== Testing: >>>>>>>> Local testing: >>>>>>>> W/o docker present: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> Detailed error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test skipped >>>>>>>> >>>>>>>> With docker installed but no permissions: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> Detailed error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test skipped >>>>>>>> >>>>>>>> With docker installed and operational: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> No error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test passed >>>>>>>> >>>>>>>> Testing via automated distributed test system: >>>>>>>> - tier1 >>>>>>>> - docker tests >>>>>>>> In progress >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> >>>>>> >> From chris.plummer at oracle.com Tue Nov 7 05:05:57 2017 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 6 Nov 2017 21:05:57 -0800 Subject: alternate pns() debug function Message-ID: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> Hi, I'd like to add an alternate version of pns() to debug.cpp, and would like some initial comments first before filing a bug and sending out an RFR. pns() is used to print out the native stack: extern "C" void pns(void* sp, void* fp, void* pc) { // print native stack ? Command c("pns"); ? static char buf[O_BUFLEN]; ? Thread* t = Thread::current_or_null(); ? // Call generic frame constructor (certain arguments may be ignored) ? frame fr(sp, fp, pc); ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); } And here's the help output for it: ? pns(void* sp, void* fp, void* pc)? - print native (i.e. mixed) stack trace. E.g."); ?????????????????? pns($sp, $rbp, $pc) on Linux/amd64 and Solaris/amd64 or"); ?????????????????? pns($sp, $ebp, $pc) on Linux/x86 or"); ?????????????????? pns($sp, 0, $pc)??? on Linux/ppc64 or"); ?????????????????? pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC"); ???????????????? - in gdb do 'set overload-resolution off' before calling pns()"); ???????????????? - in dbx do 'frame 1' before calling pns()"); The intent is to call it from gdb, but it is potentially useful to call it from within hotspot when debugging. I'd like to propose the following alternative, since it does away with needing to pass in register values (not easy to do from C): extern "C" void pns2() { // print native stack ? Command c("pns2"); ? static char buf[O_BUFLEN]; ? Thread* t = Thread::current_or_null(); ? frame fr = os::current_frame(); ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); } I actually used this quite a bit with some recent debugging. There were places in hotspot where I wanted to know what the native stack looked like when called, and it was a lot easier to just add a call to pns2() than to setup the test to run in gdb. This seems to work on Linux/x64, macosx/64, and solaris/sparc. For some reason it crashes on Windows/x64, but I don't see any indication from the above help for pns() that it works on Windows/x64 anyway. # # A fatal error has been detected by the Java Runtime Environment: # #? EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000003bcfae1466, pid=9872, tid=12212 # # JRE version: Java(TM) SE Runtime Environment (10.0) (fastdebug build 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs, mixed mode, tiered, compressed oops, g1 gc, windows-amd64) # Problematic frame: # v? ~StubRoutines::get_previous_fp # I'm starting to suspect that os::current_frame() normally is never even called on Windows (there are calls to it in the VmError code, but I suspect they are never triggered on Windows) and that StubRoutines::get_previous_fp() is not working on Windows, but I could be wrong. Let me know what you think. thanks, Chris From mikhailo.seledtsov at oracle.com Tue Nov 7 05:20:27 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Mon, 06 Nov 2017 21:20:27 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> Message-ID: <5A01429B.4070806@oracle.com> Hi Igor, Thank you for reviewing the code. On 11/6/17, 4:31 PM, Igor Ignatyev wrote: > Hi Misha, > > I have several comments: > 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) Fixed > 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) Thank you. I will try it out. > 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >> 73 Common.run( Common.newOpts(imageName) >> 74 .addDockerOpts("--memory", valueToSet)) >> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); > or >> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >> 98 opts.classParams.add("" + sizeToAllocInMb); OK. I will unwind these expressions, and make sure indentation looks good. > 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 OK > 5. Test*: there is no need to have parentheses in '@requires (docker.support)' OK > 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it Will fix. > 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format OK. > 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? Oops. Missed it. Will add code to check it. > 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness Common.run() already checks for .shouldHaveExitValue(0) after running the container. I will apply feedback from Bob and you, and will post a new webrev. Thank you, Misha > > Thanks, > -- Igor > >> On Nov 1, 2017, at 8:11 PM, mikhailo wrote: >> >> Please review these tests that were developed to test JVM's container awareness in Docker environment. >> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >> Webrev: >> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >> Testing: >> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >> Ran the developed tests via jtreg >> Pass >> >> 2. Automated testing system - run these tests >> In progress >> >> Thank you, >> Misha >> From jiangli.zhou at oracle.com Tue Nov 7 06:24:40 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 6 Nov 2017 22:24:40 -0800 Subject: RFR(S) 8189840: CheckCachedResolvedReferencesApp has no cached resolved references In-Reply-To: References: Message-ID: <233C1DAA-2541-4239-8BD4-351B9F884DD5@oracle.com> Hi Ioi, Thanks for the suggestion. Will do. Thanks, Jiangli > On Nov 6, 2017, at 7:47 PM, Ioi Lam wrote: > > Hi Jiangli, > > Just one nit - I think the method openArchiveHeapObjectsMapped should be renamed to areOpenArchiveHeapObjectsMapped to be consistent with the other methods. > > {CC"isShared", CC"(Ljava/lang/Object;)Z", (void*)&WB_IsShared }, > {CC"isSharedClass", CC"(Ljava/lang/Class;)Z", (void*)&WB_IsSharedClass }, > {CC"areSharedStringsIgnored", CC"()Z", (void*)&WB_AreSharedStringsIgnored }, > {CC"getResolvedReferences", CC"(Ljava/lang/Class;)Ljava/lang/Object;", (void*)&WB_GetResolvedReferences}, > + {CC"openArchiveHeapObjectsMapped", CC"()Z", (void*)&WB_OpenArchiveHeapObjectsMapped}, > > Thanks > - Ioi > >> On 11/3/17 4:46 PM, Jiangli Zhou wrote: >> Please review the fix for 8189840. CheckCachedResolvedReferencesApp currently is still a closed AppCDS test. The following webrev only contains the WhiteBox change with the new API (WhiteBox.openrchiveHeapObjectsMapped()) added. CheckCachedResolvedReferencesApp calls the new API to detect if the ?open archive? heap objects are mapped successfully in the current JVM execution. CheckCachedResolvedReferencesApp test change is not included in the webrev. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8189840/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8189840?filter=14921 >> >> Tested on linux-x64. Also running hs-tier1, hs-tier2 tests. >> >> Thanks, >> Jiangli > From ioi.lam at oracle.com Tue Nov 7 07:20:31 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 6 Nov 2017 23:20:31 -0800 Subject: alternate pns() debug function In-Reply-To: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> Message-ID: <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> On 11/6/17 9:05 PM, Chris Plummer wrote: > Hi, > > I'd like to add an alternate version of pns() to debug.cpp, and would > like some initial comments first before filing a bug and sending out > an RFR. pns() is used to print out the native stack: > > extern "C" void pns(void* sp, void* fp, void* pc) { // print native stack > ? Command c("pns"); > ? static char buf[O_BUFLEN]; > ? Thread* t = Thread::current_or_null(); > ? // Call generic frame constructor (certain arguments may be ignored) > ? frame fr(sp, fp, pc); > ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); > } > > And here's the help output for it: > > ? pns(void* sp, void* fp, void* pc)? - print native (i.e. mixed) stack > trace. E.g."); > ?????????????????? pns($sp, $rbp, $pc) on Linux/amd64 and > Solaris/amd64 or"); > ?????????????????? pns($sp, $ebp, $pc) on Linux/x86 or"); > ?????????????????? pns($sp, 0, $pc)??? on Linux/ppc64 or"); > ?????????????????? pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC"); > ???????????????? - in gdb do 'set overload-resolution off' before > calling pns()"); > ???????????????? - in dbx do 'frame 1' before calling pns()"); > > The intent is to call it from gdb, but it is potentially useful to > call it from within hotspot when debugging. I'd like to propose the > following alternative, since it does away with needing to pass in > register values (not easy to do from C): > > extern "C" void pns2() { // print native stack > ? Command c("pns2"); > ? static char buf[O_BUFLEN]; > ? Thread* t = Thread::current_or_null(); > ? frame fr = os::current_frame(); > ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); > } > > I actually used this quite a bit with some recent debugging. There > were places in hotspot where I wanted to know what the native stack > looked like when called, and it was a lot easier to just add a call to > pns2() than to setup the test to run in gdb. This seems to work on > Linux/x64, macosx/64, and solaris/sparc. For some reason it crashes on > Windows/x64, but I don't see any indication from the above help for > pns() that it works on Windows/x64 anyway. > > # > # A fatal error has been detected by the Java Runtime Environment: > # > #? EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000003bcfae1466, > pid=9872, tid=12212 > # > # JRE version: Java(TM) SE Runtime Environment (10.0) (fastdebug build > 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug > 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs, mixed mode, > tiered, compressed oops, g1 gc, windows-amd64) > # Problematic frame: > # v? ~StubRoutines::get_previous_fp > # > > I'm starting to suspect that os::current_frame() normally is never > even called on Windows (there are calls to it in the VmError code, but > I suspect they are never triggered on Windows) and that > StubRoutines::get_previous_fp() is not working on Windows, but I could > be wrong. > Hi Chris, I like this new pns2() function. To make this work on Windows, you probably need to follow this pattern in vmError.cpp. ???? if (os::platform_print_native_stack(st, _context, buf, sizeof(buf))) { ?????? // We have printed the native stack in platform-specific code ?????? // Windows/x64 needs special handling. ???? } else { ?????? frame fr = _context ? os::fetch_frame_from_context(_context) ?????????????????????????? : os::current_frame(); ?????? print_native_stack(st, fr, _thread, buf, sizeof(buf)); ???? } Thanks - Ioi > Let me know what you think. > > thanks, > > Chris > From jini.george at oracle.com Tue Nov 7 09:24:54 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 7 Nov 2017 14:54:54 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> <6b40cba4-fc4e-dc54-a506-62f64599331e@oracle.com> Message-ID: <810c7db7-fe2a-45e1-92f9-2e34f1cc328a@oracle.com> Thank you very much, Sharath, for the review. My response inline: On 11/3/2017 3:44 PM, Sharath Ballal wrote: > Hi Jini, > > You have appended 'Field' for most of the SA variables. Example: > > private static CIntegerField pcOffsetField; > pcOffsetField = type.getCIntegerField("_pc_offset"); > > However that is not the case in > private static long MinChunkSizeInBytes; > MinChunkSizeInBytes = (type.getCIntegerField("_min_chunk_size_in_bytes")).getValue(); > > You may want to follow the same convention here. [Jini]: Unlike in the other cases, for MinChunkSizeInBytes, we are getting and storing the __value__ of the Field rather than the Field itself. Hence, I feel it might make more sense to not have 'Field' in this name. Thanks, Jini. > Rest of the changes look ok. > > Thanks, > Sharath (not a reviewer) > > > -----Original Message----- > From: Jini George > Sent: Thursday, November 02, 2017 10:24 AM > To: Serguei Spitsyn; serviceability-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-dev at openjdk.java.net > Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 > > Could I please get one more review done for this ? > > Thanks, > Jini. > > On 10/27/2017 9:19 PM, Jini George wrote: >> Thank you very much, Serguei. >> >> -Jini. >> >> On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Jini, >>> >>> The fix looks good to me. >>> >>> Thanks, >>> Serguei >>> >>> >>> On 10/24/17 00:31, Jini George wrote: >>>> Adding hotspot-dev too. >>>> >>>> Thanks, >>>> Jini. >>>> >>>> On 10/24/2017 12:05 PM, Jini George wrote: >>>>> Hello, >>>>> >>>>> As a part of SA next, I am working on writing a test case which >>>>> compares the fields and the types of the fields of the SA java >>>>> classes with the corresponding entries in the vmStructs tables. >>>>> This, to some extent, would help in preventing errors in SA due to >>>>> the changes in hotspot. As a precursor to this, I am in the process >>>>> of making some cleanup related changes (mostly in SA). I plan to >>>>> have the changes done in parts. For this webrev, most of the >>>>> changes are for: >>>>> >>>>> 1. Avoiding having some values being redefined in SA. Instead have >>>>> those exported through vmStructs, and read it in SA. >>>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>>> CompactibleFreeListSpace::IndexSetSize) >>>>> >>>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>>> value gets altered in hotspot and the corresponding modification >>>>> gets missed out in SA. >>>>> >>>>> 2. To remove some unused code (JNIid.java). >>>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>>> names, so that the comparison of the fields become easier. Most of >>>>> the changes belong to this group. >>>>> >>>>> Could I please get reviews done for these precursor changes ? >>>>> >>>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>>> >>>>> Thank you, >>>>> Jini. >>>>> >>> > X-sender: > X-Receiver: > X-CreatedBy: MSExchange15 > X-HeloDomain: mx.expurgate.net > X-ExtendedProps: BQBjAAoAqbJ+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAGcrAAB2AAAABQBkAA8ABAAAAEVkZ2U= > X-Source: SMTP:External Receive Connector > X-SourceIPAddress: 195.190.135.10 > X-EndOfInjectedXHeaders: 7733 > Received: from mx.expurgate.net (195.190.135.10) by smtpgw03.sap-ag.de > (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov > 2017 05:54:39 +0100 > Received: from mx.expurgate.net (helo=localhost) > by mx.expurgate.net with esmtp > id 1eA7WZ-000EZw-D4 > for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:39 +0100 > Received: from [156.151.31.69] (helo=ucsinet41.oracle.com) > by mxtls.expurgate.net with ESMTPS (eXpurgate 4.2.1-3) > (envelope-from ) > id 59faa50b-7c01-c0a8033409dd-9c971f454783-3 > for ; Thu, 02 Nov 2017 05:54:38 +0100 > Received: from aojmv0009 (unknown [137.254.59.6]) by ucsinet41.oracle.com with smtp > id 4eb7_00c7_3ecf1c93_28fd_4a9f_97f0_f31de9ddb295; > Thu, 02 Nov 2017 04:54:17 +0000 > Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) > by aojmv0009 (Postfix) with ESMTP id 15A99229CEC; > Thu, 2 Nov 2017 04:57:26 +0000 (UTC) > X-Original-To: serviceability-dev at openjdk.java.net > Delivered-To: serviceability-dev at openjdk.java.net > Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) > Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp > (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) > id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; > Thu, 02 Nov 2017 04:54:08 +0000 > Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id > vA24s8KW028461 > (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT > Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 > (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT > Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT > Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 > Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 > From: Jini George > To: "serguei.spitsyn at oracle.com" , > "serviceability-dev at openjdk.java.net" , > "hotspot-runtime-dev at openjdk.java.net" > , > References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> > > <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> > > Organization: Oracle Corporation > Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> > Date: Thu, 2 Nov 2017 10:24:03 +0530 > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 > Thunderbird/52.4.0 > MIME-Version: 1.0 > In-Reply-To: > Content-Type: text/plain; charset="utf-8"; format=flowed > Content-Language: en-US > Content-Transfer-Encoding: 7bit > X-Source-IP: userv0022.oracle.com [156.151.31.74] > X-BeenThere: serviceability-dev at openjdk.java.net > X-Mailman-Version: 2.1.17 > Precedence: list > List-Id: "Technical discussion about the development of serviceability technologies \(debugging, profiling, monitoring, and management\)" > List-Unsubscribe: , > > List-Archive: > List-Post: > List-Help: > List-Subscribe: , > > Errors-To: serviceability-dev-bounces at openjdk.java.net > Sender: serviceability-dev > X-purgate-ID: tlsNG-9b91ac/1509598479-00007C01-219CB02A/0/0 > X-purgate-type: clean > X-purgate-size: 2000 > X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de > Comment: eleven transport token MTU2LjE1MS4zMS42OQBzZXJ2aWNlYWJpbGl0eS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADgwYzQ2ZTI0N2Q2MTBlYjNiNTQ5MmE5YWE2NzJjNzJjN2NiNjcwNDQ= > X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) > X-purgate: clean > X-SAP-SPAM-Status: clean > Return-Path: serviceability-dev-bounces at openjdk.java.net > X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:39.6785 > (UTC) > X-MS-Exchange-Organization-OriginalClientIPAddress: 195.190.135.10 > X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 > X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp > X-MS-Exchange-Organization-AuthAs: Anonymous > X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp > X-MS-Exchange-Organization-Network-Message-Id: 8fba1fcd-01cd-4a67-1ee6-08d521add313 > X-MS-Exchange-Organization-OriginalSize: 7400 > X-MS-Exchange-Organization-HygienePolicy: Standard > X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: > LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11997.291|UTH=0.001|RST=11997.244|QS=0.029|CATCM-McAfeeTxRoutingAgent=0.011|CATCM=0.011|CAT=0.012;2017-11-02T08:14:36.969Z > > Could I please get one more review done for this ? > > Thanks, > Jini. > > On 10/27/2017 9:19 PM, Jini George wrote: >> Thank you very much, Serguei. >> >> -Jini. >> >> On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Jini, >>> >>> The fix looks good to me. >>> >>> Thanks, >>> Serguei >>> >>> >>> On 10/24/17 00:31, Jini George wrote: >>>> Adding hotspot-dev too. >>>> >>>> Thanks, >>>> Jini. >>>> >>>> On 10/24/2017 12:05 PM, Jini George wrote: >>>>> Hello, >>>>> >>>>> As a part of SA next, I am working on writing a test case which >>>>> compares the fields and the types of the fields of the SA java >>>>> classes with the corresponding entries in the vmStructs tables. >>>>> This, to some extent, would help in preventing errors in SA due to >>>>> the changes in hotspot. As a precursor to this, I am in the process >>>>> of making some cleanup related changes (mostly in SA). I plan to >>>>> have the changes done in parts. For this webrev, most of the >>>>> changes are for: >>>>> >>>>> 1. Avoiding having some values being redefined in SA. Instead have >>>>> those exported through vmStructs, and read it in SA. >>>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>>> CompactibleFreeListSpace::IndexSetSize) >>>>> >>>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>>> value gets altered in hotspot and the corresponding modification >>>>> gets missed out in SA. >>>>> >>>>> 2. To remove some unused code (JNIid.java). >>>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>>> names, so that the comparison of the fields become easier. Most of >>>>> the changes belong to this group. >>>>> >>>>> Could I please get reviews done for these precursor changes ? >>>>> >>>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>>> >>>>> Thank you, >>>>> Jini. >>>>> >>> > X-sender: > X-Receiver: > X-CreatedBy: MSExchange15 > X-HeloDomain: mx.expurgate.net > X-ExtendedProps: BQBjAAoAALN+Vtsd1QgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwAJAAAAQ0lBdWRpdGVkAgABBQBJAAIAAQUAYgAKAIMrAAB2AAAABQBkAA8ABAAAAEVkZ2U= > X-Source: SMTP:External Receive Connector > X-SourceIPAddress: 194.145.224.110 > X-EndOfInjectedXHeaders: 7704 > Received: from mx.expurgate.net (194.145.224.110) by smtpgw03.sap-ag.de > (155.56.66.98) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Thu, 2 Nov > 2017 05:54:46 +0100 > Received: from mx.expurgate.net (helo=localhost) > by mx.expurgate.net with esmtp > id 1eA7Wg-0004R0-FP > for axel.siebenborn at sap.com; Thu, 02 Nov 2017 05:54:46 +0100 > Received: from [141.146.126.229] (helo=acsinet41.oracle.com) > by mxtls.expurgate.net with ESMTPS (eXpurgate 4.3.1) > (envelope-from ) > id 59faa50f-65eb-c0a8029b09dd-8d927ee53db9-3 > for ; Thu, 02 Nov 2017 05:54:45 +0100 > Received: from aojmv0009 (aojmv0009.oracle.com [137.254.59.6]) by acsinet41.oracle.com with smtp > id 6461_15f0_154b24ba_5aa0_408c_88bd_1cdb37c9c8f4; > Thu, 02 Nov 2017 04:54:31 +0000 > Received: from aojmv0009.oracle.com (localhost [127.0.0.1]) > by aojmv0009 (Postfix) with ESMTP id 3B280229CF4; > Thu, 2 Nov 2017 04:57:26 +0000 (UTC) > X-Original-To: hotspot-runtime-dev at openjdk.java.net > Delivered-To: hotspot-runtime-dev at openjdk.java.net > Received: from ucsinet40.oracle.com (ucsinet40.oracle.com [156.151.31.68]) by aojmv0009 (Postfix) with ESMTP id 9CFF6229CC3; Thu, 2 Nov 2017 04:57:20 +0000 (UTC) > Received: from userp1040.oracle.com (unknown [156.151.31.81]) by ucsinet40.oracle.com with smtp > (TLS: TLSv1/SSLv3,256bits,ECDHE-RSA-AES256-GCM-SHA384) > id 37c3_8726_79982056_f63b_491d_a8be_57819e8c7dcc; > Thu, 02 Nov 2017 04:54:08 +0000 > Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id > vA24s8KW028461 > (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT > Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s84O013997 > (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 04:54:08 GMT > Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id vA24s7KK017107; Thu, 2 Nov 2017 04:54:08 GMT > Received: from [10.177.158.142] (/10.177.158.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 01 Nov 2017 21:54:07 -0700 > Subject: Re: RFR: SA: JDK-8189798: SA cleanup - part 1 > From: Jini George > To: "serguei.spitsyn at oracle.com" , > "serviceability-dev at openjdk.java.net" , > "hotspot-runtime-dev at openjdk.java.net" > , > References: <18501902-23db-de6c-b83d-640cd33df836 at oracle.com> > > <691d8166-5395-906a-4256-ef0ab2e2773a at oracle.com> > > Organization: Oracle Corporation > Message-ID: <6b40cba4-fc4e-dc54-a506-62f64599331e at oracle.com> > Date: Thu, 2 Nov 2017 10:24:03 +0530 > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 > Thunderbird/52.4.0 > MIME-Version: 1.0 > In-Reply-To: > Content-Type: text/plain; charset="utf-8"; format=flowed > Content-Language: en-US > Content-Transfer-Encoding: 7bit > X-Source-IP: userv0022.oracle.com [156.151.31.74] > X-BeenThere: hotspot-runtime-dev at openjdk.java.net > X-Mailman-Version: 2.1.17 > Precedence: list > List-Id: Technical discussion about the development of the HotSpot runtime subsystem > List-Unsubscribe: , > > List-Archive: > List-Post: > List-Help: > List-Subscribe: , > > Errors-To: hotspot-runtime-dev-bounces at openjdk.java.net > Sender: hotspot-runtime-dev > X-purgate-ID: tlsNG-81024b/1509598486-000065EB-186BA960/0/0 > X-purgate-type: clean > X-purgate-size: 2000 > X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de > Comment: eleven transport token MTQxLjE0Ni4xMjYuMjI5AGhvdHNwb3QtcnVudGltZS1kZXYtYm91bmNlc0BvcGVuamRrLmphdmEubmV0AGF4ZWwuc2llYmVuYm9ybkBzYXAuY29tADE0ZjI2NjVmYTY4ZGRiOWMwNGJiMGU0MmE2Njk2M2ZiNWQxNzAyOGI= > X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) > X-purgate: clean > X-SAP-SPAM-Status: clean > Return-Path: hotspot-runtime-dev-bounces at openjdk.java.net > X-MS-Exchange-Organization-OriginalArrivalTime: 02 Nov 2017 04:54:46.7566 > (UTC) > X-MS-Exchange-Organization-OriginalClientIPAddress: 194.145.224.110 > X-MS-Exchange-Organization-OriginalServerIPAddress: 155.56.66.98 > X-MS-Exchange-Organization-AuthSource: DEWDFE13EDGE01.wdf.sap.corp > X-MS-Exchange-Organization-AuthAs: Anonymous > X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: DEWDFE13EDGE01.wdf.sap.corp > X-MS-Exchange-Organization-Network-Message-Id: 9365fa06-08de-47df-2865-08d521add74b > X-MS-Exchange-Organization-OriginalSize: 7380 > X-MS-Exchange-Organization-HygienePolicy: Standard > X-MS-Exchange-Organization-MessageHighPrecisionLatencyInProgress: > LSRV=DEWDFE13EDGE01.wdf.sap.corp:TOTAL-EDGE=11990.260|UTH=0.001|RST=11990.260|CATCM-McAfeeTxRoutingAgent=0.004|CATCM=0.005|CAT=0.006;2017-11-02T08:14:37.016Z > > Could I please get one more review done for this ? > > Thanks, > Jini. > > On 10/27/2017 9:19 PM, Jini George wrote: >> Thank you very much, Serguei. >> >> -Jini. >> >> On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Jini, >>> >>> The fix looks good to me. >>> >>> Thanks, >>> Serguei >>> >>> >>> On 10/24/17 00:31, Jini George wrote: >>>> Adding hotspot-dev too. >>>> >>>> Thanks, >>>> Jini. >>>> >>>> On 10/24/2017 12:05 PM, Jini George wrote: >>>>> Hello, >>>>> >>>>> As a part of SA next, I am working on writing a test case which >>>>> compares the fields and the types of the fields of the SA java >>>>> classes with the corresponding entries in the vmStructs tables. >>>>> This, to some extent, would help in preventing errors in SA due to >>>>> the changes in hotspot. As a precursor to this, I am in the process >>>>> of making some cleanup related changes (mostly in SA). I plan to >>>>> have the changes done in parts. For this webrev, most of the >>>>> changes are for: >>>>> >>>>> 1. Avoiding having some values being redefined in SA. Instead have >>>>> those exported through vmStructs, and read it in SA. >>>>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>>>> CompactibleFreeListSpace::IndexSetSize) >>>>> >>>>> Redefinition of hotspot values in SA makes SA error prone, when the >>>>> value gets altered in hotspot and the corresponding modification >>>>> gets missed out in SA. >>>>> >>>>> 2. To remove some unused code (JNIid.java). >>>>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>>>> 4. Modify variable names in SA and hotspot to match the counterpart >>>>> names, so that the comparison of the fields become easier. Most of >>>>> the changes belong to this group. >>>>> >>>>> Could I please get reviews done for these precursor changes ? >>>>> >>>>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>>>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>>>> >>>>> Thank you, >>>>> Jini. >>>>> >>> From per.liden at oracle.com Tue Nov 7 09:49:39 2017 From: per.liden at oracle.com (Per Liden) Date: Tue, 7 Nov 2017 10:49:39 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> Message-ID: <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> Looks good Roman! /Per On 2017-11-06 12:13, Roman Kennke wrote: > Am 26.10.2017 um 11:43 schrieb Per Liden: >> Hi, >> >> On 2017-10-25 18:11, Erik Osterlund wrote: >> [...] >>>> So let me just put your changes up for review (again), if you don't >>>> mind: >>>> >>>> Full webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>> >> >> I like this. Just two naming suggestions: >> >> src/hotspot/share/gc/shared/gcArguments.hpp: >> >> 39 static jint create_instance(); >> 40 static bool is_initialized(); >> 41 static GCArguments* instance(); >> >> Could we call these: >> >> 39 static jint initialize(); >> 40 static bool is_initialized(); >> 41 static GCArguments* arguments(); >> >> Reasoning: initialize() maps better to its companion is_initialized(). >> GCArguments::arguments() maps better to the existing patterns we have >> with CollectedHeap::heap(). > Ok, that sounds good to me. Actually, it's almost full-circle back to my > original proposal ;-) > > Differential: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ > > Full: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ > > > Ok to push now? > > Roman From rkennke at redhat.com Tue Nov 7 11:01:15 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 7 Nov 2017 12:01:15 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> Message-ID: Hi Per & Erik, thanks for the reviews! Now I need a sponsor. Final webrev (same as before, including Reviewed-by line): http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ Thanks, Roman > Looks good Roman! > > /Per > > On 2017-11-06 12:13, Roman Kennke wrote: >> Am 26.10.2017 um 11:43 schrieb Per Liden: >>> Hi, >>> >>> On 2017-10-25 18:11, Erik Osterlund wrote: >>> [...] >>>>> So let me just put your changes up for review (again), if you don't >>>>> mind: >>>>> >>>>> Full webrev: >>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>> >>> >>> I like this. Just two naming suggestions: >>> >>> src/hotspot/share/gc/shared/gcArguments.hpp: >>> >>> ? 39?? static jint create_instance(); >>> ? 40?? static bool is_initialized(); >>> ? 41?? static GCArguments* instance(); >>> >>> Could we call these: >>> >>> ? 39?? static jint initialize(); >>> ? 40?? static bool is_initialized(); >>> ? 41?? static GCArguments* arguments(); >>> >>> Reasoning: initialize() maps better to its companion is_initialized(). >>> GCArguments::arguments() maps better to the existing patterns we have >>> with CollectedHeap::heap(). >> Ok, that sounds good to me. Actually, it's almost full-circle back to my >> original proposal ;-) >> >> Differential: >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >> >> Full: >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >> >> >> Ok to push now? >> >> Roman From thomas.stuefe at gmail.com Tue Nov 7 12:29:25 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 7 Nov 2017 13:29:25 +0100 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> Message-ID: Hi Zhengyu, sorry for dropping out of the conversation, I was out of the office for some time. The latest webrev looks good, thank you for taking my suggestion. As for the final report, I am fine with doing this in another issue, seems it is more complicated than I initially thought. Kind Regards, Thomas On Mon, Nov 6, 2017 at 9:37 PM, Zhengyu Gu wrote: > Ping ... > > I need second reviewer and sponsor. > > Latest webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > Thanks, > > -Zhengyu > > On 10/30/2017 11:29 AM, Zhengyu Gu wrote: > >> >>> >>> Yes, this is not trivial. I hacked it in (because I wanted to see your >>> output at the end of my tests) and had to disable the assert. I never ran >>> into problems. Should java threads not all be stopped at that point? >>> >>> Yes, it looks like that all Java threads are stopped at that point. >> However, I am not so sure about workers. Could there be active workers >> while JVM exiting? >> >> >> But this also can be done in a follow up issue. >>> >> >> Yes, let's address it separately. >> https://bugs.openjdk.java.net/browse/JDK-8190357 >> >> >> Updated webrev based on your early comments: >> >> Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >> >> Thanks, >> >> -Zhengyu >> >> >>> Thanks, Thomas >>> >>> >>> Thanks! Thomas >>> >>> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>> >>> >> >> wrote: >>> >>> Hi Zhengyu, >>> >>> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu >> >>> >> wrote: >>> >>> Hi Thomas, >>> >>> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>> >>> Hi Zhengyu, >>> >>> - VM_PrintMetadata still has the _unit member, but >>> I see it >>> nowhere used. >>> >>> - I would have split the summary between class and >>> non-class >>> area, that may have been more useful. But this is a >>> matter >>> of taste, I leave it up to you. >>> >>> Agree, should go little further. >>> >>> Summary: >>> Total class loaders= 754 capacity= 67528.00KB >>> used= 61216.88KB free= 9453.11KB waste= 38.73KB >>> Metadata capacity= 58780.00KB >>> used= 54053.45KB free= 4726.55KB waste= 36.38KB >>> Class data capacity= 8748.00KB >>> used= 7163.43KB free= 1584.57KB waste= 2.35KB >>> >>> For anonymous classes= 580 capacity= 2732.00KB >>> used= 1065.06KB free= 1666.94KB waste= 16.27KB >>> Metadata capacity= 2152.00KB >>> used= 738.46KB free= 1413.54KB waste= 16.27KB >>> Class data capacity= 580.00KB >>> used= 326.60KB free= 253.40KB waste= 0.00KB >>> >>> >>> >>> Nice, thanks :) >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>> >>> >> > >>> >>> >>> All is well. Note that you could reduce the footprint of >>> your change >>> somewhat by defining a structure like this: >>> >>> struct numbers_t { size_t cap; size_t used; size_t free; >>> size_t waste; } >>> >>> and replacing the many members in >>> PrintCLDMetaspaceInfoClosure with >>> instances of this structure: >>> >>> numbers_t total; >>> numbers_t total_class; >>> numbers_t total_anon; >>> numbers_t total_anon_class; >>> Depending on how far you go, your code could deflate quite >>> a bit. >>> Give the structure a default ctor and your large >>> initializer list >>> goes away; give it a printing function and you reduce >>> print-summary() to four function calls; with something like >>> an >>> numbers_t::add(size_t cap, size_t used, size_t free, size_t >>> waste) >>> you can reduce the additions in >>> PrintCLDMetaspaceInfoClosure::print_metaspace() to four >>> lines. >>> >>> Just a suggestion, I leave it up to you if you do this. >>> >>> Lines 3108 and 3129 miss each a space before BytesPerWord. >>> Change >>> looks fine otherwise. >>> >>> Thanks, Thomas >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> All else looks fine to me now. I do not need >>> another review. >>> >>> Thanks for your work, this will come in handy! >>> >>> ..Thomas >>> >>> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >>> >>> > >>> >>> >>> >>> wrote: >>> >>> Hi Thomas, >>> >>> >>> Not sure whether we misunderstood each >>> other. I was >>> thinking of >>> something in the line of: >>> >>> <<<< >>> .... >>> ClassLoader: >>> jdk/internal/reflect/DelegatingClassLoader >>> Metadata: >>> capacity: 5.00KB used: >>> 3.32KB free: 1.68KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: >>> 0.54KB free: 0.46KB >>> waste: 0.00KB >>> >>> ClassLoader: for anonymous class >>> Metadata: >>> capacity: 1.00KB used: >>> 1.00KB free: 0.00KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: >>> 0.64KB free: 0.36KB >>> waste: 0.00KB >>> .... >>> >>> Summary: >>> XX class loaders encountered, total >>> capacity: xx, >>> total waste: xx. >>> >>> Peak usage by class loader: xxxx >>> >>>> >>> >>> Added summary lines: >>> >>> Total class loaders= 56 capacity= >>> 6378.00KB >>> used= 6205.08KB free= 172.92KB waste= >>> 1.44KB >>> For anonymous classes= 54 capacity= >>> 212.00KB >>> used= 96.95KB >>> free= 115.05KB waste= 0.94KB >>> >>> >>> >>> >>> These numbers would not be interpretation, >>> just a >>> convenience to >>> the reader to get a clear picture. >>> >>> It even may be useful to separate the >>> output >>> between basic and >>> detailed mode, and in basic mode omit all >>> the >>> single class >>> loaders and just print the summary line. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >> >>>> >>> >>> >>> >>> metaspace.hpp: >>> >>> I would have preferred to keep scale_unit >>> file >>> local static >>> instead of exposing it via MetaspaceAux, >>> because it >>> does not >>> really have anything to do with metaspace. >>> >>> Fixed >>> >>> >>> Otherwise ok. >>> >>> --- >>> >>> metaspace.cpp >>> >>> - ChunkManager::print_statistics(): >>> thanks for >>> reusing this >>> function! >>> -> Scale can only be ever 1, K, M, >>> G, yes? >>> So, could you >>> add an assert at the start of the >>> function, and a >>> comment at the >>> prototype or function head? >>> -> Also, now that >>> ChunkManager::print_statistics() takes a >>> scale, could you please change the >>> printout to use >>> floats like >>> you did in >>> PrintCLDMetaspaceInfoClosure::print_metaspace()? >>> >>> Done. >>> >>> >>> Chunkmanager (non-class): >>> 0 specialized (128 bytes) chunks, total >>> 0.00KB >>> 0 small (512 bytes) chunks, total 0.00KB >>> 0 medium (8192 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> Chunkmanager (class): >>> 0 specialized (128 bytes) chunks, total >>> 0.00KB >>> 0 small (256 bytes) chunks, total 0.00KB >>> 0 medium (4096 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> >>> >>> - I am concerned about races in >>> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>> Maybe my understanding is limited. We bail >>> if >>> cld->is_unloading. >>> But nothing prevents a >>> ClassLoaderDataGraph from >>> starting to >>> unload a ClassLoaderData and tearing down >>> the >>> Metaspace after we >>> started printing it in another thread, >>> right? Also, >>> I do not see >>> any fences. >>> >>> So, I wonder whether >>> PrintCLDMetaspaceInfoClosure >>> should run in >>> lock protection. And then, I wonder if it >>> would be >>> good to >>> separate data gathering and printing. We >>> print to a >>> (at least in >>> principle) unknown output stream, which >>> may be >>> doing slow File >>> IO or even Network IO. Doing this under >>> lock >>> protection seems >>> not a good idea (but I see other places >>> where this >>> happens too). >>> >>> For an example, see >>> ChunkManager::get_statistic() vs >>> ChunkManager::print_statistic(). Could you >>> do the >>> same, have a >>> data gathering step where you collect your >>> quadrupel in a >>> variable >>> sized list of >>> structures in memory, and print it out in a >>> separate step? I >>> realize though that the effort would be >>> larger than >>> for what we >>> did in ChunkManager::get_statistic(), >>> where we only >>> fill a >>> structure. >>> >>> >>> Unlike other NMT queries, this query is >>> executed at a >>> safepoint by >>> VMThread, so it should be okay. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>> >>> >> > >>> >> >>> >> >> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> ------ >>> >>> vm_operations.hpp >>> >>> - VM_PrintMetadata : do we still need >>> _unit? >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> Thanks! >>> >>> Thomas >>> >>> >>> >>> On 10/23/2017 11:08 AM, Thomas St?fe >>> wrote: >>> >>> >>> >>> On Mon, Oct 23, 2017 at 5:03 PM, >>> Zhengyu Gu >>> >>> > >>> >>> >> >>> >> >>> > >>> >>> >>> >>> >> >> > >>> >>> >> >>> >> >>> > >>> >>> >>>>> >>> wrote: >>> >>> >>> >>> Okay. So this is >>> important for >>> understanding cases >>> where we have >>> a large number of >>> miniscule class >>> loaders, >>> each one >>> only having >>> a few numbers of chunks. >>> In this >>> case, would it be >>> useful to >>> have a running total of >>> "free" >>> and "waste", >>> too? Or >>> even better, >>> also average and peak >>> values of >>> "free" and >>> "waste" (to tell >>> apart cases where you >>> have one >>> outlier vs >>> cases where >>> just all >>> metaspaces waste a lot >>> of memory). >>> Just out of curiosity, I >>> looked at >>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> >>> and >>> it seems that "free" is >>> the >>> problem, not >>> "waste"? So I >>> guess >>> what happens is that all >>> the >>> little classloaders >>> allocate just >>> enough memory to push >>> them over >>> "_small_chunk_limit=4", >>> so they >>> allocate the first >>> medium chunk, >>> from which >>> then they >>> take a >>> small bite and leave the >>> rest >>> laying around? >>> >>> Yes. The free space of >>> anonymous >>> classes should be >>> counted >>> as waste, >>> in my opinion. I am not sure >>> if >>> everyone agrees, >>> so I took the >>> summary line out of this >>> patch. >>> >>> I would be more than happy >>> to restore >>> the summary >>> line, if >>> you find >>> it is useful :-) >>> >>> >>> I agree with you. Typically class >>> loaders >>> stop loading >>> at some >>> point, especially these tine >>> ones, and >>> often will not >>> resume >>> loading. So, effectivly, the >>> space is >>> wasted. It still >>> helps to >>> distinguish "free" in the current >>> chunks >>> from "free" in the >>> other chunks to tell this >>> situation apart >>> from a >>> situation where >>> we have just a bug in metaspace >>> chunk handling >>> preventing us to >>> use up our chunks. >>> >>> >>> The latter >>> would be an >>> important >>> addition, >>> regardless >>> if this is >>> for customers >>> or for us. >>> Chunks idling in >>> freelists can >>> make a >>> huge impact, and >>> unfortunately this >>> may happen >>> in perfectly >>> valid cases. >>> See e.g. >>> our JEP >>> "https://bugs.openjdk.java.net/browse/JDK-8166690 >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>>>". We >>> have >>> already a >>> printing >>> routine for free >>> chunks, in >>> ChunkManager::print_all_chunkmanagers(). Could >>> you add >>> this to >>> your output? >>> >>> >>> Make sense! Could >>> you >>> recommend what data and >>> format you >>> would like >>> to see? >>> >>> >>> >>> Would not >>> ChunkManager::print_all_chunkmanagers() be >>> sufficient? >>> >>> Okay, I might need to tweak >>> output >>> format. >>> >>> >>> Fine by me. Nothing depends on it >>> yet. >>> >>> Thanks, Thomas >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> ------------- >>> >>> Other than >>> above notes, >>> the changes seem >>> straighforward, I did >>> not see >>> anything wrong. >>> Minor nits: >>> >>> - >>> PrintCLDMetaspaceInfoClosure::_out, >>> _scale >>> and _unit >>> could all >>> be constant >>> (with _out >>> being a >>> outputStream* >>> const). >>> - We also do >>> not need to >> >> From zgu at redhat.com Tue Nov 7 13:02:56 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 7 Nov 2017 08:02:56 -0500 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> Message-ID: <33c94ed8-1a0e-dece-cf09-6b8c19b5563f@redhat.com> Thanks, Coleen. On 11/06/2017 08:00 PM, coleen.phillimore at oracle.com wrote: > > Zhengyu, I've reviewed the code and this looks fine. There is so > much overlapping functionality for printing sizes of metaspaces in this > file. It would be nice to have just one way of printing the metaspaces, > and have the ::dump functions go away, and have the "print" functions > consistently print sizes and statistics. There is also CLD walking and > metaspace printing in ClassLoaderDataGraph::dump which is used for > debugging. I'd be happy if that was only printing statistics about the > CLD and not details of the metaspace that they point to. This function > was only for debugging though. > > I think your version may be the most currently useful so I'm fine with > it. I hope we can have a cleanup of the rest of the printing though. > I filed JDK-8190864 (https://bugs.openjdk.java.net/browse/JDK-8190864) for cleanup. -Zhengyu > I will sponsor this. > > Thanks, > Coleen > > On 11/6/17 3:37 PM, Zhengyu Gu wrote: >> Ping ... >> >> I need second reviewer and sponsor. >> >> Latest webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >> >> Thanks, >> >> -Zhengyu >> >> On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >>>> >>>> >>>> Yes, this is not trivial. I hacked it in (because I wanted to see >>>> your output at the end of my tests) and had to disable the assert. I >>>> never ran into problems. Should java threads not all be stopped at >>>> that point? >>>> >>> Yes, it looks like that all Java threads are stopped at that point. >>> However, I am not so sure about workers. Could there be active >>> workers while JVM exiting? >>> >>> >>>> But this also can be done in a follow up issue. >>> >>> Yes, let's address it separately. >>> https://bugs.openjdk.java.net/browse/JDK-8190357 >>> >>> >>> Updated webrev based on your early comments: >>> >>> Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >>> >>> Thanks, >>> >>> -Zhengyu >>> >>>> >>>> Thanks, Thomas >>>> >>>> >>>> Thanks! Thomas >>>> >>>> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>>> >>>> >>> >> wrote: >>>> >>>> Hi Zhengyu, >>>> >>>> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu >>>> >>> >>>> >> wrote: >>>> >>>> Hi Thomas, >>>> >>>> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>>> >>>> Hi Zhengyu, >>>> >>>> - VM_PrintMetadata still has the _unit member, but >>>> I see it >>>> nowhere used. >>>> >>>> - I would have split the summary between class and >>>> non-class >>>> area, that may have been more useful. But this >>>> is a >>>> matter >>>> of taste, I leave it up to you. >>>> >>>> Agree, should go little further. >>>> >>>> Summary: >>>> Total class loaders= 754 capacity= 67528.00KB >>>> used= 61216.88KB free= 9453.11KB waste= 38.73KB >>>> Metadata capacity= 58780.00KB >>>> used= 54053.45KB free= 4726.55KB waste= 36.38KB >>>> Class data capacity= 8748.00KB >>>> used= 7163.43KB free= 1584.57KB waste= 2.35KB >>>> >>>> For anonymous classes= 580 capacity= 2732.00KB >>>> used= 1065.06KB free= 1666.94KB waste= 16.27KB >>>> Metadata capacity= 2152.00KB >>>> used= 738.46KB free= 1413.54KB waste= 16.27KB >>>> Class data capacity= 580.00KB >>>> used= 326.60KB free= 253.40KB waste= 0.00KB >>>> >>>> >>>> >>>> Nice, thanks :) >>>> >>>> Updated webrev: >>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>>> >>>> >>> > >>>> >>>> >>>> All is well. Note that you could reduce the footprint of >>>> your change >>>> somewhat by defining a structure like this: >>>> >>>> struct numbers_t { size_t cap; size_t used; size_t free; >>>> size_t waste; } >>>> >>>> and replacing the many members in >>>> PrintCLDMetaspaceInfoClosure with >>>> instances of this structure: >>>> >>>> numbers_t total; >>>> numbers_t total_class; >>>> numbers_t total_anon; >>>> numbers_t total_anon_class; >>>> Depending on how far you go, your code could deflate quite >>>> a bit. >>>> Give the structure a default ctor and your large >>>> initializer list >>>> goes away; give it a printing function and you reduce >>>> print-summary() to four function calls; with something >>>> like an >>>> numbers_t::add(size_t cap, size_t used, size_t free, >>>> size_t >>>> waste) >>>> you can reduce the additions in >>>> PrintCLDMetaspaceInfoClosure::print_metaspace() to four >>>> lines. >>>> >>>> Just a suggestion, I leave it up to you if you do this. >>>> >>>> Lines 3108 and 3129 miss each a space before BytesPerWord. >>>> Change >>>> looks fine otherwise. >>>> >>>> Thanks, Thomas >>>> >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>> All else looks fine to me now. I do not need >>>> another review. >>>> >>>> Thanks for your work, this will come in handy! >>>> >>>> ..Thomas >>>> >>>> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >>>> >>>> > >>>> >>>> >>> >>>> wrote: >>>> >>>> Hi Thomas, >>>> >>>> >>>> Not sure whether we misunderstood each >>>> other. I was >>>> thinking of >>>> something in the line of: >>>> >>>> <<<< >>>> .... >>>> ClassLoader: >>>> jdk/internal/reflect/DelegatingClassLoader >>>> Metadata: >>>> capacity: 5.00KB used: >>>> 3.32KB free: 1.68KB >>>> waste: 0.00KB >>>> Class data: >>>> capacity: 1.00KB used: >>>> 0.54KB free: 0.46KB >>>> waste: 0.00KB >>>> >>>> ClassLoader: for anonymous class >>>> Metadata: >>>> capacity: 1.00KB used: >>>> 1.00KB free: 0.00KB >>>> waste: 0.00KB >>>> Class data: >>>> capacity: 1.00KB used: >>>> 0.64KB free: 0.36KB >>>> waste: 0.00KB >>>> .... >>>> >>>> Summary: >>>> XX class loaders encountered, total >>>> capacity: xx, >>>> total waste: xx. >>>> >>>> Peak usage by class loader: xxxx >>>> >>>> >>>> >>>> Added summary lines: >>>> >>>> Total class loaders= 56 capacity= >>>> 6378.00KB >>>> used= 6205.08KB free= 172.92KB waste= >>>> 1.44KB >>>> For anonymous classes= 54 capacity= >>>> 212.00KB >>>> used= 96.95KB >>>> free= 115.05KB waste= 0.94KB >>>> >>>> >>>> >>>> >>>> These numbers would not be >>>> interpretation, >>>> just a >>>> convenience to >>>> the reader to get a clear picture. >>>> >>>> It even may be useful to separate the >>>> output >>>> between basic and >>>> detailed mode, and in basic mode omit >>>> all the >>>> single class >>>> loaders and just print the summary line. >>>> >>>> Updated webrev: >>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>>> >>>> >>>> metaspace.hpp: >>>> >>>> I would have preferred to keep >>>> scale_unit file >>>> local static >>>> instead of exposing it via MetaspaceAux, >>>> because it >>>> does not >>>> really have anything to do with >>>> metaspace. >>>> >>>> Fixed >>>> >>>> >>>> Otherwise ok. >>>> >>>> --- >>>> >>>> metaspace.cpp >>>> >>>> - ChunkManager::print_statistics(): >>>> thanks for >>>> reusing this >>>> function! >>>> -> Scale can only be ever 1, K, M, >>>> G, yes? >>>> So, could you >>>> add an assert at the start of the >>>> function, and a >>>> comment at the >>>> prototype or function head? >>>> -> Also, now that >>>> ChunkManager::print_statistics() takes a >>>> scale, could you please change the >>>> printout to use >>>> floats like >>>> you did in >>>> PrintCLDMetaspaceInfoClosure::print_metaspace()? >>>> >>>> Done. >>>> >>>> >>>> Chunkmanager (non-class): >>>> 0 specialized (128 bytes) chunks, total >>>> 0.00KB >>>> 0 small (512 bytes) chunks, total 0.00KB >>>> 0 medium (8192 bytes) chunks, total 0.00KB >>>> 0 humongous chunks, total 0.00KB >>>> total size: 0.00KB. >>>> Chunkmanager (class): >>>> 0 specialized (128 bytes) chunks, total >>>> 0.00KB >>>> 0 small (256 bytes) chunks, total 0.00KB >>>> 0 medium (4096 bytes) chunks, total 0.00KB >>>> 0 humongous chunks, total 0.00KB >>>> total size: 0.00KB. >>>> >>>> >>>> - I am concerned about races in >>>> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>>> Maybe my understanding is limited. We >>>> bail if >>>> cld->is_unloading. >>>> But nothing prevents a >>>> ClassLoaderDataGraph from >>>> starting to >>>> unload a ClassLoaderData and tearing >>>> down the >>>> Metaspace after we >>>> started printing it in another thread, >>>> right? Also, >>>> I do not see >>>> any fences. >>>> >>>> So, I wonder whether >>>> PrintCLDMetaspaceInfoClosure >>>> should run in >>>> lock protection. And then, I wonder if it >>>> would be >>>> good to >>>> separate data gathering and printing. We >>>> print to a >>>> (at least in >>>> principle) unknown output stream, >>>> which may be >>>> doing slow File >>>> IO or even Network IO. Doing this >>>> under lock >>>> protection seems >>>> not a good idea (but I see other places >>>> where this >>>> happens too). >>>> >>>> For an example, see >>>> ChunkManager::get_statistic() vs >>>> ChunkManager::print_statistic(). Could >>>> you >>>> do the >>>> same, have a >>>> data gathering step where you collect >>>> your >>>> quadrupel in a >>>> variable >>>> sized list of >>>> structures in memory, and print it out >>>> in a >>>> separate step? I >>>> realize though that the effort would be >>>> larger than >>>> for what we >>>> did in ChunkManager::get_statistic(), >>>> where we only >>>> fill a >>>> structure. >>>> >>>> >>>> Unlike other NMT queries, this query is >>>> executed at a >>>> safepoint by >>>> VMThread, so it should be okay. >>>> >>>> Updated webrev: >>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>> ------ >>>> >>>> vm_operations.hpp >>>> >>>> - VM_PrintMetadata : do we still need >>>> _unit? >>>> >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Thomas >>>> >>>> >>>> >>>> On 10/23/2017 11:08 AM, Thomas St?fe >>>> wrote: >>>> >>>> >>>> >>>> On Mon, Oct 23, 2017 at 5:03 PM, >>>> Zhengyu Gu >>>> >>>> > >>>> >>>> >> >>>> >>> >>>> > >>>> >>>> >>> >>>> >>> >>> > >>>> >>>> >> >>>> >>> >>>> > >>>> >>>> >>> >>>>> >>>> wrote: >>>> >>>> >>>> >>>> Okay. So this is >>>> important for >>>> understanding cases >>>> where we have >>>> a large number of >>>> miniscule class >>>> loaders, >>>> each one >>>> only having >>>> a few numbers of >>>> chunks. >>>> In this >>>> case, would it be >>>> useful to >>>> have a running total of >>>> "free" >>>> and "waste", >>>> too? Or >>>> even better, >>>> also average and peak >>>> values of >>>> "free" and >>>> "waste" (to tell >>>> apart cases where you >>>> have one >>>> outlier vs >>>> cases where >>>> just all >>>> metaspaces waste a lot >>>> of memory). >>>> Just out of >>>> curiosity, I >>>> looked at >>>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>> >>>> and >>>> it seems that "free" >>>> is the >>>> problem, not >>>> "waste"? So I >>>> guess >>>> what happens is that >>>> all the >>>> little classloaders >>>> allocate just >>>> enough memory to push >>>> them over >>>> "_small_chunk_limit=4", >>>> so they >>>> allocate the first >>>> medium chunk, >>>> from which >>>> then they >>>> take a >>>> small bite and leave >>>> the >>>> rest >>>> laying around? >>>> >>>> Yes. The free space of >>>> anonymous >>>> classes should be >>>> counted >>>> as waste, >>>> in my opinion. I am not >>>> sure if >>>> everyone agrees, >>>> so I took the >>>> summary line out of this >>>> patch. >>>> >>>> I would be more than happy >>>> to restore >>>> the summary >>>> line, if >>>> you find >>>> it is useful :-) >>>> >>>> >>>> I agree with you. Typically >>>> class >>>> loaders >>>> stop loading >>>> at some >>>> point, especially these tine >>>> ones, and >>>> often will not >>>> resume >>>> loading. So, effectivly, the >>>> space is >>>> wasted. It still >>>> helps to >>>> distinguish "free" in the >>>> current >>>> chunks >>>> from "free" in the >>>> other chunks to tell this >>>> situation apart >>>> from a >>>> situation where >>>> we have just a bug in metaspace >>>> chunk handling >>>> preventing us to >>>> use up our chunks. >>>> >>>> >>>> The latter >>>> would be an >>>> important >>>> addition, >>>> regardless >>>> if this is >>>> for customers >>>> or for us. >>>> Chunks idling in >>>> freelists can >>>> make a >>>> huge >>>> impact, and >>>> unfortunately this >>>> may happen >>>> in perfectly >>>> valid cases. >>>> See e.g. >>>> our JEP >>>> "https://bugs.openjdk.java.net/browse/JDK-8166690 >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>>>". We have >>>> already a printing >>>> routine for free >>>> chunks, in >>>> ChunkManager::print_all_chunkmanagers(). Could >>>> you add >>>> this to >>>> your output? >>>> >>>> >>>> Make sense! >>>> Could you >>>> recommend what data and >>>> format you >>>> would like >>>> to see? >>>> >>>> >>>> >>>> Would not >>>> ChunkManager::print_all_chunkmanagers() be >>>> sufficient? >>>> >>>> Okay, I might need to tweak >>>> output >>>> format. >>>> >>>> >>>> Fine by me. Nothing depends >>>> on it >>>> yet. >>>> >>>> Thanks, Thomas >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>> >>>> ------------- >>>> >>>> Other than >>>> above notes, >>>> the changes seem >>>> straighforward, I did >>>> not see >>>> anything wrong. >>>> Minor nits: >>>> >>>> - >>>> PrintCLDMetaspaceInfoClosure::_out, >>>> _scale >>>> and _unit >>>> could all >>>> be constant >>>> (with _out >>>> being a >>>> outputStream* >>>> const). >>>> - We also do >>>> not need to >>>> provide >>>> scale *and* >>>> unit as >>>> argument, >>>> one can be >>>> deduced from >>>> the other. >>>> This would >>>> also prevent >>>> mismatches.We >>>> do need >>>> scale here, >>>> since nmt >>>> command >>>> line can set >>>> scale and >>>> we do >>>> >>>> have to honor >>>> that. >>>> >>>> However, passing >>>> unit is >>>> ugly! I don't >>>> want to >>>> introduce >>>> inverse >>>> dependence >>>> (metaspace -> >>>> nmt), which is >>>> also ugly. >>>> >>>> >>>> Yes, that would be >>>> regrettable. >>>> >>>> Probably should >>>> move scale >>>> mapping code from >>>> NMTUtil to a >>>> common util? >>>> >>>> >>>> >>>> That would be possible, >>>> these >>>> functions >>>> (scale_from_name etc) >>>> are simple enough to be >>>> moved to >>>> a generic file. >>>> However, if you >>>> pass scale, deducing >>>> the >>>> string >>>> that goes with >>>> it is >>>> trivial and >>>> I would have no qualms >>>> doing this >>>> by hand in >>>> metaspace.cpp. But >>>> it is a matter of >>>> taste. >>>> >>>> >>>> I did not look >>>> closely >>>> at locking >>>> issues, I assume >>>> MetaspaceAux::print_metadata() is >>>> supposed to >>>> run only in >>>> Safepoints? >>>> >>>> >>>> No. >>>> sum_xxx_xxx_in_use >>>> queries are >>>> guarded by locks. >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>> Thanks, Thomas >>>> >>>> >>>> >>>> Thanks & Kind >>>> Regards, >>>> Thomas >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Oct >>>> 20, >>>> 2017 at >>>> 4:00 PM, >>>> Zhengyu Gu >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>> >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>>> >>>> >>>> > >>>> >>> >>> >> >>>> >>>> > >>>> >>> >>> >>> >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>>>> >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>> >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>>> >>>> >>>> >>>> > >>>> >>> >>> >> >>>> >>>> > >>>> >>> >>> >>> >>>> >>> >>>> > >>>> >>>> >> >>>> >>> >>> > >>>> >>>> >>>>>>> wrote: >>>> >>>> Up to now, >>>> there is >>>> little >>>> visibility >>>> into how >>>> metaspaces >>>> are used, >>>> and by whom. >>>> >>>> This proposed >>>> enhancement gives >>>> NMT the >>>> ability to >>>> report >>>> metaspace >>>> usages on >>>> per-classloader level, >>>> via a >>>> new NMT command >>>> "metadata" >>>> >>>> >>>> For example: >>>> >>>> 2496: >>>> Metaspaces: >>>> Metadata space: >>>> reserved= 8192KB >>>> committed= 5888KB >>>> Class space: >>>> reserved= 1048576KB >>>> committed= 640KB >>>> >>>> Per-classloader >>>> metadata: >>>> >>>> ClassLoader: for >>>> anonymous class >>>> Metadata capacity= 5.00KB >>>> used= 1.16KB >>>> free= >>>> 3.84KB >>>> waste= 0.05KB >>>> Class data >>>> capacity= 1.00KB >>>> used= 0.59KB >>>> free= >>>> 0.41KB >>>> waste= 0.00KB >>>> >>>> .... >>>> >>>> ClassLoader: >>>> >>>> Metadata capacity= 5640.00KB >>>> used= 5607.42KB >>>> free= >>>> 32.58KB >>>> waste= 0.46KB >>>> Class data >>>> capacity= 520.00KB >>>> used= 500.94KB >>>> free= >>>> 19.06KB >>>> waste= 0.02KB >>>> >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>>> > From thomas.stuefe at gmail.com Tue Nov 7 13:33:00 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 7 Nov 2017 14:33:00 +0100 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <33c94ed8-1a0e-dece-cf09-6b8c19b5563f@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> <33c94ed8-1a0e-dece-cf09-6b8c19b5563f@redhat.com> Message-ID: Hi Zhengyu, On Tue, Nov 7, 2017 at 2:02 PM, Zhengyu Gu wrote: > Thanks, Coleen. > > On 11/06/2017 08:00 PM, coleen.phillimore at oracle.com wrote: > >> >> Zhengyu, I've reviewed the code and this looks fine. There is so much >> overlapping functionality for printing sizes of metaspaces in this file. >> It would be nice to have just one way of printing the metaspaces, and have >> the ::dump functions go away, and have the "print" functions consistently >> print sizes and statistics. There is also CLD walking and metaspace >> printing in ClassLoaderDataGraph::dump which is used for debugging. I'd be >> happy if that was only printing statistics about the CLD and not details of >> the metaspace that they point to. This function was only for debugging >> though. >> >> I think your version may be the most currently useful so I'm fine with >> it. I hope we can have a cleanup of the rest of the printing though. >> >> I filed JDK-8190864 (https://bugs.openjdk.java.net/browse/JDK-8190864) > for cleanup. > > could you merge this please with https://bugs.openjdk.java.net/browse/JDK-8185034? Thanks! ..Thomas > -Zhengyu > > > I will sponsor this. >> >> Thanks, >> Coleen >> >> On 11/6/17 3:37 PM, Zhengyu Gu wrote: >> >>> Ping ... >>> >>> I need second reviewer and sponsor. >>> >>> Latest webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >>> >>>> >>>>> >>>>> Yes, this is not trivial. I hacked it in (because I wanted to see your >>>>> output at the end of my tests) and had to disable the assert. I never ran >>>>> into problems. Should java threads not all be stopped at that point? >>>>> >>>>> Yes, it looks like that all Java threads are stopped at that point. >>>> However, I am not so sure about workers. Could there be active workers >>>> while JVM exiting? >>>> >>>> >>>> But this also can be done in a follow up issue. >>>>> >>>> >>>> Yes, let's address it separately. >>>> https://bugs.openjdk.java.net/browse/JDK-8190357 >>>> >>>> >>>> Updated webrev based on your early comments: >>>> >>>> Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>> >>>>> Thanks, Thomas >>>>> >>>>> >>>>> Thanks! Thomas >>>>> >>>>> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>>>> >>>>> >>>> >> wrote: >>>>> >>>>> Hi Zhengyu, >>>>> >>>>> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu < >>>>> zgu at redhat.com >>>>> >>>>> >> wrote: >>>>> >>>>> Hi Thomas, >>>>> >>>>> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>>>> >>>>> Hi Zhengyu, >>>>> >>>>> - VM_PrintMetadata still has the _unit member, but >>>>> I see it >>>>> nowhere used. >>>>> >>>>> - I would have split the summary between class and >>>>> non-class >>>>> area, that may have been more useful. But this is >>>>> a >>>>> matter >>>>> of taste, I leave it up to you. >>>>> >>>>> Agree, should go little further. >>>>> >>>>> Summary: >>>>> Total class loaders= 754 capacity= 67528.00KB >>>>> used= 61216.88KB free= 9453.11KB waste= 38.73KB >>>>> Metadata capacity= 58780.00KB >>>>> used= 54053.45KB free= 4726.55KB waste= 36.38KB >>>>> Class data capacity= 8748.00KB >>>>> used= 7163.43KB free= 1584.57KB waste= 2.35KB >>>>> >>>>> For anonymous classes= 580 capacity= 2732.00KB >>>>> used= 1065.06KB free= 1666.94KB waste= 16.27KB >>>>> Metadata capacity= 2152.00KB >>>>> used= 738.46KB free= 1413.54KB waste= 16.27KB >>>>> Class data capacity= 580.00KB >>>>> used= 326.60KB free= 253.40KB waste= 0.00KB >>>>> >>>>> >>>>> >>>>> Nice, thanks :) >>>>> >>>>> Updated webrev: >>>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>>>> >>>>> >>>> > >>>>> >>>>> >>>>> All is well. Note that you could reduce the footprint of >>>>> your change >>>>> somewhat by defining a structure like this: >>>>> >>>>> struct numbers_t { size_t cap; size_t used; size_t free; >>>>> size_t waste; } >>>>> >>>>> and replacing the many members in >>>>> PrintCLDMetaspaceInfoClosure with >>>>> instances of this structure: >>>>> >>>>> numbers_t total; >>>>> numbers_t total_class; >>>>> numbers_t total_anon; >>>>> numbers_t total_anon_class; >>>>> Depending on how far you go, your code could deflate quite >>>>> a bit. >>>>> Give the structure a default ctor and your large >>>>> initializer list >>>>> goes away; give it a printing function and you reduce >>>>> print-summary() to four function calls; with something >>>>> like an >>>>> numbers_t::add(size_t cap, size_t used, size_t free, >>>>> size_t >>>>> waste) >>>>> you can reduce the additions in >>>>> PrintCLDMetaspaceInfoClosure::print_metaspace() to four >>>>> lines. >>>>> >>>>> Just a suggestion, I leave it up to you if you do this. >>>>> >>>>> Lines 3108 and 3129 miss each a space before BytesPerWord. >>>>> Change >>>>> looks fine otherwise. >>>>> >>>>> Thanks, Thomas >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> -Zhengyu >>>>> >>>>> >>>>> All else looks fine to me now. I do not need >>>>> another review. >>>>> >>>>> Thanks for your work, this will come in handy! >>>>> >>>>> ..Thomas >>>>> >>>>> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >>>>> >>>>> > >>>>> >>>>> >>> >>>>> wrote: >>>>> >>>>> Hi Thomas, >>>>> >>>>> >>>>> Not sure whether we misunderstood each >>>>> other. I was >>>>> thinking of >>>>> something in the line of: >>>>> >>>>> <<<< >>>>> .... >>>>> ClassLoader: >>>>> jdk/internal/reflect/DelegatingClassLoader >>>>> Metadata: >>>>> capacity: 5.00KB used: >>>>> 3.32KB free: 1.68KB >>>>> waste: 0.00KB >>>>> Class data: >>>>> capacity: 1.00KB used: >>>>> 0.54KB free: 0.46KB >>>>> waste: 0.00KB >>>>> >>>>> ClassLoader: for anonymous class >>>>> Metadata: >>>>> capacity: 1.00KB used: >>>>> 1.00KB free: 0.00KB >>>>> waste: 0.00KB >>>>> Class data: >>>>> capacity: 1.00KB used: >>>>> 0.64KB free: 0.36KB >>>>> waste: 0.00KB >>>>> .... >>>>> >>>>> Summary: >>>>> XX class loaders encountered, total >>>>> capacity: xx, >>>>> total waste: xx. >>>>> >>>>> Peak usage by class loader: xxxx >>>>> >>>> >>>>> >>>>> Added summary lines: >>>>> >>>>> Total class loaders= 56 capacity= >>>>> 6378.00KB >>>>> used= 6205.08KB free= 172.92KB waste= >>>>> 1.44KB >>>>> For anonymous classes= 54 capacity= >>>>> 212.00KB >>>>> used= 96.95KB >>>>> free= 115.05KB waste= 0.94KB >>>>> >>>>> >>>>> >>>>> >>>>> These numbers would not be >>>>> interpretation, >>>>> just a >>>>> convenience to >>>>> the reader to get a clear picture. >>>>> >>>>> It even may be useful to separate the >>>>> output >>>>> between basic and >>>>> detailed mode, and in basic mode omit >>>>> all the >>>>> single class >>>>> loaders and just print the summary line. >>>>> >>>>> Updated webrev: >>>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>> >>>>> >>>>> >>>>> >>>>> metaspace.hpp: >>>>> >>>>> I would have preferred to keep >>>>> scale_unit file >>>>> local static >>>>> instead of exposing it via MetaspaceAux, >>>>> because it >>>>> does not >>>>> really have anything to do with >>>>> metaspace. >>>>> >>>>> Fixed >>>>> >>>>> >>>>> Otherwise ok. >>>>> >>>>> --- >>>>> >>>>> metaspace.cpp >>>>> >>>>> - ChunkManager::print_statistics(): >>>>> thanks for >>>>> reusing this >>>>> function! >>>>> -> Scale can only be ever 1, K, M, >>>>> G, yes? >>>>> So, could you >>>>> add an assert at the start of the >>>>> function, and a >>>>> comment at the >>>>> prototype or function head? >>>>> -> Also, now that >>>>> ChunkManager::print_statistics() takes a >>>>> scale, could you please change the >>>>> printout to use >>>>> floats like >>>>> you did in >>>>> PrintCLDMetaspaceInfoClosure::print_metaspace()? >>>>> >>>>> Done. >>>>> >>>>> >>>>> Chunkmanager (non-class): >>>>> 0 specialized (128 bytes) chunks, total >>>>> 0.00KB >>>>> 0 small (512 bytes) chunks, total 0.00KB >>>>> 0 medium (8192 bytes) chunks, total 0.00KB >>>>> 0 humongous chunks, total 0.00KB >>>>> total size: 0.00KB. >>>>> Chunkmanager (class): >>>>> 0 specialized (128 bytes) chunks, total >>>>> 0.00KB >>>>> 0 small (256 bytes) chunks, total 0.00KB >>>>> 0 medium (4096 bytes) chunks, total 0.00KB >>>>> 0 humongous chunks, total 0.00KB >>>>> total size: 0.00KB. >>>>> >>>>> >>>>> - I am concerned about races in >>>>> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>>>> Maybe my understanding is limited. We >>>>> bail if >>>>> cld->is_unloading. >>>>> But nothing prevents a >>>>> ClassLoaderDataGraph from >>>>> starting to >>>>> unload a ClassLoaderData and tearing >>>>> down the >>>>> Metaspace after we >>>>> started printing it in another thread, >>>>> right? Also, >>>>> I do not see >>>>> any fences. >>>>> >>>>> So, I wonder whether >>>>> PrintCLDMetaspaceInfoClosure >>>>> should run in >>>>> lock protection. And then, I wonder if it >>>>> would be >>>>> good to >>>>> separate data gathering and printing. We >>>>> print to a >>>>> (at least in >>>>> principle) unknown output stream, which >>>>> may be >>>>> doing slow File >>>>> IO or even Network IO. Doing this under >>>>> lock >>>>> protection seems >>>>> not a good idea (but I see other places >>>>> where this >>>>> happens too). >>>>> >>>>> For an example, see >>>>> ChunkManager::get_statistic() vs >>>>> ChunkManager::print_statistic(). Could >>>>> you >>>>> do the >>>>> same, have a >>>>> data gathering step where you collect >>>>> your >>>>> quadrupel in a >>>>> variable >>>>> sized list of >>>>> structures in memory, and print it out >>>>> in a >>>>> separate step? I >>>>> realize though that the effort would be >>>>> larger than >>>>> for what we >>>>> did in ChunkManager::get_statistic(), >>>>> where we only >>>>> fill a >>>>> structure. >>>>> >>>>> >>>>> Unlike other NMT queries, this query is >>>>> executed at a >>>>> safepoint by >>>>> VMThread, so it should be okay. >>>>> >>>>> Updated webrev: >>>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>>> Thanks, >>>>> >>>>> -Zhengyu >>>>> >>>>> >>>>> ------ >>>>> >>>>> vm_operations.hpp >>>>> >>>>> - VM_PrintMetadata : do we still need >>>>> _unit? >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> -Zhengyu >>>>> >>>>> >>>>> >>>>> Thanks! >>>>> >>>>> Thomas >>>>> >>>>> >>>>> >>>>> On 10/23/2017 11:08 AM, Thomas St?fe >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Oct 23, 2017 at 5:03 PM, >>>>> Zhengyu Gu >>>>> >>>>> > >>>>> >>>>> >> >>>>> >>>> >>>>> > >>>>> >>>>> >>> >>>>> >>>> >>>> > >>>>> >>>>> >> >>>>> >>>> >>>>> > >>>>> >>>>> >>>> >>>>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> Okay. So this is >>>>> important for >>>>> understanding cases >>>>> where we have >>>>> a large number of >>>>> miniscule class >>>>> loaders, >>>>> each one >>>>> only having >>>>> a few numbers of >>>>> chunks. >>>>> In this >>>>> case, would it be >>>>> useful to >>>>> have a running total of >>>>> "free" >>>>> and "waste", >>>>> too? Or >>>>> even better, >>>>> also average and peak >>>>> values of >>>>> "free" and >>>>> "waste" (to tell >>>>> apart cases where you >>>>> have one >>>>> outlier vs >>>>> cases where >>>>> just all >>>>> metaspaces waste a lot >>>>> of memory). >>>>> Just out of curiosity, >>>>> I >>>>> looked at >>>>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>>> >>>>> and >>>>> it seems that "free" >>>>> is the >>>>> problem, not >>>>> "waste"? So I >>>>> guess >>>>> what happens is that >>>>> all the >>>>> little classloaders >>>>> allocate just >>>>> enough memory to push >>>>> them over >>>>> "_small_chunk_limit=4", >>>>> so they >>>>> allocate the first >>>>> medium chunk, >>>>> from which >>>>> then they >>>>> take a >>>>> small bite and leave >>>>> the >>>>> rest >>>>> laying around? >>>>> >>>>> Yes. The free space of >>>>> anonymous >>>>> classes should be >>>>> counted >>>>> as waste, >>>>> in my opinion. I am not >>>>> sure if >>>>> everyone agrees, >>>>> so I took the >>>>> summary line out of this >>>>> patch. >>>>> >>>>> I would be more than happy >>>>> to restore >>>>> the summary >>>>> line, if >>>>> you find >>>>> it is useful :-) >>>>> >>>>> >>>>> I agree with you. Typically >>>>> class >>>>> loaders >>>>> stop loading >>>>> at some >>>>> point, especially these tine >>>>> ones, and >>>>> often will not >>>>> resume >>>>> loading. So, effectivly, the >>>>> space is >>>>> wasted. It still >>>>> helps to >>>>> distinguish "free" in the >>>>> current >>>>> chunks >>>>> from "free" in the >>>>> other chunks to tell this >>>>> situation apart >>>>> from a >>>>> situation where >>>>> we have just a bug in metaspace >>>>> chunk handling >>>>> preventing us to >>>>> use up our chunks. >>>>> >>>>> >>>>> The latter >>>>> would be an >>>>> important >>>>> addition, >>>>> regardless >>>>> if this is >>>>> for customers >>>>> or for us. >>>>> Chunks idling in >>>>> freelists can >>>>> make a >>>>> huge impact, >>>>> and >>>>> unfortunately this >>>>> may happen >>>>> in perfectly >>>>> valid cases. >>>>> See e.g. >>>>> our JEP >>>>> "https://bugs.openjdk.java.net/browse/JDK-8166690 >>>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>>> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >> >>>>> >>>> >>>>> >>>> > >>>>> >>>> >>>>> >>>> >>>>>". We have >>>>> already a printing >>>>> routine for free >>>>> chunks, in >>>>> ChunkManager::print_all_chunkmanagers(). Could >>>>> you add >>>>> this to >>>>> your output? >>>>> >>>>> >>>>> Make sense! Could >>>>> you >>>>> recommend what data and >>>>> format you >>>>> would like >>>>> to see? >>>>> >>>>> >>>>> >>>>> Would not >>>>> ChunkManager::print_all_chunkmanagers() be >>>>> sufficient? >>>>> >>>>> Okay, I might need to tweak >>>>> output >>>>> format. >>>>> >>>>> >>>>> Fine by me. Nothing depends on >>>>> it >>>>> yet. >>>>> >>>>> Thanks, Thomas >>>>> >>>>> Thanks, >>>>> >>>>> -Zhengyu >>>>> >>>>> >>>>> >>>>> ------------- >>>>> >>>>> Other than >>>>> above notes, >>>>> the changes seem >>>>> straighforward, I did >>>>> not see >>>>> anything wrong. >>>>> Minor nits: >>>>> >>>>> - >>>>> PrintCLDMetaspaceInfoClosure::_out, >>>>> _scale >>>>> and _unit >>>>> could all >>>>> be constant >>>>> (with _out >>>>> being a >>>>> outputStream* >>>>> const). >>>>> - We also do >>>>> not need to >>>>> provide >>>>> scale *and* >>>>> unit as >>>>> argument, >>>>> one can be >>>>> deduced from >>>>> the other. >>>>> This would >>>>> also prevent >>>>> mismatches.We >>>>> do need >>>>> scale here, >>>>> since nmt >>>>> command >>>>> line can set >>>>> scale and we >>>>> do >>>>> >>>>> have to honor >>>>> that. >>>>> >>>>> However, passing >>>>> unit is >>>>> ugly! I don't >>>>> want to >>>>> introduce >>>>> inverse >>>>> dependence >>>>> (metaspace -> >>>>> nmt), which is >>>>> also ugly. >>>>> >>>>> >>>>> Yes, that would be >>>> >>>> From zgu at redhat.com Tue Nov 7 13:53:22 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 7 Nov 2017 08:53:22 -0500 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> <33c94ed8-1a0e-dece-cf09-6b8c19b5563f@redhat.com> Message-ID: <47dc5270-3cda-9ed9-1402-8217db2f4b4d@redhat.com> Hi Thomas, Sorry for filing JDK-8185034 premature. I closed it as a duplicate. Thanks, -Zhengyu > > I filed JDK-8190864 > (https://bugs.openjdk.java.net/browse/JDK-8190864 > ) for cleanup. > > > could you merge this please with > https://bugs.openjdk.java.net/browse/JDK-8185034? > > Thanks! > > ..Thomas > > -Zhengyu > > > I will sponsor this. > > Thanks, > Coleen > > On 11/6/17 3:37 PM, Zhengyu Gu wrote: > > Ping ... > > I need second reviewer and sponsor. > > Latest webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > > Thanks, > > -Zhengyu > > On 10/30/2017 11:29 AM, Zhengyu Gu wrote: > > > > Yes, this is not trivial. I hacked it in (because I > wanted to see your output at the end of my tests) > and had to disable the assert. I never ran into > problems. Should java threads not all be stopped at > that point? > > Yes, it looks like that all Java threads are stopped at > that point. However, I am not so sure about workers. > Could there be active workers while JVM exiting? > > > But this also can be done in a follow up issue. > > > Yes, let's address it separately. > https://bugs.openjdk.java.net/browse/JDK-8190357 > > > > Updated webrev based on your early comments: > > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > Thanks! Thomas > > On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe > > > > > >>> wrote: > > Hi Zhengyu, > > On Tue, Oct 24, 2017 at 8:04 PM, > Zhengyu Gu > > > >>> wrote: > > Hi Thomas, > > On 10/24/2017 12:08 PM, Thomas > St?fe wrote: > > Hi Zhengyu, > > - VM_PrintMetadata still has > the _unit member, but > I see it > nowhere used. > > - I would have split the > summary between class and > non-class > area, that may have been more > useful. But this is a > matter > of taste, I leave it up to you. > > Agree, should go little further. > > Summary: > Total class loaders= 754 > capacity= 67528.00KB > used= 61216.88KB free= 9453.11KB > waste= 38.73KB > Metadata > capacity= 58780.00KB > used= 54053.45KB free= 4726.55KB > waste= 36.38KB > Class data > capacity= 8748.00KB > used= 7163.43KB free= 1584.57KB > waste= 2.35KB > > For anonymous classes= 580 > capacity= 2732.00KB > used= 1065.06KB free= 1666.94KB > waste= 16.27KB > Metadata > capacity= 2152.00KB > used= 738.46KB free= 1413.54KB > waste= 16.27KB > Class data > capacity= 580.00KB > used= 326.60KB free= 253.40KB > waste= 0.00KB > > > > Nice, thanks :) > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ > > > > > >> > > > All is well. Note that you could > reduce the footprint of > your change > somewhat by defining a structure like > this: > > struct numbers_t { size_t cap; size_t > used; size_t free; > size_t waste; } > > and replacing the many members in > PrintCLDMetaspaceInfoClosure with > instances of this structure: > > numbers_t total; > numbers_t total_class; > numbers_t total_anon; > numbers_t total_anon_class; > Depending on how far you go, your code > could deflate quite > a bit. > Give the structure a default ctor and > your large > initializer list > goes away; give it a printing function > and you reduce > print-summary() to four function > calls; with something like an > numbers_t::add(size_t cap, size_t > used, size_t free, size_t > waste) > you can reduce the additions in > > PrintCLDMetaspaceInfoClosure::print_metaspace() to > four lines. > > Just a suggestion, I leave it up to > you if you do this. > > Lines 3108 and 3129 miss each a space > before BytesPerWord. > Change > looks fine otherwise. > > Thanks, Thomas > > > Thanks, > > -Zhengyu > > > All else looks fine to me now. > I do not need > another review. > > Thanks for your work, this > will come in handy! > > ..Thomas > > On Tue, Oct 24, 2017 at 5:08 > PM, Zhengyu Gu > > > > >> > > > >>>> > wrote: > > Hi Thomas, > > > Not sure whether we > misunderstood each > other. I was > thinking of > something in the line of: > > <<<< > .... > ClassLoader: > jdk/internal/reflect/DelegatingClassLoader > Metadata: > capacity: > 5.00KB used: 3.32KB free: > 1.68KB > waste: 0.00KB > Class data: > capacity: > 1.00KB used: 0.54KB free: > 0.46KB > waste: 0.00KB > > ClassLoader: for > anonymous class > Metadata: > capacity: > 1.00KB used: 1.00KB free: > 0.00KB > waste: 0.00KB > Class data: > capacity: > 1.00KB used: 0.64KB free: > 0.36KB > waste: 0.00KB > .... > > Summary: > XX class loaders > encountered, total > capacity: xx, > total waste: xx. > > Peak usage by class > loader: xxxx > >>>> > > Added summary lines: > > Total class loaders= > 56 capacity= 6378.00KB > used= 6205.08KB free= > 172.92KB waste= 1.44KB > For anonymous classes= > 54 capacity= 212.00KB > used= 96.95KB > free= 115.05KB waste= > 0.94KB > > > > > These numbers would > not be interpretation, > just a > convenience to > the reader to get a > clear picture. > > It even may be useful > to separate the output > between basic and > detailed mode, and in > basic mode omit all the > single class > loaders and just > print the summary line. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>> > > > > metaspace.hpp: > > I would have > preferred to keep scale_unit file > local static > instead of exposing > it via MetaspaceAux, > because it > does not > really have anything > to do with metaspace. > > Fixed > > > Otherwise ok. > > --- > > metaspace.cpp > > - > ChunkManager::print_statistics(): thanks for > reusing this > function! > -> Scale can > only be ever 1, K, M, > G, yes? > So, could you > add an assert at the > start of the > function, and a > comment at the > prototype or function > head? > -> Also, now that > > ChunkManager::print_statistics() takes a > scale, could you > please change the > printout to use > floats like > you did in > PrintCLDMetaspaceInfoClosure::print_metaspace()? > > Done. > > > Chunkmanager (non-class): > 0 specialized (128 > bytes) chunks, total 0.00KB > 0 small (512 bytes) > chunks, total 0.00KB > 0 medium (8192 bytes) > chunks, total 0.00KB > 0 humongous chunks, > total 0.00KB > total size: 0.00KB. > Chunkmanager (class): > 0 specialized (128 > bytes) chunks, total 0.00KB > 0 small (256 bytes) > chunks, total 0.00KB > 0 medium (4096 bytes) > chunks, total 0.00KB > 0 humongous chunks, > total 0.00KB > total size: 0.00KB. > > > - I am concerned > about races in > > PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). > Maybe my > understanding is limited. We bail if > cld->is_unloading. > But nothing prevents a > ClassLoaderDataGraph from > starting to > unload a > ClassLoaderData and tearing down the > Metaspace after we > started printing it > in another thread, > right? Also, > I do not see > any fences. > > So, I wonder whether > PrintCLDMetaspaceInfoClosure > should run in > lock protection. And > then, I wonder if it > would be > good to > separate data > gathering and printing. We > print to a > (at least in > principle) unknown > output stream, which may be > doing slow File > IO or even Network > IO. Doing this under lock > protection seems > not a good idea (but > I see other places > where this > happens too). > > For an example, see > ChunkManager::get_statistic() vs > > ChunkManager::print_statistic(). Could you > do the > same, have a > data gathering step > where you collect your > > quadrupel in a > variable > sized list of > structures in memory, > and print it out in a > separate step? I > realize though that > the effort would be > larger than > for what we > did in > ChunkManager::get_statistic(), > where we only > fill a > structure. > > > Unlike other NMT queries, > this query is > executed at a > safepoint by > VMThread, so it should be > okay. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ > > > > > >> > > > > > >>> > > Thanks, > > -Zhengyu > > > ------ > > vm_operations.hpp > > - VM_PrintMetadata : > do we still need _unit? > > > Thanks, > > -Zhengyu > > > > Thanks! > > Thomas > > > > On 10/23/2017 > 11:08 AM, Thomas St?fe > wrote: > > > > On Mon, Oct > 23, 2017 at 5:03 PM, > Zhengyu Gu > > > >> > > > >>> > > > > >> > > > >>>> > > > > > >> > > > >>> > > > > >> > > > >>>>>> > wrote: > > > > > Okay. So this is > important for > understanding cases > where we have > a > large number of > miniscule class > loaders, > each one > only having > a > few numbers of chunks. > In this > case, would it be > useful to > > have a running total of > "free" > and "waste", > too? Or > even better, > > also average and peak > values of > "free" and > "waste" (to tell > > apart cases where you > have one > outlier vs > cases where > just all > > metaspaces waste a lot > of memory). > > Just out of curiosity, I > looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>>> > and > it > seems that "free" is the > problem, not > "waste"? So I > guess > > what happens is that all the > little classloaders > allocate just > > enough memory to push > them over > "_small_chunk_limit=4", > so they > > allocate the first > medium chunk, > from which > then they > take a > > small bite and leave the > rest > laying around? > > Yes. > The free space of anonymous > classes should be > counted > as waste, > in my > opinion. I am not sure if > everyone agrees, > so I took the > summary > line out of this patch. > > I would > be more than happy > to restore > the summary > line, if > you find > it is > useful :-) > > > I agree with > you. Typically class > loaders > stop loading > at some > point, > especially these tine > ones, and > often will not > resume > loading. So, > effectivly, the space is > wasted. It still > helps to > distinguish > "free" in the current > chunks > from "free" in the > other chunks > to tell this > situation apart > from a > situation where > we have just > a bug in metaspace > chunk handling > preventing us to > use up our > chunks. > > > > The latter > would be an > important > addition, > regardless > if > this is > > for customers > or for us. > Chunks idling in > freelists can > make a > > huge impact, and > unfortunately this > may happen > in perfectly > > valid cases. > See e.g. > our JEP > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>>>> > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>>>>>". > We have > already a printing > routine for free > chunks, in > ChunkManager::print_all_chunkmanagers(). > Could > you add > this to > > your output? > > > > Make sense! Could you > recommend what data and > format you > > would like > > to see? > > > > > Would not > ChunkManager::print_all_chunkmanagers() be > sufficient? > > Okay, I > might need to tweak > output > format. > > > Fine by me. > Nothing depends on it > yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > > Other than > above notes, > the changes seem > straighforward, I did > > not see > anything wrong. > Minor nits: > > > - > > PrintCLDMetaspaceInfoClosure::_out, > _scale > and _unit > > could all > > be constant > (with _out > being a > outputStream* > const). > > - We also do > not need to > provide > scale *and* > unit as > > argument, > > one can be > deduced from > the other. > This would > also prevent > mismatches.We > do need > scale here, > since nmt > command > > line can set > > scale and we do > > > have to honor that. > > > However, passing > unit is > ugly! I don't > want to > introduce > inverse > dependence > (metaspace -> > nmt), which is > also ugly. > > > > Yes, that would be > > From thomas.stuefe at gmail.com Tue Nov 7 14:12:17 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 7 Nov 2017 15:12:17 +0100 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59FA1B6E.4090000@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> Message-ID: Hi Calvin, On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung wrote: > > > On 10/27/17, 12:55 AM, Thomas St?fe wrote: > > Hi Calvin, > > On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung > wrote: > >> Hi Thomas, >> >> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >> >>> Hi Calvin, >>> >>> this is a worthwhile addition, thank you for your work! >>> >> Thanks for your review. > > > Thanks for your work :) > > First off, there is another issue in file_attribute_data_to_stat(). We > actually had this issue before, but your work uncovered it: > > According to POSIX (http://pubs.opengroup.org/ > onlinepubs/009695399/basedefs/sys/stat.h.html) and every unix manpage I > looked at (solaris, linux, aix..), st_ctime is not the file creation time > but the last time file status was changed. Only Microsoft gets this wrong > in their posix layer, its stat() function returns ( > https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx) creation time in > st_ctime. > > But as os::stat() is a platform independent layer, I'd say we should > behave the same on all platforms, and that would be the Posix way. > > I did not find any code calling os::stat() and using st_ctime, so this is > still save to change for windows. (Note that I found that > perfMemory_xxx.cpp uses plain OS ::stat and misuses ctime as "creation > time" - I opened a bug for that (https://bugs.openjdk.java. > net/browse/JDK-8190260 - but that does not affect us, as they do not call > os::stat()). > > There is one more complication: in os::stat() we use plain ::stat() in the > single byte case.: > > *+ if (strlen(path) < MAX_PATH) {* > > *+ ret = ::stat(pathbuf, sbuf);**+ } else {* > > > But ::stat() on Windows is broken, as I wrote above. We should not use it, > especially not if we change the meaing of st_ctime in the double byte path. > > My suggestion would be to just always call GetFileAttributes(), also for > the single byte path. A very simple solution would be to just always go the > double byte path with UNC translation, regardless of the path length. Or, > if you are worried about performance, for paths shorter than MAX_PATH we > use GetFileAttributesA(), for longer paths GetFileAttributesW with UNC > translation. In both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you > can feed tp your file_attribute_data_to_stat() routine and have the struct > stat you return be always consistent. > > I ran into an issue with the dwFileAttributes value for a jar file. On > Windows Server O/S, the value is set to 0x2020 which is > (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) but on a > desktop O/S like Windows 7, it is set to 0x0020 which is just > FILE_ATTRIBUTE_ARCHIVE. I've fixed it in file_attribute_data_to_stat(). > > Oh.. :( good catch! But that opens a new can of worms I did not see before: FILE_ATTRIBUTE_NORMAL is documented as "A file that does not have other attributes set. This attribute is valid only when used alone." In addition to this flag, we have a multitude of things like FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this is a normal file in Unix terminology and where we would expect os::stat to return S_IFREG, but where according to the msdn doc FILE_ATTRIBUTE_NORMAL is not set. Looking at what others do in this scenario (Looked at mingw sources and at ReactOS - I am not posting any source code here for legal reasons but feel free to look for yourself), the usual way to translate WIN32_FILE_ATTRIBUTES to struct stat seems to be: "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, no dependency on FILE_ATTRIBUTE_NORMAL). I wonder about other special cases which should work too: - read-only files should be S_IFREG and !S_IWRITE, read-only directories should be S_IFDIR and !S_IWRITE. - We should tread the device root ("C:\") as a directory (so, os::stat("c:\") should return S_IFDIR). Does this work? > I've also used GetTickCounts() to measure the performance of ::stat() vs > GetFileAttributesA() plus file_attribute_data_to_stat(). There's no > difference at the ms resolution. Using the high-resolution timestamp ( > https://msdn.microsoft.com/en-us/library/windows/desktop/ > dn553408(v=vs.85).aspx), it doesn't show much difference. > > stat() seems to be implemented using FindFirstFile() (see crt sources if you can), and we call GetFileAttributesA(), so I do not think this differs much. > > But onward: > > >> >>> ========================= >>> >>> src/hotspot/os/windows/os_windows.cpp >>> >>> Could you please use wchar_t instead of WCHAR? >>> >> Fixed. >> >>> >>> --------------- >>> in os::stat(): >>> >>> This cumbersome malloc/strcpy sequence: >>> >>> ! size_t path_len = strlen(path) + 1; >>> + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), >>> mtInternal); >>> + if (pathbuf == NULL) { >>> + errno = ENOMEM; >>> return -1; >>> } >>> os::native_path(strcpy(pathbuf, path)); >>> >>> can be reduced to a simple strdup: >>> >>> char* pathbuf = os::strdup(path, mtInternal); >>> if (pathbuf == NULL) { >>> errno = ENOMEM; >>> return -1; >>> } >>> os::native_path(pathbuf); >>> >>> This also would separate strcpy() from os::native_path() which is a bit >>> unreadable. >>> >> I've made the change. >> >>> >>> -------------------- >>> >>> The single-byte path (strdup, ::stat()), together with its free(), can >>> be moved inside the (strlen(path) < MAX_PATH) condition. os::native_path >>> will not change the path length (it better not, as it operates on the input >>> buffer). That avoids having two allocations on the doublebyte path. >>> >> >> os::native_path() converts a path like C:\\\\somedir to C:\\somedir >> So I'll need the converted path in the wchar_t case too. The freeing of >> the pathbuf is being done at the end of the function or in the middle of >> the wchar_t case if there's an error. > > > Oh, you are right,of course. I missed that it this is needed for both > paths. > > >> >> >>> ----------------------- >>> >>> But seeing that translation to UNC paths is something we might want more >>> often, I would combine allocation and UNC prefix adding to one function >>> like this, to avoid the memove and increase reusability: >>> >>> // Creates an unc path from a single byte path. Return buffer is >>> allocated in C heap and needs to be freed by caller. Returns NULL on error. >>> static whchar_t* create_unc_path(const char* s) { >>> - does s start with \\?\ ? >>> - yes: >>> - os::malloc(strlen(s) + 1) and mbstowcs_s >>> - no: >>> - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to fourth >>> position in string, prefix with L"\\?\" >>> } >>> >> >> I also include the case for adding L"\\\\?\\UNC\0" at the beginning to >> be consistent with libjava/canonicalize_md.c. > > >>> We also need error checking to mbstowcs_s. >>> >> I've added assert like the following after the call: >> >> assert(converted_chars == path_len, "sanity"); > > > > create_unc_path() : > > - could you convert the /**/ to // comments, please? > > Fixed. > thanks. > > > - not sure about the assert after mbstowcs_s: if we happen to encounter an > invalid multibyte character, function will fail and return an error: > > https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx > "If mbstowcs_s encounters an invalid multibyte character, it puts 0 in > *``pReturnValue, sets the destination buffer to an empty string, sets errno > to EILSEQ, and returns EILSEQ." > > I've changed create_unc_path() so that the caller will get the errno and > removed the assert. > > static wchar_t* create_unc_path(const char* path, errno_t &err) > Okay, works for me. > > > As this is dependent on user data, we should not assert, but handle the > return code of mbstowcs_s gracefully. Same goes for > libjava/canonicalize_md.c. > > - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + 7, path, > path_len); > third parameter is wrong, as we hand in an offset into the buffer, we must > decrement the buffer size by this offset, so correct would be path_len +7 - > 7 or just path_len. > > - Same error below: + ::mbstowcs_s(&converted_chars, &wpath[4], path_len + > 4, path, path_len); > > Fixed in both places. > Okay. > > > > >> >> >>> Just for cleanliness, I would then wrap deallocation into an own >>> function. >>> >>> static viud destroy_unc_path(whchar_t* s) { os::free(s); } >>> >> I've added the destroy function. >> >>> >>> These two functions could be candidates of putting into os::win32 >>> namespace, as this form of ANSI->UNC path translation is quite common - >>> whoever wants to use the xxxxW() functions must do this. >>> >> I'm leaving them in os_windows.cpp since they're being used only within >> that file. > > > Fine by me. > > >> >> >>> ----------------------- >>> >>> FindFirstFileW: >>> >>> I am pretty sure that you can achieve the same result with >>> GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. >>> >> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very similar to >> WIN32_FIND_DATAW. >> >>> >>> It is way more straightforward to use than FindFirstFileW, as it does >>> not require you to write a callback hook. >>> >> I've switched to using GetFileAttributesExW(). > > > Thank you, this is way more readable. > > Another issue: do we still need the fix for 6539723 if we switch from > stat() to GetFileAttributesExW()? This fix looks important, but the > comment > indicates that it could break things if the original bug is not present. > > Btw, this is another strong argument for scrapping ::stat() altogether on > all code paths, not only for long input paths, because stat() and > GetFileAttributesExW() may behave > differently. So it would be good to use the same API on all code paths, in > order to get the best test coverage. > > For this round of change, I'm using GetFileAttributesEx[A|W]() for both > code paths. > > webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ > > thanks, > Calvin > Okay, all good apart from the issues mentioned above. Thanks for your work! Best Regards, Thomas > > >> >>> ------- >>> >>> eval_find_data(): This is more of a generic helper function, could you >>> rename this to something clearer, e.g. make_double_word() ? >>> >> Ok. I've renamed it. >> >>> >>> Also, setup_stat(), could this be renamed more clearly to something like >>> WIN32_FIND_DATA_to_stat? or lowercase if this bothers you :) >>> >> I'm naming the function as file_attribute_data_to_stat() to match with >> the data structure name. > > > Thanks for taking my suggestions. > > >> >> >>> ================================== >>> src/hotspot/share/classfile/classLoader.cpp >>> >>> In ClassPathDirEntry::open_stream(), I would feel better if we were >>> asserting _dir and name to be != NULL before feeding it to strlen. >>> >> I've added an assert statement. >> >>> >>> =================== >>> >> Here's an updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >> >> > Thanks! > > Thomas > > >> thanks, >> Calvin >> >> >>> Thanks! >>> >>> Thomas >>> >>> >>> >>> >>> >>> >>> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >> > wrote: >>> >>> I've reworked this fix by using the Unicode version of open >>> (wopen) to handle path name longer than max path with the path >>> prefixed to indicate an extended length path as described in [1]. >>> >>> The Unicode version of stat (wstat) doesn't work well with long >>> path [2]. So FindFirstFileW will be used.The data in >>> WIN32_FIND_DATA returned from FindFirstFileW needs to be >>> transferred to the stat structure since the caller expects a >>> return stat structure and other platforms return a stat structure. >>> >>> In classLoader.cpp, calculate the size of buffer required instead >>> of limiting it to JVM_MAXPATHLEN. >>> In os_windows.cpp, dynamically allocate buffers in os::open and >>> os::stat. >>> >>> updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>> >>> >>> It passed hs-tier2 testing using mach5. >>> >>> thanks, >>> Calvin >>> >>> [1] >>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa3 >>> 65247(v=vs.85).aspx#MAX_PATH >>> >> 365247%28v=vs.85%29.aspx#MAX_PATH> >>> >>> [2] >>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093 >>> ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>> >> 3ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral> >>> >>> >>> >>> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >>> >>> >>> Adding a warning message if the full path or the directory >>> length exceeds MAX_PATH on windows. >>> >>> Example warning messages. >>> >>> 1) The full path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: construct full path >>> name failed: total length 270 exceeds max length of 260 >>> dir >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >>> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> length 259 >>> name Hello.class length 11 >>> >>> 2) The directory path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: >>> path length 265 exceeds max length 260 >>> path >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >>> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>> >>> webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>> >>> >>> Testing: >>> JPRT (including the new test) >>> >>> thanks, >>> Calvin >>> >>> >>> > From thomas.stuefe at gmail.com Tue Nov 7 14:17:24 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 7 Nov 2017 15:17:24 +0100 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <47dc5270-3cda-9ed9-1402-8217db2f4b4d@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> <64e47f30-415c-eb0f-8df0-6db51e5f04b9@oracle.com> <33c94ed8-1a0e-dece-cf09-6b8c19b5563f@redhat.com> <47dc5270-3cda-9ed9-1402-8217db2f4b4d@redhat.com> Message-ID: Thanks, Zhengyu. On Tue, Nov 7, 2017 at 2:53 PM, Zhengyu Gu wrote: > Hi Thomas, > > Sorry for filing JDK-8185034 premature. I closed it as a duplicate. > > Thanks, > > -Zhengyu > > >> I filed JDK-8190864 >> (https://bugs.openjdk.java.net/browse/JDK-8190864 >> ) for cleanup. >> >> >> could you merge this please with https://bugs.openjdk.java.net/ >> browse/JDK-8185034? >> >> Thanks! >> >> ..Thomas >> >> -Zhengyu >> >> >> I will sponsor this. >> >> Thanks, >> Coleen >> >> On 11/6/17 3:37 PM, Zhengyu Gu wrote: >> >> Ping ... >> >> I need second reviewer and sponsor. >> >> Latest webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >> >> >> Thanks, >> >> -Zhengyu >> >> On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >> >> >> >> Yes, this is not trivial. I hacked it in (because I >> wanted to see your output at the end of my tests) >> and had to disable the assert. I never ran into >> problems. Should java threads not all be stopped at >> that point? >> >> Yes, it looks like that all Java threads are stopped at >> that point. However, I am not so sure about workers. >> Could there be active workers while JVM exiting? >> >> >> But this also can be done in a follow up issue. >> >> >> Yes, let's address it separately. >> https://bugs.openjdk.java.net/browse/JDK-8190357 >> >> >> >> Updated webrev based on your early comments: >> >> Webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ >> >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> Thanks! Thomas >> >> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >> > >> > > >> > >> > >>> wrote: >> >> Hi Zhengyu, >> >> On Tue, Oct 24, 2017 at 8:04 PM, >> Zhengyu Gu >> > zgu at redhat.com>> >> > > >>> wrote: >> >> Hi Thomas, >> >> On 10/24/2017 12:08 PM, Thomas >> St?fe wrote: >> >> Hi Zhengyu, >> >> - VM_PrintMetadata still has >> the _unit member, but >> I see it >> nowhere used. >> >> - I would have split the >> summary between class and >> non-class >> area, that may have been more >> useful. But this is a >> matter >> of taste, I leave it up to you. >> >> Agree, should go little further. >> >> Summary: >> Total class loaders= 754 >> capacity= 67528.00KB >> used= 61216.88KB free= 9453.11KB >> waste= 38.73KB >> Metadata >> capacity= 58780.00KB >> used= 54053.45KB free= 4726.55KB >> waste= 36.38KB >> Class data >> capacity= 8748.00KB >> used= 7163.43KB free= 1584.57KB >> waste= 2.35KB >> >> For anonymous classes= 580 >> capacity= 2732.00KB >> used= 1065.06KB free= 1666.94KB >> waste= 16.27KB >> Metadata >> capacity= 2152.00KB >> used= 738.46KB free= 1413.54KB >> waste= 16.27KB >> Class data >> capacity= 580.00KB >> used= 326.60KB free= 253.40KB >> waste= 0.00KB >> >> >> >> Nice, thanks :) >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> >> > > >> > >> > > >>> >> >> >> All is well. Note that you could >> reduce the footprint of >> your change >> somewhat by defining a structure like >> this: >> >> struct numbers_t { size_t cap; size_t >> used; size_t free; >> size_t waste; } >> >> and replacing the many members in >> PrintCLDMetaspaceInfoClosure with >> instances of this structure: >> >> numbers_t total; >> numbers_t total_class; >> numbers_t total_anon; >> numbers_t total_anon_class; >> Depending on how far you go, your code >> could deflate quite >> a bit. >> Give the structure a default ctor and >> your large >> initializer list >> goes away; give it a printing function >> and you reduce >> print-summary() to four function >> calls; with something like an >> numbers_t::add(size_t cap, size_t >> used, size_t free, size_t >> waste) >> you can reduce the additions in >> >> PrintCLDMetaspaceInfoClosure::print_metaspace() to >> four lines. >> >> Just a suggestion, I leave it up to >> you if you do this. >> >> Lines 3108 and 3129 miss each a space >> before BytesPerWord. >> Change >> looks fine otherwise. >> >> Thanks, Thomas >> >> >> Thanks, >> >> -Zhengyu >> >> >> All else looks fine to me now. >> I do not need >> another review. >> >> Thanks for your work, this >> will come in handy! >> >> ..Thomas >> >> On Tue, Oct 24, 2017 at 5:08 >> PM, Zhengyu Gu >> >> > >> > > >> >> > > > >> > > >>>> >> wrote: >> >> Hi Thomas, >> >> >> Not sure whether we >> misunderstood each >> other. I was >> thinking of >> something in the line >> of: >> >> <<<< >> .... >> ClassLoader: >> jdk/internal/reflect/DelegatingClassLoader >> Metadata: >> capacity: >> 5.00KB used: 3.32KB free: >> 1.68KB >> waste: 0.00KB >> Class data: >> capacity: >> 1.00KB used: 0.54KB free: >> 0.46KB >> waste: 0.00KB >> >> ClassLoader: for >> anonymous class >> Metadata: >> capacity: >> 1.00KB used: 1.00KB free: >> 0.00KB >> waste: 0.00KB >> Class data: >> capacity: >> 1.00KB used: 0.64KB free: >> 0.36KB >> waste: 0.00KB >> .... >> >> Summary: >> XX class loaders >> encountered, total >> capacity: xx, >> total waste: xx. >> >> Peak usage by class >> loader: xxxx >> >>>> >> >> Added summary lines: >> >> Total class loaders= >> 56 capacity= 6378.00KB >> used= 6205.08KB free= >> 172.92KB waste= 1.44KB >> For anonymous classes= >> 54 capacity= 212.00KB >> used= 96.95KB >> free= 115.05KB waste= >> 0.94KB >> >> >> >> >> These numbers would >> not be interpretation, >> just a >> convenience to >> the reader to get a >> clear picture. >> >> It even may be useful >> to separate the output >> between basic and >> detailed mode, and in >> basic mode omit all the >> single class >> loaders and just >> print the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zg >> u/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>>>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html >> > gu/8189688/webrev.01/index.html>>>>> >> >> >> >> metaspace.hpp: >> >> I would have >> preferred to keep scale_unit file >> local static >> instead of exposing >> it via MetaspaceAux, >> because it >> does not >> really have anything >> to do with metaspace. >> >> Fixed >> >> >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - >> ChunkManager::print_statistics(): thanks for >> reusing this >> function! >> -> Scale can >> only be ever 1, K, M, >> G, yes? >> So, could you >> add an assert at the >> start of the >> function, and a >> comment at the >> prototype or function >> head? >> -> Also, now that >> >> ChunkManager::print_statistics() takes a >> scale, could you >> please change the >> printout to use >> floats like >> you did in >> PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> Done. >> >> >> Chunkmanager (non-class): >> 0 specialized (128 >> bytes) chunks, total 0.00KB >> 0 small (512 bytes) >> chunks, total 0.00KB >> 0 medium (8192 bytes) >> chunks, total 0.00KB >> 0 humongous chunks, >> total 0.00KB >> total size: 0.00KB. >> Chunkmanager (class): >> 0 specialized (128 >> bytes) chunks, total 0.00KB >> 0 small (256 bytes) >> chunks, total 0.00KB >> 0 medium (4096 bytes) >> chunks, total 0.00KB >> 0 humongous chunks, >> total 0.00KB >> total size: 0.00KB. >> >> >> - I am concerned >> about races in >> >> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >> Maybe my >> understanding is limited. We bail if >> cld->is_unloading. >> But nothing prevents a >> ClassLoaderDataGraph from >> starting to >> unload a >> ClassLoaderData and tearing down the >> Metaspace after we >> started printing it >> in another thread, >> right? Also, >> I do not see >> any fences. >> >> So, I wonder whether >> PrintCLDMetaspaceInfoClosure >> should run in >> lock protection. And >> then, I wonder if it >> would be >> good to >> separate data >> gathering and printing. We >> print to a >> (at least in >> principle) unknown >> output stream, which may be >> doing slow File >> IO or even Network >> IO. Doing this under lock >> protection seems >> not a good idea (but >> I see other places >> where this >> happens too). >> >> For an example, see >> ChunkManager::get_statistic() vs >> >> ChunkManager::print_statistic(). Could you >> do the >> same, have a >> data gathering step >> where you collect your >> >> quadrupel in a >> variable >> sized list of >> structures in memory, >> and print it out in a >> separate step? I >> realize though that >> the effort would be >> larger than >> for what we >> did in >> ChunkManager::get_statistic(), >> where we only >> fill a >> structure. >> >> >> Unlike other NMT queries, >> this query is >> executed at a >> safepoint by >> VMThread, so it should be >> okay. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> >> > > >> > >> > > >>> >> > >> > > >> > >> > > >>>> >> >> Thanks, >> >> -Zhengyu >> >> >> ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : >> do we still need _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 >> 11:08 AM, Thomas St?fe >> wrote: >> >> >> >> On Mon, Oct >> 23, 2017 at 5:03 PM, >> Zhengyu Gu >> > > > >> > > >> >> > > > >> > > >>> >> >> > zgu at redhat.com>> >> > > >> >> > > > >> > > >>>> >> >> >> > > > >> > >> >> > > > >> > > >>> >> >> > zgu at redhat.com>> >> > > >> >> > > > >> > > >>>>>> >> wrote: >> >> >> >> >> Okay. So this is >> important for >> understanding cases >> where we have >> a >> large number of >> miniscule class >> loaders, >> each one >> only having >> a >> few numbers of chunks. >> In this >> case, would it be >> useful to >> >> have a running total of >> "free" >> and "waste", >> too? Or >> even better, >> >> also average and peak >> values of >> "free" and >> "waste" (to tell >> >> apart cases where you >> have one >> outlier vs >> cases where >> just all >> >> metaspaces waste a lot >> of memory). >> >> Just out of curiosity, I >> looked at >> http://cr.openjdk.java.net/~zg >> u/cld_metaspace/wildfly.txt >> > gu/cld_metaspace/wildfly.txt> >> > gu/cld_metaspace/wildfly.txt >> > gu/cld_metaspace/wildfly.txt>> >> > gu/cld_metaspace/wildfly.txt >> > gu/cld_metaspace/wildfly.txt> >> > gu/cld_metaspace/wildfly.txt >> > gu/cld_metaspace/wildfly.txt>>> >> > gu/cld_metaspace/wildfly.txt > > From bob.vandette at oracle.com Tue Nov 7 16:14:41 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Tue, 7 Nov 2017 11:14:41 -0500 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A013118.6030101@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <30EFA226-1F76-4AF9-8838-6B0E2E39CF55@oracle.com> <5A013118.6030101@oracle.com> Message-ID: <9D64FC51-D58C-47CB-ADC0-5187979B0B91@oracle.com> > On Nov 6, 2017, at 11:05 PM, Mikhailo Seledtsov wrote: > > Hi Bob, > > Thank you for review. Please see my comments inline: > > On 11/3/17, 11:04 AM, Bob Vandette wrote: >> http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/CPUSetsReader.java.html >> >> Not sure this is a problem but If you specify --cpuset-cpus 2-3,1 in docker you end up with cpuset.cpus containing 1-3. >> Your cpusets test cases are all increasing in order so you won?t hit this issue. >> > I did not do the exhaustive test of all combinations of cpu sets. I tried to cover more common cases while balancing vs complexity and execution time. > In TestCPUSets.java I read all available cpu sets from the system, flattened them into a list/set, and then created a container with > a subset of one, a full subset available on the system, and rounded half of the subset. > If you wish I could file an RFE for 11 to add more corner test cases for this. Please let me know. >> >> The read function in this file has a hard coded /sys/fs/cgroup/cpuset directory. This may not be where this >> ends up being mounted. Is this read function even used? > Yes. This function is used from " TestCPUSets.testTheSet()" to read the sets. > I did design the method and test such that if the file can not be read, the test case for a given set will be skipped, to avoid false failures. > I did not realize the location for this directory could vary. I can introduce a property jdk.test.docker.cpuset.location that test users or test system could specify to point to the location of cpuset directory; if not specify the default value would be as it is now, /sys/fs/cgroup/cpuset. Is this reasonable? I think this needs to be more dynamic. One of my internal systems has, cpuset.cpus is in /cgroup/cpuset. Who knows how the mach5 systems are configured. Maybe you can use /proc/self/status and extract Cpus_allowed_list? >> >> >> http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/TestCPUAwareness.java.html >> >> Your test assumes that there are at least two physical processors on the host. You might want to check first. >> >> 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); >> 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2) > Makes sense. Will do. >> >> Everything else looks good, >> bob. > Thank you, Once you get all of your changes done, maybe you can commit them in a local forest and do an hg export of the changeset. I can then import it and push it for you with my changes. Assuming this works, this should preserve you as the committer and allow the push to have both implementation and tests. Bob. > > Misha >> >> >>> On Nov 1, 2017, at 11:11 PM, mikhailo wrote: >>> >>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>> Webrev: >>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>> Testing: >>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>> Ran the developed tests via jtreg >>> Pass >>> >>> 2. Automated testing system - run these tests >>> In progress >>> >>> Thank you, >>> Misha >>> >> From bob.vandette at oracle.com Tue Nov 7 17:04:12 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Tue, 7 Nov 2017 12:04:12 -0500 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> Message-ID: > On Nov 6, 2017, at 5:42 PM, mikhailo wrote: > > Hi David, > > > On 11/05/2017 07:28 PM, David Holmes wrote: >> Hi Misha, >> >> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>> Hi David, >>> >>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>> >>> Please see my comments below. >>> >>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>> Hi Misha, >>>> >>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>> Hi David, >>>>> >>>>> Thank you for taking a look at this change. >>>>> >>>>> >>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>> Hi Misha, >>>>>> >>>>>> I don't see anything else in VMProps that writes anything out so to me the fix is to delete the System.err.println completely. >>>>> Removal is an option, and seems a simpler one. However, we will loose the diagnostic information. >>>>> On the other hand, if an issue arises on a specific host or system, I can patch this file with extra trace statements, and diagnose the failure that way. >>>>> If you think removing the print statement is a better option, and I do not hear objections or other opinions, I can just remove the print statement. >>>> >>>> Delete please :) >>>> >>> Done. >>>>>> That said I did not look at this in the past but I feel the whole docker test here is not really appropriate/suitable to always be run on linux-x64. It has to exec another process and wait up to 10 seconds to get a result! Can't this actually be done via a WhiteBox check once Bob's container support updates are in place? >>>>> Actually, WhiteBox is not necessary for such check. The ability to run docker on a given host/node can be determined in a body of a jtreg test, via a test utility. In fact, that was my original intent. When this change was presented for a discussion, this aspect was debated and reviewers have reached a consensus to do such check in VMProps.java. We mainly discussed within VM SQE team. The main argument was that using @requires is a more proper way of "skipping" the test(s) on unsupported environments, rather then checking this in the body of the test and then returning. When using @requires, the test execution system will report the test as "not run". When skipping the test using "return" from the body of a test, the execution system will report test as "ran" and passed; jtreg does not yet support the "skipped" state. >>>>> >>>>> Unfortunately, the way our "@requires" mechanism works is that it runs each time prior to starting a test run, even if the test being run is not concerned with a specific property. It was my concern that this could be a burden on the test run startup, but I eventually agreed to the opinion of other reviewers. >>>>> >>>>> If you have a strong opinion or concern about this, let me know. However, I am not sure what is the best process to use to overturn a decision on a prior review or design discussion. I guess one option is to file a bug or an RFE, and bring it up for a discussion. >>>> >>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using WhiteBox connected to Bob's new container support code is better than exec'ing a separate process (which I'm not even sure what it does). >>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy" >>> >>> Also, in current webrev, I though I could do something quick and easy to address your concerns, before we discuss and agree on a final solution. >>> I have timed running "docker ps" on 2 different hosts, 30 times each. I saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec to 500ms. I thought 500 ms gives enough time cushion for slower test nodes/hosts. >> >> I think a slowness would arise on systems that do not have docker installed. If I time "docker ps" on my system I get >> >> > time docker ps >> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >> >> real 0m4.103s >> user 0m0.044s >> sys 0m0.036s >> >> but the second call is much quicker: >> >> > time docker ps >> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >> >> real 0m0.078s >> user 0m0.052s >> sys 0m0.012s >> >> It may well depend on the user's path. If "docker" is present early in the path then it will be found and executed quickly, but if not present, or if further down the path it may take a while to occur. >> >> So I think the timeout change, as a short term proposal, may be more risky and so not worthwhile. >> >> I'd prefer to see the whole "docker ps" disappear altogether. Such an approach is not scalable if there may be different container systems. > I have discussed this topic again with Igor and Leonid, and explained your concerns. > > > We came up with alternative ideas: > > 1. VMProps.java will check the existence of docker on a given test machine. We will look at "/bin/docker", "/usr/bin/docker", "/usr/local/bin/docker" > For now we only support Linux x64. If any of these files exist, we return 'true' for a corresponding at-requires property, otherwise false. > > 2. In case someone wishes to specify a custom location for a docker binary, we thought we could use a property for that: > jdk.test.lib.docker.path="" > If this property is specified, VMProps.java will check the docker at that location. > > 3. No more "docker ps" execution in VMProps.java. > > Please let me know what you think. Just having checking for the existence of /bin/docker doesn?t means that it?s properly configured for use by any test process. There?s permission setup, possible proxy setup etc. We had issues with our internal testing systems not having the tools we?ve needed and it?s a pain to have to go to each system in order to clear the way to successfully run all the tests. This is probably breaking new ground, but can we cache the result of ?docker ps? in /tmp in order to eliminate the performance overhead for all but the first time a new user runs these tests? Bob. > > > > Thank you, > Misha > >> >> Thanks, >> David >> >>> Let me know what you think. I can revert this part of the change or reduce the timeout value a bit more. >>> >>> Thank you, >>> Misha >>>> >>>>> >>>>> For this change under review, if I do not hear opinions from other reviewers, I can simply delete the print statement. >>>> >>>> Sounds good to me. >>>> >>>> Thanks, >>>> David >>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>> Please review this simple change that makes the error message in jtreg-ext/requires/VMProps.java conditional >>>>>>> on java property "vmprops.trace.verbose". W/o this condition an error message is printed each time any jtreg >>>>>>> test is ran, including non-docker tests. >>>>>>> >>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> ======= Background and details: >>>>>>> VMProps.java is called for each test run of jtreg (be it a run containing a singe test or multiple tests). >>>>>>> VMProps evaluates the @requires properties. Therefore, in case of this bug, any time a user runs any jtreg VM test >>>>>>> on a machine w/o an operational docker engine, this error will pop up. >>>>>>> The fix makes printing of this error message conditional, off by default. The message can be turend on >>>>>>> via a property vmprops.trace.verbose when detailed diagnostic info is desired. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ======== Testing: >>>>>>> Local testing: >>>>>>> W/o docker present: >>>>>>> jtreg >>>>>>> No error message >>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>> Detailed error message >>>>>>> jtreg DockerBasicTest.java >>>>>>> Test skipped >>>>>>> >>>>>>> With docker installed but no permissions: >>>>>>> jtreg >>>>>>> No error message >>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>> Detailed error message >>>>>>> jtreg DockerBasicTest.java >>>>>>> Test skipped >>>>>>> >>>>>>> With docker installed and operational: >>>>>>> jtreg >>>>>>> No error message >>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>> No error message >>>>>>> jtreg DockerBasicTest.java >>>>>>> Test passed >>>>>>> >>>>>>> Testing via automated distributed test system: >>>>>>> - tier1 >>>>>>> - docker tests >>>>>>> In progress >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>> > From calvin.cheung at oracle.com Tue Nov 7 17:24:39 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 07 Nov 2017 09:24:39 -0800 Subject: RFR(S) 8189840: CheckCachedResolvedReferencesApp has no cached resolved references In-Reply-To: <233C1DAA-2541-4239-8BD4-351B9F884DD5@oracle.com> References: <233C1DAA-2541-4239-8BD4-351B9F884DD5@oracle.com> Message-ID: <5A01EC57.1080908@oracle.com> Hi Jiangli, I agree with Ioi's suggestion on the method name. Looks good otherwise. thanks, Calvin On 11/6/17, 10:24 PM, Jiangli Zhou wrote: > Hi Ioi, > > Thanks for the suggestion. Will do. > > Thanks, > Jiangli > >> On Nov 6, 2017, at 7:47 PM, Ioi Lam wrote: >> >> Hi Jiangli, >> >> Just one nit - I think the method openArchiveHeapObjectsMapped should be renamed to areOpenArchiveHeapObjectsMapped to be consistent with the other methods. >> >> {CC"isShared", CC"(Ljava/lang/Object;)Z", (void*)&WB_IsShared }, >> {CC"isSharedClass", CC"(Ljava/lang/Class;)Z", (void*)&WB_IsSharedClass }, >> {CC"areSharedStringsIgnored", CC"()Z", (void*)&WB_AreSharedStringsIgnored }, >> {CC"getResolvedReferences", CC"(Ljava/lang/Class;)Ljava/lang/Object;", (void*)&WB_GetResolvedReferences}, >> + {CC"openArchiveHeapObjectsMapped", CC"()Z", (void*)&WB_OpenArchiveHeapObjectsMapped}, >> >> Thanks >> - Ioi >> >>> On 11/3/17 4:46 PM, Jiangli Zhou wrote: >>> Please review the fix for 8189840. CheckCachedResolvedReferencesApp currently is still a closed AppCDS test. The following webrev only contains the WhiteBox change with the new API (WhiteBox.openrchiveHeapObjectsMapped()) added. CheckCachedResolvedReferencesApp calls the new API to detect if the ?open archive? heap objects are mapped successfully in the current JVM execution. CheckCachedResolvedReferencesApp test change is not included in the webrev. >>> >>> webrev: http://cr.openjdk.java.net/~jiangli/8189840/webrev.00/ >>> bug: https://bugs.openjdk.java.net/browse/JDK-8189840?filter=14921 >>> >>> Tested on linux-x64. Also running hs-tier1, hs-tier2 tests. >>> >>> Thanks, >>> Jiangli From jiangli.zhou at oracle.com Tue Nov 7 17:33:06 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 7 Nov 2017 09:33:06 -0800 Subject: RFR(S) 8189840: CheckCachedResolvedReferencesApp has no cached resolved references In-Reply-To: <5A01EC57.1080908@oracle.com> References: <233C1DAA-2541-4239-8BD4-351B9F884DD5@oracle.com> <5A01EC57.1080908@oracle.com> Message-ID: Thanks for the review, Calvin! Jiangli > On Nov 7, 2017, at 9:24 AM, Calvin Cheung wrote: > > Hi Jiangli, > > I agree with Ioi's suggestion on the method name. Looks good otherwise. > > thanks, > Calvin > > On 11/6/17, 10:24 PM, Jiangli Zhou wrote: >> Hi Ioi, >> >> Thanks for the suggestion. Will do. >> >> Thanks, >> Jiangli >> >>> On Nov 6, 2017, at 7:47 PM, Ioi Lam wrote: >>> >>> Hi Jiangli, >>> >>> Just one nit - I think the method openArchiveHeapObjectsMapped should be renamed to areOpenArchiveHeapObjectsMapped to be consistent with the other methods. >>> >>> {CC"isShared", CC"(Ljava/lang/Object;)Z", (void*)&WB_IsShared }, >>> {CC"isSharedClass", CC"(Ljava/lang/Class;)Z", (void*)&WB_IsSharedClass }, >>> {CC"areSharedStringsIgnored", CC"()Z", (void*)&WB_AreSharedStringsIgnored }, >>> {CC"getResolvedReferences", CC"(Ljava/lang/Class;)Ljava/lang/Object;", (void*)&WB_GetResolvedReferences}, >>> + {CC"openArchiveHeapObjectsMapped", CC"()Z", (void*)&WB_OpenArchiveHeapObjectsMapped}, >>> >>> Thanks >>> - Ioi >>> >>>> On 11/3/17 4:46 PM, Jiangli Zhou wrote: >>>> Please review the fix for 8189840. CheckCachedResolvedReferencesApp currently is still a closed AppCDS test. The following webrev only contains the WhiteBox change with the new API (WhiteBox.openrchiveHeapObjectsMapped()) added. CheckCachedResolvedReferencesApp calls the new API to detect if the ?open archive? heap objects are mapped successfully in the current JVM execution. CheckCachedResolvedReferencesApp test change is not included in the webrev. >>>> >>>> webrev: http://cr.openjdk.java.net/~jiangli/8189840/webrev.00/ >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8189840?filter=14921 >>>> >>>> Tested on linux-x64. Also running hs-tier1, hs-tier2 tests. >>>> >>>> Thanks, >>>> Jiangli From ioi.lam at oracle.com Tue Nov 7 17:33:07 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 7 Nov 2017 09:33:07 -0800 Subject: RFR[XXS]: 8186778 Make obsolete VM options for shared region size control Message-ID: <26645083-96ad-9fb8-9370-6dbf77319b68@oracle.com> Hi, Please review this very small change. These VM arguments are no longer used when creating a CDS archive, so they should be declared obsolete: https://bugs.openjdk.java.net/browse/JDK-8186778 http://cr.openjdk.java.net/~iklam/jdk10/8186778-make-onsolete-shared-region-size-args.v01/ Thanks - Ioi ----- $ java -XX:SharedReadWriteSize=123? -XX:SharedReadOnlySize=123 -XX:SharedMiscDataSize=123 -XX:SharedMiscCodeSize=123 -version Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedReadWriteSize; support was removed in 10.0 Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedReadOnlySize; support was removed in 10.0 Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedMiscDataSize; support was removed in 10.0 Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedMiscCodeSize; support was removed in 10.0 java version "10-internal" Java(TM) SE Runtime Environment (build 10-internal+0-adhoc.iklam.open) Java HotSpot(TM) 64-Bit Server VM (build 10-internal+0-adhoc.iklam.open, mixed mode) From vladimir.kozlov at oracle.com Tue Nov 7 18:56:36 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 7 Nov 2017 10:56:36 -0800 Subject: [10] RFR(S): 8190797: OSR compilation fails with "assert(__the_thread__->can_call_java()) failed: can not load classes with compiler thread" In-Reply-To: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> References: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> Message-ID: <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> Looks good. Thanks, Vladimir On 11/7/17 4:04 AM, Tobias Hartmann wrote: > Hi, > > please review the following patch: > https://bugs.openjdk.java.net/browse/JDK-8190797 > http://cr.openjdk.java.net/~thartmann/8190797/webrev.00/ > > When oop map creation fails, we try to throw a LinkageError to propagate the error message. If this happens in the > compiler thread (for example, during OSR compilation), we fail because a compiler thread cannot initialize an exception > object. > > I've fixed this by bailing out with a meaningful message in case !Thread::can_call_java(). Please note that even if we > are able to instantiate an exception, we will still fail with ShouldNotReachHere because compute_map(TRAPS) is called > with CATCH (see comments in the bug for a detailed explanation). This fix is not about changing this behavior but to > fail with a meaningful error message during compilation. This should only happen if something is seriously broken (for > example, incorrect bytecode with -noverify, see TestLinkageErrorInGenerateOopMap). In this case we would probably hit > other issues as well if we would continue execution. > > Thanks, > Tobias > From mikhailo.seledtsov at oracle.com Tue Nov 7 19:25:20 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Tue, 7 Nov 2017 11:25:20 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> Message-ID: On 11/07/2017 09:04 AM, Bob Vandette wrote: >> On Nov 6, 2017, at 5:42 PM, mikhailo wrote: >> >> Hi David, >> >> >> On 11/05/2017 07:28 PM, David Holmes wrote: >>> Hi Misha, >>> >>> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>>> Hi David, >>>> >>>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>>> >>>> Please see my comments below. >>>> >>>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>>> Hi Misha, >>>>> >>>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>>> Hi David, >>>>>> >>>>>> Thank you for taking a look at this change. >>>>>> >>>>>> >>>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>>> Hi Misha, >>>>>>> >>>>>>> I don't see anything else in VMProps that writes anything out so to me the fix is to delete the System.err.println completely. >>>>>> Removal is an option, and seems a simpler one. However, we will loose the diagnostic information. >>>>>> On the other hand, if an issue arises on a specific host or system, I can patch this file with extra trace statements, and diagnose the failure that way. >>>>>> If you think removing the print statement is a better option, and I do not hear objections or other opinions, I can just remove the print statement. >>>>> Delete please :) >>>>> >>>> Done. >>>>>>> That said I did not look at this in the past but I feel the whole docker test here is not really appropriate/suitable to always be run on linux-x64. It has to exec another process and wait up to 10 seconds to get a result! Can't this actually be done via a WhiteBox check once Bob's container support updates are in place? >>>>>> Actually, WhiteBox is not necessary for such check. The ability to run docker on a given host/node can be determined in a body of a jtreg test, via a test utility. In fact, that was my original intent. When this change was presented for a discussion, this aspect was debated and reviewers have reached a consensus to do such check in VMProps.java. We mainly discussed within VM SQE team. The main argument was that using @requires is a more proper way of "skipping" the test(s) on unsupported environments, rather then checking this in the body of the test and then returning. When using @requires, the test execution system will report the test as "not run". When skipping the test using "return" from the body of a test, the execution system will report test as "ran" and passed; jtreg does not yet support the "skipped" state. >>>>>> >>>>>> Unfortunately, the way our "@requires" mechanism works is that it runs each time prior to starting a test run, even if the test being run is not concerned with a specific property. It was my concern that this could be a burden on the test run startup, but I eventually agreed to the opinion of other reviewers. >>>>>> >>>>>> If you have a strong opinion or concern about this, let me know. However, I am not sure what is the best process to use to overturn a decision on a prior review or design discussion. I guess one option is to file a bug or an RFE, and bring it up for a discussion. >>>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using WhiteBox connected to Bob's new container support code is better than exec'ing a separate process (which I'm not even sure what it does). >>>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy" >>>> >>>> Also, in current webrev, I though I could do something quick and easy to address your concerns, before we discuss and agree on a final solution. >>>> I have timed running "docker ps" on 2 different hosts, 30 times each. I saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec to 500ms. I thought 500 ms gives enough time cushion for slower test nodes/hosts. >>> I think a slowness would arise on systems that do not have docker installed. If I time "docker ps" on my system I get >>> >>> > time docker ps >>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>> >>> real 0m4.103s >>> user 0m0.044s >>> sys 0m0.036s >>> >>> but the second call is much quicker: >>> >>> > time docker ps >>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>> >>> real 0m0.078s >>> user 0m0.052s >>> sys 0m0.012s >>> >>> It may well depend on the user's path. If "docker" is present early in the path then it will be found and executed quickly, but if not present, or if further down the path it may take a while to occur. >>> >>> So I think the timeout change, as a short term proposal, may be more risky and so not worthwhile. >>> >>> I'd prefer to see the whole "docker ps" disappear altogether. Such an approach is not scalable if there may be different container systems. >> I have discussed this topic again with Igor and Leonid, and explained your concerns. >> >> >> We came up with alternative ideas: >> >> 1. VMProps.java will check the existence of docker on a given test machine. We will look at "/bin/docker", "/usr/bin/docker", "/usr/local/bin/docker" >> For now we only support Linux x64. If any of these files exist, we return 'true' for a corresponding at-requires property, otherwise false. >> >> 2. In case someone wishes to specify a custom location for a docker binary, we thought we could use a property for that: >> jdk.test.lib.docker.path="" >> If this property is specified, VMProps.java will check the docker at that location. >> >> 3. No more "docker ps" execution in VMProps.java. >> >> Please let me know what you think. > Just having checking for the existence of /bin/docker doesn?t means that it?s properly configured for use by any test process. > There?s permission setup, possible proxy setup etc. We had issues with our internal testing systems not having the tools we?ve > needed and it?s a pain to have to go to each system in order to clear the way to successfully run all the tests. Yes, I agree with Bob, it is important to check that the docker is runnable by the test user or test agent. Given this, I propose to check 'docker ps' with a small timeout, say 300ms. However, instead of doing 'docker ps', I will do the '/docker ps' with the docker binary that I have found on a given system. This way we run 'docker ps' ONLY if we found it on the system. And, we give a process runner an exact location of the binary which will eliminate a search on the PATH, which in turn allows us to make wait timeout smaller, hence reducing impact/cost on VMProps evaluation. Does it work for you Bob? David, are you OK with this approach? > > This is probably breaking new ground, but can we cache the result of ?docker ps? in /tmp in order to eliminate the performance > overhead for all but the first time a new user runs these tests? I try to keep this solution simple. If you and David approve the solution above, I will implement and test it. If we find any problems, either during my testing or during test production, we could look into caching mechanisms. Thank you, Misha > > Bob. >> >> >> Thank you, >> Misha >> >>> Thanks, >>> David >>> >>>> Let me know what you think. I can revert this part of the change or reduce the timeout value a bit more. >>>> >>>> Thank you, >>>> Misha >>>>>> For this change under review, if I do not hear opinions from other reviewers, I can simply delete the print statement. >>>>> Sounds good to me. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>> Please review this simple change that makes the error message in jtreg-ext/requires/VMProps.java conditional >>>>>>>> on java property "vmprops.trace.verbose". W/o this condition an error message is printed each time any jtreg >>>>>>>> test is ran, including non-docker tests. >>>>>>>> >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ======= Background and details: >>>>>>>> VMProps.java is called for each test run of jtreg (be it a run containing a singe test or multiple tests). >>>>>>>> VMProps evaluates the @requires properties. Therefore, in case of this bug, any time a user runs any jtreg VM test >>>>>>>> on a machine w/o an operational docker engine, this error will pop up. >>>>>>>> The fix makes printing of this error message conditional, off by default. The message can be turend on >>>>>>>> via a property vmprops.trace.verbose when detailed diagnostic info is desired. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ======== Testing: >>>>>>>> Local testing: >>>>>>>> W/o docker present: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> Detailed error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test skipped >>>>>>>> >>>>>>>> With docker installed but no permissions: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> Detailed error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test skipped >>>>>>>> >>>>>>>> With docker installed and operational: >>>>>>>> jtreg >>>>>>>> No error message >>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>> No error message >>>>>>>> jtreg DockerBasicTest.java >>>>>>>> Test passed >>>>>>>> >>>>>>>> Testing via automated distributed test system: >>>>>>>> - tier1 >>>>>>>> - docker tests >>>>>>>> In progress >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> From bob.vandette at oracle.com Tue Nov 7 19:47:52 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Tue, 7 Nov 2017 14:47:52 -0500 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> Message-ID: <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> > On Nov 7, 2017, at 2:25 PM, mikhailo wrote: > > > > On 11/07/2017 09:04 AM, Bob Vandette wrote: >>> On Nov 6, 2017, at 5:42 PM, mikhailo wrote: >>> >>> Hi David, >>> >>> >>> On 11/05/2017 07:28 PM, David Holmes wrote: >>>> Hi Misha, >>>> >>>> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>>>> Hi David, >>>>> >>>>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>>>> >>>>> Please see my comments below. >>>>> >>>>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>>>> Hi Misha, >>>>>> >>>>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>>>> Hi David, >>>>>>> >>>>>>> Thank you for taking a look at this change. >>>>>>> >>>>>>> >>>>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>>>> Hi Misha, >>>>>>>> >>>>>>>> I don't see anything else in VMProps that writes anything out so to me the fix is to delete the System.err.println completely. >>>>>>> Removal is an option, and seems a simpler one. However, we will loose the diagnostic information. >>>>>>> On the other hand, if an issue arises on a specific host or system, I can patch this file with extra trace statements, and diagnose the failure that way. >>>>>>> If you think removing the print statement is a better option, and I do not hear objections or other opinions, I can just remove the print statement. >>>>>> Delete please :) >>>>>> >>>>> Done. >>>>>>>> That said I did not look at this in the past but I feel the whole docker test here is not really appropriate/suitable to always be run on linux-x64. It has to exec another process and wait up to 10 seconds to get a result! Can't this actually be done via a WhiteBox check once Bob's container support updates are in place? >>>>>>> Actually, WhiteBox is not necessary for such check. The ability to run docker on a given host/node can be determined in a body of a jtreg test, via a test utility. In fact, that was my original intent. When this change was presented for a discussion, this aspect was debated and reviewers have reached a consensus to do such check in VMProps.java. We mainly discussed within VM SQE team. The main argument was that using @requires is a more proper way of "skipping" the test(s) on unsupported environments, rather then checking this in the body of the test and then returning. When using @requires, the test execution system will report the test as "not run". When skipping the test using "return" from the body of a test, the execution system will report test as "ran" and passed; jtreg does not yet support the "skipped" state. >>>>>>> >>>>>>> Unfortunately, the way our "@requires" mechanism works is that it runs each time prior to starting a test run, even if the test being run is not concerned with a specific property. It was my concern that this could be a burden on the test run startup, but I eventually agreed to the opinion of other reviewers. >>>>>>> >>>>>>> If you have a strong opinion or concern about this, let me know. However, I am not sure what is the best process to use to overturn a decision on a prior review or design discussion. I guess one option is to file a bug or an RFE, and bring it up for a discussion. >>>>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using WhiteBox connected to Bob's new container support code is better than exec'ing a separate process (which I'm not even sure what it does). >>>>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy" >>>>> >>>>> Also, in current webrev, I though I could do something quick and easy to address your concerns, before we discuss and agree on a final solution. >>>>> I have timed running "docker ps" on 2 different hosts, 30 times each. I saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec to 500ms. I thought 500 ms gives enough time cushion for slower test nodes/hosts. >>>> I think a slowness would arise on systems that do not have docker installed. If I time "docker ps" on my system I get >>>> >>>> > time docker ps >>>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>>> >>>> real 0m4.103s >>>> user 0m0.044s >>>> sys 0m0.036s >>>> >>>> but the second call is much quicker: >>>> >>>> > time docker ps >>>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>>> >>>> real 0m0.078s >>>> user 0m0.052s >>>> sys 0m0.012s >>>> >>>> It may well depend on the user's path. If "docker" is present early in the path then it will be found and executed quickly, but if not present, or if further down the path it may take a while to occur. >>>> >>>> So I think the timeout change, as a short term proposal, may be more risky and so not worthwhile. >>>> >>>> I'd prefer to see the whole "docker ps" disappear altogether. Such an approach is not scalable if there may be different container systems. >>> I have discussed this topic again with Igor and Leonid, and explained your concerns. >>> >>> >>> We came up with alternative ideas: >>> >>> 1. VMProps.java will check the existence of docker on a given test machine. We will look at "/bin/docker", "/usr/bin/docker", "/usr/local/bin/docker" >>> For now we only support Linux x64. If any of these files exist, we return 'true' for a corresponding at-requires property, otherwise false. >>> >>> 2. In case someone wishes to specify a custom location for a docker binary, we thought we could use a property for that: >>> jdk.test.lib.docker.path="" >>> If this property is specified, VMProps.java will check the docker at that location. >>> >>> 3. No more "docker ps" execution in VMProps.java. >>> >>> Please let me know what you think. >> Just having checking for the existence of /bin/docker doesn?t means that it?s properly configured for use by any test process. >> There?s permission setup, possible proxy setup etc. We had issues with our internal testing systems not having the tools we?ve >> needed and it?s a pain to have to go to each system in order to clear the way to successfully run all the tests. > Yes, I agree with Bob, it is important to check that the docker is runnable by the test user or test agent. > Given this, I propose to check 'docker ps' with a small timeout, say 300ms. However, instead of doing 'docker ps', I will do the '/docker ps' with the docker binary that I have found on a given system. > This way we run 'docker ps' ONLY if we found it on the system. And, we give a process runner an exact location of the binary which will eliminate a search on the PATH, which in turn allows us to make wait timeout smaller, hence reducing impact/cost on VMProps evaluation. > > Does it work for you Bob? David, are you OK with this approach? I think this is an improvement and I?m ok with this approach but I don?t see that it will save that much time. Starting docker takes a while (worse the first time). I?ve seen 1sec startup times. Let see what David thinks of the /tmp caching idea. You?d have to put the user that found and successfully ran docker and have to deal with multi-process write access to the file. Bob. > >> >> This is probably breaking new ground, but can we cache the result of ?docker ps? in /tmp in order to eliminate the performance >> overhead for all but the first time a new user runs these tests? > I try to keep this solution simple. If you and David approve the solution above, I will implement and test it. If we find any problems, either during my testing or during test production, we could look into caching mechanisms. > > > Thank you, > Misha >> >> Bob. > >>> >>> >>> Thank you, >>> Misha >>> >>>> Thanks, >>>> David >>>> >>>>> Let me know what you think. I can revert this part of the change or reduce the timeout value a bit more. >>>>> >>>>> Thank you, >>>>> Misha >>>>>>> For this change under review, if I do not hear opinions from other reviewers, I can simply delete the print statement. >>>>>> Sounds good to me. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>> Please review this simple change that makes the error message in jtreg-ext/requires/VMProps.java conditional >>>>>>>>> on java property "vmprops.trace.verbose". W/o this condition an error message is printed each time any jtreg >>>>>>>>> test is ran, including non-docker tests. >>>>>>>>> >>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ======= Background and details: >>>>>>>>> VMProps.java is called for each test run of jtreg (be it a run containing a singe test or multiple tests). >>>>>>>>> VMProps evaluates the @requires properties. Therefore, in case of this bug, any time a user runs any jtreg VM test >>>>>>>>> on a machine w/o an operational docker engine, this error will pop up. >>>>>>>>> The fix makes printing of this error message conditional, off by default. The message can be turend on >>>>>>>>> via a property vmprops.trace.verbose when detailed diagnostic info is desired. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ======== Testing: >>>>>>>>> Local testing: >>>>>>>>> W/o docker present: >>>>>>>>> jtreg >>>>>>>>> No error message >>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>> Detailed error message >>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>> Test skipped >>>>>>>>> >>>>>>>>> With docker installed but no permissions: >>>>>>>>> jtreg >>>>>>>>> No error message >>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>> Detailed error message >>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>> Test skipped >>>>>>>>> >>>>>>>>> With docker installed and operational: >>>>>>>>> jtreg >>>>>>>>> No error message >>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>> No error message >>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>> Test passed >>>>>>>>> >>>>>>>>> Testing via automated distributed test system: >>>>>>>>> - tier1 >>>>>>>>> - docker tests >>>>>>>>> In progress >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>> > From chris.plummer at oracle.com Tue Nov 7 20:18:33 2017 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 7 Nov 2017 12:18:33 -0800 Subject: alternate pns() debug function In-Reply-To: <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> Message-ID: <13c53eb0-8232-dcd3-e563-f2d8688f3bca@oracle.com> On 11/6/17 11:20 PM, Ioi Lam wrote: > > > On 11/6/17 9:05 PM, Chris Plummer wrote: >> Hi, >> >> I'd like to add an alternate version of pns() to debug.cpp, and would >> like some initial comments first before filing a bug and sending out >> an RFR. pns() is used to print out the native stack: >> >> extern "C" void pns(void* sp, void* fp, void* pc) { // print native >> stack >> ? Command c("pns"); >> ? static char buf[O_BUFLEN]; >> ? Thread* t = Thread::current_or_null(); >> ? // Call generic frame constructor (certain arguments may be ignored) >> ? frame fr(sp, fp, pc); >> ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >> } >> >> And here's the help output for it: >> >> ? pns(void* sp, void* fp, void* pc)? - print native (i.e. mixed) >> stack trace. E.g."); >> ?????????????????? pns($sp, $rbp, $pc) on Linux/amd64 and >> Solaris/amd64 or"); >> ?????????????????? pns($sp, $ebp, $pc) on Linux/x86 or"); >> ?????????????????? pns($sp, 0, $pc)??? on Linux/ppc64 or"); >> ?????????????????? pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC"); >> ???????????????? - in gdb do 'set overload-resolution off' before >> calling pns()"); >> ???????????????? - in dbx do 'frame 1' before calling pns()"); >> >> The intent is to call it from gdb, but it is potentially useful to >> call it from within hotspot when debugging. I'd like to propose the >> following alternative, since it does away with needing to pass in >> register values (not easy to do from C): >> >> extern "C" void pns2() { // print native stack >> ? Command c("pns2"); >> ? static char buf[O_BUFLEN]; >> ? Thread* t = Thread::current_or_null(); >> ? frame fr = os::current_frame(); >> ? VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >> } >> >> I actually used this quite a bit with some recent debugging. There >> were places in hotspot where I wanted to know what the native stack >> looked like when called, and it was a lot easier to just add a call >> to pns2() than to setup the test to run in gdb. This seems to work on >> Linux/x64, macosx/64, and solaris/sparc. For some reason it crashes >> on Windows/x64, but I don't see any indication from the above help >> for pns() that it works on Windows/x64 anyway. >> >> # >> # A fatal error has been detected by the Java Runtime Environment: >> # >> #? EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000003bcfae1466, >> pid=9872, tid=12212 >> # >> # JRE version: Java(TM) SE Runtime Environment (10.0) (fastdebug >> build 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs) >> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug >> 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs, mixed mode, >> tiered, compressed oops, g1 gc, windows-amd64) >> # Problematic frame: >> # v? ~StubRoutines::get_previous_fp >> # >> >> I'm starting to suspect that os::current_frame() normally is never >> even called on Windows (there are calls to it in the VmError code, >> but I suspect they are never triggered on Windows) and that >> StubRoutines::get_previous_fp() is not working on Windows, but I >> could be wrong. >> > Hi Chris, > > I like this new pns2() function. > > To make this work on Windows, you probably need to follow this pattern > in vmError.cpp. > > > ???? if (os::platform_print_native_stack(st, _context, buf, > sizeof(buf))) { > ?????? // We have printed the native stack in platform-specific code > ?????? // Windows/x64 needs special handling. > ???? } else { > ?????? frame fr = _context ? os::fetch_frame_from_context(_context) > ?????????????????????????? : os::current_frame(); > > ?????? print_native_stack(st, fr, _thread, buf, sizeof(buf)); > ???? } Thanks for the tip. I'll give that a try. Looks like it's needed for AIX also. Chris > > Thanks > - Ioi > >> Let me know what you think. >> >> thanks, >> >> Chris >> > From mikhailo.seledtsov at oracle.com Tue Nov 7 21:36:13 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Tue, 7 Nov 2017 13:36:13 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> Message-ID: On 11/07/2017 11:47 AM, Bob Vandette wrote: >> On Nov 7, 2017, at 2:25 PM, mikhailo wrote: >> >> >> >> On 11/07/2017 09:04 AM, Bob Vandette wrote: >>>> On Nov 6, 2017, at 5:42 PM, mikhailo wrote: >>>> >>>> Hi David, >>>> >>>> >>>> On 11/05/2017 07:28 PM, David Holmes wrote: >>>>> Hi Misha, >>>>> >>>>> On 4/11/2017 10:21 AM, Mikhailo Seledtsov wrote: >>>>>> Hi David, >>>>>> >>>>>> Updated webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.01/ >>>>>> >>>>>> Please see my comments below. >>>>>> >>>>>> On 11/2/17, 6:02 PM, David Holmes wrote: >>>>>>> Hi Misha, >>>>>>> >>>>>>> On 3/11/2017 10:22 AM, mikhailo wrote: >>>>>>>> Hi David, >>>>>>>> >>>>>>>> Thank you for taking a look at this change. >>>>>>>> >>>>>>>> >>>>>>>> On 11/02/2017 04:47 PM, David Holmes wrote: >>>>>>>>> Hi Misha, >>>>>>>>> >>>>>>>>> I don't see anything else in VMProps that writes anything out so to me the fix is to delete the System.err.println completely. >>>>>>>> Removal is an option, and seems a simpler one. However, we will loose the diagnostic information. >>>>>>>> On the other hand, if an issue arises on a specific host or system, I can patch this file with extra trace statements, and diagnose the failure that way. >>>>>>>> If you think removing the print statement is a better option, and I do not hear objections or other opinions, I can just remove the print statement. >>>>>>> Delete please :) >>>>>>> >>>>>> Done. >>>>>>>>> That said I did not look at this in the past but I feel the whole docker test here is not really appropriate/suitable to always be run on linux-x64. It has to exec another process and wait up to 10 seconds to get a result! Can't this actually be done via a WhiteBox check once Bob's container support updates are in place? >>>>>>>> Actually, WhiteBox is not necessary for such check. The ability to run docker on a given host/node can be determined in a body of a jtreg test, via a test utility. In fact, that was my original intent. When this change was presented for a discussion, this aspect was debated and reviewers have reached a consensus to do such check in VMProps.java. We mainly discussed within VM SQE team. The main argument was that using @requires is a more proper way of "skipping" the test(s) on unsupported environments, rather then checking this in the body of the test and then returning. When using @requires, the test execution system will report the test as "not run". When skipping the test using "return" from the body of a test, the execution system will report test as "ran" and passed; jtreg does not yet support the "skipped" state. >>>>>>>> >>>>>>>> Unfortunately, the way our "@requires" mechanism works is that it runs each time prior to starting a test run, even if the test being run is not concerned with a specific property. It was my concern that this could be a burden on the test run startup, but I eventually agreed to the opinion of other reviewers. >>>>>>>> >>>>>>>> If you have a strong opinion or concern about this, let me know. However, I am not sure what is the best process to use to overturn a decision on a prior review or design discussion. I guess one option is to file a bug or an RFE, and bring it up for a discussion. >>>>>>> Yes let's file a bug/RFE. VMProps can use WhiteBox and I think using WhiteBox connected to Bob's new container support code is better than exec'ing a separate process (which I'm not even sure what it does). >>>>>> I have filed a bug: "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy" >>>>>> >>>>>> Also, in current webrev, I though I could do something quick and easy to address your concerns, before we discuss and agree on a final solution. >>>>>> I have timed running "docker ps" on 2 different hosts, 30 times each. I saw a spread between 20ms to 150ms, mostly centered around 30ms. Hence, I reduced the timeout in the VMProps.java from 10 sec to 500ms. I thought 500 ms gives enough time cushion for slower test nodes/hosts. >>>>> I think a slowness would arise on systems that do not have docker installed. If I time "docker ps" on my system I get >>>>> >>>>> > time docker ps >>>>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>>>> >>>>> real 0m4.103s >>>>> user 0m0.044s >>>>> sys 0m0.036s >>>>> >>>>> but the second call is much quicker: >>>>> >>>>> > time docker ps >>>>> The program 'docker' is currently not installed. To run 'docker' please ask your administrator to install the package 'docker' >>>>> >>>>> real 0m0.078s >>>>> user 0m0.052s >>>>> sys 0m0.012s >>>>> >>>>> It may well depend on the user's path. If "docker" is present early in the path then it will be found and executed quickly, but if not present, or if further down the path it may take a while to occur. >>>>> >>>>> So I think the timeout change, as a short term proposal, may be more risky and so not worthwhile. >>>>> >>>>> I'd prefer to see the whole "docker ps" disappear altogether. Such an approach is not scalable if there may be different container systems. >>>> I have discussed this topic again with Igor and Leonid, and explained your concerns. >>>> >>>> >>>> We came up with alternative ideas: >>>> >>>> 1. VMProps.java will check the existence of docker on a given test machine. We will look at "/bin/docker", "/usr/bin/docker", "/usr/local/bin/docker" >>>> For now we only support Linux x64. If any of these files exist, we return 'true' for a corresponding at-requires property, otherwise false. >>>> >>>> 2. In case someone wishes to specify a custom location for a docker binary, we thought we could use a property for that: >>>> jdk.test.lib.docker.path="" >>>> If this property is specified, VMProps.java will check the docker at that location. >>>> >>>> 3. No more "docker ps" execution in VMProps.java. >>>> >>>> Please let me know what you think. >>> Just having checking for the existence of /bin/docker doesn?t means that it?s properly configured for use by any test process. >>> There?s permission setup, possible proxy setup etc. We had issues with our internal testing systems not having the tools we?ve >>> needed and it?s a pain to have to go to each system in order to clear the way to successfully run all the tests. >> Yes, I agree with Bob, it is important to check that the docker is runnable by the test user or test agent. >> Given this, I propose to check 'docker ps' with a small timeout, say 300ms. However, instead of doing 'docker ps', I will do the '/docker ps' with the docker binary that I have found on a given system. >> This way we run 'docker ps' ONLY if we found it on the system. And, we give a process runner an exact location of the binary which will eliminate a search on the PATH, which in turn allows us to make wait timeout smaller, hence reducing impact/cost on VMProps evaluation. >> >> Does it work for you Bob? David, are you OK with this approach? > I think this is an improvement and I?m ok with this approach but I don?t see that it will save that much time. > Starting docker takes a while (worse the first time). I?ve seen 1sec startup times. Let see what David thinks > of the /tmp caching idea. You?d have to put the user that found and successfully ran docker and have to deal > with multi-process write access to the file. OK. There is another option that will work well, but requires one extra input from user. We could designate a property that would state whether a host is docker capable. When user wishes to execute docker tests locally she/he will set the property to true. When executed via Oracle automated test system, the system will set this property to true on docker capable test nodes. All that VMProps will do is check the property, so no docker ps any more, not even searching for docker on a known locations. As for naming, I can propose this: jdk.test.lib.docker.capable.node. Alternative suggestions of names welcome. This makes the check clear, simple and reliable. What do you think? Thank you, Misha > > Bob. > > >>> This is probably breaking new ground, but can we cache the result of ?docker ps? in /tmp in order to eliminate the performance >>> overhead for all but the first time a new user runs these tests? >> I try to keep this solution simple. If you and David approve the solution above, I will implement and test it. If we find any problems, either during my testing or during test production, we could look into caching mechanisms. >> >> >> Thank you, >> Misha >>> Bob. >>>> >>>> Thank you, >>>> Misha >>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Let me know what you think. I can revert this part of the change or reduce the timeout value a bit more. >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>>> For this change under review, if I do not hear opinions from other reviewers, I can simply delete the print statement. >>>>>>> Sounds good to me. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>>> Please review this simple change that makes the error message in jtreg-ext/requires/VMProps.java conditional >>>>>>>>>> on java property "vmprops.trace.verbose". W/o this condition an error message is printed each time any jtreg >>>>>>>>>> test is ran, including non-docker tests. >>>>>>>>>> >>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ======= Background and details: >>>>>>>>>> VMProps.java is called for each test run of jtreg (be it a run containing a singe test or multiple tests). >>>>>>>>>> VMProps evaluates the @requires properties. Therefore, in case of this bug, any time a user runs any jtreg VM test >>>>>>>>>> on a machine w/o an operational docker engine, this error will pop up. >>>>>>>>>> The fix makes printing of this error message conditional, off by default. The message can be turend on >>>>>>>>>> via a property vmprops.trace.verbose when detailed diagnostic info is desired. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ======== Testing: >>>>>>>>>> Local testing: >>>>>>>>>> W/o docker present: >>>>>>>>>> jtreg >>>>>>>>>> No error message >>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>> Detailed error message >>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>> Test skipped >>>>>>>>>> >>>>>>>>>> With docker installed but no permissions: >>>>>>>>>> jtreg >>>>>>>>>> No error message >>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>> Detailed error message >>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>> Test skipped >>>>>>>>>> >>>>>>>>>> With docker installed and operational: >>>>>>>>>> jtreg >>>>>>>>>> No error message >>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>> No error message >>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>> Test passed >>>>>>>>>> >>>>>>>>>> Testing via automated distributed test system: >>>>>>>>>> - tier1 >>>>>>>>>> - docker tests >>>>>>>>>> In progress >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Misha >>>>>>>>>> From mikhailo.seledtsov at oracle.com Tue Nov 7 23:12:24 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Tue, 7 Nov 2017 15:12:24 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <9D64FC51-D58C-47CB-ADC0-5187979B0B91@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <30EFA226-1F76-4AF9-8838-6B0E2E39CF55@oracle.com> <5A013118.6030101@oracle.com> <9D64FC51-D58C-47CB-ADC0-5187979B0B91@oracle.com> Message-ID: On 11/07/2017 08:14 AM, Bob Vandette wrote: >> On Nov 6, 2017, at 11:05 PM, Mikhailo Seledtsov wrote: >> >> Hi Bob, >> >> Thank you for review. Please see my comments inline: >> >> On 11/3/17, 11:04 AM, Bob Vandette wrote: >>> http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/CPUSetsReader.java.html >>> >>> Not sure this is a problem but If you specify --cpuset-cpus 2-3,1 in docker you end up with cpuset.cpus containing 1-3. >>> Your cpusets test cases are all increasing in order so you won?t hit this issue. >>> >> I did not do the exhaustive test of all combinations of cpu sets. I tried to cover more common cases while balancing vs complexity and execution time. >> In TestCPUSets.java I read all available cpu sets from the system, flattened them into a list/set, and then created a container with >> a subset of one, a full subset available on the system, and rounded half of the subset. >> If you wish I could file an RFE for 11 to add more corner test cases for this. Please let me know. >>> The read function in this file has a hard coded /sys/fs/cgroup/cpuset directory. This may not be where this >>> ends up being mounted. Is this read function even used? >> Yes. This function is used from " TestCPUSets.testTheSet()" to read the sets. >> I did design the method and test such that if the file can not be read, the test case for a given set will be skipped, to avoid false failures. >> I did not realize the location for this directory could vary. I can introduce a property jdk.test.docker.cpuset.location that test users or test system could specify to point to the location of cpuset directory; if not specify the default value would be as it is now, /sys/fs/cgroup/cpuset. Is this reasonable? > I think this needs to be more dynamic. One of my internal systems has, cpuset.cpus is in /cgroup/cpuset. > Who knows how the mach5 systems are configured. > > Maybe you can use /proc/self/status and extract Cpus_allowed_list? OK. I will change my tests to extract values from /proc/self/status. I assume this method is more standard and reliable than reading from /sys/fs/cgroup/cpuset. Right? Thank you, Misha > >>> >>> http://cr.openjdk.java.net/~mseledtsov/8189762.00/test/hotspot/jtreg/runtime/containers/docker/TestCPUAwareness.java.html >>> >>> Your test assumes that there are at least two physical processors on the host. You might want to check first. >>> >>> 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); >>> 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2) >> Makes sense. Will do. >>> Everything else looks good, >>> bob. >> Thank you, > Once you get all of your changes done, maybe you can commit them in a local forest and do an hg export of the > changeset. I can then import it and push it for you with my changes. Assuming this works, this should preserve > you as the committer and allow the push to have both implementation and tests. > > Bob. > >> Misha >>> >>>> On Nov 1, 2017, at 11:11 PM, mikhailo wrote: >>>> >>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>> Webrev: >>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>> Testing: >>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>> Ran the developed tests via jtreg >>>> Pass >>>> >>>> 2. Automated testing system - run these tests >>>> In progress >>>> >>>> Thank you, >>>> Misha >>>> From david.holmes at oracle.com Wed Nov 8 02:01:40 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 12:01:40 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> Message-ID: <9fa4064a-5764-d3fa-c398-ebd05c24e3ed@oracle.com> Hi Misha, On 8/11/2017 7:36 AM, mikhailo wrote: >>> Does it work for you Bob? David, are you OK with this approach? >> I think this is an improvement and I?m ok with this approach but I >> don?t see that it will save that much time. >> Starting docker takes? a while (worse the first time).? I?ve seen 1sec >> startup times.? Let see what David thinks >> of the /tmp caching idea.? You?d have to put the user that found and >> successfully ran docker and have to deal >> with multi-process write access to the file. > OK. There is another option that will work well, but requires one extra > input from user. We could designate a property that would state whether > a host is docker capable. When user wishes to execute docker tests > locally she/he will set the property to true. When executed via Oracle > automated test system, the system will set this property to true on > docker capable test nodes. All that VMProps will do is check the > property, so no docker ps any more, not even searching for docker on a > known locations. > > As for naming, I can propose this: jdk.test.lib.docker.capable.node. > Alternative suggestions of names welcome. > > This makes the check clear, simple and reliable. What do you think? This doesn't work if the intent is that we happen to add a docker-enabled machine to a test pool. You can't selectively set the property only when run on the right host (if you could then we'd only run the tests on that host in the first place). But you seemed to have moved from fixing this bug - 8189213 - to trying to address the new RFE 8190730. Thanks, David > > Thank you, > Misha > > >> >> Bob. >> >> >>>> This is probably breaking new ground, but can we cache the result of >>>> ?docker ps? in /tmp in order to eliminate the performance >>>> overhead for all but the first time a new user runs these tests? >>> I try to keep this solution simple. If you and David approve the >>> solution above, I will implement and test it. If we find any >>> problems, either during my testing or during test production, we >>> could look into caching mechanisms. >>> >>> >>> Thank you, >>> Misha >>>> Bob. >>>>> >>>>> Thank you, >>>>> Misha >>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> Let me know what you think. I can revert this part of the change >>>>>>> or reduce the timeout value a bit more. >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>>>> For this change under review, if I do not hear opinions from >>>>>>>>> other reviewers, I can simply delete the print statement. >>>>>>>> Sounds good to me. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>>> Thanks, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>>>> Please review this simple change that makes the error message >>>>>>>>>>> in jtreg-ext/requires/VMProps.java conditional >>>>>>>>>>> on java property "vmprops.trace.verbose". W/o this condition >>>>>>>>>>> an error message is printed each time any jtreg >>>>>>>>>>> test is ran, including non-docker tests. >>>>>>>>>>> >>>>>>>>>>> ????? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>>>> ????? Webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ======= Background and details: >>>>>>>>>>> VMProps.java is called for each test run of jtreg (be it a >>>>>>>>>>> run containing a singe test or multiple tests). >>>>>>>>>>> VMProps evaluates the @requires properties. Therefore, in >>>>>>>>>>> case of this bug, any time a user runs any jtreg VM test >>>>>>>>>>> on a machine w/o an operational docker engine, this error >>>>>>>>>>> will pop up. >>>>>>>>>>> The fix makes printing of this error message conditional, off >>>>>>>>>>> by default. The message can be turend on >>>>>>>>>>> via a property vmprops.trace.verbose when detailed diagnostic >>>>>>>>>>> info is desired. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ======== Testing: >>>>>>>>>>> Local testing: >>>>>>>>>>> ??? W/o docker present: >>>>>>>>>>> ????? jtreg >>>>>>>>>>> ??????? No error message >>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>> ??????? Test skipped >>>>>>>>>>> >>>>>>>>>>> ??? With docker installed but no permissions: >>>>>>>>>>> ????? jtreg >>>>>>>>>>> ??????? No error message >>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>> ??????? Test skipped >>>>>>>>>>> >>>>>>>>>>> ??? With docker installed and operational: >>>>>>>>>>> ????? jtreg >>>>>>>>>>> ??????? No error message >>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>> ??????? No error message >>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>> ??????? Test passed >>>>>>>>>>> >>>>>>>>>>> Testing via automated distributed test system: >>>>>>>>>>> ??? - tier1 >>>>>>>>>>> ??? - docker tests >>>>>>>>>>> ??? In progress >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Misha >>>>>>>>>>> > From david.holmes at oracle.com Wed Nov 8 02:23:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 12:23:49 +1000 Subject: RFR[XXS]: 8186778 Make obsolete VM options for shared region size control In-Reply-To: <26645083-96ad-9fb8-9370-6dbf77319b68@oracle.com> References: <26645083-96ad-9fb8-9370-6dbf77319b68@oracle.com> Message-ID: <99874707-e0e4-829a-929e-bce41b8e7da9@oracle.com> Hi Ioi, On 8/11/2017 3:33 AM, Ioi Lam wrote: > Hi, > > Please review this very small change. These VM arguments are no longer > used when creating a CDS archive, so they should be declared obsolete: > > https://bugs.openjdk.java.net/browse/JDK-8186778 > http://cr.openjdk.java.net/~iklam/jdk10/8186778-make-onsolete-shared-region-size-args.v01/ Okay. This is correcting an oversight made when the flags actually stopped being examined. Thanks, David > > Thanks > - Ioi > > ----- > $ java -XX:SharedReadWriteSize=123? -XX:SharedReadOnlySize=123 > -XX:SharedMiscDataSize=123 -XX:SharedMiscCodeSize=123 -version > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option > SharedReadWriteSize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option > SharedReadOnlySize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option > SharedMiscDataSize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option > SharedMiscCodeSize; support was removed in 10.0 > java version "10-internal" > Java(TM) SE Runtime Environment (build 10-internal+0-adhoc.iklam.open) > Java HotSpot(TM) 64-Bit Server VM (build 10-internal+0-adhoc.iklam.open, > mixed mode) > From tobias.hartmann at oracle.com Wed Nov 8 07:58:21 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Wed, 8 Nov 2017 08:58:21 +0100 Subject: [10] RFR(S): 8190797: OSR compilation fails with "assert(__the_thread__->can_call_java()) failed: can not load classes with compiler thread" In-Reply-To: <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> References: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> Message-ID: Hi Vladimir, thanks for the review! Best regards, Tobias On 07.11.2017 19:56, Vladimir Kozlov wrote: > Looks good. > > Thanks, > Vladimir > > On 11/7/17 4:04 AM, Tobias Hartmann wrote: >> Hi, >> >> please review the following patch: >> https://bugs.openjdk.java.net/browse/JDK-8190797 >> http://cr.openjdk.java.net/~thartmann/8190797/webrev.00/ >> >> When oop map creation fails, we try to throw a LinkageError to propagate the error message. If this happens in the >> compiler thread (for example, during OSR compilation), we fail because a compiler thread cannot initialize an exception >> object. >> >> I've fixed this by bailing out with a meaningful message in case !Thread::can_call_java(). Please note that even if we >> are able to instantiate an exception, we will still fail with ShouldNotReachHere because compute_map(TRAPS) is called >> with CATCH (see comments in the bug for a detailed explanation). This fix is not about changing this behavior but to >> fail with a meaningful error message during compilation. This should only happen if something is seriously broken (for >> example, incorrect bytecode with -noverify, see TestLinkageErrorInGenerateOopMap). In this case we would probably hit >> other issues as well if we would continue execution. >> >> Thanks, >> Tobias >> From mikhailo.seledtsov at oracle.com Wed Nov 8 15:22:25 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 8 Nov 2017 07:22:25 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <9fa4064a-5764-d3fa-c398-ebd05c24e3ed@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> <9fa4064a-5764-d3fa-c398-ebd05c24e3ed@oracle.com> Message-ID: <6f99fcdb-d3b8-adf7-2d8d-21232c145a73@oracle.com> On 11/07/2017 06:01 PM, David Holmes wrote: > Hi Misha, > On 8/11/2017 7:36 AM, mikhailo wrote: >>>> Does it work for you Bob? David, are you OK with this approach? >>> I think this is an improvement and I?m ok with this approach but I >>> don?t see that it will save that much time. >>> Starting docker takes? a while (worse the first time).? I?ve seen >>> 1sec startup times.? Let see what David thinks >>> of the /tmp caching idea.? You?d have to put the user that found and >>> successfully ran docker and have to deal >>> with multi-process write access to the file. >> OK. There is another option that will work well, but requires one >> extra input from user. We could designate a property that would state >> whether a host is docker capable. When user wishes to execute docker >> tests locally she/he will set the property to true. When executed via >> Oracle automated test system, the system will set this property to >> true on docker capable test nodes. All that VMProps will do is check >> the property, so no docker ps any more, not even searching for docker >> on a known locations. >> >> As for naming, I can propose this: jdk.test.lib.docker.capable.node. >> Alternative suggestions of names welcome. >> >> This makes the check clear, simple and reliable. What do you think? > > This doesn't work if the intent is that we happen to add a > docker-enabled machine to a test pool. You can't selectively set the > property only when run on the right host (if you could then we'd only > run the tests on that host in the first place). > > But you seemed to have moved from fixing this bug - 8189213 - to > trying to address the new RFE 8190730. You are right. I got carried away with this discussion. I should fix the "JDK-8189213 - [TESTBUG] Running jtreg tests on machine without docker shows extra message" first, and then work on "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to determine presence of docker engine is deemed too heavy". So, the fix for this issue is simple, as we agreed before - remove the message. Here is a webrev: http://cr.openjdk.java.net/~mseledtsov/8189213.02/ May I count you as a Reviewer for JDK-8189213? Thank you, Misha P.S. As for JDK-8190730, I will try to summarize the discussion in the comments of the JBS issue. > > Thanks, > David > >> >> Thank you, >> Misha >> >> >>> >>> Bob. >>> >>> >>>>> This is probably breaking new ground, but can we cache the result >>>>> of ?docker ps? in /tmp in order to eliminate the performance >>>>> overhead for all but the first time a new user runs these tests? >>>> I try to keep this solution simple. If you and David approve the >>>> solution above, I will implement and test it. If we find any >>>> problems, either during my testing or during test production, we >>>> could look into caching mechanisms. >>>> >>>> >>>> Thank you, >>>> Misha >>>>> Bob. >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>>> Let me know what you think. I can revert this part of the >>>>>>>> change or reduce the timeout value a bit more. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>>>> For this change under review, if I do not hear opinions from >>>>>>>>>> other reviewers, I can simply delete the print statement. >>>>>>>>> Sounds good to me. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Misha >>>>>>>>>>> Thanks, >>>>>>>>>>> David >>>>>>>>>>> >>>>>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>>>>> Please review this simple change that makes the error >>>>>>>>>>>> message in jtreg-ext/requires/VMProps.java conditional >>>>>>>>>>>> on java property "vmprops.trace.verbose". W/o this >>>>>>>>>>>> condition an error message is printed each time any jtreg >>>>>>>>>>>> test is ran, including non-docker tests. >>>>>>>>>>>> >>>>>>>>>>>> ????? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>>>>> ????? Webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ======= Background and details: >>>>>>>>>>>> VMProps.java is called for each test run of jtreg (be it a >>>>>>>>>>>> run containing a singe test or multiple tests). >>>>>>>>>>>> VMProps evaluates the @requires properties. Therefore, in >>>>>>>>>>>> case of this bug, any time a user runs any jtreg VM test >>>>>>>>>>>> on a machine w/o an operational docker engine, this error >>>>>>>>>>>> will pop up. >>>>>>>>>>>> The fix makes printing of this error message conditional, >>>>>>>>>>>> off by default. The message can be turend on >>>>>>>>>>>> via a property vmprops.trace.verbose when detailed >>>>>>>>>>>> diagnostic info is desired. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ======== Testing: >>>>>>>>>>>> Local testing: >>>>>>>>>>>> ??? W/o docker present: >>>>>>>>>>>> ????? jtreg >>>>>>>>>>>> ??????? No error message >>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>> >>>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>> ??????? Test skipped >>>>>>>>>>>> >>>>>>>>>>>> ??? With docker installed but no permissions: >>>>>>>>>>>> ????? jtreg >>>>>>>>>>>> ??????? No error message >>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>> >>>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>> ??????? Test skipped >>>>>>>>>>>> >>>>>>>>>>>> ??? With docker installed and operational: >>>>>>>>>>>> ????? jtreg >>>>>>>>>>>> ??????? No error message >>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>> >>>>>>>>>>>> ??????? No error message >>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>> ??????? Test passed >>>>>>>>>>>> >>>>>>>>>>>> Testing via automated distributed test system: >>>>>>>>>>>> ??? - tier1 >>>>>>>>>>>> ??? - docker tests >>>>>>>>>>>> ??? In progress >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Misha >>>>>>>>>>>> >> From thomas.stuefe at gmail.com Wed Nov 8 15:33:25 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 8 Nov 2017 16:33:25 +0100 Subject: alternate pns() debug function In-Reply-To: <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> Message-ID: Hi Chris, Ioi, On Tue, Nov 7, 2017 at 8:20 AM, Ioi Lam wrote: > > > On 11/6/17 9:05 PM, Chris Plummer wrote: > >> Hi, >> >> I'd like to add an alternate version of pns() to debug.cpp, and would >> like some initial comments first before filing a bug and sending out an >> RFR. pns() is used to print out the native stack: >> >> extern "C" void pns(void* sp, void* fp, void* pc) { // print native stack >> Command c("pns"); >> static char buf[O_BUFLEN]; >> Thread* t = Thread::current_or_null(); >> // Call generic frame constructor (certain arguments may be ignored) >> frame fr(sp, fp, pc); >> VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >> } >> >> And here's the help output for it: >> >> pns(void* sp, void* fp, void* pc) - print native (i.e. mixed) stack >> trace. E.g."); >> pns($sp, $rbp, $pc) on Linux/amd64 and Solaris/amd64 >> or"); >> pns($sp, $ebp, $pc) on Linux/x86 or"); >> pns($sp, 0, $pc) on Linux/ppc64 or"); >> pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC"); >> - in gdb do 'set overload-resolution off' before calling >> pns()"); >> - in dbx do 'frame 1' before calling pns()"); >> >> The intent is to call it from gdb, but it is potentially useful to call >> it from within hotspot when debugging. I'd like to propose the following >> alternative, since it does away with needing to pass in register values >> (not easy to do from C): >> >> extern "C" void pns2() { // print native stack >> Command c("pns2"); >> static char buf[O_BUFLEN]; >> Thread* t = Thread::current_or_null(); >> frame fr = os::current_frame(); >> VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >> } >> >> I actually used this quite a bit with some recent debugging. There were >> places in hotspot where I wanted to know what the native stack looked like >> when called, and it was a lot easier to just add a call to pns2() than to >> setup the test to run in gdb. This seems to work on Linux/x64, macosx/64, >> and solaris/sparc. For some reason it crashes on Windows/x64, but I don't >> see any indication from the above help for pns() that it works on >> Windows/x64 anyway. >> >> # >> # A fatal error has been detected by the Java Runtime Environment: >> # >> # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000003bcfae1466, >> pid=9872, tid=12212 >> # >> # JRE version: Java(TM) SE Runtime Environment (10.0) (fastdebug build >> 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs) >> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug >> 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs, mixed mode, >> tiered, compressed oops, g1 gc, windows-amd64) >> # Problematic frame: >> # v ~StubRoutines::get_previous_fp >> # >> >> I'm starting to suspect that os::current_frame() normally is never even >> called on Windows (there are calls to it in the VmError code, but I suspect >> they are never triggered on Windows) and that StubRoutines::get_previous_fp() >> is not working on Windows, but I could be wrong. >> >> Hi Chris, > > I like this new pns2() function. > > To make this work on Windows, you probably need to follow this pattern in > vmError.cpp. > > > if (os::platform_print_native_stack(st, _context, buf, sizeof(buf))) > { > // We have printed the native stack in platform-specific code > // Windows/x64 needs special handling. > } else { > frame fr = _context ? os::fetch_frame_from_context(_context) > : os::current_frame(); > > print_native_stack(st, fr, _thread, buf, sizeof(buf)); > } > > Thanks > - Ioi > > Note that os::platform_print_native_stack() on Windows and AIX only print native C/C++ frames. So, calling this from the debugger may not be that useful. On AIX, it probably will not even be possible to invoke a function in the debugger - never tried, but given that not much else works in that thing, I am pessimistic. But pns() may still be useful as a tracing option, though. On Windows x64, we do not have frame pointers, so we cannot walk the frame VM-Style, so os::platform_print_native_stack() is the only way to print a stack. On AIX, both ways work - VMError::print_native_stack() and os::platform_print_native_stack() - the former gives a mixed stack, the latter the pure C/C++ stack, but with many more information. In the end, for consistency, I'd suggest the same as Ioi, to follow the pattern in VMError(). And we could probably factor that out to an own function? Kind Regards, Thomas > > Let me know what you think. >> >> thanks, >> >> Chris >> >> > From thomas.stuefe at gmail.com Wed Nov 8 16:21:26 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 8 Nov 2017 17:21:26 +0100 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: Hi Coleen, Goetz, new webrev, rebased to the new jdk-hs repo. http://cr.openjdk.java.net/~stuefe/webrevs/8189864-metaspace-map/webrev.01/webrev/ Nothing changed. @Coleen, would you be able to sponsor this? Thanks a lot! Kind Regards, Thomas On Wed, Nov 1, 2017 at 8:20 AM, Thomas St?fe wrote: > Hi Goetz, > > On Mon, Oct 30, 2017 at 1:49 PM, Lindenmaier, Goetz < > goetz.lindenmaier at sap.com> wrote: > >> Hi Thomas, >> >> the change looks good and I know we have made good use of this >> already. >> >> Can you look into the chunks? It could be useful to know that, >> for example, a medium chunk is used by a class loader, but >> still mostly empty (i.e., empty but not free). Well, there is no >> third variant after lower and upper case, but maybe empty >> parts could be indicated in the line with the dots by 'e's (obviously >> at the granularity of small chunks). >> >> Thanks, >> Goetz. >> >> > thank you for the review! > > Indicating which chunks are filled to what degree can certainly be done. > Question is how useful this is (see Zhengyu's work for > https://bugs.openjdk.java.net/browse/JDK-8189688), because within chunks > there is no fragmentation, so the in-chunk map could look rather > unexciting. I will take a look when I am back next week. > > Kind Regards, Thomas > > > -----Original Message----- >> > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- >> > bounces at openjdk.java.net] On Behalf Of Thomas St?fe >> > Sent: Wednesday, October 25, 2017 6:52 AM >> > To: hotspot-runtime-dev at openjdk.java.net >> > Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace >> > fragmentation >> > >> > Hi all, >> > >> > could I please have your thoughts and reviews for this enhancement: >> > >> > Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 >> > Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- >> > metaspace-map/webrev.00/webrev/ >> > >> > At SAP, we added a something we call a metaspace map to the metaspace >> > analysis functions. This is a very simple ASCII print of the metaspace >> > layout. It shows the composition of the VirtualSpaceNodes on a chunk >> basis. >> > It facilitates examining fragmentation and has been proven useful when >> > experimenting with the metaspace allocator. >> > >> > This change adds the map printing code and invokes it if a Metaspace OOM >> > occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, >> we >> > already print out a bunch of things). We also may consider adding this >> to >> > NMT or as a jcmd addition, but I did not want to overload the patch. >> > >> > Example output looks like this (mind that this will only make sense >> with a >> > monospaced font). Dots indicate starts of chunks. Letters indicate chunk >> > type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case >> > letters indicating >> > a free chunk, upper case letters indicating a chunk in use. >> > >> > 0x0000000100000000: ...... >> > xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > 0x0000000100020000: >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > 0x0000000100040000: >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > HHHHHHHHH >> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> > 0x0000000100060000: . . . . . . . . . . . . . . >> > SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > M >> > 0x0000000100080000: . . . . >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >> > mmmmmmmmmmmmmmmmMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > M >> > 0x00000001000a0000: . . . . >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > M >> > 0x00000001000c0000: . . . . >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >> > mmmmmmmmmmmmmmmmMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > M >> > 0x00000001000e0000: . . . . >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > M >> > 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . >> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM >> > MMMMMMMMMMMMMMMMMMMSS >> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> > 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . >> . . >> > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . >> . >> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> > >> > >> > Thank you! >> > >> > Thomas >> > > From thomas.stuefe at gmail.com Wed Nov 8 16:23:25 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 8 Nov 2017 17:23:25 +0100 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: Oh, forgot mention: Goetz and I discussed offlist his idea of adding in-chunk occupation to the map. The idea is good, but we decided to keep this for a separate RFE later and go with the current patch now as it is. Thanks! Thomas On Wed, Nov 8, 2017 at 5:21 PM, Thomas St?fe wrote: > Hi Coleen, Goetz, > > new webrev, rebased to the new jdk-hs repo. > > http://cr.openjdk.java.net/~stuefe/webrevs/8189864- > metaspace-map/webrev.01/webrev/ > > Nothing changed. > > @Coleen, would you be able to sponsor this? Thanks a lot! > > Kind Regards, Thomas > > On Wed, Nov 1, 2017 at 8:20 AM, Thomas St?fe > wrote: > >> Hi Goetz, >> >> On Mon, Oct 30, 2017 at 1:49 PM, Lindenmaier, Goetz < >> goetz.lindenmaier at sap.com> wrote: >> >>> Hi Thomas, >>> >>> the change looks good and I know we have made good use of this >>> already. >>> >>> Can you look into the chunks? It could be useful to know that, >>> for example, a medium chunk is used by a class loader, but >>> still mostly empty (i.e., empty but not free). Well, there is no >>> third variant after lower and upper case, but maybe empty >>> parts could be indicated in the line with the dots by 'e's (obviously >>> at the granularity of small chunks). >>> >>> Thanks, >>> Goetz. >>> >>> >> thank you for the review! >> >> Indicating which chunks are filled to what degree can certainly be done. >> Question is how useful this is (see Zhengyu's work for >> https://bugs.openjdk.java.net/browse/JDK-8189688), because within chunks >> there is no fragmentation, so the in-chunk map could look rather >> unexciting. I will take a look when I am back next week. >> >> Kind Regards, Thomas >> >> > -----Original Message----- >>> > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- >>> > bounces at openjdk.java.net] On Behalf Of Thomas St?fe >>> > Sent: Wednesday, October 25, 2017 6:52 AM >>> > To: hotspot-runtime-dev at openjdk.java.net >>> > Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace >>> > fragmentation >>> > >>> > Hi all, >>> > >>> > could I please have your thoughts and reviews for this enhancement: >>> > >>> > Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 >>> > Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- >>> > metaspace-map/webrev.00/webrev/ >>> > >>> > At SAP, we added a something we call a metaspace map to the metaspace >>> > analysis functions. This is a very simple ASCII print of the metaspace >>> > layout. It shows the composition of the VirtualSpaceNodes on a chunk >>> basis. >>> > It facilitates examining fragmentation and has been proven useful when >>> > experimenting with the metaspace allocator. >>> > >>> > This change adds the map printing code and invokes it if a Metaspace >>> OOM >>> > occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, >>> we >>> > already print out a bunch of things). We also may consider adding this >>> to >>> > NMT or as a jcmd addition, but I did not want to overload the patch. >>> > >>> > Example output looks like this (mind that this will only make sense >>> with a >>> > monospaced font). Dots indicate starts of chunks. Letters indicate >>> chunk >>> > type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case >>> > letters indicating >>> > a free chunk, upper case letters indicating a chunk in use. >>> > >>> > 0x0000000100000000: ...... >>> > xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > 0x0000000100020000: >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > 0x0000000100040000: >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > HHHHHHHHH >>> > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >>> > 0x0000000100060000: . . . . . . . . . . . . . . >>> > SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > M >>> > 0x0000000100080000: . . . . >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >>> > mmmmmmmmmmmmmmmmMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > M >>> > 0x00000001000a0000: . . . . >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > M >>> > 0x00000001000c0000: . . . . >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >>> > mmmmmmmmmmmmmmmmMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > M >>> > 0x00000001000e0000: . . . . >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > M >>> > 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . >>> > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM >>> > MMMMMMMMMMMMMMMMMMMSS >>> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> > 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . >>> . . . >>> > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . >>> . . >>> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >>> > >>> > >>> > Thank you! >>> > >>> > Thomas >>> >> >> > From coleen.phillimore at oracle.com Wed Nov 8 16:44:25 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 8 Nov 2017 11:44:25 -0500 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: Happy to sponsor. Coleen On 11/8/17 11:21 AM, Thomas St?fe wrote: > Hi Coleen, Goetz, > > new webrev, rebased to the new jdk-hs repo. > > http://cr.openjdk.java.net/~stuefe/webrevs/8189864-metaspace-map/webrev.01/webrev/ > > > Nothing changed. > > @Coleen, would you be able to sponsor this? Thanks a lot! > > Kind Regards, Thomas > > On Wed, Nov 1, 2017 at 8:20 AM, Thomas St?fe > wrote: > > Hi Goetz, > > On Mon, Oct 30, 2017 at 1:49 PM, Lindenmaier, Goetz > > wrote: > > Hi Thomas, > > the change looks good and I know we have made good use of this > already. > > Can you look into the chunks? It could be useful to know that, > for example, a medium chunk is used by a? class loader, but > still mostly empty (i.e., empty but not free). Well, there is no > third variant after lower and upper case,? but maybe empty > parts could be indicated in the line with the dots by 'e's > (obviously > at the granularity of small chunks). > > Thanks, > ? Goetz. > > > thank you for the review! > > Indicating which chunks are filled to what degree can certainly be > done. Question is how useful this is (see Zhengyu's work for > https://bugs.openjdk.java.net/browse/JDK-8189688 > ), because > within chunks there is no fragmentation, so the in-chunk map could > look rather unexciting.?I will take a look when I am back next week. > > Kind Regards, Thomas > > > -----Original Message----- > > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > > > bounces at openjdk.java.net ] > On Behalf Of Thomas St?fe > > Sent: Wednesday, October 25, 2017 6:52 AM > > To: hotspot-runtime-dev at openjdk.java.net > > > Subject: RFR(s): 8189864: Provide an ascii map to visualize > metaspace > > fragmentation > > > > Hi all, > > > > could I please have your thoughts and reviews for this > enhancement: > > > > Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 > > > Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- > > > metaspace-map/webrev.00/webrev/ > > > > At SAP, we added a something we call a metaspace map to the > metaspace > > analysis functions. This is a very simple ASCII print of the > metaspace > > layout. It shows the composition of the VirtualSpaceNodes on > a chunk basis. > > It facilitates examining fragmentation and has been proven > useful when > > experimenting with the metaspace allocator. > > > > This change adds the map printing code and invokes it if a > Metaspace OOM > > occurs and -Xlog:gc+metaspace+freelist=debug is active (in > this case, we > > already print out a bunch of things). We also may consider > adding this to > > NMT or as a jcmd addition, but I did not want to overload > the patch. > > > > Example output looks like this (mind that this will only > make sense with a > > monospaced font). Dots indicate starts of chunks. Letters > indicate chunk > > type - (H)umongous, (M)edium, (S)mall (X)specialized - with > lower case > > letters indicating > > a free chunk, upper case letters indicating a chunk in use. > > > > 0x0000000100000000:? ?...... > > xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100020000: > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100040000: > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > HHHHHHHHH > > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > > 0x0000000100060000:? ?. . . . . . . . . . . . . . > > SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x0000000100080000:? ?. . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > > mmmmmmmmmmmmmmmmMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000a0000:? ?. . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000c0000:? ?. . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > > mmmmmmmmmmmmmmmmMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x00000001000e0000:? ?. . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > M > > 0x0000000100100000:? ?. . . . . . . . . . . . . . . . . . . > . . . . . > > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM > > MMMMMMMMMMMMMMMMMMMSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > 0x0000000100120000:? ?. . . . . . . . . . . . . . . . . . . > . . . . . . . . > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > . . . . . . . > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > > > > > Thank you! > > > > Thomas > > > From calvin.cheung at oracle.com Wed Nov 8 17:27:14 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 08 Nov 2017 09:27:14 -0800 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> Message-ID: <5A033E72.90800@oracle.com> On 11/7/17, 6:12 AM, Thomas St?fe wrote: > Hi Calvin, > > On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung > > wrote: > > > > On 10/27/17, 12:55 AM, Thomas St?fe wrote: >> Hi Calvin, >> >> On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >> > wrote: >> >> Hi Thomas, >> >> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >> >> Hi Calvin, >> >> this is a worthwhile addition, thank you for your work! >> >> Thanks for your review. >> >> >> Thanks for your work :) >> >> First off, there is another issue in >> file_attribute_data_to_stat(). We actually had this issue before, >> but your work uncovered it: >> >> According to POSIX >> (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html >> ) >> and every unix manpage I looked at (solaris, linux, aix..), >> st_ctime is not the file creation time but the last time file >> status was changed. Only Microsoft gets this wrong in their posix >> layer, its stat() function returns >> (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx >> ) >> creation time in st_ctime. >> >> But as os::stat() is a platform independent layer, I'd say we >> should behave the same on all platforms, and that would be the >> Posix way. >> >> I did not find any code calling os::stat() and using st_ctime, so >> this is still save to change for windows. (Note that I found that >> perfMemory_xxx.cpp uses plain OS ::stat and misuses ctime as >> "creation time" - I opened a bug for that >> (https://bugs.openjdk.java.net/browse/JDK-8190260 >> - but that >> does not affect us, as they do not call os::stat()). >> >> There is one more complication: in os::stat() we use plain >> ::stat() in the single byte case.: >> >> *+ if (strlen(path) < MAX_PATH) {* >> *+ ret = ::stat(pathbuf, sbuf);* >> *+ } else {* >> * >> * >> But ::stat() on Windows is broken, as I wrote above. We should >> not use it, especially not if we change the meaing of st_ctime in >> the double byte path. >> >> My suggestion would be to just always call GetFileAttributes(), >> also for the single byte path. A very simple solution would be to >> just always go the double byte path with UNC translation, >> regardless of the path length. Or, if you are worried about >> performance, for paths shorter than MAX_PATH we use >> GetFileAttributesA(), for longer paths GetFileAttributesW with >> UNC translation. In both cases you get >> a WIN32_FILE_ATTRIBUTE_DATA which you can feed tp your >> file_attribute_data_to_stat() routine and have the struct stat >> you return be always consistent. > I ran into an issue with the dwFileAttributes value for a jar > file. On Windows Server O/S, the value is set to 0x2020 which is > (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) but > on a desktop O/S like Windows 7, it is set to 0x0020 which is just > FILE_ATTRIBUTE_ARCHIVE. I've fixed it in > file_attribute_data_to_stat(). > > > Oh.. :( good catch! But that opens a new can of worms I did not see > before: > > FILE_ATTRIBUTE_NORMAL is documented as "A file that does not have > other attributes set. This attribute is valid only when used alone." > In addition to this flag, we have a multitude of things like > FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, > FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this is a > normal file in Unix terminology and where we would expect os::stat to > return S_IFREG, but where according to the msdn doc > FILE_ATTRIBUTE_NORMAL is not set. > > Looking at what others do in this scenario (Looked at mingw sources > and at ReactOS - I am not posting any source code here for legal > reasons but feel free to look for yourself), the usual way to > translate WIN32_FILE_ATTRIBUTES to struct stat seems to be: > "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, no > dependency on FILE_ATTRIBUTE_NORMAL). This makes sense but I ran into similar problem as before - the dwFileAttributes has a different value on a windows server O/S vs desktop O/S. So I need to do the check as follows: + // A directory has the dwFileAttributes value of 0x2010 which is + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) + // on Windows Server O/S but the value is 0x0010 on Windows desktop O/S + // such as Windows 7. + if ((file_data.dwFileAttributes& FILE_ATTRIBUTE_DIRECTORY) != 0) { + sbuf->st_mode |= S_IFDIR; + } else { + sbuf->st_mode |= S_IFREG; + } updated webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ Diff from webrev.03: < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 08:50:40.170786397 -0800 < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 08:50:39.803751877 -0800 < @@ -4060,41 +4060,119 @@ --- > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 09:40:13.657460425 -0700 > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 09:40:13.261423192 -0700 > @@ -4060,41 +4060,121 @@ 25,29c25 < + // A directory has the dwFileAttributes value of 0x2010 which is < + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) < + // on Windows Server O/S but the value is 0x0010 on Windows desktop O/S < + // such as Windows 7. < + if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0) { --- > + if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { 31c27,33 < + } else { --- > + } > + if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || > + // an archive file such as a jar file has the dwFileAttributes value of > + // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) > + // on Windows Server O/S but the value is 0x0020 on > + // Windows desktop O/S such as Windows 7. > + ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) != 0)) { > > I wonder about other special cases which should work too: > - read-only files should be S_IFREG and !S_IWRITE, For a read-only system file under the user's home dir. st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | FILE_ATTRIBUTE_READONLY) > read-only directories should be S_IFDIR and !S_IWRITE. I've tried the C:\progra~1 dir. st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | FILE_ATTRIBUTE_READONLY) > - We should tread the device root ("C:\") as a directory (so, > os::stat("c:\") should return S_IFDIR). Does this work? This one works too. st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) > > I've also used GetTickCounts() to measure the performance of > ::stat() vs GetFileAttributesA() plus > file_attribute_data_to_stat(). There's no difference at the ms > resolution. Using the high-resolution timestamp > (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx) > , > it doesn't show much difference. > > > stat() seems to be implemented using FindFirstFile() (see crt sources > if you can), and we call GetFileAttributesA(), so I do not think this > differs much. Yes, I saw the same in my Visual Studio installation. thanks, Calvin > >> >> But onward: >> >> >> >> ========================= >> >> src/hotspot/os/windows/os_windows.cpp >> >> Could you please use wchar_t instead of WCHAR? >> >> Fixed. >> >> >> --------------- >> in os::stat(): >> >> This cumbersome malloc/strcpy sequence: >> >> ! size_t path_len = strlen(path) + 1; >> + char* pathbuf = (char*)os::malloc(path_len * >> sizeof(char), mtInternal); >> + if (pathbuf == NULL) { >> + errno = ENOMEM; >> return -1; >> } >> os::native_path(strcpy(pathbuf, path)); >> >> can be reduced to a simple strdup: >> >> char* pathbuf = os::strdup(path, mtInternal); >> if (pathbuf == NULL) { >> errno = ENOMEM; >> return -1; >> } >> os::native_path(pathbuf); >> >> This also would separate strcpy() from os::native_path() >> which is a bit unreadable. >> >> I've made the change. >> >> >> -------------------- >> >> The single-byte path (strdup, ::stat()), together with >> its free(), can be moved inside the (strlen(path) < >> MAX_PATH) condition. os::native_path will not change the >> path length (it better not, as it operates on the input >> buffer). That avoids having two allocations on the >> doublebyte path. >> >> >> os::native_path() converts a path like C:\\\\somedir to >> C:\\somedir >> So I'll need the converted path in the wchar_t case too. The >> freeing of the pathbuf is being done at the end of the >> function or in the middle of the wchar_t case if there's an >> error. >> >> >> Oh, you are right,of course. I missed that it this is needed for >> both paths. >> >> >> >> ----------------------- >> >> But seeing that translation to UNC paths is something we >> might want more often, I would combine allocation and UNC >> prefix adding to one function like this, to avoid the >> memove and increase reusability: >> >> // Creates an unc path from a single byte path. Return >> buffer is allocated in C heap and needs to be freed by >> caller. Returns NULL on error. >> static whchar_t* create_unc_path(const char* s) { >> - does s start with \\?\ ? >> - yes: >> - os::malloc(strlen(s) + 1) and mbstowcs_s >> - no: >> - os::malloc(strlen(s) + 1 + 4), mbstowcs_s >> to fourth position in string, prefix with L"\\?\" >> } >> >> >> I also include the case for adding L"\\\\?\\UNC\0" at the >> beginning to be consistent with libjava/canonicalize_md.c. >> >> >> We also need error checking to mbstowcs_s. >> >> I've added assert like the following after the call: >> >> assert(converted_chars == path_len, "sanity"); >> >> >> >> create_unc_path() : >> >> - could you convert the /**/ to // comments, please? > Fixed. > > > thanks. > > >> >> - not sure about the assert after mbstowcs_s: if we happen to >> encounter an invalid multibyte character, function will fail and >> return an error: >> >> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >> >> "If mbstowcs_s encounters an invalid multibyte character, it puts >> 0 in *``pReturnValue, sets the destination buffer to an empty >> string, sets errno to EILSEQ, and returns EILSEQ." > I've changed create_unc_path() so that the caller will get the > errno and removed the assert. > > static wchar_t* create_unc_path(const char* path, errno_t &err) > > > Okay, works for me. > > >> >> As this is dependent on user data, we should not assert, but >> handle the return code of mbstowcs_s gracefully. Same goes for >> libjava/canonicalize_md.c. >> >> - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + 7, >> path, path_len); >> third parameter is wrong, as we hand in an offset into the >> buffer, we must decrement the buffer size by this offset, so >> correct would be path_len +7 - 7 or just path_len. >> >> - Same error below: + ::mbstowcs_s(&converted_chars, &wpath[4], >> path_len + 4, path, path_len); > Fixed in both places. > > > Okay. > > >> >> >> Just for cleanliness, I would then wrap deallocation into >> an own function. >> >> static viud destroy_unc_path(whchar_t* s) { os::free(s); } >> >> I've added the destroy function. >> >> >> These two functions could be candidates of putting into >> os::win32 namespace, as this form of ANSI->UNC path >> translation is quite common - whoever wants to use the >> xxxxW() functions must do this. >> >> I'm leaving them in os_windows.cpp since they're being used >> only within that file. >> >> >> Fine by me. >> >> >> >> ----------------------- >> >> FindFirstFileW: >> >> I am pretty sure that you can achieve the same result >> with GetFileAttributesExW(). It also returns >> WIN32_FIND_DATAW. >> >> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very >> similar to WIN32_FIND_DATAW. >> >> >> It is way more straightforward to use than >> FindFirstFileW, as it does not require you to write a >> callback hook. >> >> I've switched to using GetFileAttributesExW(). >> >> >> Thank you, this is way more readable. >> Another issue: do we still need the fix for 6539723 if we switch >> from stat() to GetFileAttributesExW()? This fix looks important, >> but the comment >> indicates that it could break things if the original bug is not >> present. >> >> Btw, this is another strong argument for scrapping ::stat() >> altogether on all code paths, not only for long input paths, >> because stat() and GetFileAttributesExW() may behave >> differently. So it would be good to use the same API on all code >> paths, in order to get the best test coverage. > For this round of change, I'm using GetFileAttributesEx[A|W]() for > both code paths. > > webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ > > > thanks, > Calvin > > > Okay, all good apart from the issues mentioned above. Thanks for your > work! > > Best Regards, Thomas > > > >> >> >> >> ------- >> >> eval_find_data(): This is more of a generic helper >> function, could you rename this to something clearer, >> e.g. make_double_word() ? >> >> Ok. I've renamed it. >> >> >> Also, setup_stat(), could this be renamed more clearly to >> something like WIN32_FIND_DATA_to_stat? or lowercase if >> this bothers you :) >> >> I'm naming the function as file_attribute_data_to_stat() to >> match with the data structure name. >> >> >> Thanks for taking my suggestions. >> >> >> >> ================================== >> src/hotspot/share/classfile/classLoader.cpp >> >> In ClassPathDirEntry::open_stream(), I would feel better >> if we were asserting _dir and name to be != NULL before >> feeding it to strlen. >> >> I've added an assert statement. >> >> >> =================== >> >> Here's an updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >> >> >> >> Thanks! >> >> Thomas >> >> thanks, >> Calvin >> >> >> Thanks! >> >> Thomas >> >> >> >> >> >> >> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >> > >> > >> wrote: >> >> I've reworked this fix by using the Unicode version >> of open >> (wopen) to handle path name longer than max path with >> the path >> prefixed to indicate an extended length path as >> described in [1]. >> >> The Unicode version of stat (wstat) doesn't work well >> with long >> path [2]. So FindFirstFileW will be used.The data in >> WIN32_FIND_DATA returned from FindFirstFileW needs to be >> transferred to the stat structure since the caller >> expects a >> return stat structure and other platforms return a >> stat structure. >> >> In classLoader.cpp, calculate the size of buffer >> required instead >> of limiting it to JVM_MAXPATHLEN. >> In os_windows.cpp, dynamically allocate buffers in >> os::open and >> os::stat. >> >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >> >> > > >> >> It passed hs-tier2 testing using mach5. >> >> thanks, >> Calvin >> >> [1] >> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH >> >> > > >> >> [2] >> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >> >> > > >> >> >> >> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >> >> JBS: >> https://bugs.openjdk.java.net/browse/JDK-8188122 >> >> > > >> >> Adding a warning message if the full path or the >> directory >> length exceeds MAX_PATH on windows. >> >> Example warning messages. >> >> 1) The full path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: >> construct full path >> name failed: total length 270 exceeds max length >> of 260 >> dir >> >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> length 259 >> name Hello.class length 11 >> >> 2) The directory path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: >> os::stat failed: >> path length 265 exceeds max length 260 >> path >> >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >> >> > > >> >> Testing: >> JPRT (including the new test) >> >> thanks, >> Calvin >> >> >> > From daniel.daugherty at oracle.com Wed Nov 8 17:49:43 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 8 Nov 2017 12:49:43 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: Message-ID: <543b5bd6-1f5e-37df-d4fc-199b4e67fca3@oracle.com> Greetings, This code review from Stefan Karlsson was originally posted on an Oracle internal alias that we use for discussing HotSpot SMR development issues. That subject was: "Thread-SMR (8167108)(JDK10): CR round 0 changes". I did not have time to address Stefan's review before I went on vacation and Stefan graciously allowed me to defer his review to the OpenJDK review. With Stefan's permission, I'm replying to his code review on the OpenJDK aliases that we're using for the JDK-8167108 code review. On 10/6/17 1:19 PM, Stefan Karlsson wrote: > Hi Dan, > > Great that this is getting done! Thanks! It has been an adventure for all three primary contributors... > Erik already mentioned the file so I think that's on track, and that's > what I was most concerned about. > > > I have not been involved in the review of this patch at all, but now > that I've been looking at it I have a few comments. I hope you don't > mind. I don't want them to block the open review request, but at the > same time I'd like to share my opinion and have it weighted in for the > future of this code. > > 1) ThreadsListHandle allows a safepoint to block and allow GCs to run > and restructure objects and metadata, iff the ThreadsListHandle is > nested. This forced me to review all usages of ThreadsListHandle in > great detail to visually verify that it didn't break the surrounding code. > > If ThreadsListHandle didn't ever block for safepoints, I wouldn't have > had to care about that aspect when reading and reviewing the code. > This also means that we in the future runs the risk of someone > accidentally adding a nested ThreadsListHandle somewhere where they > don't expect a safepoint to happen. We included the following to automatically sanity check the placement of the ThreadsListHandle: src/hotspot/share/runtime/thread.cpp: 3596 ThreadsList *Threads::acquire_stable_list(Thread *self, bool is_ThreadsListSetter) { 3597?? assert(self != NULL, "sanity check"); 3598?? // acquire_stable_list_nested_path() will grab the Threads_lock 3599?? // so let's make sure the ThreadsListHandle is in a safe place. 3600?? // ThreadsListSetter cannot make this check on this code path. 3601?? debug_only(if (!is_ThreadsListSetter && StrictSafepointChecks) self->check_for_valid_safepoint_state(/* potential_vm_operation */ false);) So each ThreadsListHandle location is checked for being a valid safepoint state. We tested this idea with our work on JvmtiEnv::GetThreadLocalStorage(). GetThreadLocalStorage() is called from the 'in native' state so we cannot place a common ThreadsListHandle for all code paths because it would fail the assertion when called with 'thread == NULL', i.e., the thread is operating on itself. Update: Erik is going to explore a lock free solution for the nesting algorithm. That will likely mean that a ThreadsListHandle will not require placement in a safepoint safe location... > If the lock protecting the threads list could be taken with > no_safepoint_check, then the described problem above would completely > vanish. Unfortunately, we can't acquire the Threads_lock with > no_safepoint_check. A solution to this would be to introduced a > low-rank (rank == access) lock, say ThreadsList_lock, and always take > it with no_safepoint_check. The problem with switching locks is that we are protecting the scanning phase of the SMR algorithm with the Threads_lock so we would have to switch that lock also. I believe we use the Threads_lock with the scanning because we're using closures... Of course, Erik or Robbin should probably jump in here... :-) Update: Erik is going to explore a lock-free solution for the nesting algorithm. > That solution would also solve a potential lock-rank problem, we have > that ThreadsListHandle can't be used if a higher-rank lock is held. So far we haven't run into that problem and we think that the check_for_valid_safepoint_state() will catch that if the code should evolve to introduce one. Update: Erik is going to explore a lock-free solution for the nesting algorithm. > 2) The following one-liner: > - for (JavaThread* t = Threads::first(); t; t = t->next()) { > has been converted to a five-liner: > > + { > + ThreadsListHandle tlh; > + JavaThreadIterator jti(tlh.list()); > + for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { > ... + } I find that unfortunate, and would prefer if this was > rewritten. For example: for (JavaThreadIterator jti; JavaThread* t = > jit.next();) { jti will implicitly hold the ThreadListHandle on the > stack. I know that the implicit conversion of pointer to bool is > non-conventional in the HotSpot code, but in this case I prefer that > over five lines of extra noise in the GC code. Coleen made a similar comment in her OpenJDK review: > Hi,? From my initial look at this, I really like new interface for > walking the threads list, except every instance but 2 uses of > JavaThreadIterator has a preceding ThreadsListHandle declaration. > > +? ThreadsListHandle tlh; > +? JavaThreadIterator jti(tlh.list()); > > > Could there be a wrapper that embeds the ThreadsListHandle, like: > > class JavaThreadsListIterator { > ??? ThreadsListHandle _tlh; > ... > } Yup a definite code noise problem. We originally didn't have an iterator and used direct thread_at(foo) calls so we had a pretty close correspondence between the old for-loop and the new for-loop. Early reviewers asked for an iterator abstraction so we modeled JavaThreadIterator after other existing iterators in the JVM/TI area... (Yes, Dan stayed pretty close to "home" here...) There are places where we still need the existing JavaThreadIterator because a ThreadsList is passed down a call stack... So we've added a JavaThreadIteratorWithHandle class. Here's an example of the noisy code in the GC area: src/hotspot/share/gc/g1/dirtyCardQueue.cpp: @@ -319,8 +320,12 @@ ?? clear(); ?? // Since abandon is done only at safepoints, we can safely manipulate ?? // these queues. -? for (JavaThread* t = Threads::first(); t; t = t->next()) { -??? t->dirty_card_queue().reset(); +? { +??? ThreadsListHandle tlh; +??? JavaThreadIterator jti(tlh.list()); +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { +????? t->dirty_card_queue().reset(); +??? } ?? } ?? shared_dirty_card_queue()->reset(); ?} Here's an example of the revised code in the GC area: src/hotspot/share/gc/g1/dirtyCardQueue.cpp: @@ -319,7 +320,7 @@ ?? clear(); ?? // Since abandon is done only at safepoints, we can safely manipulate ?? // these queues. -? for (JavaThread* t = Threads::first(); t; t = t->next()) { +? for (JavaThreadIteratorWithHandle jtiwh; JavaThread *t = jtiwh.next(); ) { ???? t->dirty_card_queue().reset(); ?? } ?? shared_dirty_card_queue()->reset(); This is definitely much less noisy and I'm hoping I don't catch too much grief for the implied boolean usage... > 3) cv_internal_thread_to_JavaThread Has one return value and two > output parameter. It forces the callers to setup stack lock variables > for the two new variables. --- Used to be --- > - oop java_thread = JNIHandles::resolve_non_null(jthread); > java_lang_Thread::set_priority(java_thread, (ThreadPriority)prio); - > JavaThread* thr = java_lang_Thread::thread(java_thread); - if (thr != > NULL) { // Thread not yet started; priority pushed down when it is - > Thread::set_priority(thr, (ThreadPriority)prio); > --- Now is --- > + oop java_thread = NULL; + JavaThread* receiver = NULL; + bool > is_alive = tlh.cv_internal_thread_to_JavaThread(jthread, &receiver, > &java_thread); java_lang_Thread::set_priority(java_thread, > (ThreadPriority)prio); + + if (is_alive) { + // jthread refers to a > live JavaThread. + Thread::set_priority(receiver, > (ThreadPriority)prio); --- Maybe could be --- > oop java_thread = JNIHandles::resolve_non_null(jthread); > java_lang_Thread::set_priority(java_thread, (ThreadPriority)prio); > JavaThread* thr = tlh.cv_internal_thread_to_JavaThread(java_thread); > if (thr != NULL) { // Thread not yet started; priority pushed down > when it is Thread::set_priority(thr, (ThreadPriority)prio); I don't > see the need to for the extra complexity of the two output parameters. > The return value is always true when thr is non-null, and always false > when thr is null. There are three cv_*_to_JavaThread() functions and we had to craft them very, very carefully to avoid changing some of the subtle semantics at the original call sites. If you carefully examine the output parameters and return value usages at all of the call sites, you'll see that each was added to fit some existing semantic that we didn't want to risk changing. Of course, not all of the call sites need all of the special behaviors individually, but they do as a union. In short, compatibility and risk management led us here. > But the usage of cv_internal_thread_to_JavaThread is contained within > the Runtime code, so my opinion should probably not weigh as much as > other's opinions. :) We definitely agree this is a mess, but not one we're willing to risk changing at this time. Dan has made the observation that the JVM_* entry points, like Y2K code, shows a variety of different coding patterns, each with different (potential) issues... Thanks for being flexible on accepting cv_internal_thread_to_JavaThread() as is for now! > 4) I'd prefer if abbreviations where expanded, so that the casual > reader would immediately understood the code. For example: t_list -> > thread_list cv_internal_thread_to_JavaThread -> > internal_thread_to_JavaThread We've had requests to shorten names, spell out names, use different names, etc. It seems that no one is going to be happy with all of the names we used in this code. Our guidelines: - use the same consistently, e.g., t_list for ThreadsLists - use a name that doesn't show up in an existing grep (if possible) We hope you don't mind, but we're not planning to make any more name changes... > 5) This macro and the jesting is pretty bad. Yup! > I complained about it to Erik, and then he confessed that he wrote it :D > +// Possibly the ugliest for loop the world has seen. C++ does not > allow +// multiple types in the declaration section of the for loop. > In this case +// we are only dealing with pointers and hence can cast > them. It looks ugly +// but macros are ugly and therefore it's fine to > make things absurdly ugly. +#define DO_JAVA_THREADS(LIST, X) \ + for > (JavaThread *MACRO_scan_interval = > (JavaThread*)(uintptr_t)PrefetchScanIntervalInBytes, \ + *MACRO_list = > (JavaThread*)(LIST), \ + **MACRO_end = > ((JavaThread**)((ThreadsList*)MACRO_list)->threads()) + > ((ThreadsList*)MACRO_list)->length(), \ + **MACRO_current_p = > (JavaThread**)((ThreadsList*)MACRO_list)->threads(), \ + *X = > (JavaThread*)prefetch_and_load_ptr((void**)MACRO_current_p, > (intx)MACRO_scan_interval); \ + MACRO_current_p != MACRO_end; \ + > MACRO_current_p++, \ + X = > (JavaThread*)prefetch_and_load_ptr((void**)MACRO_current_p, > (intx)MACRO_scan_interval)) + This can be rewritten without all these > cast, and minimal usage of the macros expansions: struct > JavaThreadPrefetchedIterator { ThreadList* _list; JavaThread** _end; > JavaThread** _current; JavaThreadIteror(ThreadList* list) : > _list(list), _end(list->threads() + list->length()), > _current(list->threads()) {} JavaThread* current() { return > context._current != context._end ?? > prefetch_and_load_ptr(context._current, PrefetchScanIntervalInBytes) > ?: NULL) // ^ prefetch would be rewritten to return JavaThread* and > not void* } void next() { _current++; }; #define DO_JAVA_THREADS(LIST, > X) \ for (JavaThreadPrefetchedIterator iter(LIST); JavaThread* X = > iter.current(); iter.next()) (I did some changes to the code above and > haven't compiled this exact version) Erik hasn't chimed in on this comment so I (Dan) am not planning to try to resolve this comment in this round. Update: Erik is planning to look at cleaning up the ugly macro... > 6) I think it would be nice if the SMR stuff in thread.hpp were > encapsulated into an class instead of added directly to Thread and > Threads. I sort-of expected the SMR variables to be moved to > threadSMR.hpp. For example: > class Threads: AllStatic { friend class VMStructs; private: + // Safe > Memory Reclamation (SMR) support: + static Monitor* _smr_delete_lock; > + // The '_cnt', '_max' and '_times" fields are enabled via + // > -XX:+EnableThreadSMRStatistics: + static uint > _smr_delete_lock_wait_cnt; + static uint _smr_delete_lock_wait_max; + > static volatile int _smr_delete_notify; + static volatile jint > _smr_deleted_thread_cnt; + static volatile jint > _smr_deleted_thread_time_max; + static volatile jint > _smr_deleted_thread_times; + static ThreadsList* volatile > _smr_java_thread_list; + static ThreadsList* > get_smr_java_thread_list() { + return > (ThreadsList*)OrderAccess::load_ptr_acquire((void* > volatile*)&_smr_java_thread_list); + } + static ThreadsList* > xchg_smr_java_thread_list(ThreadsList* new_list) { + return > (ThreadsList*)Atomic::xchg_ptr((void*)new_list, (volatile > void*)&_smr_java_thread_list); + } + static long > _smr_java_thread_list_alloc_cnt; + static long > _smr_java_thread_list_free_cnt; + static uint > _smr_java_thread_list_max; + static uint _smr_nested_thread_list_max; > + static volatile jint _smr_tlh_cnt; + static volatile jint > _smr_tlh_time_max; + static volatile jint _smr_tlh_times; + static > ThreadsList* _smr_to_delete_list; + static uint > _smr_to_delete_list_cnt; + static uint _smr_to_delete_list_max; Could > be: class Threads: AllStatic { friend class VMStructs; private: // > Safe Memory Reclamation (SMR) support: SMRSupport _smr_support; And > SMRSupport could be moved to threadSMR.hpp. I haven't read all the > code in detail, so I'm not sure if this is feasible or not, but my > gut-feeling says that it would be worth testing. The above is just an > example, and the rest of the code could probably be encapsulated and > moved as well. We already moved all of the SMR stuff that was "easy" to move based on your earlier comment. (Of course, doing that move introduced a number of compilation complications, but that's just Dan complaining :-)) We don't really want to try this migration at this time since we're still hoping to get this code into 18.3. (Also Dan doesn't see the value in moving the SMR fields... Threads is already an eclectic static class so why does SMR have to encapsulate when no other project did so?) > 7) Currently, threadSMR.hpp "only" contains the ThreadList. Unless, > you move the SMR stuff into threadSMR.hpp, maybe it should be renamed > to threadsList.hpp? Maybe it does make sense to have both > threadSMR.hpp and threadsList.hpp? We're planning to stick with the threadSMR.hpp and threadSMR.cpp file names. Currently they contain the more standalone pieces of thread SMR (more than just ThreadsList) so we're planning to keep those names. Thanks for the detailed review!! Dan, Erik and Robbin > Cheers and thanks! StefanK From daniel.daugherty at oracle.com Wed Nov 8 17:53:22 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 8 Nov 2017 12:53:22 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> Message-ID: <10022dbf-74a6-5b38-265e-079ebaad22c8@oracle.com> Added back the other three OpenJDK aliases being used for this review... Coleen, thanks for chiming in on this review thread. Sorry for the delay in responding... I was on vacation... On 10/11/17 11:32 AM, coleen.phillimore at oracle.com wrote: > > Hi,? From my initial look at this, I really like new interface for > walking the threads list, except every instance but 2 uses of > JavaThreadIterator has a preceding ThreadsListHandle declaration. > > +? ThreadsListHandle tlh; > +? JavaThreadIterator jti(tlh.list()); > > > Could there be a wrapper that embeds the ThreadsListHandle, like: > > class JavaThreadsListIterator { > ??? ThreadsListHandle _tlh; > ... > } This issue has been addressed as part of my response to Stefan K's code review comments. I quoted the above comment in that reply so it should be easy to find that resolution... Dan > > Thanks, > Coleen > > On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> We have a (eXtra Large) fix for the following bug: >> >> 8167108 inconsistent handling of SR_lock can lead to crashes >> https://bugs.openjdk.java.net/browse/JDK-8167108 >> >> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >> Hazard Pointers to manage JavaThread lifecycle. >> >> Here's a PDF for the internal wiki that we've been using to describe >> and track the work on this project: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >> >> >> Dan has noticed that the indenting is wrong in some of the code quotes >> in the PDF that are not present in the internal wiki. We don't have a >> solution for that problem yet. >> >> Here's the webrev for current JDK10 version of this fix: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >> >> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >> testing, additional stress testing on Dan's Solaris X64 server, and >> additional testing on Erik and Robbin's machines. >> >> We welcome comments, suggestions and feedback. >> >> Daniel Daugherty >> Erik Osterlund >> Robbin Ehn > From daniel.daugherty at oracle.com Wed Nov 8 18:05:57 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 8 Nov 2017 13:05:57 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Message-ID: Greetings, Resolving one of the code review comments (from both Stefan K and Coleen) on jdk10-04-full required quite a few changes so it is being done as a standalone patch and corresponding webrevs: Here's the mq comment for the change: ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use ??? JavaThreadIteratorWithHandle. Here is the full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ And here is the delta webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: > Greetings, > > We have a (eXtra Large) fix for the following bug: > > 8167108 inconsistent handling of SR_lock can lead to crashes > https://bugs.openjdk.java.net/browse/JDK-8167108 > > This fix adds a Safe Memory Reclamation (SMR) mechanism based on > Hazard Pointers to manage JavaThread lifecycle. > > Here's a PDF for the internal wiki that we've been using to describe > and track the work on this project: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf > > > Dan has noticed that the indenting is wrong in some of the code quotes > in the PDF that are not present in the internal wiki. We don't have a > solution for that problem yet. > > Here's the webrev for current JDK10 version of this fix: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full > > This fix has been run through many rounds of JPRT and Mach5 tier[2-5] > testing, additional stress testing on Dan's Solaris X64 server, and > additional testing on Erik and Robbin's machines. > > We welcome comments, suggestions and feedback. > > Daniel Daugherty > Erik Osterlund > Robbin Ehn > From jiangli.zhou at oracle.com Wed Nov 8 18:09:11 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Wed, 8 Nov 2017 10:09:11 -0800 Subject: RFR[XXS]: 8186778 Make obsolete VM options for shared region size control In-Reply-To: <26645083-96ad-9fb8-9370-6dbf77319b68@oracle.com> References: <26645083-96ad-9fb8-9370-6dbf77319b68@oracle.com> Message-ID: <6637C9C8-FFE2-4695-B9C6-B4654A209B81@oracle.com> Looks good. Thanks, Jiangli > On Nov 7, 2017, at 9:33 AM, Ioi Lam wrote: > > Hi, > > Please review this very small change. These VM arguments are no longer used when creating a CDS archive, so they should be declared obsolete: > > https://bugs.openjdk.java.net/browse/JDK-8186778 > http://cr.openjdk.java.net/~iklam/jdk10/8186778-make-onsolete-shared-region-size-args.v01/ > > Thanks > - Ioi > > ----- > $ java -XX:SharedReadWriteSize=123 -XX:SharedReadOnlySize=123 -XX:SharedMiscDataSize=123 -XX:SharedMiscCodeSize=123 -version > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedReadWriteSize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedReadOnlySize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedMiscDataSize; support was removed in 10.0 > Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option SharedMiscCodeSize; support was removed in 10.0 > java version "10-internal" > Java(TM) SE Runtime Environment (build 10-internal+0-adhoc.iklam.open) > Java HotSpot(TM) 64-Bit Server VM (build 10-internal+0-adhoc.iklam.open, mixed mode) > From thomas.stuefe at gmail.com Wed Nov 8 19:26:44 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 08 Nov 2017 19:26:44 +0000 Subject: alternate pns() debug function In-Reply-To: References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> Message-ID: Hi Chris, On Wed 8. Nov 2017 at 19:46, Chris Plummer wrote: > Hi Thomas, > > > On 11/8/17 7:33 AM, Thomas St?fe wrote: > > Hi Chris, Ioi, > > On Tue, Nov 7, 2017 at 8:20 AM, Ioi Lam wrote: > >> >> >> On 11/6/17 9:05 PM, Chris Plummer wrote: >> >>> Hi, >>> >>> I'd like to add an alternate version of pns() to debug.cpp, and would >>> like some initial comments first before filing a bug and sending out an >>> RFR. pns() is used to print out the native stack: >>> >>> extern "C" void pns(void* sp, void* fp, void* pc) { // print native stack >>> Command c("pns"); >>> static char buf[O_BUFLEN]; >>> Thread* t = Thread::current_or_null(); >>> // Call generic frame constructor (certain arguments may be ignored) >>> frame fr(sp, fp, pc); >>> VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >>> } >>> >>> And here's the help output for it: >>> >>> pns(void* sp, void* fp, void* pc) - print native (i.e. mixed) stack >>> trace. E.g."); >>> pns($sp, $rbp, $pc) on Linux/amd64 and Solaris/amd64 >>> or"); >>> pns($sp, $ebp, $pc) on Linux/x86 or"); >>> pns($sp, 0, $pc) on Linux/ppc64 or"); >>> pns($sp + 0x7ff, 0, $pc) on Solaris/SPARC"); >>> - in gdb do 'set overload-resolution off' before >>> calling pns()"); >>> - in dbx do 'frame 1' before calling pns()"); >>> >>> The intent is to call it from gdb, but it is potentially useful to call >>> it from within hotspot when debugging. I'd like to propose the following >>> alternative, since it does away with needing to pass in register values >>> (not easy to do from C): >>> >>> extern "C" void pns2() { // print native stack >>> Command c("pns2"); >>> static char buf[O_BUFLEN]; >>> Thread* t = Thread::current_or_null(); >>> frame fr = os::current_frame(); >>> VMError::print_native_stack(tty, fr, t, buf, sizeof(buf)); >>> } >>> >>> I actually used this quite a bit with some recent debugging. There were >>> places in hotspot where I wanted to know what the native stack looked like >>> when called, and it was a lot easier to just add a call to pns2() than to >>> setup the test to run in gdb. This seems to work on Linux/x64, macosx/64, >>> and solaris/sparc. For some reason it crashes on Windows/x64, but I don't >>> see any indication from the above help for pns() that it works on >>> Windows/x64 anyway. >>> >>> # >>> # A fatal error has been detected by the Java Runtime Environment: >>> # >>> # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000003bcfae1466, >>> pid=9872, tid=12212 >>> # >>> # JRE version: Java(TM) SE Runtime Environment (10.0) (fastdebug build >>> 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs) >>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug >>> 10-internal+0-2017-11-07-0256309.chris.plummer.jdk10-hs, mixed mode, >>> tiered, compressed oops, g1 gc, windows-amd64) >>> # Problematic frame: >>> # v ~StubRoutines::get_previous_fp >>> # >>> >>> I'm starting to suspect that os::current_frame() normally is never even >>> called on Windows (there are calls to it in the VmError code, but I suspect >>> they are never triggered on Windows) and that >>> StubRoutines::get_previous_fp() is not working on Windows, but I could be >>> wrong. >>> >>> Hi Chris, >> >> I like this new pns2() function. >> >> To make this work on Windows, you probably need to follow this pattern in >> vmError.cpp. >> >> >> if (os::platform_print_native_stack(st, _context, buf, sizeof(buf))) >> { >> // We have printed the native stack in platform-specific code >> // Windows/x64 needs special handling. >> } else { >> frame fr = _context ? os::fetch_frame_from_context(_context) >> : os::current_frame(); >> >> print_native_stack(st, fr, _thread, buf, sizeof(buf)); >> } >> >> Thanks >> - Ioi >> >> > Note that os::platform_print_native_stack() on Windows and AIX only print > native C/C++ frames. So, calling this from the debugger may not be that > useful. > > With the recent debug session I had, it was useful even with just the VM > part of the backtrace. > Interesting. What did you see what the debugger would not show you? > On AIX, it probably will not even be possible to invoke a function in the > debugger - never tried, but given that not much else works in that thing, I > am pessimistic. > > I was also pessimistic with gdb, so I tried it yesterday, and it didn't > work. It's not in the proper context for os::current_frame() to provide the > right information. So this will only be useful when inserted in hotspot > code for debugging purposes. > > > But pns() may still be useful as a tracing option, though. > > Not sure if you are talking about the original or my version. Both have > their use: pns() from the debugger and pns2() from within hotspot code. > > > On Windows x64, we do not have frame pointers, so we cannot walk the frame > VM-Style, so os::platform_print_native_stack() is the only way to print a > stack. > > Yes, and I got this to work Windows/x64, but unfortunately no symbols are > printed. > When you let the Vm crash with e.g. -XX:ErrorHandlerTest=14, does the resulting hs-err file show a correct callstack with symbols? If not, what does the dbghelp.dll loader say (the one line below the loaded libraries section)? > On AIX, both ways work - VMError::print_native_stack() and > os::platform_print_native_stack() - the former gives a mixed stack, the > latter the pure C/C++ stack, but with many more information. > > In the end, for consistency, I'd suggest the same as Ioi, to follow the > pattern in VMError(). > > I've made that change already. > > And we could probably factor that out to an own function? > > I see your point, but given the simplicity of the change (the addition of > one function with no disruption to existing code), I'm inclined to choose > simplicity over elegance. > > Sure! But having the generic os::current_frame function assert on windows x64 is a bit of a trap. Maybe we should fix this at some point. thanks, > > Chris > > Best regards, Thomas > > Kind Regards, Thomas > > > >> >> Let me know what you think. >>> >>> thanks, >>> >>> Chris >>> >>> >> > > From daniel.daugherty at oracle.com Wed Nov 8 23:09:41 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 8 Nov 2017 18:09:41 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Message-ID: <1c0edbe2-1b9c-dc7e-c610-4ea053831790@oracle.com> The jdk10-05-full bits have passed JPRT testing (hs-tier1 testing) and hs-tier[2-5] testing via Mach 5. No unexpected test failures. Dan On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: > Greetings, > > Resolving one of the code review comments (from both Stefan K and Coleen) > on jdk10-04-full required quite a few changes so it is being done as a > standalone patch and corresponding webrevs: > > Here's the mq comment for the change: > > ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use > ??? JavaThreadIteratorWithHandle. > > Here is the full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ > > And here is the delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> We have a (eXtra Large) fix for the following bug: >> >> 8167108 inconsistent handling of SR_lock can lead to crashes >> https://bugs.openjdk.java.net/browse/JDK-8167108 >> >> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >> Hazard Pointers to manage JavaThread lifecycle. >> >> Here's a PDF for the internal wiki that we've been using to describe >> and track the work on this project: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >> >> >> Dan has noticed that the indenting is wrong in some of the code quotes >> in the PDF that are not present in the internal wiki. We don't have a >> solution for that problem yet. >> >> Here's the webrev for current JDK10 version of this fix: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >> >> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >> testing, additional stress testing on Dan's Solaris X64 server, and >> additional testing on Erik and Robbin's machines. >> >> We welcome comments, suggestions and feedback. >> >> Daniel Daugherty >> Erik Osterlund >> Robbin Ehn >> > > From mikhailo.seledtsov at oracle.com Thu Nov 9 01:26:41 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 08 Nov 2017 17:26:41 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A01429B.4070806@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> Message-ID: <5A03AED1.3060203@oracle.com> Here is an update webrev: http://cr.openjdk.java.net/~mseledtsov/8189762.02/ I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). Summary for updated webrev.02: - implemented review feedback from Bob and Igor - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) My next steps: - rebase to new jdk tree - update/merge - retest Thank you, Misha On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: > Hi Igor, > > Thank you for reviewing the code. > > On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >> Hi Misha, >> >> I have several comments: >> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it >> should be WB_IsContainerized(JNIEnv*, jobject) > Fixed >> 2. CPUSetsReader.java : listToString can be rewritten using stream >> api as >> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) > Thank you. I will try it out. >> 3. in several files, I have noticed some problems w/ indents which >> make code quite hard to read, for example >>> 73 Common.run( Common.newOpts(imageName) >>> 74 .addDockerOpts("--memory", valueToSet)) >>> 75 .shouldMatch("Memory Limit is:.*" + >>> expectedTraceValue); >> or >>> 96 DockerRunOptions opts = Common.newOpts(imageName, >>> "AttemptOOM") >>> 97 .addDockerOpts("--memory", dockerMemLimit, >>> "--memory-swap", dockerMemLimit); >>> 98 opts.classParams.add("" + sizeToAllocInMb); > OK. I will unwind these expressions, and make sure indentation looks > good. >> 4. TestCPUSets.java : it's better to use >> Assert.assertEquals(parts.length, 2) than >> Asserts.assertTrue(parts.length == 2) as the former will also print >> the actual value of parts.length. so you won't need to have L#103 > OK >> 5. Test*: there is no need to have parentheses in '@requires >> (docker.support)' > OK >> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under >> test, so you can use '@run driver' to run it > Will fix. >> 7. TestCPUSets.java : L#72, to get a proper new line symbol you >> should use %n in format string. also System.out::printf seems to be >> more popular in our code than System.out::format > OK. >> 8. TestCPUSets.java: shouldn't checkResult assert that we found >> lineMarker? > Oops. Missed it. Will add code to check it. >> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all >> Common.run calls which aren't expected to fail, e.g. all test* in >> TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in >> TestMemoryAwareness > Common.run() already checks for .shouldHaveExitValue(0) after running > the container. > > > I will apply feedback from Bob and you, and will post a new webrev. > > > Thank you, > Misha >> >> Thanks, >> -- Igor >> >>> On Nov 1, 2017, at 8:11 PM, mikhailo >>> wrote: >>> >>> Please review these tests that were developed to test JVM's >>> container awareness in Docker environment. >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>> Webrev: >>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>> WB API: >>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>> Testing: >>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>> Ran the developed tests via jtreg >>> Pass >>> >>> 2. Automated testing system - run these tests >>> In progress >>> >>> Thank you, >>> Misha >>> From david.holmes at oracle.com Thu Nov 9 01:40:42 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 11:40:42 +1000 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: <6f99fcdb-d3b8-adf7-2d8d-21232c145a73@oracle.com> References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> <9fa4064a-5764-d3fa-c398-ebd05c24e3ed@oracle.com> <6f99fcdb-d3b8-adf7-2d8d-21232c145a73@oracle.com> Message-ID: On 9/11/2017 1:22 AM, mikhailo wrote: > > > On 11/07/2017 06:01 PM, David Holmes wrote: >> Hi Misha, >> On 8/11/2017 7:36 AM, mikhailo wrote: >>>>> Does it work for you Bob? David, are you OK with this approach? >>>> I think this is an improvement and I?m ok with this approach but I >>>> don?t see that it will save that much time. >>>> Starting docker takes? a while (worse the first time).? I?ve seen >>>> 1sec startup times.? Let see what David thinks >>>> of the /tmp caching idea.? You?d have to put the user that found and >>>> successfully ran docker and have to deal >>>> with multi-process write access to the file. >>> OK. There is another option that will work well, but requires one >>> extra input from user. We could designate a property that would state >>> whether a host is docker capable. When user wishes to execute docker >>> tests locally she/he will set the property to true. When executed via >>> Oracle automated test system, the system will set this property to >>> true on docker capable test nodes. All that VMProps will do is check >>> the property, so no docker ps any more, not even searching for docker >>> on a known locations. >>> >>> As for naming, I can propose this: jdk.test.lib.docker.capable.node. >>> Alternative suggestions of names welcome. >>> >>> This makes the check clear, simple and reliable. What do you think? >> >> This doesn't work if the intent is that we happen to add a >> docker-enabled machine to a test pool. You can't selectively set the >> property only when run on the right host (if you could then we'd only >> run the tests on that host in the first place). >> >> But you seemed to have moved from fixing this bug - 8189213 - to >> trying to address the new RFE 8190730. > You are right. I got carried away with this discussion. > > I should fix the "JDK-8189213 - [TESTBUG] Running jtreg tests on machine > without docker shows extra message" first, and then work on > > "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to > determine presence of docker engine is deemed too heavy". > > > So, the fix for this issue is simple, as we agreed before - remove the > message. Here is a webrev: > http://cr.openjdk.java.net/~mseledtsov/8189213.02/ > > May I count you as a Reviewer for JDK-8189213? Yes - and this is a trivial change so only one review needed. Thanks, David > > Thank you, > Misha > > > P.S. As for JDK-8190730, I will try to summarize the discussion in the > comments of the JBS issue. > >> >> Thanks, >> David >> >>> >>> Thank you, >>> Misha >>> >>> >>>> >>>> Bob. >>>> >>>> >>>>>> This is probably breaking new ground, but can we cache the result >>>>>> of ?docker ps? in /tmp in order to eliminate the performance >>>>>> overhead for all but the first time a new user runs these tests? >>>>> I try to keep this solution simple. If you and David approve the >>>>> solution above, I will implement and test it. If we find any >>>>> problems, either during my testing or during test production, we >>>>> could look into caching mechanisms. >>>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>>> Bob. >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>>> Let me know what you think. I can revert this part of the >>>>>>>>> change or reduce the timeout value a bit more. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>>>> For this change under review, if I do not hear opinions from >>>>>>>>>>> other reviewers, I can simply delete the print statement. >>>>>>>>>> Sounds good to me. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Misha >>>>>>>>>>>> Thanks, >>>>>>>>>>>> David >>>>>>>>>>>> >>>>>>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>>>>>> Please review this simple change that makes the error >>>>>>>>>>>>> message in jtreg-ext/requires/VMProps.java conditional >>>>>>>>>>>>> on java property "vmprops.trace.verbose". W/o this >>>>>>>>>>>>> condition an error message is printed each time any jtreg >>>>>>>>>>>>> test is ran, including non-docker tests. >>>>>>>>>>>>> >>>>>>>>>>>>> ????? JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>>>>>> ????? Webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ======= Background and details: >>>>>>>>>>>>> VMProps.java is called for each test run of jtreg (be it a >>>>>>>>>>>>> run containing a singe test or multiple tests). >>>>>>>>>>>>> VMProps evaluates the @requires properties. Therefore, in >>>>>>>>>>>>> case of this bug, any time a user runs any jtreg VM test >>>>>>>>>>>>> on a machine w/o an operational docker engine, this error >>>>>>>>>>>>> will pop up. >>>>>>>>>>>>> The fix makes printing of this error message conditional, >>>>>>>>>>>>> off by default. The message can be turend on >>>>>>>>>>>>> via a property vmprops.trace.verbose when detailed >>>>>>>>>>>>> diagnostic info is desired. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ======== Testing: >>>>>>>>>>>>> Local testing: >>>>>>>>>>>>> ??? W/o docker present: >>>>>>>>>>>>> ????? jtreg >>>>>>>>>>>>> ??????? No error message >>>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>> >>>>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>>> ??????? Test skipped >>>>>>>>>>>>> >>>>>>>>>>>>> ??? With docker installed but no permissions: >>>>>>>>>>>>> ????? jtreg >>>>>>>>>>>>> ??????? No error message >>>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>> >>>>>>>>>>>>> ??????? Detailed error message >>>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>>> ??????? Test skipped >>>>>>>>>>>>> >>>>>>>>>>>>> ??? With docker installed and operational: >>>>>>>>>>>>> ????? jtreg >>>>>>>>>>>>> ??????? No error message >>>>>>>>>>>>> ????? jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>> >>>>>>>>>>>>> ??????? No error message >>>>>>>>>>>>> ????? jtreg DockerBasicTest.java >>>>>>>>>>>>> ??????? Test passed >>>>>>>>>>>>> >>>>>>>>>>>>> Testing via automated distributed test system: >>>>>>>>>>>>> ??? - tier1 >>>>>>>>>>>>> ??? - docker tests >>>>>>>>>>>>> ??? In progress >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> Misha >>>>>>>>>>>>> >>> > From mikhailo.seledtsov at oracle.com Thu Nov 9 04:04:52 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 08 Nov 2017 20:04:52 -0800 Subject: RFR(XS): 8189213: [TESTBUG] Running jtreg tests on machine without docker shows extra message In-Reply-To: References: <7a8cf00f-6eea-6f84-1020-1a56815caa01@oracle.com> <44c57c94-5ba3-5d4b-9cce-5b5d2916b90f@oracle.com> <59FD0806.5000200@oracle.com> <5b393c49-807e-5196-2589-53b88b8c8b8f@oracle.com> <84573c8c-567b-7d03-2a9b-66adc1125ea9@oracle.com> <7CBF5614-9A24-4F8F-B31A-5A41FE7F7B80@oracle.com> <9fa4064a-5764-d3fa-c398-ebd05c24e3ed@oracle.com> <6f99fcdb-d3b8-adf7-2d8d-21232c145a73@oracle.com> Message-ID: <5A03D3E4.3000807@oracle.com> Thank you, Misha On 11/8/17, 5:40 PM, David Holmes wrote: > On 9/11/2017 1:22 AM, mikhailo wrote: >> >> >> On 11/07/2017 06:01 PM, David Holmes wrote: >>> Hi Misha, >>> On 8/11/2017 7:36 AM, mikhailo wrote: >>>>>> Does it work for you Bob? David, are you OK with this approach? >>>>> I think this is an improvement and I?m ok with this approach but I >>>>> don?t see that it will save that much time. >>>>> Starting docker takes a while (worse the first time). I?ve seen >>>>> 1sec startup times. Let see what David thinks >>>>> of the /tmp caching idea. You?d have to put the user that found >>>>> and successfully ran docker and have to deal >>>>> with multi-process write access to the file. >>>> OK. There is another option that will work well, but requires one >>>> extra input from user. We could designate a property that would >>>> state whether a host is docker capable. When user wishes to execute >>>> docker tests locally she/he will set the property to true. When >>>> executed via Oracle automated test system, the system will set this >>>> property to true on docker capable test nodes. All that VMProps >>>> will do is check the property, so no docker ps any more, not even >>>> searching for docker on a known locations. >>>> >>>> As for naming, I can propose this: >>>> jdk.test.lib.docker.capable.node. Alternative suggestions of names >>>> welcome. >>>> >>>> This makes the check clear, simple and reliable. What do you think? >>> >>> This doesn't work if the intent is that we happen to add a >>> docker-enabled machine to a test pool. You can't selectively set the >>> property only when run on the right host (if you could then we'd >>> only run the tests on that host in the first place). >>> >>> But you seemed to have moved from fixing this bug - 8189213 - to >>> trying to address the new RFE 8190730. >> You are right. I got carried away with this discussion. >> >> I should fix the "JDK-8189213 - [TESTBUG] Running jtreg tests on >> machine without docker shows extra message" first, and then work on >> >> "JDK-8190730 - [TESTBUG] Calling 'docker ps' from VMProps.java to >> determine presence of docker engine is deemed too heavy". >> >> >> So, the fix for this issue is simple, as we agreed before - remove >> the message. Here is a webrev: >> http://cr.openjdk.java.net/~mseledtsov/8189213.02/ >> >> May I count you as a Reviewer for JDK-8189213? > > Yes - and this is a trivial change so only one review needed. > > Thanks, > David > >> >> Thank you, >> Misha >> >> >> P.S. As for JDK-8190730, I will try to summarize the discussion in >> the comments of the JBS issue. >> >>> >>> Thanks, >>> David >>> >>>> >>>> Thank you, >>>> Misha >>>> >>>> >>>>> >>>>> Bob. >>>>> >>>>> >>>>>>> This is probably breaking new ground, but can we cache the >>>>>>> result of ?docker ps? in /tmp in order to eliminate the performance >>>>>>> overhead for all but the first time a new user runs these tests? >>>>>> I try to keep this solution simple. If you and David approve the >>>>>> solution above, I will implement and test it. If we find any >>>>>> problems, either during my testing or during test production, we >>>>>> could look into caching mechanisms. >>>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>> Bob. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>>>> Let me know what you think. I can revert this part of the >>>>>>>>>> change or reduce the timeout value a bit more. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Misha >>>>>>>>>>>> For this change under review, if I do not hear opinions >>>>>>>>>>>> from other reviewers, I can simply delete the print statement. >>>>>>>>>>> Sounds good to me. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> David >>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Misha >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> David >>>>>>>>>>>>> >>>>>>>>>>>>> On 3/11/2017 7:03 AM, mikhailo wrote: >>>>>>>>>>>>>> Please review this simple change that makes the error >>>>>>>>>>>>>> message in jtreg-ext/requires/VMProps.java conditional >>>>>>>>>>>>>> on java property "vmprops.trace.verbose". W/o this >>>>>>>>>>>>>> condition an error message is printed each time any jtreg >>>>>>>>>>>>>> test is ran, including non-docker tests. >>>>>>>>>>>>>> >>>>>>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189213 >>>>>>>>>>>>>> Webrev: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189213.00/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ======= Background and details: >>>>>>>>>>>>>> VMProps.java is called for each test run of jtreg (be it >>>>>>>>>>>>>> a run containing a singe test or multiple tests). >>>>>>>>>>>>>> VMProps evaluates the @requires properties. Therefore, in >>>>>>>>>>>>>> case of this bug, any time a user runs any jtreg VM test >>>>>>>>>>>>>> on a machine w/o an operational docker engine, this error >>>>>>>>>>>>>> will pop up. >>>>>>>>>>>>>> The fix makes printing of this error message conditional, >>>>>>>>>>>>>> off by default. The message can be turend on >>>>>>>>>>>>>> via a property vmprops.trace.verbose when detailed >>>>>>>>>>>>>> diagnostic info is desired. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ======== Testing: >>>>>>>>>>>>>> Local testing: >>>>>>>>>>>>>> W/o docker present: >>>>>>>>>>>>>> jtreg >>>>>>>>>>>>>> No error message >>>>>>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>>> >>>>>>>>>>>>>> Detailed error message >>>>>>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>>>>>> Test skipped >>>>>>>>>>>>>> >>>>>>>>>>>>>> With docker installed but no permissions: >>>>>>>>>>>>>> jtreg >>>>>>>>>>>>>> No error message >>>>>>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>>> >>>>>>>>>>>>>> Detailed error message >>>>>>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>>>>>> Test skipped >>>>>>>>>>>>>> >>>>>>>>>>>>>> With docker installed and operational: >>>>>>>>>>>>>> jtreg >>>>>>>>>>>>>> No error message >>>>>>>>>>>>>> jtreg -Dvmprops.trace.verbose=true >>>>>>>>>>>>>> >>>>>>>>>>>>>> No error message >>>>>>>>>>>>>> jtreg DockerBasicTest.java >>>>>>>>>>>>>> Test passed >>>>>>>>>>>>>> >>>>>>>>>>>>>> Testing via automated distributed test system: >>>>>>>>>>>>>> - tier1 >>>>>>>>>>>>>> - docker tests >>>>>>>>>>>>>> In progress >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> Misha >>>>>>>>>>>>>> >>>> >> From david.holmes at oracle.com Thu Nov 9 05:38:21 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 15:38:21 +1000 Subject: (trivial) RFR: 8190881: [TESTBUG] test.runtime.ErrorHandling.TestOnError comment is incomplete Message-ID: Bug: https://bugs.openjdk.java.net/browse/JDK-8190881 Can a Reviewer please spare me 5 seconds to okay this totally trivial deletion of a partial comment. Thanks, David diff -r 2cd7d700217f test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java --- a/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java +++ b/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, 2016, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2015, 2017, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -44,7 +44,6 @@ String msg = "Test Succeeded"; - // Execute the VM so that a ProcessBuilder pb = ProcessTools.createJavaProcessBuilder( "-XX:-TransmitErrorReport", "-XX:-CreateCoredumpOnCrash", From igor.ignatyev at oracle.com Thu Nov 9 05:41:28 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Wed, 8 Nov 2017 21:41:28 -0800 Subject: (trivial) RFR: 8190881: [TESTBUG] test.runtime.ErrorHandling.TestOnError comment is incomplete In-Reply-To: References: Message-ID: <54F4B860-C6C0-4C72-A8EE-BB66F8212B77@oracle.com> looks good to me. -- Igor > On Nov 8, 2017, at 9:38 PM, David Holmes wrote: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190881 > > Can a Reviewer please spare me 5 seconds to okay this totally trivial deletion of a partial comment. > > Thanks, > David > > diff -r 2cd7d700217f test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java > --- a/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java > +++ b/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2015, 2016, Oracle and/or its affiliates. All rights reserved. > + * Copyright (c) 2015, 2017, Oracle and/or its affiliates. All rights reserved. > * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > * > * This code is free software; you can redistribute it and/or modify it > @@ -44,7 +44,6 @@ > > String msg = "Test Succeeded"; > > - // Execute the VM so that a > ProcessBuilder pb = ProcessTools.createJavaProcessBuilder( > "-XX:-TransmitErrorReport", > "-XX:-CreateCoredumpOnCrash", > From david.holmes at oracle.com Thu Nov 9 05:51:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 15:51:06 +1000 Subject: (trivial) RFR: 8190881: [TESTBUG] test.runtime.ErrorHandling.TestOnError comment is incomplete In-Reply-To: <54F4B860-C6C0-4C72-A8EE-BB66F8212B77@oracle.com> References: <54F4B860-C6C0-4C72-A8EE-BB66F8212B77@oracle.com> Message-ID: Thanks Igor! David On 9/11/2017 3:41 PM, Igor Ignatyev wrote: > looks good to me. > > -- Igor >> On Nov 8, 2017, at 9:38 PM, David Holmes wrote: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8190881 >> >> Can a Reviewer please spare me 5 seconds to okay this totally trivial deletion of a partial comment. >> >> Thanks, >> David >> >> diff -r 2cd7d700217f test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java >> --- a/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java >> +++ b/test/hotspot/jtreg/runtime/ErrorHandling/TestOnError.java >> @@ -1,5 +1,5 @@ >> /* >> - * Copyright (c) 2015, 2016, Oracle and/or its affiliates. All rights reserved. >> + * Copyright (c) 2015, 2017, Oracle and/or its affiliates. All rights reserved. >> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. >> * >> * This code is free software; you can redistribute it and/or modify it >> @@ -44,7 +44,6 @@ >> >> String msg = "Test Succeeded"; >> >> - // Execute the VM so that a >> ProcessBuilder pb = ProcessTools.createJavaProcessBuilder( >> "-XX:-TransmitErrorReport", >> "-XX:-CreateCoredumpOnCrash", >> > From thomas.stuefe at gmail.com Thu Nov 9 06:24:17 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 9 Nov 2017 07:24:17 +0100 Subject: alternate pns() debug function In-Reply-To: References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> Message-ID: Hi Chris, I went accidentally off-list with my last response yesterday. I continue on-list. On Thu, Nov 9, 2017 at 2:27 AM, Chris Plummer wrote: > Hi Thomas, > > > >> Interesting. What did you see what the debugger would not show you? >> >> (gdb) call pns2() >> >> "Executing pns2" >> Native frames: (J=compiled Java code, A=aot compiled Java code, >> j=interpreted, Vv=VM code, C=native code) >> C 0x00007ffff5d5e7df >> C 0xcccccccccccccccc >> C 0xcccccccccccccccc >> >> (gdb) bt >> #0 0x00007fffd79eef5c in ?? () >> #1 0x00007ffff73d79a5 in SafeFetch32 (errValue=2748, >> adr=0xabc0000000000abc) at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/runtime/stubRoutines.hpp:462 >> #2 test_safefetch32 () at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/runtime/stubRoutines.cpp:244 >> #3 StubRoutines::initialize2 () at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/runtime/stubRoutines.cpp:365 >> #4 0x00007ffff73d80ba in stubRoutines_init2 () at >> /local/ws/jdk10/jdk10-hs.clean/open/src/hotspot/share/ >> runtime/stubRoutines.cpp:374 >> #5 0x00007ffff6bc2fa3 in init_globals () at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/runtime/init.cpp:150 >> #6 0x00007ffff7474453 in Threads::create_vm (args=args at entry=0x7ffff5d5ee60, >> canTryAgain=canTryAgain at entry=0x7ffff5d5ed8f) at >> /local/ws/jdk10/jdk10-hs.clean/open/src/hotspot/share/ >> runtime/thread.cpp:3616 >> #7 0x00007ffff6d19d0b in JNI_CreateJavaVM_inner (args=0x7ffff5d5ee60, >> penv=0x7ffff5d5ee58, vm=0x7ffff5d5ee50) at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/prims/jni.cpp:3938 >> #8 JNI_CreateJavaVM (vm=0x7ffff5d5ee50, penv=0x7ffff5d5ee58, >> args=0x7ffff5d5ee60) at /local/ws/jdk10/jdk10-hs. >> clean/open/src/hotspot/share/prims/jni.cpp:4033 >> #9 0x00007ffff7dd89d1 in InitializeJVM () at /local/ws/jdk10/jdk10-hs. >> clean/open/src/java.base/share/native/libjli/java.c:1478 >> #10 JavaMain () at /local/ws/jdk10/jdk10-hs.clean/open/src/java.base/ >> share/native/libjli/java.c:411 >> #11 0x00000034f8a07a51 in start_thread () from /lib64/libpthread.so.0 >> #12 0x00000034f86e893d in clone () from /lib64/libc.so.6 >> >> Actually would have been more useful to compare pns2() with pns() instead > of a gdb backtrace: > > (gdb) call pns($sp, $rbp, $pc) > > "Executing pns" > Native frames: (J=compiled Java code, A=aot compiled Java code, > j=interpreted, Vv=VM code, C=native code) > V [libjvm.so+0x1677a07] StubRoutines::initialize2()+0x737 > V [libjvm.so+0xe62fa3] init_globals()+0xf3 > V [libjvm.so+0x1714453] Threads::create_vm(JavaVMInitArgs*, bool*)+0x2e3 > V [libjvm.so+0xfb9d0b] JNI_CreateJavaVM+0x9b > C [libjli.so+0x39d1] JavaMain+0x81 > C [libpthread.so.0+0x7a51] > > I think gdb is making use of inlining info from the .debuginfo file since > it has a lot of extra frames. > > I still do not understand - here, the debugger shows me more than both pns() and pns2(). Especially, in this case, the real crash point, including file/line number. Is that not what you want? So, why do you need to invoke pns/pns2() ? Note that this is just curiosity, I still think pns/pns2() can be valuable for dumping stacks within the hotspot. > >>> On Windows x64, we do not have frame pointers, so we cannot walk the >>> frame VM-Style, so os::platform_print_native_stack() is the only way to >>> print a stack. >>> >>> Yes, and I got this to work Windows/x64, but unfortunately no symbols >>> are printed. >>> >> >> When you let the Vm crash with e.g. -XX:ErrorHandlerTest=14, does the >> resulting hs-err file show a correct callstack with symbols? If not, what >> does the dbghelp.dll loader say (the one line below the loaded libraries >> section)? >> >> I'll need to give this a try, but I suspect it will not include symbols. >> I recall that you need to run windows stack traces through some special >> tool that converts addresses to symbols. >> > > I do not think this is true, at least not to my knowledge. Dbghelp.dll > must be installed on the system - it usually is - and you need the pdb > files laying beside the binaries. The latter depends on how you build. > -with-native-debug-symbols=external will place the pdb files beside the > binaries, but =zipped (or so, I am away from my machine atm) will zip them. > In zipped form they are unusable and they need to be unzipped before they > can be used. At least jvm.pdb, which must be located beside jvm.dll. > > This might be the issue. I'm using our build+test infrastructure to do > this testing, so I'm not in control of how it's built or deployed. I've > downloaded the binary bundle used for the testing so I could have a look, > and I see that it does not include the symbols. However, there is a > separate bundle with just the symbols. I'm unsure if it's being installed > for the testing. I'll ask around. > > > I am curious about this because I recently changed the coding responsible > for the symbol resolution on windows. I?d like to understand why you do not > see symbols, because in our tests everything works. > > The is from the hs_err file when running TestOnError.java, which uses > -XX:ErrorHandlerTest=12: > > Stack: [0x00000038a0f40000,0x00000038a1040000], sp=0x00000038a103f740, > free space=1021k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [jvm.dll+0xc3255a] > V [jvm.dll+0x7a6198] > V [jvm.dll+0x7a8e8f] > C [java.exe+0x3609] > C [java.exe+0xe83f] > C [java.exe+0xe9e6] > C [KERNEL32.DLL+0x13d2] > C [ntdll.dll+0x154f4] > > thanks, > > Chris > Yes, this may be a case of missing infos. I had a discussion with Coleen some weeks ago offlist in the context of testing the changes for https://bugs.openjdk.java.net/browse/JDK-8185712. I had to jtreg tests disable tests which ran nicely here at SAP but would not yield callstacks at Oracle. At that time I was wondering if there was something wrong with my code but now I am quite certain you guys either remove the debug info after building or you build with zipped debug info (which is the default on all platforms but AIX). Or both. I'll ask on build.dev. Thanks, Thomas From thomas.stuefe at gmail.com Thu Nov 9 06:40:05 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 9 Nov 2017 07:40:05 +0100 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <5A033E72.90800@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> <5A033E72.90800@oracle.com> Message-ID: Hi Calvin, On Wed, Nov 8, 2017 at 6:27 PM, Calvin Cheung wrote: > > > On 11/7/17, 6:12 AM, Thomas St?fe wrote: > > Hi Calvin, > > On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung > wrote: > >> >> >> On 10/27/17, 12:55 AM, Thomas St?fe wrote: >> >> Hi Calvin, >> >> On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >> wrote: >> >>> Hi Thomas, >>> >>> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >>> >>>> Hi Calvin, >>>> >>>> this is a worthwhile addition, thank you for your work! >>>> >>> Thanks for your review. >> >> >> Thanks for your work :) >> >> First off, there is another issue in file_attribute_data_to_stat(). We >> actually had this issue before, but your work uncovered it: >> >> According to POSIX (http://pubs.opengroup.org/onl >> inepubs/009695399/basedefs/sys/stat.h.html) and every unix manpage I >> looked at (solaris, linux, aix..), st_ctime is not the file creation time >> but the last time file status was changed. Only Microsoft gets this wrong >> in their posix layer, its stat() function returns ( >> https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx) creation time in >> st_ctime. >> >> But as os::stat() is a platform independent layer, I'd say we should >> behave the same on all platforms, and that would be the Posix way. >> >> I did not find any code calling os::stat() and using st_ctime, so this is >> still save to change for windows. (Note that I found that >> perfMemory_xxx.cpp uses plain OS ::stat and misuses ctime as "creation >> time" - I opened a bug for that (https://bugs.openjdk.java.net >> /browse/JDK-8190260 - but that does not affect us, as they do not call >> os::stat()). >> >> There is one more complication: in os::stat() we use plain ::stat() in >> the single byte case.: >> >> *+ if (strlen(path) < MAX_PATH) {* >> >> *+ ret = ::stat(pathbuf, sbuf);**+ } else {* >> >> >> But ::stat() on Windows is broken, as I wrote above. We should not use >> it, especially not if we change the meaing of st_ctime in the double byte >> path. >> >> My suggestion would be to just always call GetFileAttributes(), also for >> the single byte path. A very simple solution would be to just always go the >> double byte path with UNC translation, regardless of the path length. Or, >> if you are worried about performance, for paths shorter than MAX_PATH we >> use GetFileAttributesA(), for longer paths GetFileAttributesW with UNC >> translation. In both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you >> can feed tp your file_attribute_data_to_stat() routine and have the struct >> stat you return be always consistent. >> >> I ran into an issue with the dwFileAttributes value for a jar file. On >> Windows Server O/S, the value is set to 0x2020 which is >> (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) but on a >> desktop O/S like Windows 7, it is set to 0x0020 which is just >> FILE_ATTRIBUTE_ARCHIVE. I've fixed it in file_attribute_data_to_stat(). >> >> > Oh.. :( good catch! But that opens a new can of worms I did not see before: > > FILE_ATTRIBUTE_NORMAL is documented as "A file that does not have other > attributes set. This attribute is valid only when used alone." In addition > to this flag, we have a multitude of things like FILE_ATTRIBUTE_ARCHIVE, > FILE_ATTRIBUTE_ENCRYPTED, FILE_ATTRIBUTE_READONLY ... etc, all cases where > we assume this is a normal file in Unix terminology and where we would > expect os::stat to return S_IFREG, but where according to the msdn doc > FILE_ATTRIBUTE_NORMAL is not set. > > Looking at what others do in this scenario (Looked at mingw sources and at > ReactOS - I am not posting any source code here for legal reasons but feel > free to look for yourself), the usual way to translate > WIN32_FILE_ATTRIBUTES to struct stat seems to be: > "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, no > dependency on FILE_ATTRIBUTE_NORMAL). > > This makes sense but I ran into similar problem as before - the > dwFileAttributes has a different value on a windows server O/S vs desktop > O/S. So I need to do the check as follows: > > + // A directory has the dwFileAttributes value of 0x2010 which is+ // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE)+ // on Windows Server O/S but the value is 0x0010 on Windows desktop O/S+ // such as Windows 7.+ if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0) {+ sbuf->st_mode |= S_IFDIR;+ } else {+ sbuf->st_mode |= S_IFREG;+ } > > I scratched my head a bit about the comment till I understood that you mean "if FILE_ATTRIBUTE_DIRECTORY bit is set it is a directory regardless of which other flags are set" instead of "if flags==FILE_ATTRIBUTE_DIRECTORY it is a directory". Sure, I guess my comment above was sloppy, but this was what I meant. I am not even sure the comment is needed, this is quite self-explaining. > updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ > > I am fine with all the changes now. Thank you for your work and patience. Kind Regards, Thomas > Diff from webrev.03: > > < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 > 08:50:40.170786397 -0800 > < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 > 08:50:39.803751877 -0800 > < @@ -4060,41 +4060,119 @@ > --- > > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 > 09:40:13.657460425 -0700 > > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 > 09:40:13.261423192 -0700 > > @@ -4060,41 +4060,121 @@ > 25,29c25 > < + // A directory has the dwFileAttributes value of 0x2010 which is > < + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) > < + // on Windows Server O/S but the value is 0x0010 on Windows desktop > O/S > < + // such as Windows 7. > < + if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0) { > --- > > + if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { > 31c27,33 > < + } else { > --- > > + } > > + if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || > > + // an archive file such as a jar file has the dwFileAttributes > value of > > + // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | > FILE_ATTRIBUTE_ARCHIVE) > > + // on Windows Server O/S but the value is 0x0020 on > > + // Windows desktop O/S such as Windows 7. > > + ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) != 0)) { > > > I wonder about other special cases which should work too: > > - read-only files should be S_IFREG and !S_IWRITE, > > For a read-only system file under the user's home dir. > > st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD > dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | FILE_ATTRIBUTE_HIDDEN | > FILE_ATTRIBUTE_SYSTEM | FILE_ATTRIBUTE_READONLY) > > read-only directories should be S_IFDIR and !S_IWRITE. > > I've tried the C:\progra~1 dir. > > st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD > dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | > FILE_ATTRIBUTE_READONLY) > > - We should tread the device root ("C:\") as a directory (so, > os::stat("c:\") should return S_IFDIR). Does this work? > > This one works too. > > st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD > dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | FILE_ATTRIBUTE_HIDDEN > | FILE_ATTRIBUTE_SYSTEM) > > > >> I've also used GetTickCounts() to measure the performance of ::stat() vs >> GetFileAttributesA() plus file_attribute_data_to_stat(). There's no >> difference at the ms resolution. Using the high-resolution timestamp ( >> https://msdn.microsoft.com/en-us/library/windows/desktop/dn >> 553408(v=vs.85).aspx), it doesn't show much difference. >> >> > stat() seems to be implemented using FindFirstFile() (see crt sources if > you can), and we call GetFileAttributesA(), so I do not think this differs > much. > > Yes, I saw the same in my Visual Studio installation. > > thanks, > Calvin > > > >> >> But onward: >> >> >>> >>>> ========================= >>>> >>>> src/hotspot/os/windows/os_windows.cpp >>>> >>>> Could you please use wchar_t instead of WCHAR? >>>> >>> Fixed. >>> >>>> >>>> --------------- >>>> in os::stat(): >>>> >>>> This cumbersome malloc/strcpy sequence: >>>> >>>> ! size_t path_len = strlen(path) + 1; >>>> + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), >>>> mtInternal); >>>> + if (pathbuf == NULL) { >>>> + errno = ENOMEM; >>>> return -1; >>>> } >>>> os::native_path(strcpy(pathbuf, path)); >>>> >>>> can be reduced to a simple strdup: >>>> >>>> char* pathbuf = os::strdup(path, mtInternal); >>>> if (pathbuf == NULL) { >>>> errno = ENOMEM; >>>> return -1; >>>> } >>>> os::native_path(pathbuf); >>>> >>>> This also would separate strcpy() from os::native_path() which is a bit >>>> unreadable. >>>> >>> I've made the change. >>> >>>> >>>> -------------------- >>>> >>>> The single-byte path (strdup, ::stat()), together with its free(), can >>>> be moved inside the (strlen(path) < MAX_PATH) condition. os::native_path >>>> will not change the path length (it better not, as it operates on the input >>>> buffer). That avoids having two allocations on the doublebyte path. >>>> >>> >>> os::native_path() converts a path like C:\\\\somedir to C:\\somedir >>> So I'll need the converted path in the wchar_t case too. The freeing of >>> the pathbuf is being done at the end of the function or in the middle of >>> the wchar_t case if there's an error. >> >> >> Oh, you are right,of course. I missed that it this is needed for both >> paths. >> >> >>> >>> >>>> ----------------------- >>>> >>>> But seeing that translation to UNC paths is something we might want >>>> more often, I would combine allocation and UNC prefix adding to one >>>> function like this, to avoid the memove and increase reusability: >>>> >>>> // Creates an unc path from a single byte path. Return buffer is >>>> allocated in C heap and needs to be freed by caller. Returns NULL on error. >>>> static whchar_t* create_unc_path(const char* s) { >>>> - does s start with \\?\ ? >>>> - yes: >>>> - os::malloc(strlen(s) + 1) and mbstowcs_s >>>> - no: >>>> - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to fourth >>>> position in string, prefix with L"\\?\" >>>> } >>>> >>> >>> I also include the case for adding L"\\\\?\\UNC\0" at the beginning to >>> be consistent with libjava/canonicalize_md.c. >> >> >>>> We also need error checking to mbstowcs_s. >>>> >>> I've added assert like the following after the call: >>> >>> assert(converted_chars == path_len, "sanity"); >> >> >> >> create_unc_path() : >> >> - could you convert the /**/ to // comments, please? >> >> Fixed. >> > > thanks. > > >> >> >> - not sure about the assert after mbstowcs_s: if we happen to encounter >> an invalid multibyte character, function will fail and return an error: >> >> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >> "If mbstowcs_s encounters an invalid multibyte character, it puts 0 in >> *``pReturnValue, sets the destination buffer to an empty string, sets errno >> to EILSEQ, and returns EILSEQ." >> >> I've changed create_unc_path() so that the caller will get the errno and >> removed the assert. >> >> static wchar_t* create_unc_path(const char* path, errno_t &err) >> > > Okay, works for me. > > >> >> >> As this is dependent on user data, we should not assert, but handle the >> return code of mbstowcs_s gracefully. Same goes for >> libjava/canonicalize_md.c. >> >> - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + 7, path, >> path_len); >> third parameter is wrong, as we hand in an offset into the buffer, we >> must decrement the buffer size by this offset, so correct would be path_len >> +7 - 7 or just path_len. >> >> - Same error below: + ::mbstowcs_s(&converted_chars, &wpath[4], path_len >> + 4, path, path_len); >> >> Fixed in both places. >> > > Okay. > > >> >> >> >> >>> >>> >>>> Just for cleanliness, I would then wrap deallocation into an own >>>> function. >>>> >>>> static viud destroy_unc_path(whchar_t* s) { os::free(s); } >>>> >>> I've added the destroy function. >>> >>>> >>>> These two functions could be candidates of putting into os::win32 >>>> namespace, as this form of ANSI->UNC path translation is quite common - >>>> whoever wants to use the xxxxW() functions must do this. >>>> >>> I'm leaving them in os_windows.cpp since they're being used only within >>> that file. >> >> >> Fine by me. >> >> >>> >>> >>>> ----------------------- >>>> >>>> FindFirstFileW: >>>> >>>> I am pretty sure that you can achieve the same result with >>>> GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. >>>> >>> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very similar to >>> WIN32_FIND_DATAW. >>> >>>> >>>> It is way more straightforward to use than FindFirstFileW, as it does >>>> not require you to write a callback hook. >>>> >>> I've switched to using GetFileAttributesExW(). >> >> >> Thank you, this is way more readable. >> >> Another issue: do we still need the fix for 6539723 if we switch from >> stat() to GetFileAttributesExW()? This fix looks important, but the >> comment >> indicates that it could break things if the original bug is not present. >> >> Btw, this is another strong argument for scrapping ::stat() altogether on >> all code paths, not only for long input paths, because stat() and >> GetFileAttributesExW() may behave >> differently. So it would be good to use the same API on all code paths, >> in order to get the best test coverage. >> >> For this round of change, I'm using GetFileAttributesEx[A|W]() for both >> code paths. >> >> webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ >> >> thanks, >> Calvin >> > > Okay, all good apart from the issues mentioned above. Thanks for your work! > > Best Regards, Thomas > > > >> >> >>> >>>> ------- >>>> >>>> eval_find_data(): This is more of a generic helper function, could you >>>> rename this to something clearer, e.g. make_double_word() ? >>>> >>> Ok. I've renamed it. >>> >>>> >>>> Also, setup_stat(), could this be renamed more clearly to something >>>> like WIN32_FIND_DATA_to_stat? or lowercase if this bothers you :) >>>> >>> I'm naming the function as file_attribute_data_to_stat() to match with >>> the data structure name. >> >> >> Thanks for taking my suggestions. >> >> >>> >>> >>>> ================================== >>>> src/hotspot/share/classfile/classLoader.cpp >>>> >>>> In ClassPathDirEntry::open_stream(), I would feel better if we were >>>> asserting _dir and name to be != NULL before feeding it to strlen. >>>> >>> I've added an assert statement. >>> >>>> >>>> =================== >>>> >>> Here's an updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >>> >>> >> Thanks! >> >> Thomas >> >> >>> thanks, >>> Calvin >>> >>> >>>> Thanks! >>>> >>>> Thomas >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung < >>>> calvin.cheung at oracle.com > wrote: >>>> >>>> I've reworked this fix by using the Unicode version of open >>>> (wopen) to handle path name longer than max path with the path >>>> prefixed to indicate an extended length path as described in [1]. >>>> >>>> The Unicode version of stat (wstat) doesn't work well with long >>>> path [2]. So FindFirstFileW will be used.The data in >>>> WIN32_FIND_DATA returned from FindFirstFileW needs to be >>>> transferred to the stat structure since the caller expects a >>>> return stat structure and other platforms return a stat structure. >>>> >>>> In classLoader.cpp, calculate the size of buffer required instead >>>> of limiting it to JVM_MAXPATHLEN. >>>> In os_windows.cpp, dynamically allocate buffers in os::open and >>>> os::stat. >>>> >>>> updated webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>>> >>>> >>>> It passed hs-tier2 testing using mach5. >>>> >>>> thanks, >>>> Calvin >>>> >>>> [1] >>>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa3 >>>> 65247(v=vs.85).aspx#MAX_PATH >>>> >>> 365247%28v=vs.85%29.aspx#MAX_PATH> >>>> >>>> [2] >>>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093 >>>> ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>>> >>> 3ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral> >>>> >>>> >>>> >>>> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >>>> >>>> >>>> Adding a warning message if the full path or the directory >>>> length exceeds MAX_PATH on windows. >>>> >>>> Example warning messages. >>>> >>>> 1) The full path exceeds MAX_PATH: >>>> >>>> Java HotSpot(TM) 64-Bit Server VM warning: construct full path >>>> name failed: total length 270 exceeds max length of 260 >>>> dir >>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >>>> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> length 259 >>>> name Hello.class length 11 >>>> >>>> 2) The directory path exceeds MAX_PATH: >>>> >>>> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: >>>> path length 265 exceeds max length 260 >>>> path >>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >>>> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>>> >>>> webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>>> >>>> >>>> Testing: >>>> JPRT (including the new test) >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> >> > From thomas.stuefe at gmail.com Thu Nov 9 06:55:16 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 9 Nov 2017 07:55:16 +0100 Subject: alternate pns() debug function In-Reply-To: <3080e5e2-aa7c-4a2a-2de7-5bfbd8294625@oracle.com> References: <4c68b6bf-77d0-e3c0-f1c6-d0ce28038b26@oracle.com> <2c52c8df-ba7f-1fbd-fed1-147b40c7d2fc@oracle.com> <3080e5e2-aa7c-4a2a-2de7-5bfbd8294625@oracle.com> Message-ID: On Thu, Nov 9, 2017 at 7:52 AM, Chris Plummer wrote: > On 11/8/17 10:24 PM, Thomas St?fe wrote: > > Hi Chris, > > I went accidentally off-list with my last response yesterday. I continue > on-list. > > On Thu, Nov 9, 2017 at 2:27 AM, Chris Plummer > wrote: > >> Hi Thomas, >> >> > >> >>> Interesting. What did you see what the debugger would not show you? >>> >>> (gdb) call pns2() >>> >>> "Executing pns2" >>> Native frames: (J=compiled Java code, A=aot compiled Java code, >>> j=interpreted, Vv=VM code, C=native code) >>> C 0x00007ffff5d5e7df >>> C 0xcccccccccccccccc >>> C 0xcccccccccccccccc >>> >>> (gdb) bt >>> #0 0x00007fffd79eef5c in ?? () >>> #1 0x00007ffff73d79a5 in SafeFetch32 (errValue=2748, >>> adr=0xabc0000000000abc) at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/hotspot/share/runtime/stubRoutines.hpp:462 >>> #2 test_safefetch32 () at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/hotspot/share/runtime/stubRoutines.cpp:244 >>> #3 StubRoutines::initialize2 () at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/hotspot/share/runtime/stubRoutines.cpp:365 >>> #4 0x00007ffff73d80ba in stubRoutines_init2 () at >>> /local/ws/jdk10/jdk10-hs.clean/open/src/hotspot/share/runtim >>> e/stubRoutines.cpp:374 >>> #5 0x00007ffff6bc2fa3 in init_globals () at >>> /local/ws/jdk10/jdk10-hs.clean/open/src/hotspot/share/runtim >>> e/init.cpp:150 >>> #6 0x00007ffff7474453 in Threads::create_vm (args=args at entry >>> =0x7ffff5d5ee60, canTryAgain=canTryAgain at entry=0x7ffff5d5ed8f) at >>> /local/ws/jdk10/jdk10-hs.clean/open/src/hotspot/share/runtim >>> e/thread.cpp:3616 >>> #7 0x00007ffff6d19d0b in JNI_CreateJavaVM_inner (args=0x7ffff5d5ee60, >>> penv=0x7ffff5d5ee58, vm=0x7ffff5d5ee50) at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/hotspot/share/prims/jni.cpp:3938 >>> #8 JNI_CreateJavaVM (vm=0x7ffff5d5ee50, penv=0x7ffff5d5ee58, >>> args=0x7ffff5d5ee60) at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/hotspot/share/prims/jni.cpp:4033 >>> #9 0x00007ffff7dd89d1 in InitializeJVM () at >>> /local/ws/jdk10/jdk10-hs.clean/open/src/java.base/share/ >>> native/libjli/java.c:1478 >>> #10 JavaMain () at /local/ws/jdk10/jdk10-hs.clean >>> /open/src/java.base/share/native/libjli/java.c:411 >>> #11 0x00000034f8a07a51 in start_thread () from /lib64/libpthread.so.0 >>> #12 0x00000034f86e893d in clone () from /lib64/libc.so.6 >>> >>> Actually would have been more useful to compare pns2() with pns() >> instead of a gdb backtrace: >> >> (gdb) call pns($sp, $rbp, $pc) >> >> "Executing pns" >> Native frames: (J=compiled Java code, A=aot compiled Java code, >> j=interpreted, Vv=VM code, C=native code) >> V [libjvm.so+0x1677a07] StubRoutines::initialize2()+0x737 >> V [libjvm.so+0xe62fa3] init_globals()+0xf3 >> V [libjvm.so+0x1714453] Threads::create_vm(JavaVMInitArgs*, >> bool*)+0x2e3 >> V [libjvm.so+0xfb9d0b] JNI_CreateJavaVM+0x9b >> C [libjli.so+0x39d1] JavaMain+0x81 >> C [libpthread.so.0+0x7a51] >> >> I think gdb is making use of inlining info from the .debuginfo file since >> it has a lot of extra frames. >> >> > I still do not understand - here, the debugger shows me more than both > pns() and pns2(). Especially, in this case, the real crash point, including > file/line number. Is that not what you want? So, why do you need to invoke > pns/pns2() ? Note that this is just curiosity, I still think pns/pns2() can > be valuable for dumping stacks within the hotspot. > > Well pns2() is not even an option within the debugger. I believe the > advantage of pns() is that it includes java frames. > > > > >> >>>> On Windows x64, we do not have frame pointers, so we cannot walk the >>>> frame VM-Style, so os::platform_print_native_stack() is the only way >>>> to print a stack. >>>> >>>> Yes, and I got this to work Windows/x64, but unfortunately no symbols >>>> are printed. >>>> >>> >>> When you let the Vm crash with e.g. -XX:ErrorHandlerTest=14, does the >>> resulting hs-err file show a correct callstack with symbols? If not, what >>> does the dbghelp.dll loader say (the one line below the loaded libraries >>> section)? >>> >>> I'll need to give this a try, but I suspect it will not include symbols. >>> I recall that you need to run windows stack traces through some special >>> tool that converts addresses to symbols. >>> >> >> I do not think this is true, at least not to my knowledge. Dbghelp.dll >> must be installed on the system - it usually is - and you need the pdb >> files laying beside the binaries. The latter depends on how you build. >> -with-native-debug-symbols=external will place the pdb files beside the >> binaries, but =zipped (or so, I am away from my machine atm) will zip them. >> In zipped form they are unusable and they need to be unzipped before they >> can be used. At least jvm.pdb, which must be located beside jvm.dll. >> >> This might be the issue. I'm using our build+test infrastructure to do >> this testing, so I'm not in control of how it's built or deployed. I've >> downloaded the binary bundle used for the testing so I could have a look, >> and I see that it does not include the symbols. However, there is a >> separate bundle with just the symbols. I'm unsure if it's being installed >> for the testing. I'll ask around. >> > >> I am curious about this because I recently changed the coding responsible >> for the symbol resolution on windows. I?d like to understand why you do not >> see symbols, because in our tests everything works. >> >> The is from the hs_err file when running TestOnError.java, which uses >> -XX:ErrorHandlerTest=12: >> >> Stack: [0x00000038a0f40000,0x00000038a1040000], sp=0x00000038a103f740, >> free space=1021k >> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native >> code) >> V [jvm.dll+0xc3255a] >> V [jvm.dll+0x7a6198] >> V [jvm.dll+0x7a8e8f] >> C [java.exe+0x3609] >> C [java.exe+0xe83f] >> C [java.exe+0xe9e6] >> C [KERNEL32.DLL+0x13d2] >> C [ntdll.dll+0x154f4] >> >> thanks, >> >> Chris >> > > Yes, this may be a case of missing infos. I had a discussion with Coleen > some weeks ago offlist in the context of testing the changes for > https://bugs.openjdk.java.net/browse/JDK-8185712. I had to jtreg tests > disable tests which ran nicely here at SAP but would not yield callstacks > at Oracle. At that time I was wondering if there was something wrong with > my code but now I am quite certain you guys either remove the debug info > after building or you build with zipped debug info (which is the default on > all platforms but AIX). Or both. I'll ask on build.dev. > > I've confirmed that it is a known issue that debug symbols are not > included in our testing (or at least that is the case on Windows). pns2() > is showing symbols on linux, but that might just be the result of having > pretty much all the symbols exported, which allows lookup with dladdr(). > > thanks, > > Chris > Thanks for confirming that! ..Thomas > > Thanks, Thomas > > > From calvin.cheung at oracle.com Thu Nov 9 17:23:47 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 09 Nov 2017 09:23:47 -0800 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> <5A033E72.90800@oracle.com> Message-ID: <5A048F23.9030806@oracle.com> Hi Thomas, On 11/8/17, 10:40 PM, Thomas St?fe wrote: > Hi Calvin, > > On Wed, Nov 8, 2017 at 6:27 PM, Calvin Cheung > > wrote: > > > > On 11/7/17, 6:12 AM, Thomas St?fe wrote: >> Hi Calvin, >> >> On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung >> > wrote: >> >> >> >> On 10/27/17, 12:55 AM, Thomas St?fe wrote: >>> Hi Calvin, >>> >>> On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >>> > >>> wrote: >>> >>> Hi Thomas, >>> >>> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >>> >>> Hi Calvin, >>> >>> this is a worthwhile addition, thank you for your work! >>> >>> Thanks for your review. >>> >>> >>> Thanks for your work :) >>> >>> First off, there is another issue in >>> file_attribute_data_to_stat(). We actually had this issue >>> before, but your work uncovered it: >>> >>> According to POSIX >>> (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html >>> ) >>> and every unix manpage I looked at (solaris, linux, aix..), >>> st_ctime is not the file creation time but the last time >>> file status was changed. Only Microsoft gets this wrong in >>> their posix layer, its stat() function returns >>> (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx >>> ) >>> creation time in st_ctime. >>> >>> But as os::stat() is a platform independent layer, I'd say >>> we should behave the same on all platforms, and that would >>> be the Posix way. >>> >>> I did not find any code calling os::stat() and using >>> st_ctime, so this is still save to change for windows. (Note >>> that I found that perfMemory_xxx.cpp uses plain OS ::stat >>> and misuses ctime as "creation time" - I opened a bug for >>> that (https://bugs.openjdk.java.net/browse/JDK-8190260 >>> - but >>> that does not affect us, as they do not call os::stat()). >>> >>> There is one more complication: in os::stat() we use plain >>> ::stat() in the single byte case.: >>> >>> *+ if (strlen(path) < MAX_PATH) {* >>> *+ ret = ::stat(pathbuf, sbuf);* >>> *+ } else {* >>> * >>> * >>> But ::stat() on Windows is broken, as I wrote above. We >>> should not use it, especially not if we change the meaing of >>> st_ctime in the double byte path. >>> >>> My suggestion would be to just always call >>> GetFileAttributes(), also for the single byte path. A very >>> simple solution would be to just always go the double byte >>> path with UNC translation, regardless of the path >>> length. Or, if you are worried about performance, for paths >>> shorter than MAX_PATH we use GetFileAttributesA(), for >>> longer paths GetFileAttributesW with UNC translation. In >>> both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you can >>> feed tp your file_attribute_data_to_stat() routine and have >>> the struct stat you return be always consistent. >> I ran into an issue with the dwFileAttributes value for a jar >> file. On Windows Server O/S, the value is set to 0x2020 which >> is (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >> FILE_ATTRIBUTE_ARCHIVE) but on a desktop O/S like Windows 7, >> it is set to 0x0020 which is just FILE_ATTRIBUTE_ARCHIVE. >> I've fixed it in file_attribute_data_to_stat(). >> >> >> Oh.. :( good catch! But that opens a new can of worms I did not >> see before: >> >> FILE_ATTRIBUTE_NORMAL is documented as "A file that does not have >> other attributes set. This attribute is valid only when used >> alone." In addition to this flag, we have a multitude of things >> like FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, >> FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this >> is a normal file in Unix terminology and where we would expect >> os::stat to return S_IFREG, but where according to the msdn doc >> FILE_ATTRIBUTE_NORMAL is not set. >> >> Looking at what others do in this scenario (Looked at mingw >> sources and at ReactOS - I am not posting any source code here >> for legal reasons but feel free to look for yourself), the usual >> way to translate WIN32_FILE_ATTRIBUTES to struct stat seems to be: >> "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, >> no dependency on FILE_ATTRIBUTE_NORMAL). > This makes sense but I ran into similar problem as before - the > dwFileAttributes has a different value on a windows server O/S vs > desktop O/S. So I need to do the check as follows: > > + // A directory has the dwFileAttributes value of 0x2010 which is > + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) > + // on Windows Server O/S but the value is 0x0010 on Windows desktop O/S > + // such as Windows 7. > + if ((file_data.dwFileAttributes& FILE_ATTRIBUTE_DIRECTORY) != 0) { > + sbuf->st_mode |= S_IFDIR; > + } else { > + sbuf->st_mode |= S_IFREG; > + } > > I scratched my head a bit about the comment till I understood that you > mean "if FILE_ATTRIBUTE_DIRECTORY bit is set it is a directory > regardless of which other flags are set" instead of "if > flags==FILE_ATTRIBUTE_DIRECTORY it is a directory". Sure, I guess my > comment above was sloppy, but this was what I meant. I am not even > sure the comment is needed, this is quite self-explaining. I've noticed a typo in the above comment: + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) FILE_ATTRIBUTE_ARCHIVE should be FILE_ATTRIBUTE_DIRECTORY I'll correct it before push. > > updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ > > > > I am fine with all the changes now. Thank you for your work and patience. Thanks for your discussions and review. thanks, Calvin > > Kind Regards, Thomas > > > > > Diff from webrev.03: > > < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 > 08:50:40.170786397 -0800 > < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 > 08:50:39.803751877 -0800 > < @@ -4060,41 +4060,119 @@ > --- > > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 > 09:40:13.657460425 -0700 > > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 > 09:40:13.261423192 -0700 > > @@ -4060,41 +4060,121 @@ > 25,29c25 > < + // A directory has the dwFileAttributes value of 0x2010 which is > < + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) > < + // on Windows Server O/S but the value is 0x0010 on Windows > desktop O/S > < + // such as Windows 7. > < + if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) > != 0) { > --- > > + if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { > 31c27,33 > < + } else { > --- > > + } > > + if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || > > + // an archive file such as a jar file has the > dwFileAttributes value of > > + // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | > FILE_ATTRIBUTE_ARCHIVE) > > + // on Windows Server O/S but the value is 0x0020 on > > + // Windows desktop O/S such as Windows 7. > > + ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) != > 0)) { > >> >> I wonder about other special cases which should work too: >> - read-only files should be S_IFREG and !S_IWRITE, > For a read-only system file under the user's home dir. > > st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD > dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | > FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | > FILE_ATTRIBUTE_READONLY) >> read-only directories should be S_IFDIR and !S_IWRITE. > I've tried the C:\progra~1 dir. > > st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD > dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | > FILE_ATTRIBUTE_READONLY) >> - We should tread the device root ("C:\") as a directory (so, >> os::stat("c:\") should return S_IFDIR). Does this work? > This one works too. > > st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD > dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | > FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) >> >> I've also used GetTickCounts() to measure the performance of >> ::stat() vs GetFileAttributesA() plus >> file_attribute_data_to_stat(). There's no difference at the >> ms resolution. Using the high-resolution timestamp >> (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx) >> , >> it doesn't show much difference. >> >> >> stat() seems to be implemented using FindFirstFile() (see crt >> sources if you can), and we call GetFileAttributesA(), so I do >> not think this differs much. > Yes, I saw the same in my Visual Studio installation. > > thanks, > Calvin > >>> >>> But onward: >>> >>> >>> >>> ========================= >>> >>> src/hotspot/os/windows/os_windows.cpp >>> >>> Could you please use wchar_t instead of WCHAR? >>> >>> Fixed. >>> >>> >>> --------------- >>> in os::stat(): >>> >>> This cumbersome malloc/strcpy sequence: >>> >>> ! size_t path_len = strlen(path) + 1; >>> + char* pathbuf = (char*)os::malloc(path_len * >>> sizeof(char), mtInternal); >>> + if (pathbuf == NULL) { >>> + errno = ENOMEM; >>> return -1; >>> } >>> os::native_path(strcpy(pathbuf, path)); >>> >>> can be reduced to a simple strdup: >>> >>> char* pathbuf = os::strdup(path, mtInternal); >>> if (pathbuf == NULL) { >>> errno = ENOMEM; >>> return -1; >>> } >>> os::native_path(pathbuf); >>> >>> This also would separate strcpy() from >>> os::native_path() which is a bit unreadable. >>> >>> I've made the change. >>> >>> >>> -------------------- >>> >>> The single-byte path (strdup, ::stat()), together >>> with its free(), can be moved inside the >>> (strlen(path) < MAX_PATH) condition. os::native_path >>> will not change the path length (it better not, as >>> it operates on the input buffer). That avoids >>> having two allocations on the doublebyte path. >>> >>> >>> os::native_path() converts a path like C:\\\\somedir to >>> C:\\somedir >>> So I'll need the converted path in the wchar_t case too. >>> The freeing of the pathbuf is being done at the end of >>> the function or in the middle of the wchar_t case if >>> there's an error. >>> >>> >>> Oh, you are right,of course. I missed that it this is needed >>> for both paths. >>> >>> >>> >>> ----------------------- >>> >>> But seeing that translation to UNC paths is >>> something we might want more often, I would combine >>> allocation and UNC prefix adding to one function >>> like this, to avoid the memove and increase reusability: >>> >>> // Creates an unc path from a single byte path. >>> Return buffer is allocated in C heap and needs to be >>> freed by caller. Returns NULL on error. >>> static whchar_t* create_unc_path(const char* s) { >>> - does s start with \\?\ ? >>> - yes: >>> - os::malloc(strlen(s) + 1) and mbstowcs_s >>> - no: >>> - os::malloc(strlen(s) + 1 + 4), >>> mbstowcs_s to fourth position in string, prefix with >>> L"\\?\" >>> } >>> >>> >>> I also include the case for adding L"\\\\?\\UNC\0" at >>> the beginning to be consistent with >>> libjava/canonicalize_md.c. >>> >>> >>> We also need error checking to mbstowcs_s. >>> >>> I've added assert like the following after the call: >>> >>> assert(converted_chars == path_len, "sanity"); >>> >>> >>> >>> create_unc_path() : >>> >>> - could you convert the /**/ to // comments, please? >> Fixed. >> >> >> thanks. >> >> >>> >>> - not sure about the assert after mbstowcs_s: if we happen >>> to encounter an invalid multibyte character, function will >>> fail and return an error: >>> >>> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >>> >>> "If mbstowcs_s encounters an invalid multibyte character, it >>> puts 0 in *``pReturnValue, sets the destination buffer to an >>> empty string, sets errno to EILSEQ, and returns EILSEQ." >> I've changed create_unc_path() so that the caller will get >> the errno and removed the assert. >> >> static wchar_t* create_unc_path(const char* path, errno_t &err) >> >> >> Okay, works for me. >> >> >>> >>> As this is dependent on user data, we should not assert, >>> but handle the return code of mbstowcs_s gracefully. Same >>> goes for libjava/canonicalize_md.c. >>> >>> - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + >>> 7, path, path_len); >>> third parameter is wrong, as we hand in an offset into the >>> buffer, we must decrement the buffer size by this offset, so >>> correct would be path_len +7 - 7 or just path_len. >>> >>> - Same error below: + ::mbstowcs_s(&converted_chars, >>> &wpath[4], path_len + 4, path, path_len); >> Fixed in both places. >> >> >> Okay. >> >> >>> >>> >>> Just for cleanliness, I would then wrap deallocation >>> into an own function. >>> >>> static viud destroy_unc_path(whchar_t* s) { >>> os::free(s); } >>> >>> I've added the destroy function. >>> >>> >>> These two functions could be candidates of putting >>> into os::win32 namespace, as this form of ANSI->UNC >>> path translation is quite common - whoever wants to >>> use the xxxxW() functions must do this. >>> >>> I'm leaving them in os_windows.cpp since they're being >>> used only within that file. >>> >>> >>> Fine by me. >>> >>> >>> >>> ----------------------- >>> >>> FindFirstFileW: >>> >>> I am pretty sure that you can achieve the same >>> result with GetFileAttributesExW(). It also returns >>> WIN32_FIND_DATAW. >>> >>> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is >>> very similar to WIN32_FIND_DATAW. >>> >>> >>> It is way more straightforward to use than >>> FindFirstFileW, as it does not require you to write >>> a callback hook. >>> >>> I've switched to using GetFileAttributesExW(). >>> >>> >>> Thank you, this is way more readable. >>> Another issue: do we still need the fix for 6539723 if we >>> switch from stat() to GetFileAttributesExW()? This fix looks >>> important, but the comment >>> indicates that it could break things if the original bug is >>> not present. >>> >>> Btw, this is another strong argument for scrapping ::stat() >>> altogether on all code paths, not only for long input paths, >>> because stat() and GetFileAttributesExW() may behave >>> differently. So it would be good to use the same API on all >>> code paths, in order to get the best test coverage. >> For this round of change, I'm using >> GetFileAttributesEx[A|W]() for both code paths. >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ >> >> >> thanks, >> Calvin >> >> >> Okay, all good apart from the issues mentioned above. Thanks for >> your work! >> >> Best Regards, Thomas >> >> >> >>> >>> >>> >>> ------- >>> >>> eval_find_data(): This is more of a generic helper >>> function, could you rename this to something >>> clearer, e.g. make_double_word() ? >>> >>> Ok. I've renamed it. >>> >>> >>> Also, setup_stat(), could this be renamed more >>> clearly to something like WIN32_FIND_DATA_to_stat? >>> or lowercase if this bothers you :) >>> >>> I'm naming the function as file_attribute_data_to_stat() >>> to match with the data structure name. >>> >>> >>> Thanks for taking my suggestions. >>> >>> >>> >>> ================================== >>> src/hotspot/share/classfile/classLoader.cpp >>> >>> In ClassPathDirEntry::open_stream(), I would feel >>> better if we were asserting _dir and name to be != >>> NULL before feeding it to strlen. >>> >>> I've added an assert statement. >>> >>> >>> =================== >>> >>> Here's an updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >>> >>> >>> >>> Thanks! >>> >>> Thomas >>> >>> thanks, >>> Calvin >>> >>> >>> Thanks! >>> >>> Thomas >>> >>> >>> >>> >>> >>> >>> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >>> >> >>> >> >> wrote: >>> >>> I've reworked this fix by using the Unicode >>> version of open >>> (wopen) to handle path name longer than max path >>> with the path >>> prefixed to indicate an extended length path as >>> described in [1]. >>> >>> The Unicode version of stat (wstat) doesn't work >>> well with long >>> path [2]. So FindFirstFileW will be used.The data in >>> WIN32_FIND_DATA returned from FindFirstFileW >>> needs to be >>> transferred to the stat structure since the >>> caller expects a >>> return stat structure and other platforms return >>> a stat structure. >>> >>> In classLoader.cpp, calculate the size of buffer >>> required instead >>> of limiting it to JVM_MAXPATHLEN. >>> In os_windows.cpp, dynamically allocate buffers >>> in os::open and >>> os::stat. >>> >>> updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>> >>> >> > >>> >>> It passed hs-tier2 testing using mach5. >>> >>> thanks, >>> Calvin >>> >>> [1] >>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH >>> >>> >> > >>> >>> [2] >>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>> >>> >> > >>> >>> >>> >>> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>> >>> JBS: >>> https://bugs.openjdk.java.net/browse/JDK-8188122 >>> >>> >> > >>> >>> Adding a warning message if the full path or >>> the directory >>> length exceeds MAX_PATH on windows. >>> >>> Example warning messages. >>> >>> 1) The full path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: >>> construct full path >>> name failed: total length 270 exceeds max >>> length of 260 >>> dir >>> >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> length 259 >>> name Hello.class length 11 >>> >>> 2) The directory path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: >>> os::stat failed: >>> path length 265 exceeds max length 260 >>> path >>> >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>> >>> webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>> >>> >> > >>> >>> Testing: >>> JPRT (including the new test) >>> >>> thanks, >>> Calvin >>> >>> >>> >> > From mikhailo.seledtsov at oracle.com Thu Nov 9 19:15:38 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Thu, 09 Nov 2017 11:15:38 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A03AED1.3060203@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> Message-ID: <5A04A95A.8010106@oracle.com> Here is a differential webrev: http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ Misha On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: > Here is an update webrev: > http://cr.openjdk.java.net/~mseledtsov/8189762.02/ > I also tried to generate a diff between 01 and 02, but could not. My > script does not seem to work any more, or I am missing something. > Igor, please let me know if you have scripts or command line to > generate incremental webrev (I committed my initial changes locally, > then applied the new changes on top). > > Summary for updated webrev.02: > - implemented review feedback from Bob and Igor > - added @driver to all tests, since the actual testing happens in the > child docker process; the main is just a test driver > - configured docker directory for exclusive access by jtreg tests, > since there are potential and actual issues when running these tests > concurrently > - modified test groups to add a hotspot_docker test group, and to > exclude docker testing from lower tiers > (once the infra is ready, I plan to add this execution initially > at higher tiers; first will run it for a while as custom runs) > > My next steps: > - rebase to new jdk tree > - update/merge > - retest > > Thank you, > Misha > > > On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >> Hi Igor, >> >> Thank you for reviewing the code. >> >> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>> Hi Misha, >>> >>> I have several comments: >>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it >>> should be WB_IsContainerized(JNIEnv*, jobject) >> Fixed >>> 2. CPUSetsReader.java : listToString can be rewritten using stream >>> api as >>> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >> Thank you. I will try it out. >>> 3. in several files, I have noticed some problems w/ indents which >>> make code quite hard to read, for example >>>> 73 Common.run( Common.newOpts(imageName) >>>> 74 .addDockerOpts("--memory", valueToSet)) >>>> 75 .shouldMatch("Memory Limit is:.*" + >>>> expectedTraceValue); >>> or >>>> 96 DockerRunOptions opts = Common.newOpts(imageName, >>>> "AttemptOOM") >>>> 97 .addDockerOpts("--memory", dockerMemLimit, >>>> "--memory-swap", dockerMemLimit); >>>> 98 opts.classParams.add("" + sizeToAllocInMb); >> OK. I will unwind these expressions, and make sure indentation looks >> good. >>> 4. TestCPUSets.java : it's better to use >>> Assert.assertEquals(parts.length, 2) than >>> Asserts.assertTrue(parts.length == 2) as the former will also print >>> the actual value of parts.length. so you won't need to have L#103 >> OK >>> 5. Test*: there is no need to have parentheses in '@requires >>> (docker.support)' >> OK >>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under >>> test, so you can use '@run driver' to run it >> Will fix. >>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you >>> should use %n in format string. also System.out::printf seems to be >>> more popular in our code than System.out::format >> OK. >>> 8. TestCPUSets.java: shouldn't checkResult assert that we found >>> lineMarker? >> Oops. Missed it. Will add code to check it. >>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all >>> Common.run calls which aren't expected to fail, e.g. all test* in >>> TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in >>> TestMemoryAwareness >> Common.run() already checks for .shouldHaveExitValue(0) after running >> the container. >> >> >> I will apply feedback from Bob and you, and will post a new webrev. >> >> >> Thank you, >> Misha >>> >>> Thanks, >>> -- Igor >>> >>>> On Nov 1, 2017, at 8:11 PM, >>>> mikhailo wrote: >>>> >>>> Please review these tests that were developed to test JVM's >>>> container awareness in Docker environment. >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>> Webrev: >>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>> WB API: >>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>> Testing: >>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>> Ran the developed tests via jtreg >>>> Pass >>>> >>>> 2. Automated testing system - run these tests >>>> In progress >>>> >>>> Thank you, >>>> Misha >>>> From zgu at redhat.com Thu Nov 9 19:33:39 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 9 Nov 2017 14:33:39 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on Message-ID: This patch adds metadata reporting in NMT final report. What complicates the matter, is that, reporting per-class loader metadata requires a safepoint, so that it can safely walk class loaders. So, we only report it when JVM is about to exit in good state. Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ Thanks, -Zhengyu From bob.vandette at oracle.com Thu Nov 9 20:00:43 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Thu, 9 Nov 2017 15:00:43 -0500 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A04A95A.8010106@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> Message-ID: Do you still need this function? I don?t see it being used. 84 public static String read(String fileName) { 85 String path = CGROUP_CPUSET_PATH + fileName; 86 try { 87 return readLineFromFile(path); 88 } catch (Exception e) { 89 System.err.println("Exception reading " + path); 90 System.err.println("Exception: " + e); 91 } 92 93 return null; 94 } Bob. > On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov wrote: > > Here is a differential webrev: > http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ > > Misha > > On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >> Here is an update webrev: >> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >> I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. >> Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). >> >> Summary for updated webrev.02: >> - implemented review feedback from Bob and Igor >> - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver >> - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently >> - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers >> (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) >> >> My next steps: >> - rebase to new jdk tree >> - update/merge >> - retest >> >> Thank you, >> Misha >> >> >> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>> Hi Igor, >>> >>> Thank you for reviewing the code. >>> >>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>> Hi Misha, >>>> >>>> I have several comments: >>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>> Fixed >>>> 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>> Thank you. I will try it out. >>>> 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >>>>> 73 Common.run( Common.newOpts(imageName) >>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); >>>> or >>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >>>>> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>> OK. I will unwind these expressions, and make sure indentation looks good. >>>> 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 >>> OK >>>> 5. Test*: there is no need to have parentheses in '@requires (docker.support)' >>> OK >>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it >>> Will fix. >>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format >>> OK. >>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? >>> Oops. Missed it. Will add code to check it. >>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness >>> Common.run() already checks for .shouldHaveExitValue(0) after running the container. >>> >>> >>> I will apply feedback from Bob and you, and will post a new webrev. >>> >>> >>> Thank you, >>> Misha >>>> >>>> Thanks, >>>> -- Igor >>>> >>>>> On Nov 1, 2017, at 8:11 PM, mikhailo wrote: >>>>> >>>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>> Webrev: >>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>> Testing: >>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>> Ran the developed tests via jtreg >>>>> Pass >>>>> >>>>> 2. Automated testing system - run these tests >>>>> In progress >>>>> >>>>> Thank you, >>>>> Misha >>>>> From mikhailo.seledtsov at oracle.com Thu Nov 9 20:31:47 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Thu, 09 Nov 2017 12:31:47 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> Message-ID: <5A04BB33.2030207@oracle.com> I am not using it. I thought it may be helpful if we go back to that method of extracting cpuset values for some reason. On the other hand, dead code is not a good thing. So I am a bit undecided. I will delete it if you think it is better to do so. Misha On 11/9/17, 12:00 PM, Bob Vandette wrote: > Do you still need this function? I don?t see it being used. > > 84 public static String read(String fileName) { > 85 String path = CGROUP_CPUSET_PATH + fileName; > 86 try { > 87 return readLineFromFile(path); > 88 } catch (Exception e) { > 89 System.err.println("Exception reading " + path); > 90 System.err.println("Exception: " + e); > 91 } > 92 > 93 return null; > 94 } > > Bob. > >> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov >> > > wrote: >> >> Here is a differential webrev: >> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >> >> >> Misha >> >> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>> Here is an update webrev: >>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>> >>> I also tried to generate a diff between 01 and 02, but could not. My >>> script does not seem to work any more, or I am missing something. >>> Igor, please let me know if you have scripts or command line to >>> generate incremental webrev (I committed my initial changes locally, >>> then applied the new changes on top). >>> >>> Summary for updated webrev.02: >>> - implemented review feedback from Bob and Igor >>> - added @driver to all tests, since the actual testing happens in >>> the child docker process; the main is just a test driver >>> - configured docker directory for exclusive access by jtreg tests, >>> since there are potential and actual issues when running these tests >>> concurrently >>> - modified test groups to add a hotspot_docker test group, and to >>> exclude docker testing from lower tiers >>> (once the infra is ready, I plan to add this execution initially >>> at higher tiers; first will run it for a while as custom runs) >>> >>> My next steps: >>> - rebase to new jdk tree >>> - update/merge >>> - retest >>> >>> Thank you, >>> Misha >>> >>> >>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>> Hi Igor, >>>> >>>> Thank you for reviewing the code. >>>> >>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>> Hi Misha, >>>>> >>>>> I have several comments: >>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, >>>>> it should be WB_IsContainerized(JNIEnv*, jobject) >>>> Fixed >>>>> 2. CPUSetsReader.java : listToString can be rewritten using stream >>>>> api as >>>>> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>> Thank you. I will try it out. >>>>> 3. in several files, I have noticed some problems w/ indents which >>>>> make code quite hard to read, for example >>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>> 75 .shouldMatch("Memory Limit is:.*" + >>>>>> expectedTraceValue); >>>>> or >>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, >>>>>> "AttemptOOM") >>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, >>>>>> "--memory-swap", dockerMemLimit); >>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>> OK. I will unwind these expressions, and make sure indentation >>>> looks good. >>>>> 4. TestCPUSets.java : it's better to use >>>>> Assert.assertEquals(parts.length, 2) than >>>>> Asserts.assertTrue(parts.length == 2) as the former will also >>>>> print the actual value of parts.length. so you won't need to have >>>>> L#103 >>>> OK >>>>> 5. Test*: there is no need to have parentheses in '@requires >>>>> (docker.support)' >>>> OK >>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under >>>>> test, so you can use '@run driver' to run it >>>> Will fix. >>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you >>>>> should use %n in format string. also System.out::printf seems to >>>>> be more popular in our code than System.out::format >>>> OK. >>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found >>>>> lineMarker? >>>> Oops. Missed it. Will add code to check it. >>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all >>>>> Common.run calls which aren't expected to fail, e.g. all test* in >>>>> TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in >>>>> TestMemoryAwareness >>>> Common.run() already checks for .shouldHaveExitValue(0) after >>>> running the container. >>>> >>>> >>>> I will apply feedback from Bob and you, and will post a new webrev. >>>> >>>> >>>> Thank you, >>>> Misha >>>>> >>>>> Thanks, >>>>> -- Igor >>>>> >>>>>> On Nov 1, 2017, at 8:11 PM, >>>>>> mikhailo>>>>> > wrote: >>>>>> >>>>>> Please review these tests that were developed to test JVM's >>>>>> container awareness in Docker environment. >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>> Webrev: >>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>> >>>>>> WB API: >>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>> >>>>>> Testing: >>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>> Ran the developed tests via jtreg >>>>>> Pass >>>>>> >>>>>> 2. Automated testing system - run these tests >>>>>> In progress >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> > From bob.vandette at oracle.com Thu Nov 9 20:38:04 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Thu, 9 Nov 2017 15:38:04 -0500 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A04BB33.2030207@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> Message-ID: <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> Since that path is not guaranteed to work, I?d remove CGROUP_CPUSET_PATH and the function. Bob. > On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov wrote: > > I am not using it. I thought it may be helpful if we go back to that method of extracting cpuset values for some reason. > On the other hand, dead code is not a good thing. So I am a bit undecided. > I will delete it if you think it is better to do so. > > Misha > > On 11/9/17, 12:00 PM, Bob Vandette wrote: >> >> Do you still need this function? I don?t see it being used. >> >> 84 public static String read(String fileName) { >> 85 String path = CGROUP_CPUSET_PATH + fileName; >> 86 try { >> 87 return readLineFromFile(path); >> 88 } catch (Exception e) { >> 89 System.err.println("Exception reading " + path); >> 90 System.err.println("Exception: " + e); >> 91 } >> 92 >> 93 return null; >> 94 } >> >> Bob. >> >>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov > wrote: >>> >>> Here is a differential webrev: >>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>> >>> Misha >>> >>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>> Here is an update webrev: >>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>> I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. >>>> Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). >>>> >>>> Summary for updated webrev.02: >>>> - implemented review feedback from Bob and Igor >>>> - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver >>>> - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently >>>> - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers >>>> (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) >>>> >>>> My next steps: >>>> - rebase to new jdk tree >>>> - update/merge >>>> - retest >>>> >>>> Thank you, >>>> Misha >>>> >>>> >>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>> Hi Igor, >>>>> >>>>> Thank you for reviewing the code. >>>>> >>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>> Hi Misha, >>>>>> >>>>>> I have several comments: >>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>> Fixed >>>>>> 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>> Thank you. I will try it out. >>>>>> 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); >>>>>> or >>>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>> OK. I will unwind these expressions, and make sure indentation looks good. >>>>>> 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 >>>>> OK >>>>>> 5. Test*: there is no need to have parentheses in '@requires (docker.support)' >>>>> OK >>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it >>>>> Will fix. >>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format >>>>> OK. >>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? >>>>> Oops. Missed it. Will add code to check it. >>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness >>>>> Common.run() already checks for .shouldHaveExitValue(0) after running the container. >>>>> >>>>> >>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>>> >>>>>> Thanks, >>>>>> -- Igor >>>>>> >>>>>>> On Nov 1, 2017, at 8:11 PM, mikhailo> wrote: >>>>>>> >>>>>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>> Webrev: >>>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>> Testing: >>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>> Ran the developed tests via jtreg >>>>>>> Pass >>>>>>> >>>>>>> 2. Automated testing system - run these tests >>>>>>> In progress >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >> From mikhailo.seledtsov at oracle.com Thu Nov 9 21:04:21 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Thu, 09 Nov 2017 13:04:21 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> Message-ID: <5A04C2D5.8090404@oracle.com> OK, will remove it. Misha On 11/9/17, 12:38 PM, Bob Vandette wrote: > Since that path is not guaranteed to work, I?d remove > CGROUP_CPUSET_PATH and the function. > > Bob. > >> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov >> > > wrote: >> >> I am not using it. I thought it may be helpful if we go back to that >> method of extracting cpuset values for some reason. >> On the other hand, dead code is not a good thing. So I am a bit >> undecided. >> I will delete it if you think it is better to do so. >> >> Misha >> >> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>> Do you still need this function? I don?t see it being used. >>> >>> 84 public static String read(String fileName) { >>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>> 86 try { >>> 87 return readLineFromFile(path); >>> 88 } catch (Exception e) { >>> 89 System.err.println("Exception reading " + path); >>> 90 System.err.println("Exception: " + e); >>> 91 } >>> 92 >>> 93 return null; >>> 94 } >>> >>> Bob. >>> >>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov >>>> >>> > wrote: >>>> >>>> Here is a differential webrev: >>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>> >>>> >>>> Misha >>>> >>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>> Here is an update webrev: >>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>> >>>>> I also tried to generate a diff between 01 and 02, but could not. >>>>> My script does not seem to work any more, or I am missing something. >>>>> Igor, please let me know if you have scripts or command line to >>>>> generate incremental webrev (I committed my initial changes >>>>> locally, then applied the new changes on top). >>>>> >>>>> Summary for updated webrev.02: >>>>> - implemented review feedback from Bob and Igor >>>>> - added @driver to all tests, since the actual testing happens in >>>>> the child docker process; the main is just a test driver >>>>> - configured docker directory for exclusive access by jtreg tests, >>>>> since there are potential and actual issues when running these >>>>> tests concurrently >>>>> - modified test groups to add a hotspot_docker test group, and to >>>>> exclude docker testing from lower tiers >>>>> (once the infra is ready, I plan to add this execution >>>>> initially at higher tiers; first will run it for a while as custom >>>>> runs) >>>>> >>>>> My next steps: >>>>> - rebase to new jdk tree >>>>> - update/merge >>>>> - retest >>>>> >>>>> Thank you, >>>>> Misha >>>>> >>>>> >>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>> Hi Igor, >>>>>> >>>>>> Thank you for reviewing the code. >>>>>> >>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>> Hi Misha, >>>>>>> >>>>>>> I have several comments: >>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, >>>>>>> it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>> Fixed >>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using >>>>>>> stream api as >>>>>>> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>> Thank you. I will try it out. >>>>>>> 3. in several files, I have noticed some problems w/ indents >>>>>>> which make code quite hard to read, for example >>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + >>>>>>>> expectedTraceValue); >>>>>>> or >>>>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, >>>>>>>> "AttemptOOM") >>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, >>>>>>>> "--memory-swap", dockerMemLimit); >>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>> OK. I will unwind these expressions, and make sure indentation >>>>>> looks good. >>>>>>> 4. TestCPUSets.java : it's better to use >>>>>>> Assert.assertEquals(parts.length, 2) than >>>>>>> Asserts.assertTrue(parts.length == 2) as the former will also >>>>>>> print the actual value of parts.length. so you won't need to >>>>>>> have L#103 >>>>>> OK >>>>>>> 5. Test*: there is no need to have parentheses in '@requires >>>>>>> (docker.support)' >>>>>> OK >>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under >>>>>>> test, so you can use '@run driver' to run it >>>>>> Will fix. >>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you >>>>>>> should use %n in format string. also System.out::printf seems to >>>>>>> be more popular in our code than System.out::format >>>>>> OK. >>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found >>>>>>> lineMarker? >>>>>> Oops. Missed it. Will add code to check it. >>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to >>>>>>> all Common.run calls which aren't expected to fail, e.g. all >>>>>>> test* in TestCPUAwareness, testMemorySoftLimit and >>>>>>> testMemoryLimit in TestMemoryAwareness >>>>>> Common.run() already checks for .shouldHaveExitValue(0) after >>>>>> running the container. >>>>>> >>>>>> >>>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>> >>>>>>> Thanks, >>>>>>> -- Igor >>>>>>> >>>>>>>> On Nov 1, 2017, at 8:11 PM, >>>>>>>> mikhailo>>>>>>> > wrote: >>>>>>>> >>>>>>>> Please review these tests that were developed to test JVM's >>>>>>>> container awareness in Docker environment. >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>> Webrev: >>>>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>> >>>>>>>> WB API: >>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>> >>>>>>>> Testing: >>>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>>> Ran the developed tests via jtreg >>>>>>>> Pass >>>>>>>> >>>>>>>> 2. Automated testing system - run these tests >>>>>>>> In progress >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> >>> > From manc at google.com Fri Nov 10 02:31:38 2017 From: manc at google.com (Man Cao) Date: Thu, 9 Nov 2017 18:31:38 -0800 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: <79bcec31-20d1-1aef-b9a6-1134676aa289@oracle.com> References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> <79bcec31-20d1-1aef-b9a6-1134676aa289@oracle.com> Message-ID: Hi Yasumasa, I'm looking at if this patch can fix the original issue with hsperfdata counters sun_gc_metaspace_maxCapacity and sun_gc_compressedclassspace_maxCapacity being too large when user has set -XX:MaxMetaspaceSize=100m or some value less than 1GB. However, it does not, these counters still reports around 1GB if MaxMetaspaceSize is less than 1GB. The reason is that the code added in this patch to Metaspace::ergo_initialize() is AFTER the two lines: CompressedClassSpaceSize = align_size_down_bounded(CompressedClassSpaceSize, _reserve_alignment); set_compressed_class_space_size(CompressedClassSpaceSize); So set_compressed_class_space_size() still gets the old value of CompressedClassSpaceSize that is too large, making the reserved size of metaspace too large. Should these two lines be moved after the code added in this patch? Thanks, -Man On Tue, Oct 17, 2017 at 12:46 PM, wrote: > > Hi Yasumasa, > This looks good. I can sponsor this for you. > thanks, > Coleen > > > On 10/12/17 1:16 AM, Yasumasa Suenaga wrote: > >> Thanks David! >> >> I've fixed: >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ >> >> >> Yasumasa >> >> >> 2017-10-12 13:55 GMT+09:00 David Holmes : >> >>> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >>> >>>> Hi David, >>>> >>>> You may have time to fix these in place before anyone else sees the >>>>> webrev >>>>> >>>> >>>> I've fixed that: >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >>>> >>>> >>>> I checked raw text of your email, so I believe the indent is correct. >>>> If I have mistake(s), please tell me. >>>> >>> >>> Sorry they are off. Basic rules: >>> >>> var = >>> some long value; >>> >>> indent by 4. >>> >>> foo(arg1, arg2, >>> arg3, arg4); >>> >>> align the args on the two lines as shown. >>> >>> Hope that is clearer. >>> >>> Thanks, >>> David >>> >>> >>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> 2017-10-12 13:22 GMT+09:00 David Holmes : >>>> >>>>> Hi Yasumasa, >>>>> >>>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>>> >>>>>> >>>>>> Hi all, >>>>>> >>>>>> This change was tried to push via JPRT, but it failed because >>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSiz >>>>>> eTest.java >>>>>> was failed. >>>>>> Also I heard the comment that the changes in arguments.cpp which set >>>>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>>>> Metaspace::ergo_initialize(). >>>>>> >>>>>> I uploaded new webrev. Could you review again? >>>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>>>>> >>>>> >>>>> >>>>> I'll let Coleen cover the technical details, I'll just pick up a few >>>>> style >>>>> nits: >>>>> >>>>> 3328 /* Initial virtual space size will be calculated at >>>>> global_initialize() */ >>>>> >>>>> Use // comment >>>>> >>>>> 3329 size_t min_metaspace_sz = >>>>> 3330 VIRTUALSPACEMULTIPLIER * >>>>> InitialBootClassLoaderMetaspaceSize; >>>>> >>>>> ^ should indent only to here >>>>> >>>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>>>> 3337 MaxMetaspaceSize - >>>>> min_metaspace_sz); >>>>> ^ should indent only to here >>>>> >>>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>>>> 3342 min_metaspace_sz); >>>>> >>>>> ^ should indent only to here >>>>> >>>>> You may have time to fix these in place before anyone else sees the >>>>> webrev >>>>> :) >>>>> >>>>> Thanks, >>>>> David >>>>> ----- >>>>> >>>>> >>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> >>>>>> 2017-10-10 21:02 GMT+09:00 : >>>>>> >>>>>>> >>>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it >>>>>>> only >>>>>>> needs >>>>>>> one reviewer, unless there were more? Please send me the result of >>>>>>> hg >>>>>>> export. >>>>>>> thanks, >>>>>>> Coleen >>>>>>> >>>>>>> >>>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi Coleen, >>>>>>>> >>>>>>>> I will sponsor this for you. >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>>> check_metaspace_sizes() >>>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>>> is. >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>>>>> to get minimum metaspace size, and uses return value from this >>>>>>>> function in metaspace initialization and metaspace size check. >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>>>> >>>>>>>> Could you review again? >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>>>> >>>>>>>>> -Man >>>>>>>>> >>>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>>> check_metaspace_sizes() >>>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>>> is. >>>>>>>>>> >>>>>>>>>> I will sponsor this for you. >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> Coleen >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~y >>>>>>>>>>> suenaga/JDK-8087291/webrev.04/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga >>>>>>>>>> >: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>>>> Could you review it? >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~y >>>>>>>>>>>> suenaga/JDK-8087291/webrev.03/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Yasumasa >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>>>> >>>>>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>>>>> internal >>>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>>>> >>>>>>>>>>>>> -Man >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>>> class >>>>>>>>>>>>>>> space >>>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>>>> after >>>>>>>>>>>>>> repos >>>>>>>>>>>>>> are opened. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>> since we >>>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>>> class >>>>>>>>>>>>>>> space >>>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> StefanK >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Man >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I wonder if there is any recent update on the patch >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> its >>>>>>>>>>>>>>>> low >>>>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>>>> getting >>>>>>>>>>>>>>>> approved >>>>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>>>>> encountered >>>>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root >>>>>>>>>>>>>>>> cause >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>> capped >>>>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>>>> during VM initialization, and this patch seems fix >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> root >>>>>>>>>>>>>>>> cause. >>>>>>>>>>>>>>>> (I'm aware that even after this patch, the reserved >>>>>>>>>>>>>>>> size >>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Man >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> suenaga/JDK-8087291/webrev.02/> >>>>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > On a linux system I get this when I build >>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>> patch. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> argument >>>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>> >> # A fatal error has been detected by the >>>>>>>>>>>>>>>> Java >>>>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-co >>>>>>>>>>>>>>>> de_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://mail.openjdk.java.net/p >>>>>>>>>>>>>>>> ipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> pipermail/hotspot-gc-dev/2015-June/013873.html> >>>>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> show >>>>>>>>>>>>>>>> error message? >>>>>>>>>>>>>>>> >>> If you add slightly better heuristics for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> setup >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>>>> lowering >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> set, >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>>>> settings. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> suenaga/JDK-8087291/webrev.01/> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> > From yasuenag at gmail.com Fri Nov 10 08:15:50 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Fri, 10 Nov 2017 17:15:50 +0900 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> <79bcec31-20d1-1aef-b9a6-1134676aa289@oracle.com> Message-ID: Hi Man, 2017-11-10 11:31 GMT+09:00 Man Cao : > Hi Yasumasa, > > I'm looking at if this patch can fix the original issue with hsperfdata > counters sun_gc_metaspace_maxCapacity and > sun_gc_compressedclassspace_maxCapacity being too large when user has set > -XX:MaxMetaspaceSize=100m or some value less than 1GB. However, it does not, > these counters still reports around 1GB if MaxMetaspaceSize is less than > 1GB. > > The reason is that the code added in this patch to > Metaspace::ergo_initialize() is AFTER the two lines: > > CompressedClassSpaceSize = > align_size_down_bounded(CompressedClassSpaceSize, _reserve_alignment); > set_compressed_class_space_size(CompressedClassSpaceSize); > > So set_compressed_class_space_size() still gets the old value of > CompressedClassSpaceSize that is too large, making the reserved size of > metaspace too large. > > Should these two lines be moved after the code added in this patch? I agree with you. Can you send review request for it? According to OCA signatories [1], Google signs OCA, so I think you can contribute it to OpenJDK. I can file this to JBS, and create webrev and changest. But I cannot push it to jdk/hs repo. So you need to find a sponsor. I'm a reviewer of OpenJDK. So I can +1 to your change when you send review request. Thanks, Yasumasa [1] http://www.oracle.com/technetwork/community/oca-486395.html > Thanks, > -Man > > On Tue, Oct 17, 2017 at 12:46 PM, wrote: >> >> >> Hi Yasumasa, >> This looks good. I can sponsor this for you. >> thanks, >> Coleen >> >> >> On 10/12/17 1:16 AM, Yasumasa Suenaga wrote: >>> >>> Thanks David! >>> >>> I've fixed: >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ >>> >>> >>> Yasumasa >>> >>> >>> 2017-10-12 13:55 GMT+09:00 David Holmes : >>>> >>>> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >>>>> >>>>> Hi David, >>>>> >>>>>> You may have time to fix these in place before anyone else sees the >>>>>> webrev >>>>> >>>>> >>>>> I've fixed that: >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >>>>> >>>>> >>>>> I checked raw text of your email, so I believe the indent is correct. >>>>> If I have mistake(s), please tell me. >>>> >>>> >>>> Sorry they are off. Basic rules: >>>> >>>> var = >>>> some long value; >>>> >>>> indent by 4. >>>> >>>> foo(arg1, arg2, >>>> arg3, arg4); >>>> >>>> align the args on the two lines as shown. >>>> >>>> Hope that is clearer. >>>> >>>> Thanks, >>>> David >>>> >>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> 2017-10-12 13:22 GMT+09:00 David Holmes : >>>>>> >>>>>> Hi Yasumasa, >>>>>> >>>>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> This change was tried to push via JPRT, but it failed because >>>>>>> >>>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>>>>>> was failed. >>>>>>> Also I heard the comment that the changes in arguments.cpp which set >>>>>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>>>>> Metaspace::ergo_initialize(). >>>>>>> >>>>>>> I uploaded new webrev. Could you review again? >>>>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>>>>> >>>>>> >>>>>> >>>>>> I'll let Coleen cover the technical details, I'll just pick up a few >>>>>> style >>>>>> nits: >>>>>> >>>>>> 3328 /* Initial virtual space size will be calculated at >>>>>> global_initialize() */ >>>>>> >>>>>> Use // comment >>>>>> >>>>>> 3329 size_t min_metaspace_sz = >>>>>> 3330 VIRTUALSPACEMULTIPLIER * >>>>>> InitialBootClassLoaderMetaspaceSize; >>>>>> >>>>>> ^ should indent only to here >>>>>> >>>>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>>>>> 3337 MaxMetaspaceSize - >>>>>> min_metaspace_sz); >>>>>> ^ should indent only to here >>>>>> >>>>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>>>>> 3342 min_metaspace_sz); >>>>>> >>>>>> ^ should indent only to here >>>>>> >>>>>> You may have time to fix these in place before anyone else sees the >>>>>> webrev >>>>>> :) >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> ----- >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2017-10-10 21:02 GMT+09:00 : >>>>>>>> >>>>>>>> >>>>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it >>>>>>>> only >>>>>>>> needs >>>>>>>> one reviewer, unless there were more? Please send me the result of >>>>>>>> hg >>>>>>>> export. >>>>>>>> thanks, >>>>>>>> Coleen >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Coleen, >>>>>>>>> >>>>>>>>>>> I will sponsor this for you. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> >>>>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>>>> check_metaspace_sizes() >>>>>>>>>>> so that you don't have to tell arguments what the MediumChunk >>>>>>>>>>> size >>>>>>>>>>> is. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I added new function calculate_min_metaspace_size() in >>>>>>>>> metaspace.cpp >>>>>>>>> to get minimum metaspace size, and uses return value from this >>>>>>>>> function in metaspace initialization and metaspace size check. >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>>>>> >>>>>>>>> Could you review again? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> >>>>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>>>>> >>>>>>>>>> -Man >>>>>>>>>> >>>>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>>>> check_metaspace_sizes() >>>>>>>>>>> so that you don't have to tell arguments what the MediumChunk >>>>>>>>>>> size >>>>>>>>>>> is. >>>>>>>>>>> >>>>>>>>>>> I will sponsor this for you. >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> Coleen >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Yasumasa >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga >>>>>>>>>>>> : >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>>>>> Could you review it? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Good to know that the patch is not blocked due to breaking >>>>>>>>>>>>>> some >>>>>>>>>>>>>> internal >>>>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Man >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>>>> class >>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>>>>> after >>>>>>>>>>>>>>> repos >>>>>>>>>>>>>>> are opened. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>>> since we >>>>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>>>> class >>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> StefanK >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Man >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I wonder if there is any recent update on the >>>>>>>>>>>>>>>>> patch >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? >>>>>>>>>>>>>>>>> Except >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> its >>>>>>>>>>>>>>>>> low >>>>>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>>>>> getting >>>>>>>>>>>>>>>>> approved >>>>>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We >>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> encountered >>>>>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root >>>>>>>>>>>>>>>>> cause >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>>> capped >>>>>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>>>>> during VM initialization, and this patch seems fix >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> root >>>>>>>>>>>>>>>>> cause. >>>>>>>>>>>>>>>>> (I'm aware that even after this patch, the >>>>>>>>>>>>>>>>> reserved >>>>>>>>>>>>>>>>> size >>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Man >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>>>>> > Try running a debug JVM with your patch >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Try running a debug JVM with your patch >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > On a linux system I get this when I build >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>> patch. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> argument >>>>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>>> >> # A fatal error has been detected by the >>>>>>>>>>>>>>>>> Java >>>>>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> show >>>>>>>>>>>>>>>>> error message? >>>>>>>>>>>>>>>>> >>> If you add slightly better heuristics for >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> setup >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>>>>> lowering >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> set, >>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>>>>> settings. >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will >>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >> > From tobias.hartmann at oracle.com Fri Nov 10 08:35:52 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Fri, 10 Nov 2017 09:35:52 +0100 Subject: [10] RFR(S): 8190797: OSR compilation fails with "assert(__the_thread__->can_call_java()) failed: can not load classes with compiler thread" In-Reply-To: References: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> Message-ID: <34b31a7b-aacf-c12c-85b2-2a917d6ffa42@oracle.com> Hi, could I please get a second review from the runtime team? Thanks, Tobias On 08.11.2017 08:58, Tobias Hartmann wrote: > Hi Vladimir, > > thanks for the review! > > Best regards, > Tobias > > On 07.11.2017 19:56, Vladimir Kozlov wrote: >> Looks good. >> >> Thanks, >> Vladimir >> >> On 11/7/17 4:04 AM, Tobias Hartmann wrote: >>> Hi, >>> >>> please review the following patch: >>> https://bugs.openjdk.java.net/browse/JDK-8190797 >>> http://cr.openjdk.java.net/~thartmann/8190797/webrev.00/ >>> >>> When oop map creation fails, we try to throw a LinkageError to propagate the error message. If this happens in the >>> compiler thread (for example, during OSR compilation), we fail because a compiler thread cannot initialize an exception >>> object. >>> >>> I've fixed this by bailing out with a meaningful message in case !Thread::can_call_java(). Please note that even if we >>> are able to instantiate an exception, we will still fail with ShouldNotReachHere because compute_map(TRAPS) is called >>> with CATCH (see comments in the bug for a detailed explanation). This fix is not about changing this behavior but to >>> fail with a meaningful error message during compilation. This should only happen if something is seriously broken (for >>> example, incorrect bytecode with -noverify, see TestLinkageErrorInGenerateOopMap). In this case we would probably hit >>> other issues as well if we would continue execution. >>> >>> Thanks, >>> Tobias >>> From adinn at redhat.com Fri Nov 10 09:22:29 2017 From: adinn at redhat.com (Andrew Dinn) Date: Fri, 10 Nov 2017 09:22:29 +0000 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: References: Message-ID: <43f776c6-c873-3d0e-c7e2-1f3209012946@redhat.com> Hi Zhengyu, On 09/11/17 19:33, Zhengyu Gu wrote: > This patch adds metadata reporting in NMT final report. > > What complicates the matter, is that, reporting per-class loader > metadata requires a safepoint, so that it can safely walk class loaders. > So, we only report it when JVM is about to exit in good state. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 > Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ I understand the need here to avoid reporting if we are under a fatal error. However, that assert is not going to work in product code. So, does that not imply that execution of the vmop needs to be conditional on VMError::fatal_error_in_progress() returning false? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From david.holmes at oracle.com Fri Nov 10 12:00:21 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 10 Nov 2017 22:00:21 +1000 Subject: [10] RFR(S): 8190797: OSR compilation fails with "assert(__the_thread__->can_call_java()) failed: can not load classes with compiler thread" In-Reply-To: <34b31a7b-aacf-c12c-85b2-2a917d6ffa42@oracle.com> References: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> <34b31a7b-aacf-c12c-85b2-2a917d6ffa42@oracle.com> Message-ID: <44af395f-5614-778a-63e4-e122af6300a7@oracle.com> On 10/11/2017 6:35 PM, Tobias Hartmann wrote: > Hi, > > could I please get a second review from the runtime team? Reviewed. Thanks, David PS. Never knew that -noverify existed as an alias for -Xverify:none :) > Thanks, > Tobias > > On 08.11.2017 08:58, Tobias Hartmann wrote: >> Hi Vladimir, >> >> thanks for the review! >> >> Best regards, >> Tobias >> >> On 07.11.2017 19:56, Vladimir Kozlov wrote: >>> Looks good. >>> >>> Thanks, >>> Vladimir >>> >>> On 11/7/17 4:04 AM, Tobias Hartmann wrote: >>>> Hi, >>>> >>>> please review the following patch: >>>> https://bugs.openjdk.java.net/browse/JDK-8190797 >>>> http://cr.openjdk.java.net/~thartmann/8190797/webrev.00/ >>>> >>>> When oop map creation fails, we try to throw a LinkageError to propagate the error message. If this happens in the >>>> compiler thread (for example, during OSR compilation), we fail because a compiler thread cannot initialize an exception >>>> object. >>>> >>>> I've fixed this by bailing out with a meaningful message in case !Thread::can_call_java(). Please note that even if we >>>> are able to instantiate an exception, we will still fail with ShouldNotReachHere because compute_map(TRAPS) is called >>>> with CATCH (see comments in the bug for a detailed explanation). This fix is not about changing this behavior but to >>>> fail with a meaningful error message during compilation. This should only happen if something is seriously broken (for >>>> example, incorrect bytecode with -noverify, see TestLinkageErrorInGenerateOopMap). In this case we would probably hit >>>> other issues as well if we would continue execution. >>>> >>>> Thanks, >>>> Tobias >>>> From tobias.hartmann at oracle.com Fri Nov 10 12:08:24 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Fri, 10 Nov 2017 13:08:24 +0100 Subject: [10] RFR(S): 8190797: OSR compilation fails with "assert(__the_thread__->can_call_java()) failed: can not load classes with compiler thread" In-Reply-To: <44af395f-5614-778a-63e4-e122af6300a7@oracle.com> References: <9ab0bc1c-605f-7b28-57e9-c522b40ca4c1@oracle.com> <01dbd4cb-0683-a36d-5ee9-b4950e10c517@oracle.com> <34b31a7b-aacf-c12c-85b2-2a917d6ffa42@oracle.com> <44af395f-5614-778a-63e4-e122af6300a7@oracle.com> Message-ID: Hi David, On 10.11.2017 13:00, David Holmes wrote: > Reviewed. Thank you! Best regards, Tobias From zgu at redhat.com Fri Nov 10 12:47:56 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 10 Nov 2017 07:47:56 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: <43f776c6-c873-3d0e-c7e2-1f3209012946@redhat.com> References: <43f776c6-c873-3d0e-c7e2-1f3209012946@redhat.com> Message-ID: <75af76f8-85d9-f4bb-2da8-545d9334412a@redhat.com> Hi Andrew, On 11/10/2017 04:22 AM, Andrew Dinn wrote: > Hi Zhengyu, > > On 09/11/17 19:33, Zhengyu Gu wrote: >> This patch adds metadata reporting in NMT final report. >> >> What complicates the matter, is that, reporting per-class loader >> metadata requires a safepoint, so that it can safely walk class loaders. >> So, we only report it when JVM is about to exit in good state. >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 >> Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ > I understand the need here to avoid reporting if we are under a fatal > error. However, that assert is not going to work in product code. So, > does that not imply that execution of the vmop needs to be conditional > on VMError::fatal_error_in_progress() returning false? MemTracker::report() is a private method, only called by MemTracker::final_report() and MemTracker::error_report(). This assertion ensures that metadata report should never be included in MemTracker::error_report() if it decides to extend beyond summary report. Of course, this is under assumption that final_report() is called during JVM normal shutdown (good state) and error_report() is by error handler (bad state). Besides, VMError::fatal_error_in_progress() is inherently racy. Thanks, -Zhengyu > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From thomas.stuefe at gmail.com Fri Nov 10 13:20:25 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 10 Nov 2017 13:20:25 +0000 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: References: Message-ID: Hi Zhengyu, Cannot test it before monday, but change looks good. Thank you for doing this. ..Thomas On Thu 9. Nov 2017 at 20:33, Zhengyu Gu wrote: > This patch adds metadata reporting in NMT final report. > > What complicates the matter, is that, reporting per-class loader > metadata requires a safepoint, so that it can safely walk class loaders. > So, we only report it when JVM is about to exit in good state. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 > Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ > > > Thanks, > > -Zhengyu > From zgu at redhat.com Fri Nov 10 13:23:47 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 10 Nov 2017 08:23:47 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: References: Message-ID: On 11/10/2017 08:20 AM, Thomas St?fe wrote: > Hi Zhengyu, > > Cannot test it before monday, but change looks good. Thank you for > doing this. No problem. Thanks, -Zhengyu > > ..Thomas > > On Thu 9. Nov 2017 at 20:33, Zhengyu Gu > wrote: > > This patch adds metadata reporting in NMT final report. > > What complicates the matter, is that, reporting per-class loader > metadata requires a safepoint, so that it can safely walk class loaders. > So, we only report it when JVM is about to exit in good state. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 > Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ > > > Thanks, > > -Zhengyu > From adinn at redhat.com Fri Nov 10 14:08:59 2017 From: adinn at redhat.com (Andrew Dinn) Date: Fri, 10 Nov 2017 14:08:59 +0000 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: <75af76f8-85d9-f4bb-2da8-545d9334412a@redhat.com> References: <43f776c6-c873-3d0e-c7e2-1f3209012946@redhat.com> <75af76f8-85d9-f4bb-2da8-545d9334412a@redhat.com> Message-ID: <282ade0c-9c77-f100-c8c7-fac0f81d2eec@redhat.com> On 10/11/17 12:47, Zhengyu Gu wrote: > On 11/10/2017 04:22 AM, Andrew Dinn wrote: >> I understand the need here to avoid reporting if we are under a fatal >> error. However, that assert is not going to work in product code. So, >> does that not imply that execution of the vmop needs to be conditional >> on VMError::fatal_error_in_progress() returning false? > > MemTracker::report() is a private method, only called by > MemTracker::final_report() and MemTracker::error_report(). This > assertion ensures that metadata report should never be included in > MemTracker::error_report() if it decides to extend beyond summary report. > > Of course, this is under assumption that final_report() is called during > JVM normal shutdown (good state) and error_report() is by error handler > (bad state). Ok, thanks for the explanation. The change looks fine. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From zgu at redhat.com Fri Nov 10 14:22:42 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 10 Nov 2017 09:22:42 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: <282ade0c-9c77-f100-c8c7-fac0f81d2eec@redhat.com> References: <43f776c6-c873-3d0e-c7e2-1f3209012946@redhat.com> <75af76f8-85d9-f4bb-2da8-545d9334412a@redhat.com> <282ade0c-9c77-f100-c8c7-fac0f81d2eec@redhat.com> Message-ID: Thanks for the review, Andrew. -Zhengyu On 11/10/2017 09:08 AM, Andrew Dinn wrote: > On 10/11/17 12:47, Zhengyu Gu wrote: >> On 11/10/2017 04:22 AM, Andrew Dinn wrote: > >>> I understand the need here to avoid reporting if we are under a fatal >>> error. However, that assert is not going to work in product code. So, >>> does that not imply that execution of the vmop needs to be conditional >>> on VMError::fatal_error_in_progress() returning false? >> >> MemTracker::report() is a private method, only called by >> MemTracker::final_report() and MemTracker::error_report(). This >> assertion ensures that metadata report should never be included in >> MemTracker::error_report() if it decides to extend beyond summary report. >> >> Of course, this is under assumption that final_report() is called during >> JVM normal shutdown (good state) and error_report() is by error handler >> (bad state). > > Ok, thanks for the explanation. The change looks fine. > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From manc at google.com Fri Nov 10 23:11:41 2017 From: manc at google.com (Man Cao) Date: Fri, 10 Nov 2017 15:11:41 -0800 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> <79bcec31-20d1-1aef-b9a6-1134676aa289@oracle.com> Message-ID: Hi Yasumasa, Yes, that sounds good. I haven't submitted any code to OpenJDK yet. This is a good starter patch to get me familiar with the process. -Man On Fri, Nov 10, 2017 at 12:15 AM, Yasumasa Suenaga wrote: > Hi Man, > > 2017-11-10 11:31 GMT+09:00 Man Cao : > > Hi Yasumasa, > > > > I'm looking at if this patch can fix the original issue with hsperfdata > > counters sun_gc_metaspace_maxCapacity and > > sun_gc_compressedclassspace_maxCapacity being too large when user has > set > > -XX:MaxMetaspaceSize=100m or some value less than 1GB. However, it does > not, > > these counters still reports around 1GB if MaxMetaspaceSize is less than > > 1GB. > > > > The reason is that the code added in this patch to > > Metaspace::ergo_initialize() is AFTER the two lines: > > > > CompressedClassSpaceSize = > > align_size_down_bounded(CompressedClassSpaceSize, _reserve_alignment); > > set_compressed_class_space_size(CompressedClassSpaceSize); > > > > So set_compressed_class_space_size() still gets the old value of > > CompressedClassSpaceSize that is too large, making the reserved size of > > metaspace too large. > > > > Should these two lines be moved after the code added in this patch? > > I agree with you. > Can you send review request for it? According to OCA signatories [1], > Google signs OCA, so I think you can contribute it to OpenJDK. > > I can file this to JBS, and create webrev and changest. But I cannot > push it to jdk/hs repo. So you need to find a sponsor. > I'm a reviewer of OpenJDK. So I can +1 to your change when you send > review request. > > > Thanks, > > Yasumasa > > > [1] http://www.oracle.com/technetwork/community/oca-486395.html > > > > Thanks, > > -Man > > > > On Tue, Oct 17, 2017 at 12:46 PM, wrote: > >> > >> > >> Hi Yasumasa, > >> This looks good. I can sponsor this for you. > >> thanks, > >> Coleen > >> > >> > >> On 10/12/17 1:16 AM, Yasumasa Suenaga wrote: > >>> > >>> Thanks David! > >>> > >>> I've fixed: > >>> > >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ > >>> > >>> > >>> Yasumasa > >>> > >>> > >>> 2017-10-12 13:55 GMT+09:00 David Holmes : > >>>> > >>>> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: > >>>>> > >>>>> Hi David, > >>>>> > >>>>>> You may have time to fix these in place before anyone else sees the > >>>>>> webrev > >>>>> > >>>>> > >>>>> I've fixed that: > >>>>> > >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ > >>>>> > >>>>> > >>>>> I checked raw text of your email, so I believe the indent is correct. > >>>>> If I have mistake(s), please tell me. > >>>> > >>>> > >>>> Sorry they are off. Basic rules: > >>>> > >>>> var = > >>>> some long value; > >>>> > >>>> indent by 4. > >>>> > >>>> foo(arg1, arg2, > >>>> arg3, arg4); > >>>> > >>>> align the args on the two lines as shown. > >>>> > >>>> Hope that is clearer. > >>>> > >>>> Thanks, > >>>> David > >>>> > >>>> > >>>>> Thanks, > >>>>> > >>>>> Yasumasa > >>>>> > >>>>> > >>>>> 2017-10-12 13:22 GMT+09:00 David Holmes : > >>>>>> > >>>>>> Hi Yasumasa, > >>>>>> > >>>>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: > >>>>>>> > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> This change was tried to push via JPRT, but it failed because > >>>>>>> > >>>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/ > MaxMetaspaceSizeTest.java > >>>>>>> was failed. > >>>>>>> Also I heard the comment that the changes in arguments.cpp which > set > >>>>>>> ergonomic value to CompressedClassSpaceSize should be moved to > >>>>>>> Metaspace::ergo_initialize(). > >>>>>>> > >>>>>>> I uploaded new webrev. Could you review again? > >>>>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and > >>>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. > >>>>>>> > >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ > >>>>>> > >>>>>> > >>>>>> > >>>>>> I'll let Coleen cover the technical details, I'll just pick up a few > >>>>>> style > >>>>>> nits: > >>>>>> > >>>>>> 3328 /* Initial virtual space size will be calculated at > >>>>>> global_initialize() */ > >>>>>> > >>>>>> Use // comment > >>>>>> > >>>>>> 3329 size_t min_metaspace_sz = > >>>>>> 3330 VIRTUALSPACEMULTIPLIER * > >>>>>> InitialBootClassLoaderMetaspaceSize; > >>>>>> > >>>>>> ^ should indent only to here > >>>>>> > >>>>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, > >>>>>> 3337 MaxMetaspaceSize - > >>>>>> min_metaspace_sz); > >>>>>> ^ should indent only to here > >>>>>> > >>>>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, > >>>>>> 3342 min_metaspace_sz); > >>>>>> > >>>>>> ^ should indent only to here > >>>>>> > >>>>>> You may have time to fix these in place before anyone else sees the > >>>>>> webrev > >>>>>> :) > >>>>>> > >>>>>> Thanks, > >>>>>> David > >>>>>> ----- > >>>>>> > >>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Yasumasa > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2017-10-10 21:02 GMT+09:00 : > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it > >>>>>>>> only > >>>>>>>> needs > >>>>>>>> one reviewer, unless there were more? Please send me the result > of > >>>>>>>> hg > >>>>>>>> export. > >>>>>>>> thanks, > >>>>>>>> Coleen > >>>>>>>> > >>>>>>>> > >>>>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Hi Coleen, > >>>>>>>>> > >>>>>>>>>>> I will sponsor this for you. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>>> Hi, this looks reasonable but I would prefer that the code in > >>>>>>>>>>> arguments.cpp call something in metaspace.cpp, like > >>>>>>>>>>> check_metaspace_sizes() > >>>>>>>>>>> so that you don't have to tell arguments what the MediumChunk > >>>>>>>>>>> size > >>>>>>>>>>> is. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I added new function calculate_min_metaspace_size() in > >>>>>>>>> metaspace.cpp > >>>>>>>>> to get minimum metaspace size, and uses return value from this > >>>>>>>>> function in metaspace initialization and metaspace size check. > >>>>>>>>> > >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev. > 05/ > >>>>>>>>> > >>>>>>>>> Could you review again? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Yasumasa > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Thanks for you both updating and sponsoring the webrev! > >>>>>>>>>> We plan to back-port it to our JDK9 once it is submitted. > >>>>>>>>>> > >>>>>>>>>> -Man > >>>>>>>>>> > >>>>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Hi, this looks reasonable but I would prefer that the code in > >>>>>>>>>>> arguments.cpp call something in metaspace.cpp, like > >>>>>>>>>>> check_metaspace_sizes() > >>>>>>>>>>> so that you don't have to tell arguments what the MediumChunk > >>>>>>>>>>> size > >>>>>>>>>>> is. > >>>>>>>>>>> > >>>>>>>>>>> I will sponsor this for you. > >>>>>>>>>>> > >>>>>>>>>>> thanks, > >>>>>>>>>>> Coleen > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Hi all, > >>>>>>>>>>>> > >>>>>>>>>>>> I added a testcase for this issue in new webrev: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> > >>>>>>>>>>>> Yasumasa > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga > >>>>>>>>>>>> : > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. > >>>>>>>>>>>>> Could you review it? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I cannot access JPRT. So I need a sponsor. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yasumasa > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Good to know that the patch is not blocked due to breaking > >>>>>>>>>>>>>> some > >>>>>>>>>>>>>> internal > >>>>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. > >>>>>>>>>>>>>> Is it possible to push it to P4? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -Man > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I think the reason is that this bug is a P5. The > compressed > >>>>>>>>>>>>>>>> class > >>>>>>>>>>>>>>>> space > >>>>>>>>>>>>>>>> belongs to the runtime code, so you might get more > traction > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>> on the > >>>>>>>>>>>>>>>> hotspot-runtime-dev list. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs > >>>>>>>>>>>>>>> after > >>>>>>>>>>>>>>> repos > >>>>>>>>>>>>>>> are opened. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yasumasa > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Man, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Yasumasa, Stefan, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been > pending > >>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> 2+ > >>>>>>>>>>>>>>>>> years? This patch could really save us from some annoying > >>>>>>>>>>>>>>>>> issues > >>>>>>>>>>>>>>>>> since we > >>>>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I think the reason is that this bug is a P5. The > compressed > >>>>>>>>>>>>>>>> class > >>>>>>>>>>>>>>>> space > >>>>>>>>>>>>>>>> belongs to the runtime code, so you might get more > traction > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>> on the > >>>>>>>>>>>>>>>> hotspot-runtime-dev list. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> StefanK > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -Man > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao < > manc at google.com > >>>>>>>>>>>>>>>>> > wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I wonder if there is any recent update on the > >>>>>>>>>>>>>>>>> patch > >>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> JDK-8087291. > >>>>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? > >>>>>>>>>>>>>>>>> Except > >>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> its > >>>>>>>>>>>>>>>>> low > >>>>>>>>>>>>>>>>> priority (P5), > >>>>>>>>>>>>>>>>> is there any complication that prevents this > patch > >>>>>>>>>>>>>>>>> getting > >>>>>>>>>>>>>>>>> approved > >>>>>>>>>>>>>>>>> (for example, some JVM logic requires > >>>>>>>>>>>>>>>>> CompressedClassSpaceSize > >>>>>>>>>>>>>>>>> to be > >>>>>>>>>>>>>>>>> 1GB by default)? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We > >>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> encountered > >>>>>>>>>>>>>>>>> annoying issues that the hsperfdata counter > >>>>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting > >>>>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets > >>>>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log > shows > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> confusing 1GB > >>>>>>>>>>>>>>>>> memory reserved by metaspace, > >>>>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root > >>>>>>>>>>>>>>>>> cause > >>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> these > >>>>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not > >>>>>>>>>>>>>>>>> automatically > >>>>>>>>>>>>>>>>> capped > >>>>>>>>>>>>>>>>> by MaxMetaspaceSize > >>>>>>>>>>>>>>>>> during VM initialization, and this patch seems > fix > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> root > >>>>>>>>>>>>>>>>> cause. > >>>>>>>>>>>>>>>>> (I'm aware that even after this patch, the > >>>>>>>>>>>>>>>>> reserved > >>>>>>>>>>>>>>>>> size > >>>>>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, > >>>>>>>>>>>>>>>>> but it is better than the current situation.) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> Man > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thank you for your comment! > >>>>>>>>>>>>>>>>> > Try running a debug JVM with your patch > >>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>> line. > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 > -version > >>>>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev. > 02/ > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ysuenaga/JDK-8087291/webrev.02/> > >>>>>>>>>>>>>>>>> It works on fastdebug VM. > >>>>>>>>>>>>>>>>> Please review again. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> Yasumasa > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: > >>>>>>>>>>>>>>>>> > Yasumasa, > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > Try running a debug JVM with your patch > >>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>> line. > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 > -version > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > On a linux system I get this when I build > >>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>> patch. > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 > -version > >>>>>>>>>>>>>>>>> >> # To suppress the following error > report, > >>>>>>>>>>>>>>>>> specify > >>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> argument > >>>>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: > >>>>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 > >>>>>>>>>>>>>>>>> >> # > >>>>>>>>>>>>>>>>> >> # A fatal error has been detected by the > >>>>>>>>>>>>>>>>> Java > >>>>>>>>>>>>>>>>> Runtime > >>>>>>>>>>>>>>>>> Environment: > >>>>>>>>>>>>>>>>> >> # > >>>>>>>>>>>>>>>>> >> # Internal Error > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/ > memory/metaspace.cpp:2324), > >>>>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 > >>>>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > > >>>>>>>>>>>>>>>>> ClassMediumChunk) > >>>>>>>>>>>>>>>>> failed: Not a > >>>>>>>>>>>>>>>>> >> humongous chunk > >>>>>>>>>>>>>>>>> >> # > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > Jon > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> >> I want to continue to discuss about > >>>>>>>>>>>>>>>>> CompressedClassSpace and > >>>>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> http://mail.openjdk.java.net/ > pipermail/hotspot-gc-dev/2015-June/013873.html > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> pipermail/hotspot-gc-dev/2015-June/013873.html> > >>>>>>>>>>>>>>>>> >>>> Should I resize > CompressedClassSpaceSize > >>>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> show > >>>>>>>>>>>>>>>>> error message? > >>>>>>>>>>>>>>>>> >>> If you add slightly better heuristics > for > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> setup > >>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example > >>>>>>>>>>>>>>>>> lowering > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> CompressedClassSpaceSize when > MaxMetaspaceSize > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> set, > >>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>> might be less likely that you'll hit the > >>>>>>>>>>>>>>>>> OutOfMemoryError > >>>>>>>>>>>>>>>>> when > >>>>>>>>>>>>>>>>> the system is set up with strict overcommit > >>>>>>>>>>>>>>>>> settings. > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> I've uploaded new webrev: > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev. > 01/ > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ysuenaga/JDK-8087291/webrev.01/> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, > >>>>>>>>>>>>>>>>> CompressedClassSpaceSize, and > >>>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize > in > >>>>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). > >>>>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize > is > >>>>>>>>>>>>>>>>> greater > >>>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>> MaxMetaspaceSize, > >>>>>>>>>>>>>>>>> >> VM will fail with error message. > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize > will > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>> set > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> MaxMetaspaceSize > >>>>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not > set > >>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>> Metaspace::ergo_initialize(). > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> Thanks, > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> Yasumasa > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >> > > > From daniel.daugherty at oracle.com Mon Nov 13 17:30:57 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 13 Nov 2017 12:30:57 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Message-ID: <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> Greetings, I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. Here are the updated webrevs: Here's the mq comment for the change: ? Rebase to 2017.10.25 PIT snapshot. Here is the full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ And here is the delta webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ I ran the above bits throught Mach5 tier[1-5] testing over the holiday weekend. Didn't see any issues in a quick look. Going to take a closer look today. We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: > Greetings, > > Resolving one of the code review comments (from both Stefan K and Coleen) > on jdk10-04-full required quite a few changes so it is being done as a > standalone patch and corresponding webrevs: > > Here's the mq comment for the change: > > ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use > ??? JavaThreadIteratorWithHandle. > > Here is the full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ > > And here is the delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> We have a (eXtra Large) fix for the following bug: >> >> 8167108 inconsistent handling of SR_lock can lead to crashes >> https://bugs.openjdk.java.net/browse/JDK-8167108 >> >> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >> Hazard Pointers to manage JavaThread lifecycle. >> >> Here's a PDF for the internal wiki that we've been using to describe >> and track the work on this project: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >> >> >> Dan has noticed that the indenting is wrong in some of the code quotes >> in the PDF that are not present in the internal wiki. We don't have a >> solution for that problem yet. >> >> Here's the webrev for current JDK10 version of this fix: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >> >> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >> testing, additional stress testing on Dan's Solaris X64 server, and >> additional testing on Erik and Robbin's machines. >> >> We welcome comments, suggestions and feedback. >> >> Daniel Daugherty >> Erik Osterlund >> Robbin Ehn >> > > From kevin.walls at oracle.com Mon Nov 13 18:08:27 2017 From: kevin.walls at oracle.com (Kevin Walls) Date: Mon, 13 Nov 2017 18:08:27 +0000 Subject: RFR(8u): 8038636, 8055008, 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links ...and 8057570. Message-ID: Hi, I'd like to get a hotspot review of these backports from 9 to 8u.? This is mainly runtime territory but some of the history is from the compiler side. webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ The one we need is: 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links jbs: https://bugs.openjdk.java.net/browse/JDK-8156137 9 changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/882e8cda60b3 The 9 changeset is short, just changing klass.cpp, but not possible in 8 without additional work, so... 1. 8038636: speculative traps break when classes are redefined https://bugs.openjdk.java.net/browse/JDK-8038636 9 changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/a7784ddacbef 9 review thread: http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2014-April/013948.html (in 9 since April 2014) That one imports cleanly into 8u. 2. With that imported, we need: 8055008:Clean up code that saves the previous versions of redefined classes https://bugs.openjdk.java.net/browse/JDK-8055008 9 changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/e3fb51ae8d7d This is the change to stop using PreviousVersionNode and get an InstanceKlass* from previous_versions(). My webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ ...is 8055008 and 8156137 in 8u. The klass.cpp change here is 8156137.? The rest is 8055008, so I can commit them separately. 8156137 doesn't have its own test, but I have found if you get this area wrong, the test runtime/RedefineObject crashes. Notes: For 8055008 there was a change in src/share/vm/classfile/classLoaderData.cpp which is already in 8u. src/share/vm/classfile/metadataOnStackMark.cpp: +MetadataOnStackMark::MetadataOnStackMark(bool has_redefined_a_class) { The change is to stop using JvmtiExport::has_redefined_a_class() here which is already in 8u at this point, but the param is already there with a different name, so renamed as per 8055008. universe.cpp: had a change in 8055008 but following that there was: 8057570: RedefineClasses() tests fail assert(((Metadata*)obj)->is_valid()) failed: obj is valid https://bugs.openjdk.java.net/browse/JDK-8057570 9 changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/479ed4234a9d ..this puts back a few lines in nmethod.cpp which 8055008 removed: + // Call function Method*, not embedded in these other places. + if (_method != NULL) f(_method); ..and in universe.cpp: + // Make the dependent methods not entrant (in VM_Deoptimize they are made zombies) + CodeCache::make_marked_nmethods_not_entrant(); ..and removes: CodeCache::make_marked_nmethods_zombies(); My webrev takes these into account. Then there are 8038636's two follow-on bugs which I'd like to follow-up in a separate thread. 8039960 is a test change 8040237: nsk/jvmti/RetransformClasses/retransform001 crashed the VM on all platforms when run with with -server -Xcomp https://bugs.openjdk.java.net/browse/JDK-8040237 Thanks! Kevin From kishor.kharbas at intel.com Mon Nov 13 19:40:56 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Mon, 13 Nov 2017 19:40:56 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> Message-ID: Greetings, I have an updated webrev to remove compilation warning on Windows 32-bit. http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15_to_14/ Sorry missed this earlier. I request for a review on this update. Thanks Kishor From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Friday, November 3, 2017 4:07 PM To: Kharbas, Kishor ; Thomas Schatzl ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: Hi Sangheon, Here is link to the updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ Looks good to me. Thanks, Sangheon Thanks Kishor From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Friday, November 3, 2017 2:38 PM To: Kharbas, Kishor ; Thomas Schatzl ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: Thanks a lot! Link to updated webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ Thank you for fixing all. Looks good to me except below. Could you update the copyright format in TestAllocateHeapAt.java? 2 * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. Thanks, Sangheon @Sangheon: Please let me know if you see any corrections needed. -Kishor -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Friday, November 3, 2017 7:31 AM To: Kharbas, Kishor ; sangheon.kim ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime- dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi, On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: Hi Sangheon, Thanks for the review and comments. Here is an updated webrev- http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 In addition to your suggested corrections, I added code to set Linux core dump filter ensuring Heap is dumped correctly when this feature is used. This is follow-up to Jini George?s comment (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- allocation-on-alternative-memory-devices-td300109.html#a300450). Some minor nits: - os_posix.cpp:300: please move the else next to the brace - arguments.cpp:4624: please add a space between "if" and the bracket I do not need to see a new webrev for these changes. Looks good. Thanks, Thomas From thomas.schatzl at oracle.com Mon Nov 13 20:40:10 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 13 Nov 2017 21:40:10 +0100 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> Message-ID: <1510605610.7132.3.camel@oracle.com> Hi Kishor, On Mon, 2017-11-13 at 19:40 +0000, Kharbas, Kishor wrote: > Greetings, > ? > I have an updated webrev to remove compilation warning on Windows 32- > bit. > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15/ > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15_to_14/ > ? > Sorry missed this earlier. I request for a review on this update. looks good. The other changes from webrev.13 on also look good. Thanks, Thomas > ? > Thanks > Kishor > ? > From: sangheon.kim [mailto:sangheon.kim at oracle.com]? > Sent: Friday, November 3, 2017 4:07 PM > To: Kharbas, Kishor ; Thomas Schatzl s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8190308: Supporting heap allocation on > alternative memory devices and CSR review > ? > Hi Kishor, > > > On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: > Hi Sangheon, > ? > Here is link to the updated webrev- > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ > Looks good to me. > > Thanks, > Sangheon > > > > ? > Thanks > Kishor > ? > From: sangheon.kim [mailto:sangheon.kim at oracle.com]? > Sent: Friday, November 3, 2017 2:38 PM > To: Kharbas, Kishor ; Thomas Schatzl s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8190308: Supporting heap allocation on > alternative memory devices and CSR review > ? > Hi Kishor, > > On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: > Thanks a lot! > ? > Link to updated webrevs - > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ > Thank you for fixing all. > Looks good to me except below. > > Could you update the copyright format in TestAllocateHeapAt.java? > 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights > reserved. > - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. > All rights reserved. > > Thanks, > Sangheon > > > > > ? > ? > @Sangheon: Please let me know if you see any corrections needed. > ? > -Kishor > ? > -----Original Message----- > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > Sent: Friday, November 3, 2017 7:31 AM > To: Kharbas, Kishor ; sangheon.kim > ; 'hotspot-gc-dev at openjdk.java.net' > ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8190308: Supporting heap allocation on > alternative > memory devices and CSR review > ? > Hi, > ? > On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: > Hi Sangheon, > ? > Thanks for the review and comments. Here is an updated webrev- > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 > http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 > ? > In addition to your suggested corrections, I added code to set Linux > core dump filter ensuring Heap is dumped correctly when this feature > is used. This is follow-up to Jini George?s comment > (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- > allocation-on-alternative-memory-devices-td300109.html#a300450). > ? > Some minor nits: > ? > ?- os_posix.cpp:300: please move the else next to the brace > ?- arguments.cpp:4624: please add a space between "if" and the > bracket > ? > I do not need to see a new webrev for these changes. Looks good. > ? > Thanks, > ? Thomas > ? > ? > ? From sangheon.kim at oracle.com Mon Nov 13 20:41:09 2017 From: sangheon.kim at oracle.com (sangheon.kim) Date: Mon, 13 Nov 2017 12:41:09 -0800 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <1510605610.7132.3.camel@oracle.com> References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> <1510605610.7132.3.camel@oracle.com> Message-ID: <3e6131cd-bac3-c8db-971f-297bb13b087d@oracle.com> Hi Kishor, On 11/13/2017 12:40 PM, Thomas Schatzl wrote: > Hi Kishor, > > On Mon, 2017-11-13 at 19:40 +0000, Kharbas, Kishor wrote: >> Greetings, >> >> I have an updated webrev to remove compilation warning on Windows 32- >> bit. >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15_to_14/ >> >> Sorry missed this earlier. I request for a review on this update. > looks good. The other changes from webrev.13 on also look good. +1 Thanks, Sangheon > > Thanks, > Thomas > >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 4:07 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> >> On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Here is link to the updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ >> Looks good to me. >> >> Thanks, >> Sangheon >> >> >> >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 2:38 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: >> Thanks a lot! >> >> Link to updated webrevs - >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ >> Thank you for fixing all. >> Looks good to me except below. >> >> Could you update the copyright format in TestAllocateHeapAt.java? >> 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights >> reserved. >> - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. >> All rights reserved. >> >> Thanks, >> Sangheon >> >> >> >> >> >> >> @Sangheon: Please let me know if you see any corrections needed. >> >> -Kishor >> >> -----Original Message----- >> From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] >> Sent: Friday, November 3, 2017 7:31 AM >> To: Kharbas, Kishor ; sangheon.kim >> ; 'hotspot-gc-dev at openjdk.java.net' >> ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative >> memory devices and CSR review >> >> Hi, >> >> On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Thanks for the review and comments. Here is an updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 >> >> In addition to your suggested corrections, I added code to set Linux >> core dump filter ensuring Heap is dumped correctly when this feature >> is used. This is follow-up to Jini George?s comment >> (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- >> allocation-on-alternative-memory-devices-td300109.html#a300450). >> >> Some minor nits: >> >> ?- os_posix.cpp:300: please move the else next to the brace >> ?- arguments.cpp:4624: please add a space between "if" and the >> bracket >> >> I do not need to see a new webrev for these changes. Looks good. >> >> Thanks, >> ? Thomas >> >> >> From kishor.kharbas at intel.com Mon Nov 13 21:33:21 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Mon, 13 Nov 2017 21:33:21 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: <3e6131cd-bac3-c8db-971f-297bb13b087d@oracle.com> References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> <1510605610.7132.3.camel@oracle.com> <3e6131cd-bac3-c8db-971f-297bb13b087d@oracle.com> Message-ID: Thanks Sangheon and Thomas! -----Original Message----- From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Monday, November 13, 2017 12:41 PM To: Thomas Schatzl ; Kharbas, Kishor ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/13/2017 12:40 PM, Thomas Schatzl wrote: > Hi Kishor, > > On Mon, 2017-11-13 at 19:40 +0000, Kharbas, Kishor wrote: >> Greetings, >> >> I have an updated webrev to remove compilation warning on Windows 32- >> bit. >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15_to_14/ >> >> Sorry missed this earlier. I request for a review on this update. > looks good. The other changes from webrev.13 on also look good. +1 Thanks, Sangheon > > Thanks, > Thomas > >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 4:07 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> >> On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Here is link to the updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ >> Looks good to me. >> >> Thanks, >> Sangheon >> >> >> >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 2:38 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: >> Thanks a lot! >> >> Link to updated webrevs - >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ >> Thank you for fixing all. >> Looks good to me except below. >> >> Could you update the copyright format in TestAllocateHeapAt.java? >> 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights >> reserved. >> - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. >> All rights reserved. >> >> Thanks, >> Sangheon >> >> >> >> >> >> >> @Sangheon: Please let me know if you see any corrections needed. >> >> -Kishor >> >> -----Original Message----- >> From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] >> Sent: Friday, November 3, 2017 7:31 AM >> To: Kharbas, Kishor ; sangheon.kim >> ; 'hotspot-gc-dev at openjdk.java.net' >> ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi, >> >> On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Thanks for the review and comments. Here is an updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 >> >> In addition to your suggested corrections, I added code to set Linux >> core dump filter ensuring Heap is dumped correctly when this feature >> is used. This is follow-up to Jini George?s comment >> (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- >> allocation-on-alternative-memory-devices-td300109.html#a300450). >> >> Some minor nits: >> >> ?- os_posix.cpp:300: please move the else next to the brace >> ?- arguments.cpp:4624: please add a space between "if" and the >> bracket >> >> I do not need to see a new webrev for these changes. Looks good. >> >> Thanks, >> ? Thomas >> >> >> From david.holmes at oracle.com Tue Nov 14 10:39:24 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 14 Nov 2017 20:39:24 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM Message-ID: Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ There are three parts to this which made it somewhat more complicated than I had envisaged: 1. Add the flag and disable the guards. This is relatively straight-forward: void JavaThread::create_stack_guard_pages() { if (!os::uses_stack_guard_pages() || _stack_guard_state != stack_guard_unused || (DisablePrimordialThreadGuardPages && os::is_primordial_thread())) { log_info(os, thread)("Stack guard page creation for thread " UINTX_FORMAT " disabled", os::current_thread_id()); return; with a tweak in JavaThread::stack_guards_enabled as well. 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() We have three platforms that already implement this functionality: Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do anything different for the primordial thread, and we don't have a way to detect the primordial thread - so is_primordial_thread() just returns false. I also tidied up some comments regarding terminology. 3. Thomas noticed that we unconditionally subtract 2*page_size() from the rlimit stack size, without checking it was bigger than 2*page_size() to start with. I was unsure how to tackle this. It's no good leaving stack_size at zero so I opted to ensure we would have at least one page left. Of course in such cases we would hit the bug in libc (if it still exists, which seems unlikely but time consuming to establish). Testing: - Tier 1 (JPRT) testing with a modified launcher that uses the primordial thread - with guard pages enabled - with guard pages disabled - Tier 1 (JPRT) normal JDK build (which should be unaffected by this change) The testing was primarily to ensure that disabling the stack guards on the primordial thread wasn't totally broken. A number of tests fail: - setting -Xss32m fails for the primordial thread when guards are enabled and the rlimit is 8m - tests that would hit StackOverflowError crash the VM when guards are disabled (as you would expect). - runtime/execstack/Testexecstack.java crashes when the guards are disabled Thanks, David From thomas.stuefe at gmail.com Tue Nov 14 11:07:14 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 14 Nov 2017 12:07:14 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: Hi David, Looks mostly fine, nice cleanup. Minor nits: os_bsd.cpp, os_windows.cpp: I do not like the fact that we offer a generic function but do not implement it. The comment +// Unlike Linux we don't try to handle the initial process +// thread in any special way - so just report 'false' assumes a given usage of this function, but the function looks general purpose and may be used in different contexts in the future. I'd either invest the work to implement it on bsd/windows or at least add a comment that the function should not be relied upon on these platforms. Or, something like this: static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) BSD_ONLY({return false;}) ----- src/hotspot/os/linux/os_linux.cpp + // But ensure we don't underflow the stack size - allow 1 page spare + if (stack_size >= (size_t)(3 * page_size())) stack_size -= 2 * page_size(); Could you please add {} ? ---- Sidenote (not touched by your thread): 3045 uintptr_t stack_extent = (uintptr_t) os::Linux::initial_thread_stack_bottom(); 3046 unsigned char vec[1]; 3047 3048 if (mincore((address)stack_extent, os::vm_page_size(), vec) == -1) { I feel a tiny bit queasy about this coding. mincore(2) is defined to work with pages as returned by sysconf(_SC_PAGESIZE). Should the implementation of os::vm_page_size() ever be something different than sysconf(_SC_PAGESIZE), this may overflow vec. ----- src/hotspot/share/runtime/os.hpp s/Some platforms have to special-case handling/Some platforms need special-case handling ---- Kind Regards, Thomas On Tue, Nov 14, 2017 at 11:39 AM, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > > There are three parts to this which made it somewhat more complicated than > I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > void JavaThread::create_stack_guard_pages() { > if (!os::uses_stack_guard_pages() || > _stack_guard_state != stack_guard_unused || > (DisablePrimordialThreadGuardPages && os::is_primordial_thread())) > { > log_info(os, thread)("Stack guard page creation for thread " > UINTX_FORMAT " disabled", > os::current_thread_id()); > return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: Linux, > Solaris and AIX. For BSD/OSX and Windows we don't attempt to do anything > different for the primordial thread, and we don't have a way to detect the > primordial thread - so is_primordial_thread() just returns false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() from the > rlimit stack size, without checking it was bigger than 2*page_size() to > start with. I was unsure how to tackle this. It's no good leaving > stack_size at zero so I opted to ensure we would have at least one page > left. Of course in such cases we would hit the bug in libc (if it still > exists, which seems unlikely but time consuming to establish). > > Testing: > - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > - with guard pages enabled > - with guard pages disabled > - Tier 1 (JPRT) normal JDK build (which should be unaffected by this > change) > > The testing was primarily to ensure that disabling the stack guards on the > primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are enabled > and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards are > disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are > disabled > > Thanks, > David > From david.holmes at oracle.com Tue Nov 14 12:10:02 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 14 Nov 2017 22:10:02 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: Hi Thomas, Thanks for taking a look so quickly. On 14/11/2017 9:07 PM, Thomas St?fe wrote: > Hi David, > > Looks mostly fine, nice cleanup. > > Minor nits: > > os_bsd.cpp, os_windows.cpp: > > I do not like the fact that we offer a generic function but do not > implement it. The comment > > +// Unlike Linux we don't try to handle the initial process > +// thread in any special way - so just report 'false' > > assumes a given usage of this function, but the function looks general > purpose and may be used in different contexts in the future. I'd either > invest the work to implement it on bsd/windows or at least add a comment > that the function should not be relied upon on these platforms. I went hunting for a way to implement it and could not find anything. I hadn't realized initially that we didn't already have something on each platform. Given the narrow problem this is trying to solve it's already exceeded the time budget I can allocate to it. I was tempted to drop back to having a Linux only solution ... but three platforms can support it, while three platforms can't. > Or, something like this: > static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) > BSD_ONLY({return false;}) I can do that. I'll see if anyone else has any comments on this. > ----- > > src/hotspot/os/linux/os_linux.cpp > > +? //? ?But ensure we don't underflow the stack size - allow 1 page spare > +? if (stack_size >= (size_t)(3 * page_size())) > ? ?stack_size -= 2 * page_size(); > > Could you please add {} ? Fixed. > ---- > > Sidenote (not touched by your thread): > > 3045? ? ?uintptr_t stack_extent = (uintptr_t) > os::Linux::initial_thread_stack_bottom(); > 3046? ? ?unsigned char vec[1]; > 3047 > 3048? ? ?if (mincore((address)stack_extent, os::vm_page_size(), vec) == > -1) { > > I feel a tiny bit queasy about this coding. mincore(2) is defined to > work with pages as returned by sysconf(_SC_PAGESIZE). Should the > implementation of os::vm_page_size() ever be something different than > sysconf(_SC_PAGESIZE), this may overflow vec. Is there any reason to think it would ever be something different? On Linux os::vm_page_size() returns os::Linux::page_size() which is set by: Linux::set_page_size(sysconf(_SC_PAGESIZE)); > ----- > > src/hotspot/share/runtime/os.hpp > > s/Some platforms have to special-case handling/Some platforms need > special-case handling Fixed. Thanks, David > ---- > > Kind Regards, Thomas > > > On Tue, Nov 14, 2017 at 11:39 AM, David Holmes > wrote: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > > webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > > > There are three parts to this which made it somewhat more > complicated than I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > ?void JavaThread::create_stack_guard_pages() { > ? ?if (!os::uses_stack_guard_pages() || > ? ? ? ?_stack_guard_state != stack_guard_unused || > ? ? ? ?(DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ? ? ? ?log_info(os, thread)("Stack guard page creation for thread " > ? ? ? ? ? ? ? ? ? ? ? ? ? ? UINTX_FORMAT " disabled", > os::current_thread_id()); > ? ? ?return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: > Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to > do anything different for the primordial thread, and we don't have a > way to detect the primordial thread - so is_primordial_thread() just > returns false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() > from the rlimit stack size, without checking it was bigger than > 2*page_size() to start with. I was unsure how to tackle this. It's > no good leaving stack_size at zero so I opted to ensure we would > have at least one page left. Of course in such cases we would hit > the bug in libc (if it still exists, which seems unlikely but time > consuming to establish). > > Testing: > ? - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > ? ? - with guard pages enabled > ? ? - with guard pages disabled > ? - Tier 1 (JPRT) normal JDK build (which should be unaffected by > this change) > > The testing was primarily to ensure that disabling the stack guards > on the primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are > enabled and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards > are disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are > disabled > > Thanks, > David > > From thomas.stuefe at gmail.com Tue Nov 14 12:52:45 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 14 Nov 2017 13:52:45 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: Hi David, On Tue, Nov 14, 2017 at 1:10 PM, David Holmes wrote: > Hi Thomas, > > Thanks for taking a look so quickly. > > On 14/11/2017 9:07 PM, Thomas St?fe wrote: > >> Hi David, >> >> Looks mostly fine, nice cleanup. >> >> Minor nits: >> >> os_bsd.cpp, os_windows.cpp: >> >> I do not like the fact that we offer a generic function but do not >> implement it. The comment >> >> +// Unlike Linux we don't try to handle the initial process >> +// thread in any special way - so just report 'false' >> >> assumes a given usage of this function, but the function looks general >> purpose and may be used in different contexts in the future. I'd either >> invest the work to implement it on bsd/windows or at least add a comment >> that the function should not be relied upon on these platforms. >> > > I went hunting for a way to implement it and could not find anything. I > hadn't realized initially that we didn't already have something on each > platform. Given the narrow problem this is trying to solve it's already > exceeded the time budget I can allocate to it. I was tempted to drop back > to having a Linux only solution ... but three platforms can support it, > while three platforms can't. > > Or, something like this: >> static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) >> BSD_ONLY({return false;}) >> > > I can do that. I'll see if anyone else has any comments on this. > > ----- >> >> src/hotspot/os/linux/os_linux.cpp >> >> + // But ensure we don't underflow the stack size - allow 1 page spare >> + if (stack_size >= (size_t)(3 * page_size())) >> stack_size -= 2 * page_size(); >> >> Could you please add {} ? >> > > Fixed. > > ---- >> >> Sidenote (not touched by your thread): >> >> 3045 uintptr_t stack_extent = (uintptr_t) >> os::Linux::initial_thread_stack_bottom(); >> 3046 unsigned char vec[1]; >> 3047 >> 3048 if (mincore((address)stack_extent, os::vm_page_size(), vec) == >> -1) { >> >> I feel a tiny bit queasy about this coding. mincore(2) is defined to work >> with pages as returned by sysconf(_SC_PAGESIZE). Should the implementation >> of os::vm_page_size() ever be something different than >> sysconf(_SC_PAGESIZE), this may overflow vec. >> > > Is there any reason to think it would ever be something different? On > Linux os::vm_page_size() returns os::Linux::page_size() which is set by: > > Linux::set_page_size(sysconf(_SC_PAGESIZE)); > > We had https://bugs.openjdk.java.net/browse/JDK-8186665 and since then I'm wary around mincore(2).. Basically, on AIX we decide - for complicated reasons - to make os::vm_page_size() a larger size than the system page size. This triggered the overflow with mincore. But you are right, the implementation of os::vm_page_size() is is not very probable to change. One could add a sentinel like this: unsigned char vec[2]; vec[1] = 'X'; mincore((address)stack_extent, os::vm_page_size(), vec); assert(vec[1] == 'X'); but maybe it is not worth the trouble. > ----- >> >> src/hotspot/share/runtime/os.hpp >> >> s/Some platforms have to special-case handling/Some platforms need >> special-case handling >> > > Fixed. > > Thanks, > David > > Thanks, Thomas > ---- >> >> Kind Regards, Thomas >> >> >> >> On Tue, Nov 14, 2017 at 11:39 AM, David Holmes > > wrote: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> >> webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ >> >> >> >> There are three parts to this which made it somewhat more >> complicated than I had envisaged: >> >> 1. Add the flag and disable the guards. >> >> This is relatively straight-forward: >> >> void JavaThread::create_stack_guard_pages() { >> if (!os::uses_stack_guard_pages() || >> _stack_guard_state != stack_guard_unused || >> (DisablePrimordialThreadGuardPages && >> os::is_primordial_thread())) { >> log_info(os, thread)("Stack guard page creation for thread " >> UINTX_FORMAT " disabled", >> os::current_thread_id()); >> return; >> >> with a tweak in JavaThread::stack_guards_enabled as well. >> >> 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() >> >> We have three platforms that already implement this functionality: >> Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to >> do anything different for the primordial thread, and we don't have a >> way to detect the primordial thread - so is_primordial_thread() just >> returns false. >> >> I also tidied up some comments regarding terminology. >> >> 3. Thomas noticed that we unconditionally subtract 2*page_size() >> from the rlimit stack size, without checking it was bigger than >> 2*page_size() to start with. I was unsure how to tackle this. It's >> no good leaving stack_size at zero so I opted to ensure we would >> have at least one page left. Of course in such cases we would hit >> the bug in libc (if it still exists, which seems unlikely but time >> consuming to establish). >> >> Testing: >> - Tier 1 (JPRT) testing with a modified launcher that uses the >> primordial thread >> - with guard pages enabled >> - with guard pages disabled >> - Tier 1 (JPRT) normal JDK build (which should be unaffected by >> this change) >> >> The testing was primarily to ensure that disabling the stack guards >> on the primordial thread wasn't totally broken. A number of tests >> fail: >> - setting -Xss32m fails for the primordial thread when guards are >> enabled and the rlimit is 8m >> - tests that would hit StackOverflowError crash the VM when guards >> are disabled (as you would expect). >> - runtime/execstack/Testexecstack.java crashes when the guards are >> disabled >> >> Thanks, >> David >> >> >> From martin.doerr at sap.com Tue Nov 14 14:55:22 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Tue, 14 Nov 2017 14:55:22 +0000 Subject: RFR(S): 8191212: AIX: Build and polling page allocation broken after 8189941 Message-ID: <3002aa602a0e415bbc1c9d9963bbfece@sap.com> Hi, the VM doesn't build any more on AIX after 8189941. In addition, the attempt_reserve_memory_at can't be used for polling pages because it returns non-protectable memory. Please review this small fix: http://cr.openjdk.java.net/~mdoerr/8191212_aix_polling_page/webrev.00/ Best regards, Martin From thomas.stuefe at gmail.com Tue Nov 14 14:59:52 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 14 Nov 2017 15:59:52 +0100 Subject: RFR(S): 8191212: AIX: Build and polling page allocation broken after 8189941 In-Reply-To: <3002aa602a0e415bbc1c9d9963bbfece@sap.com> References: <3002aa602a0e415bbc1c9d9963bbfece@sap.com> Message-ID: Looks good. Thanks for fixing. Gruss Thomas On Tue, Nov 14, 2017 at 3:55 PM, Doerr, Martin wrote: > Hi, > > the VM doesn't build any more on AIX after 8189941. In addition, the > attempt_reserve_memory_at can't be used for polling pages because it > returns non-protectable memory. > > Please review this small fix: > http://cr.openjdk.java.net/~mdoerr/8191212_aix_polling_page/webrev.00/ > > Best regards, > Martin > > > From martin.doerr at sap.com Tue Nov 14 15:46:40 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Tue, 14 Nov 2017 15:46:40 +0000 Subject: RFR(S): 8191212: AIX: Build and polling page allocation broken after 8189941 In-Reply-To: References: <3002aa602a0e415bbc1c9d9963bbfece@sap.com> Message-ID: <576ac62da5ff4d1785577a5a044b2e59@sap.com> Hi Thomas, thanks for the review. Pushed. Best regards, Martin From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] Sent: Dienstag, 14. November 2017 16:00 To: Doerr, Martin Cc: hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(S): 8191212: AIX: Build and polling page allocation broken after 8189941 Looks good. Thanks for fixing. Gruss Thomas On Tue, Nov 14, 2017 at 3:55 PM, Doerr, Martin > wrote: Hi, the VM doesn't build any more on AIX after 8189941. In addition, the attempt_reserve_memory_at can't be used for polling pages because it returns non-protectable memory. Please review this small fix: http://cr.openjdk.java.net/~mdoerr/8191212_aix_polling_page/webrev.00/ Best regards, Martin From dean.long at oracle.com Tue Nov 14 18:27:52 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Tue, 14 Nov 2017 10:27:52 -0800 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: References: Message-ID: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> Adding runtime alias... comments inlined below. On 11/13/17 9:10 PM, jamsheed wrote: > Hi, request for review, jbs: > https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: > http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) > changes equivalent to JDK-4454115 is done for windows. It looks like "nm" can be uninitialized if "in_java" is false. > 2) added guard to multiple value access sites, Unsafe_CopyMemory0, > Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. Can you narrow the scope of the unsafe access using something like GuardUnsafeAccess, instead of marking the whole function as doing unsafe access? There are some risks with trying to? abort a copy function. First, won't we get multiple exceptions until we finally hit the end of the range?? What if the bad range is very large? Second, what if the loop is using auto-increment instructions? Skipping to the next instruction would mean we loop forever if the increment never happens. I think if we are going to safely abort copy functions then we need to register them as a kind of CodeBlob that has a special abort entry point or exception handler we can redirect to, or maybe pop the whole frame and return. Is there really a problem with these copy functions?? I'm wondering why Mikael did not identify these as a problem in 8154592. > 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed > thread->thread_state() == _thread_in_vm checks from signal handler How about adding a check for _thread_in_native instead of removing the check entirely? dl > Best regards, Jamsheed > From harold.seigel at oracle.com Tue Nov 14 18:53:14 2017 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 14 Nov 2017 13:53:14 -0500 Subject: RFR 8191132: assert condition should not be in quotes Message-ID: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> Hi, Please review this simple fix for a bad assert condition. Open Webrev: http://javaweb.us.oracle.com/~hseigel/webrev/bug_8191132/webrev/index.html JBS Bug:? https://bugs.openjdk.java.net/browse/JDK-8191132 The fix was tested with JPRT, JCK Lang and VM, and Mach5 tier1 - tier5 tests to make sure the new assert did not trigger. Thanks, Harold From harold.seigel at oracle.com Tue Nov 14 18:55:33 2017 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 14 Nov 2017 13:55:33 -0500 Subject: RFR 8191132: assert condition should not be in quotes In-Reply-To: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> References: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> Message-ID: <7a97f1ec-027b-dcc0-5fe9-e70b5010bce1@oracle.com> Here's the correct webrev: http://cr.openjdk.java.net/~hseigel/bug_8191132/webrev/ On 11/14/2017 1:53 PM, harold seigel wrote: > Hi, > > Please review this simple fix for a bad assert condition. > > Open Webrev: > http://javaweb.us.oracle.com/~hseigel/webrev/bug_8191132/webrev/index.html > > JBS Bug:? https://bugs.openjdk.java.net/browse/JDK-8191132 > > The fix was tested with JPRT, JCK Lang and VM, and Mach5 tier1 - tier5 > tests to make sure the new assert did not trigger. > > Thanks, Harold > From coleen.phillimore at oracle.com Tue Nov 14 19:00:37 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 14 Nov 2017 14:00:37 -0500 Subject: RFR 8191132: assert condition should not be in quotes In-Reply-To: <7a97f1ec-027b-dcc0-5fe9-e70b5010bce1@oracle.com> References: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> <7a97f1ec-027b-dcc0-5fe9-e70b5010bce1@oracle.com> Message-ID: <734fe2ea-3cc7-8640-2438-7ac2023a2ae9@oracle.com> Looks good!? I think this is the definition of trivial change, and reassuring that you ran all the tests to see that the real assert never fires. Thanks, Coleen On 11/14/17 1:55 PM, harold seigel wrote: > Here's the correct webrev: > http://cr.openjdk.java.net/~hseigel/bug_8191132/webrev/ > > > On 11/14/2017 1:53 PM, harold seigel wrote: >> Hi, >> >> Please review this simple fix for a bad assert condition. >> >> Open Webrev: >> http://javaweb.us.oracle.com/~hseigel/webrev/bug_8191132/webrev/index.html >> >> JBS Bug:? https://bugs.openjdk.java.net/browse/JDK-8191132 >> >> The fix was tested with JPRT, JCK Lang and VM, and Mach5 tier1 - >> tier5 tests to make sure the new assert did not trigger. >> >> Thanks, Harold >> > From harold.seigel at oracle.com Tue Nov 14 19:03:29 2017 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 14 Nov 2017 14:03:29 -0500 Subject: RFR 8191132: assert condition should not be in quotes In-Reply-To: <734fe2ea-3cc7-8640-2438-7ac2023a2ae9@oracle.com> References: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> <7a97f1ec-027b-dcc0-5fe9-e70b5010bce1@oracle.com> <734fe2ea-3cc7-8640-2438-7ac2023a2ae9@oracle.com> Message-ID: <248150b5-2673-9b4d-7c78-d5062ca62459@oracle.com> Thanks Coleen! Harold On 11/14/2017 2:00 PM, coleen.phillimore at oracle.com wrote: > Looks good!? I think this is the definition of trivial change, and > reassuring that you ran all the tests to see that the real assert > never fires. > > Thanks, > Coleen > > On 11/14/17 1:55 PM, harold seigel wrote: >> Here's the correct webrev: >> http://cr.openjdk.java.net/~hseigel/bug_8191132/webrev/ >> >> >> On 11/14/2017 1:53 PM, harold seigel wrote: >>> Hi, >>> >>> Please review this simple fix for a bad assert condition. >>> >>> Open Webrev: >>> http://javaweb.us.oracle.com/~hseigel/webrev/bug_8191132/webrev/index.html >>> >>> JBS Bug:? https://bugs.openjdk.java.net/browse/JDK-8191132 >>> >>> The fix was tested with JPRT, JCK Lang and VM, and Mach5 tier1 - >>> tier5 tests to make sure the new assert did not trigger. >>> >>> Thanks, Harold >>> >> > From jamsheed.c.m at oracle.com Tue Nov 14 19:31:32 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 01:01:32 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> Message-ID: <4e401096-2d7c-317d-4ff4-28c59984d1fb@oracle.com> Hi Dean, Thank you for the feedback, On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: > Adding runtime alias... > > comments inlined below. > > > On 11/13/17 9:10 PM, jamsheed wrote: > >> Hi, request for review, jbs: >> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >> changes equivalent to JDK-4454115 is done for windows. > > It looks like "nm" can be uninitialized if "in_java" is false. that was a miss, thank you for pointing that. > >> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. > > Can you narrow the scope of the unsafe access using something like > GuardUnsafeAccess, instead of marking the whole function as doing > unsafe access? can do that, but i don't find an issue with putting guard for the whole function. > > There are some risks with trying to? abort a copy function. > > First, won't we get multiple exceptions until we finally hit the end > of the range?? What if the bad range is very large? we may get multiple exception, and we may reach safepoint a bit late, this should be case in compiled code too, where we use intrinsics > > Second, what if the loop is using auto-increment instructions? > Skipping to the next instruction would mean we loop forever if the > increment never happens. we move to next instruction, so will exit loop, no backward branching right ? > > I think if we are going to safely abort copy functions then we need to > register them as a kind of CodeBlob that has a special abort entry > point or exception handler we can redirect to, or maybe pop the whole > frame and return. > > Is there really a problem with these copy functions?? I'm wondering > why Mikael did not identify these as a problem in 8154592. > >> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >> thread->thread_state() == _thread_in_vm checks from signal handler > > How about adding a check for _thread_in_native instead of removing the > check entirely? ok i will add that. Best regards, Jamsheed > > dl > >> Best regards, Jamsheed >> > From jamsheed.c.m at oracle.com Tue Nov 14 19:32:02 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 01:02:02 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> Message-ID: <200961d9-f62d-ab2c-f0fe-eb2892c52440@oracle.com> Hi Dean, Thank you for the feedback, On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: > Adding runtime alias... > > comments inlined below. > > > On 11/13/17 9:10 PM, jamsheed wrote: > >> Hi, request for review, jbs: >> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >> changes equivalent to JDK-4454115 is done for windows. > > It looks like "nm" can be uninitialized if "in_java" is false. that was a miss, thank you for pointing that. > >> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. > > Can you narrow the scope of the unsafe access using something like > GuardUnsafeAccess, instead of marking the whole function as doing > unsafe access? can do that, but i don't find an issue with putting guard for the whole function. > > There are some risks with trying to? abort a copy function. > > First, won't we get multiple exceptions until we finally hit the end > of the range?? What if the bad range is very large? we may get multiple exception, and we may reach safepoint a bit late, this should be case in compiled code too, where we use intrinsics > > Second, what if the loop is using auto-increment instructions? > Skipping to the next instruction would mean we loop forever if the > increment never happens. we move to next instruction, so will exit loop, no backward branching right ? > > I think if we are going to safely abort copy functions then we need to > register them as a kind of CodeBlob that has a special abort entry > point or exception handler we can redirect to, or maybe pop the whole > frame and return. > > Is there really a problem with these copy functions?? I'm wondering > why Mikael did not identify these as a problem in 8154592. > >> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >> thread->thread_state() == _thread_in_vm checks from signal handler > > How about adding a check for _thread_in_native instead of removing the > check entirely? ok i will add that. Best regards, Jamsheed > > dl > >> Best regards, Jamsheed >> > From jamsheed.c.m at oracle.com Tue Nov 14 19:58:11 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 01:28:11 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <200961d9-f62d-ab2c-f0fe-eb2892c52440@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <200961d9-f62d-ab2c-f0fe-eb2892c52440@oracle.com> Message-ID: <8feaa96f-50cb-daa3-3e4d-6f4b0b255c3c@oracle.com> sorry, we do loop, even for autoincrement case only if both from and to are invalid we may loop for ever, i didn't assume that case Best regards, Jamsheed On Wednesday 15 November 2017 01:02 AM, jamsheed wrote: > Second, what if the loop is using auto-increment instructions? > Skipping to the next instruction would mean we loop forever if the > increment never happens. From jamsheed.c.m at oracle.com Tue Nov 14 20:23:18 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 01:53:18 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <8feaa96f-50cb-daa3-3e4d-6f4b0b255c3c@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <200961d9-f62d-ab2c-f0fe-eb2892c52440@oracle.com> <8feaa96f-50cb-daa3-3e4d-6f4b0b255c3c@oracle.com> Message-ID: only problem i see in autoincrement is in? unsafe.setMemory(base, size, (byte) 0), may be i will skip this case. and maintain same implementation for copy and copyswap, as i assume autoincrement? for count wouldn't happen in single instruction for these two case. Best regards, Jamsheed On Wednesday 15 November 2017 01:28 AM, jamsheed wrote: > sorry, we do loop, even for autoincrement case only if both from and > to are invalid we may loop for ever, i didn't assume that case > > Best regards, > > Jamsheed > > > On Wednesday 15 November 2017 01:02 AM, jamsheed wrote: >> Second, what if the loop is using auto-increment instructions? >> Skipping to the next instruction would mean we loop forever if the >> increment never happens. > From jamsheed.c.m at oracle.com Tue Nov 14 21:11:08 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 02:41:08 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> Message-ID: <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> Hi Dean, revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ i agree to the comment that it is potentially unsafe to assume the implementation, and count can be in autoincrement mode. so with this bug i would like to deal with only the single value access error handling. revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ Best regards, Jamsheed On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: > Adding runtime alias... > > comments inlined below. > > > On 11/13/17 9:10 PM, jamsheed wrote: > >> Hi, request for review, jbs: >> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >> changes equivalent to JDK-4454115 is done for windows. > > It looks like "nm" can be uninitialized if "in_java" is false. > >> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. > > Can you narrow the scope of the unsafe access using something like > GuardUnsafeAccess, instead of marking the whole function as doing > unsafe access? > > There are some risks with trying to? abort a copy function. > > First, won't we get multiple exceptions until we finally hit the end > of the range?? What if the bad range is very large? > > Second, what if the loop is using auto-increment instructions? > Skipping to the next instruction would mean we loop forever if the > increment never happens. > > I think if we are going to safely abort copy functions then we need to > register them as a kind of CodeBlob that has a special abort entry > point or exception handler we can redirect to, or maybe pop the whole > frame and return. > > Is there really a problem with these copy functions?? I'm wondering > why Mikael did not identify these as a problem in 8154592. > >> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >> thread->thread_state() == _thread_in_vm checks from signal handler > > How about adding a check for _thread_in_native instead of removing the > check entirely? > > dl > >> Best regards, Jamsheed >> > From david.holmes at oracle.com Tue Nov 14 21:15:36 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 15 Nov 2017 07:15:36 +1000 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> Message-ID: On 15/11/2017 4:27 AM, dean.long at oracle.com wrote: > Adding runtime alias... Thanks Dean! > comments inlined below. > > > On 11/13/17 9:10 PM, jamsheed wrote: > >> Hi, request for review, jbs: >> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >> changes equivalent to JDK-4454115 is done for windows. > > It looks like "nm" can be uninitialized if "in_java" is false. > >> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. > > Can you narrow the scope of the unsafe access using something like > GuardUnsafeAccess, instead of marking the whole function as doing unsafe > access? I tend to agree with this suggestion. The unsafe region should be as narrow as possible. > There are some risks with trying to? abort a copy function. > > First, won't we get multiple exceptions until we finally hit the end of > the range?? What if the bad range is very large? > > Second, what if the loop is using auto-increment instructions? Skipping > to the next instruction would mean we loop forever if the increment > never happens. > > I think if we are going to safely abort copy functions then we need to > register them as a kind of CodeBlob that has a special abort entry point > or exception handler we can redirect to, or maybe pop the whole frame > and return. Jamsheed and I has some discussions on this on IM last week. I also flagged the multiple exceptions as potentially problematic. I'm not sure how things will behave if we trigger hundreds of faults in succession - nor how long it will take. Aborting the whole loop, or frame, seems preferable but I don't know how you would do that. > Is there really a problem with these copy functions?? I'm wondering why > Mikael did not identify these as a problem in 8154592. Good question. :) It seems quite obvious to me that if a single load/store needs protection then bulk load/store would also need protection. And Jamsheeds tests confirmed that. Cheers, David >> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >> thread->thread_state() == _thread_in_vm checks from signal handler > > How about adding a check for _thread_in_native instead of removing the > check entirely? > > dl > >> Best regards, Jamsheed >> > From david.holmes at oracle.com Tue Nov 14 21:18:18 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 15 Nov 2017 07:18:18 +1000 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> Message-ID: On 15/11/2017 7:11 AM, jamsheed wrote: > Hi Dean, > > revised webrev: > > http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > i agree to the comment that it is potentially unsafe to assume the > implementation, and count can be in autoincrement mode. so with this bug > i would like to deal with only the single value access error handling. > > revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ Okay so now this just brings to windows what currently exists on the other platforms - right? Seems reasonable for now. Longer term we should look at the bulk operation problem. Thanks, David > Best regards, > > Jamsheed > > > > On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: >> Adding runtime alias... >> >> comments inlined below. >> >> >> On 11/13/17 9:10 PM, jamsheed wrote: >> >>> Hi, request for review, jbs: >>> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >>> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >>> changes equivalent to JDK-4454115 is done for windows. >> >> It looks like "nm" can be uninitialized if "in_java" is false. >> >>> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >>> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. >> >> Can you narrow the scope of the unsafe access using something like >> GuardUnsafeAccess, instead of marking the whole function as doing >> unsafe access? >> >> There are some risks with trying to? abort a copy function. >> >> First, won't we get multiple exceptions until we finally hit the end >> of the range?? What if the bad range is very large? >> >> Second, what if the loop is using auto-increment instructions? >> Skipping to the next instruction would mean we loop forever if the >> increment never happens. >> >> I think if we are going to safely abort copy functions then we need to >> register them as a kind of CodeBlob that has a special abort entry >> point or exception handler we can redirect to, or maybe pop the whole >> frame and return. >> >> Is there really a problem with these copy functions?? I'm wondering >> why Mikael did not identify these as a problem in 8154592. >> >>> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >>> thread->thread_state() == _thread_in_vm checks from signal handler >> >> How about adding a check for _thread_in_native instead of removing the >> check entirely? >> >> dl >> >>> Best regards, Jamsheed >>> >> > From jamsheed.c.m at oracle.com Tue Nov 14 21:23:55 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 02:53:55 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> Message-ID: Hi David, Hope you too agree here. Best regards, Jamsheed On Wednesday 15 November 2017 02:41 AM, jamsheed wrote: > Hi Dean, > > revised webrev: > > http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > i agree to the comment that it is potentially unsafe to assume the > implementation, and count can be in autoincrement mode. so with this > bug i would like to deal with only > > the single value access error handling. > > revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > Best regards, > > Jamsheed > > > > On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: >> Adding runtime alias... >> >> comments inlined below. >> >> >> On 11/13/17 9:10 PM, jamsheed wrote: >> >>> Hi, request for review, jbs: >>> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >>> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >>> changes equivalent to JDK-4454115 is done for windows. >> >> It looks like "nm" can be uninitialized if "in_java" is false. >> >>> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >>> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. >> >> Can you narrow the scope of the unsafe access using something like >> GuardUnsafeAccess, instead of marking the whole function as doing >> unsafe access? >> >> There are some risks with trying to? abort a copy function. >> >> First, won't we get multiple exceptions until we finally hit the end >> of the range?? What if the bad range is very large? >> >> Second, what if the loop is using auto-increment instructions? >> Skipping to the next instruction would mean we loop forever if the >> increment never happens. >> >> I think if we are going to safely abort copy functions then we need >> to register them as a kind of CodeBlob that has a special abort entry >> point or exception handler we can redirect to, or maybe pop the whole >> frame and return. >> >> Is there really a problem with these copy functions?? I'm wondering >> why Mikael did not identify these as a problem in 8154592. >> >>> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >>> thread->thread_state() == _thread_in_vm checks from signal handler >> >> How about adding a check for _thread_in_native instead of removing >> the check entirely? >> >> dl >> >>> Best regards, Jamsheed >>> >> > From jamsheed.c.m at oracle.com Tue Nov 14 21:26:53 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 02:56:53 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> Message-ID: <41491f82-94fd-57d8-d6da-677443762f32@oracle.com> Hi David, On Wednesday 15 November 2017 02:48 AM, David Holmes wrote: > On 15/11/2017 7:11 AM, jamsheed wrote: >> Hi Dean, >> >> revised webrev: >> >> http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ >> >> i agree to the comment that it is potentially unsafe to assume the >> implementation, and count can be in autoincrement mode. so with this >> bug i would like to deal with only the single value access error >> handling. >> >> revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > Okay so now this just brings to windows what currently exists on the > other platforms - right? Yes, > > Seems reasonable for now. Longer term we should look at the bulk > operation problem. Sure, Best regards, Jamsheed > > Thanks, > David > >> Best regards, >> >> Jamsheed >> >> >> >> On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: >>> Adding runtime alias... >>> >>> comments inlined below. >>> >>> >>> On 11/13/17 9:10 PM, jamsheed wrote: >>> >>>> Hi, request for review, jbs: >>>> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >>>> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >>>> changes equivalent to JDK-4454115 is done for windows. >>> >>> It looks like "nm" can be uninitialized if "in_java" is false. >>> >>>> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >>>> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. >>> >>> Can you narrow the scope of the unsafe access using something like >>> GuardUnsafeAccess, instead of marking the whole function as doing >>> unsafe access? >>> >>> There are some risks with trying to? abort a copy function. >>> >>> First, won't we get multiple exceptions until we finally hit the end >>> of the range?? What if the bad range is very large? >>> >>> Second, what if the loop is using auto-increment instructions? >>> Skipping to the next instruction would mean we loop forever if the >>> increment never happens. >>> >>> I think if we are going to safely abort copy functions then we need >>> to register them as a kind of CodeBlob that has a special abort >>> entry point or exception handler we can redirect to, or maybe pop >>> the whole frame and return. >>> >>> Is there really a problem with these copy functions?? I'm wondering >>> why Mikael did not identify these as a problem in 8154592. >>> >>>> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >>>> thread->thread_state() == _thread_in_vm checks from signal handler >>> >>> How about adding a check for _thread_in_native instead of removing >>> the check entirely? >>> >>> dl >>> >>>> Best regards, Jamsheed >>>> >>> >> From daniel.daugherty at oracle.com Tue Nov 14 21:48:42 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 14 Nov 2017 16:48:42 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> Message-ID: <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> Greetings, I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. (Yes, we're playing chase-the-repo...) Here is the updated full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ Unlike the previous rebase, there were no changes required to the open code to get the rebased bits to build so there is no delta webrev. These bits have been run through JPRT and I've submitted the usual Mach5 tier[1-5] test run... We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: > Greetings, > > I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. > > Here are the updated webrevs: > > Here's the mq comment for the change: > > ? Rebase to 2017.10.25 PIT snapshot. > > Here is the full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ > > And here is the delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ > > I ran the above bits throught Mach5 tier[1-5] testing over the holiday > weekend. Didn't see any issues in a quick look. Going to take a closer > look today. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Resolving one of the code review comments (from both Stefan K and >> Coleen) >> on jdk10-04-full required quite a few changes so it is being done as a >> standalone patch and corresponding webrevs: >> >> Here's the mq comment for the change: >> >> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >> ??? JavaThreadIteratorWithHandle. >> >> Here is the full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >> >> And here is the delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> We have a (eXtra Large) fix for the following bug: >>> >>> 8167108 inconsistent handling of SR_lock can lead to crashes >>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>> >>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>> Hazard Pointers to manage JavaThread lifecycle. >>> >>> Here's a PDF for the internal wiki that we've been using to describe >>> and track the work on this project: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>> >>> >>> Dan has noticed that the indenting is wrong in some of the code quotes >>> in the PDF that are not present in the internal wiki. We don't have a >>> solution for that problem yet. >>> >>> Here's the webrev for current JDK10 version of this fix: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>> >>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>> testing, additional stress testing on Dan's Solaris X64 server, and >>> additional testing on Erik and Robbin's machines. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Daniel Daugherty >>> Erik Osterlund >>> Robbin Ehn >>> >> >> > > From jiangli.zhou at oracle.com Tue Nov 14 22:20:28 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 14 Nov 2017 14:20:28 -0800 Subject: RFR 8191132: assert condition should not be in quotes In-Reply-To: <734fe2ea-3cc7-8640-2438-7ac2023a2ae9@oracle.com> References: <9646714c-b0ac-3aa3-fe1f-2848b067659a@oracle.com> <7a97f1ec-027b-dcc0-5fe9-e70b5010bce1@oracle.com> <734fe2ea-3cc7-8640-2438-7ac2023a2ae9@oracle.com> Message-ID: <5ACDD547-8015-4491-84E3-702102199D0F@oracle.com> +1 Jiangli > On Nov 14, 2017, at 11:00 AM, coleen.phillimore at oracle.com wrote: > > Looks good! I think this is the definition of trivial change, and reassuring that you ran all the tests to see that the real assert never fires. > > Thanks, > Coleen > > On 11/14/17 1:55 PM, harold seigel wrote: >> Here's the correct webrev: http://cr.openjdk.java.net/~hseigel/bug_8191132/webrev/ >> >> >> On 11/14/2017 1:53 PM, harold seigel wrote: >>> Hi, >>> >>> Please review this simple fix for a bad assert condition. >>> >>> Open Webrev: http://javaweb.us.oracle.com/~hseigel/webrev/bug_8191132/webrev/index.html >>> >>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8191132 >>> >>> The fix was tested with JPRT, JCK Lang and VM, and Mach5 tier1 - tier5 tests to make sure the new assert did not trigger. >>> >>> Thanks, Harold >>> >> > From dean.long at oracle.com Tue Nov 14 22:38:34 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Tue, 14 Nov 2017 14:38:34 -0800 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> Message-ID: <43f85d9a-89fd-0ecf-2c18-c8c10ce466f5@oracle.com> This version looks good.? If you wanted you could also do the following: CompiledMethod* nm = NULL; JavaThread* thread = (JavaThread*)t; if (in_java) { CodeBlob* cb= CodeCache::find_blob_unsafe(pc); because "cb" is only used in that inner block. dl On 11/14/17 1:11 PM, jamsheed wrote: > Hi Dean, > > revised webrev: > > http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > i agree to the comment that it is potentially unsafe to assume the > implementation, and count can be in autoincrement mode. so with this > bug i would like to deal with only > > the single value access error handling. > > revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ > > Best regards, > > Jamsheed > > > > On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: >> Adding runtime alias... >> >> comments inlined below. >> >> >> On 11/13/17 9:10 PM, jamsheed wrote: >> >>> Hi, request for review, jbs: >>> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >>> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >>> changes equivalent to JDK-4454115 is done for windows. >> >> It looks like "nm" can be uninitialized if "in_java" is false. >> >>> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >>> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. >> >> Can you narrow the scope of the unsafe access using something like >> GuardUnsafeAccess, instead of marking the whole function as doing >> unsafe access? >> >> There are some risks with trying to? abort a copy function. >> >> First, won't we get multiple exceptions until we finally hit the end >> of the range?? What if the bad range is very large? >> >> Second, what if the loop is using auto-increment instructions? >> Skipping to the next instruction would mean we loop forever if the >> increment never happens. >> >> I think if we are going to safely abort copy functions then we need >> to register them as a kind of CodeBlob that has a special abort entry >> point or exception handler we can redirect to, or maybe pop the whole >> frame and return. >> >> Is there really a problem with these copy functions?? I'm wondering >> why Mikael did not identify these as a problem in 8154592. >> >>> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >>> thread->thread_state() == _thread_in_vm checks from signal handler >> >> How about adding a check for _thread_in_native instead of removing >> the check entirely? >> >> dl >> >>> Best regards, Jamsheed >>> >> > From igor.ignatyev at oracle.com Wed Nov 15 00:00:58 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Tue, 14 Nov 2017 16:00:58 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A04C2D5.8090404@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> <5A04C2D5.8090404@oracle.com> Message-ID: <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> Hi Misha, a few more comments: - are you using CPUSetsReader::listToString(boolean abc, List list, int maxCount) method? I wasn't able to find any usages of it. - CPUSetsReader uses Asserts.assertTrue(parts.length == 2) instead of Asserts.assertEQ(parts.length, 2) - instead of using Stream::collect in CPUSetsReader::readFromProcStatus, you can use Stream::findFirst(). so it'll be smth like > + try (Stream stream = Files.lines(Paths.get(path))) { > + o = stream > + .filter(line -> line.contains(setType)) > + .findFirst(); > + } catch (IOException e) { > + return null; > + } > + > + if (!o.isPresent()) { > + return null; > + } > + > + String[] parts = o.get().replaceAll("\\s","").split(":"); - you don't need to box int into Integer in TestCPUSets::checkResult L#108: > Asserts.assertEquals(new Integer(parts.length), new Integer(2)); - IIRC, in order to properly use whitebox API, you have to have both sun.hotspot.WhiteBox and sun.hotspot.WhiteBox$WhiteBoxPermission in BCP, but '@run driver ClassFileInstaller -jar whitebox.jar sun.hotspot.WhiteBox', which is used in all your tests, will put only s.h.WhiteBox in whitebox.jar - how do you plan to use :hotspot_containers test group? can't you use the test directory instead? the rest looks good to me. Thanks, -- Igor > On Nov 9, 2017, at 1:04 PM, Mikhailo Seledtsov wrote: > > OK, will remove it. > > Misha > > On 11/9/17, 12:38 PM, Bob Vandette wrote: >> >> Since that path is not guaranteed to work, I?d remove CGROUP_CPUSET_PATH and the function. >> >> Bob. >> >>> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov > wrote: >>> >>> I am not using it. I thought it may be helpful if we go back to that method of extracting cpuset values for some reason. >>> On the other hand, dead code is not a good thing. So I am a bit undecided. >>> I will delete it if you think it is better to do so. >>> >>> Misha >>> >>> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>>> >>>> Do you still need this function? I don?t see it being used. >>>> >>>> 84 public static String read(String fileName) { >>>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>>> 86 try { >>>> 87 return readLineFromFile(path); >>>> 88 } catch (Exception e) { >>>> 89 System.err.println("Exception reading " + path); >>>> 90 System.err.println("Exception: " + e); >>>> 91 } >>>> 92 >>>> 93 return null; >>>> 94 } >>>> >>>> Bob. >>>> >>>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov > wrote: >>>>> >>>>> Here is a differential webrev: >>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>>> >>>>> Misha >>>>> >>>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>>> Here is an update webrev: >>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>>> I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. >>>>>> Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). >>>>>> >>>>>> Summary for updated webrev.02: >>>>>> - implemented review feedback from Bob and Igor >>>>>> - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver >>>>>> - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently >>>>>> - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers >>>>>> (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) >>>>>> >>>>>> My next steps: >>>>>> - rebase to new jdk tree >>>>>> - update/merge >>>>>> - retest >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> >>>>>> >>>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>>> Hi Igor, >>>>>>> >>>>>>> Thank you for reviewing the code. >>>>>>> >>>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>>> Hi Misha, >>>>>>>> >>>>>>>> I have several comments: >>>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>>> Fixed >>>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>>> Thank you. I will try it out. >>>>>>>> 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >>>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); >>>>>>>> or >>>>>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >>>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >>>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>>> OK. I will unwind these expressions, and make sure indentation looks good. >>>>>>>> 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 >>>>>>> OK >>>>>>>> 5. Test*: there is no need to have parentheses in '@requires (docker.support)' >>>>>>> OK >>>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it >>>>>>> Will fix. >>>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format >>>>>>> OK. >>>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? >>>>>>> Oops. Missed it. Will add code to check it. >>>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness >>>>>>> Common.run() already checks for .shouldHaveExitValue(0) after running the container. >>>>>>> >>>>>>> >>>>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -- Igor >>>>>>>> >>>>>>>>> On Nov 1, 2017, at 8:11 PM, mikhailo> wrote: >>>>>>>>> >>>>>>>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>>> Webrev: >>>>>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>>> Testing: >>>>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>>>> Ran the developed tests via jtreg >>>>>>>>> Pass >>>>>>>>> >>>>>>>>> 2. Automated testing system - run these tests >>>>>>>>> In progress >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>> >>>> >> From jamsheed.c.m at oracle.com Wed Nov 15 03:30:22 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 15 Nov 2017 09:00:22 +0530 Subject: [10]RFR: 6415680: (bf) MappedByteBuffer.get() can provoke crash with EXCEPTION_IN_PAGE_ERROR In-Reply-To: <43f85d9a-89fd-0ecf-2c18-c8c10ce466f5@oracle.com> References: <05a52b8b-8a14-83c1-f1b3-5798de2d8af8@oracle.com> <2c16f1ff-57fc-9fed-2290-7e3a99c6a741@oracle.com> <43f85d9a-89fd-0ecf-2c18-c8c10ce466f5@oracle.com> Message-ID: <12e5f50a-d59d-e6ce-b04e-d3c34298658c@oracle.com> Thank you, Dean, David for review, created a new CR for tracking bulk access failure graceful handling https://bugs.openjdk.java.net/browse/JDK-8191278 On Wednesday 15 November 2017 04:08 AM, dean.long at oracle.com wrote: > > This version looks good.? If you wanted you could also do the following: > > CompiledMethod* nm = NULL; > JavaThread* thread = (JavaThread*)t; > if (in_java) { > CodeBlob* cb= CodeCache::find_blob_unsafe(pc); > > because "cb" is only used in that inner block. Ok Best regards, Jamsheed > > dl > > On 11/14/17 1:11 PM, jamsheed wrote: >> Hi Dean, >> >> revised webrev: >> >> http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ >> >> i agree to the comment that it is potentially unsafe to assume the >> implementation, and count can be in autoincrement mode. so with this >> bug i would like to deal with only >> >> the single value access error handling. >> >> revised webrev: http://cr.openjdk.java.net/~jcm/6415680/webrev.01/ >> >> Best regards, >> >> Jamsheed >> >> >> >> On Tuesday 14 November 2017 11:57 PM, dean.long at oracle.com wrote: >>> Adding runtime alias... >>> >>> comments inlined below. >>> >>> >>> On 11/13/17 9:10 PM, jamsheed wrote: >>> >>>> Hi, request for review, jbs: >>>> https://bugs.openjdk.java.net/browse/JDK-6415680 webrev: >>>> http://cr.openjdk.java.net/~jcm/6415680/webrev.00/ Description: 1) >>>> changes equivalent to JDK-4454115 is done for windows. >>> >>> It looks like "nm" can be uninitialized if "in_java" is false. >>> >>>> 2) added guard to multiple value access sites, Unsafe_CopyMemory0, >>>> Unsafe_SetMemory0 and Unsafe_CopySwapMemory0. >>> >>> Can you narrow the scope of the unsafe access using something like >>> GuardUnsafeAccess, instead of marking the whole function as doing >>> unsafe access? >>> >>> There are some risks with trying to? abort a copy function. >>> >>> First, won't we get multiple exceptions until we finally hit the end >>> of the range?? What if the bad range is very large? >>> >>> Second, what if the loop is using auto-increment instructions? >>> Skipping to the next instruction would mean we loop forever if the >>> increment never happens. >>> >>> I think if we are going to safely abort copy functions then we need >>> to register them as a kind of CodeBlob that has a special abort >>> entry point or exception handler we can redirect to, or maybe pop >>> the whole frame and return. >>> >>> Is there really a problem with these copy functions?? I'm wondering >>> why Mikael did not identify these as a problem in 8154592. >>> >>>> 3) Unsafe_CopySwapMemory0 is JVM_LEAF so removed >>>> thread->thread_state() == _thread_in_vm checks from signal handler >>> >>> How about adding a check for _thread_in_native instead of removing >>> the check entirely? >>> >>> dl >>> >>>> Best regards, Jamsheed >>>> >>> >> > From yumin.qi at gmail.com Wed Nov 15 17:52:17 2017 From: yumin.qi at gmail.com (yumin qi) Date: Wed, 15 Nov 2017 09:52:17 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> Message-ID: HI, Ioi Only one minor comment, in cdsoffsets.hpp: #ifndef CLOSED_VM_PRIMS_CDSWHITEBOX_HPP Should it be changed to SHARE_PRIMS_CDOFFSETS_HPP to keep it with same style with other head files. Since the hotspot directory changed, I don't know if all head files should change to follow that (changes). Thanks Yumin On Tue, Oct 31, 2017 at 7:43 PM, Ioi Lam wrote: > Hi, > > Here's the new webrev for both the AppCDS implementation and tests. During > internal review of the JEP, we have decided to integrate both > implementation and tests at the same time. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ > > As mentioned before, most of the "diffs" shown in this webrev are the > result of copying the closed source files on top of files of the same name > in the open repo. So in reviewing, instead of focusing on what's "changed", > please focus on the entire content of the new version of each file. > > Testing: I did an OpenJDK linux-x64 build (without Oracle closed sources) > and all the new appcds tests passed. > > Thanks > > - Ioi > > > On 10/30/17 8:52 AM, Ioi Lam wrote: > >> Hi Dmitry, >> >> In the latest JDK 10 repo, is_jrt has been renamed to is_modules_image. >> Please change the code accordingly. >> >> I will post my latest diff soon, with some test cases as well. >> >> Thanks >> >> - Ioi >> >> >> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >> >>> Ioi, >>> >>> I'd tried to apply your patch to latest open JDK10 and >>> the compilation fails with: >>> >>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/s >>> ystemDictionaryShared.cpp:400:16: >>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>> >>> Did I miss something? >>> >>> -Dmitry >>> >>> On 13.10.2017 02:48, Ioi Lam wrote: >>> >>>> Hi, >>>> >>>> Please review this change set. >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>> >>>> This is the first step of implementing the following JEP, which moves >>>> AppCDS from >>>> closed repos into the openjdk repo: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>> >>>> In JDK 9, significant portion of AppCDS code resided in the closed repo. >>>> As part >>>> of the open-sourcing effort of JDK 18.3, we will move the source code >>>> into the >>>> open repo. >>>> >>>> In this changeset, the code is moved verbatim as much as possible. The >>>> intention is >>>> only to relocate the sources, not to changing existing behaviors, and >>>> not >>>> to do any sort of refactoring. >>>> >>>> Most of the "diffs" shown in this webrev are the result of copying the >>>> closed source >>>> files on top of files of the same name in the open repo. So in >>>> reviewing, instead of >>>> focusing on what's "changed", it's better to focus on the entire content >>>> of the new >>>> version of each file. >>>> >>>> The only functional change in this task is that the UseAppCDS flag is >>>> changed from >>>> a "commercial" flag to a regular "product" flag. This is because >>>> "commercial" >>>> flags are not supported by the OpenJDK build. >>>> >>>> Source code refactoring may be desirable, because the old open/closed >>>> source >>>> code structure had introduced some intermediary APIs to connect code >>>> between >>>> the two repos. Such API should be removed in a separate RFE. >>>> >>>> Also, some AppCDS tests are currently in the closed repo. These tests >>>> will be >>>> moved in a separate task. See JDK-8188792 for details. >>>> >>>> All the AppCDS tests (currently still in closed sources) passed with >>>> both Oracle JDK >>>> and OpenJDK. >>>> >>>> Thanks >>>> - Ioi >>>> >>> >>> >> > From daniel.daugherty at oracle.com Wed Nov 15 20:32:13 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 15 Nov 2017 15:32:13 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> Message-ID: <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> Greetings, Robbin rebased the project last night/this morning to merge with Thread Local Handshakes (TLH) and also picked up additional changesets up thru: > Changeset: fa736014cf28 > Author: cjplummer > Date: 2017-11-14 18:08 -0800 > URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 > > 8191049: Add alternate version of pns() that is callable from within hotspot source > Summary: added pns2() to debug.cpp > Reviewed-by: stuefe, gthornbr This is the first time we've rebased the project to bits that are this fresh (< 12 hours old at rebase time). We've done this because we think we're done with this project and are in the final review-change-retest cycle(s)... In other words, we're not planning on making any more major changes for this project... :-) *** Time for code reviewers to chime in on this thread! *** Here is the updated full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ Here's is the delta webrev from the 2017.11.10 rebase: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ Dan has submitted the bits for the usual Mach5 tier[1-5] testing (and the new baseline also)... We're expecting this round to be a little noisier than usual because we did not rebase on a PIT snapshot so the new baseline has not been through Jesper's usual care-and-feeding of integration_blockers, etc. We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: > Greetings, > > I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. > (Yes, we're playing chase-the-repo...) > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ > > Unlike the previous rebase, there were no changes required to the > open code to get the rebased bits to build so there is no delta > webrev. > > These bits have been run through JPRT and I've submitted the usual > Mach5 tier[1-5] test run... > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >> >> Here are the updated webrevs: >> >> Here's the mq comment for the change: >> >> ? Rebase to 2017.10.25 PIT snapshot. >> >> Here is the full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >> >> And here is the delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >> >> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >> weekend. Didn't see any issues in a quick look. Going to take a closer >> look today. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Resolving one of the code review comments (from both Stefan K and >>> Coleen) >>> on jdk10-04-full required quite a few changes so it is being done as a >>> standalone patch and corresponding webrevs: >>> >>> Here's the mq comment for the change: >>> >>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>> ??? JavaThreadIteratorWithHandle. >>> >>> Here is the full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>> >>> And here is the delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> We have a (eXtra Large) fix for the following bug: >>>> >>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>> >>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>> Hazard Pointers to manage JavaThread lifecycle. >>>> >>>> Here's a PDF for the internal wiki that we've been using to describe >>>> and track the work on this project: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>> >>>> >>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>> in the PDF that are not present in the internal wiki. We don't have a >>>> solution for that problem yet. >>>> >>>> Here's the webrev for current JDK10 version of this fix: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>> >>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>> additional testing on Erik and Robbin's machines. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Daniel Daugherty >>>> Erik Osterlund >>>> Robbin Ehn >>>> >>> >>> >> >> > > From mikhailo.seledtsov at oracle.com Wed Nov 15 21:37:22 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 15 Nov 2017 13:37:22 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> <5A04C2D5.8090404@oracle.com> <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> Message-ID: <5A0CB392.7040803@oracle.com> Hi Igor, Bob, Webrev containing latest changes based on feedback from you: Diff from 02: http://cr.openjdk.java.net/~mseledtsov/8189762.02-03/ Full: http://cr.openjdk.java.net/~mseledtsov/8189762.03/ Thank you, Misha On 11/14/17, 4:00 PM, Igor Ignatyev wrote: > Hi Misha, > > a few more comments: > - are you using CPUSetsReader::listToString(boolean abc, List > list, int maxCount) method? I wasn't able to find any usages of it. > - CPUSetsReader uses Asserts.assertTrue(parts.length == 2) instead of > Asserts.assertEQ(parts.length, 2) > - instead of using Stream::collect in > CPUSetsReader::readFromProcStatus, you can use Stream::findFirst(). so > it'll be smth like >> + try (Stream stream = Files.lines(Paths.get(path))) { >> + o = stream >> + .filter(line -> line.contains(setType)) >> + .findFirst(); >> + } catch (IOException e) { >> + return null; >> + } >> + >> + if (!o.isPresent()) { >> + return null; >> + } >> + >> + String[] parts = o.get().replaceAll("\\s","").split(":"); > - you don't need to box int into Integer in TestCPUSets::checkResult > L#108: >> Asserts.assertEquals(new Integer(parts.length), new Integer(2)); > - IIRC, in order to properly use whitebox API, you have to have > both sun.hotspot.WhiteBox and sun.hotspot.WhiteBox$WhiteBoxPermission > in BCP, but '@run driver ClassFileInstaller -jar whitebox.jar > sun.hotspot.WhiteBox', which is used in all your tests, will put only > s.h.WhiteBox in whitebox.jar > - how do you plan to use :hotspot_containers test group? can't you use > the test directory instead? > > the rest looks good to me. > > Thanks, > -- Igor > >> On Nov 9, 2017, at 1:04 PM, Mikhailo Seledtsov >> > > wrote: >> >> OK, will remove it. >> >> Misha >> >> On 11/9/17, 12:38 PM, Bob Vandette wrote: >>> Since that path is not guaranteed to work, I?d remove >>> CGROUP_CPUSET_PATH and the function. >>> >>> Bob. >>> >>>> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov >>>> >>> > wrote: >>>> >>>> I am not using it. I thought it may be helpful if we go back to >>>> that method of extracting cpuset values for some reason. >>>> On the other hand, dead code is not a good thing. So I am a bit >>>> undecided. >>>> I will delete it if you think it is better to do so. >>>> >>>> Misha >>>> >>>> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>>>> Do you still need this function? I don?t see it being used. >>>>> >>>>> 84 public static String read(String fileName) { >>>>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>>>> 86 try { >>>>> 87 return readLineFromFile(path); >>>>> 88 } catch (Exception e) { >>>>> 89 System.err.println("Exception reading " + path); >>>>> 90 System.err.println("Exception: " + e); >>>>> 91 } >>>>> 92 >>>>> 93 return null; >>>>> 94 } >>>>> >>>>> Bob. >>>>> >>>>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov >>>>>> >>>>> > wrote: >>>>>> >>>>>> Here is a differential webrev: >>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>>>> >>>>>> >>>>>> Misha >>>>>> >>>>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>>>> Here is an update webrev: >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>>>> >>>>>>> I also tried to generate a diff between 01 and 02, but could >>>>>>> not. My script does not seem to work any more, or I am missing >>>>>>> something. >>>>>>> Igor, please let me know if you have scripts or command line to >>>>>>> generate incremental webrev (I committed my initial changes >>>>>>> locally, then applied the new changes on top). >>>>>>> >>>>>>> Summary for updated webrev.02: >>>>>>> - implemented review feedback from Bob and Igor >>>>>>> - added @driver to all tests, since the actual testing happens >>>>>>> in the child docker process; the main is just a test driver >>>>>>> - configured docker directory for exclusive access by jtreg >>>>>>> tests, since there are potential and actual issues when running >>>>>>> these tests concurrently >>>>>>> - modified test groups to add a hotspot_docker test group, and >>>>>>> to exclude docker testing from lower tiers >>>>>>> (once the infra is ready, I plan to add this execution >>>>>>> initially at higher tiers; first will run it for a while as >>>>>>> custom runs) >>>>>>> >>>>>>> My next steps: >>>>>>> - rebase to new jdk tree >>>>>>> - update/merge >>>>>>> - retest >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>>>> >>>>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>>>> Hi Igor, >>>>>>>> >>>>>>>> Thank you for reviewing the code. >>>>>>>> >>>>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>>>> Hi Misha, >>>>>>>>> >>>>>>>>> I have several comments: >>>>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect >>>>>>>>> signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>>>> Fixed >>>>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using >>>>>>>>> stream api as >>>>>>>>> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>>>> Thank you. I will try it out. >>>>>>>>> 3. in several files, I have noticed some problems w/ indents >>>>>>>>> which make code quite hard to read, for example >>>>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + >>>>>>>>>> expectedTraceValue); >>>>>>>>> or >>>>>>>>>> 96 DockerRunOptions opts = >>>>>>>>>> Common.newOpts(imageName, "AttemptOOM") >>>>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, >>>>>>>>>> "--memory-swap", dockerMemLimit); >>>>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>>>> OK. I will unwind these expressions, and make sure indentation >>>>>>>> looks good. >>>>>>>>> 4. TestCPUSets.java : it's better to use >>>>>>>>> Assert.assertEquals(parts.length, 2) than >>>>>>>>> Asserts.assertTrue(parts.length == 2) as the former will also >>>>>>>>> print the actual value of parts.length. so you won't need to >>>>>>>>> have L#103 >>>>>>>> OK >>>>>>>>> 5. Test*: there is no need to have parentheses in '@requires >>>>>>>>> (docker.support)' >>>>>>>> OK >>>>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK >>>>>>>>> under test, so you can use '@run driver' to run it >>>>>>>> Will fix. >>>>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol >>>>>>>>> you should use %n in format string. also System.out::printf >>>>>>>>> seems to be more popular in our code than System.out::format >>>>>>>> OK. >>>>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we >>>>>>>>> found lineMarker? >>>>>>>> Oops. Missed it. Will add code to check it. >>>>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to >>>>>>>>> all Common.run calls which aren't expected to fail, e.g. all >>>>>>>>> test* in TestCPUAwareness, testMemorySoftLimit and >>>>>>>>> testMemoryLimit in TestMemoryAwareness >>>>>>>> Common.run() already checks for .shouldHaveExitValue(0) after >>>>>>>> running the container. >>>>>>>> >>>>>>>> >>>>>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> -- Igor >>>>>>>>> >>>>>>>>>> On Nov 1, 2017, at 8:11 PM, >>>>>>>>>> mikhailo>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>> Please review these tests that were developed to test JVM's >>>>>>>>>> container awareness in Docker environment. >>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>>>> Webrev: >>>>>>>>>> Tests: >>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>>>> >>>>>>>>>> WB API: >>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>>>> >>>>>>>>>> Testing: >>>>>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>>>>> Ran the developed tests via jtreg >>>>>>>>>> Pass >>>>>>>>>> >>>>>>>>>> 2. Automated testing system - run these tests >>>>>>>>>> In progress >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Misha >>>>>>>>>> >>>>> >>> > From igor.ignatyev at oracle.com Wed Nov 15 21:49:19 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Wed, 15 Nov 2017 13:49:19 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A0CB392.7040803@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> <5A04C2D5.8090404@oracle.com> <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> <5A0CB392.7040803@oracle.com> Message-ID: looks good to me -- Igor > On Nov 15, 2017, at 1:37 PM, Mikhailo Seledtsov wrote: > > Hi Igor, Bob, > > Webrev containing latest changes based on feedback from you: > Diff from 02: http://cr.openjdk.java.net/~mseledtsov/8189762.02-03/ > Full: http://cr.openjdk.java.net/~mseledtsov/8189762.03/ > > > Thank you, > Misha > > > On 11/14/17, 4:00 PM, Igor Ignatyev wrote: >> >> Hi Misha, >> >> a few more comments: >> - are you using CPUSetsReader::listToString(boolean abc, List list, int maxCount) method? I wasn't able to find any usages of it. >> - CPUSetsReader uses Asserts.assertTrue(parts.length == 2) instead of Asserts.assertEQ(parts.length, 2) >> - instead of using Stream::collect in CPUSetsReader::readFromProcStatus, you can use Stream::findFirst(). so it'll be smth like >>> + try (Stream stream = Files.lines(Paths.get(path))) { >>> + o = stream >>> + .filter(line -> line.contains(setType)) >>> + .findFirst(); >>> + } catch (IOException e) { >>> + return null; >>> + } >>> + >>> + if (!o.isPresent()) { >>> + return null; >>> + } >>> + >>> + String[] parts = o.get().replaceAll("\\s","").split(":"); >> - you don't need to box int into Integer in TestCPUSets::checkResult L#108: >>> Asserts.assertEquals(new Integer(parts.length), new Integer(2)); >> - IIRC, in order to properly use whitebox API, you have to have both sun.hotspot.WhiteBox and sun.hotspot.WhiteBox$WhiteBoxPermission in BCP, but '@run driver ClassFileInstaller -jar whitebox.jar sun.hotspot.WhiteBox', which is used in all your tests, will put only s.h.WhiteBox in whitebox.jar >> - how do you plan to use :hotspot_containers test group? can't you use the test directory instead? >> >> the rest looks good to me. >> >> Thanks, >> -- Igor >> >>> On Nov 9, 2017, at 1:04 PM, Mikhailo Seledtsov > wrote: >>> >>> OK, will remove it. >>> >>> Misha >>> >>> On 11/9/17, 12:38 PM, Bob Vandette wrote: >>>> >>>> Since that path is not guaranteed to work, I?d remove CGROUP_CPUSET_PATH and the function. >>>> >>>> Bob. >>>> >>>>> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov > wrote: >>>>> >>>>> I am not using it. I thought it may be helpful if we go back to that method of extracting cpuset values for some reason. >>>>> On the other hand, dead code is not a good thing. So I am a bit undecided. >>>>> I will delete it if you think it is better to do so. >>>>> >>>>> Misha >>>>> >>>>> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>>>>> >>>>>> Do you still need this function? I don?t see it being used. >>>>>> >>>>>> 84 public static String read(String fileName) { >>>>>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>>>>> 86 try { >>>>>> 87 return readLineFromFile(path); >>>>>> 88 } catch (Exception e) { >>>>>> 89 System.err.println("Exception reading " + path); >>>>>> 90 System.err.println("Exception: " + e); >>>>>> 91 } >>>>>> 92 >>>>>> 93 return null; >>>>>> 94 } >>>>>> >>>>>> Bob. >>>>>> >>>>>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov > wrote: >>>>>>> >>>>>>> Here is a differential webrev: >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>>>>> >>>>>>> Misha >>>>>>> >>>>>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>>>>> Here is an update webrev: >>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>>>>> I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. >>>>>>>> Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). >>>>>>>> >>>>>>>> Summary for updated webrev.02: >>>>>>>> - implemented review feedback from Bob and Igor >>>>>>>> - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver >>>>>>>> - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently >>>>>>>> - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers >>>>>>>> (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) >>>>>>>> >>>>>>>> My next steps: >>>>>>>> - rebase to new jdk tree >>>>>>>> - update/merge >>>>>>>> - retest >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> >>>>>>>> >>>>>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>>>>> Hi Igor, >>>>>>>>> >>>>>>>>> Thank you for reviewing the code. >>>>>>>>> >>>>>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>>>>> Hi Misha, >>>>>>>>>> >>>>>>>>>> I have several comments: >>>>>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>>>>> Fixed >>>>>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>>>>> Thank you. I will try it out. >>>>>>>>>> 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >>>>>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); >>>>>>>>>> or >>>>>>>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >>>>>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >>>>>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>>>>> OK. I will unwind these expressions, and make sure indentation looks good. >>>>>>>>>> 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 >>>>>>>>> OK >>>>>>>>>> 5. Test*: there is no need to have parentheses in '@requires (docker.support)' >>>>>>>>> OK >>>>>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it >>>>>>>>> Will fix. >>>>>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format >>>>>>>>> OK. >>>>>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? >>>>>>>>> Oops. Missed it. Will add code to check it. >>>>>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness >>>>>>>>> Common.run() already checks for .shouldHaveExitValue(0) after running the container. >>>>>>>>> >>>>>>>>> >>>>>>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> -- Igor >>>>>>>>>> >>>>>>>>>>> On Nov 1, 2017, at 8:11 PM, mikhailo> wrote: >>>>>>>>>>> >>>>>>>>>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>>>>> Webrev: >>>>>>>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>>>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>>>>> Testing: >>>>>>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>>>>>> Ran the developed tests via jtreg >>>>>>>>>>> Pass >>>>>>>>>>> >>>>>>>>>>> 2. Automated testing system - run these tests >>>>>>>>>>> In progress >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Misha >>>>>>>>>>> >>>>>> >>>> >> From ioi.lam at oracle.com Wed Nov 15 22:04:37 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 15 Nov 2017 14:04:37 -0800 Subject: RFR(XXS) JDK-8191042 Deprecate VM option CheckEndorsedAndExtDirs Message-ID: Hi, please review this very small change: ??? https://bugs.openjdk.java.net/browse/JDK-8191042 The endorsed standards override mechanism and the extensions mechanism were removed in JDK 9 via JEP 220. The CheckEndorsedAndExtDirs option is disabled by default. However, if it's enabled in the command-line, the VM would check the endorsed and ext directories so that the VM terminates if they exist. These checks are no longer needed in JDK 10 and beyond and can be removed. This task is the first step in the removal -- we need to mark this option as deprecated and print a warning message when it's used. The option itself, and the code that uses it, will be removed in a subsequent JDK release. Tested with hotspot tier1/2 Thanks - Ioi =================================== diff -r d76a6042f5d7 src/hotspot/share/runtime/arguments.cpp --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 08 09:03:24 2017 -0800 +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 15 14:00:58 2017 -0800 @@ -383,6 +383,7 @@ ?? { "MinRAMFraction",?????????????? JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() }, ?? { "InitialRAMFraction",?????????? JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() }, ?? { "IgnoreUnverifiableClassesDuringDump", JDK_Version::jdk(10),? JDK_Version::undefined(), JDK_Version::undefined() }, +? { "CheckEndorsedAndExtDirs",????? JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() }, ?? // --- Deprecated alias flags (see also aliased_jvm_flags) - sorted by obsolete_in then expired_in: ?? { "DefaultMaxRAMFraction",??????? JDK_Version::jdk(8), JDK_Version::undefined(), JDK_Version::undefined() }, From zgu at redhat.com Wed Nov 15 22:25:28 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 15 Nov 2017 17:25:28 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking Message-ID: This is Linux only enhancement for now, can be extended for other platforms. (See bug for details) Current implementation, thread stack is tracked when thread is created, its available stack space is tagged as reserved and committed. However, this is not how stack works. Stack pages are committed on demand, so this approach overstates its memory usage. This enhancement gathers thread stack usage at NMT query time, so that it can report more accurate usage. Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html Example: Summary shows thread stacks only use small faction of reserved space. - Thread (reserved=41216KB, committed=632KB) (thread #41) (stack: reserved=41032KB, committed=448KB) (malloc=137KB #222) (arena=47KB #76) Detail tracking shows stack guards and actually used/committed stack area. [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread Stack from [0x00007f67c0b57d5c] JavaThread::run()+0x2c [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 [0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB from [0x00007f67c08b7319] os::pd_create_stack_guard_pages(char*, unsigned long)+0x29 [0x00007f67c0b56851] JavaThread::create_stack_guard_pages() [clone .part.125]+0xa1 [0x00007f67c0b57d6e] JavaThread::run()+0x3e [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB Thanks, -Zhengyu From bob.vandette at oracle.com Wed Nov 15 23:12:29 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 15 Nov 2017 18:12:29 -0500 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <5A0CB392.7040803@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> <5A04C2D5.8090404@oracle.com> <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> <5A0CB392.7040803@oracle.com> Message-ID: <39A85F90-2DAF-4C96-AFE2-B3CDB5D9E5DF@oracle.com> Looks good to me. Bob. > On Nov 15, 2017, at 4:37 PM, Mikhailo Seledtsov wrote: > > Hi Igor, Bob, > > Webrev containing latest changes based on feedback from you: > Diff from 02: http://cr.openjdk.java.net/~mseledtsov/8189762.02-03/ > Full: http://cr.openjdk.java.net/~mseledtsov/8189762.03/ > > > Thank you, > Misha > > > On 11/14/17, 4:00 PM, Igor Ignatyev wrote: >> >> Hi Misha, >> >> a few more comments: >> - are you using CPUSetsReader::listToString(boolean abc, List list, int maxCount) method? I wasn't able to find any usages of it. >> - CPUSetsReader uses Asserts.assertTrue(parts.length == 2) instead of Asserts.assertEQ(parts.length, 2) >> - instead of using Stream::collect in CPUSetsReader::readFromProcStatus, you can use Stream::findFirst(). so it'll be smth like >>> + try (Stream stream = Files.lines(Paths.get(path))) { >>> + o = stream >>> + .filter(line -> line.contains(setType)) >>> + .findFirst(); >>> + } catch (IOException e) { >>> + return null; >>> + } >>> + >>> + if (!o.isPresent()) { >>> + return null; >>> + } >>> + >>> + String[] parts = o.get().replaceAll("\\s","").split(":"); >> - you don't need to box int into Integer in TestCPUSets::checkResult L#108: >>> Asserts.assertEquals(new Integer(parts.length), new Integer(2)); >> - IIRC, in order to properly use whitebox API, you have to have both sun.hotspot.WhiteBox and sun.hotspot.WhiteBox$WhiteBoxPermission in BCP, but '@run driver ClassFileInstaller -jar whitebox.jar sun.hotspot.WhiteBox', which is used in all your tests, will put only s.h.WhiteBox in whitebox.jar >> - how do you plan to use :hotspot_containers test group? can't you use the test directory instead? >> >> the rest looks good to me. >> >> Thanks, >> -- Igor >> >>> On Nov 9, 2017, at 1:04 PM, Mikhailo Seledtsov > wrote: >>> >>> OK, will remove it. >>> >>> Misha >>> >>> On 11/9/17, 12:38 PM, Bob Vandette wrote: >>>> >>>> Since that path is not guaranteed to work, I?d remove CGROUP_CPUSET_PATH and the function. >>>> >>>> Bob. >>>> >>>>> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov > wrote: >>>>> >>>>> I am not using it. I thought it may be helpful if we go back to that method of extracting cpuset values for some reason. >>>>> On the other hand, dead code is not a good thing. So I am a bit undecided. >>>>> I will delete it if you think it is better to do so. >>>>> >>>>> Misha >>>>> >>>>> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>>>>> >>>>>> Do you still need this function? I don?t see it being used. >>>>>> >>>>>> 84 public static String read(String fileName) { >>>>>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>>>>> 86 try { >>>>>> 87 return readLineFromFile(path); >>>>>> 88 } catch (Exception e) { >>>>>> 89 System.err.println("Exception reading " + path); >>>>>> 90 System.err.println("Exception: " + e); >>>>>> 91 } >>>>>> 92 >>>>>> 93 return null; >>>>>> 94 } >>>>>> >>>>>> Bob. >>>>>> >>>>>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov > wrote: >>>>>>> >>>>>>> Here is a differential webrev: >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>>>>> >>>>>>> Misha >>>>>>> >>>>>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>>>>> Here is an update webrev: >>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>>>>> I also tried to generate a diff between 01 and 02, but could not. My script does not seem to work any more, or I am missing something. >>>>>>>> Igor, please let me know if you have scripts or command line to generate incremental webrev (I committed my initial changes locally, then applied the new changes on top). >>>>>>>> >>>>>>>> Summary for updated webrev.02: >>>>>>>> - implemented review feedback from Bob and Igor >>>>>>>> - added @driver to all tests, since the actual testing happens in the child docker process; the main is just a test driver >>>>>>>> - configured docker directory for exclusive access by jtreg tests, since there are potential and actual issues when running these tests concurrently >>>>>>>> - modified test groups to add a hotspot_docker test group, and to exclude docker testing from lower tiers >>>>>>>> (once the infra is ready, I plan to add this execution initially at higher tiers; first will run it for a while as custom runs) >>>>>>>> >>>>>>>> My next steps: >>>>>>>> - rebase to new jdk tree >>>>>>>> - update/merge >>>>>>>> - retest >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> >>>>>>>> >>>>>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>>>>> Hi Igor, >>>>>>>>> >>>>>>>>> Thank you for reviewing the code. >>>>>>>>> >>>>>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>>>>> Hi Misha, >>>>>>>>>> >>>>>>>>>> I have several comments: >>>>>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>>>>> Fixed >>>>>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using stream api as list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>>>>> Thank you. I will try it out. >>>>>>>>>> 3. in several files, I have noticed some problems w/ indents which make code quite hard to read, for example >>>>>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + expectedTraceValue); >>>>>>>>>> or >>>>>>>>>>> 96 DockerRunOptions opts = Common.newOpts(imageName, "AttemptOOM") >>>>>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, "--memory-swap", dockerMemLimit); >>>>>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>>>>> OK. I will unwind these expressions, and make sure indentation looks good. >>>>>>>>>> 4. TestCPUSets.java : it's better to use Assert.assertEquals(parts.length, 2) than Asserts.assertTrue(parts.length == 2) as the former will also print the actual value of parts.length. so you won't need to have L#103 >>>>>>>>> OK >>>>>>>>>> 5. Test*: there is no need to have parentheses in '@requires (docker.support)' >>>>>>>>> OK >>>>>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK under test, so you can use '@run driver' to run it >>>>>>>>> Will fix. >>>>>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol you should use %n in format string. also System.out::printf seems to be more popular in our code than System.out::format >>>>>>>>> OK. >>>>>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we found lineMarker? >>>>>>>>> Oops. Missed it. Will add code to check it. >>>>>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) to all Common.run calls which aren't expected to fail, e.g. all test* in TestCPUAwareness, testMemorySoftLimit and testMemoryLimit in TestMemoryAwareness >>>>>>>>> Common.run() already checks for .shouldHaveExitValue(0) after running the container. >>>>>>>>> >>>>>>>>> >>>>>>>>> I will apply feedback from Bob and you, and will post a new webrev. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> -- Igor >>>>>>>>>> >>>>>>>>>>> On Nov 1, 2017, at 8:11 PM, mikhailo> wrote: >>>>>>>>>>> >>>>>>>>>>> Please review these tests that were developed to test JVM's container awareness in Docker environment. >>>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>>>>> Webrev: >>>>>>>>>>> Tests: http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>>>>> WB API: http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>>>>> Testing: >>>>>>>>>>> 1. Locally: Linux-x64, docker engine version: 17.06.2-ce >>>>>>>>>>> Ran the developed tests via jtreg >>>>>>>>>>> Pass >>>>>>>>>>> >>>>>>>>>>> 2. Automated testing system - run these tests >>>>>>>>>>> In progress >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Misha >>>>>>>>>>> >>>>>> >>>> >> From ioi.lam at oracle.com Wed Nov 15 23:37:34 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 15 Nov 2017 15:37:34 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> Message-ID: <558d63a5-da5a-7bd7-ba44-048868a37f12@oracle.com> Hi Yumin, Thanks for the review. I have changed it to SHARE_PRIMS_CDOFFSETS_HPP. - Ioi On 11/15/17 9:52 AM, yumin qi wrote: > HI, Ioi > > Only one minor comment,? in cdsoffsets.hpp: > #ifndef CLOSED_VM_PRIMS_CDSWHITEBOX_HPP > Should it be changed to SHARE_PRIMS_CDOFFSETS_HPP to keep it with same > style with other head files. > Since the hotspot directory changed, I don't know if all head files > should change to follow that (changes). > Thanks > Yumin > > On Tue, Oct 31, 2017 at 7:43 PM, Ioi Lam > wrote: > > Hi, > > Here's the new webrev for both the AppCDS implementation and > tests. During internal review of the JEP, we have decided to > integrate both implementation and tests at the same time. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ > > > As mentioned before, most of the "diffs" shown in this webrev are > the result of copying the closed source files on top of files of > the same name in the open repo. So in reviewing, instead of > focusing on what's "changed", please focus on the entire content > of the new version of each file. > > Testing: I did an OpenJDK linux-x64 build (without Oracle closed > sources) and all the new appcds tests passed. > > Thanks > > - Ioi > > > On 10/30/17 8:52 AM, Ioi Lam wrote: > > Hi Dmitry, > > In the latest JDK 10 repo, is_jrt has been renamed to > is_modules_image. Please change the code accordingly. > > I will post my latest diff soon, with some test cases as well. > > Thanks > > - Ioi > > > On 10/30/17 4:04 AM, Dmitry Samersoff wrote: > > Ioi, > > I'd tried to apply your patch to latest open JDK10 and > the compilation fails with: > > /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: > > error: ?class SharedClassPathEntry? has no member named > ?is_jrt? > > Did I miss something? > > -Dmitry > > On 13.10.2017 02:48, Ioi Lam wrote: > > Hi, > > Please review this change set. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ > > https://bugs.openjdk.java.net/browse/JDK-8188791 > > > This is the first step of implementing the following > JEP, which moves > AppCDS from > closed repos into the openjdk repo: > > https://bugs.openjdk.java.net/browse/JDK-8185996 > > > In JDK 9, significant portion of AppCDS code resided > in the closed repo. > As part > of the open-sourcing effort of JDK 18.3, we will move > the source code > into the > open repo. > > In this changeset, the code is moved verbatim as much > as possible. The > intention is > only to relocate the sources, not to changing existing > behaviors, and not > to do any sort of refactoring. > > Most of the "diffs" shown in this webrev are the > result of copying the > closed source > files on top of files of the same name in the open > repo. So in > reviewing, instead of > focusing on what's "changed", it's better to focus on > the entire content > of the new > version of each file. > > The only functional change in this task is that the > UseAppCDS flag is > changed from > a "commercial" flag to a regular "product" flag. This > is because > "commercial" > flags are not supported by the OpenJDK build. > > Source code refactoring may be desirable, because the > old open/closed > source > code structure had introduced some intermediary APIs > to connect code > between > the two repos. Such API should be removed in a > separate RFE. > > Also, some AppCDS tests are currently in the closed > repo. These tests > will be > moved in a separate task. See JDK-8188792 for details. > > All the AppCDS tests (currently still in closed > sources) passed with > both Oracle JDK > and OpenJDK. > > Thanks > - Ioi > > > > > From ioi.lam at oracle.com Thu Nov 16 00:01:43 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 15 Nov 2017 16:01:43 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> Message-ID: Hi Dmitry, Thanks for the review. On 11/6/17 7:07 AM, Dmitry Samersoff wrote: > Ioi, > > I tested new patch on aarch64 and it works fine with the changes below[1]. I've fixed it. Thanks for the patch. > Also tests doesn't work with exploded image - is it possible to check > for this case and produce appropriate error message? CDS is not supported on exploded images. I think the best thing to do is to add something like "@requires vm.cds" into the test cases (similar to the use of "@require vm.aot" in other tests). That way, none of the CDS tests will be executed for It's too late to do this in JDK 10 now, but I will file an RFE to do it in the next release. I'll also file another RFE to print a better message. Today if you use an exploded build the error message is kind of confusing: $ ./jdk/bin/java -Xshare:dump Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' Hint: enable -Xlog:class+path=info to diagnose the failure Error occurred during initialization of VM CDS allows only empty directories in archived classpaths It should say something like "CDS is not supported on exploded images". Thanks - Ioi > 1. > diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp > --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 > 09:45:23 2017 +0000 > +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 > 15:02:51 2017 +0000 > @@ -31,7 +31,6 @@ > -#include "prims/jvm.h" > > diff -r dbf9cec6a568 src/hotspot/share/classfile/systemDictionaryShared.cpp > --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov > 06 09:45:23 2017 +0000 > +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov > 06 15:02:51 2017 +0000 > @@ -518,7 +518,7 @@ > > { > MutexLocker mu(SystemDictionary_lock, THREAD); > - Klass* check = find_class(d_index, d_hash, name, dictionary); > + Klass* check = dictionary->find_class(d_index, d_hash, name); > if (check != NULL) { > return InstanceKlass::cast(check); > } > > -Dmitry > > On 11/01/2017 05:43 AM, Ioi Lam wrote: >> Hi, >> >> Here's the new webrev for both the AppCDS implementation and tests. >> During internal review of the JEP, we have decided to integrate both >> implementation and tests at the same time. >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >> >> As mentioned before, most of the "diffs" shown in this webrev are the >> result of copying the closed source files on top of files of the same >> name in the open repo. So in reviewing, instead of focusing on what's >> "changed", please focus on the entire content of the new version of each >> file. >> >> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >> sources) and all the new appcds tests passed. >> >> Thanks >> >> - Ioi >> >> >> On 10/30/17 8:52 AM, Ioi Lam wrote: >>> Hi Dmitry, >>> >>> In the latest JDK 10 repo, is_jrt has been renamed to >>> is_modules_image. Please change the code accordingly. >>> >>> I will post my latest diff soon, with some test cases as well. >>> >>> Thanks >>> >>> - Ioi >>> >>> >>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>> Ioi, >>>> >>>> I'd tried to apply your patch to latest open JDK10 and >>>> the compilation fails with: >>>> >>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>> >>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>> >>>> Did I miss something? >>>> >>>> -Dmitry >>>> >>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>> Hi, >>>>> >>>>> Please review this change set. >>>>> >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>> ???? https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>> >>>>> This is the first step of implementing the following JEP, which moves >>>>> AppCDS from >>>>> closed repos into the openjdk repo: >>>>> >>>>> ???? https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>> >>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>> repo. >>>>> As part >>>>> of the open-sourcing effort of JDK 18.3, we will move the source code >>>>> into the >>>>> open repo. >>>>> >>>>> In this changeset, the code is moved verbatim as much as possible. The >>>>> intention is >>>>> only to relocate the sources, not to changing existing behaviors, >>>>> and not >>>>> to do any sort of refactoring. >>>>> >>>>> Most of the "diffs" shown in this webrev are the result of copying the >>>>> closed source >>>>> files on top of files of the same name in the open repo. So in >>>>> reviewing, instead of >>>>> focusing on what's "changed", it's better to focus on the entire >>>>> content >>>>> of the new >>>>> version of each file. >>>>> >>>>> The only functional change in this task is that the UseAppCDS flag is >>>>> changed from >>>>> a "commercial" flag to a regular "product" flag. This is because >>>>> "commercial" >>>>> flags are not supported by the OpenJDK build. >>>>> >>>>> Source code refactoring may be desirable, because the old open/closed >>>>> source >>>>> code structure had introduced some intermediary APIs to connect code >>>>> between >>>>> the two repos. Such API should be removed in a separate RFE. >>>>> >>>>> Also, some AppCDS tests are currently in the closed repo. These tests >>>>> will be >>>>> moved in a separate task. See JDK-8188792 for details. >>>>> >>>>> All the AppCDS tests (currently still in closed sources) passed with >>>>> both Oracle JDK >>>>> and OpenJDK. >>>>> >>>>> Thanks >>>>> - Ioi From ioi.lam at oracle.com Thu Nov 16 00:10:46 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 15 Nov 2017 16:10:46 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> Message-ID: I filed: https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg VMProps to filter out CDS tests https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message when CDS is not supported Thanks - Ioi On 11/15/17 4:01 PM, Ioi Lam wrote: > Hi Dmitry, > > Thanks for the review. > > On 11/6/17 7:07 AM, Dmitry Samersoff wrote: > >> Ioi, >> >> I tested new patch on aarch64 and it works fine with the changes >> below[1]. > I've fixed it. Thanks for the patch. >> Also tests doesn't work with exploded image - is it possible to check >> for this case and produce appropriate error message? > CDS is not supported on exploded images. I think the best thing to do > is to add something like "@requires vm.cds" into the test cases > (similar to the use of "@require vm.aot" in other tests). That way, > none of the CDS tests will be executed for > > It's too late to do this in JDK 10 now, but I will file an RFE to do > it in the next release. > > I'll also file another RFE to print a better message. Today if you use > an exploded build the error message is kind of confusing: > > $ ./jdk/bin/java -Xshare:dump > Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' > Hint: enable -Xlog:class+path=info to diagnose the failure > Error occurred during initialization of VM > CDS allows only empty directories in archived classpaths > > It should say something like "CDS is not supported on exploded images". > > Thanks > - Ioi > >> 1. >> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >> --- a/src/hotspot/share/classfile/classListParser.cpp?? Mon Nov 06 >> 09:45:23 2017 +0000 >> +++ b/src/hotspot/share/classfile/classListParser.cpp?? Mon Nov 06 >> 15:02:51 2017 +0000 >> @@ -31,7 +31,6 @@ >> -#include "prims/jvm.h" >> >> diff -r dbf9cec6a568 >> src/hotspot/share/classfile/systemDictionaryShared.cpp >> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >> 06 09:45:23 2017 +0000 >> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >> 06 15:02:51 2017 +0000 >> @@ -518,7 +518,7 @@ >> >> ????? { >> ??????? MutexLocker mu(SystemDictionary_lock, THREAD); >> -????? Klass* check = find_class(d_index, d_hash, name, dictionary); >> +????? Klass* check = dictionary->find_class(d_index, d_hash, name); >> ??????? if (check != NULL) { >> ????????? return InstanceKlass::cast(check); >> ??????? } >> >> -Dmitry >> >> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>> Hi, >>> >>> Here's the new webrev for both the AppCDS implementation and tests. >>> During internal review of the JEP, we have decided to integrate both >>> implementation and tests at the same time. >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>> >>> As mentioned before, most of the "diffs" shown in this webrev are the >>> result of copying the closed source files on top of files of the same >>> name in the open repo. So in reviewing, instead of focusing on what's >>> "changed", please focus on the entire content of the new version of >>> each >>> file. >>> >>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>> sources) and all the new appcds tests passed. >>> >>> Thanks >>> >>> - Ioi >>> >>> >>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>> Hi Dmitry, >>>> >>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>> is_modules_image. Please change the code accordingly. >>>> >>>> I will post my latest diff soon, with some test cases as well. >>>> >>>> Thanks >>>> >>>> - Ioi >>>> >>>> >>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>> Ioi, >>>>> >>>>> I'd tried to apply your patch to latest open JDK10 and >>>>> the compilation fails with: >>>>> >>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>> >>>>> >>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>> >>>>> Did I miss something? >>>>> >>>>> -Dmitry >>>>> >>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>> Hi, >>>>>> >>>>>> Please review this change set. >>>>>> >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>> >>>>>> ????? https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>> >>>>>> This is the first step of implementing the following JEP, which >>>>>> moves >>>>>> AppCDS from >>>>>> closed repos into the openjdk repo: >>>>>> >>>>>> ????? https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>> >>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>> repo. >>>>>> As part >>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>> code >>>>>> into the >>>>>> open repo. >>>>>> >>>>>> In this changeset, the code is moved verbatim as much as >>>>>> possible. The >>>>>> intention is >>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>> and not >>>>>> to do any sort of refactoring. >>>>>> >>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>> copying the >>>>>> closed source >>>>>> files on top of files of the same name in the open repo. So in >>>>>> reviewing, instead of >>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>> content >>>>>> of the new >>>>>> version of each file. >>>>>> >>>>>> The only functional change in this task is that the UseAppCDS >>>>>> flag is >>>>>> changed from >>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>> "commercial" >>>>>> flags are not supported by the OpenJDK build. >>>>>> >>>>>> Source code refactoring may be desirable, because the old >>>>>> open/closed >>>>>> source >>>>>> code structure had introduced some intermediary APIs to connect code >>>>>> between >>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>> >>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>> tests >>>>>> will be >>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>> >>>>>> All the AppCDS tests (currently still in closed sources) passed with >>>>>> both Oracle JDK >>>>>> and OpenJDK. >>>>>> >>>>>> Thanks >>>>>> - Ioi > From mikhailo.seledtsov at oracle.com Thu Nov 16 00:20:30 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 15 Nov 2017 16:20:30 -0800 Subject: RFR(M): 8189762: [TESTBUG] Create tests for JDK-8146115 container awareness and resource configuration In-Reply-To: <39A85F90-2DAF-4C96-AFE2-B3CDB5D9E5DF@oracle.com> References: <4c62fbeb-fc17-ec0e-5662-75642ee821e6@oracle.com> <3DDC3F75-57CD-4A51-8E99-3A598FB11309@oracle.com> <5A01429B.4070806@oracle.com> <5A03AED1.3060203@oracle.com> <5A04A95A.8010106@oracle.com> <5A04BB33.2030207@oracle.com> <06025509-0650-4FBB-AACE-102583015D8C@oracle.com> <5A04C2D5.8090404@oracle.com> <6583B764-7DEE-4375-AFA9-B764744BC93B@oracle.com> <5A0CB392.7040803@oracle.com> <39A85F90-2DAF-4C96-AFE2-B3CDB5D9E5DF@oracle.com> Message-ID: <5A0CD9CE.4020907@oracle.com> Bob, Igor, Thank you for review. Misha On 11/15/17, 3:12 PM, Bob Vandette wrote: > Looks good to me. > > Bob. > > >> On Nov 15, 2017, at 4:37 PM, Mikhailo Seledtsov >> > > wrote: >> >> Hi Igor, Bob, >> >> Webrev containing latest changes based on feedback from you: >> Diff from 02: http://cr.openjdk.java.net/~mseledtsov/8189762.02-03/ >> Full: http://cr.openjdk.java.net/~mseledtsov/8189762.03/ >> >> >> Thank you, >> Misha >> >> >> On 11/14/17, 4:00 PM, Igor Ignatyev wrote: >>> Hi Misha, >>> >>> a few more comments: >>> - are you using CPUSetsReader::listToString(boolean abc, >>> List list, int maxCount) method? I wasn't able to find any >>> usages of it. >>> - CPUSetsReader uses Asserts.assertTrue(parts.length == 2) instead >>> of Asserts.assertEQ(parts.length, 2) >>> - instead of using Stream::collect in >>> CPUSetsReader::readFromProcStatus, you can use Stream::findFirst(). >>> so it'll be smth like >>>> + try (Stream stream = Files.lines(Paths.get(path))) { >>>> + o = stream >>>> + .filter(line -> line.contains(setType)) >>>> + .findFirst(); >>>> + } catch (IOException e) { >>>> + return null; >>>> + } >>>> + >>>> + if (!o.isPresent()) { >>>> + return null; >>>> + } >>>> + >>>> + String[] parts = o.get().replaceAll("\\s","").split(":"); >>> - you don't need to box int into Integer in TestCPUSets::checkResult >>> L#108: >>>> Asserts.assertEquals(new Integer(parts.length), new Integer(2)); >>> - IIRC, in order to properly use whitebox API, you have to have >>> both sun.hotspot.WhiteBox and >>> sun.hotspot.WhiteBox$WhiteBoxPermission in BCP, but '@run driver >>> ClassFileInstaller -jar whitebox.jar sun.hotspot.WhiteBox', which is >>> used in all your tests, will put only s.h.WhiteBox in whitebox.jar >>> - how do you plan to use :hotspot_containers test group? can't you >>> use the test directory instead? >>> >>> the rest looks good to me. >>> >>> Thanks, >>> -- Igor >>> >>>> On Nov 9, 2017, at 1:04 PM, Mikhailo Seledtsov >>>> >>> > wrote: >>>> >>>> OK, will remove it. >>>> >>>> Misha >>>> >>>> On 11/9/17, 12:38 PM, Bob Vandette wrote: >>>>> Since that path is not guaranteed to work, I?d remove >>>>> CGROUP_CPUSET_PATH and the function. >>>>> >>>>> Bob. >>>>> >>>>>> On Nov 9, 2017, at 3:31 PM, Mikhailo Seledtsov >>>>>> >>>>> > wrote: >>>>>> >>>>>> I am not using it. I thought it may be helpful if we go back to >>>>>> that method of extracting cpuset values for some reason. >>>>>> On the other hand, dead code is not a good thing. So I am a bit >>>>>> undecided. >>>>>> I will delete it if you think it is better to do so. >>>>>> >>>>>> Misha >>>>>> >>>>>> On 11/9/17, 12:00 PM, Bob Vandette wrote: >>>>>>> Do you still need this function? I don?t see it being used. >>>>>>> >>>>>>> 84 public static String read(String fileName) { >>>>>>> 85 String path = CGROUP_CPUSET_PATH + fileName; >>>>>>> 86 try { >>>>>>> 87 return readLineFromFile(path); >>>>>>> 88 } catch (Exception e) { >>>>>>> 89 System.err.println("Exception reading " + path); >>>>>>> 90 System.err.println("Exception: " + e); >>>>>>> 91 } >>>>>>> 92 >>>>>>> 93 return null; >>>>>>> 94 } >>>>>>> >>>>>>> Bob. >>>>>>> >>>>>>>> On Nov 9, 2017, at 2:15 PM, Mikhailo Seledtsov >>>>>>>> >>>>>>> > wrote: >>>>>>>> >>>>>>>> Here is a differential webrev: >>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.01-02/webrev/ >>>>>>>> >>>>>>>> >>>>>>>> Misha >>>>>>>> >>>>>>>> On 11/8/17, 5:26 PM, Mikhailo Seledtsov wrote: >>>>>>>>> Here is an update webrev: >>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.02/ >>>>>>>>> >>>>>>>>> I also tried to generate a diff between 01 and 02, but could >>>>>>>>> not. My script does not seem to work any more, or I am missing >>>>>>>>> something. >>>>>>>>> Igor, please let me know if you have scripts or command line >>>>>>>>> to generate incremental webrev (I committed my initial changes >>>>>>>>> locally, then applied the new changes on top). >>>>>>>>> >>>>>>>>> Summary for updated webrev.02: >>>>>>>>> - implemented review feedback from Bob and Igor >>>>>>>>> - added @driver to all tests, since the actual testing happens >>>>>>>>> in the child docker process; the main is just a test driver >>>>>>>>> - configured docker directory for exclusive access by jtreg >>>>>>>>> tests, since there are potential and actual issues when >>>>>>>>> running these tests concurrently >>>>>>>>> - modified test groups to add a hotspot_docker test group, and >>>>>>>>> to exclude docker testing from lower tiers >>>>>>>>> (once the infra is ready, I plan to add this execution >>>>>>>>> initially at higher tiers; first will run it for a while as >>>>>>>>> custom runs) >>>>>>>>> >>>>>>>>> My next steps: >>>>>>>>> - rebase to new jdk tree >>>>>>>>> - update/merge >>>>>>>>> - retest >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Misha >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/6/17, 9:20 PM, Mikhailo Seledtsov wrote: >>>>>>>>>> Hi Igor, >>>>>>>>>> >>>>>>>>>> Thank you for reviewing the code. >>>>>>>>>> >>>>>>>>>> On 11/6/17, 4:31 PM, Igor Ignatyev wrote: >>>>>>>>>>> Hi Misha, >>>>>>>>>>> >>>>>>>>>>> I have several comments: >>>>>>>>>>> 1. whitebox.cpp : WB_IsContainerized has an incorrect >>>>>>>>>>> signature, it should be WB_IsContainerized(JNIEnv*, jobject) >>>>>>>>>> Fixed >>>>>>>>>>> 2. CPUSetsReader.java : listToString can be rewritten using >>>>>>>>>>> stream api as >>>>>>>>>>> list.stream().limit(maxCount).map(Object::toString).collect(Collectors.joining(",")) >>>>>>>>>> Thank you. I will try it out. >>>>>>>>>>> 3. in several files, I have noticed some problems w/ indents >>>>>>>>>>> which make code quite hard to read, for example >>>>>>>>>>>> 73 Common.run( Common.newOpts(imageName) >>>>>>>>>>>> 74 .addDockerOpts("--memory", valueToSet)) >>>>>>>>>>>> 75 .shouldMatch("Memory Limit is:.*" + >>>>>>>>>>>> expectedTraceValue); >>>>>>>>>>> or >>>>>>>>>>>> 96 DockerRunOptions opts = >>>>>>>>>>>> Common.newOpts(imageName, "AttemptOOM") >>>>>>>>>>>> 97 .addDockerOpts("--memory", dockerMemLimit, >>>>>>>>>>>> "--memory-swap", dockerMemLimit); >>>>>>>>>>>> 98 opts.classParams.add("" + sizeToAllocInMb); >>>>>>>>>> OK. I will unwind these expressions, and make sure >>>>>>>>>> indentation looks good. >>>>>>>>>>> 4. TestCPUSets.java : it's better to use >>>>>>>>>>> Assert.assertEquals(parts.length, 2) than >>>>>>>>>>> Asserts.assertTrue(parts.length == 2) as the former will >>>>>>>>>>> also print the actual value of parts.length. so you won't >>>>>>>>>>> need to have L#103 >>>>>>>>>> OK >>>>>>>>>>> 5. Test*: there is no need to have parentheses in '@requires >>>>>>>>>>> (docker.support)' >>>>>>>>>> OK >>>>>>>>>>> 6. Test*: ClassFileInstaller doesn't have to be run by JDK >>>>>>>>>>> under test, so you can use '@run driver' to run it >>>>>>>>>> Will fix. >>>>>>>>>>> 7. TestCPUSets.java : L#72, to get a proper new line symbol >>>>>>>>>>> you should use %n in format string. also System.out::printf >>>>>>>>>>> seems to be more popular in our code than System.out::format >>>>>>>>>> OK. >>>>>>>>>>> 8. TestCPUSets.java: shouldn't checkResult assert that we >>>>>>>>>>> found lineMarker? >>>>>>>>>> Oops. Missed it. Will add code to check it. >>>>>>>>>>> 9. Test*: would you consider adding .shouldHaveExitValue(0) >>>>>>>>>>> to all Common.run calls which aren't expected to fail, e.g. >>>>>>>>>>> all test* in TestCPUAwareness, testMemorySoftLimit and >>>>>>>>>>> testMemoryLimit in TestMemoryAwareness >>>>>>>>>> Common.run() already checks for .shouldHaveExitValue(0) after >>>>>>>>>> running the container. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I will apply feedback from Bob and you, and will post a new >>>>>>>>>> webrev. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Misha >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> -- Igor >>>>>>>>>>> >>>>>>>>>>>> On Nov 1, 2017, at 8:11 PM, >>>>>>>>>>>> mikhailo>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>> Please review these tests that were developed to test JVM's >>>>>>>>>>>> container awareness in Docker environment. >>>>>>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8189762 >>>>>>>>>>>> Webrev: >>>>>>>>>>>> Tests: >>>>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00/ >>>>>>>>>>>> >>>>>>>>>>>> WB API: >>>>>>>>>>>> http://cr.openjdk.java.net/~mseledtsov/8189762.00.whitebox/ >>>>>>>>>>>> >>>>>>>>>>>> Testing: >>>>>>>>>>>> 1. Locally: Linux-x64, docker engine version: >>>>>>>>>>>> 17.06.2-ce >>>>>>>>>>>> Ran the developed tests via jtreg >>>>>>>>>>>> Pass >>>>>>>>>>>> >>>>>>>>>>>> 2. Automated testing system - run these tests >>>>>>>>>>>> In progress >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Misha >>>>>>>>>>>> >>>>>>> >>>>> >>> > From david.holmes at oracle.com Thu Nov 16 02:19:00 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 16 Nov 2017 12:19:00 +1000 Subject: RFR(XXS) JDK-8191042 Deprecate VM option CheckEndorsedAndExtDirs In-Reply-To: References: Message-ID: Hi Ioi, Looks good. Just need to wait for the CSR to be approved before pushing. Thanks, David On 16/11/2017 8:04 AM, Ioi Lam wrote: > Hi, please review this very small change: > > ??? https://bugs.openjdk.java.net/browse/JDK-8191042 > > The endorsed standards override mechanism and the extensions mechanism > were removed in JDK 9 via JEP 220. > > The CheckEndorsedAndExtDirs option is disabled by default. However, if > it's enabled in the command-line, the VM would check the endorsed and ext > directories so that the VM terminates if they exist. These checks are no > longer needed in JDK 10 and beyond and can be removed. > > This task is the first step in the removal -- we need to mark this option > as deprecated and print a warning message when it's used. The option > itself, and the code that uses it, will be removed in a subsequent JDK > release. > > Tested with hotspot tier1/2 > > Thanks > - Ioi > > =================================== > > diff -r d76a6042f5d7 src/hotspot/share/runtime/arguments.cpp > --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 08 09:03:24 > 2017 -0800 > +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 15 14:00:58 > 2017 -0800 > @@ -383,6 +383,7 @@ > ?? { "MinRAMFraction",?????????????? JDK_Version::jdk(10), > JDK_Version::undefined(), JDK_Version::undefined() }, > ?? { "InitialRAMFraction",?????????? JDK_Version::jdk(10), > JDK_Version::undefined(), JDK_Version::undefined() }, > ?? { "IgnoreUnverifiableClassesDuringDump", JDK_Version::jdk(10), > JDK_Version::undefined(), JDK_Version::undefined() }, > +? { "CheckEndorsedAndExtDirs",????? JDK_Version::jdk(10), > JDK_Version::undefined(), JDK_Version::undefined() }, > > ?? // --- Deprecated alias flags (see also aliased_jvm_flags) - sorted > by obsolete_in then expired_in: > ?? { "DefaultMaxRAMFraction",??????? JDK_Version::jdk(8), > JDK_Version::undefined(), JDK_Version::undefined() }, > From david.holmes at oracle.com Thu Nov 16 06:58:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 16 Nov 2017 16:58:06 +1000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: Hi Zhengyu, I can't review this in detail as I'm not familiar with NMT workings, but have a couple of comments. On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > This is Linux only enhancement for now, can be extended for other > platforms. (See bug for details) I'm concerned about unnecessary platform skew. I added some comments to the bug. I think the mincore trick can be used on Solaris and AIX as well - but not on OS X / BSD. It may be that VirtualQuery can be used on Windows to get the same information - but I'm unclear if the memory states support what you're looking for. It would be good to extend this to other platforms that will support it. And on a minor note can you please correct the existing spelling error in get_stack_commited_bottom. :) > Current implementation, thread stack is tracked when thread is created, > its available stack space is tagged as reserved and committed. > > However, this is not how stack works. Stack pages are committed on > demand, so this approach overstates its memory usage. > > This enhancement gathers thread stack usage at NMT query time, so that > it can report more accurate usage. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > > Example: > > Summary shows thread stacks only use small faction of reserved space. > > -??????????????????? Thread (reserved=41216KB, committed=632KB) > ??????????????????????????? (thread #41) > ??????????????????????????? (stack: reserved=41032KB, committed=448KB) > ??????????????????????????? (malloc=137KB #222) > ??????????????????????????? (arena=47KB #76) > > > Detail tracking shows stack guards and actually used/committed stack area. > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread > Stack from > ??? [0x00007f67c0b57d5c] JavaThread::run()+0x2c > ??? [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > ??????? [0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB from > ??????????? [0x00007f67c08b7319] os::pd_create_stack_guard_pages(char*, > unsigned long)+0x29 > ??????????? [0x00007f67c0b56851] JavaThread::create_stack_guard_pages() > [clone .part.125]+0xa1 > ??????????? [0x00007f67c0b57d6e] JavaThread::run()+0x3e > ??????????? [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > ??????? [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB Can we capture this in a test so we can tell that the tracking has improved? Thanks, David ----- > Thanks, > > -Zhengyu > From thomas.stuefe at gmail.com Thu Nov 16 07:14:57 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 16 Nov 2017 08:14:57 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: On Thu, Nov 16, 2017 at 7:58 AM, David Holmes wrote: > Hi Zhengyu, > > I can't review this in detail as I'm not familiar with NMT workings, but > have a couple of comments. > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > >> This is Linux only enhancement for now, can be extended for other >> platforms. (See bug for details) >> > > I'm concerned about unnecessary platform skew. I added some comments to > the bug. I think the mincore trick can be used on Solaris and AIX as well - > but not on OS X / BSD. It may be that VirtualQuery can be used on Windows > to get the same information - but I'm unclear if the memory states support > what you're looking for. It would be good to extend this to other platforms > that will support it. > Just 5c: I think this is definitely useful even with only the Linux implementation provided. On AIX we have commit-on-touch and hence mincore may actually force commit the thread :P - would have to try it but have no high hopes. Well. On Windows this is possible, we already use VirtualQuery in our fork to print out the commit-state of the current thread stack in case of an error. ..Thomas > And on a minor note can you please correct the existing spelling error in > get_stack_commited_bottom. :) > > Current implementation, thread stack is tracked when thread is created, >> its available stack space is tagged as reserved and committed. >> >> However, this is not how stack works. Stack pages are committed on >> demand, so this approach overstates its memory usage. >> >> This enhancement gathers thread stack usage at NMT query time, so that it >> can report more accurate usage. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >> Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html >> >> >> Example: >> >> Summary shows thread stacks only use small faction of reserved space. >> >> - Thread (reserved=41216KB, committed=632KB) >> (thread #41) >> (stack: reserved=41032KB, committed=448KB) >> (malloc=137KB #222) >> (arena=47KB #76) >> >> >> Detail tracking shows stack guards and actually used/committed stack area. >> >> [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread >> Stack from >> [0x00007f67c0b57d5c] JavaThread::run()+0x2c >> [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> >> [0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB from >> [0x00007f67c08b7319] os::pd_create_stack_guard_pages(char*, >> unsigned long)+0x29 >> [0x00007f67c0b56851] JavaThread::create_stack_guard_pages() >> [clone .part.125]+0xa1 >> [0x00007f67c0b57d6e] JavaThread::run()+0x3e >> [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> >> [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB >> > > Can we capture this in a test so we can tell that the tracking has > improved? > > Thanks, > David > ----- > > Thanks, >> >> -Zhengyu >> >> From sgehwolf at redhat.com Thu Nov 16 09:23:04 2017 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Thu, 16 Nov 2017 10:23:04 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: <1510824184.3968.3.camel@redhat.com> On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes > wrote: > > > Hi Zhengyu, > > > > I can't review this in detail as I'm not familiar with NMT workings, but > > have a couple of comments. > > > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > > > > > This is Linux only enhancement for now, can be extended for other > > > platforms. (See bug for details) > > > > > > > I'm concerned about unnecessary platform skew. I added some comments to > > the bug. I think the mincore trick can be used on Solaris and AIX as well - > > but not on OS X / BSD. It may be that VirtualQuery can be used on Windows > > to get the same information - but I'm unclear if the memory states support > > what you're looking for. It would be good to extend this to other platforms > > that will support it. > > > > Just 5c: > > I think this is definitely useful even with only the Linux implementation > provided. +1 While plaform skew is a valid concern, I'm doubtful it's "unnecessary". I think there is value to get one platform fixed and then look at other platforms one by one as follow-up bugs. For some platforms it might be hard to do (or undesirable) as Thomas says. Thanks, Severin > On AIX we have commit-on-touch and hence mincore may actually force commit > the thread :P - would have to try it but have no high hopes. Well. > > On Windows this is possible, we already use VirtualQuery in our fork to > print out the commit-state of the current thread stack in case of an error. > > ..Thomas > > > > And on a minor note can you please correct the existing spelling error in > > get_stack_commited_bottom. :) > > > > Current implementation, thread stack is tracked when thread is created, > > > its available stack space is tagged as reserved and committed. > > > > > > However, this is not how stack works. Stack pages are committed on > > > demand, so this approach overstates its memory usage. > > > > > > This enhancement gathers thread stack usage at NMT query time, so that it > > > can report more accurate usage. > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > > > Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > > > > > > > > Example: > > > > > > Summary shows thread stacks only use small faction of reserved space. > > > > > > -????????????????????Thread (reserved=41216KB, committed=632KB) > > > ?????????????????????????????(thread #41) > > > ?????????????????????????????(stack: reserved=41032KB, committed=448KB) > > > ?????????????????????????????(malloc=137KB #222) > > > ?????????????????????????????(arena=47KB #76) > > > > > > > > > Detail tracking shows stack guards and actually used/committed stack area. > > > > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread > > > Stack from > > > ?????[0x00007f67c0b57d5c] JavaThread::run()+0x2c > > > ?????[0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > ?????????[0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB from > > > ?????????????[0x00007f67c08b7319] os::pd_create_stack_guard_pages(char*, > > > unsigned long)+0x29 > > > ?????????????[0x00007f67c0b56851] JavaThread::create_stack_guard_pages() > > > [clone .part.125]+0xa1 > > > ?????????????[0x00007f67c0b57d6e] JavaThread::run()+0x3e > > > ?????????????[0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > ?????????[0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB > > > > > > > Can we capture this in a test so we can tell that the tracking has > > improved? > > > > Thanks, > > David > > ----- > > > > Thanks, > > > > > > -Zhengyu > > > > > > From thomas.stuefe at gmail.com Thu Nov 16 09:45:44 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 16 Nov 2017 10:45:44 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <1510824184.3968.3.camel@redhat.com> References: <1510824184.3968.3.camel@redhat.com> Message-ID: On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf wrote: > On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: > > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes > > wrote: > > > > > Hi Zhengyu, > > > > > > I can't review this in detail as I'm not familiar with NMT workings, > but > > > have a couple of comments. > > > > > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > > > > > > > This is Linux only enhancement for now, can be extended for other > > > > platforms. (See bug for details) > > > > > > > > > > I'm concerned about unnecessary platform skew. I added some comments to > > > the bug. I think the mincore trick can be used on Solaris and AIX as > well - > > > but not on OS X / BSD. It may be that VirtualQuery can be used on > Windows > > > to get the same information - but I'm unclear if the memory states > support > > > what you're looking for. It would be good to extend this to other > platforms > > > that will support it. > > > > > > > Just 5c: > > > > I think this is definitely useful even with only the Linux implementation > > provided. > > +1 > > While plaform skew is a valid concern, I'm doubtful it's "unnecessary". > I think there is value to get one platform fixed and then look at other > platforms one by one as follow-up bugs. For some platforms it might be > hard to do (or undesirable) as Thomas says. > > Thanks, > Severin > I volunteer to implement this for windows if it can be done as non-time-critical follow up, if no-one else has stepped up in until then. ..Thomas > > On AIX we have commit-on-touch and hence mincore may actually force > commit > > the thread :P - would have to try it but have no high hopes. Well. > > > > On Windows this is possible, we already use VirtualQuery in our fork to > > print out the commit-state of the current thread stack in case of an > error. > > > > ..Thomas > > > > > > > And on a minor note can you please correct the existing spelling error > in > > > get_stack_commited_bottom. :) > > > > > > Current implementation, thread stack is tracked when thread is created, > > > > its available stack space is tagged as reserved and committed. > > > > > > > > However, this is not how stack works. Stack pages are committed on > > > > demand, so this approach overstates its memory usage. > > > > > > > > This enhancement gathers thread stack usage at NMT query time, so > that it > > > > can report more accurate usage. > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > > > > Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > > > > > > > > > > > Example: > > > > > > > > Summary shows thread stacks only use small faction of reserved space. > > > > > > > > - Thread (reserved=41216KB, committed=632KB) > > > > (thread #41) > > > > (stack: reserved=41032KB, > committed=448KB) > > > > (malloc=137KB #222) > > > > (arena=47KB #76) > > > > > > > > > > > > Detail tracking shows stack guards and actually used/committed stack > area. > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread > > > > Stack from > > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c > > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB > from > > > > [0x00007f67c08b7319] os::pd_create_stack_guard_ > pages(char*, > > > > unsigned long)+0x29 > > > > [0x00007f67c0b56851] JavaThread::create_stack_ > guard_pages() > > > > [clone .part.125]+0xa1 > > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e > > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB > > > > > > > > > > Can we capture this in a test so we can tell that the tracking has > > > improved? > > > > > > Thanks, > > > David > > > ----- > > > > > > Thanks, > > > > > > > > -Zhengyu > > > > > > > > > > From adinn at redhat.com Thu Nov 16 09:48:10 2017 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 16 Nov 2017 09:48:10 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: <37a2a4e1-9b1f-20ca-90c3-459093b2cd7e@redhat.com> On 15/11/17 22:25, Zhengyu Gu wrote: > This is Linux only enhancement for now, can be extended for other > platforms. (See bug for details) > > Current implementation, thread stack is tracked when thread is created, > its available stack space is tagged as reserved and committed. > > However, this is not how stack works. Stack pages are committed on > demand, so this approach overstates its memory usage. > > This enhancement gathers thread stack usage at NMT query time, so that > it can report more accurate usage. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html The fix looks good to me. It was very handy that get_stack_commited_bottom was already present in os_linux.cpp for other reasons -- although as David says it really ought to be spelled correctly with a double t in committed :-) It is a good idea to try to extend this to other Unix-variant OSes that support mincore and to Windows (by whatever other means) but I don't think that should stop this being added to Linux now. This is too useful and important to omit because . . . Footprint is a major concern for cloud deployments (where memory use is part of costing) and Linux underpins a significant proportion of cloud deployments. More accurate measurement of JVM footprint on Linux is going to be a great help to Java users when it comes to identifying and reducing the cost of cloud deployments. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From dms at samersoff.net Thu Nov 16 11:22:16 2017 From: dms at samersoff.net (Dmitry Samersoff) Date: Thu, 16 Nov 2017 14:22:16 +0300 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> Message-ID: <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> Ioi, Thank you! I did some additional testing and find that four tests fails the same way on both x86_64 and AARCH64 (see below). runtime/appcds/VerifierTest_0.java runtime/appcds/cacheObject/GCStressTest.java runtime/appcds/cacheObject/RedefineClassTest.java runtime/appcds/sharedStrings/InternSharedString.java Is it expected? test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'Please remove the unverifiable cl asses' missing from stdout/stderr Exception in thread "main" java.lang.RuntimeException: FAILED. GCStressApp_nonshared_string should not be shared at GCStressApp.main(GCStressApp.java:73) FAILED. buzzshould not be shared Exception in thread "main" java.lang.RuntimeException: Failed: unexpected shared string. at InternStringTest.main(InternStringTest.java:63) -Dmitry On 16.11.2017 03:10, Ioi Lam wrote: > I filed: > > https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg > VMProps to filter out CDS tests > > https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message > when CDS is not supported > > Thanks > > - Ioi > > > > On 11/15/17 4:01 PM, Ioi Lam wrote: >> Hi Dmitry, >> >> Thanks for the review. >> >> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >> >>> Ioi, >>> >>> I tested new patch on aarch64 and it works fine with the changes >>> below[1]. >> I've fixed it. Thanks for the patch. >>> Also tests doesn't work with exploded image - is it possible to check >>> for this case and produce appropriate error message? >> CDS is not supported on exploded images. I think the best thing to do >> is to add something like "@requires vm.cds" into the test cases >> (similar to the use of "@require vm.aot" in other tests). That way, >> none of the CDS tests will be executed for >> >> It's too late to do this in JDK 10 now, but I will file an RFE to do >> it in the next release. >> >> I'll also file another RFE to print a better message. Today if you use >> an exploded build the error message is kind of confusing: >> >> $ ./jdk/bin/java -Xshare:dump >> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >> Hint: enable -Xlog:class+path=info to diagnose the failure >> Error occurred during initialization of VM >> CDS allows only empty directories in archived classpaths >> >> It should say something like "CDS is not supported on exploded images". >> >> Thanks >> - Ioi >> >>> 1. >>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>> --- a/src/hotspot/share/classfile/classListParser.cpp?? Mon Nov 06 >>> 09:45:23 2017 +0000 >>> +++ b/src/hotspot/share/classfile/classListParser.cpp?? Mon Nov 06 >>> 15:02:51 2017 +0000 >>> @@ -31,7 +31,6 @@ >>> -#include "prims/jvm.h" >>> >>> diff -r dbf9cec6a568 >>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>> 06 09:45:23 2017 +0000 >>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>> 06 15:02:51 2017 +0000 >>> @@ -518,7 +518,7 @@ >>> >>> ????? { >>> ??????? MutexLocker mu(SystemDictionary_lock, THREAD); >>> -????? Klass* check = find_class(d_index, d_hash, name, dictionary); >>> +????? Klass* check = dictionary->find_class(d_index, d_hash, name); >>> ??????? if (check != NULL) { >>> ????????? return InstanceKlass::cast(check); >>> ??????? } >>> >>> -Dmitry >>> >>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>> Hi, >>>> >>>> Here's the new webrev for both the AppCDS implementation and tests. >>>> During internal review of the JEP, we have decided to integrate both >>>> implementation and tests at the same time. >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>> >>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>> result of copying the closed source files on top of files of the same >>>> name in the open repo. So in reviewing, instead of focusing on what's >>>> "changed", please focus on the entire content of the new version of >>>> each >>>> file. >>>> >>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>> sources) and all the new appcds tests passed. >>>> >>>> Thanks >>>> >>>> - Ioi >>>> >>>> >>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>> Hi Dmitry, >>>>> >>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>> is_modules_image. Please change the code accordingly. >>>>> >>>>> I will post my latest diff soon, with some test cases as well. >>>>> >>>>> Thanks >>>>> >>>>> - Ioi >>>>> >>>>> >>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>> Ioi, >>>>>> >>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>> the compilation fails with: >>>>>> >>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>> >>>>>> >>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>> >>>>>> Did I miss something? >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Please review this change set. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>> >>>>>>> ????? https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>> >>>>>>> This is the first step of implementing the following JEP, which >>>>>>> moves >>>>>>> AppCDS from >>>>>>> closed repos into the openjdk repo: >>>>>>> >>>>>>> ????? https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>> >>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>> repo. >>>>>>> As part >>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>> code >>>>>>> into the >>>>>>> open repo. >>>>>>> >>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>> possible. The >>>>>>> intention is >>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>> and not >>>>>>> to do any sort of refactoring. >>>>>>> >>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>> copying the >>>>>>> closed source >>>>>>> files on top of files of the same name in the open repo. So in >>>>>>> reviewing, instead of >>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>> content >>>>>>> of the new >>>>>>> version of each file. >>>>>>> >>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>> flag is >>>>>>> changed from >>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>> "commercial" >>>>>>> flags are not supported by the OpenJDK build. >>>>>>> >>>>>>> Source code refactoring may be desirable, because the old >>>>>>> open/closed >>>>>>> source >>>>>>> code structure had introduced some intermediary APIs to connect code >>>>>>> between >>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>> >>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>> tests >>>>>>> will be >>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>> >>>>>>> All the AppCDS tests (currently still in closed sources) passed with >>>>>>> both Oracle JDK >>>>>>> and OpenJDK. >>>>>>> >>>>>>> Thanks >>>>>>> - Ioi >> > From zgu at redhat.com Thu Nov 16 13:09:25 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 08:09:25 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: <774f2ece-0353-a87f-d715-06c110df63da@redhat.com> Hi David, Thanks for the suggestions. On 11/16/2017 01:58 AM, David Holmes wrote: > Hi Zhengyu, > > I can't review this in detail as I'm not familiar with NMT workings, but > have a couple of comments. > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >> This is Linux only enhancement for now, can be extended for other >> platforms. (See bug for details) > > I'm concerned about unnecessary platform skew. I added some comments to > the bug. I think the mincore trick can be used on Solaris and AIX as > well - but not on OS X / BSD. It may be that VirtualQuery can be used on > Windows to get the same information - but I'm unclear if the memory > states support what you're looking for. It would be good to extend this > to other platforms that will support it. Windows is definitely doable, I can add Windows support if it helps the case. But I don't have access to other platforms. > > And on a minor note can you please correct the existing spelling error > in get_stack_commited_bottom. :) > Will do. Thanks, -Zhengyu >> Current implementation, thread stack is tracked when thread is >> created, its available stack space is tagged as reserved and committed. >> >> However, this is not how stack works. Stack pages are committed on >> demand, so this approach overstates its memory usage. >> >> This enhancement gathers thread stack usage at NMT query time, so that >> it can report more accurate usage. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >> Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html >> >> >> Example: >> >> Summary shows thread stacks only use small faction of reserved space. >> >> - Thread (reserved=41216KB, committed=632KB) >> (thread #41) >> (stack: reserved=41032KB, committed=448KB) >> (malloc=137KB #222) >> (arena=47KB #76) >> >> >> Detail tracking shows stack guards and actually used/committed stack >> area. >> >> [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for Thread >> Stack from >> [0x00007f67c0b57d5c] JavaThread::run()+0x2c >> [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> >> [0x00007f66e95e7000 - 0x00007f66e95eb000] committed 16KB from >> [0x00007f67c08b7319] >> os::pd_create_stack_guard_pages(char*, unsigned long)+0x29 >> [0x00007f67c0b56851] >> JavaThread::create_stack_guard_pages() [clone .part.125]+0xa1 >> [0x00007f67c0b57d6e] JavaThread::run()+0x3e >> [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> >> [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB > > Can we capture this in a test so we can tell that the tracking has > improved? > > Thanks, > David > ----- > >> Thanks, >> >> -Zhengyu >> From zgu at redhat.com Thu Nov 16 13:25:56 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 08:25:56 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: <1510824184.3968.3.camel@redhat.com> Message-ID: <3e85a8de-2dff-b08a-aa56-c07eb043e5e8@redhat.com> Hi Thomas, On 11/16/2017 04:45 AM, Thomas St?fe wrote: > > > On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf > wrote: > > On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: > > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes > > > wrote: > > > > > Hi Zhengyu, > > > > > > I can't review this in detail as I'm not familiar with NMT workings, but > > > have a couple of comments. > > > > > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > > > > > > > This is Linux only enhancement for now, can be extended for other > > > > platforms. (See bug for details) > > > > > > > > > > I'm concerned about unnecessary platform skew. I added some comments to > > > the bug. I think the mincore trick can be used on Solaris and AIX as well - > > > but not on OS X / BSD. It may be that VirtualQuery can be used on Windows > > > to get the same information - but I'm unclear if the memory states support > > > what you're looking for. It would be good to extend this to other platforms > > > that will support it. > > > > > > > Just 5c: > > > > I think this is definitely useful even with only the Linux implementation > > provided. > > +1 > > While plaform skew is a valid concern, I'm doubtful it's "unnecessary". > I think there is value to get one platform fixed and then look at other > platforms one by one as follow-up bugs. For some platforms it might be > hard to do (or undesirable) as Thomas says. > > Thanks, > Severin > > > I volunteer to implement this for windows if it can be done as > non-time-critical follow up, if no-one else has stepped up in until then. I can work on Windows solution today if it helps the case :-) Thanks, -Zhengyu > > ..Thomas > > > > On AIX we have commit-on-touch and hence mincore may actually > force commit > > the thread :P - would have to try it but have no high hopes. Well. > > > > On Windows this is possible, we already use VirtualQuery in our > fork to > > print out the commit-state of the current thread stack in case of > an error. > > > > ..Thomas > > > > > > > And on a minor note can you please correct the existing > spelling error in > > > get_stack_commited_bottom. :) > > > > > > Current implementation, thread stack is tracked when thread is > created, > > > > its available stack space is tagged as reserved and committed. > > > > > > > > However, this is not how stack works. Stack pages are > committed on > > > > demand, so this approach overstates its memory usage. > > > > > > > > This enhancement gathers thread stack usage at NMT query > time, so that it > > > > can report more accurate usage. > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > > > > > Webrev: > http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > > > > > > > > > > > > Example: > > > > > > > > Summary shows thread stacks only use small faction of > reserved space. > > > > > > > > - Thread (reserved=41216KB, committed=632KB) > > > > (thread #41) > > > > (stack: reserved=41032KB, > committed=448KB) > > > > (malloc=137KB #222) > > > > (arena=47KB #76) > > > > > > > > > > > > Detail tracking shows stack guards and actually > used/committed stack area. > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for > Thread > > > > Stack from > > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c > > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed > 16KB from > > > > [0x00007f67c08b7319] > os::pd_create_stack_guard_pages(char*, > > > > unsigned long)+0x29 > > > > [0x00007f67c0b56851] > JavaThread::create_stack_guard_pages() > > > > [clone .part.125]+0xa1 > > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e > > > > [0x00007f67c08bbea2] > thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB > > > > > > > > > > Can we capture this in a test so we can tell that the tracking has > > > improved? > > > > > > Thanks, > > > David > > > ----- > > > > > > Thanks, > > > > > > > > -Zhengyu > > > > > > > > > > From thomas.stuefe at gmail.com Thu Nov 16 13:29:03 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 16 Nov 2017 14:29:03 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <3e85a8de-2dff-b08a-aa56-c07eb043e5e8@redhat.com> References: <1510824184.3968.3.camel@redhat.com> <3e85a8de-2dff-b08a-aa56-c07eb043e5e8@redhat.com> Message-ID: On Thu, Nov 16, 2017 at 2:25 PM, Zhengyu Gu wrote: > Hi Thomas, > > On 11/16/2017 04:45 AM, Thomas St?fe wrote: > >> >> >> On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf > > wrote: >> >> On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: >> > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes < >> david.holmes at oracle.com > >> >> > wrote: >> > >> > > Hi Zhengyu, >> > > >> > > I can't review this in detail as I'm not familiar with NMT >> workings, but >> > > have a couple of comments. >> > > >> > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >> > > >> > > > This is Linux only enhancement for now, can be extended for >> other >> > > > platforms. (See bug for details) >> > > > >> > > >> > > I'm concerned about unnecessary platform skew. I added some >> comments to >> > > the bug. I think the mincore trick can be used on Solaris and AIX >> as well - >> > > but not on OS X / BSD. It may be that VirtualQuery can be used on >> Windows >> > > to get the same information - but I'm unclear if the memory >> states support >> > > what you're looking for. It would be good to extend this to other >> platforms >> > > that will support it. >> > > >> > >> > Just 5c: >> > >> > I think this is definitely useful even with only the Linux >> implementation >> > provided. >> >> +1 >> >> While plaform skew is a valid concern, I'm doubtful it's >> "unnecessary". >> I think there is value to get one platform fixed and then look at >> other >> platforms one by one as follow-up bugs. For some platforms it might be >> hard to do (or undesirable) as Thomas says. >> >> Thanks, >> Severin >> >> >> I volunteer to implement this for windows if it can be done as >> non-time-critical follow up, if no-one else has stepped up in until then. >> > > I can work on Windows solution today if it helps the case :-) > > Thanks, > > -Zhengyu > > > :) Sure, fine by me! ..Thomas > > >> ..Thomas >> >> >> > On AIX we have commit-on-touch and hence mincore may actually >> force commit >> > the thread :P - would have to try it but have no high hopes. Well. >> > >> > On Windows this is possible, we already use VirtualQuery in our >> fork to >> > print out the commit-state of the current thread stack in case of >> an error. >> > >> > ..Thomas >> > >> > >> > > And on a minor note can you please correct the existing >> spelling error in >> > > get_stack_commited_bottom. :) >> > > >> > > Current implementation, thread stack is tracked when thread is >> created, >> > > > its available stack space is tagged as reserved and committed. >> > > > >> > > > However, this is not how stack works. Stack pages are >> committed on >> > > > demand, so this approach overstates its memory usage. >> > > > >> > > > This enhancement gathers thread stack usage at NMT query >> time, so that it >> > > > can report more accurate usage. >> > > > >> > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >> >> > > > Webrev: >> http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html >> >> > > > >> > > > >> > > > Example: >> > > > >> > > > Summary shows thread stacks only use small faction of >> reserved space. >> > > > >> > > > - Thread (reserved=41216KB, committed=632KB) >> > > > (thread #41) >> > > > (stack: reserved=41032KB, >> committed=448KB) >> > > > (malloc=137KB #222) >> > > > (arena=47KB #76) >> > > > >> > > > >> > > > Detail tracking shows stack guards and actually >> used/committed stack area. >> > > > >> > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for >> Thread >> > > > Stack from >> > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c >> > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> > > > >> > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed >> 16KB from >> > > > [0x00007f67c08b7319] >> os::pd_create_stack_guard_pages(char*, >> > > > unsigned long)+0x29 >> > > > [0x00007f67c0b56851] >> JavaThread::create_stack_guard_pages() >> > > > [clone .part.125]+0xa1 >> > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e >> > > > [0x00007f67c08bbea2] >> thread_native_entry(Thread*)+0x112 >> > > > >> > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] committed >> 4KB >> > > > >> > > >> > > Can we capture this in a test so we can tell that the tracking >> has >> > > improved? >> > > >> > > Thanks, >> > > David >> > > ----- >> > > >> > > Thanks, >> > > > >> > > > -Zhengyu >> > > > >> > > > >> >> >> From harold.seigel at oracle.com Thu Nov 16 13:38:35 2017 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 16 Nov 2017 08:38:35 -0500 Subject: RFR 8167372: Add code to check for getting oops while thread is in native Message-ID: Hi, Please review this JDK-10 enhancement for bug JDK-8167372.? As described in the bug, this fix helps check for naked oops.? The fix was tested with JCK lang and VM tests, JTReg hotspot and many JTReg JDK tests, M5 tier1 - tier5 tests, and JPRT. Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8167372 Thanks, Harold From zgu at redhat.com Thu Nov 16 14:01:57 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 09:01:57 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <37a2a4e1-9b1f-20ca-90c3-459093b2cd7e@redhat.com> References: <37a2a4e1-9b1f-20ca-90c3-459093b2cd7e@redhat.com> Message-ID: <48a5dc48-2824-7a87-a441-2f5e80a88c50@redhat.com> Thanks for the quick review, Andrew! -Zhengyu On 11/16/2017 04:48 AM, Andrew Dinn wrote: > On 15/11/17 22:25, Zhengyu Gu wrote: >> This is Linux only enhancement for now, can be extended for other >> platforms. (See bug for details) >> >> Current implementation, thread stack is tracked when thread is created, >> its available stack space is tagged as reserved and committed. >> >> However, this is not how stack works. Stack pages are committed on >> demand, so this approach overstates its memory usage. >> >> This enhancement gathers thread stack usage at NMT query time, so that >> it can report more accurate usage. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >> Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > The fix looks good to me. > > It was very handy that get_stack_commited_bottom was already present in > os_linux.cpp for other reasons -- although as David says it really ought > to be spelled correctly with a double t in committed :-) > > It is a good idea to try to extend this to other Unix-variant OSes that > support mincore and to Windows (by whatever other means) but I don't > think that should stop this being added to Linux now. This is too useful > and important to omit because . . . > > Footprint is a major concern for cloud deployments (where memory use is > part of costing) and Linux underpins a significant proportion of cloud > deployments. More accurate measurement of JVM footprint on Linux is > going to be a great help to Java users when it comes to identifying and > reducing the cost of cloud deployments. > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From george.triantafillou at oracle.com Thu Nov 16 15:01:38 2017 From: george.triantafillou at oracle.com (George Triantafillou) Date: Thu, 16 Nov 2017 10:01:38 -0500 Subject: RFR 8167372: Add code to check for getting oops while thread is in native In-Reply-To: References: Message-ID: <7475e0fc-d565-7648-ed39-2cb545198296@oracle.com> Hi Harold, Looks good. -George On 11/16/2017 8:38 AM, harold seigel wrote: > Hi, > > Please review this JDK-10 enhancement for bug JDK-8167372.? As > described in the bug, this fix helps check for naked oops.? The fix > was tested with JCK lang and VM tests, JTReg hotspot and many JTReg > JDK tests, M5 tier1 - tier5 tests, and JPRT. > > Open Webrev: > http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html > > JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8167372 > > Thanks, Harold > From shade at redhat.com Thu Nov 16 15:06:59 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 16 Nov 2017 16:06:59 +0100 Subject: RFR 8167372: Add code to check for getting oops while thread is in native In-Reply-To: References: Message-ID: <96631f21-5266-59a7-bd70-cfbb2d7dfa37@redhat.com> On 11/16/2017 02:38 PM, harold seigel wrote: > Hi, > > Please review this JDK-10 enhancement for bug JDK-8167372.? As described in the bug, this fix helps > check for naked oops.? The fix was tested with JCK lang and VM tests, JTReg hotspot and many JTReg > JDK tests, M5 tier1 - tier5 tests, and JPRT. > > Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html This looks good. It probably makes sense to assert that JNIHandle::resolve path that unpacks the naked oops also has thread in proper state? This would expose failures like in JDK-8191227 [1] better. Thanks, -Aleksey [1] https://bugs.openjdk.java.net/browse/JDK-8191227 From bob.vandette at oracle.com Thu Nov 16 15:16:06 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Thu, 16 Nov 2017 10:16:06 -0500 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> Message-ID: Roman, It looks like this change may have caused the build of the minimal VM to fail. The minimal VM is no longer a configuration that we regularly build and test but it is still a build option that can be selected via "--with-jvm-variants=minimal1" /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: In static member function 'static jint GCArguments::initialize()': /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: error: 'defaultStream' has not been declared jio_fprintf(defaultStream::error_stream(), "UseParallelGC not supported in this VM.\n"); ^ /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: error: 'defaultStream' has not been declared jio_fprintf(defaultStream::error_stream(), "UseG1GC not supported in this VM.\n"); ^ /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: error: 'defaultStream' has not been declared jio_fprintf(defaultStream::error_stream(), "UseConcMarkSweepGC not supported in this VM.\n"); ^ gmake[3]: *** [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] Error 1 Bob. > On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: > > Hi Per & Erik, > > thanks for the reviews! > > Now I need a sponsor. > > Final webrev (same as before, including Reviewed-by line): > http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ > > Thanks, Roman > > >> Looks good Roman! >> >> /Per >> >> On 2017-11-06 12:13, Roman Kennke wrote: >>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>> Hi, >>>> >>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>> [...] >>>>>> So let me just put your changes up for review (again), if you don't >>>>>> mind: >>>>>> >>>>>> Full webrev: >>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>> >>>> >>>> I like this. Just two naming suggestions: >>>> >>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>> >>>> 39 static jint create_instance(); >>>> 40 static bool is_initialized(); >>>> 41 static GCArguments* instance(); >>>> >>>> Could we call these: >>>> >>>> 39 static jint initialize(); >>>> 40 static bool is_initialized(); >>>> 41 static GCArguments* arguments(); >>>> >>>> Reasoning: initialize() maps better to its companion is_initialized(). >>>> GCArguments::arguments() maps better to the existing patterns we have >>>> with CollectedHeap::heap(). >>> Ok, that sounds good to me. Actually, it's almost full-circle back to my >>> original proposal ;-) >>> >>> Differential: >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>> >>> Full: >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>> >>> >>> Ok to push now? >>> >>> Roman > > From aph at redhat.com Thu Nov 16 15:46:01 2017 From: aph at redhat.com (Andrew Haley) Date: Thu, 16 Nov 2017 15:46:01 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: On 16/11/17 06:58, David Holmes wrote: > I can't review this in detail as I'm not familiar with NMT workings, but > have a couple of comments. > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >> This is Linux only enhancement for now, can be extended for other >> platforms. (See bug for details) > > I'm concerned about unnecessary platform skew. I added some comments to > the bug. I think the mincore trick can be used on Solaris and AIX as > well - but not on OS X / BSD. It may be that VirtualQuery can be used on > Windows to get the same information - but I'm unclear if the memory > states support what you're looking for. It would be good to extend this > to other platforms that will support it. Sure, but as OpenJDK becomes more open and hopefully more diverse, it becomes less and less practical to keep everything in step. I favour the classic free software approach, which is to encourage people to create new features and everyone else can fill in the gaps, depending on how much time and effort is available. It's this approach that allows GCC to support a vast range of targets (and hosts) while not holding back development. I have consistently argued for this in the Governing Board. So, let's get this done. It's very important for container memory monitoring, which right now usually means Linux. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From rkennke at redhat.com Thu Nov 16 16:17:30 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 16 Nov 2017 17:17:30 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> Message-ID: <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> Hi Bob, thanks for letting me know. I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 and posted for review: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html Thanks, Roman > Roman, > > It looks like this change may have caused the build of the minimal VM to > fail. The minimal VM is no longer a configuration that we regularly build and test > but it is still a build option that can be selected via "--with-jvm-variants=minimal1" > > > /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: In static member function 'static jint GCArguments::initialize()': > /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: error: 'defaultStream' has not been declared > jio_fprintf(defaultStream::error_stream(), "UseParallelGC not supported in this VM.\n"); > ^ > /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: error: 'defaultStream' has not been declared > jio_fprintf(defaultStream::error_stream(), "UseG1GC not supported in this VM.\n"); > ^ > /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: error: 'defaultStream' has not been declared > jio_fprintf(defaultStream::error_stream(), "UseConcMarkSweepGC not supported in this VM.\n"); > ^ > gmake[3]: *** [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] Error 1 > > Bob. > > >> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >> >> Hi Per & Erik, >> >> thanks for the reviews! >> >> Now I need a sponsor. >> >> Final webrev (same as before, including Reviewed-by line): >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >> >> Thanks, Roman >> >> >>> Looks good Roman! >>> >>> /Per >>> >>> On 2017-11-06 12:13, Roman Kennke wrote: >>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>> Hi, >>>>> >>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>> [...] >>>>>>> So let me just put your changes up for review (again), if you don't >>>>>>> mind: >>>>>>> >>>>>>> Full webrev: >>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>> >>>>> I like this. Just two naming suggestions: >>>>> >>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>> >>>>> 39 static jint create_instance(); >>>>> 40 static bool is_initialized(); >>>>> 41 static GCArguments* instance(); >>>>> >>>>> Could we call these: >>>>> >>>>> 39 static jint initialize(); >>>>> 40 static bool is_initialized(); >>>>> 41 static GCArguments* arguments(); >>>>> >>>>> Reasoning: initialize() maps better to its companion is_initialized(). >>>>> GCArguments::arguments() maps better to the existing patterns we have >>>>> with CollectedHeap::heap(). >>>> Ok, that sounds good to me. Actually, it's almost full-circle back to my >>>> original proposal ;-) >>>> >>>> Differential: >>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>> >>>> Full: >>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>> >>>> >>>> Ok to push now? >>>> >>>> Roman >> From bob.vandette at oracle.com Thu Nov 16 16:33:12 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Thu, 16 Nov 2017 11:33:12 -0500 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> Message-ID: Yes, that?s the fix. I built it correctly with that added include. There are still a few other issues with building and using the minimal VM but those existed before your change. 1. You cannot build the minimal VM without also building the server VM. There is a Makefile dependency on the server VM. 2. The jvm.cfg produced in the jvm and jre images does not add the minimalvm as one of the possible VM to select. As a work-around jlink produces a workable jvm.cfg. Thanks, Bob. > On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: > > Hi Bob, > thanks for letting me know. > > I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 > and posted for review: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html > > Thanks, Roman > >> Roman, >> >> It looks like this change may have caused the build of the minimal VM to >> fail. The minimal VM is no longer a configuration that we regularly build and test >> but it is still a build option that can be selected via "--with-jvm-variants=minimal1" >> >> >> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: In static member function 'static jint GCArguments::initialize()': >> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: error: 'defaultStream' has not been declared >> jio_fprintf(defaultStream::error_stream(), "UseParallelGC not supported in this VM.\n"); >> ^ >> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: error: 'defaultStream' has not been declared >> jio_fprintf(defaultStream::error_stream(), "UseG1GC not supported in this VM.\n"); >> ^ >> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: error: 'defaultStream' has not been declared >> jio_fprintf(defaultStream::error_stream(), "UseConcMarkSweepGC not supported in this VM.\n"); >> ^ >> gmake[3]: *** [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] Error 1 >> >> Bob. >> >> >>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>> >>> Hi Per & Erik, >>> >>> thanks for the reviews! >>> >>> Now I need a sponsor. >>> >>> Final webrev (same as before, including Reviewed-by line): >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>> >>> Thanks, Roman >>> >>> >>>> Looks good Roman! >>>> >>>> /Per >>>> >>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>> Hi, >>>>>> >>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>> [...] >>>>>>>> So let me just put your changes up for review (again), if you don't >>>>>>>> mind: >>>>>>>> >>>>>>>> Full webrev: >>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>> >>>>>> I like this. Just two naming suggestions: >>>>>> >>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>> >>>>>> 39 static jint create_instance(); >>>>>> 40 static bool is_initialized(); >>>>>> 41 static GCArguments* instance(); >>>>>> >>>>>> Could we call these: >>>>>> >>>>>> 39 static jint initialize(); >>>>>> 40 static bool is_initialized(); >>>>>> 41 static GCArguments* arguments(); >>>>>> >>>>>> Reasoning: initialize() maps better to its companion is_initialized(). >>>>>> GCArguments::arguments() maps better to the existing patterns we have >>>>>> with CollectedHeap::heap(). >>>>> Ok, that sounds good to me. Actually, it's almost full-circle back to my >>>>> original proposal ;-) >>>>> >>>>> Differential: >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>> >>>>> Full: >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>> >>>>> >>>>> Ok to push now? >>>>> >>>>> Roman >>> > From ioi.lam at oracle.com Thu Nov 16 17:24:41 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 16 Nov 2017 09:24:41 -0800 Subject: RFR(XXS) JDK-8191042 Deprecate VM option CheckEndorsedAndExtDirs In-Reply-To: References: Message-ID: <515de28f-9b50-2a5c-569b-3b2f7289351b@oracle.com> Hi David, Thanks for the review. The CSR has been approved, so I will push this. Thanks - Ioi On 11/15/17 6:19 PM, David Holmes wrote: > Hi Ioi, > > Looks good. > > Just need to wait for the CSR to be approved before pushing. > > Thanks, > David > > On 16/11/2017 8:04 AM, Ioi Lam wrote: >> Hi, please review this very small change: >> >> ???? https://bugs.openjdk.java.net/browse/JDK-8191042 >> >> The endorsed standards override mechanism and the extensions mechanism >> were removed in JDK 9 via JEP 220. >> >> The CheckEndorsedAndExtDirs option is disabled by default. However, if >> it's enabled in the command-line, the VM would check the endorsed and >> ext >> directories so that the VM terminates if they exist. These checks are no >> longer needed in JDK 10 and beyond and can be removed. >> >> This task is the first step in the removal -- we need to mark this >> option >> as deprecated and print a warning message when it's used. The option >> itself, and the code that uses it, will be removed in a subsequent JDK >> release. >> >> Tested with hotspot tier1/2 >> >> Thanks >> - Ioi >> >> =================================== >> >> diff -r d76a6042f5d7 src/hotspot/share/runtime/arguments.cpp >> --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 08 09:03:24 >> 2017 -0800 >> +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 15 14:00:58 >> 2017 -0800 >> @@ -383,6 +383,7 @@ >> ??? { "MinRAMFraction",?????????????? JDK_Version::jdk(10), >> JDK_Version::undefined(), JDK_Version::undefined() }, >> ??? { "InitialRAMFraction",?????????? JDK_Version::jdk(10), >> JDK_Version::undefined(), JDK_Version::undefined() }, >> ??? { "IgnoreUnverifiableClassesDuringDump", JDK_Version::jdk(10), >> JDK_Version::undefined(), JDK_Version::undefined() }, >> +? { "CheckEndorsedAndExtDirs",????? JDK_Version::jdk(10), >> JDK_Version::undefined(), JDK_Version::undefined() }, >> >> ??? // --- Deprecated alias flags (see also aliased_jvm_flags) - >> sorted by obsolete_in then expired_in: >> ??? { "DefaultMaxRAMFraction",??????? JDK_Version::jdk(8), >> JDK_Version::undefined(), JDK_Version::undefined() }, >> From zgu at redhat.com Thu Nov 16 18:30:00 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 13:30:00 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: <1510824184.3968.3.camel@redhat.com> Message-ID: Hi, I added Windows support and updated bug to make this enhancement Linux and Windows only. As Thomas suggested, other platforms can be addressed by followup fixes. Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ I reran hotspot_tier1_runtime on Linux x64 and Windows 10 (fastdebug + release) Thanks, -Zhengyu On 11/16/2017 04:45 AM, Thomas St?fe wrote: > > > On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf > wrote: > > On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: > > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes > > > wrote: > > > > > Hi Zhengyu, > > > > > > I can't review this in detail as I'm not familiar with NMT workings, but > > > have a couple of comments. > > > > > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: > > > > > > > This is Linux only enhancement for now, can be extended for other > > > > platforms. (See bug for details) > > > > > > > > > > I'm concerned about unnecessary platform skew. I added some comments to > > > the bug. I think the mincore trick can be used on Solaris and AIX as well - > > > but not on OS X / BSD. It may be that VirtualQuery can be used on Windows > > > to get the same information - but I'm unclear if the memory states support > > > what you're looking for. It would be good to extend this to other platforms > > > that will support it. > > > > > > > Just 5c: > > > > I think this is definitely useful even with only the Linux implementation > > provided. > > +1 > > While plaform skew is a valid concern, I'm doubtful it's "unnecessary". > I think there is value to get one platform fixed and then look at other > platforms one by one as follow-up bugs. For some platforms it might be > hard to do (or undesirable) as Thomas says. > > Thanks, > Severin > > > I volunteer to implement this for windows if it can be done as > non-time-critical follow up, if no-one else has stepped up in until then. > > ..Thomas > > > > On AIX we have commit-on-touch and hence mincore may actually > force commit > > the thread :P - would have to try it but have no high hopes. Well. > > > > On Windows this is possible, we already use VirtualQuery in our > fork to > > print out the commit-state of the current thread stack in case of > an error. > > > > ..Thomas > > > > > > > And on a minor note can you please correct the existing > spelling error in > > > get_stack_commited_bottom. :) > > > > > > Current implementation, thread stack is tracked when thread is > created, > > > > its available stack space is tagged as reserved and committed. > > > > > > > > However, this is not how stack works. Stack pages are > committed on > > > > demand, so this approach overstates its memory usage. > > > > > > > > This enhancement gathers thread stack usage at NMT query > time, so that it > > > > can report more accurate usage. > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 > > > > > Webrev: > http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html > > > > > > > > > > > > > Example: > > > > > > > > Summary shows thread stacks only use small faction of > reserved space. > > > > > > > > - Thread (reserved=41216KB, committed=632KB) > > > > (thread #41) > > > > (stack: reserved=41032KB, > committed=448KB) > > > > (malloc=137KB #222) > > > > (arena=47KB #76) > > > > > > > > > > > > Detail tracking shows stack guards and actually > used/committed stack area. > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for > Thread > > > > Stack from > > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c > > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed > 16KB from > > > > [0x00007f67c08b7319] > os::pd_create_stack_guard_pages(char*, > > > > unsigned long)+0x29 > > > > [0x00007f67c0b56851] > JavaThread::create_stack_guard_pages() > > > > [clone .part.125]+0xa1 > > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e > > > > [0x00007f67c08bbea2] > thread_native_entry(Thread*)+0x112 > > > > > > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] committed 4KB > > > > > > > > > > Can we capture this in a test so we can tell that the tracking has > > > improved? > > > > > > Thanks, > > > David > > > ----- > > > > > > Thanks, > > > > > > > > -Zhengyu > > > > > > > > > > From harold.seigel at oracle.com Thu Nov 16 19:42:22 2017 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 16 Nov 2017 14:42:22 -0500 Subject: RFR 8167372: Add code to check for getting oops while thread is in native In-Reply-To: <7475e0fc-d565-7648-ed39-2cb545198296@oracle.com> References: <7475e0fc-d565-7648-ed39-2cb545198296@oracle.com> Message-ID: <99ca035b-4931-ac4e-9369-25bf9069c500@oracle.com> Thanks George for the review! Harold On 11/16/2017 10:01 AM, George Triantafillou wrote: > Hi Harold, > > Looks good. > > -George > > On 11/16/2017 8:38 AM, harold seigel wrote: >> Hi, >> >> Please review this JDK-10 enhancement for bug JDK-8167372.? As >> described in the bug, this fix helps check for naked oops.? The fix >> was tested with JCK lang and VM tests, JTReg hotspot and many JTReg >> JDK tests, M5 tier1 - tier5 tests, and JPRT. >> >> Open Webrev: >> http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html >> >> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8167372 >> >> Thanks, Harold >> > From harold.seigel at oracle.com Thu Nov 16 19:51:40 2017 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 16 Nov 2017 14:51:40 -0500 Subject: RFR 8167372: Add code to check for getting oops while thread is in native In-Reply-To: <96631f21-5266-59a7-bd70-cfbb2d7dfa37@redhat.com> References: <96631f21-5266-59a7-bd70-cfbb2d7dfa37@redhat.com> Message-ID: Hi Aleksey, Thanks for the suggestion.? I added an assert, that checks for the proper thread state, to JNIHandles::resolve_impl() but was then unable to build because of JDK-8191227 . So I'm withdrawing this fix until that bug is fixed. I added the text of this new change, showing the added assert, as a comment to JDK-8167372. Thanks, Harold On 11/16/2017 10:06 AM, Aleksey Shipilev wrote: > On 11/16/2017 02:38 PM, harold seigel wrote: >> Hi, >> >> Please review this JDK-10 enhancement for bug JDK-8167372.? As described in the bug, this fix helps >> check for naked oops.? The fix was tested with JCK lang and VM tests, JTReg hotspot and many JTReg >> JDK tests, M5 tier1 - tier5 tests, and JPRT. >> >> Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html > This looks good. > > It probably makes sense to assert that JNIHandle::resolve path that unpacks the naked oops also has > thread in proper state? This would expose failures like in JDK-8191227 [1] better. > > Thanks, > -Aleksey > > [1] https://bugs.openjdk.java.net/browse/JDK-8191227 > From zgu at redhat.com Thu Nov 16 20:42:15 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 15:42:15 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: <1510824184.3968.3.camel@redhat.com> Message-ID: Sorry, I have to withdraw this code review request, since the stack sizes reported on Linux do not match /proc//smaps. I am debugging the problem, will submit new request later. Thanks, -Zhengyu On 11/16/2017 01:30 PM, Zhengyu Gu wrote: > Hi, > > I added Windows support and updated bug to make this enhancement Linux > and Windows only. > > As Thomas suggested, other platforms can be addressed by followup fixes. > > Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ > > I reran hotspot_tier1_runtime on Linux x64 and Windows 10 (fastdebug + > release) > > Thanks, > > -Zhengyu > > > > On 11/16/2017 04:45 AM, Thomas St?fe wrote: >> >> >> On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf > > wrote: >> >> On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: >> > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes >> > >> > wrote: >> > >> > > Hi Zhengyu, >> > > >> > > I can't review this in detail as I'm not familiar with NMT >> workings, but >> > > have a couple of comments. >> > > >> > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >> > > >> > > > This is Linux only enhancement for now, can be extended for >> other >> > > > platforms. (See bug for details) >> > > > >> > > >> > > I'm concerned about unnecessary platform skew. I added some >> comments to >> > > the bug. I think the mincore trick can be used on Solaris and >> AIX as well - >> > > but not on OS X / BSD. It may be that VirtualQuery can be used >> on Windows >> > > to get the same information - but I'm unclear if the memory >> states support >> > > what you're looking for. It would be good to extend this to >> other platforms >> > > that will support it. >> > > >> > >> > Just 5c: >> > >> > I think this is definitely useful even with only the Linux >> implementation >> > provided. >> >> +1 >> >> While plaform skew is a valid concern, I'm doubtful it's >> "unnecessary". >> I think there is value to get one platform fixed and then look at >> other >> platforms one by one as follow-up bugs. For some platforms it >> might be >> hard to do (or undesirable) as Thomas says. >> >> Thanks, >> Severin >> >> >> I volunteer to implement this for windows if it can be done as >> non-time-critical follow up, if no-one else has stepped up in until then. >> >> ..Thomas >> >> >> > On AIX we have commit-on-touch and hence mincore may actually >> force commit >> > the thread :P - would have to try it but have no high hopes. Well. >> > >> > On Windows this is possible, we already use VirtualQuery in our >> fork to >> > print out the commit-state of the current thread stack in case of >> an error. >> > >> > ..Thomas >> > >> > >> > > And on a minor note can you please correct the existing >> spelling error in >> > > get_stack_commited_bottom. :) >> > > >> > > Current implementation, thread stack is tracked when thread is >> created, >> > > > its available stack space is tagged as reserved and committed. >> > > > >> > > > However, this is not how stack works. Stack pages are >> committed on >> > > > demand, so this approach overstates its memory usage. >> > > > >> > > > This enhancement gathers thread stack usage at NMT query >> time, so that it >> > > > can report more accurate usage. >> > > > >> > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >> >> > > > Webrev: >> http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html >> >> > > > >> > > > >> > > > Example: >> > > > >> > > > Summary shows thread stacks only use small faction of >> reserved space. >> > > > >> > > > - Thread (reserved=41216KB, >> committed=632KB) >> > > > (thread #41) >> > > > (stack: reserved=41032KB, >> committed=448KB) >> > > > (malloc=137KB #222) >> > > > (arena=47KB #76) >> > > > >> > > > >> > > > Detail tracking shows stack guards and actually >> used/committed stack area. >> > > > >> > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for >> Thread >> > > > Stack from >> > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c >> > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >> > > > >> > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed >> 16KB from >> > > > [0x00007f67c08b7319] >> os::pd_create_stack_guard_pages(char*, >> > > > unsigned long)+0x29 >> > > > [0x00007f67c0b56851] >> JavaThread::create_stack_guard_pages() >> > > > [clone .part.125]+0xa1 >> > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e >> > > > [0x00007f67c08bbea2] >> thread_native_entry(Thread*)+0x112 >> > > > >> > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] >> committed 4KB >> > > > >> > > >> > > Can we capture this in a test so we can tell that the >> tracking has >> > > improved? >> > > >> > > Thanks, >> > > David >> > > ----- >> > > >> > > Thanks, >> > > > >> > > > -Zhengyu >> > > > >> > > > >> >> From coleen.phillimore at oracle.com Thu Nov 16 23:08:35 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 16 Nov 2017 18:08:35 -0500 Subject: RFR(8u): 8038636, 8055008, 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links ...and 8057570. In-Reply-To: References: Message-ID: Hi Kevin, These changes look good.? It's nice to have the simpler version of the InstanceKlass::_previous_versions in jdk8.? There was a bug tail so there may be more changes after 8040237.? It's worth following through all the bugs to get something close to what we have in jdk9/10.? But this change looks good so far. Thanks, Coleen On 11/13/17 1:08 PM, Kevin Walls wrote: > Hi, > > I'd like to get a hotspot review of these backports from 9 to 8u. This > is mainly runtime territory but some of the history is from the > compiler side. > > webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ > > The one we need is: > > 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links > jbs: https://bugs.openjdk.java.net/browse/JDK-8156137 > 9 changeset: > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/882e8cda60b3 > > The 9 changeset is short, just changing klass.cpp, but not possible in > 8 without additional work, so... > > 1. > > 8038636: speculative traps break when classes are redefined > https://bugs.openjdk.java.net/browse/JDK-8038636 > > 9 changeset: > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/a7784ddacbef > > 9 review thread: > http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2014-April/013948.html > (in 9 since April 2014) > > That one imports cleanly into 8u. > > > 2. > > With that imported, we need: > > 8055008:Clean up code that saves the previous versions of redefined > classes > https://bugs.openjdk.java.net/browse/JDK-8055008 > > 9 changeset: > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/e3fb51ae8d7d > > This is the change to stop using PreviousVersionNode and get an > InstanceKlass* from previous_versions(). > > > My webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ > > ...is 8055008 and 8156137 in 8u. The klass.cpp change here is > 8156137.? The rest is 8055008, so I can commit them separately. > > 8156137 doesn't have its own test, but I have found if you get this > area wrong, the test runtime/RedefineObject crashes. > > > Notes: > For 8055008 there was a change in > src/share/vm/classfile/classLoaderData.cpp which is already in 8u. > > src/share/vm/classfile/metadataOnStackMark.cpp: > +MetadataOnStackMark::MetadataOnStackMark(bool has_redefined_a_class) { > > The change is to stop using JvmtiExport::has_redefined_a_class() here > which is already in 8u at this point, but the param is already there > with a different name, so renamed as per 8055008. > > universe.cpp: had a change in 8055008 but following that there was: > > 8057570: RedefineClasses() tests fail > assert(((Metadata*)obj)->is_valid()) failed: obj is valid > https://bugs.openjdk.java.net/browse/JDK-8057570 > 9 changeset: > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/479ed4234a9d > > ..this puts back a few lines in nmethod.cpp which 8055008 removed: > > + // Call function Method*, not embedded in these other places. > + if (_method != NULL) f(_method); > > ..and in universe.cpp: > > + // Make the dependent methods not entrant (in VM_Deoptimize they are > made zombies) > + CodeCache::make_marked_nmethods_not_entrant(); > > ..and removes: CodeCache::make_marked_nmethods_zombies(); > > My webrev takes these into account. > > > Then there are 8038636's two follow-on bugs which I'd like to > follow-up in a separate thread. > 8039960 is a test change > 8040237: nsk/jvmti/RetransformClasses/retransform001 crashed the VM on > all platforms when run with with -server -Xcomp > https://bugs.openjdk.java.net/browse/JDK-8040237 > > > Thanks! > Kevin > > > From zgu at redhat.com Thu Nov 16 23:31:40 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 18:31:40 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: References: Message-ID: <4735d7db-fd09-feac-2804-f73430b66be2@redhat.com> Hi Thomas, Have you gotten chance to review this? I would like to get this in before door closes for jdk10. Thanks, -Zhengyu On 11/10/2017 08:20 AM, Thomas St?fe wrote: > Hi Zhengyu, > > Cannot test it before monday, but change looks good. Thank you for > doing this. > > ..Thomas > > On Thu 9. Nov 2017 at 20:33, Zhengyu Gu > wrote: > > This patch adds metadata reporting in NMT final report. > > What complicates the matter, is that, reporting per-class loader > metadata requires a safepoint, so that it can safely walk class loaders. > So, we only report it when JVM is about to exit in good state. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 > Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ > > > Thanks, > > -Zhengyu > From zgu at redhat.com Fri Nov 17 01:02:44 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 16 Nov 2017 20:02:44 -0500 Subject: RFR(XXS) 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on In-Reply-To: References: <4735d7db-fd09-feac-2804-f73430b66be2@redhat.com> Message-ID: <5620046d-d8a6-db05-2300-a9a2b623a6c6@redhat.com> Thanks for the review, Thomas. -Zhengyu On 11/16/2017 07:27 PM, Thomas St?fe wrote: > Hi zhengyu, > > The change looks fine. No objection from my side. > > Thanks, Thomas > > On Fri 17. Nov 2017 at 00:31, Zhengyu Gu > wrote: > > Hi Thomas, > > Have you gotten chance to review this? I would like to get this in > before door closes for jdk10. > > Thanks, > > -Zhengyu > > > > On 11/10/2017 08:20 AM, Thomas St?fe wrote: > > Hi Zhengyu, > > > > Cannot test it before monday, but change looks good. Thank you for > > doing this. > > > > ..Thomas > > > > On Thu 9. Nov 2017 at 20:33, Zhengyu Gu > > >> wrote: > > > > This patch adds metadata reporting in NMT final report. > > > > What complicates the matter, is that, reporting per-class loader > > metadata requires a safepoint, so that it can safely walk > class loaders. > > So, we only report it when JVM is about to exit in good state. > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8190357 > > Webrev: http://cr.openjdk.java.net/~zgu/8190357/webrev.00/ > > > > > > > Thanks, > > > > -Zhengyu > > > From david.holmes at oracle.com Fri Nov 17 01:46:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 17 Nov 2017 11:46:15 +1000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: Hi Andrew, On 17/11/2017 1:46 AM, Andrew Haley wrote: > On 16/11/17 06:58, David Holmes wrote: > >> I can't review this in detail as I'm not familiar with NMT workings, but >> have a couple of comments. >> >> On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >>> This is Linux only enhancement for now, can be extended for other >>> platforms. (See bug for details) >> >> I'm concerned about unnecessary platform skew. I added some comments to >> the bug. I think the mincore trick can be used on Solaris and AIX as >> well - but not on OS X / BSD. It may be that VirtualQuery can be used on >> Windows to get the same information - but I'm unclear if the memory >> states support what you're looking for. It would be good to extend this >> to other platforms that will support it. > > Sure, but as OpenJDK becomes more open and hopefully more diverse, it > becomes less and less practical to keep everything in step. I favour > the classic free software approach, which is to encourage people to > create new features and everyone else can fill in the gaps, depending > on how much time and effort is available. It's this approach that > allows GCC to support a vast range of targets (and hosts) while not > holding back development. I have consistently argued for this in the > Governing Board. For OpenJDK I hope there are sufficient safety-nets in place to ensure such an approach does not lead to a product where you never know what will and won't work on each platform. Whenever we have such a case we should at least file sub-tasks for supplying the functionality on the missing platforms. We can then determine whether any such missing functionality is critical to fix before a release. David ----- > So, let's get this done. It's very important for container memory > monitoring, which right now usually means Linux. > From david.holmes at oracle.com Fri Nov 17 02:40:35 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 17 Nov 2017 12:40:35 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: <061fe195-2f79-b0e2-65d4-0aea07672d76@oracle.com> Looking for a runtime Reviewer please. Also Tomas: are you able to test this patch with R? Thanks, David On 14/11/2017 8:39 PM, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > > There are three parts to this which made it somewhat more complicated > than I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > ?void JavaThread::create_stack_guard_pages() { > ?? if (!os::uses_stack_guard_pages() || > ?????? _stack_guard_state != stack_guard_unused || > ?????? (DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ?????? log_info(os, thread)("Stack guard page creation for thread " > ??????????????????????????? UINTX_FORMAT " disabled", > os::current_thread_id()); > ???? return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: > Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do > anything different for the primordial thread, and we don't have a way to > detect the primordial thread - so is_primordial_thread() just returns > false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() from > the rlimit stack size, without checking it was bigger than 2*page_size() > to start with. I was unsure how to tackle this. It's no good leaving > stack_size at zero so I opted to ensure we would have at least one page > left. Of course in such cases we would hit the bug in libc (if it still > exists, which seems unlikely but time consuming to establish). > > Testing: > ? - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > ??? - with guard pages enabled > ??? - with guard pages disabled > ? - Tier 1 (JPRT) normal JDK build (which should be unaffected by this > change) > > The testing was primarily to ensure that disabling the stack guards on > the primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are > enabled and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards are > disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are disabled > > Thanks, > David From thomas.stuefe at gmail.com Fri Nov 17 06:54:52 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 17 Nov 2017 07:54:52 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: On Fri, Nov 17, 2017 at 2:46 AM, David Holmes wrote: > Hi Andrew, > > On 17/11/2017 1:46 AM, Andrew Haley wrote: > >> On 16/11/17 06:58, David Holmes wrote: >> >> I can't review this in detail as I'm not familiar with NMT workings, but >>> have a couple of comments. >>> >>> On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >>> >>>> This is Linux only enhancement for now, can be extended for other >>>> platforms. (See bug for details) >>>> >>> >>> I'm concerned about unnecessary platform skew. I added some comments to >>> the bug. I think the mincore trick can be used on Solaris and AIX as >>> well - but not on OS X / BSD. It may be that VirtualQuery can be used on >>> Windows to get the same information - but I'm unclear if the memory >>> states support what you're looking for. It would be good to extend this >>> to other platforms that will support it. >>> >> >> Sure, but as OpenJDK becomes more open and hopefully more diverse, it >> becomes less and less practical to keep everything in step. I favour >> the classic free software approach, which is to encourage people to >> create new features and everyone else can fill in the gaps, depending >> on how much time and effort is available. It's this approach that >> allows GCC to support a vast range of targets (and hosts) while not >> holding back development. I have consistently argued for this in the >> Governing Board. >> > > For OpenJDK I hope there are sufficient safety-nets in place to ensure > such an approach does not lead to a product where you never know what will > and won't work on each platform. Whenever we have such a case we should at > least file sub-tasks for supplying the functionality on the missing > platforms. We can then determine whether any such missing functionality is > critical to fix before a release. > > I agree. Also, I find that for platform-independent monitoring facilities like NMT, I really just want to be able to trust the numbers without having to peek at the implementation. ..Thomas > David > ----- > > > So, let's get this done. It's very important for container memory >> monitoring, which right now usually means Linux. >> >> From robbin.ehn at oracle.com Fri Nov 17 08:41:01 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 17 Nov 2017 09:41:01 +0100 Subject: RFR: 8191326/8191327/8191329: Deprecate SafepointSpinBeforeYield/DeferThrSuspendLoopCount/DeferPollingPageLoopCount Message-ID: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> Hi please review, Parent issue: 8191325: Deprecate unstable options for SafepointSynchronize: https://bugs.openjdk.java.net/browse/JDK-8191325 8191326: Deprecate SafepointSpinBeforeYield Issue: https://bugs.openjdk.java.net/browse/JDK-8191326 CSR: https://bugs.openjdk.java.net/browse/JDK-8191330 8191327: Deprecate DeferThrSuspendLoopCount Issue: https://bugs.openjdk.java.net/browse/JDK-8191327 CSR: https://bugs.openjdk.java.net/browse/JDK-8191331 8191329: Deprecate DeferPollingPageLoopCount Issue: https://bugs.openjdk.java.net/browse/JDK-8191329 CSR: https://bugs.openjdk.java.net/browse/JDK-8191336 Code: http://cr.openjdk.java.net/~rehn/8191325/webrev/ Thanks Robbin From shade at redhat.com Fri Nov 17 08:44:11 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 17 Nov 2017 09:44:11 +0100 Subject: RFR: 8191326/8191327/8191329: Deprecate SafepointSpinBeforeYield/DeferThrSuspendLoopCount/DeferPollingPageLoopCount In-Reply-To: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> References: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> Message-ID: On 11/17/2017 09:41 AM, Robbin Ehn wrote: > Hi please review, > > Parent issue: 8191325: Deprecate unstable options for SafepointSynchronize: > https://bugs.openjdk.java.net/browse/JDK-8191325 > > 8191326: Deprecate SafepointSpinBeforeYield > Issue: https://bugs.openjdk.java.net/browse/JDK-8191326 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191330 > > 8191327: Deprecate DeferThrSuspendLoopCount > Issue: https://bugs.openjdk.java.net/browse/JDK-8191327 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191331 > > 8191329: Deprecate DeferPollingPageLoopCount > Issue: https://bugs.openjdk.java.net/browse/JDK-8191329 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191336 > > Code: > http://cr.openjdk.java.net/~rehn/8191325/webrev/ This change is very welcome. Looks good. Thanks, -Aleksey From david.holmes at oracle.com Fri Nov 17 09:26:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 17 Nov 2017 19:26:06 +1000 Subject: RFR: 8191326/8191327/8191329: Deprecate SafepointSpinBeforeYield/DeferThrSuspendLoopCount/DeferPollingPageLoopCount In-Reply-To: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> References: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> Message-ID: Looks good. Thanks, David On 17/11/2017 6:41 PM, Robbin Ehn wrote: > Hi please review, > > Parent issue: 8191325: Deprecate unstable options for SafepointSynchronize: > https://bugs.openjdk.java.net/browse/JDK-8191325 > > 8191326: Deprecate SafepointSpinBeforeYield > Issue: https://bugs.openjdk.java.net/browse/JDK-8191326 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191330 > > 8191327: Deprecate DeferThrSuspendLoopCount > Issue: https://bugs.openjdk.java.net/browse/JDK-8191327 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191331 > > 8191329: Deprecate DeferPollingPageLoopCount > Issue: https://bugs.openjdk.java.net/browse/JDK-8191329 > CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191336 > > Code: > http://cr.openjdk.java.net/~rehn/8191325/webrev/ > > Thanks Robbin From aph at redhat.com Fri Nov 17 09:35:47 2017 From: aph at redhat.com (Andrew Haley) Date: Fri, 17 Nov 2017 09:35:47 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: On 17/11/17 01:46, David Holmes wrote: >> Sure, but as OpenJDK becomes more open and hopefully more diverse, it >> becomes less and less practical to keep everything in step. I favour >> the classic free software approach, which is to encourage people to >> create new features and everyone else can fill in the gaps, depending >> on how much time and effort is available. It's this approach that >> allows GCC to support a vast range of targets (and hosts) while not >> holding back development. I have consistently argued for this in the >> Governing Board. > > For OpenJDK I hope there are sufficient safety-nets in place to ensure > such an approach does not lead to a product where you never know what > will and won't work on each platform. I don't believe that's even desirable for OpenJDK. Inevitably some minority platforms won't get that much attention, but as long as every platform is Java-compatible (in the TCK sense) we're doing what we should. > Whenever we have such a case we should at least file sub-tasks for > supplying the functionality on the missing platforms. We can then > determine whether any such missing functionality is critical to fix > before a release. Fair enough. But with quick-fire releases we'll inevitably have to slip some features on some platforms. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From aph at redhat.com Fri Nov 17 09:36:33 2017 From: aph at redhat.com (Andrew Haley) Date: Fri, 17 Nov 2017 09:36:33 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: Message-ID: <3995207c-4a67-62ea-949c-1218f0ca5862@redhat.com> On 17/11/17 06:54, Thomas St?fe wrote: > I agree. Also, I find that for platform-independent monitoring facilities > like NMT, I really just want to be able to trust the numbers without having > to peek at the implementation. Right now you can't trust the numbers. We're talking about fixing that on some platforms. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From robbin.ehn at oracle.com Fri Nov 17 09:40:23 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 17 Nov 2017 10:40:23 +0100 Subject: RFR: 8191326/8191327/8191329: Deprecate SafepointSpinBeforeYield/DeferThrSuspendLoopCount/DeferPollingPageLoopCount In-Reply-To: References: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> Message-ID: Thanks, Robbin On 11/17/2017 09:44 AM, Aleksey Shipilev wrote: > On 11/17/2017 09:41 AM, Robbin Ehn wrote: >> Hi please review, >> >> Parent issue: 8191325: Deprecate unstable options for SafepointSynchronize: >> https://bugs.openjdk.java.net/browse/JDK-8191325 >> >> 8191326: Deprecate SafepointSpinBeforeYield >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191326 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191330 >> >> 8191327: Deprecate DeferThrSuspendLoopCount >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191327 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191331 >> >> 8191329: Deprecate DeferPollingPageLoopCount >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191329 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191336 >> >> Code: >> http://cr.openjdk.java.net/~rehn/8191325/webrev/ > > This change is very welcome. Looks good. > > Thanks, > -Aleksey > From robbin.ehn at oracle.com Fri Nov 17 09:40:32 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 17 Nov 2017 10:40:32 +0100 Subject: RFR: 8191326/8191327/8191329: Deprecate SafepointSpinBeforeYield/DeferThrSuspendLoopCount/DeferPollingPageLoopCount In-Reply-To: References: <60c39fcc-6c0e-cf06-65d8-c165cdd1c226@oracle.com> Message-ID: Thanks, Robbin On 11/17/2017 10:26 AM, David Holmes wrote: > Looks good. > > Thanks, > David > > On 17/11/2017 6:41 PM, Robbin Ehn wrote: >> Hi please review, >> >> Parent issue: 8191325: Deprecate unstable options for SafepointSynchronize: >> https://bugs.openjdk.java.net/browse/JDK-8191325 >> >> 8191326: Deprecate SafepointSpinBeforeYield >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191326 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191330 >> >> 8191327: Deprecate DeferThrSuspendLoopCount >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191327 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191331 >> >> 8191329: Deprecate DeferPollingPageLoopCount >> Issue: https://bugs.openjdk.java.net/browse/JDK-8191329 >> CSR:?? https://bugs.openjdk.java.net/browse/JDK-8191336 >> >> Code: >> http://cr.openjdk.java.net/~rehn/8191325/webrev/ >> >> Thanks Robbin From thomas.stuefe at gmail.com Fri Nov 17 09:57:48 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 17 Nov 2017 10:57:48 +0100 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <3995207c-4a67-62ea-949c-1218f0ca5862@redhat.com> References: <3995207c-4a67-62ea-949c-1218f0ca5862@redhat.com> Message-ID: On Fri, Nov 17, 2017 at 10:36 AM, Andrew Haley wrote: > On 17/11/17 06:54, Thomas St?fe wrote: > > I agree. Also, I find that for platform-independent monitoring facilities > > like NMT, I really just want to be able to trust the numbers without > having > > to peek at the implementation. > > Right now you can't trust the numbers. We're talking about fixing that > on some platforms. > > I did not argue against the patch. If you read the thread, you'll see that I was in favour of it even if with only the Linux implementation provided. But I'd like to see unimplemented platforms tracked in follow up issues as David proposed. Because it is easy to loose track of the many things to port. In this case, we'd need a follow up issue for Solaris and AIX; the latter you can assign to me. Best Regards, Thomas > -- > Andrew Haley > Java Platform Lead Engineer > Red Hat UK Ltd. > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > From volker.simonis at gmail.com Fri Nov 17 13:42:37 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 17 Nov 2017 14:42:37 +0100 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> Message-ID: Hi Ioi, I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX ports. I applied your patch to the current jdk-hs repo as of today and first of all I see the same compilation problems already reported by Dmitry before. Can you please update you webrev with the corresponding changes or create a new one. That will help other who wish to look at this change. After building, I ran the appcds jtreg tests (i.e. test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see the same four failures like Dmitry. Is that expected? I see a lot more errors on Linux/ppc64le but before going into more detail I'd like to know what is the correct baseline on which we can rely. Finally, I saw that with a product build, I get two additional test failures: runtime/appcds/ProhibitedPackage.java runtime/appcds/DumpClassList.java because of: Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is available only in debug version of VM. So these two tests should probably be adjusted to only run for slow/fastdebug builds. Thank you and best regards, Volker On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff wrote: > Ioi, > > Thank you! > > I did some additional testing and find that four tests > fails the same way on both x86_64 and AARCH64 (see below). > > runtime/appcds/VerifierTest_0.java > runtime/appcds/cacheObject/GCStressTest.java > runtime/appcds/cacheObject/RedefineClassTest.java > runtime/appcds/sharedStrings/InternSharedString.java > > Is it expected? > > test result: Failed. Execution failed: `main' threw exception: > java.lang.RuntimeException: 'Please remove the unverifiable cl > asses' missing from stdout/stderr > > Exception in thread "main" java.lang.RuntimeException: FAILED. > GCStressApp_nonshared_string should not be shared > at GCStressApp.main(GCStressApp.java:73) > > > FAILED. buzzshould not be shared > > Exception in thread "main" java.lang.RuntimeException: Failed: > unexpected shared string. > at InternStringTest.main(InternStringTest.java:63) > > -Dmitry > > On 16.11.2017 03:10, Ioi Lam wrote: >> I filed: >> >> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >> VMProps to filter out CDS tests >> >> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >> when CDS is not supported >> >> Thanks >> >> - Ioi >> >> >> >> On 11/15/17 4:01 PM, Ioi Lam wrote: >>> Hi Dmitry, >>> >>> Thanks for the review. >>> >>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>> >>>> Ioi, >>>> >>>> I tested new patch on aarch64 and it works fine with the changes >>>> below[1]. >>> I've fixed it. Thanks for the patch. >>>> Also tests doesn't work with exploded image - is it possible to check >>>> for this case and produce appropriate error message? >>> CDS is not supported on exploded images. I think the best thing to do >>> is to add something like "@requires vm.cds" into the test cases >>> (similar to the use of "@require vm.aot" in other tests). That way, >>> none of the CDS tests will be executed for >>> >>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>> it in the next release. >>> >>> I'll also file another RFE to print a better message. Today if you use >>> an exploded build the error message is kind of confusing: >>> >>> $ ./jdk/bin/java -Xshare:dump >>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>> Hint: enable -Xlog:class+path=info to diagnose the failure >>> Error occurred during initialization of VM >>> CDS allows only empty directories in archived classpaths >>> >>> It should say something like "CDS is not supported on exploded images". >>> >>> Thanks >>> - Ioi >>> >>>> 1. >>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>> 09:45:23 2017 +0000 >>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>> 15:02:51 2017 +0000 >>>> @@ -31,7 +31,6 @@ >>>> -#include "prims/jvm.h" >>>> >>>> diff -r dbf9cec6a568 >>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>> 06 09:45:23 2017 +0000 >>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>> 06 15:02:51 2017 +0000 >>>> @@ -518,7 +518,7 @@ >>>> >>>> { >>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>> if (check != NULL) { >>>> return InstanceKlass::cast(check); >>>> } >>>> >>>> -Dmitry >>>> >>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>> Hi, >>>>> >>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>> During internal review of the JEP, we have decided to integrate both >>>>> implementation and tests at the same time. >>>>> >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>> >>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>> result of copying the closed source files on top of files of the same >>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>> "changed", please focus on the entire content of the new version of >>>>> each >>>>> file. >>>>> >>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>> sources) and all the new appcds tests passed. >>>>> >>>>> Thanks >>>>> >>>>> - Ioi >>>>> >>>>> >>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>> is_modules_image. Please change the code accordingly. >>>>>> >>>>>> I will post my latest diff soon, with some test cases as well. >>>>>> >>>>>> Thanks >>>>>> >>>>>> - Ioi >>>>>> >>>>>> >>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>> Ioi, >>>>>>> >>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>> the compilation fails with: >>>>>>> >>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>> >>>>>>> >>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>> >>>>>>> Did I miss something? >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Please review this change set. >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>> >>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>> moves >>>>>>>> AppCDS from >>>>>>>> closed repos into the openjdk repo: >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>> >>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>> repo. >>>>>>>> As part >>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>> code >>>>>>>> into the >>>>>>>> open repo. >>>>>>>> >>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>> possible. The >>>>>>>> intention is >>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>> and not >>>>>>>> to do any sort of refactoring. >>>>>>>> >>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>> copying the >>>>>>>> closed source >>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>> reviewing, instead of >>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>> content >>>>>>>> of the new >>>>>>>> version of each file. >>>>>>>> >>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>> flag is >>>>>>>> changed from >>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>> "commercial" >>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>> >>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>> open/closed >>>>>>>> source >>>>>>>> code structure had introduced some intermediary APIs to connect code >>>>>>>> between >>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>> >>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>> tests >>>>>>>> will be >>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>> >>>>>>>> All the AppCDS tests (currently still in closed sources) passed with >>>>>>>> both Oracle JDK >>>>>>>> and OpenJDK. >>>>>>>> >>>>>>>> Thanks >>>>>>>> - Ioi >>> >> > > From robbin.ehn at oracle.com Fri Nov 17 14:10:57 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 17 Nov 2017 15:10:57 +0100 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected Message-ID: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Hi all, please review. Windows needs to run os::init_2 before allocation polling page to proper setup numa. I saw no reason not to have it for all platforms after init_2. Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 Code below. Tested tier 1-5/1-3 Thanks, Robbin diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 +0100 +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 +0100 @@ -3560,10 +3560,10 @@ TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); - SafepointMechanism::initialize(); - // Initialize the os module after parsing the args jint os_init_2_result = os::init_2(); if (os_init_2_result != JNI_OK) return os_init_2_result; + SafepointMechanism::initialize(); + jint adjust_after_os_result = Arguments::adjust_after_os(); if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From volker.simonis at gmail.com Fri Nov 17 14:21:11 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 17 Nov 2017 15:21:11 +0100 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: Hi Gustavo, as our "PowerPC NUMA expert", do you see any problem with this change? Thank yo uand best regards, Volker On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: > Hi all, please review. > > Windows needs to run os::init_2 before allocation polling page to proper > setup numa. I saw no reason not to have it for all platforms after init_2. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 > > Code below. > > Tested tier 1-5/1-3 > > Thanks, Robbin > > diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp > --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 > +0100 > +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 > +0100 > @@ -3560,10 +3560,10 @@ > TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); > > - SafepointMechanism::initialize(); > - > // Initialize the os module after parsing the args > jint os_init_2_result = os::init_2(); > if (os_init_2_result != JNI_OK) return os_init_2_result; > > + SafepointMechanism::initialize(); > + > jint adjust_after_os_result = Arguments::adjust_after_os(); > if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From daniel.daugherty at oracle.com Fri Nov 17 14:36:29 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 17 Nov 2017 09:36:29 -0500 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: On 11/17/17 9:10 AM, Robbin Ehn wrote: > Hi all, please review. > > Windows needs to run os::init_2 before allocation polling page to > proper setup numa. I saw no reason not to have it for all platforms > after init_2. Do we understand why this failure mode is intermittent? Dan > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 > > Code below. > > Tested tier 1-5/1-3 > > Thanks, Robbin > > diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp > --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 2017 > +0100 > +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 2017 > +0100 > @@ -3560,10 +3560,10 @@ > ?? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); > > -? SafepointMechanism::initialize(); > - > ?? // Initialize the os module after parsing the args > ?? jint os_init_2_result = os::init_2(); > ?? if (os_init_2_result != JNI_OK) return os_init_2_result; > > +? SafepointMechanism::initialize(); > + > ?? jint adjust_after_os_result = Arguments::adjust_after_os(); > ?? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From daniel.daugherty at oracle.com Fri Nov 17 15:39:13 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 17 Nov 2017 10:39:13 -0500 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: <73a1f468-33d7-73ce-8766-3c5e798fc733@oracle.com> On 11/17/17 9:36 AM, Daniel D. Daugherty wrote: > On 11/17/17 9:10 AM, Robbin Ehn wrote: >> Hi all, please review. >> >> Windows needs to run os::init_2 before allocation polling page to >> proper setup numa. I saw no reason not to have it for all platforms >> after init_2. > > Do we understand why this failure mode is intermittent? Update: The TestOptionsWithRanges.java failure mode is solid and reproducible. And it does not reproduce with Robbin's fix in place. When Robbin was testing the fix a couple of other tests failed intermittently and Robbin initially thought his fix provoked those intermittent failures. Those failures have since reproduced in the jdk/hs nightly so they are unrelated to Robbin's fix. Sorry for any confusion. Thumbs up on the fix! Dan > > Dan > > >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >> >> Code below. >> >> Tested tier 1-5/1-3 >> >> Thanks, Robbin >> >> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >> --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 >> 2017 +0100 >> +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 >> 2017 +0100 >> @@ -3560,10 +3560,10 @@ >> ?? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >> >> -? SafepointMechanism::initialize(); >> - >> ?? // Initialize the os module after parsing the args >> ?? jint os_init_2_result = os::init_2(); >> ?? if (os_init_2_result != JNI_OK) return os_init_2_result; >> >> +? SafepointMechanism::initialize(); >> + >> ?? jint adjust_after_os_result = Arguments::adjust_after_os(); >> ?? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; > > From daniel.daugherty at oracle.com Fri Nov 17 15:40:45 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 17 Nov 2017 10:40:45 -0500 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> On 11/14/17 5:39 AM, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ src/hotspot/os/aix/os_aix.cpp ??? No comments. src/hotspot/os/aix/os_aix.hpp ??? No comments. src/hotspot/os/bsd/os_bsd.cpp ??? L880: // thread in any special way - so just report 'false' ?? ? ?? Nit - needs a period after 'false' to make it a sentence. ??? L882: ??? return false; ??????? Nit - indent should be 2 spaces. src/hotspot/os/bsd/os_bsd.hpp ??? No comments. src/hotspot/os/linux/os_linux.cpp ??? L947: ? if (stack_size >= (size_t)(3 * page_size())) ??????? Nit - needs '{' and '}' on the if-statement body. ??? L3070: // always place it right after end of the mapped region ??????? Nit - needs a period after 'region' to make it a sentence. ??????? (Not your problem, but you did update the sentence...) src/hotspot/os/linux/os_linux.hpp ??? No comments. src/hotspot/os/solaris/os_solaris.cpp ??? No comments other than I learned something new about thr_main() :-) ??? Thanks for cleaning up that mess. src/hotspot/os/windows/os_windows.cpp ??? L452: // thread in any special way - so just report 'false' ??????? Nit - needs a period after 'false' to make it a sentence. ??? L454: ??? return false; ??????? Nit - indent should be 2 spaces. src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp ??? No comments. src/hotspot/share/runtime/globals.hpp ??? L1181: ? experimental(bool, DisablePrimordialThreadGuardPages, false, ??????? Experimental or diagnostic? src/hotspot/share/runtime/os.hpp ??? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM ??????? Nit - needs a period after 'JNI_CreateJavaVM'. ??? L453: ? // launcher nevers uses the primordial thread as the main thread, but ??????? Typo - 'nevers' -> 'never' ??? L455: ? // have to special-case handling of the primordial thread if it attaches ??????? Typo - 'have to' -> 'have' src/hotspot/share/runtime/thread.cpp ??? I like the new log message. src/hotspot/share/runtime/thread.inline.hpp ??? L159: ? if (os::uses_stack_guard_pages() && ??? L160: ????? !(DisablePrimordialThreadGuardPages && os::is_primordial_thread())) { ??????? I'm not sure why, but I had to read these two lines several ??????? times before I convinced myself the logic is right. I'm good with the code changes. Your call on whether to fix the bits or not. Regarding this part of Thomas' review: >> Or, something like this: >> static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) >> BSD_ONLY({return false;}) > > I can do that. I'll see if anyone else has any comments on this. I'm assuming that Thomas means for the above to be at the platform independent layer (runtime/os.hpp?) so it is more obvious to the reader that the function doesn't do anything on those two platforms... I know we dislike such things at the platform independent layer, but it does feel appropriate to make this limitation more obvious. I would be okay if you made the change that Thomas is suggesting. Thumbs up! Dan > > > There are three parts to this which made it somewhat more complicated > than I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > ?void JavaThread::create_stack_guard_pages() { > ?? if (!os::uses_stack_guard_pages() || > ?????? _stack_guard_state != stack_guard_unused || > ?????? (DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ?????? log_info(os, thread)("Stack guard page creation for thread " > ??????????????????????????? UINTX_FORMAT " disabled", > os::current_thread_id()); > ???? return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: > Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do > anything different for the primordial thread, and we don't have a way > to detect the primordial thread - so is_primordial_thread() just > returns false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() from > the rlimit stack size, without checking it was bigger than > 2*page_size() to start with. I was unsure how to tackle this. It's no > good leaving stack_size at zero so I opted to ensure we would have at > least one page left. Of course in such cases we would hit the bug in > libc (if it still exists, which seems unlikely but time consuming to > establish). > > Testing: > ? - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > ??? - with guard pages enabled > ??? - with guard pages disabled > ? - Tier 1 (JPRT) normal JDK build (which should be unaffected by this > change) > > The testing was primarily to ensure that disabling the stack guards on > the primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are > enabled and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards are > disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are > disabled > > Thanks, > David > From ioi.lam at oracle.com Fri Nov 17 15:51:08 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 17 Nov 2017 07:51:08 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> Message-ID: <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> Hi Volker, Thanks for trying the patch out. On 11/17/17 5:42 AM, Volker Simonis wrote: > Hi Ioi, > > I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX ports. > > I applied your patch to the current jdk-hs repo as of today and first > of all I see the same compilation problems already reported by Dmitry > before. Can you please update you webrev with the corresponding > changes or create a new one. That will help other who wish to look at > this change. I've updated the patch inside the webrev. You can download it from http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch The previous patch (which you had the problems with) is still available as http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old I will post a new webrev, but it's big and I might be running out of quota from cr.openjdk.java.net, so I need to figure out how to handle that. The jdk/hs changeset corresponding to the new patch is the following: changeset:?? 47860:564882d918d4 user:??????? zgu date:??????? Thu Nov 16 20:21:11 2017 -0500 files:?????? src/hotspot/share/services/memTracker.cpp description: 8190357: NMT: Include metadata information in NMT final report when PrintNMTStatistics is on Summary: Include metadata information in NMT final report Reviewed-by: adinn, stuefe > After building, I ran the appcds jtreg tests (i.e. > test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see > the same four failures like Dmitry. Those are also due to merge issues. With the updated patch, I passed all tests under on Linux/x64: ??? test/hotspot/jtreg/runtime/SharedArchiveFile ??? test/hotspot/jtreg/runtime/CDSCompressedKPtrs test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java ??? test/hotspot/jtreg/runtime/appcds $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 -retain -agentvm -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ open/test/hotspot/jtreg/runtime/SharedArchiveFile open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java open/test/hotspot/jtreg/runtime/appcds [...] Test results: passed: 128 Results written to /jdk/tmp/jtreg/work > Is that expected? I see a lot more errors on Linux/ppc64le but before > going into more detail I'd like to know what is the correct baseline > on which we can rely. > > Finally, I saw that with a product build, I get two additional test failures: > > runtime/appcds/ProhibitedPackage.java > runtime/appcds/DumpClassList.java > > because of: > > Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is > available only in debug version of VM. > > So these two tests should probably be adjusted to only run for > slow/fastdebug builds. You're correct. I will exclude those tests from product builds, and file an RFE to fix them later. Thanks - Ioi > Thank you and best regards, > Volker > > > On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff wrote: >> Ioi, >> >> Thank you! >> >> I did some additional testing and find that four tests >> fails the same way on both x86_64 and AARCH64 (see below). >> >> runtime/appcds/VerifierTest_0.java >> runtime/appcds/cacheObject/GCStressTest.java >> runtime/appcds/cacheObject/RedefineClassTest.java >> runtime/appcds/sharedStrings/InternSharedString.java >> >> Is it expected? >> >> test result: Failed. Execution failed: `main' threw exception: >> java.lang.RuntimeException: 'Please remove the unverifiable cl >> asses' missing from stdout/stderr >> >> Exception in thread "main" java.lang.RuntimeException: FAILED. >> GCStressApp_nonshared_string should not be shared >> at GCStressApp.main(GCStressApp.java:73) >> >> >> FAILED. buzzshould not be shared >> >> Exception in thread "main" java.lang.RuntimeException: Failed: >> unexpected shared string. >> at InternStringTest.main(InternStringTest.java:63) >> >> -Dmitry >> >> On 16.11.2017 03:10, Ioi Lam wrote: >>> I filed: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>> VMProps to filter out CDS tests >>> >>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>> when CDS is not supported >>> >>> Thanks >>> >>> - Ioi >>> >>> >>> >>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>> Hi Dmitry, >>>> >>>> Thanks for the review. >>>> >>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>> >>>>> Ioi, >>>>> >>>>> I tested new patch on aarch64 and it works fine with the changes >>>>> below[1]. >>>> I've fixed it. Thanks for the patch. >>>>> Also tests doesn't work with exploded image - is it possible to check >>>>> for this case and produce appropriate error message? >>>> CDS is not supported on exploded images. I think the best thing to do >>>> is to add something like "@requires vm.cds" into the test cases >>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>> none of the CDS tests will be executed for >>>> >>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>> it in the next release. >>>> >>>> I'll also file another RFE to print a better message. Today if you use >>>> an exploded build the error message is kind of confusing: >>>> >>>> $ ./jdk/bin/java -Xshare:dump >>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>> Error occurred during initialization of VM >>>> CDS allows only empty directories in archived classpaths >>>> >>>> It should say something like "CDS is not supported on exploded images". >>>> >>>> Thanks >>>> - Ioi >>>> >>>>> 1. >>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>> 09:45:23 2017 +0000 >>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>> 15:02:51 2017 +0000 >>>>> @@ -31,7 +31,6 @@ >>>>> -#include "prims/jvm.h" >>>>> >>>>> diff -r dbf9cec6a568 >>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>> 06 09:45:23 2017 +0000 >>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>> 06 15:02:51 2017 +0000 >>>>> @@ -518,7 +518,7 @@ >>>>> >>>>> { >>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>> if (check != NULL) { >>>>> return InstanceKlass::cast(check); >>>>> } >>>>> >>>>> -Dmitry >>>>> >>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>> Hi, >>>>>> >>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>> During internal review of the JEP, we have decided to integrate both >>>>>> implementation and tests at the same time. >>>>>> >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>> >>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>> result of copying the closed source files on top of files of the same >>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>> "changed", please focus on the entire content of the new version of >>>>>> each >>>>>> file. >>>>>> >>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>> sources) and all the new appcds tests passed. >>>>>> >>>>>> Thanks >>>>>> >>>>>> - Ioi >>>>>> >>>>>> >>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>> Hi Dmitry, >>>>>>> >>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>> is_modules_image. Please change the code accordingly. >>>>>>> >>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> - Ioi >>>>>>> >>>>>>> >>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>> Ioi, >>>>>>>> >>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>> the compilation fails with: >>>>>>>> >>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>> >>>>>>>> >>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>> >>>>>>>> Did I miss something? >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Please review this change set. >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>> >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>> >>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>> moves >>>>>>>>> AppCDS from >>>>>>>>> closed repos into the openjdk repo: >>>>>>>>> >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>> >>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>> repo. >>>>>>>>> As part >>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>> code >>>>>>>>> into the >>>>>>>>> open repo. >>>>>>>>> >>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>> possible. The >>>>>>>>> intention is >>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>> and not >>>>>>>>> to do any sort of refactoring. >>>>>>>>> >>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>> copying the >>>>>>>>> closed source >>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>> reviewing, instead of >>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>> content >>>>>>>>> of the new >>>>>>>>> version of each file. >>>>>>>>> >>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>> flag is >>>>>>>>> changed from >>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>> "commercial" >>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>> >>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>> open/closed >>>>>>>>> source >>>>>>>>> code structure had introduced some intermediary APIs to connect code >>>>>>>>> between >>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>> >>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>> tests >>>>>>>>> will be >>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>> >>>>>>>>> All the AppCDS tests (currently still in closed sources) passed with >>>>>>>>> both Oracle JDK >>>>>>>>> and OpenJDK. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> - Ioi >> From gromero at linux.vnet.ibm.com Fri Nov 17 17:30:53 2017 From: gromero at linux.vnet.ibm.com (Gustavo Romero) Date: Fri, 17 Nov 2017 15:30:53 -0200 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: <5A0F1CCD.1020007@linux.vnet.ibm.com> Hi Volker, Robbin, @Robbin Could you please tell me against which repo/commit id you created that diff? It's after consolidation (hs or master repo), right? Just asking because: gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ fgrep -i SafepointMechanism -RIn gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg id d85284ccd1bd tip gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg path default = http://hg.openjdk.java.net/jdk10/hs/ gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ cd ~/hg/jdk10/master/src/hotspot/ gromero at gromero16:~/hg/jdk10/master/src/hotspot$ fgrep -i SafepointMechanism -RIn gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg id be620a591379 tip gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg path default = http://hg.openjdk.java.net/jdk10/master/ But maybe I'm missing something obvious... Thanks! Regards, Gustavo On 17-11-2017 12:21, Volker Simonis wrote: > Hi Gustavo, > > as our "PowerPC NUMA expert", do you see any problem with this change? > > Thank yo uand best regards, > Volker > > > On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >> Hi all, please review. >> >> Windows needs to run os::init_2 before allocation polling page to proper >> setup numa. I saw no reason not to have it for all platforms after init_2. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >> >> Code below. >> >> Tested tier 1-5/1-3 >> >> Thanks, Robbin >> >> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >> --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 >> +0100 >> +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 >> +0100 >> @@ -3560,10 +3560,10 @@ >> TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >> >> - SafepointMechanism::initialize(); >> - >> // Initialize the os module after parsing the args >> jint os_init_2_result = os::init_2(); >> if (os_init_2_result != JNI_OK) return os_init_2_result; >> >> + SafepointMechanism::initialize(); >> + >> jint adjust_after_os_result = Arguments::adjust_after_os(); >> if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; > From gromero at linux.vnet.ibm.com Fri Nov 17 17:44:59 2017 From: gromero at linux.vnet.ibm.com (Gustavo Romero) Date: Fri, 17 Nov 2017 15:44:59 -0200 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <5A0F1CCD.1020007@linux.vnet.ibm.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> <5A0F1CCD.1020007@linux.vnet.ibm.com> Message-ID: <5A0F201B.4040904@linux.vnet.ibm.com> Hi Robbin, On 17-11-2017 15:30, Gustavo Romero wrote: > Hi Volker, Robbin, > > @Robbin Could you please tell me against which repo/commit id you created that > diff? It's after consolidation (hs or master repo), right? Just asking because: > > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ fgrep -i SafepointMechanism -RIn > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg id > d85284ccd1bd tip > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg path > default = http://hg.openjdk.java.net/jdk10/hs/ > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ cd ~/hg/jdk10/master/src/hotspot/ > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ fgrep -i SafepointMechanism -RIn > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg id > be620a591379 tip > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg path > default = http://hg.openjdk.java.net/jdk10/master/ > > But maybe I'm missing something obvious... I think that this change depends upon: http://cr.openjdk.java.net/~rehn/8185640/v0/PollingPage-1/webrev/open.patch which is not pushed yet, right? Thanks and best regards, Gustavo > > Thanks! > > Regards, > Gustavo > > On 17-11-2017 12:21, Volker Simonis wrote: >> Hi Gustavo, >> >> as our "PowerPC NUMA expert", do you see any problem with this change? >> >> Thank yo uand best regards, >> Volker >> >> >> On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >>> Hi all, please review. >>> >>> Windows needs to run os::init_2 before allocation polling page to proper >>> setup numa. I saw no reason not to have it for all platforms after init_2. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >>> >>> Code below. >>> >>> Tested tier 1-5/1-3 >>> >>> Thanks, Robbin >>> >>> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >>> --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 >>> +0100 >>> +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 >>> +0100 >>> @@ -3560,10 +3560,10 @@ >>> TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >>> >>> - SafepointMechanism::initialize(); >>> - >>> // Initialize the os module after parsing the args >>> jint os_init_2_result = os::init_2(); >>> if (os_init_2_result != JNI_OK) return os_init_2_result; >>> >>> + SafepointMechanism::initialize(); >>> + >>> jint adjust_after_os_result = Arguments::adjust_after_os(); >>> if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; >> > From robbin.ehn at oracle.com Fri Nov 17 17:45:49 2017 From: robbin.ehn at oracle.com (robbin.ehn) Date: Fri, 17 Nov 2017 18:45:49 +0100 Subject: SV: Re: RFR(integration blocker): 8191373: Multiple NUMA nodes expected Message-ID: Hi, http://hg.openjdk.java.net/jdk/hs/ /Robbin from phone -------- Originalmeddelande --------Fr?n: Gustavo Romero Datum: 2017-11-17 18:30 (GMT+01:00) Till: Robbin Ehn Kopia: Volker Simonis , hotspot-runtime-dev at openjdk.java.net, Jesper Wilhelmsson Rubrik: Re: RFR(integration blocker): 8191373: Multiple NUMA nodes expected Hi Volker, Robbin, @Robbin Could you please tell me against which repo/commit id you created that diff? It's after consolidation (hs or master repo), right? Just asking because: gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ fgrep -i SafepointMechanism -RIn gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg id d85284ccd1bd tip gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg path default = http://hg.openjdk.java.net/jdk10/hs/ gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ cd ~/hg/jdk10/master/src/hotspot/ gromero at gromero16:~/hg/jdk10/master/src/hotspot$ fgrep -i SafepointMechanism -RIn gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg id be620a591379 tip gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg path default = http://hg.openjdk.java.net/jdk10/master/ But maybe I'm missing something obvious... Thanks! Regards, Gustavo On 17-11-2017 12:21, Volker Simonis wrote: > Hi Gustavo, > > as our "PowerPC NUMA expert", do you see any problem with this change? > > Thank yo uand best regards, > Volker > > > On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >> Hi all, please review. >> >> Windows needs to run os::init_2 before allocation polling page to proper >> setup numa. I saw no reason not to have it for all platforms after init_2. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >> >> Code below. >> >> Tested tier 1-5/1-3 >> >> Thanks, Robbin >> >> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >> --- a/src/hotspot/share/runtime/thread.cpp????? Fri Nov 17 02:50:51 2017 >> +0100 >> +++ b/src/hotspot/share/runtime/thread.cpp????? Fri Nov 17 13:30:28 2017 >> +0100 >> @@ -3560,10 +3560,10 @@ >>??? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >> >> -? SafepointMechanism::initialize(); >> - >>??? // Initialize the os module after parsing the args >>??? jint os_init_2_result = os::init_2(); >>??? if (os_init_2_result != JNI_OK) return os_init_2_result; >> >> +? SafepointMechanism::initialize(); >> + >>??? jint adjust_after_os_result = Arguments::adjust_after_os(); >>??? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; > From gromero at linux.vnet.ibm.com Fri Nov 17 17:49:55 2017 From: gromero at linux.vnet.ibm.com (Gustavo Romero) Date: Fri, 17 Nov 2017 15:49:55 -0200 Subject: SV: Re: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: References: Message-ID: <5A0F2143.5070703@linux.vnet.ibm.com> Hi Robbin, On 17-11-2017 15:45, robbin.ehn wrote: > Hi, > http://hg.openjdk.java.net/jdk/hs/ > /Robbin from phone OK. Got it. Thanks. Regards, Gustavo > -------- Originalmeddelande --------Fr?n: Gustavo Romero Datum: 2017-11-17 18:30 (GMT+01:00) Till: Robbin Ehn Kopia: Volker Simonis , hotspot-runtime-dev at openjdk.java.net, Jesper Wilhelmsson Rubrik: Re: RFR(integration blocker): 8191373: Multiple NUMA nodes expected > Hi Volker, Robbin, > > @Robbin Could you please tell me against which repo/commit id you created that > diff? It's after consolidation (hs or master repo), right? Just asking because: > > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ fgrep -i SafepointMechanism -RIn > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg id > d85284ccd1bd tip > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ hg path > default = http://hg.openjdk.java.net/jdk10/hs/ > gromero at gromero16:~/hg/jdk10/hs/src/hotspot$ cd ~/hg/jdk10/master/src/hotspot/ > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ fgrep -i SafepointMechanism -RIn > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg id > be620a591379 tip > gromero at gromero16:~/hg/jdk10/master/src/hotspot$ hg path > default = http://hg.openjdk.java.net/jdk10/master/ > > But maybe I'm missing something obvious... > > > Thanks! > > Regards, > Gustavo > > On 17-11-2017 12:21, Volker Simonis wrote: >> Hi Gustavo, >> >> as our "PowerPC NUMA expert", do you see any problem with this change? >> >> Thank yo uand best regards, >> Volker >> >> >> On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >>> Hi all, please review. >>> >>> Windows needs to run os::init_2 before allocation polling page to proper >>> setup numa. I saw no reason not to have it for all platforms after init_2. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >>> >>> Code below. >>> >>> Tested tier 1-5/1-3 >>> >>> Thanks, Robbin >>> >>> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >>> --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 >>> +0100 >>> +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 >>> +0100 >>> @@ -3560,10 +3560,10 @@ >>> TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >>> >>> - SafepointMechanism::initialize(); >>> - >>> // Initialize the os module after parsing the args >>> jint os_init_2_result = os::init_2(); >>> if (os_init_2_result != JNI_OK) return os_init_2_result; >>> >>> + SafepointMechanism::initialize(); >>> + >>> jint adjust_after_os_result = Arguments::adjust_after_os(); >>> if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; >> > From thomas.stuefe at gmail.com Fri Nov 17 18:06:26 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 17 Nov 2017 19:06:26 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> Message-ID: On Fri, Nov 17, 2017 at 4:40 PM, Daniel D. Daugherty < daniel.daugherty at oracle.com> wrote: > On 11/14/17 5:39 AM, David Holmes wrote: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > > src/hotspot/os/aix/os_aix.cpp > No comments. > > src/hotspot/os/aix/os_aix.hpp > No comments. > > src/hotspot/os/bsd/os_bsd.cpp > L880: // thread in any special way - so just report 'false' > Nit - needs a period after 'false' to make it a sentence. > > L882: return false; > Nit - indent should be 2 spaces. > > src/hotspot/os/bsd/os_bsd.hpp > No comments. > > src/hotspot/os/linux/os_linux.cpp > L947: if (stack_size >= (size_t)(3 * page_size())) > Nit - needs '{' and '}' on the if-statement body. > > L3070: // always place it right after end of the mapped region > Nit - needs a period after 'region' to make it a sentence. > (Not your problem, but you did update the sentence...) > > src/hotspot/os/linux/os_linux.hpp > No comments. > > src/hotspot/os/solaris/os_solaris.cpp > No comments other than I learned something new about thr_main() :-) > Thanks for cleaning up that mess. > > src/hotspot/os/windows/os_windows.cpp > L452: // thread in any special way - so just report 'false' > Nit - needs a period after 'false' to make it a sentence. > > L454: return false; > Nit - indent should be 2 spaces. > > src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp > No comments. > > src/hotspot/share/runtime/globals.hpp > L1181: experimental(bool, DisablePrimordialThreadGuardPages, false, > Experimental or diagnostic? > > src/hotspot/share/runtime/os.hpp > L450: // that loads/creates the JVM via JNI_CreateJavaVM > Nit - needs a period after 'JNI_CreateJavaVM'. > > L453: // launcher nevers uses the primordial thread as the main > thread, but > Typo - 'nevers' -> 'never' > > L455: // have to special-case handling of the primordial thread if > it attaches > Typo - 'have to' -> 'have' > > src/hotspot/share/runtime/thread.cpp > I like the new log message. > > src/hotspot/share/runtime/thread.inline.hpp > L159: if (os::uses_stack_guard_pages() && > L160: !(DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > I'm not sure why, but I had to read these two lines several > times before I convinced myself the logic is right. > > I'm good with the code changes. Your call on whether to fix the > bits or not. > > Regarding this part of Thomas' review: > > Or, something like this: > static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) > BSD_ONLY({return false;}) > > > I can do that. I'll see if anyone else has any comments on this. > > > I'm assuming that Thomas means for the above to be at the > platform independent layer (runtime/os.hpp?) so it is more > obvious to the reader that the function doesn't do anything > on those two platforms... I know we dislike such things at > the platform independent layer, but it does feel appropriate > to make this limitation more obvious. > > I would be okay if you made the change that Thomas is suggesting. > > Thumbs up! > > Dan > > > > > There are three parts to this which made it somewhat more complicated than > I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > void JavaThread::create_stack_guard_pages() { > if (!os::uses_stack_guard_pages() || > _stack_guard_state != stack_guard_unused || > (DisablePrimordialThreadGuardPages && os::is_primordial_thread())) > { > log_info(os, thread)("Stack guard page creation for thread " > UINTX_FORMAT " disabled", > os::current_thread_id()); > return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: Linux, > Solaris and AIX. For BSD/OSX and Windows we don't attempt to do anything > different for the primordial thread, and we don't have a way to detect the > primordial thread - so is_primordial_thread() just returns false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() from the > rlimit stack size, without checking it was bigger than 2*page_size() to > start with. I was unsure how to tackle this. It's no good leaving > stack_size at zero so I opted to ensure we would have at least one page > left. Of course in such cases we would hit the bug in libc (if it still > exists, which seems unlikely but time consuming to establish). > > Testing: > - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > - with guard pages enabled > - with guard pages disabled > - Tier 1 (JPRT) normal JDK build (which should be unaffected by this > change) > > The testing was primarily to ensure that disabling the stack guards on the > primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are enabled > and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards are > disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are > disabled > > Thanks, > David > > > From thomas.stuefe at gmail.com Fri Nov 17 18:08:35 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 17 Nov 2017 19:08:35 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> Message-ID: Ooops, pressed Send to soon... ... > > Regarding this part of Thomas' review: > > Or, something like this: > static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) > BSD_ONLY({return false;}) > > > I can do that. I'll see if anyone else has any comments on this. > > > I'm assuming that Thomas means for the above to be at the > platform independent layer (runtime/os.hpp?) so it is more > obvious to the reader that the function doesn't do anything > on those two platforms... I know we dislike such things at > the platform independent layer, but it does feel appropriate > to make this limitation more obvious. > > I would be okay if you made the change that Thomas is suggesting. > > Yes, that's what I meant. Putting that into os.hpp makes it sprint up readily, e.g. in Tooltips in Eclipse CDT when hovering over the function call. ..Thomas > Thumbs up! > > Dan > > > > > There are three parts to this which made it somewhat more complicated than > I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > void JavaThread::create_stack_guard_pages() { > if (!os::uses_stack_guard_pages() || > _stack_guard_state != stack_guard_unused || > (DisablePrimordialThreadGuardPages && os::is_primordial_thread())) > { > log_info(os, thread)("Stack guard page creation for thread " > UINTX_FORMAT " disabled", > os::current_thread_id()); > return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() > > We have three platforms that already implement this functionality: Linux, > Solaris and AIX. For BSD/OSX and Windows we don't attempt to do anything > different for the primordial thread, and we don't have a way to detect the > primordial thread - so is_primordial_thread() just returns false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract 2*page_size() from the > rlimit stack size, without checking it was bigger than 2*page_size() to > start with. I was unsure how to tackle this. It's no good leaving > stack_size at zero so I opted to ensure we would have at least one page > left. Of course in such cases we would hit the bug in libc (if it still > exists, which seems unlikely but time consuming to establish). > > Testing: > - Tier 1 (JPRT) testing with a modified launcher that uses the > primordial thread > - with guard pages enabled > - with guard pages disabled > - Tier 1 (JPRT) normal JDK build (which should be unaffected by this > change) > > The testing was primarily to ensure that disabling the stack guards on the > primordial thread wasn't totally broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when guards are enabled > and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM when guards are > disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the guards are > disabled > > Thanks, > David > > > From gromero at linux.vnet.ibm.com Fri Nov 17 20:01:57 2017 From: gromero at linux.vnet.ibm.com (Gustavo Romero) Date: Fri, 17 Nov 2017 18:01:57 -0200 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: <5A0F4035.9010302@linux.vnet.ibm.com> Hi Volker, Robbin! I could not find any damage to JVM NUMA detection on PPC64 regarding this change. So thumbs up from my side. Regards, Gustavo On 17-11-2017 12:21, Volker Simonis wrote: > Hi Gustavo, > > as our "PowerPC NUMA expert", do you see any problem with this change? > > Thank yo uand best regards, > Volker > > > On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >> Hi all, please review. >> >> Windows needs to run os::init_2 before allocation polling page to proper >> setup numa. I saw no reason not to have it for all platforms after init_2. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >> >> Code below. >> >> Tested tier 1-5/1-3 >> >> Thanks, Robbin >> >> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >> --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 >> +0100 >> +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 >> +0100 >> @@ -3560,10 +3560,10 @@ >> TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >> >> - SafepointMechanism::initialize(); >> - >> // Initialize the os module after parsing the args >> jint os_init_2_result = os::init_2(); >> if (os_init_2_result != JNI_OK) return os_init_2_result; >> >> + SafepointMechanism::initialize(); >> + >> jint adjust_after_os_result = Arguments::adjust_after_os(); >> if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; > From zgu at redhat.com Fri Nov 17 20:43:08 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 17 Nov 2017 15:43:08 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: References: <1510824184.3968.3.camel@redhat.com> Message-ID: <4ffdb4b6-8902-b5e7-b68c-e7a7e5210ed9@redhat.com> There were two bugs that resulted NMT report incorrect thread stacks. 1) NMT did not handle overlapped regions correctly. 2) There are two issues with Linux: get_stack_commited_bottom() * The method name is misleading, it does not return *committed* bottom, but *mapped* bottom, where address space is valid, not necessary committed. I renamed this function to get_stack_mapped_bottom() with option to return committed bottom with proper parameter. * The binary search algorithm mishandled edge cases. The test case for verifying fixes for above two issues can be found here: http://cr.openjdk.java.net/~zgu/8191369/test_mmap.c As mentioned in early email, new patch also contains implementation for Windows. Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ Test: Reran hotspot_tier1 tests on Windows x64 and Linux x64 (fastdebug and release) Thanks, -Zhengyu On 11/16/2017 03:42 PM, Zhengyu Gu wrote: > Sorry, I have to withdraw this code review request, since the stack > sizes reported on Linux do not match /proc//smaps. > > I am debugging the problem, will submit new request later. > > Thanks, > > -Zhengyu > > On 11/16/2017 01:30 PM, Zhengyu Gu wrote: >> Hi, >> >> I added Windows support and updated bug to make this enhancement Linux >> and Windows only. >> >> As Thomas suggested, other platforms can be addressed by followup fixes. >> >> Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ >> >> I reran hotspot_tier1_runtime on Linux x64 and Windows 10 (fastdebug + >> release) >> >> Thanks, >> >> -Zhengyu >> >> >> >> On 11/16/2017 04:45 AM, Thomas St?fe wrote: >>> >>> >>> On Thu, Nov 16, 2017 at 10:23 AM, Severin Gehwolf >>> > wrote: >>> >>> On Thu, 2017-11-16 at 08:14 +0100, Thomas St?fe wrote: >>> > On Thu, Nov 16, 2017 at 7:58 AM, David Holmes >>> > >>> > wrote: >>> > >>> > > Hi Zhengyu, >>> > > >>> > > I can't review this in detail as I'm not familiar with NMT >>> workings, but >>> > > have a couple of comments. >>> > > >>> > > On 16/11/2017 8:25 AM, Zhengyu Gu wrote: >>> > > >>> > > > This is Linux only enhancement for now, can be extended for >>> other >>> > > > platforms. (See bug for details) >>> > > > >>> > > >>> > > I'm concerned about unnecessary platform skew. I added some >>> comments to >>> > > the bug. I think the mincore trick can be used on Solaris and >>> AIX as well - >>> > > but not on OS X / BSD. It may be that VirtualQuery can be >>> used on Windows >>> > > to get the same information - but I'm unclear if the memory >>> states support >>> > > what you're looking for. It would be good to extend this to >>> other platforms >>> > > that will support it. >>> > > >>> > >>> > Just 5c: >>> > >>> > I think this is definitely useful even with only the Linux >>> implementation >>> > provided. >>> >>> +1 >>> >>> While plaform skew is a valid concern, I'm doubtful it's >>> "unnecessary". >>> I think there is value to get one platform fixed and then look at >>> other >>> platforms one by one as follow-up bugs. For some platforms it >>> might be >>> hard to do (or undesirable) as Thomas says. >>> >>> Thanks, >>> Severin >>> >>> >>> I volunteer to implement this for windows if it can be done as >>> non-time-critical follow up, if no-one else has stepped up in until >>> then. >>> >>> ..Thomas >>> >>> >>> > On AIX we have commit-on-touch and hence mincore may actually >>> force commit >>> > the thread :P - would have to try it but have no high hopes. >>> Well. >>> > >>> > On Windows this is possible, we already use VirtualQuery in our >>> fork to >>> > print out the commit-state of the current thread stack in case of >>> an error. >>> > >>> > ..Thomas >>> > >>> > >>> > > And on a minor note can you please correct the existing >>> spelling error in >>> > > get_stack_commited_bottom. :) >>> > > >>> > > Current implementation, thread stack is tracked when thread is >>> created, >>> > > > its available stack space is tagged as reserved and >>> committed. >>> > > > >>> > > > However, this is not how stack works. Stack pages are >>> committed on >>> > > > demand, so this approach overstates its memory usage. >>> > > > >>> > > > This enhancement gathers thread stack usage at NMT query >>> time, so that it >>> > > > can report more accurate usage. >>> > > > >>> > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191369 >>> >>> > > > Webrev: >>> http://cr.openjdk.java.net/~zgu/8191369/webrev.00/index.html >>> >>> > > > >>> > > > >>> > > > Example: >>> > > > >>> > > > Summary shows thread stacks only use small faction of >>> reserved space. >>> > > > >>> > > > - Thread (reserved=41216KB, >>> committed=632KB) >>> > > > (thread #41) >>> > > > (stack: reserved=41032KB, >>> committed=448KB) >>> > > > (malloc=137KB #222) >>> > > > (arena=47KB #76) >>> > > > >>> > > > >>> > > > Detail tracking shows stack guards and actually >>> used/committed stack area. >>> > > > >>> > > > [0x00007f66e95e7000 - 0x00007f66e96e8000] reserved 1028KB for >>> Thread >>> > > > Stack from >>> > > > [0x00007f67c0b57d5c] JavaThread::run()+0x2c >>> > > > [0x00007f67c08bbea2] thread_native_entry(Thread*)+0x112 >>> > > > >>> > > > [0x00007f66e95e7000 - 0x00007f66e95eb000] committed >>> 16KB from >>> > > > [0x00007f67c08b7319] >>> os::pd_create_stack_guard_pages(char*, >>> > > > unsigned long)+0x29 >>> > > > [0x00007f67c0b56851] >>> JavaThread::create_stack_guard_pages() >>> > > > [clone .part.125]+0xa1 >>> > > > [0x00007f67c0b57d6e] JavaThread::run()+0x3e >>> > > > [0x00007f67c08bbea2] >>> thread_native_entry(Thread*)+0x112 >>> > > > >>> > > > [0x00007f66e96e7000 - 0x00007f66e96e8000] >>> committed 4KB >>> > > > >>> > > >>> > > Can we capture this in a test so we can tell that the >>> tracking has >>> > > improved? >>> > > >>> > > Thanks, >>> > > David >>> > > ----- >>> > > >>> > > Thanks, >>> > > > >>> > > > -Zhengyu >>> > > > >>> > > > >>> >>> From jiangli.zhou at Oracle.COM Sat Nov 18 00:05:16 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Fri, 17 Nov 2017 16:05:16 -0800 Subject: RFR:8187118, 8187119: Removing -cp class path entries from boot append list at CDS dump time and other cleanups Message-ID: Please review the changes for following two RFEs. 8187118: Remove appending -cp path to the boot class path at AppCDS dump time. https://bugs.openjdk.java.net/browse/JDK-8187118?filter=14921 8187119: Consolidate record_shared_class_loader_type() and record_result(). https://bugs.openjdk.java.net/browse/JDK-8187119?filter=14921 webrev: http://cr.openjdk.java.net/~jiangli/8187118_8187119/webrev.01 With the changes in above webrev, I?ve done some cleanups in CDS/AppCDS code: - Removed -cp class path entries from boot append list at CDS dump time as the application classes are no longer loaded by the boot loader at CDS dump time. The -cp path entries are maintained in a separate list by ClassLoader now. CDS code creates the shared class path entry table by combining the boot append list an -cp path entry list (see FileMapInfo::allocate_classpath_entry_table()). Split ClassLoaderExt::add_class_path_entry() logic into add_to_boot_append_entries() (renamed from ClassLoader::add_to_list()) and add_to_classpath_entries(). - Cleaned up SystemDictionary::load_instance_class() to remove the handling of DumpSharedSpaces cases from general class loading code. - Separated ClassLoader::add_package() call from ClassLoaderExt::Context::record_result(). Calling add_package() in record_result() complicates things unnecessarily. ClassLoader::add_package() is needed for classes loaded by the boot loader, while Context::record_result() is only needed for CDS dump time. Logically they are unrelated. Separating the two makes the code much cleaner. - Renamed ClassLoader::record_shared_class_loader_type() to ClassLoader::record_result(). Cleaned up the usage of ClassLoader::record_shared_class_loader_type() and Context::record_result(). Before the clean up, Context::record_result() was called twice during loading of classes for the boot loader at CDS dump time, and the classes for boot loader and PlatformClassLoader/AppClassLoader were handled differently. Now all classes loaded by the builtin class loaders at CDS dump time are handled the same. - Removed the use of ClassLoader::_num_entries and _num_boot_entries. - Renamed ClassLoader::setup_search_path() to setup_boot_search_path(). - Removed unused ClassLoader::add_to_list(const char*) Tested with hotspot/runtime locally on linux-x64. Ran tier1, tier2, tier3, teir4 and tier5 tests. Thanks, Jiangli From lois.foltan at oracle.com Sat Nov 18 00:54:25 2017 From: lois.foltan at oracle.com (Lois Foltan) Date: Fri, 17 Nov 2017 19:54:25 -0500 Subject: RFR:8187118, 8187119: Removing -cp class path entries from boot append list at CDS dump time and other cleanups In-Reply-To: References: Message-ID: <6a41e358-5077-487d-884c-31828201accf@oracle.com> Looks good! Lois On 11/17/2017 7:05 PM, Jiangli Zhou wrote: > Please review the changes for following two RFEs. > > 8187118: Remove appending -cp path to the boot class path at AppCDS dump time. https://bugs.openjdk.java.net/browse/JDK-8187118?filter=14921 > 8187119: Consolidate record_shared_class_loader_type() and record_result(). https://bugs.openjdk.java.net/browse/JDK-8187119?filter=14921 > > webrev: http://cr.openjdk.java.net/~jiangli/8187118_8187119/webrev.01 > > With the changes in above webrev, I?ve done some cleanups in CDS/AppCDS code: > > - Removed -cp class path entries from boot append list at CDS dump time as the application classes are no longer loaded by the boot loader at CDS dump time. The -cp path entries are maintained in a separate list by ClassLoader now. CDS code creates the shared class path entry table by combining the boot append list an -cp path entry list (see FileMapInfo::allocate_classpath_entry_table()). Split ClassLoaderExt::add_class_path_entry() logic into add_to_boot_append_entries() (renamed from ClassLoader::add_to_list()) and add_to_classpath_entries(). > > - Cleaned up SystemDictionary::load_instance_class() to remove the handling of DumpSharedSpaces cases from general class loading code. > > - Separated ClassLoader::add_package() call from ClassLoaderExt::Context::record_result(). Calling add_package() in record_result() complicates things unnecessarily. ClassLoader::add_package() is needed for classes loaded by the boot loader, while Context::record_result() is only needed for CDS dump time. Logically they are unrelated. Separating the two makes the code much cleaner. > > - Renamed ClassLoader::record_shared_class_loader_type() to ClassLoader::record_result(). Cleaned up the usage of ClassLoader::record_shared_class_loader_type() and Context::record_result(). Before the clean up, Context::record_result() was called twice during loading of classes for the boot loader at CDS dump time, and the classes for boot loader and PlatformClassLoader/AppClassLoader were handled differently. Now all classes loaded by the builtin class loaders at CDS dump time are handled the same. > > - Removed the use of ClassLoader::_num_entries and _num_boot_entries. > - Renamed ClassLoader::setup_search_path() to setup_boot_search_path(). > - Removed unused ClassLoader::add_to_list(const char*) > > Tested with hotspot/runtime locally on linux-x64. Ran tier1, tier2, tier3, teir4 and tier5 tests. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Sat Nov 18 00:55:34 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 17 Nov 2017 16:55:34 -0800 Subject: RFR:8187118, 8187119: Removing -cp class path entries from boot append list at CDS dump time and other cleanups In-Reply-To: <6a41e358-5077-487d-884c-31828201accf@oracle.com> References: <6a41e358-5077-487d-884c-31828201accf@oracle.com> Message-ID: <586D9348-AC00-4939-AD9B-8353FFC0DA69@oracle.com> Thank you for the review and testing suggestions, Lois! Jiangli > On Nov 17, 2017, at 4:54 PM, Lois Foltan wrote: > > Looks good! > Lois > > On 11/17/2017 7:05 PM, Jiangli Zhou wrote: >> Please review the changes for following two RFEs. >> >> 8187118: Remove appending -cp path to the boot class path at AppCDS dump time. https://bugs.openjdk.java.net/browse/JDK-8187118?filter=14921 >> 8187119: Consolidate record_shared_class_loader_type() and record_result(). https://bugs.openjdk.java.net/browse/JDK-8187119?filter=14921 >> >> webrev: http://cr.openjdk.java.net/~jiangli/8187118_8187119/webrev.01 >> >> With the changes in above webrev, I?ve done some cleanups in CDS/AppCDS code: >> >> - Removed -cp class path entries from boot append list at CDS dump time as the application classes are no longer loaded by the boot loader at CDS dump time. The -cp path entries are maintained in a separate list by ClassLoader now. CDS code creates the shared class path entry table by combining the boot append list an -cp path entry list (see FileMapInfo::allocate_classpath_entry_table()). Split ClassLoaderExt::add_class_path_entry() logic into add_to_boot_append_entries() (renamed from ClassLoader::add_to_list()) and add_to_classpath_entries(). >> >> - Cleaned up SystemDictionary::load_instance_class() to remove the handling of DumpSharedSpaces cases from general class loading code. >> >> - Separated ClassLoader::add_package() call from ClassLoaderExt::Context::record_result(). Calling add_package() in record_result() complicates things unnecessarily. ClassLoader::add_package() is needed for classes loaded by the boot loader, while Context::record_result() is only needed for CDS dump time. Logically they are unrelated. Separating the two makes the code much cleaner. >> >> - Renamed ClassLoader::record_shared_class_loader_type() to ClassLoader::record_result(). Cleaned up the usage of ClassLoader::record_shared_class_loader_type() and Context::record_result(). Before the clean up, Context::record_result() was called twice during loading of classes for the boot loader at CDS dump time, and the classes for boot loader and PlatformClassLoader/AppClassLoader were handled differently. Now all classes loaded by the builtin class loaders at CDS dump time are handled the same. >> >> - Removed the use of ClassLoader::_num_entries and _num_boot_entries. >> - Renamed ClassLoader::setup_search_path() to setup_boot_search_path(). >> - Removed unused ClassLoader::add_to_list(const char*) >> >> Tested with hotspot/runtime locally on linux-x64. Ran tier1, tier2, tier3, teir4 and tier5 tests. >> >> Thanks, >> Jiangli > From ioi.lam at oracle.com Sat Nov 18 02:22:14 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 17 Nov 2017 18:22:14 -0800 Subject: RFR:8187118, 8187119: Removing -cp class path entries from boot append list at CDS dump time and other cleanups In-Reply-To: <6a41e358-5077-487d-884c-31828201accf@oracle.com> References: <6a41e358-5077-487d-884c-31828201accf@oracle.com> Message-ID: Looks good. Thanks! - Ioi On 11/17/17 4:54 PM, Lois Foltan wrote: > Looks good! > Lois > > On 11/17/2017 7:05 PM, Jiangli Zhou wrote: >> Please review the changes for following two RFEs. >> >> 8187118: Remove appending -cp path to the boot class path at AppCDS >> dump time. >> https://bugs.openjdk.java.net/browse/JDK-8187118?filter=14921 >> >> 8187119: Consolidate record_shared_class_loader_type() and >> record_result(). >> https://bugs.openjdk.java.net/browse/JDK-8187119?filter=14921 >> >> >> webrev: http://cr.openjdk.java.net/~jiangli/8187118_8187119/webrev.01 >> >> >> With the changes in above webrev, I?ve done some cleanups in >> CDS/AppCDS code: >> >> - Removed -cp class path entries from boot append list at CDS dump >> time as the application classes are no longer loaded by the boot >> loader at CDS dump time. The -cp path entries are maintained in a >> separate list by ClassLoader now. CDS code creates the shared class >> path entry table by combining the boot append list an -cp path entry >> list (see FileMapInfo::allocate_classpath_entry_table()). Split >> ClassLoaderExt::add_class_path_entry() logic into >> add_to_boot_append_entries() (renamed from >> ClassLoader::add_to_list()) and add_to_classpath_entries(). >> >> - Cleaned up SystemDictionary::load_instance_class() to remove the >> handling of DumpSharedSpaces cases from general class loading code. >> >> - Separated ClassLoader::add_package() call from >> ClassLoaderExt::Context::record_result(). Calling add_package() in >> record_result() complicates things unnecessarily. >> ClassLoader::add_package() is needed for classes loaded by the boot >> loader, while Context::record_result() is only needed for CDS dump >> time. Logically they are unrelated. Separating the two makes the code >> much cleaner. >> >> - Renamed ClassLoader::record_shared_class_loader_type() to >> ClassLoader::record_result(). Cleaned up the usage of >> ClassLoader::record_shared_class_loader_type() and >> Context::record_result(). Before the clean up, >> Context::record_result() was called twice during loading of classes >> for the boot loader at CDS dump time, and the classes for boot loader >> and PlatformClassLoader/AppClassLoader were handled differently. Now >> all classes loaded by the builtin class loaders at CDS dump time are >> handled the same. >> >> - Removed the use of ClassLoader::_num_entries and _num_boot_entries. >> - Renamed ClassLoader::setup_search_path() to setup_boot_search_path(). >> - Removed unused ClassLoader::add_to_list(const char*) >> >> Tested with hotspot/runtime locally on linux-x64. Ran tier1, tier2, >> tier3, teir4 and tier5 tests. >> >> Thanks, >> Jiangli > From jiangli.zhou at oracle.com Sat Nov 18 04:54:09 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 17 Nov 2017 20:54:09 -0800 Subject: RFR:8187118, 8187119: Removing -cp class path entries from boot append list at CDS dump time and other cleanups In-Reply-To: References: <6a41e358-5077-487d-884c-31828201accf@oracle.com> Message-ID: Thanks, Ioi! Thanks, Jiangli > On Nov 17, 2017, at 6:22 PM, Ioi Lam wrote: > > Looks good. Thanks! > > - Ioi > > >> On 11/17/17 4:54 PM, Lois Foltan wrote: >> Looks good! >> Lois >> >>> On 11/17/2017 7:05 PM, Jiangli Zhou wrote: >>> Please review the changes for following two RFEs. >>> >>> 8187118: Remove appending -cp path to the boot class path at AppCDS dump time. https://bugs.openjdk.java.net/browse/JDK-8187118?filter=14921 >>> 8187119: Consolidate record_shared_class_loader_type() and record_result(). https://bugs.openjdk.java.net/browse/JDK-8187119?filter=14921 >>> >>> webrev: http://cr.openjdk.java.net/~jiangli/8187118_8187119/webrev.01 >>> >>> With the changes in above webrev, I?ve done some cleanups in CDS/AppCDS code: >>> >>> - Removed -cp class path entries from boot append list at CDS dump time as the application classes are no longer loaded by the boot loader at CDS dump time. The -cp path entries are maintained in a separate list by ClassLoader now. CDS code creates the shared class path entry table by combining the boot append list an -cp path entry list (see FileMapInfo::allocate_classpath_entry_table()). Split ClassLoaderExt::add_class_path_entry() logic into add_to_boot_append_entries() (renamed from ClassLoader::add_to_list()) and add_to_classpath_entries(). >>> >>> - Cleaned up SystemDictionary::load_instance_class() to remove the handling of DumpSharedSpaces cases from general class loading code. >>> >>> - Separated ClassLoader::add_package() call from ClassLoaderExt::Context::record_result(). Calling add_package() in record_result() complicates things unnecessarily. ClassLoader::add_package() is needed for classes loaded by the boot loader, while Context::record_result() is only needed for CDS dump time. Logically they are unrelated. Separating the two makes the code much cleaner. >>> >>> - Renamed ClassLoader::record_shared_class_loader_type() to ClassLoader::record_result(). Cleaned up the usage of ClassLoader::record_shared_class_loader_type() and Context::record_result(). Before the clean up, Context::record_result() was called twice during loading of classes for the boot loader at CDS dump time, and the classes for boot loader and PlatformClassLoader/AppClassLoader were handled differently. Now all classes loaded by the builtin class loaders at CDS dump time are handled the same. >>> >>> - Removed the use of ClassLoader::_num_entries and _num_boot_entries. >>> - Renamed ClassLoader::setup_search_path() to setup_boot_search_path(). >>> - Removed unused ClassLoader::add_to_list(const char*) >>> >>> Tested with hotspot/runtime locally on linux-x64. Ran tier1, tier2, tier3, teir4 and tier5 tests. >>> >>> Thanks, >>> Jiangli > From daniel.daugherty at oracle.com Sun Nov 19 01:49:43 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Sat, 18 Nov 2017 20:49:43 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> Message-ID: <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Greetings, Testing of the last round of changes revealed a hang in one of the new TLH tests. Robbin has fixed the hang, updated the existing TLH test, and added another TLH test for good measure. Here is the updated full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ Here is the updated delta webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are no unexpected failures. We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: > Greetings, > > Robbin rebased the project last night/this morning to merge with Thread > Local Handshakes (TLH) and also picked up additional changesets up thru: > >> Changeset: fa736014cf28 >> Author:??? cjplummer >> Date:????? 2017-11-14 18:08 -0800 >> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >> >> 8191049: Add alternate version of pns() that is callable from within >> hotspot source >> Summary: added pns2() to debug.cpp >> Reviewed-by: stuefe, gthornbr > > This is the first time we've rebased the project to bits that are this > fresh (< 12 hours old at rebase time). We've done this because we think > we're done with this project and are in the final review-change-retest > cycle(s)... In other words, we're not planning on making any more major > changes for this project... :-) > > *** Time for code reviewers to chime in on this thread! *** > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ > > Here's is the delta webrev from the 2017.11.10 rebase: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ > > Dan has submitted the bits for the usual Mach5 tier[1-5] testing > (and the new baseline also)... > > We're expecting this round to be a little noisier than usual because > we did not rebase on a PIT snapshot so the new baseline has not been > through Jesper's usual care-and-feeding of integration_blockers, etc. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >> (Yes, we're playing chase-the-repo...) >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >> >> Unlike the previous rebase, there were no changes required to the >> open code to get the rebased bits to build so there is no delta >> webrev. >> >> These bits have been run through JPRT and I've submitted the usual >> Mach5 tier[1-5] test run... >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>> >>> Here are the updated webrevs: >>> >>> Here's the mq comment for the change: >>> >>> ? Rebase to 2017.10.25 PIT snapshot. >>> >>> Here is the full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>> >>> And here is the delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>> >>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>> weekend. Didn't see any issues in a quick look. Going to take a closer >>> look today. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Resolving one of the code review comments (from both Stefan K and >>>> Coleen) >>>> on jdk10-04-full required quite a few changes so it is being done as a >>>> standalone patch and corresponding webrevs: >>>> >>>> Here's the mq comment for the change: >>>> >>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>> ??? JavaThreadIteratorWithHandle. >>>> >>>> Here is the full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>> >>>> And here is the delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> We have a (eXtra Large) fix for the following bug: >>>>> >>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>> >>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>> >>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>> and track the work on this project: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>> >>>>> >>>>> Dan has noticed that the indenting is wrong in some of the code >>>>> quotes >>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>> solution for that problem yet. >>>>> >>>>> Here's the webrev for current JDK10 version of this fix: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>> >>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>> additional testing on Erik and Robbin's machines. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Daniel Daugherty >>>>> Erik Osterlund >>>>> Robbin Ehn >>>>> >>>> >>>> >>> >>> >> >> > > From david.holmes at oracle.com Sun Nov 19 12:34:28 2017 From: david.holmes at oracle.com (David Holmes) Date: Sun, 19 Nov 2017 22:34:28 +1000 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: Hi Robbin, I need to study this more closely. Need to check the original code before the TLH code, to evaluate impact. Initialization order issues are rarely simple to fix as there are so many hidden dependencies. Will get back to you asap. David On 18/11/2017 12:10 AM, Robbin Ehn wrote: > Hi all, please review. > > Windows needs to run os::init_2 before allocation polling page to proper > setup numa. I saw no reason not to have it for all platforms after init_2. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 > > Code below. > > Tested tier 1-5/1-3 > > Thanks, Robbin > > diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp > --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 2017 > +0100 > +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 2017 > +0100 > @@ -3560,10 +3560,10 @@ > ?? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); > > -? SafepointMechanism::initialize(); > - > ?? // Initialize the os module after parsing the args > ?? jint os_init_2_result = os::init_2(); > ?? if (os_init_2_result != JNI_OK) return os_init_2_result; > > +? SafepointMechanism::initialize(); > + > ?? jint adjust_after_os_result = Arguments::adjust_after_os(); > ?? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From david.holmes at oracle.com Mon Nov 20 01:09:31 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 11:09:31 +1000 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> Message-ID: <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> On 17/11/2017 2:33 AM, Bob Vandette wrote: > Yes, that?s the fix. I built it correctly with that added include. How? I can't build the MinimalVM in JPRT because the UNSUPPORTED_OPTION macro is only locally defined in arguments.cpp, but is now used in gcArguments.cpp: /opt/jprt/T/P1/003543.daholme/s/open/src/hotspot/share/gc/shared/gcArguments.cpp:77:29: error: 'UNSUPPORTED_OPTION' was not declared in this scope UNSUPPORTED_OPTION(UseG1GC); ^ make[3]: *** [/opt/jprt/T/P1/003543.daholme/s/build/linux-x86-debug/hotspot/variant-minimal/libjvm/objs/gcArguments.o] Error 1 --- Roman/Erik: If changing code that has #include guards you must ensure all the paths are tested! Thanks, David > There are still a few other issues with building and using the minimal VM but those > existed before your change. > > 1. You cannot build the minimal VM without also building the server VM. There > is a Makefile dependency on the server VM. > 2. The jvm.cfg produced in the jvm and jre images does not add the minimalvm > as one of the possible VM to select. As a work-around jlink produces a workable jvm.cfg. > > Thanks, > Bob. > > >> On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: >> >> Hi Bob, >> thanks for letting me know. >> >> I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 >> and posted for review: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html >> >> Thanks, Roman >> >>> Roman, >>> >>> It looks like this change may have caused the build of the minimal VM to >>> fail. The minimal VM is no longer a configuration that we regularly build and test >>> but it is still a build option that can be selected via "--with-jvm-variants=minimal1" >>> >>> >>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: In static member function 'static jint GCArguments::initialize()': >>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: error: 'defaultStream' has not been declared >>> jio_fprintf(defaultStream::error_stream(), "UseParallelGC not supported in this VM.\n"); >>> ^ >>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: error: 'defaultStream' has not been declared >>> jio_fprintf(defaultStream::error_stream(), "UseG1GC not supported in this VM.\n"); >>> ^ >>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: error: 'defaultStream' has not been declared >>> jio_fprintf(defaultStream::error_stream(), "UseConcMarkSweepGC not supported in this VM.\n"); >>> ^ >>> gmake[3]: *** [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] Error 1 >>> >>> Bob. >>> >>> >>>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>>> >>>> Hi Per & Erik, >>>> >>>> thanks for the reviews! >>>> >>>> Now I need a sponsor. >>>> >>>> Final webrev (same as before, including Reviewed-by line): >>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>>> >>>> Thanks, Roman >>>> >>>> >>>>> Looks good Roman! >>>>> >>>>> /Per >>>>> >>>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>>> Hi, >>>>>>> >>>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>>> [...] >>>>>>>>> So let me just put your changes up for review (again), if you don't >>>>>>>>> mind: >>>>>>>>> >>>>>>>>> Full webrev: >>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>>> >>>>>>> I like this. Just two naming suggestions: >>>>>>> >>>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>>> >>>>>>> 39 static jint create_instance(); >>>>>>> 40 static bool is_initialized(); >>>>>>> 41 static GCArguments* instance(); >>>>>>> >>>>>>> Could we call these: >>>>>>> >>>>>>> 39 static jint initialize(); >>>>>>> 40 static bool is_initialized(); >>>>>>> 41 static GCArguments* arguments(); >>>>>>> >>>>>>> Reasoning: initialize() maps better to its companion is_initialized(). >>>>>>> GCArguments::arguments() maps better to the existing patterns we have >>>>>>> with CollectedHeap::heap(). >>>>>> Ok, that sounds good to me. Actually, it's almost full-circle back to my >>>>>> original proposal ;-) >>>>>> >>>>>> Differential: >>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>>> >>>>>> Full: >>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>>> >>>>>> >>>>>> Ok to push now? >>>>>> >>>>>> Roman >>>> >> > From david.holmes at oracle.com Mon Nov 20 01:31:04 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 11:31:04 +1000 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> Message-ID: <0944d8e4-1f00-8c3c-2c50-a598acd098ad@oracle.com> Typo ... On 20/11/2017 11:09 AM, David Holmes wrote: > On 17/11/2017 2:33 AM, Bob Vandette wrote: >> Yes, that?s the fix.? I built it correctly with that added include. > > How? I can't build the MinimalVM in JPRT because the UNSUPPORTED_OPTION > macro is only locally defined in arguments.cpp, but is now used in > gcArguments.cpp: > > /opt/jprt/T/P1/003543.daholme/s/open/src/hotspot/share/gc/shared/gcArguments.cpp:77:29: > error: 'UNSUPPORTED_OPTION' was not declared in this scope > ?? UNSUPPORTED_OPTION(UseG1GC); > ???????????????????????????? ^ > make[3]: *** > [/opt/jprt/T/P1/003543.daholme/s/build/linux-x86-debug/hotspot/variant-minimal/libjvm/objs/gcArguments.o] > Error 1 > > --- > > Roman/Erik: If changing code that has #include guards you must ensure > all the paths are tested! #include -> #ifdef David > Thanks, > David > >> There are still a few other issues with building and using the minimal >> VM but those >> existed before your change. >> >> 1. You cannot build the minimal VM without also building the server >> VM.? There >> is a Makefile dependency on the server VM. >> 2. The jvm.cfg produced in the jvm and jre images does not add the >> minimalvm >> as one of the possible VM to select.? As a work-around jlink produces >> a workable jvm.cfg. >> >> Thanks, >> Bob. >> >> >>> On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: >>> >>> Hi Bob, >>> thanks for letting me know. >>> >>> I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 >>> and posted for review: >>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html >>> >>> >>> Thanks, Roman >>> >>>> Roman, >>>> >>>> It looks like this change may have caused the build of the minimal >>>> VM to >>>> fail.? The minimal VM is no longer a configuration that we regularly >>>> build and test >>>> but it is still a build option that can be selected via >>>> "--with-jvm-variants=minimal1" >>>> >>>> >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: >>>> In static member function 'static jint GCArguments::initialize()': >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseParallelGC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseG1GC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseConcMarkSweepGC >>>> not supported in this VM.\n"); >>>> ????????????????? ^ >>>> gmake[3]: *** >>>> [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] >>>> Error 1 >>>> >>>> Bob. >>>> >>>> >>>>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>>>> >>>>> Hi Per & Erik, >>>>> >>>>> thanks for the reviews! >>>>> >>>>> Now I need a sponsor. >>>>> >>>>> Final webrev (same as before, including Reviewed-by line): >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>>>> >>>>> >>>>> Thanks, Roman >>>>> >>>>> >>>>>> Looks good Roman! >>>>>> >>>>>> /Per >>>>>> >>>>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>>>> [...] >>>>>>>>>> So let me just put your changes up for review (again), if you >>>>>>>>>> don't >>>>>>>>>> mind: >>>>>>>>>> >>>>>>>>>> Full webrev: >>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>>>> >>>>>>>> I like this. Just two naming suggestions: >>>>>>>> >>>>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>>>> >>>>>>>> ?? 39?? static jint create_instance(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* instance(); >>>>>>>> >>>>>>>> Could we call these: >>>>>>>> >>>>>>>> ?? 39?? static jint initialize(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* arguments(); >>>>>>>> >>>>>>>> Reasoning: initialize() maps better to its companion >>>>>>>> is_initialized(). >>>>>>>> GCArguments::arguments() maps better to the existing patterns we >>>>>>>> have >>>>>>>> with CollectedHeap::heap(). >>>>>>> Ok, that sounds good to me. Actually, it's almost full-circle >>>>>>> back to my >>>>>>> original proposal ;-) >>>>>>> >>>>>>> Differential: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>>>> >>>>>>> Full: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>>>> >>>>>>> >>>>>>> Ok to push now? >>>>>>> >>>>>>> Roman >>>>> >>> >> From david.holmes at oracle.com Mon Nov 20 03:44:45 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 13:44:45 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> Message-ID: <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> Hi Dan, Thanks for the review. I've addressed all of yours and Thomas's comments. http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ The main change is: + static bool is_primordial_thread(void) + #if defined(_WINDOWS) || defined(BSD) + // No way to identify the primordial thread. + { return false; } + #else + ; + #endif Also to clarify this is logically a "product" flag for the systems that need it, not a diagnostic one. But as we don't want this flag to be in common use and it is of such narrow applicability, we have made it experimental. This was discussed previously in the CSR review thread for this: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html Once Tomas K. has been able to test this, and you and Thomas sign-off on the updates, I'll get this pushed. But as I'm away from Thursday it may not be until next week. Thanks, David ----- On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: > On 11/14/17 5:39 AM, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > src/hotspot/os/aix/os_aix.cpp > ??? No comments. > > src/hotspot/os/aix/os_aix.hpp > ??? No comments. > > src/hotspot/os/bsd/os_bsd.cpp > ??? L880: // thread in any special way - so just report 'false' > ?? ? ?? Nit - needs a period after 'false' to make it a sentence. > > ??? L882: ??? return false; > ??????? Nit - indent should be 2 spaces. > > src/hotspot/os/bsd/os_bsd.hpp > ??? No comments. > > src/hotspot/os/linux/os_linux.cpp > ??? L947: ? if (stack_size >= (size_t)(3 * page_size())) > ??????? Nit - needs '{' and '}' on the if-statement body. > > ??? L3070: // always place it right after end of the mapped region > ??????? Nit - needs a period after 'region' to make it a sentence. > ??????? (Not your problem, but you did update the sentence...) > > src/hotspot/os/linux/os_linux.hpp > ??? No comments. > > src/hotspot/os/solaris/os_solaris.cpp > ??? No comments other than I learned something new about thr_main() :-) > ??? Thanks for cleaning up that mess. > > src/hotspot/os/windows/os_windows.cpp > ??? L452: // thread in any special way - so just report 'false' > ??????? Nit - needs a period after 'false' to make it a sentence. > > ??? L454: ??? return false; > ??????? Nit - indent should be 2 spaces. > > src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp > ??? No comments. > > src/hotspot/share/runtime/globals.hpp > ??? L1181: ? experimental(bool, DisablePrimordialThreadGuardPages, false, > ??????? Experimental or diagnostic? > > src/hotspot/share/runtime/os.hpp > ??? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM > ??????? Nit - needs a period after 'JNI_CreateJavaVM'. > > ??? L453: ? // launcher nevers uses the primordial thread as the main > thread, but > ??????? Typo - 'nevers' -> 'never' > > ??? L455: ? // have to special-case handling of the primordial thread > if it attaches > ??????? Typo - 'have to' -> 'have' > > src/hotspot/share/runtime/thread.cpp > ??? I like the new log message. > > src/hotspot/share/runtime/thread.inline.hpp > ??? L159: ? if (os::uses_stack_guard_pages() && > ??? L160: ????? !(DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ??????? I'm not sure why, but I had to read these two lines several > ??????? times before I convinced myself the logic is right. > > I'm good with the code changes. Your call on whether to fix the > bits or not. > > Regarding this part of Thomas' review: > >>> Or, something like this: >>> static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) >>> BSD_ONLY({return false;}) >> >> I can do that. I'll see if anyone else has any comments on this. > > I'm assuming that Thomas means for the above to be at the > platform independent layer (runtime/os.hpp?) so it is more > obvious to the reader that the function doesn't do anything > on those two platforms... I know we dislike such things at > the platform independent layer, but it does feel appropriate > to make this limitation more obvious. > > I would be okay if you made the change that Thomas is suggesting. > > Thumbs up! > > Dan > > >> >> >> There are three parts to this which made it somewhat more complicated >> than I had envisaged: >> >> 1. Add the flag and disable the guards. >> >> This is relatively straight-forward: >> >> ?void JavaThread::create_stack_guard_pages() { >> ?? if (!os::uses_stack_guard_pages() || >> ?????? _stack_guard_state != stack_guard_unused || >> ?????? (DisablePrimordialThreadGuardPages && >> os::is_primordial_thread())) { >> ?????? log_info(os, thread)("Stack guard page creation for thread " >> ??????????????????????????? UINTX_FORMAT " disabled", >> os::current_thread_id()); >> ???? return; >> >> with a tweak in JavaThread::stack_guards_enabled as well. >> >> 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() >> >> We have three platforms that already implement this functionality: >> Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do >> anything different for the primordial thread, and we don't have a way >> to detect the primordial thread - so is_primordial_thread() just >> returns false. >> >> I also tidied up some comments regarding terminology. >> >> 3. Thomas noticed that we unconditionally subtract 2*page_size() from >> the rlimit stack size, without checking it was bigger than >> 2*page_size() to start with. I was unsure how to tackle this. It's no >> good leaving stack_size at zero so I opted to ensure we would have at >> least one page left. Of course in such cases we would hit the bug in >> libc (if it still exists, which seems unlikely but time consuming to >> establish). >> >> Testing: >> ? - Tier 1 (JPRT) testing with a modified launcher that uses the >> primordial thread >> ??? - with guard pages enabled >> ??? - with guard pages disabled >> ? - Tier 1 (JPRT) normal JDK build (which should be unaffected by this >> change) >> >> The testing was primarily to ensure that disabling the stack guards on >> the primordial thread wasn't totally broken. A number of tests fail: >> - setting -Xss32m fails for the primordial thread when guards are >> enabled and the rlimit is 8m >> - tests that would hit StackOverflowError crash the VM when guards are >> disabled (as you would expect). >> - runtime/execstack/Testexecstack.java crashes when the guards are >> disabled >> >> Thanks, >> David >> > From david.holmes at oracle.com Mon Nov 20 04:28:07 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 14:28:07 +1000 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> Message-ID: <65e9bb15-e9e2-8220-15a8-3630df0fd139@oracle.com> Hi Robbin, tl;dr: looks good. reviewed. So with TLH you introduced SafepointMechanism::initialize() and inserted it into the init sequence here: + SafepointMechanism::initialize(); + // Initialize the os module after parsing the args jint os_init_2_result = os::init_2(); so prior to os::init_2(). Most (all?) of the code in SafepointMechanism::initialize() was actually factored out from the platform specific os::init_2() functions. But in doing the factoring you replaced native OS calls like VirtualAlloc with more abstract os::* APIs. So now you have a method that depends on various os::* APIs but which is called before os initialization is fully complete. And it turns out on Windows those functions depended on something only done in os::init_2(), so you've now moved the initialization function to after os::init_2(). So as long as there is nothing in os::init_2() that may depend on the initialization now done afterwards via SafepointMechanism::initialize(), this should be a good fix - and handle any other unforeseen init_2() dependencies that may not have surfaced. As should not be encountering safepoints or handshakes until well after this point in the init sequence this all seems okay to me. Thanks, David On 19/11/2017 10:34 PM, David Holmes wrote: > Hi Robbin, > > I need to study this more closely. Need to check the original code > before the TLH code, to evaluate impact. > > Initialization order issues are rarely simple to fix as there are so > many hidden dependencies. > > Will get back to you asap. > > David > > On 18/11/2017 12:10 AM, Robbin Ehn wrote: >> Hi all, please review. >> >> Windows needs to run os::init_2 before allocation polling page to >> proper setup numa. I saw no reason not to have it for all platforms >> after init_2. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >> >> Code below. >> >> Tested tier 1-5/1-3 >> >> Thanks, Robbin >> >> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >> --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 2017 >> +0100 >> +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 2017 >> +0100 >> @@ -3560,10 +3560,10 @@ >> ??? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >> >> -? SafepointMechanism::initialize(); >> - >> ??? // Initialize the os module after parsing the args >> ??? jint os_init_2_result = os::init_2(); >> ??? if (os_init_2_result != JNI_OK) return os_init_2_result; >> >> +? SafepointMechanism::initialize(); >> + >> ??? jint adjust_after_os_result = Arguments::adjust_after_os(); >> ??? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From david.holmes at oracle.com Mon Nov 20 05:51:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 15:51:06 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> Hi Dan, Figured I'd better cast an eye over this again before it gets pushed :) Only one thing (repeated many times) jumped out at me and apologies if this has already been debated back and forth. I missed the exchange where the for loop iteration was rewritten into this unusual form: for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) { On first reading I expect most people would mistake the control expression for the iteration-expression due to the next(). I didn't even know the control expression could introduce a local variable! I find this really hard to read stylistically and far too cute/clever for its own good. It also violates the style rules regarding implicit boolean checks. I know Stefan proposed this as he did not like the alternate (in a few places): + { + ThreadsListHandle tlh; + JavaThreadIterator jti(tlh.list()); + for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { ... + } but it seems to me the iterator code has evolved since then and only a couple of places need a distinct TLH separate from the actual iterator object. So I'm more in favour of the more classic for-loop, with the iterator declared before the loop. Or even convert to while loops, as this iterator seems more suited to that. Other than that just a couple of comments on variable type choices, and a few typos spotted below. Thanks, David --- src/hotspot/share/runtime/globals.hpp 2476 diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, \ 2477 "Enable Thread SMR extra validity checks") \ 2478 \ 2479 diagnostic(bool, EnableThreadSMRStatistics, true, \ 2480 "Enable Thread SMR Statistics") \ Indent of strings is off by 3 spaces. --- src/hotspot/share/runtime/handshake.cpp 140 // There is assumption in code that Threads_lock should be lock 200 // There is assumption in code that Threads_lock should be lock lock -> locked 146 // handshake_process_by_vmthread will check the semaphore for us again Needs period at end. 148 // should be okey since the new thread will not have an operation. okey -> okay 151 // We can't warn here is since the thread do cancel_handshake after it have been removed I think: // We can't warn here since the thread does cancel_handshake after it has been removed 152 // from ThreadsList. So we should just keep looping here until while below return negative. from -> from the Needs period at end. 204 // A new thread on the ThreadsList will not have an operation. Change period to comma, and ... 205 // Hence is skipped in handshake_process_by_vmthread. Hence is -> hence it is --- src/hotspot/share/runtime/thread.cpp 1482 // dcubed - Looks like the Threads_lock is for stable access 1483 // to _jvmci_old_thread_counters and _jvmci_counters. Does this comment need revisiting? 3451 volatile jint ... Why are you using jint for field types here? (Surprised Coleen hasn't spotted them! ;-) ). 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; 3457 long Threads::_smr_java_thread_list_free_cnt = 0; long? If you really want 64-bit counters here you won't get them on Windows with that declaration. If you really want variable sized counters I suggest uintptr_t; otherwise uint64_t. ---- On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: > Greetings, > > Testing of the last round of changes revealed a hang in one of the new > TLH tests. Robbin has fixed the hang, updated the existing TLH test, and > added another TLH test for good measure. > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ > > Here is the updated delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ > > Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are > no unexpected failures. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Robbin rebased the project last night/this morning to merge with Thread >> Local Handshakes (TLH) and also picked up additional changesets up thru: >> >>> Changeset: fa736014cf28 >>> Author:??? cjplummer >>> Date:????? 2017-11-14 18:08 -0800 >>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>> >>> 8191049: Add alternate version of pns() that is callable from within >>> hotspot source >>> Summary: added pns2() to debug.cpp >>> Reviewed-by: stuefe, gthornbr >> >> This is the first time we've rebased the project to bits that are this >> fresh (< 12 hours old at rebase time). We've done this because we think >> we're done with this project and are in the final review-change-retest >> cycle(s)... In other words, we're not planning on making any more major >> changes for this project... :-) >> >> *** Time for code reviewers to chime in on this thread! *** >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >> >> Here's is the delta webrev from the 2017.11.10 rebase: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >> (and the new baseline also)... >> >> We're expecting this round to be a little noisier than usual because >> we did not rebase on a PIT snapshot so the new baseline has not been >> through Jesper's usual care-and-feeding of integration_blockers, etc. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>> (Yes, we're playing chase-the-repo...) >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>> >>> Unlike the previous rebase, there were no changes required to the >>> open code to get the rebased bits to build so there is no delta >>> webrev. >>> >>> These bits have been run through JPRT and I've submitted the usual >>> Mach5 tier[1-5] test run... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>> >>>> Here are the updated webrevs: >>>> >>>> Here's the mq comment for the change: >>>> >>>> ? Rebase to 2017.10.25 PIT snapshot. >>>> >>>> Here is the full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>> >>>> And here is the delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>> >>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>> look today. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Resolving one of the code review comments (from both Stefan K and >>>>> Coleen) >>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>> standalone patch and corresponding webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>> ??? JavaThreadIteratorWithHandle. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> We have a (eXtra Large) fix for the following bug: >>>>>> >>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>> >>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>> >>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>> and track the work on this project: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>> >>>>>> >>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>> quotes >>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>> solution for that problem yet. >>>>>> >>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>> >>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>> additional testing on Erik and Robbin's machines. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Daniel Daugherty >>>>>> Erik Osterlund >>>>>> Robbin Ehn >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From robbin.ehn at oracle.com Mon Nov 20 07:58:54 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Mon, 20 Nov 2017 08:58:54 +0100 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <65e9bb15-e9e2-8220-15a8-3630df0fd139@oracle.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> <65e9bb15-e9e2-8220-15a8-3630df0fd139@oracle.com> Message-ID: <692c00cc-bd36-f8cb-b64f-4d344ebc42a2@oracle.com> Thank you David for taking the time and doing a thorough review. /Robbin On 2017-11-20 05:28, David Holmes wrote: > Hi Robbin, > > tl;dr: looks good. reviewed. > > So with TLH you introduced SafepointMechanism::initialize() and inserted it into the init sequence here: > > +? SafepointMechanism::initialize(); > + > ?? // Initialize the os module after parsing the args > ?? jint os_init_2_result = os::init_2(); > > so prior to os::init_2(). Most (all?) of the code in SafepointMechanism::initialize() was actually factored out from the platform specific os::init_2() functions. But in doing the factoring you replaced native OS calls like VirtualAlloc with more abstract os::* APIs. So now you have a method that depends on various os::* APIs but which is called before os initialization is fully complete. And it turns out on Windows those functions depended on something only done in os::init_2(), so you've now moved the initialization function to after os::init_2(). > > So as long as there is nothing in os::init_2() that may depend on the initialization now done afterwards via SafepointMechanism::initialize(), this should be a good fix - and handle any other unforeseen init_2() dependencies that may not have surfaced. > > As should not be encountering safepoints or handshakes until well after this point in the init sequence this all seems okay to me. > > Thanks, > David > > On 19/11/2017 10:34 PM, David Holmes wrote: >> Hi Robbin, >> >> I need to study this more closely. Need to check the original code before the TLH code, to evaluate impact. >> >> Initialization order issues are rarely simple to fix as there are so many hidden dependencies. >> >> Will get back to you asap. >> >> David >> >> On 18/11/2017 12:10 AM, Robbin Ehn wrote: >>> Hi all, please review. >>> >>> Windows needs to run os::init_2 before allocation polling page to proper setup numa. I saw no reason not to have it for all platforms after init_2. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >>> >>> Code below. >>> >>> Tested tier 1-5/1-3 >>> >>> Thanks, Robbin >>> >>> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >>> --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 2017 +0100 >>> +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 2017 +0100 >>> @@ -3560,10 +3560,10 @@ >>> ??? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >>> >>> -? SafepointMechanism::initialize(); >>> - >>> ??? // Initialize the os module after parsing the args >>> ??? jint os_init_2_result = os::init_2(); >>> ??? if (os_init_2_result != JNI_OK) return os_init_2_result; >>> >>> +? SafepointMechanism::initialize(); >>> + >>> ??? jint adjust_after_os_result = Arguments::adjust_after_os(); >>> ??? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; From robbin.ehn at oracle.com Mon Nov 20 07:59:49 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Mon, 20 Nov 2017 08:59:49 +0100 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <5A0F4035.9010302@linux.vnet.ibm.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> <5A0F4035.9010302@linux.vnet.ibm.com> Message-ID: <771a3e29-93fb-2d50-3654-00e2a509655b@oracle.com> Great, thanks Gustavo! /Robbin On 2017-11-17 21:01, Gustavo Romero wrote: > Hi Volker, Robbin! > > I could not find any damage to JVM NUMA detection on PPC64 regarding this > change. So thumbs up from my side. > > Regards, > Gustavo > > On 17-11-2017 12:21, Volker Simonis wrote: >> Hi Gustavo, >> >> as our "PowerPC NUMA expert", do you see any problem with this change? >> >> Thank yo uand best regards, >> Volker >> >> >> On Fri, Nov 17, 2017 at 3:10 PM, Robbin Ehn wrote: >>> Hi all, please review. >>> >>> Windows needs to run os::init_2 before allocation polling page to proper >>> setup numa. I saw no reason not to have it for all platforms after init_2. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >>> >>> Code below. >>> >>> Tested tier 1-5/1-3 >>> >>> Thanks, Robbin >>> >>> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >>> --- a/src/hotspot/share/runtime/thread.cpp Fri Nov 17 02:50:51 2017 >>> +0100 >>> +++ b/src/hotspot/share/runtime/thread.cpp Fri Nov 17 13:30:28 2017 >>> +0100 >>> @@ -3560,10 +3560,10 @@ >>> TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >>> >>> - SafepointMechanism::initialize(); >>> - >>> // Initialize the os module after parsing the args >>> jint os_init_2_result = os::init_2(); >>> if (os_init_2_result != JNI_OK) return os_init_2_result; >>> >>> + SafepointMechanism::initialize(); >>> + >>> jint adjust_after_os_result = Arguments::adjust_after_os(); >>> if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; >> > From robbin.ehn at oracle.com Mon Nov 20 08:00:46 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Mon, 20 Nov 2017 09:00:46 +0100 Subject: RFR(integration blocker): 8191373: Multiple NUMA nodes expected In-Reply-To: <73a1f468-33d7-73ce-8766-3c5e798fc733@oracle.com> References: <6c923656-0781-9a3d-f977-2390f7371ec0@oracle.com> <73a1f468-33d7-73ce-8766-3c5e798fc733@oracle.com> Message-ID: Thanks Dan! /Robbin On 2017-11-17 16:39, Daniel D. Daugherty wrote: > On 11/17/17 9:36 AM, Daniel D. Daugherty wrote: >> On 11/17/17 9:10 AM, Robbin Ehn wrote: >>> Hi all, please review. >>> >>> Windows needs to run os::init_2 before allocation polling page to proper setup numa. I saw no reason not to have it for all platforms after init_2. >> >> Do we understand why this failure mode is intermittent? > > Update: The TestOptionsWithRanges.java failure mode is solid and > reproducible. And it does not reproduce with Robbin's fix in place. > > When Robbin was testing the fix a couple of other tests failed > intermittently and Robbin initially thought his fix provoked those > intermittent failures. Those failures have since reproduced in the > jdk/hs nightly so they are unrelated to Robbin's fix. > > Sorry for any confusion. > > Thumbs up on the fix! > > Dan > > >> >> Dan >> >> >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191373 >>> >>> Code below. >>> >>> Tested tier 1-5/1-3 >>> >>> Thanks, Robbin >>> >>> diff -r b4d2929683b6 src/hotspot/share/runtime/thread.cpp >>> --- a/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 02:50:51 2017 +0100 >>> +++ b/src/hotspot/share/runtime/thread.cpp??? Fri Nov 17 13:30:28 2017 +0100 >>> @@ -3560,10 +3560,10 @@ >>> ?? TraceTime timer("Create VM", TRACETIME_LOG(Info, startuptime)); >>> >>> -? SafepointMechanism::initialize(); >>> - >>> ?? // Initialize the os module after parsing the args >>> ?? jint os_init_2_result = os::init_2(); >>> ?? if (os_init_2_result != JNI_OK) return os_init_2_result; >>> >>> +? SafepointMechanism::initialize(); >>> + >>> ?? jint adjust_after_os_result = Arguments::adjust_after_os(); >>> ?? if (adjust_after_os_result != JNI_OK) return adjust_after_os_result; >> >> > From thomas.stuefe at gmail.com Mon Nov 20 09:04:40 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 20 Nov 2017 10:04:40 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> Message-ID: Hi David, looks fine. Thanks, Thomas On Mon, Nov 20, 2017 at 4:44 AM, David Holmes wrote: > Hi Dan, > > Thanks for the review. I've addressed all of yours and Thomas's comments. > > http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ > > The main change is: > > + static bool is_primordial_thread(void) > + #if defined(_WINDOWS) || defined(BSD) > + // No way to identify the primordial thread. > + { return false; } > + #else > + ; > + #endif > > > Also to clarify this is logically a "product" flag for the systems that > need it, not a diagnostic one. But as we don't want this flag to be in > common use and it is of such narrow applicability, we have made it > experimental. This was discussed previously in the CSR review thread for > this: > > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2 > 017-October/024859.html > > Once Tomas K. has been able to test this, and you and Thomas sign-off on > the updates, I'll get this pushed. But as I'm away from Thursday it may not > be until next week. > > Thanks, > David > ----- > > > > On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: > >> On 11/14/17 5:39 AM, David Holmes wrote: >> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>> >>> webrev: http://cr.openjdk.java.net/~dholmes/8189170/webrev/ >>> >> >> src/hotspot/os/aix/os_aix.cpp >> No comments. >> >> src/hotspot/os/aix/os_aix.hpp >> No comments. >> >> src/hotspot/os/bsd/os_bsd.cpp >> L880: // thread in any special way - so just report 'false' >> Nit - needs a period after 'false' to make it a sentence. >> >> L882: return false; >> Nit - indent should be 2 spaces. >> >> src/hotspot/os/bsd/os_bsd.hpp >> No comments. >> >> src/hotspot/os/linux/os_linux.cpp >> L947: if (stack_size >= (size_t)(3 * page_size())) >> Nit - needs '{' and '}' on the if-statement body. >> >> L3070: // always place it right after end of the mapped region >> Nit - needs a period after 'region' to make it a sentence. >> (Not your problem, but you did update the sentence...) >> >> src/hotspot/os/linux/os_linux.hpp >> No comments. >> >> src/hotspot/os/solaris/os_solaris.cpp >> No comments other than I learned something new about thr_main() :-) >> Thanks for cleaning up that mess. >> >> src/hotspot/os/windows/os_windows.cpp >> L452: // thread in any special way - so just report 'false' >> Nit - needs a period after 'false' to make it a sentence. >> >> L454: return false; >> Nit - indent should be 2 spaces. >> >> src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp >> No comments. >> >> src/hotspot/share/runtime/globals.hpp >> L1181: experimental(bool, DisablePrimordialThreadGuardPages, >> false, >> Experimental or diagnostic? >> >> src/hotspot/share/runtime/os.hpp >> L450: // that loads/creates the JVM via JNI_CreateJavaVM >> Nit - needs a period after 'JNI_CreateJavaVM'. >> >> L453: // launcher nevers uses the primordial thread as the main >> thread, but >> Typo - 'nevers' -> 'never' >> >> L455: // have to special-case handling of the primordial thread if >> it attaches >> Typo - 'have to' -> 'have' >> >> src/hotspot/share/runtime/thread.cpp >> I like the new log message. >> >> src/hotspot/share/runtime/thread.inline.hpp >> L159: if (os::uses_stack_guard_pages() && >> L160: !(DisablePrimordialThreadGuardPages && >> os::is_primordial_thread())) { >> I'm not sure why, but I had to read these two lines several >> times before I convinced myself the logic is right. >> >> I'm good with the code changes. Your call on whether to fix the >> bits or not. >> >> Regarding this part of Thomas' review: >> >> Or, something like this: >>>> static bool is_primordial_thread(void) WINDOWS_ONLY({return false;}) >>>> BSD_ONLY({return false;}) >>>> >>> >>> I can do that. I'll see if anyone else has any comments on this. >>> >> >> I'm assuming that Thomas means for the above to be at the >> platform independent layer (runtime/os.hpp?) so it is more >> obvious to the reader that the function doesn't do anything >> on those two platforms... I know we dislike such things at >> the platform independent layer, but it does feel appropriate >> to make this limitation more obvious. >> >> I would be okay if you made the change that Thomas is suggesting. >> >> Thumbs up! >> >> Dan >> >> >> >>> >>> There are three parts to this which made it somewhat more complicated >>> than I had envisaged: >>> >>> 1. Add the flag and disable the guards. >>> >>> This is relatively straight-forward: >>> >>> void JavaThread::create_stack_guard_pages() { >>> if (!os::uses_stack_guard_pages() || >>> _stack_guard_state != stack_guard_unused || >>> (DisablePrimordialThreadGuardPages && >>> os::is_primordial_thread())) { >>> log_info(os, thread)("Stack guard page creation for thread " >>> UINTX_FORMAT " disabled", >>> os::current_thread_id()); >>> return; >>> >>> with a tweak in JavaThread::stack_guards_enabled as well. >>> >>> 2. Promote os::XXX::is_initial_thread to os::is_primordial_thread() >>> >>> We have three platforms that already implement this functionality: >>> Linux, Solaris and AIX. For BSD/OSX and Windows we don't attempt to do >>> anything different for the primordial thread, and we don't have a way to >>> detect the primordial thread - so is_primordial_thread() just returns false. >>> >>> I also tidied up some comments regarding terminology. >>> >>> 3. Thomas noticed that we unconditionally subtract 2*page_size() from >>> the rlimit stack size, without checking it was bigger than 2*page_size() to >>> start with. I was unsure how to tackle this. It's no good leaving >>> stack_size at zero so I opted to ensure we would have at least one page >>> left. Of course in such cases we would hit the bug in libc (if it still >>> exists, which seems unlikely but time consuming to establish). >>> >>> Testing: >>> - Tier 1 (JPRT) testing with a modified launcher that uses the >>> primordial thread >>> - with guard pages enabled >>> - with guard pages disabled >>> - Tier 1 (JPRT) normal JDK build (which should be unaffected by this >>> change) >>> >>> The testing was primarily to ensure that disabling the stack guards on >>> the primordial thread wasn't totally broken. A number of tests fail: >>> - setting -Xss32m fails for the primordial thread when guards are >>> enabled and the rlimit is 8m >>> - tests that would hit StackOverflowError crash the VM when guards are >>> disabled (as you would expect). >>> - runtime/execstack/Testexecstack.java crashes when the guards are >>> disabled >>> >>> Thanks, >>> David >>> >>> >> From robin.westberg at oracle.com Mon Nov 20 09:48:15 2017 From: robin.westberg at oracle.com (Robin Westberg) Date: Mon, 20 Nov 2017 10:48:15 +0100 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: Hi Dan, Overall I must say this looks very nice, can?t wait to use it! (Also a disclaimer: not a reviewer, and have not looked at the gc or jmvti specific changes nor the tests (yet)). Didn?t spot any real issues, just a few small efficiency related things: src/hotspot/share/runtime/biasedLocking.cpp 217 for (JavaThread* cur_thread = Threads::first(); cur_thread != NULL; cur_thread = cur_thread->next()) { 218 if (cur_thread == biased_thread) { 219 thread_is_alive = true; This loop could be replaced with: ThreadsListHandle tlh; thread_is_alive = tlh.includes(biased_thread); src/hotspot/share/runtime/memprofiler.cpp 55-56 grabs the Threads_lock, shouldn?t be needed anymore. src/hotspot/share/runtime/thread.inline.hpp 198 TerminatedTypes l_terminated = (TerminatedTypes) 199 OrderAccess::load_acquire((volatile jint *) &_terminated); 200 return check_is_terminated(_terminated); The variable used in the return statement was intended to be l_terminated I guess? The following are more minor suggestions / comments: src/hotspot/share/runtime/thread.cpp 3432 // operations over all threads. It is protected by its own Mutex 3433 // lock, which is also used in other contexts to protect thread Should this comment perhaps be revised to mention SMR? 4745 hash_table_size--; 4746 hash_table_size |= hash_table_size >> 1; ... This calculation is repeated around line 4922 as well, perhaps put it in a function? 4828 // If someone set a handshake on us just as we entered exit path, we simple cancel it. Perhaps something like ?.. entered the exit path, we simply cancel it.? src/hotspot/share/runtime/thread.hpp 1179 bool on_thread_list() { return _on_thread_list; } 1180 void set_on_thread_list() { _on_thread_list = true; } 1181 1182 // thread has called JavaThread::exit() or is terminated 1183 bool is_exiting() const; 1184 // thread is terminated (no longer on the threads list); we compare 1185 // against the two non-terminated values so that a freed JavaThread 1186 // will also be considered terminated. 1187 bool check_is_terminated(TerminatedTypes l_terminated) const { 1188 return l_terminated != _not_terminated && l_terminated != _thread_exiting; 1189 } 1190 bool is_terminated(); Use of const is inconsistent here, it?s added to 1183, should it perhaps be added to 1179 and 1190 as well then? src/hotspot/share/runtime/vm_operations.hpp 406 DeadlockCycle* result() { return _deadlocks; }; 407 VMOp_Type type() const { return VMOp_FindDeadlocks; } Whitespace only change that seems spurious. Best regards, Robin > On 19 Nov 2017, at 02:49, Daniel D. Daugherty wrote: > > Greetings, > > Testing of the last round of changes revealed a hang in one of the new > TLH tests. Robbin has fixed the hang, updated the existing TLH test, and > added another TLH test for good measure. > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ > > Here is the updated delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ > > Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are > no unexpected failures. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Robbin rebased the project last night/this morning to merge with Thread >> Local Handshakes (TLH) and also picked up additional changesets up thru: >> >>> Changeset: fa736014cf28 >>> Author: cjplummer >>> Date: 2017-11-14 18:08 -0800 >>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>> >>> 8191049: Add alternate version of pns() that is callable from within hotspot source >>> Summary: added pns2() to debug.cpp >>> Reviewed-by: stuefe, gthornbr >> >> This is the first time we've rebased the project to bits that are this >> fresh (< 12 hours old at rebase time). We've done this because we think >> we're done with this project and are in the final review-change-retest >> cycle(s)... In other words, we're not planning on making any more major >> changes for this project... :-) >> >> *** Time for code reviewers to chime in on this thread! *** >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >> >> Here's is the delta webrev from the 2017.11.10 rebase: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >> (and the new baseline also)... >> >> We're expecting this round to be a little noisier than usual because >> we did not rebase on a PIT snapshot so the new baseline has not been >> through Jesper's usual care-and-feeding of integration_blockers, etc. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>> (Yes, we're playing chase-the-repo...) >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>> >>> Unlike the previous rebase, there were no changes required to the >>> open code to get the rebased bits to build so there is no delta >>> webrev. >>> >>> These bits have been run through JPRT and I've submitted the usual >>> Mach5 tier[1-5] test run... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>> >>>> Here are the updated webrevs: >>>> >>>> Here's the mq comment for the change: >>>> >>>> Rebase to 2017.10.25 PIT snapshot. >>>> >>>> Here is the full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>> >>>> And here is the delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>> >>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>> look today. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Resolving one of the code review comments (from both Stefan K and Coleen) >>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>> standalone patch and corresponding webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>> JavaThreadIteratorWithHandle. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> We have a (eXtra Large) fix for the following bug: >>>>>> >>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>> >>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>> >>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>> and track the work on this project: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>> >>>>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>> solution for that problem yet. >>>>>> >>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>> >>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>> additional testing on Erik and Robbin's machines. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Daniel Daugherty >>>>>> Erik Osterlund >>>>>> Robbin Ehn >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From rkennke at redhat.com Mon Nov 20 10:15:40 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 20 Nov 2017 11:15:40 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> Message-ID: <4ac6a8ef-a42c-f44b-7135-a9fc846b1848@redhat.com> Hi David, I just built linux-x86_64-normal-minimal-slowdebug and it does not fail. That's what I did when I tested the patch(es). It's not reasonable to expect devs to try and build all possible combinations of jvm variants, debug-levels, or even arches etc. I'll file and fix a follow-up bug for it. Roman > On 17/11/2017 2:33 AM, Bob Vandette wrote: >> Yes, that?s the fix.? I built it correctly with that added include. > > How? I can't build the MinimalVM in JPRT because the > UNSUPPORTED_OPTION macro is only locally defined in arguments.cpp, but > is now used in gcArguments.cpp: > > /opt/jprt/T/P1/003543.daholme/s/open/src/hotspot/share/gc/shared/gcArguments.cpp:77:29: > error: 'UNSUPPORTED_OPTION' was not declared in this scope > ?? UNSUPPORTED_OPTION(UseG1GC); > ???????????????????????????? ^ > make[3]: *** > [/opt/jprt/T/P1/003543.daholme/s/build/linux-x86-debug/hotspot/variant-minimal/libjvm/objs/gcArguments.o] > Error 1 > > --- > > Roman/Erik: If changing code that has #include guards you must ensure > all the paths are tested! > > Thanks, > David > >> There are still a few other issues with building and using the >> minimal VM but those >> existed before your change. >> >> 1. You cannot build the minimal VM without also building the server >> VM.? There >> is a Makefile dependency on the server VM. >> 2. The jvm.cfg produced in the jvm and jre images does not add the >> minimalvm >> as one of the possible VM to select.? As a work-around jlink produces >> a workable jvm.cfg. >> >> Thanks, >> Bob. >> >> >>> On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: >>> >>> Hi Bob, >>> thanks for letting me know. >>> >>> I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 >>> and posted for review: >>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html >>> >>> Thanks, Roman >>> >>>> Roman, >>>> >>>> It looks like this change may have caused the build of the minimal >>>> VM to >>>> fail.? The minimal VM is no longer a configuration that we >>>> regularly build and test >>>> but it is still a build option that can be selected via >>>> "--with-jvm-variants=minimal1" >>>> >>>> >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: >>>> In static member function 'static jint GCArguments::initialize()': >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseParallelGC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseG1GC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), >>>> "UseConcMarkSweepGC not supported in this VM.\n"); >>>> ????????????????? ^ >>>> gmake[3]: *** >>>> [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] >>>> Error 1 >>>> >>>> Bob. >>>> >>>> >>>>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>>>> >>>>> Hi Per & Erik, >>>>> >>>>> thanks for the reviews! >>>>> >>>>> Now I need a sponsor. >>>>> >>>>> Final webrev (same as before, including Reviewed-by line): >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>>>> >>>>> >>>>> Thanks, Roman >>>>> >>>>> >>>>>> Looks good Roman! >>>>>> >>>>>> /Per >>>>>> >>>>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>>>> [...] >>>>>>>>>> So let me just put your changes up for review (again), if you >>>>>>>>>> don't >>>>>>>>>> mind: >>>>>>>>>> >>>>>>>>>> Full webrev: >>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>>>> >>>>>>>> I like this. Just two naming suggestions: >>>>>>>> >>>>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>>>> >>>>>>>> ?? 39?? static jint create_instance(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* instance(); >>>>>>>> >>>>>>>> Could we call these: >>>>>>>> >>>>>>>> ?? 39?? static jint initialize(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* arguments(); >>>>>>>> >>>>>>>> Reasoning: initialize() maps better to its companion >>>>>>>> is_initialized(). >>>>>>>> GCArguments::arguments() maps better to the existing patterns >>>>>>>> we have >>>>>>>> with CollectedHeap::heap(). >>>>>>> Ok, that sounds good to me. Actually, it's almost full-circle >>>>>>> back to my >>>>>>> original proposal ;-) >>>>>>> >>>>>>> Differential: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>>>> >>>>>>> Full: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>>>> >>>>>>> >>>>>>> Ok to push now? >>>>>>> >>>>>>> Roman >>>>> >>> >> From rkennke at redhat.com Mon Nov 20 10:17:37 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 20 Nov 2017 11:17:37 +0100 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> Message-ID: <7ad0f09d-c6ea-c4aa-eb1f-2e31e2a8a951@redhat.com> ... and minimal-fastdebug fails with something else: /home/rkennke/src/openjdk/jdk-hs/src/hotspot/cpu/x86/c1_Runtime1_x86.cpp:136:10: error: 'call_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized] ?? return call_offset; I guess I can fix this blindly by including arguments.hpp in gcArguments.cpp and hope for the best? Roman > On 17/11/2017 2:33 AM, Bob Vandette wrote: >> Yes, that?s the fix.? I built it correctly with that added include. > > How? I can't build the MinimalVM in JPRT because the > UNSUPPORTED_OPTION macro is only locally defined in arguments.cpp, but > is now used in gcArguments.cpp: > > /opt/jprt/T/P1/003543.daholme/s/open/src/hotspot/share/gc/shared/gcArguments.cpp:77:29: > error: 'UNSUPPORTED_OPTION' was not declared in this scope > ?? UNSUPPORTED_OPTION(UseG1GC); > ???????????????????????????? ^ > make[3]: *** > [/opt/jprt/T/P1/003543.daholme/s/build/linux-x86-debug/hotspot/variant-minimal/libjvm/objs/gcArguments.o] > Error 1 > > --- > > Roman/Erik: If changing code that has #include guards you must ensure > all the paths are tested! > > Thanks, > David > >> There are still a few other issues with building and using the >> minimal VM but those >> existed before your change. >> >> 1. You cannot build the minimal VM without also building the server >> VM.? There >> is a Makefile dependency on the server VM. >> 2. The jvm.cfg produced in the jvm and jre images does not add the >> minimalvm >> as one of the possible VM to select.? As a work-around jlink produces >> a workable jvm.cfg. >> >> Thanks, >> Bob. >> >> >>> On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: >>> >>> Hi Bob, >>> thanks for letting me know. >>> >>> I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 >>> and posted for review: >>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html >>> >>> Thanks, Roman >>> >>>> Roman, >>>> >>>> It looks like this change may have caused the build of the minimal >>>> VM to >>>> fail.? The minimal VM is no longer a configuration that we >>>> regularly build and test >>>> but it is still a build option that can be selected via >>>> "--with-jvm-variants=minimal1" >>>> >>>> >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: >>>> In static member function 'static jint GCArguments::initialize()': >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseParallelGC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), "UseG1GC not >>>> supported in this VM.\n"); >>>> ????????????????? ^ >>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: >>>> error: 'defaultStream' has not been declared >>>> ????? jio_fprintf(defaultStream::error_stream(), >>>> "UseConcMarkSweepGC not supported in this VM.\n"); >>>> ????????????????? ^ >>>> gmake[3]: *** >>>> [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] >>>> Error 1 >>>> >>>> Bob. >>>> >>>> >>>>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>>>> >>>>> Hi Per & Erik, >>>>> >>>>> thanks for the reviews! >>>>> >>>>> Now I need a sponsor. >>>>> >>>>> Final webrev (same as before, including Reviewed-by line): >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>>>> >>>>> >>>>> Thanks, Roman >>>>> >>>>> >>>>>> Looks good Roman! >>>>>> >>>>>> /Per >>>>>> >>>>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>>>> [...] >>>>>>>>>> So let me just put your changes up for review (again), if you >>>>>>>>>> don't >>>>>>>>>> mind: >>>>>>>>>> >>>>>>>>>> Full webrev: >>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>>>> >>>>>>>> I like this. Just two naming suggestions: >>>>>>>> >>>>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>>>> >>>>>>>> ?? 39?? static jint create_instance(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* instance(); >>>>>>>> >>>>>>>> Could we call these: >>>>>>>> >>>>>>>> ?? 39?? static jint initialize(); >>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>> ?? 41?? static GCArguments* arguments(); >>>>>>>> >>>>>>>> Reasoning: initialize() maps better to its companion >>>>>>>> is_initialized(). >>>>>>>> GCArguments::arguments() maps better to the existing patterns >>>>>>>> we have >>>>>>>> with CollectedHeap::heap(). >>>>>>> Ok, that sounds good to me. Actually, it's almost full-circle >>>>>>> back to my >>>>>>> original proposal ;-) >>>>>>> >>>>>>> Differential: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>>>> >>>>>>> Full: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>>>> >>>>>>> >>>>>>> Ok to push now? >>>>>>> >>>>>>> Roman >>>>> >>> >> From david.holmes at oracle.com Mon Nov 20 11:53:54 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 21:53:54 +1000 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <4ac6a8ef-a42c-f44b-7135-a9fc846b1848@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> <3c75a6c5-ffe5-948f-b2c3-b4950a4b1a9f@redhat.com> <3f188247-aea1-55d7-cbd4-8ddcd33aa4a8@oracle.com> <411d8f0f-82e7-88e3-a1a4-12515b59452a@redhat.com> <38c8c5c1-6231-76c2-0c33-7920101814cd@oracle.com> <4ac6a8ef-a42c-f44b-7135-a9fc846b1848@redhat.com> Message-ID: <04b02b4a-b208-846c-8dc3-a8081c474d86@oracle.com> On 20/11/2017 8:15 PM, Roman Kennke wrote: > Hi David, > > I just built linux-x86_64-normal-minimal-slowdebug and it does not fail. > That's what I did when I tested the patch(es). It's not reasonable to > expect devs to try and build all possible combinations of jvm variants, > debug-levels, or even arches etc. I couldn't understand how any minimal VM build could succeed**. I can only guess that it is due to precompiled headers pulling in arguments.hpp in one case but not another - though I can't say why fastdebug and slowdebug should behave differently ?? In JPRT we build product and fastdebug. ** I mistakenly thought the macro was declared in arguments.cpp and hence could never be available elsewhere. > I'll file and fix a follow-up bug for it. Thanks, David ----- > > Roman > >> On 17/11/2017 2:33 AM, Bob Vandette wrote: >>> Yes, that?s the fix.? I built it correctly with that added include. >> >> How? I can't build the MinimalVM in JPRT because the >> UNSUPPORTED_OPTION macro is only locally defined in arguments.cpp, but >> is now used in gcArguments.cpp: >> >> /opt/jprt/T/P1/003543.daholme/s/open/src/hotspot/share/gc/shared/gcArguments.cpp:77:29: >> error: 'UNSUPPORTED_OPTION' was not declared in this scope >> ?? UNSUPPORTED_OPTION(UseG1GC); >> ???????????????????????????? ^ >> make[3]: *** >> [/opt/jprt/T/P1/003543.daholme/s/build/linux-x86-debug/hotspot/variant-minimal/libjvm/objs/gcArguments.o] >> Error 1 >> >> --- >> >> Roman/Erik: If changing code that has #include guards you must ensure >> all the paths are tested! >> >> Thanks, >> David >> >>> There are still a few other issues with building and using the >>> minimal VM but those >>> existed before your change. >>> >>> 1. You cannot build the minimal VM without also building the server >>> VM.? There >>> is a Makefile dependency on the server VM. >>> 2. The jvm.cfg produced in the jvm and jre images does not add the >>> minimalvm >>> as one of the possible VM to select.? As a work-around jlink produces >>> a workable jvm.cfg. >>> >>> Thanks, >>> Bob. >>> >>> >>>> On Nov 16, 2017, at 11:17 AM, Roman Kennke wrote: >>>> >>>> Hi Bob, >>>> thanks for letting me know. >>>> >>>> I filed: https://bugs.openjdk.java.net/browse/JDK-8191424 >>>> and posted for review: >>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-November/020784.html >>>> >>>> >>>> Thanks, Roman >>>> >>>>> Roman, >>>>> >>>>> It looks like this change may have caused the build of the minimal >>>>> VM to >>>>> fail.? The minimal VM is no longer a configuration that we >>>>> regularly build and test >>>>> but it is still a build option that can be selected via >>>>> "--with-jvm-variants=minimal1" >>>>> >>>>> >>>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp: >>>>> In static member function 'static jint GCArguments::initialize()': >>>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:113:17: >>>>> error: 'defaultStream' has not been declared >>>>> ????? jio_fprintf(defaultStream::error_stream(), "UseParallelGC not >>>>> supported in this VM.\n"); >>>>> ????????????????? ^ >>>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:116:17: >>>>> error: 'defaultStream' has not been declared >>>>> ????? jio_fprintf(defaultStream::error_stream(), "UseG1GC not >>>>> supported in this VM.\n"); >>>>> ????????????????? ^ >>>>> /scratch/users/bobv/jdk10hs/open/src/hotspot/share/gc/shared/gcArguments.cpp:119:17: >>>>> error: 'defaultStream' has not been declared >>>>> ????? jio_fprintf(defaultStream::error_stream(), >>>>> "UseConcMarkSweepGC not supported in this VM.\n"); >>>>> ????????????????? ^ >>>>> gmake[3]: *** >>>>> [/scratch/users/bobv/jdk10hs/build/b00/linux-x64-minimal1/hotspot/variant-minimal/libjvm/objs/gcArguments.o] >>>>> Error 1 >>>>> >>>>> Bob. >>>>> >>>>> >>>>>> On Nov 7, 2017, at 6:01 AM, Roman Kennke wrote: >>>>>> >>>>>> Hi Per & Erik, >>>>>> >>>>>> thanks for the reviews! >>>>>> >>>>>> Now I need a sponsor. >>>>>> >>>>>> Final webrev (same as before, including Reviewed-by line): >>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.05/ >>>>>> >>>>>> >>>>>> Thanks, Roman >>>>>> >>>>>> >>>>>>> Looks good Roman! >>>>>>> >>>>>>> /Per >>>>>>> >>>>>>> On 2017-11-06 12:13, Roman Kennke wrote: >>>>>>>> Am 26.10.2017 um 11:43 schrieb Per Liden: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On 2017-10-25 18:11, Erik Osterlund wrote: >>>>>>>>> [...] >>>>>>>>>>> So let me just put your changes up for review (again), if you >>>>>>>>>>> don't >>>>>>>>>>> mind: >>>>>>>>>>> >>>>>>>>>>> Full webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >>>>>>>>>>> >>>>>>>>> I like this. Just two naming suggestions: >>>>>>>>> >>>>>>>>> src/hotspot/share/gc/shared/gcArguments.hpp: >>>>>>>>> >>>>>>>>> ?? 39?? static jint create_instance(); >>>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>>> ?? 41?? static GCArguments* instance(); >>>>>>>>> >>>>>>>>> Could we call these: >>>>>>>>> >>>>>>>>> ?? 39?? static jint initialize(); >>>>>>>>> ?? 40?? static bool is_initialized(); >>>>>>>>> ?? 41?? static GCArguments* arguments(); >>>>>>>>> >>>>>>>>> Reasoning: initialize() maps better to its companion >>>>>>>>> is_initialized(). >>>>>>>>> GCArguments::arguments() maps better to the existing patterns >>>>>>>>> we have >>>>>>>>> with CollectedHeap::heap(). >>>>>>>> Ok, that sounds good to me. Actually, it's almost full-circle >>>>>>>> back to my >>>>>>>> original proposal ;-) >>>>>>>> >>>>>>>> Differential: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04.diff/ >>>>>>>> >>>>>>>> Full: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.04/ >>>>>>>> >>>>>>>> >>>>>>>> Ok to push now? >>>>>>>> >>>>>>>> Roman >>>>>> >>>> >>> > From daniel.daugherty at oracle.com Mon Nov 20 13:05:27 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 20 Nov 2017 08:05:27 -0500 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> Message-ID: <1e422a91-78e2-2e5c-2543-eaa7d49c60c7@oracle.com> I'm also good with this change. Dan On 11/20/17 4:04 AM, Thomas St?fe wrote: > Hi David, > > looks fine. > > Thanks, Thomas > > On Mon, Nov 20, 2017 at 4:44 AM, David Holmes > wrote: > > Hi Dan, > > Thanks for the review. I've addressed all of yours and Thomas's > comments. > > http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ > > > The main change is: > > +? ?static bool is_primordial_thread(void) > + #if defined(_WINDOWS) || defined(BSD) > +? ? ?// No way to identify the primordial thread. > +? ? ?{ return false; } > + #else > +? ?; > + #endif > > > Also to clarify this is logically a "product" flag for the systems > that need it, not a diagnostic one. But as we don't want this flag > to be in common use and it is of such narrow applicability, we > have made it experimental. This was discussed previously in the > CSR review thread for this: > > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html > > > Once Tomas K. has been able to test this, and you and Thomas > sign-off on the updates, I'll get this pushed. But as I'm away > from Thursday it may not be until next week. > > Thanks, > David > ----- > > > > On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: > > On 11/14/17 5:39 AM, David Holmes wrote: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > > webrev: > http://cr.openjdk.java.net/~dholmes/8189170/webrev/ > > > > src/hotspot/os/aix/os_aix.cpp > ???? No comments. > > src/hotspot/os/aix/os_aix.hpp > ???? No comments. > > src/hotspot/os/bsd/os_bsd.cpp > ???? L880: // thread in any special way - so just report 'false' > ??? ? ?? Nit - needs a period after 'false' to make it a sentence. > > ???? L882: ??? return false; > ???????? Nit - indent should be 2 spaces. > > src/hotspot/os/bsd/os_bsd.hpp > ???? No comments. > > src/hotspot/os/linux/os_linux.cpp > ???? L947: ? if (stack_size >= (size_t)(3 * page_size())) > ???????? Nit - needs '{' and '}' on the if-statement body. > > ???? L3070: // always place it right after end of the mapped > region > ???????? Nit - needs a period after 'region' to make it a > sentence. > ???????? (Not your problem, but you did update the sentence...) > > src/hotspot/os/linux/os_linux.hpp > ???? No comments. > > src/hotspot/os/solaris/os_solaris.cpp > ???? No comments other than I learned something new about > thr_main() :-) > ???? Thanks for cleaning up that mess. > > src/hotspot/os/windows/os_windows.cpp > ???? L452: // thread in any special way - so just report 'false' > ???????? Nit - needs a period after 'false' to make it a sentence. > > ???? L454: ??? return false; > ???????? Nit - indent should be 2 spaces. > > src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp > ???? No comments. > > src/hotspot/share/runtime/globals.hpp > ???? L1181: ? experimental(bool, > DisablePrimordialThreadGuardPages, false, > ???????? Experimental or diagnostic? > > src/hotspot/share/runtime/os.hpp > ???? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM > ???????? Nit - needs a period after 'JNI_CreateJavaVM'. > > ???? L453: ? // launcher nevers uses the primordial thread as > the main thread, but > ???????? Typo - 'nevers' -> 'never' > > ???? L455: ? // have to special-case handling of the > primordial thread if it attaches > ???????? Typo - 'have to' -> 'have' > > src/hotspot/share/runtime/thread.cpp > ???? I like the new log message. > > src/hotspot/share/runtime/thread.inline.hpp > ???? L159: ? if (os::uses_stack_guard_pages() && > ???? L160: ????? !(DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ???????? I'm not sure why, but I had to read these two lines > several > ???????? times before I convinced myself the logic is right. > > I'm good with the code changes. Your call on whether to fix the > bits or not. > > Regarding this part of Thomas' review: > > Or, something like this: > static bool is_primordial_thread(void) > WINDOWS_ONLY({return false;}) BSD_ONLY({return false;}) > > > I can do that. I'll see if anyone else has any comments on > this. > > > I'm assuming that Thomas means for the above to be at the > platform independent layer (runtime/os.hpp?) so it is more > obvious to the reader that the function doesn't do anything > on those two platforms... I know we dislike such things at > the platform independent layer, but it does feel appropriate > to make this limitation more obvious. > > I would be okay if you made the change that Thomas is suggesting. > > Thumbs up! > > Dan > > > > > There are three parts to this which made it somewhat more > complicated than I had envisaged: > > 1. Add the flag and disable the guards. > > This is relatively straight-forward: > > ?void JavaThread::create_stack_guard_pages() { > ?? if (!os::uses_stack_guard_pages() || > ?????? _stack_guard_state != stack_guard_unused || > ?????? (DisablePrimordialThreadGuardPages && > os::is_primordial_thread())) { > ?????? log_info(os, thread)("Stack guard page creation for > thread " > ??????????????????????????? UINTX_FORMAT " disabled", > os::current_thread_id()); > ???? return; > > with a tweak in JavaThread::stack_guards_enabled as well. > > 2. Promote os::XXX::is_initial_thread to > os::is_primordial_thread() > > We have three platforms that already implement this > functionality: Linux, Solaris and AIX. For BSD/OSX and > Windows we don't attempt to do anything different for the > primordial thread, and we don't have a way to detect the > primordial thread - so is_primordial_thread() just returns > false. > > I also tidied up some comments regarding terminology. > > 3. Thomas noticed that we unconditionally subtract > 2*page_size() from the rlimit stack size, without checking > it was bigger than 2*page_size() to start with. I was > unsure how to tackle this. It's no good leaving stack_size > at zero so I opted to ensure we would have at least one > page left. Of course in such cases we would hit the bug in > libc (if it still exists, which seems unlikely but time > consuming to establish). > > Testing: > ? - Tier 1 (JPRT) testing with a modified launcher that > uses the primordial thread > ??? - with guard pages enabled > ??? - with guard pages disabled > ? - Tier 1 (JPRT) normal JDK build (which should be > unaffected by this change) > > The testing was primarily to ensure that disabling the > stack guards on the primordial thread wasn't totally > broken. A number of tests fail: > - setting -Xss32m fails for the primordial thread when > guards are enabled and the rlimit is 8m > - tests that would hit StackOverflowError crash the VM > when guards are disabled (as you would expect). > - runtime/execstack/Testexecstack.java crashes when the > guards are disabled > > Thanks, > David > > > From david.holmes at oracle.com Mon Nov 20 13:06:33 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 23:06:33 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <1e422a91-78e2-2e5c-2543-eaa7d49c60c7@oracle.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> <1e422a91-78e2-2e5c-2543-eaa7d49c60c7@oracle.com> Message-ID: Thank you both. David On 20/11/2017 11:05 PM, Daniel D. Daugherty wrote: > I'm also good with this change. > > Dan > > > On 11/20/17 4:04 AM, Thomas St?fe wrote: >> Hi David, >> >> looks fine. >> >> Thanks, Thomas >> >> On Mon, Nov 20, 2017 at 4:44 AM, David Holmes > > wrote: >> >> Hi Dan, >> >> Thanks for the review. I've addressed all of yours and Thomas's >> comments. >> >> http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ >> >> >> The main change is: >> >> +? ?static bool is_primordial_thread(void) >> + #if defined(_WINDOWS) || defined(BSD) >> +? ? ?// No way to identify the primordial thread. >> +? ? ?{ return false; } >> + #else >> +? ?; >> + #endif >> >> >> Also to clarify this is logically a "product" flag for the systems >> that need it, not a diagnostic one. But as we don't want this flag >> to be in common use and it is of such narrow applicability, we >> have made it experimental. This was discussed previously in the >> CSR review thread for this: >> >> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html >> >> >> Once Tomas K. has been able to test this, and you and Thomas >> sign-off on the updates, I'll get this pushed. But as I'm away >> from Thursday it may not be until next week. >> >> Thanks, >> David >> ----- >> >> >> >> On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: >> >> On 11/14/17 5:39 AM, David Holmes wrote: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> >> webrev: >> http://cr.openjdk.java.net/~dholmes/8189170/webrev/ >> >> >> >> src/hotspot/os/aix/os_aix.cpp >> ???? No comments. >> >> src/hotspot/os/aix/os_aix.hpp >> ???? No comments. >> >> src/hotspot/os/bsd/os_bsd.cpp >> ???? L880: // thread in any special way - so just report 'false' >> ??? ? ?? Nit - needs a period after 'false' to make it a sentence. >> >> ???? L882: ??? return false; >> ???????? Nit - indent should be 2 spaces. >> >> src/hotspot/os/bsd/os_bsd.hpp >> ???? No comments. >> >> src/hotspot/os/linux/os_linux.cpp >> ???? L947: ? if (stack_size >= (size_t)(3 * page_size())) >> ???????? Nit - needs '{' and '}' on the if-statement body. >> >> ???? L3070: // always place it right after end of the mapped >> region >> ???????? Nit - needs a period after 'region' to make it a >> sentence. >> ???????? (Not your problem, but you did update the sentence...) >> >> src/hotspot/os/linux/os_linux.hpp >> ???? No comments. >> >> src/hotspot/os/solaris/os_solaris.cpp >> ???? No comments other than I learned something new about >> thr_main() :-) >> ???? Thanks for cleaning up that mess. >> >> src/hotspot/os/windows/os_windows.cpp >> ???? L452: // thread in any special way - so just report 'false' >> ???????? Nit - needs a period after 'false' to make it a sentence. >> >> ???? L454: ??? return false; >> ???????? Nit - indent should be 2 spaces. >> >> src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp >> ???? No comments. >> >> src/hotspot/share/runtime/globals.hpp >> ???? L1181: ? experimental(bool, >> DisablePrimordialThreadGuardPages, false, >> ???????? Experimental or diagnostic? >> >> src/hotspot/share/runtime/os.hpp >> ???? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM >> ???????? Nit - needs a period after 'JNI_CreateJavaVM'. >> >> ???? L453: ? // launcher nevers uses the primordial thread as >> the main thread, but >> ???????? Typo - 'nevers' -> 'never' >> >> ???? L455: ? // have to special-case handling of the >> primordial thread if it attaches >> ???????? Typo - 'have to' -> 'have' >> >> src/hotspot/share/runtime/thread.cpp >> ???? I like the new log message. >> >> src/hotspot/share/runtime/thread.inline.hpp >> ???? L159: ? if (os::uses_stack_guard_pages() && >> ???? L160: ????? !(DisablePrimordialThreadGuardPages && >> os::is_primordial_thread())) { >> ???????? I'm not sure why, but I had to read these two lines >> several >> ???????? times before I convinced myself the logic is right. >> >> I'm good with the code changes. Your call on whether to fix the >> bits or not. >> >> Regarding this part of Thomas' review: >> >> Or, something like this: >> static bool is_primordial_thread(void) >> WINDOWS_ONLY({return false;}) BSD_ONLY({return false;}) >> >> >> I can do that. I'll see if anyone else has any comments on >> this. >> >> >> I'm assuming that Thomas means for the above to be at the >> platform independent layer (runtime/os.hpp?) so it is more >> obvious to the reader that the function doesn't do anything >> on those two platforms... I know we dislike such things at >> the platform independent layer, but it does feel appropriate >> to make this limitation more obvious. >> >> I would be okay if you made the change that Thomas is suggesting. >> >> Thumbs up! >> >> Dan >> >> >> >> >> There are three parts to this which made it somewhat more >> complicated than I had envisaged: >> >> 1. Add the flag and disable the guards. >> >> This is relatively straight-forward: >> >> ?void JavaThread::create_stack_guard_pages() { >> ?? if (!os::uses_stack_guard_pages() || >> ?????? _stack_guard_state != stack_guard_unused || >> ?????? (DisablePrimordialThreadGuardPages && >> os::is_primordial_thread())) { >> ?????? log_info(os, thread)("Stack guard page creation for >> thread " >> ??????????????????????????? UINTX_FORMAT " disabled", >> os::current_thread_id()); >> ???? return; >> >> with a tweak in JavaThread::stack_guards_enabled as well. >> >> 2. Promote os::XXX::is_initial_thread to >> os::is_primordial_thread() >> >> We have three platforms that already implement this >> functionality: Linux, Solaris and AIX. For BSD/OSX and >> Windows we don't attempt to do anything different for the >> primordial thread, and we don't have a way to detect the >> primordial thread - so is_primordial_thread() just returns >> false. >> >> I also tidied up some comments regarding terminology. >> >> 3. Thomas noticed that we unconditionally subtract >> 2*page_size() from the rlimit stack size, without checking >> it was bigger than 2*page_size() to start with. I was >> unsure how to tackle this. It's no good leaving stack_size >> at zero so I opted to ensure we would have at least one >> page left. Of course in such cases we would hit the bug in >> libc (if it still exists, which seems unlikely but time >> consuming to establish). >> >> Testing: >> ? - Tier 1 (JPRT) testing with a modified launcher that >> uses the primordial thread >> ??? - with guard pages enabled >> ??? - with guard pages disabled >> ? - Tier 1 (JPRT) normal JDK build (which should be >> unaffected by this change) >> >> The testing was primarily to ensure that disabling the >> stack guards on the primordial thread wasn't totally >> broken. A number of tests fail: >> - setting -Xss32m fails for the primordial thread when >> guards are enabled and the rlimit is 8m >> - tests that would hit StackOverflowError crash the VM >> when guards are disabled (as you would expect). >> - runtime/execstack/Testexecstack.java crashes when the >> guards are disabled >> >> Thanks, >> David >> >> >> > From tomas.kalibera at gmail.com Mon Nov 20 14:37:21 2017 From: tomas.kalibera at gmail.com (Tomas Kalibera) Date: Mon, 20 Nov 2017 15:37:21 +0100 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> <1e422a91-78e2-2e5c-2543-eaa7d49c60c7@oracle.com> Message-ID: <19f08f81-ca53-8df0-7682-174b582bb260@gmail.com> I have tested with R and it seems to be working fine. Specifically, I've tested with jdk/hs 47868 as baseline, with R-devel 73756, rJava 0.9-9 (modified to print VM args). I've run R recursive calls after initializing the jvm via .jinit(), using RJAVA_JVM_STACK_WORKAROUND=1 (detect inserted guard pages, but only print a message when they appear). The baseline inserted guard pages as expected following -Xss (default and when specified 1m..9m) and following the implied cap when rlimit stack is unlimited, and R consequenlty crashed in the baseline. The modified version with the fix activated did not insert guard pages. Tested with rlimit stack of 8m (Ubuntu default) and unlimited. From R, I activated the fix via options(java.parameters=c("-XX:+UnlockExperimentalVMOptions", "-XX:+DisablePrimordialThreadGuardPages")) Tested on Ubuntu 17.04, x86_64. jdk/hs built with gcc 4.9.4 (but still needed --disable-warnings-as-errors) Thanks a lot for the new option! Tomas On 11/20/2017 02:06 PM, David Holmes wrote: > Thank you both. > > David > > On 20/11/2017 11:05 PM, Daniel D. Daugherty wrote: >> I'm also good with this change. >> >> Dan >> >> >> On 11/20/17 4:04 AM, Thomas St?fe wrote: >>> Hi David, >>> >>> looks fine. >>> >>> Thanks, Thomas >>> >>> On Mon, Nov 20, 2017 at 4:44 AM, David Holmes >>> > wrote: >>> >>> ??? Hi Dan, >>> >>> ??? Thanks for the review. I've addressed all of yours and Thomas's >>> ??? comments. >>> >>> ??? http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ >>> >>> >>> ??? The main change is: >>> >>> ??? +? ?static bool is_primordial_thread(void) >>> ??? + #if defined(_WINDOWS) || defined(BSD) >>> ??? +? ? ?// No way to identify the primordial thread. >>> ??? +? ? ?{ return false; } >>> ??? + #else >>> ??? +? ?; >>> ??? + #endif >>> >>> >>> ??? Also to clarify this is logically a "product" flag for the systems >>> ??? that need it, not a diagnostic one. But as we don't want this flag >>> ??? to be in common use and it is of such narrow applicability, we >>> ??? have made it experimental. This was discussed previously in the >>> ??? CSR review thread for this: >>> >>> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html >>> >>> >>> ??? Once Tomas K. has been able to test this, and you and Thomas >>> ??? sign-off on the updates, I'll get this pushed. But as I'm away >>> ??? from Thursday it may not be until next week. >>> >>> ??? Thanks, >>> ??? David >>> ??? ----- >>> >>> >>> >>> ??? On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: >>> >>> ??????? On 11/14/17 5:39 AM, David Holmes wrote: >>> >>> ??????????? Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>> >>> >>> ??????????? webrev: >>> http://cr.openjdk.java.net/~dholmes/8189170/webrev/ >>> >>> >>> >>> ??????? src/hotspot/os/aix/os_aix.cpp >>> ??????? ???? No comments. >>> >>> ??????? src/hotspot/os/aix/os_aix.hpp >>> ??????? ???? No comments. >>> >>> ??????? src/hotspot/os/bsd/os_bsd.cpp >>> ??????? ???? L880: // thread in any special way - so just report >>> 'false' >>> ??????? ??? ? ?? Nit - needs a period after 'false' to make it a >>> sentence. >>> >>> ??????? ???? L882: ??? return false; >>> ??????? ???????? Nit - indent should be 2 spaces. >>> >>> ??????? src/hotspot/os/bsd/os_bsd.hpp >>> ??????? ???? No comments. >>> >>> ??????? src/hotspot/os/linux/os_linux.cpp >>> ??????? ???? L947: ? if (stack_size >= (size_t)(3 * page_size())) >>> ??????? ???????? Nit - needs '{' and '}' on the if-statement body. >>> >>> ??????? ???? L3070: // always place it right after end of the mapped >>> ??????? region >>> ??????? ???????? Nit - needs a period after 'region' to make it a >>> ??????? sentence. >>> ??????? ???????? (Not your problem, but you did update the sentence...) >>> >>> ??????? src/hotspot/os/linux/os_linux.hpp >>> ??????? ???? No comments. >>> >>> ??????? src/hotspot/os/solaris/os_solaris.cpp >>> ??????? ???? No comments other than I learned something new about >>> ??????? thr_main() :-) >>> ??????? ???? Thanks for cleaning up that mess. >>> >>> ??????? src/hotspot/os/windows/os_windows.cpp >>> ??????? ???? L452: // thread in any special way - so just report >>> 'false' >>> ??????? ???????? Nit - needs a period after 'false' to make it a >>> sentence. >>> >>> ??????? ???? L454: ??? return false; >>> ??????? ???????? Nit - indent should be 2 spaces. >>> >>> ??????? src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp >>> ??????? ???? No comments. >>> >>> ??????? src/hotspot/share/runtime/globals.hpp >>> ??????? ???? L1181: ? experimental(bool, >>> ??????? DisablePrimordialThreadGuardPages, false, >>> ??????? ???????? Experimental or diagnostic? >>> >>> ??????? src/hotspot/share/runtime/os.hpp >>> ??????? ???? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM >>> ??????? ???????? Nit - needs a period after 'JNI_CreateJavaVM'. >>> >>> ??????? ???? L453: ? // launcher nevers uses the primordial thread as >>> ??????? the main thread, but >>> ??????? ???????? Typo - 'nevers' -> 'never' >>> >>> ??????? ???? L455: ? // have to special-case handling of the >>> ??????? primordial thread if it attaches >>> ??????? ???????? Typo - 'have to' -> 'have' >>> >>> ??????? src/hotspot/share/runtime/thread.cpp >>> ??????? ???? I like the new log message. >>> >>> ??????? src/hotspot/share/runtime/thread.inline.hpp >>> ??????? ???? L159: ? if (os::uses_stack_guard_pages() && >>> ??????? ???? L160: ????? !(DisablePrimordialThreadGuardPages && >>> ??????? os::is_primordial_thread())) { >>> ??????? ???????? I'm not sure why, but I had to read these two lines >>> ??????? several >>> ??????? ???????? times before I convinced myself the logic is right. >>> >>> ??????? I'm good with the code changes. Your call on whether to fix the >>> ??????? bits or not. >>> >>> ??????? Regarding this part of Thomas' review: >>> >>> ??????????????? Or, something like this: >>> ??????????????? static bool is_primordial_thread(void) >>> ??????????????? WINDOWS_ONLY({return false;}) BSD_ONLY({return false;}) >>> >>> >>> ??????????? I can do that. I'll see if anyone else has any comments on >>> ??????????? this. >>> >>> >>> ??????? I'm assuming that Thomas means for the above to be at the >>> ??????? platform independent layer (runtime/os.hpp?) so it is more >>> ??????? obvious to the reader that the function doesn't do anything >>> ??????? on those two platforms... I know we dislike such things at >>> ??????? the platform independent layer, but it does feel appropriate >>> ??????? to make this limitation more obvious. >>> >>> ??????? I would be okay if you made the change that Thomas is >>> suggesting. >>> >>> ??????? Thumbs up! >>> >>> ??????? Dan >>> >>> >>> >>> >>> ??????????? There are three parts to this which made it somewhat more >>> ??????????? complicated than I had envisaged: >>> >>> ??????????? 1. Add the flag and disable the guards. >>> >>> ??????????? This is relatively straight-forward: >>> >>> ??????????? ?void JavaThread::create_stack_guard_pages() { >>> ??????????? ?? if (!os::uses_stack_guard_pages() || >>> ??????????? ?????? _stack_guard_state != stack_guard_unused || >>> ??????????? ?????? (DisablePrimordialThreadGuardPages && >>> ??????????? os::is_primordial_thread())) { >>> ??????????? ?????? log_info(os, thread)("Stack guard page creation for >>> ??????????? thread " >>> ??????????? ??????????????????????????? UINTX_FORMAT " disabled", >>> ??????????? os::current_thread_id()); >>> ??????????? ???? return; >>> >>> ??????????? with a tweak in JavaThread::stack_guards_enabled as well. >>> >>> ??????????? 2. Promote os::XXX::is_initial_thread to >>> ??????????? os::is_primordial_thread() >>> >>> ??????????? We have three platforms that already implement this >>> ??????????? functionality: Linux, Solaris and AIX. For BSD/OSX and >>> ??????????? Windows we don't attempt to do anything different for the >>> ??????????? primordial thread, and we don't have a way to detect the >>> ??????????? primordial thread - so is_primordial_thread() just returns >>> ??????????? false. >>> >>> ??????????? I also tidied up some comments regarding terminology. >>> >>> ??????????? 3. Thomas noticed that we unconditionally subtract >>> ??????????? 2*page_size() from the rlimit stack size, without checking >>> ??????????? it was bigger than 2*page_size() to start with. I was >>> ??????????? unsure how to tackle this. It's no good leaving stack_size >>> ??????????? at zero so I opted to ensure we would have at least one >>> ??????????? page left. Of course in such cases we would hit the bug in >>> ??????????? libc (if it still exists, which seems unlikely but time >>> ??????????? consuming to establish). >>> >>> ??????????? Testing: >>> ??????????? ? - Tier 1 (JPRT) testing with a modified launcher that >>> ??????????? uses the primordial thread >>> ??????????? ??? - with guard pages enabled >>> ??????????? ??? - with guard pages disabled >>> ??????????? ? - Tier 1 (JPRT) normal JDK build (which should be >>> ??????????? unaffected by this change) >>> >>> ??????????? The testing was primarily to ensure that disabling the >>> ??????????? stack guards on the primordial thread wasn't totally >>> ??????????? broken. A number of tests fail: >>> ??????????? - setting -Xss32m fails for the primordial thread when >>> ??????????? guards are enabled and the rlimit is 8m >>> ??????????? - tests that would hit StackOverflowError crash the VM >>> ??????????? when guards are disabled (as you would expect). >>> ??????????? - runtime/execstack/Testexecstack.java crashes when the >>> ??????????? guards are disabled >>> >>> ??????????? Thanks, >>> ??????????? David >>> >>> >>> >> From adinn at redhat.com Mon Nov 20 16:58:28 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 20 Nov 2017 16:58:28 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <4ffdb4b6-8902-b5e7-b68c-e7a7e5210ed9@redhat.com> References: <1510824184.3968.3.camel@redhat.com> <4ffdb4b6-8902-b5e7-b68c-e7a7e5210ed9@redhat.com> Message-ID: <2f9a839d-b9b9-d9eb-3f0d-67d90b0acbf2@redhat.com> Hi Zhengyu, On 17/11/17 20:43, Zhengyu Gu wrote: > There were two bugs that resulted NMT report incorrect thread stacks. > > 1) NMT did not handle overlapped regions correctly. > > 2) There are two issues with Linux: get_stack_commited_bottom() > > ?* The method name is misleading, it does not return *committed* bottom, > but *mapped* bottom, where address space is valid, not necessary committed. > ?? I renamed this function to get_stack_mapped_bottom() with option to > return committed bottom with proper parameter. > > ?* The binary search algorithm mishandled edge cases. Hmm, yes looking through the mincore man page again you are right that you need to check the returned bit vector to see if a vmem page is committed to a physical page. I'm not sure about the logic of the test when you hit the the ENOMEM/uncommited case. + mincore_return_value = mincore(test_addr, page_sz, vec); + + if (mincore_return_value == -1 || (committed_only && (vec[0] & 0x01) == 0)) { + // Page is not mapped/committed go up + // to find first mapped/committed page + if (errno != EAGAIN || (committed_only && (vec[0] & 0x01) == 0)) { + assert(mincore_return_value != -1 || errno == ENOMEM, "Unexpected mincore errno"); Should that not be + if ((mincore_return_value != -1 && errno != EAGAIN) || (committed_only && (vec[0] & 0x01) == 0)) { A successful call to mincore is not guaranteed to reset errno and it might by chance already have value EAGAIN before the call to mincore. If we happened to hit a mapped, committed page first time then the vec bit will be 1 and the code will loop and retest the same address. Unlikely but possible? The adjustment to the edge case logic now looks correct. > The test case for verifying fixes for above two issues can be found > here: http://cr.openjdk.java.net/~zgu/8191369/test_mmap.c Yes, that verifies the edge case handling e.g. I ran it with 256 pages and with 127/8/9 mapped and it gave the right results. > As mentioned in early email, new patch also contains implementation for > Windows. > > Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ > > Test: > > ? Reran hotspot_tier1 tests on Windows x64 and Linux x64 (fastdebug and > release) I have no idea about the validity of the Windows fix but the Linux one looks good modulo that if condition. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Mon Nov 20 17:08:38 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 20 Nov 2017 17:08:38 +0000 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <2f9a839d-b9b9-d9eb-3f0d-67d90b0acbf2@redhat.com> References: <1510824184.3968.3.camel@redhat.com> <4ffdb4b6-8902-b5e7-b68c-e7a7e5210ed9@redhat.com> <2f9a839d-b9b9-d9eb-3f0d-67d90b0acbf2@redhat.com> Message-ID: <1e6993e8-f30d-3e56-d908-0ed03a0330e4@redhat.com> Oops, correcting my correction: On 20/11/17 16:58, Andrew Dinn wrote: > On 17/11/17 20:43, Zhengyu Gu wrote: >> There were two bugs that resulted NMT report incorrect thread stacks. >> >> 1) NMT did not handle overlapped regions correctly. >> >> 2) There are two issues with Linux: get_stack_commited_bottom() >> >> ?* The method name is misleading, it does not return *committed* bottom, >> but *mapped* bottom, where address space is valid, not necessary committed. >> ?? I renamed this function to get_stack_mapped_bottom() with option to >> return committed bottom with proper parameter. >> >> ?* The binary search algorithm mishandled edge cases. > > Hmm, yes looking through the mincore man page again you are right that > you need to check the returned bit vector to see if a vmem page is > committed to a physical page. > > I'm not sure about the logic of the test when you hit the the > ENOMEM/uncommited case. > > + mincore_return_value = mincore(test_addr, page_sz, vec); > + > + if (mincore_return_value == -1 || (committed_only && (vec[0] & > 0x01) == 0)) { > + // Page is not mapped/committed go up > + // to find first mapped/committed page > + if (errno != EAGAIN || (committed_only && (vec[0] & 0x01) == 0)) { > + assert(mincore_return_value != -1 || errno == ENOMEM, > "Unexpected mincore errno"); > > Should that not be > > + if ((mincore_return_value != -1 && errno != EAGAIN) || > (committed_only && (vec[0] & 0x01) == 0)) { Sorry, that correction should have been: + if ((mincore_return_value == -1 && errno != EAGAIN) || (committed_only && (vec[0] & 0x01) == 0)) { > A successful call to mincore is not guaranteed to reset errno and it > might by chance already have value EAGAIN before the call to mincore. If > we happened to hit a mapped, committed page first time then the vec bit > will be 1 and the code will loop and retest the same address. Unlikely > but possible? > > > The adjustment to the edge case logic now looks correct. > >> The test case for verifying fixes for above two issues can be found >> here: http://cr.openjdk.java.net/~zgu/8191369/test_mmap.c > > Yes, that verifies the edge case handling e.g. I ran it with 256 pages > and with 127/8/9 mapped and it gave the right results. > >> As mentioned in early email, new patch also contains implementation for >> Windows. >> >> Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ >> >> Test: >> >> ? Reran hotspot_tier1 tests on Windows x64 and Linux x64 (fastdebug and >> release) > I have no idea about the validity of the Windows fix but the Linux one > looks good modulo that if condition. > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > -- regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From zgu at redhat.com Mon Nov 20 18:46:32 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 20 Nov 2017 13:46:32 -0500 Subject: RFR(XS) 8191369: NMT: Enhance thread stack tracking In-Reply-To: <1e6993e8-f30d-3e56-d908-0ed03a0330e4@redhat.com> References: <1510824184.3968.3.camel@redhat.com> <4ffdb4b6-8902-b5e7-b68c-e7a7e5210ed9@redhat.com> <2f9a839d-b9b9-d9eb-3f0d-67d90b0acbf2@redhat.com> <1e6993e8-f30d-3e56-d908-0ed03a0330e4@redhat.com> Message-ID: <8f51b0db-5a58-1b04-6d31-934a729ba418@redhat.com> Hi Andrew, Thanks for the review. >> I'm not sure about the logic of the test when you hit the the >> ENOMEM/uncommited case. >> >> + mincore_return_value = mincore(test_addr, page_sz, vec); >> + >> + if (mincore_return_value == -1 || (committed_only && (vec[0] & >> 0x01) == 0)) { >> + // Page is not mapped/committed go up >> + // to find first mapped/committed page >> + if (errno != EAGAIN || (committed_only && (vec[0] & 0x01) == 0)) { >> + assert(mincore_return_value != -1 || errno == ENOMEM, >> "Unexpected mincore errno"); >> >> Should that not be >> >> + if ((mincore_return_value != -1 && errno != EAGAIN) || >> (committed_only && (vec[0] & 0x01) == 0)) { > > Sorry, that correction should have been: > > + if ((mincore_return_value == -1 && errno != EAGAIN) || > (committed_only && (vec[0] & 0x01) == 0)) { Thanks for pointing out. Updated accordingly: Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.02/ -Zhengyu > >> A successful call to mincore is not guaranteed to reset errno and it >> might by chance already have value EAGAIN before the call to mincore. If >> we happened to hit a mapped, committed page first time then the vec bit >> will be 1 and the code will loop and retest the same address. Unlikely >> but possible? >> >> >> The adjustment to the edge case logic now looks correct. >> >>> The test case for verifying fixes for above two issues can be found >>> here: http://cr.openjdk.java.net/~zgu/8191369/test_mmap.c >> >> Yes, that verifies the edge case handling e.g. I ran it with 256 pages >> and with 127/8/9 mapped and it gave the right results. >> >>> As mentioned in early email, new patch also contains implementation for >>> Windows. >>> >>> Updated Webrev: http://cr.openjdk.java.net/~zgu/8191369/webrev.01/ >>> >>> Test: >>> >>> Reran hotspot_tier1 tests on Windows x64 and Linux x64 (fastdebug and >>> release) >> I have no idea about the validity of the Windows fix but the Linux one >> looks good modulo that if condition. >> >> regards, >> >> >> Andrew Dinn >> ----------- >> Senior Principal Software Engineer >> Red Hat UK Ltd >> Registered in England and Wales under Company Registration No. 03798903 >> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander >> > From gerard.ziemski at oracle.com Mon Nov 20 18:58:44 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 20 Nov 2017 12:58:44 -0600 Subject: RFR (XS) 8191580: open/test/hotspot/jtreg/runtime/LoadClass/TestResize fails on product build Message-ID: hi all, Please review this trivial fix to test/hotspot/jtreg/runtime/LoadClass/TestResize.java, which was failing on product build. The problem here was that TestResize launches a java process with "PrintSystemDictionaryAtExit" flag set, which is a "notproduct" flag, and will fail on product build. Thanks to Robbin Ehn for reporting it. bug: https://bugs.openjdk.java.net/browse/JDK-8191580 webrev: http://cr.openjdk.java.net/~gziemski/8191580_rev1 cheers From harold.seigel at oracle.com Mon Nov 20 19:16:34 2017 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 20 Nov 2017 14:16:34 -0500 Subject: RFR (XS) 8191580: open/test/hotspot/jtreg/runtime/LoadClass/TestResize fails on product build In-Reply-To: References: Message-ID: <4f69abb5-7605-336a-3d39-5689398e3b7e@oracle.com> Hi Gerard, This fix looks good. Thanks, Harold On 11/20/2017 1:58 PM, Gerard Ziemski wrote: > hi all, > > Please review this trivial fix to test/hotspot/jtreg/runtime/LoadClass/TestResize.java, which was failing on product build. > > The problem here was that TestResize launches a java process with "PrintSystemDictionaryAtExit" flag set, which is a "notproduct" flag, and will fail on product build. > > Thanks to Robbin Ehn for reporting it. > > bug: https://bugs.openjdk.java.net/browse/JDK-8191580 > webrev: http://cr.openjdk.java.net/~gziemski/8191580_rev1 > > > cheers > From jiangli.zhou at oracle.com Mon Nov 20 19:19:26 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 20 Nov 2017 11:19:26 -0800 Subject: RFR (XS) 8191580: open/test/hotspot/jtreg/runtime/LoadClass/TestResize fails on product build In-Reply-To: References: Message-ID: <43B0BD36-86FA-4B7E-ABAA-4E97998C50B0@oracle.com> Looks good. Thanks, Jiangli > On Nov 20, 2017, at 10:58 AM, Gerard Ziemski wrote: > > hi all, > > Please review this trivial fix to test/hotspot/jtreg/runtime/LoadClass/TestResize.java, which was failing on product build. > > The problem here was that TestResize launches a java process with "PrintSystemDictionaryAtExit" flag set, which is a "notproduct" flag, and will fail on product build. > > Thanks to Robbin Ehn for reporting it. > > bug: https://bugs.openjdk.java.net/browse/JDK-8191580 > webrev: http://cr.openjdk.java.net/~gziemski/8191580_rev1 > > > cheers > From gerard.ziemski at oracle.com Mon Nov 20 19:55:03 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 20 Nov 2017 13:55:03 -0600 Subject: RFR (XS) 8191580: open/test/hotspot/jtreg/runtime/LoadClass/TestResize fails on product build In-Reply-To: <43B0BD36-86FA-4B7E-ABAA-4E97998C50B0@oracle.com> References: <43B0BD36-86FA-4B7E-ABAA-4E97998C50B0@oracle.com> Message-ID: <90D62B68-05E8-498B-93A7-0DA20CFCD4FB@oracle.com> hi Jiangli, Thank you for the review. cheers > On Nov 20, 2017, at 1:19 PM, Jiangli Zhou wrote: > > Looks good. > > Thanks, > Jiangli > >> On Nov 20, 2017, at 10:58 AM, Gerard Ziemski wrote: >> >> hi all, >> >> Please review this trivial fix to test/hotspot/jtreg/runtime/LoadClass/TestResize.java, which was failing on product build. >> >> The problem here was that TestResize launches a java process with "PrintSystemDictionaryAtExit" flag set, which is a "notproduct" flag, and will fail on product build. >> >> Thanks to Robbin Ehn for reporting it. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8191580 >> webrev: http://cr.openjdk.java.net/~gziemski/8191580_rev1 >> >> >> cheers >> > From calvin.cheung at oracle.com Mon Nov 20 20:05:58 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 20 Nov 2017 12:05:58 -0800 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <5A048F23.9030806@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> <5A033E72.90800@oracle.com> <5A048F23.9030806@oracle.com> Message-ID: <5A1335A6.2070509@oracle.com> I've had some off-list discussion with Ioi resulting in another update to the webrev. - added relative path scenario to the test. Currently, this fix doesn't handle relative path on windows yet. The following RFE has been filed to cover long relative paths on windows: https://bugs.openjdk.java.net/browse/JDK-8191521 - renamed the test LongPath.java to LongBCP.java to better reflect what is being tested; - some comments update on os_windows.cpp updated webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.05/ thanks, Calvin On 11/9/17, 9:23 AM, Calvin Cheung wrote: > Hi Thomas, > > On 11/8/17, 10:40 PM, Thomas St?fe wrote: >> Hi Calvin, >> >> On Wed, Nov 8, 2017 at 6:27 PM, Calvin Cheung >> > wrote: >> >> >> >> On 11/7/17, 6:12 AM, Thomas St?fe wrote: >>> Hi Calvin, >>> >>> On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung >>> > wrote: >>> >>> >>> >>> On 10/27/17, 12:55 AM, Thomas St?fe wrote: >>>> Hi Calvin, >>>> >>>> On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >>>> >>> > wrote: >>>> >>>> Hi Thomas, >>>> >>>> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >>>> >>>> Hi Calvin, >>>> >>>> this is a worthwhile addition, thank you for your work! >>>> >>>> Thanks for your review. >>>> >>>> >>>> Thanks for your work :) >>>> >>>> First off, there is another issue in >>>> file_attribute_data_to_stat(). We actually had this issue >>>> before, but your work uncovered it: >>>> >>>> According to POSIX >>>> (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html >>>> ) >>>> and every unix manpage I looked at (solaris, linux, aix..), >>>> st_ctime is not the file creation time but the last time >>>> file status was changed. Only Microsoft gets this wrong in >>>> their posix layer, its stat() function returns >>>> (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx >>>> ) >>>> creation time in st_ctime. >>>> >>>> But as os::stat() is a platform independent layer, I'd say >>>> we should behave the same on all platforms, and that would >>>> be the Posix way. >>>> >>>> I did not find any code calling os::stat() and using >>>> st_ctime, so this is still save to change for windows. >>>> (Note that I found that perfMemory_xxx.cpp uses plain OS >>>> ::stat and misuses ctime as "creation time" - I opened a >>>> bug for that >>>> (https://bugs.openjdk.java.net/browse/JDK-8190260 >>>> - but >>>> that does not affect us, as they do not call os::stat()). >>>> >>>> There is one more complication: in os::stat() we use plain >>>> ::stat() in the single byte case.: >>>> >>>> *+ if (strlen(path) < MAX_PATH) {* >>>> *+ ret = ::stat(pathbuf, sbuf);* >>>> *+ } else {* >>>> * >>>> * >>>> But ::stat() on Windows is broken, as I wrote above. We >>>> should not use it, especially not if we change the meaing >>>> of st_ctime in the double byte path. >>>> >>>> My suggestion would be to just always call >>>> GetFileAttributes(), also for the single byte path. A very >>>> simple solution would be to just always go the double byte >>>> path with UNC translation, regardless of the path >>>> length. Or, if you are worried about performance, for paths >>>> shorter than MAX_PATH we use GetFileAttributesA(), for >>>> longer paths GetFileAttributesW with UNC translation. In >>>> both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you >>>> can feed tp your file_attribute_data_to_stat() routine and >>>> have the struct stat you return be always consistent. >>> I ran into an issue with the dwFileAttributes value for a >>> jar file. On Windows Server O/S, the value is set to 0x2020 >>> which is (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>> FILE_ATTRIBUTE_ARCHIVE) but on a desktop O/S like Windows 7, >>> it is set to 0x0020 which is just FILE_ATTRIBUTE_ARCHIVE. >>> I've fixed it in file_attribute_data_to_stat(). >>> >>> >>> Oh.. :( good catch! But that opens a new can of worms I did not >>> see before: >>> >>> FILE_ATTRIBUTE_NORMAL is documented as "A file that does not >>> have other attributes set. This attribute is valid only when >>> used alone." In addition to this flag, we have a multitude of >>> things like FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, >>> FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this >>> is a normal file in Unix terminology and where we would expect >>> os::stat to return S_IFREG, but where according to the msdn doc >>> FILE_ATTRIBUTE_NORMAL is not set. >>> >>> Looking at what others do in this scenario (Looked at mingw >>> sources and at ReactOS - I am not posting any source code here >>> for legal reasons but feel free to look for yourself), the usual >>> way to translate WIN32_FILE_ATTRIBUTES to struct stat seems to be: >>> "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, >>> no dependency on FILE_ATTRIBUTE_NORMAL). >> This makes sense but I ran into similar problem as before - the >> dwFileAttributes has a different value on a windows server O/S vs >> desktop O/S. So I need to do the check as follows: >> >> + // A directory has the dwFileAttributes value of 0x2010 which is >> + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) >> + // on Windows Server O/S but the value is 0x0010 on Windows desktop O/S >> + // such as Windows 7. >> + if ((file_data.dwFileAttributes& FILE_ATTRIBUTE_DIRECTORY) != 0) { >> + sbuf->st_mode |= S_IFDIR; >> + } else { >> + sbuf->st_mode |= S_IFREG; >> + } >> >> I scratched my head a bit about the comment till I understood that >> you mean "if FILE_ATTRIBUTE_DIRECTORY bit is set it is a directory >> regardless of which other flags are set" instead of "if >> flags==FILE_ATTRIBUTE_DIRECTORY it is a directory". Sure, I guess my >> comment above was sloppy, but this was what I meant. I am not even >> sure the comment is needed, this is quite self-explaining. > I've noticed a typo in the above comment: > + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) > > FILE_ATTRIBUTE_ARCHIVE should be FILE_ATTRIBUTE_DIRECTORY > > I'll correct it before push. > >> >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ >> >> >> >> I am fine with all the changes now. Thank you for your work and patience. > Thanks for your discussions and review. > > thanks, > Calvin >> >> Kind Regards, Thomas >> >> >> >> >> Diff from webrev.03: >> >> < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >> 08:50:40.170786397 -0800 >> < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >> 08:50:39.803751877 -0800 >> < @@ -4060,41 +4060,119 @@ >> --- >> > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >> 09:40:13.657460425 -0700 >> > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >> 09:40:13.261423192 -0700 >> > @@ -4060,41 +4060,121 @@ >> 25,29c25 >> < + // A directory has the dwFileAttributes value of 0x2010 which is >> < + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) >> < + // on Windows Server O/S but the value is 0x0010 on Windows >> desktop O/S >> < + // such as Windows 7. >> < + if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) >> != 0) { >> --- >> > + if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { >> 31c27,33 >> < + } else { >> --- >> > + } >> > + if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || >> > + // an archive file such as a jar file has the >> dwFileAttributes value of >> > + // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >> FILE_ATTRIBUTE_ARCHIVE) >> > + // on Windows Server O/S but the value is 0x0020 on >> > + // Windows desktop O/S such as Windows 7. >> > + ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) >> != 0)) { >> >>> >>> I wonder about other special cases which should work too: >>> - read-only files should be S_IFREG and !S_IWRITE, >> For a read-only system file under the user's home dir. >> >> st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD >> dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | >> FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | >> FILE_ATTRIBUTE_READONLY) >>> read-only directories should be S_IFDIR and !S_IWRITE. >> I've tried the C:\progra~1 dir. >> >> st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >> dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | >> FILE_ATTRIBUTE_READONLY) >>> - We should tread the device root ("C:\") as a directory (so, >>> os::stat("c:\") should return S_IFDIR). Does this work? >> This one works too. >> >> st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >> dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | >> FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) >>> >>> I've also used GetTickCounts() to measure the performance of >>> ::stat() vs GetFileAttributesA() plus >>> file_attribute_data_to_stat(). There's no difference at the >>> ms resolution. Using the high-resolution timestamp >>> (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx) >>> , >>> it doesn't show much difference. >>> >>> >>> stat() seems to be implemented using FindFirstFile() (see crt >>> sources if you can), and we call GetFileAttributesA(), so I do >>> not think this differs much. >> Yes, I saw the same in my Visual Studio installation. >> >> thanks, >> Calvin >> >>>> >>>> But onward: >>>> >>>> >>>> >>>> ========================= >>>> >>>> src/hotspot/os/windows/os_windows.cpp >>>> >>>> Could you please use wchar_t instead of WCHAR? >>>> >>>> Fixed. >>>> >>>> >>>> --------------- >>>> in os::stat(): >>>> >>>> This cumbersome malloc/strcpy sequence: >>>> >>>> ! size_t path_len = strlen(path) + 1; >>>> + char* pathbuf = (char*)os::malloc(path_len * >>>> sizeof(char), mtInternal); >>>> + if (pathbuf == NULL) { >>>> + errno = ENOMEM; >>>> return -1; >>>> } >>>> os::native_path(strcpy(pathbuf, path)); >>>> >>>> can be reduced to a simple strdup: >>>> >>>> char* pathbuf = os::strdup(path, mtInternal); >>>> if (pathbuf == NULL) { >>>> errno = ENOMEM; >>>> return -1; >>>> } >>>> os::native_path(pathbuf); >>>> >>>> This also would separate strcpy() from >>>> os::native_path() which is a bit unreadable. >>>> >>>> I've made the change. >>>> >>>> >>>> -------------------- >>>> >>>> The single-byte path (strdup, ::stat()), together >>>> with its free(), can be moved inside the >>>> (strlen(path) < MAX_PATH) condition. >>>> os::native_path will not change the path length (it >>>> better not, as it operates on the input buffer). >>>> That avoids having two allocations on the >>>> doublebyte path. >>>> >>>> >>>> os::native_path() converts a path like C:\\\\somedir to >>>> C:\\somedir >>>> So I'll need the converted path in the wchar_t case >>>> too. The freeing of the pathbuf is being done at the >>>> end of the function or in the middle of the wchar_t >>>> case if there's an error. >>>> >>>> >>>> Oh, you are right,of course. I missed that it this is >>>> needed for both paths. >>>> >>>> >>>> >>>> ----------------------- >>>> >>>> But seeing that translation to UNC paths is >>>> something we might want more often, I would combine >>>> allocation and UNC prefix adding to one function >>>> like this, to avoid the memove and increase >>>> reusability: >>>> >>>> // Creates an unc path from a single byte path. >>>> Return buffer is allocated in C heap and needs to >>>> be freed by caller. Returns NULL on error. >>>> static whchar_t* create_unc_path(const char* s) { >>>> - does s start with \\?\ ? >>>> - yes: >>>> - os::malloc(strlen(s) + 1) and mbstowcs_s >>>> - no: >>>> - os::malloc(strlen(s) + 1 + 4), >>>> mbstowcs_s to fourth position in string, prefix >>>> with L"\\?\" >>>> } >>>> >>>> >>>> I also include the case for adding L"\\\\?\\UNC\0" at >>>> the beginning to be consistent with >>>> libjava/canonicalize_md.c. >>>> >>>> >>>> We also need error checking to mbstowcs_s. >>>> >>>> I've added assert like the following after the call: >>>> >>>> assert(converted_chars == path_len, "sanity"); >>>> >>>> >>>> >>>> create_unc_path() : >>>> >>>> - could you convert the /**/ to // comments, please? >>> Fixed. >>> >>> >>> thanks. >>> >>> >>>> >>>> - not sure about the assert after mbstowcs_s: if we happen >>>> to encounter an invalid multibyte character, function will >>>> fail and return an error: >>>> >>>> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >>>> >>>> "If mbstowcs_s encounters an invalid multibyte character, >>>> it puts 0 in *``pReturnValue, sets the destination buffer >>>> to an empty string, sets errno to EILSEQ, and returns EILSEQ." >>> I've changed create_unc_path() so that the caller will get >>> the errno and removed the assert. >>> >>> static wchar_t* create_unc_path(const char* path, errno_t &err) >>> >>> >>> Okay, works for me. >>> >>> >>>> >>>> As this is dependent on user data, we should not assert, >>>> but handle the return code of mbstowcs_s gracefully. Same >>>> goes for libjava/canonicalize_md.c. >>>> >>>> - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len >>>> + 7, path, path_len); >>>> third parameter is wrong, as we hand in an offset into the >>>> buffer, we must decrement the buffer size by this offset, >>>> so correct would be path_len +7 - 7 or just path_len. >>>> >>>> - Same error below: + ::mbstowcs_s(&converted_chars, >>>> &wpath[4], path_len + 4, path, path_len); >>> Fixed in both places. >>> >>> >>> Okay. >>> >>> >>>> >>>> >>>> Just for cleanliness, I would then wrap >>>> deallocation into an own function. >>>> >>>> static viud destroy_unc_path(whchar_t* s) { >>>> os::free(s); } >>>> >>>> I've added the destroy function. >>>> >>>> >>>> These two functions could be candidates of putting >>>> into os::win32 namespace, as this form of ANSI->UNC >>>> path translation is quite common - whoever wants to >>>> use the xxxxW() functions must do this. >>>> >>>> I'm leaving them in os_windows.cpp since they're being >>>> used only within that file. >>>> >>>> >>>> Fine by me. >>>> >>>> >>>> >>>> ----------------------- >>>> >>>> FindFirstFileW: >>>> >>>> I am pretty sure that you can achieve the same >>>> result with GetFileAttributesExW(). It also returns >>>> WIN32_FIND_DATAW. >>>> >>>> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is >>>> very similar to WIN32_FIND_DATAW. >>>> >>>> >>>> It is way more straightforward to use than >>>> FindFirstFileW, as it does not require you to write >>>> a callback hook. >>>> >>>> I've switched to using GetFileAttributesExW(). >>>> >>>> >>>> Thank you, this is way more readable. >>>> Another issue: do we still need the fix for 6539723 if we >>>> switch from stat() to GetFileAttributesExW()? This fix >>>> looks important, but the comment >>>> indicates that it could break things if the original bug is >>>> not present. >>>> >>>> Btw, this is another strong argument for scrapping ::stat() >>>> altogether on all code paths, not only for long input >>>> paths, because stat() and GetFileAttributesExW() may behave >>>> differently. So it would be good to use the same API on all >>>> code paths, in order to get the best test coverage. >>> For this round of change, I'm using >>> GetFileAttributesEx[A|W]() for both code paths. >>> >>> webrev: >>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ >>> >>> >>> thanks, >>> Calvin >>> >>> >>> Okay, all good apart from the issues mentioned above. Thanks for >>> your work! >>> >>> Best Regards, Thomas >>> >>> >>> >>>> >>>> >>>> >>>> ------- >>>> >>>> eval_find_data(): This is more of a generic helper >>>> function, could you rename this to something >>>> clearer, e.g. make_double_word() ? >>>> >>>> Ok. I've renamed it. >>>> >>>> >>>> Also, setup_stat(), could this be renamed more >>>> clearly to something like WIN32_FIND_DATA_to_stat? >>>> or lowercase if this bothers you :) >>>> >>>> I'm naming the function as >>>> file_attribute_data_to_stat() to match with the data >>>> structure name. >>>> >>>> >>>> Thanks for taking my suggestions. >>>> >>>> >>>> >>>> ================================== >>>> src/hotspot/share/classfile/classLoader.cpp >>>> >>>> In ClassPathDirEntry::open_stream(), I would feel >>>> better if we were asserting _dir and name to be != >>>> NULL before feeding it to strlen. >>>> >>>> I've added an assert statement. >>>> >>>> >>>> =================== >>>> >>>> Here's an updated webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Thomas >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> Thanks! >>>> >>>> Thomas >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >>>> >>> >>>> >>> >> wrote: >>>> >>>> I've reworked this fix by using the Unicode >>>> version of open >>>> (wopen) to handle path name longer than max >>>> path with the path >>>> prefixed to indicate an extended length path as >>>> described in [1]. >>>> >>>> The Unicode version of stat (wstat) doesn't >>>> work well with long >>>> path [2]. So FindFirstFileW will be used.The >>>> data in >>>> WIN32_FIND_DATA returned from FindFirstFileW >>>> needs to be >>>> transferred to the stat structure since the >>>> caller expects a >>>> return stat structure and other platforms >>>> return a stat structure. >>>> >>>> In classLoader.cpp, calculate the size of >>>> buffer required instead >>>> of limiting it to JVM_MAXPATHLEN. >>>> In os_windows.cpp, dynamically allocate buffers >>>> in os::open and >>>> os::stat. >>>> >>>> updated webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>>> >>>> >>> > >>>> >>>> It passed hs-tier2 testing using mach5. >>>> >>>> thanks, >>>> Calvin >>>> >>>> [1] >>>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH >>>> >>>> >>> > >>>> >>>> [2] >>>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>>> >>>> >>> > >>>> >>>> >>>> >>>> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>>> >>>> JBS: >>>> https://bugs.openjdk.java.net/browse/JDK-8188122 >>>> >>>> >>> > >>>> >>>> Adding a warning message if the full path >>>> or the directory >>>> length exceeds MAX_PATH on windows. >>>> >>>> Example warning messages. >>>> >>>> 1) The full path exceeds MAX_PATH: >>>> >>>> Java HotSpot(TM) 64-Bit Server VM warning: >>>> construct full path >>>> name failed: total length 270 exceeds max >>>> length of 260 >>>> dir >>>> >>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> length 259 >>>> name Hello.class length 11 >>>> >>>> 2) The directory path exceeds MAX_PATH: >>>> >>>> Java HotSpot(TM) 64-Bit Server VM warning: >>>> os::stat failed: >>>> path length 265 exceeds max length 260 >>>> path >>>> >>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>>> >>>> webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>>> >>>> >>> > >>>> >>>> Testing: >>>> JPRT (including the new test) >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> >>> >> From coleen.phillimore at oracle.com Mon Nov 20 20:12:20 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 20 Nov 2017 15:12:20 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/os/linux/os_linux.cpp.frames.html I read David's comments about the threads list iterator, and I was going to say that it can be cleaned up later, as the bulk of the change is the SMR part but this looks truly bizarre.?? It looks like it shouldn't compile because 'jt' isn't in scope. Why isn't this the syntax: JavaThreadIteratorWithHandle jtiwh; for (JavaThread* jt = jtiwh.first(); jt != NULL; jt = jtiwh.next()) { } Which would do the obvious thing without anyone having to squint at the code. http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.hpp.html As a hater of acronmys, can this file use "Safe Memory Reclaimation" and briefly describe the concept in the beginning of the header file, so one knows why it's called threadSMR.hpp? ? It doesn't need to be long and include why Threads list needs this.? Sometimes we tell new people that the hotspot documentation is in the header files. 186 JavaThreadIteratorWithHandle() : _tlh(), _index(0) {} This _tlh() call should not be necessary.? The compiler should generate this for you in the constructor. http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html 32 ThreadsList::ThreadsList(int entries) : _length(entries), _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), _next_list(NULL) { Seems like it should be mtThread rather than mtGC. Should 62 if (EnableThreadSMRStatistics) { really be UL, ie: if (log_is_enabled(Info, thread, smr, statistics)) ? http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java.html Can you use for these tests instead (there were a couple): *@requires (vm.debug == true)* http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/thread.cpp.udiff.html +// thread, has been added the Threads list, the system is not at a Has "not" been added to the Threads list?? missing "not"? + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); Can you add a comment about where this number came from? I can't find the caller of threads_do_smr. If these functions xchg_smr_thread_list, get_smr_java_thread_list, inc_smr_deleted_thread_count are only used by thread.cpp, I think they should go in that file and not in the .inline.hpp file to be included and possibly called by other files.? I think they're private to class Threads. I don't have any in-depth comments.? This looks really good from my day of reading it. Thanks, Coleen On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: > Greetings, > > Testing of the last round of changes revealed a hang in one of the new > TLH tests. Robbin has fixed the hang, updated the existing TLH test, and > added another TLH test for good measure. > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ > > Here is the updated delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ > > Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are > no unexpected failures. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Robbin rebased the project last night/this morning to merge with Thread >> Local Handshakes (TLH) and also picked up additional changesets up thru: >> >>> Changeset: fa736014cf28 >>> Author:??? cjplummer >>> Date:????? 2017-11-14 18:08 -0800 >>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>> >>> 8191049: Add alternate version of pns() that is callable from within >>> hotspot source >>> Summary: added pns2() to debug.cpp >>> Reviewed-by: stuefe, gthornbr >> >> This is the first time we've rebased the project to bits that are this >> fresh (< 12 hours old at rebase time). We've done this because we think >> we're done with this project and are in the final review-change-retest >> cycle(s)... In other words, we're not planning on making any more major >> changes for this project... :-) >> >> *** Time for code reviewers to chime in on this thread! *** >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >> >> Here's is the delta webrev from the 2017.11.10 rebase: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >> (and the new baseline also)... >> >> We're expecting this round to be a little noisier than usual because >> we did not rebase on a PIT snapshot so the new baseline has not been >> through Jesper's usual care-and-feeding of integration_blockers, etc. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>> (Yes, we're playing chase-the-repo...) >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>> >>> Unlike the previous rebase, there were no changes required to the >>> open code to get the rebased bits to build so there is no delta >>> webrev. >>> >>> These bits have been run through JPRT and I've submitted the usual >>> Mach5 tier[1-5] test run... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>> >>>> Here are the updated webrevs: >>>> >>>> Here's the mq comment for the change: >>>> >>>> ? Rebase to 2017.10.25 PIT snapshot. >>>> >>>> Here is the full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>> >>>> And here is the delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>> >>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>> look today. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Resolving one of the code review comments (from both Stefan K and >>>>> Coleen) >>>>> on jdk10-04-full required quite a few changes so it is being done >>>>> as a >>>>> standalone patch and corresponding webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>> ??? JavaThreadIteratorWithHandle. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> We have a (eXtra Large) fix for the following bug: >>>>>> >>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>> >>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>> >>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>> and track the work on this project: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>> >>>>>> >>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>> quotes >>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>> have a >>>>>> solution for that problem yet. >>>>>> >>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>> >>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>> tier[2-5] >>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>> additional testing on Erik and Robbin's machines. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Daniel Daugherty >>>>>> Erik Osterlund >>>>>> Robbin Ehn >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From david.holmes at oracle.com Mon Nov 20 20:50:58 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 21 Nov 2017 06:50:58 +1000 Subject: RFR: 8189170: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <19f08f81-ca53-8df0-7682-174b582bb260@gmail.com> References: <4c5c1484-b57d-fc73-1fb6-984a71fd6f4a@oracle.com> <055ec303-b6f0-0c9a-3355-20fc3376d8f0@oracle.com> <1e422a91-78e2-2e5c-2543-eaa7d49c60c7@oracle.com> <19f08f81-ca53-8df0-7682-174b582bb260@gmail.com> Message-ID: Thanks for confirming Tomas! David On 21/11/2017 12:37 AM, Tomas Kalibera wrote: > > I have tested with R and it seems to be working fine. > > Specifically, I've tested with jdk/hs 47868 as baseline, with R-devel > 73756, rJava 0.9-9 (modified to print VM args). I've run R recursive > calls after initializing the jvm via .jinit(), using > RJAVA_JVM_STACK_WORKAROUND=1 (detect inserted guard pages, but only > print a message when they appear). > > The baseline inserted guard pages as expected following -Xss (default > and when specified 1m..9m) and following the implied cap when rlimit > stack is unlimited, and R consequenlty crashed in the baseline. The > modified version with the fix activated did not insert guard pages. > Tested with rlimit stack of 8m (Ubuntu default) and unlimited. > > From R, I activated the fix via > options(java.parameters=c("-XX:+UnlockExperimentalVMOptions", > "-XX:+DisablePrimordialThreadGuardPages")) > > Tested on Ubuntu 17.04, x86_64. jdk/hs built with gcc 4.9.4 (but still > needed --disable-warnings-as-errors) > > Thanks a lot for the new option! > Tomas > > On 11/20/2017 02:06 PM, David Holmes wrote: >> Thank you both. >> >> David >> >> On 20/11/2017 11:05 PM, Daniel D. Daugherty wrote: >>> I'm also good with this change. >>> >>> Dan >>> >>> >>> On 11/20/17 4:04 AM, Thomas St?fe wrote: >>>> Hi David, >>>> >>>> looks fine. >>>> >>>> Thanks, Thomas >>>> >>>> On Mon, Nov 20, 2017 at 4:44 AM, David Holmes >>>> > wrote: >>>> >>>> ??? Hi Dan, >>>> >>>> ??? Thanks for the review. I've addressed all of yours and Thomas's >>>> ??? comments. >>>> >>>> ??? http://cr.openjdk.java.net/~dholmes/8189170/webrev.v2/ >>>> >>>> >>>> ??? The main change is: >>>> >>>> ??? +? ?static bool is_primordial_thread(void) >>>> ??? + #if defined(_WINDOWS) || defined(BSD) >>>> ??? +? ? ?// No way to identify the primordial thread. >>>> ??? +? ? ?{ return false; } >>>> ??? + #else >>>> ??? +? ?; >>>> ??? + #endif >>>> >>>> >>>> ??? Also to clarify this is logically a "product" flag for the systems >>>> ??? that need it, not a diagnostic one. But as we don't want this flag >>>> ??? to be in common use and it is of such narrow applicability, we >>>> ??? have made it experimental. This was discussed previously in the >>>> ??? CSR review thread for this: >>>> >>>> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html >>>> >>>> >>>> >>>> >>>> ??? Once Tomas K. has been able to test this, and you and Thomas >>>> ??? sign-off on the updates, I'll get this pushed. But as I'm away >>>> ??? from Thursday it may not be until next week. >>>> >>>> ??? Thanks, >>>> ??? David >>>> ??? ----- >>>> >>>> >>>> >>>> ??? On 18/11/2017 1:40 AM, Daniel D. Daugherty wrote: >>>> >>>> ??????? On 11/14/17 5:39 AM, David Holmes wrote: >>>> >>>> ??????????? Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>> >>>> >>>> ??????????? webrev: >>>> http://cr.openjdk.java.net/~dholmes/8189170/webrev/ >>>> >>>> >>>> >>>> ??????? src/hotspot/os/aix/os_aix.cpp >>>> ??????? ???? No comments. >>>> >>>> ??????? src/hotspot/os/aix/os_aix.hpp >>>> ??????? ???? No comments. >>>> >>>> ??????? src/hotspot/os/bsd/os_bsd.cpp >>>> ??????? ???? L880: // thread in any special way - so just report >>>> 'false' >>>> ??????? ??? ? ?? Nit - needs a period after 'false' to make it a >>>> sentence. >>>> >>>> ??????? ???? L882: ??? return false; >>>> ??????? ???????? Nit - indent should be 2 spaces. >>>> >>>> ??????? src/hotspot/os/bsd/os_bsd.hpp >>>> ??????? ???? No comments. >>>> >>>> ??????? src/hotspot/os/linux/os_linux.cpp >>>> ??????? ???? L947: ? if (stack_size >= (size_t)(3 * page_size())) >>>> ??????? ???????? Nit - needs '{' and '}' on the if-statement body. >>>> >>>> ??????? ???? L3070: // always place it right after end of the mapped >>>> ??????? region >>>> ??????? ???????? Nit - needs a period after 'region' to make it a >>>> ??????? sentence. >>>> ??????? ???????? (Not your problem, but you did update the sentence...) >>>> >>>> ??????? src/hotspot/os/linux/os_linux.hpp >>>> ??????? ???? No comments. >>>> >>>> ??????? src/hotspot/os/solaris/os_solaris.cpp >>>> ??????? ???? No comments other than I learned something new about >>>> ??????? thr_main() :-) >>>> ??????? ???? Thanks for cleaning up that mess. >>>> >>>> ??????? src/hotspot/os/windows/os_windows.cpp >>>> ??????? ???? L452: // thread in any special way - so just report >>>> 'false' >>>> ??????? ???????? Nit - needs a period after 'false' to make it a >>>> sentence. >>>> >>>> ??????? ???? L454: ??? return false; >>>> ??????? ???????? Nit - indent should be 2 spaces. >>>> >>>> ??????? src/hotspot/share/gc/shared/threadLocalAllocBuffer.cpp >>>> ??????? ???? No comments. >>>> >>>> ??????? src/hotspot/share/runtime/globals.hpp >>>> ??????? ???? L1181: ? experimental(bool, >>>> ??????? DisablePrimordialThreadGuardPages, false, >>>> ??????? ???????? Experimental or diagnostic? >>>> >>>> ??????? src/hotspot/share/runtime/os.hpp >>>> ??????? ???? L450: ? // that loads/creates the JVM via JNI_CreateJavaVM >>>> ??????? ???????? Nit - needs a period after 'JNI_CreateJavaVM'. >>>> >>>> ??????? ???? L453: ? // launcher nevers uses the primordial thread as >>>> ??????? the main thread, but >>>> ??????? ???????? Typo - 'nevers' -> 'never' >>>> >>>> ??????? ???? L455: ? // have to special-case handling of the >>>> ??????? primordial thread if it attaches >>>> ??????? ???????? Typo - 'have to' -> 'have' >>>> >>>> ??????? src/hotspot/share/runtime/thread.cpp >>>> ??????? ???? I like the new log message. >>>> >>>> ??????? src/hotspot/share/runtime/thread.inline.hpp >>>> ??????? ???? L159: ? if (os::uses_stack_guard_pages() && >>>> ??????? ???? L160: ????? !(DisablePrimordialThreadGuardPages && >>>> ??????? os::is_primordial_thread())) { >>>> ??????? ???????? I'm not sure why, but I had to read these two lines >>>> ??????? several >>>> ??????? ???????? times before I convinced myself the logic is right. >>>> >>>> ??????? I'm good with the code changes. Your call on whether to fix the >>>> ??????? bits or not. >>>> >>>> ??????? Regarding this part of Thomas' review: >>>> >>>> ??????????????? Or, something like this: >>>> ??????????????? static bool is_primordial_thread(void) >>>> ??????????????? WINDOWS_ONLY({return false;}) BSD_ONLY({return false;}) >>>> >>>> >>>> ??????????? I can do that. I'll see if anyone else has any comments on >>>> ??????????? this. >>>> >>>> >>>> ??????? I'm assuming that Thomas means for the above to be at the >>>> ??????? platform independent layer (runtime/os.hpp?) so it is more >>>> ??????? obvious to the reader that the function doesn't do anything >>>> ??????? on those two platforms... I know we dislike such things at >>>> ??????? the platform independent layer, but it does feel appropriate >>>> ??????? to make this limitation more obvious. >>>> >>>> ??????? I would be okay if you made the change that Thomas is >>>> suggesting. >>>> >>>> ??????? Thumbs up! >>>> >>>> ??????? Dan >>>> >>>> >>>> >>>> >>>> ??????????? There are three parts to this which made it somewhat more >>>> ??????????? complicated than I had envisaged: >>>> >>>> ??????????? 1. Add the flag and disable the guards. >>>> >>>> ??????????? This is relatively straight-forward: >>>> >>>> ??????????? ?void JavaThread::create_stack_guard_pages() { >>>> ??????????? ?? if (!os::uses_stack_guard_pages() || >>>> ??????????? ?????? _stack_guard_state != stack_guard_unused || >>>> ??????????? ?????? (DisablePrimordialThreadGuardPages && >>>> ??????????? os::is_primordial_thread())) { >>>> ??????????? ?????? log_info(os, thread)("Stack guard page creation for >>>> ??????????? thread " >>>> ??????????? ??????????????????????????? UINTX_FORMAT " disabled", >>>> ??????????? os::current_thread_id()); >>>> ??????????? ???? return; >>>> >>>> ??????????? with a tweak in JavaThread::stack_guards_enabled as well. >>>> >>>> ??????????? 2. Promote os::XXX::is_initial_thread to >>>> ??????????? os::is_primordial_thread() >>>> >>>> ??????????? We have three platforms that already implement this >>>> ??????????? functionality: Linux, Solaris and AIX. For BSD/OSX and >>>> ??????????? Windows we don't attempt to do anything different for the >>>> ??????????? primordial thread, and we don't have a way to detect the >>>> ??????????? primordial thread - so is_primordial_thread() just returns >>>> ??????????? false. >>>> >>>> ??????????? I also tidied up some comments regarding terminology. >>>> >>>> ??????????? 3. Thomas noticed that we unconditionally subtract >>>> ??????????? 2*page_size() from the rlimit stack size, without checking >>>> ??????????? it was bigger than 2*page_size() to start with. I was >>>> ??????????? unsure how to tackle this. It's no good leaving stack_size >>>> ??????????? at zero so I opted to ensure we would have at least one >>>> ??????????? page left. Of course in such cases we would hit the bug in >>>> ??????????? libc (if it still exists, which seems unlikely but time >>>> ??????????? consuming to establish). >>>> >>>> ??????????? Testing: >>>> ??????????? ? - Tier 1 (JPRT) testing with a modified launcher that >>>> ??????????? uses the primordial thread >>>> ??????????? ??? - with guard pages enabled >>>> ??????????? ??? - with guard pages disabled >>>> ??????????? ? - Tier 1 (JPRT) normal JDK build (which should be >>>> ??????????? unaffected by this change) >>>> >>>> ??????????? The testing was primarily to ensure that disabling the >>>> ??????????? stack guards on the primordial thread wasn't totally >>>> ??????????? broken. A number of tests fail: >>>> ??????????? - setting -Xss32m fails for the primordial thread when >>>> ??????????? guards are enabled and the rlimit is 8m >>>> ??????????? - tests that would hit StackOverflowError crash the VM >>>> ??????????? when guards are disabled (as you would expect). >>>> ??????????? - runtime/execstack/Testexecstack.java crashes when the >>>> ??????????? guards are disabled >>>> >>>> ??????????? Thanks, >>>> ??????????? David >>>> >>>> >>>> >>> > From daniel.daugherty at oracle.com Tue Nov 21 01:50:29 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 20 Nov 2017 20:50:29 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> Message-ID: <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> On 11/20/17 12:51 AM, David Holmes wrote: > Hi Dan, > > Figured I'd better cast an eye over this again before it gets pushed :) Thanks for reviewing again!! > Only one thing (repeated many times) jumped out at me and apologies if > this has already been debated back and forth. I missed the exchange > where the for loop iteration was rewritten into this unusual form: > > for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = > jtiwh.next(); ) { > > On first reading I expect most people would mistake the control > expression for the iteration-expression due to the next(). I didn't > even know the control expression could introduce a local variable! I > find this really hard to read stylistically and far too cute/clever > for its own good. It also violates the style rules regarding implicit > boolean checks. > > I know Stefan proposed this as he did not like the alternate (in a few > places): > > +? { > +??? ThreadsListHandle tlh; > +??? JavaThreadIterator jti(tlh.list()); > +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { > ... > +? } > > but it seems to me the iterator code has evolved since then and only a > couple of places need a distinct TLH separate from the actual iterator > object. So I'm more in favour of the more classic for-loop, with the > iterator declared before the loop. Or even convert to while loops, as > this iterator seems more suited to that. Actually both Coleen and Stefan had a problem with how much additional code was added to support an iterator model. In some cases we went from 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For Coleen, she wanted 2 of the new lines compressed into 1 new line by using an iterator with an embedded ThreadsListHandle. Stefan wanted us to try and get back to 1-line where possible. The advantages of the new style are: - 1-line to 1-line in all but a few cases - automatic limited scope of the embedded ThreadsListHandle so we were ? able to remove extra braces that were added earlier in most cases The disadvantages of the new style are: - it is an unusual for-loop style (in HotSpot) - it breaks HotSpot's style rule about implied booleans Stefan K is pretty adamant that the original iterator version where we go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC code. The compiler guys haven't chimed in on this debate so I'm guessing they don't have a strong opinion either way. One thing that we don't want to do is have different styles for the different teams. So I had to make a judgement call on whether I thought Runtime could live with Stefan's idea. Originally I wasn't fond of it either and breaking the implied boolean style rule is a problem for me (I post that comment a lot in my code reviews). However, I have to say that I'm a lot happier about the compactness of the code and the fact that limited ThreadsListHandle scope comes for 'free' in most cases. We're planning to keep the new iterator style for the following reasons: - smaller change footprint - consistent style used for all of the teams - limited ThreadsListHandle scope comes for free in most cases We're hoping that you can live with this decision (for now) and maybe even grow to like it in the future. :-) > Other than that just a couple of comments on variable type choices, > and a few typos spotted below. Replies embedded below. > > Thanks, > David > --- > > src/hotspot/share/runtime/globals.hpp > > 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, > ??????? \ > 2477?????????? "Enable Thread SMR extra validity checks") \ > 2478 ??????? \ > 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, ??????? \ > 2480?????????? "Enable Thread SMR Statistics") ??????? \ > > Indent of strings is off by? 3 spaces. Fixed. > --- > > src/hotspot/share/runtime/handshake.cpp > > ?140?????? // There is assumption in code that Threads_lock should be > lock > ?200?????? // There is assumption in code that Threads_lock should be > lock > > lock -> locked Fixed. > 146???????? // handshake_process_by_vmthread will check the semaphore > for us again > > Needs period at end. Fixed. > 148???????? // should be okey since the new thread will not have an > operation. > > okey -> okay Fixed. > 151???????? // We can't warn here is since the thread do > cancel_handshake after it have been removed > > I think: > > // We can't warn here since the thread does cancel_handshake after it > has been removed Fixed. > 152???????? // from ThreadsList. So we should just keep looping here > until while below return negative. > > from -> from the > > Needs period at end. Fixed both. > > ?204???????????? // A new thread on the ThreadsList will not have an > operation. > > Change period to comma, and ... Fixed. > 205???????????? // Hence is skipped in handshake_process_by_vmthread. > > Hence is -> hence it is Fixed. > --- > > src/hotspot/share/runtime/thread.cpp > > 1482???? // dcubed - Looks like the Threads_lock is for stable access > 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. > > Does this comment need revisiting? We've been trying to decide what to do about this comment. We'll be filing a follow up bug for the Compiler team to take another look at how _jvmci_old_thread_counters and _jvmci_counters are protected. Threads_lock is a big hammer and there should be a less expensive solution. Once that bug gets filed, this comment can go away. > 3451 volatile jint ... > > Why are you using jint for field types here? (Surprised Coleen hasn't > spotted them! ;-) ). volatile jint???????? Threads::_smr_delete_notify = 0; volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; volatile jint???????? Threads::_smr_deleted_thread_times = 0; : volatile jint???????? Threads::_smr_tlh_cnt = 0; volatile jint???????? Threads::_smr_tlh_time_max = 0; volatile jint???????? Threads::_smr_tlh_times = 0; _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock notify() operation is needed. It counts up and down and should be a fairly small number. _smr_deleted_thread_cnt counts the number of Threads that have been deleted over the life of the VM. It's an always increasing number so it's size depends on how long the VM has been running. _smr_deleted_thread_time_max is the maximum time in millis it has taken to delete a thread. This field was originally a jlong, but IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have just been Atomic::add() that was not happy) _smr_deleted_thread_times accumulates the time in millis that it has taken to delete threads. It's an always increasing number so it's size depends on how long the VM has been running. This field was originally a jlong, but IIRC the Atomic::add() code was not happy on ARM... (might have just been Atomic::cmpxchg() that was not happy) _smr_tlh_cnt counts the number of ThreadsListHandles that have been deleted over the life of the VM. It's an always increasing number so it's size depends on how long the VM has been running. _smr_tlh_time_max is the maximum time in millis it has taken to delete a ThreadsListHandle. This field was originally a jlong, but IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have just been Atomic::add() that was not happy) _smr_tlh_times? accumulates the time in millis that it has taken to delete ThreadsListHandles. It's an always increasing number so it's size depends on how long the VM has been running.? This field was originally a jlong, but IIRC the Atomic::add() code was not happy on ARM... (might have just been Atomic::cmpxchg() that was not happy) It just dawned on me that I'm not sure whether you think the 'jint' fields should be larger or smaller or the same size but a different type... :-) The fact that I had to write so much to explain what these fields are for and how they're used indicates I need more comments. See below... > 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; > 3457 long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; > > long? If you really want 64-bit counters here you won't get them on > Windows with that declaration. If you really want variable sized > counters I suggest uintptr_t; otherwise uint64_t. long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that have been allocated over the life of the VM. It's an always increasing number so it's size depends on how long the VM has been running and how many Threads have been created and destroyed. _smr_java_thread_list_free_cnt counts the number of ThreadsLists that have been freed over the life of the VM. It's an always increasing number so it's size depends on how long the VM has been running and how many Threads have been created and destroyed. I can't remember why we chose 'long' and I agree it's not a good choice for Win*. Okay so it dawns on me that we haven't written down a description for the various new '_cnt', '_max' and '_times" fields so I'm adding these comments to thread.hpp: ?private: ? // Safe Memory Reclamation (SMR) support: ? static Monitor*????????????? _smr_delete_lock; ? // The '_cnt', '_max' and '_times" fields are enabled via ? // -XX:+EnableThreadSMRStatistics: ?????????????????????????????? // # of parallel threads in _smr_delete_lock->wait(). ? static uint????????????????? _smr_delete_lock_wait_cnt; ?????????????????????????????? // Max # of parallel threads in _smr_delete_lock->wait(). ? static uint????????????????? _smr_delete_lock_wait_max; ?????????????????????????????? // Flag to indicate when an _smr_delete_lock->notify() is needed. ? static volatile uint???????? _smr_delete_notify; ?????????????????????????????? // # of threads deleted over VM lifetime. ? static volatile uint???????? _smr_deleted_thread_cnt; ?????????????????????????????? // Max time in millis to delete a thread. ? static volatile uint???????? _smr_deleted_thread_time_max; ?????????????????????????????? // Cumulative time in millis to delete threads. ? static volatile uint???????? _smr_deleted_thread_times; ? static ThreadsList* volatile _smr_java_thread_list; ? static ThreadsList*????????? get_smr_java_thread_list(); ? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); ?????????????????????????????? // # of ThreadsLists allocated over VM lifetime. ? static uint64_t????????????? _smr_java_thread_list_alloc_cnt; ?????????????????????????????? // # of ThreadsLists freed over VM lifetime. ? static uint64_t????????????? _smr_java_thread_list_free_cnt; ?????????????????????????????? // Max size ThreadsList allocated. ? static uint????????????????? _smr_java_thread_list_max; ?????????????????????????????? // Max # of nested ThreadsLists for a thread. ? static uint????????????????? _smr_nested_thread_list_max; ?????????????????????????????? // # of ThreadsListHandles deleted over VM lifetime. ? static volatile uint???????? _smr_tlh_cnt; ?????????????????????????????? // Max time in millis to delete a ThreadsListHandle. ? static volatile jint???????? _smr_tlh_time_max; ?????????????????????????????? // Cumulative time in millis to delete ThreadsListHandles. ? static volatile jint???????? _smr_tlh_times; ? static ThreadsList*????????? _smr_to_delete_list; ?????????????????????????????? // # of parallel ThreadsLists on the to-delete list. ? static uint????????????????? _smr_to_delete_list_cnt; ?????????????????????????????? // Max # of parallel ThreadsLists on the to-delete list. ? static uint????????????????? _smr_to_delete_list_max; I've also gone through all the new '_cnt', '_max' and '_times" fields in thread.cpp and added an impl note to explain the choice of type: // Safe Memory Reclamation (SMR) support: Monitor*????????????? Threads::_smr_delete_lock = ????????????????????????? new Monitor(Monitor::special, "smr_delete_lock", ????????????????????????????????????? false /* allow_vm_block */, Monitor::_safepoint_check_never); // The '_cnt', '_max' and '_times" fields are enabled via // -XX:+EnableThreadSMRStatistics: ????????????????????? // # of parallel threads in _smr_delete_lock->wait(). ????????????????????? // Impl note: Hard to imagine > 64K waiting threads so ????????????????????? // this could be 16-bit, but there is no nice 16-bit ????????????????????? // _FORMAT support. uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; ????????????????????? // Max # of parallel threads in _smr_delete_lock->wait(). ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. uint????????????????? Threads::_smr_delete_lock_wait_max = 0; ????????????????????? // Flag to indicate when an _smr_delete_lock->notify() is needed. ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. volatile uint???????? Threads::_smr_delete_notify = 0; ????????????????????? // # of threads deleted over VM lifetime. ????????????????????? // Impl note: Atomically incremented over VM lifetime so ????????????????????? // use unsigned for more range. Unsigned 64-bit would ????????????????????? // be more future proof, but 64-bit atomic inc isn't ????????????????????? // available everywhere (or is it?). volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; ????????????????????? // Max time in millis to delete a thread. ????????????????????? // Impl note: 16-bit might be too small on an overloaded ????????????????????? // machine. Use unsigned since this is a time value. Set ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; ????????????????????? // Cumulative time in millis to delete threads. ????????????????????? // Impl note: Atomically added to over VM lifetime so use ????????????????????? // unsigned for more range. Unsigned 64-bit would be more ????????????????????? // future proof, but 64-bit atomic inc isn't available ????????????????????? // everywhere (or is it?). volatile uint???????? Threads::_smr_deleted_thread_times = 0; ThreadsList* volatile Threads::_smr_java_thread_list = new ThreadsList(0); ????????????????????? // # of ThreadsLists allocated over VM lifetime. ????????????????????? // Impl note: We allocate a new ThreadsList for every ????????????????????? // thread create and every thread delete so we need a ????????????????????? // bigger type than the _smr_deleted_thread_cnt field. uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; ????????????????????? // # of ThreadsLists freed over VM lifetime. ????????????????????? // Impl note: See _smr_java_thread_list_alloc_cnt note. uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; ????????????????????? // Max size ThreadsList allocated. ????????????????????? // Impl note: Max # of threads alive at one time should ????????????????????? // fit in unsigned 32-bit. uint????????????????? Threads::_smr_java_thread_list_max = 0; ????????????????????? // Max # of nested ThreadsLists for a thread. ????????????????????? // Impl note: Hard to imagine > 64K nested ThreadsLists ????????????????????? // so his could be 16-bit, but there is no nice 16-bit ????????????????????? // _FORMAT support. uint????????????????? Threads::_smr_nested_thread_list_max = 0; ????????????????????? // # of ThreadsListHandles deleted over VM lifetime. ????????????????????? // Impl note: Atomically incremented over VM lifetime so ????????????????????? // use unsigned for more range. There will be fewer ????????????????????? // ThreadsListHandles than threads so unsigned 32-bit ????????????????????? // should be fine. volatile uint???????? Threads::_smr_tlh_cnt = 0; ????????????????????? // Max time in millis to delete a ThreadsListHandle. ????????????????????? // Impl note: 16-bit might be too small on an overloaded ????????????????????? // machine. Use unsigned since this is a time value. Set ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. volatile uint???????? Threads::_smr_tlh_time_max = 0; ????????????????????? // Cumulative time in millis to delete ThreadsListHandles. ????????????????????? // Impl note: Atomically added to over VM lifetime so use ????????????????????? // unsigned for more range. Unsigned 64-bit would be more ????????????????????? // future proof, but 64-bit atomic inc isn't available ????????????????????? // everywhere (or is it?). volatile uint???????? Threads::_smr_tlh_times = 0; ThreadsList*????????? Threads::_smr_to_delete_list = NULL; ????????????????????? // # of parallel ThreadsLists on the to-delete list. ????????????????????? // Impl note: Hard to imagine > 64K ThreadsLists needing ????????????????????? // to be deleted so this could be 16-bit, but there is no ????????????????????? // nice 16-bit _FORMAT support. uint????????????????? Threads::_smr_to_delete_list_cnt = 0; ????????????????????? // Max # of parallel ThreadsLists on the to-delete list. ????????????????????? // Impl note: See _smr_to_delete_list_cnt note. uint????????????????? Threads::_smr_to_delete_list_max = 0; Yikes! With the additional comments, the additions to thread.hpp and thread.cpp grew by a bunch... I've done a test build build on my Mac with these changes and I'm about to kick off a Mach5 tier1 job... Dan > > ---- > > On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up >>> thru: >>> >>>> Changeset: fa736014cf28 >>>> Author:??? cjplummer >>>> Date:????? 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from >>>> within hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>> holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>> closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and >>>>>> Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done >>>>>> as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to >>>>>> use >>>>>> ??? JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>> describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>> quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>> have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>> tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> From david.holmes at oracle.com Tue Nov 21 02:13:40 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 21 Nov 2017 12:13:40 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> Message-ID: Hi Dan, Just to be clear my comment about use of jint's was not about expected size but the fact you were using a j-type instead of a C++ type when the field is not a Java field. (Coleen has been on a crusade to remove j-types from where they don't belong - and they were removed from the Atomic API as part of that recent templatization effort.) No further comments. :) Thanks, David On 21/11/2017 11:50 AM, Daniel D. Daugherty wrote: > On 11/20/17 12:51 AM, David Holmes wrote: >> Hi Dan, >> >> Figured I'd better cast an eye over this again before it gets pushed :) > > Thanks for reviewing again!! > > >> Only one thing (repeated many times) jumped out at me and apologies if >> this has already been debated back and forth. I missed the exchange >> where the for loop iteration was rewritten into this unusual form: >> >> for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = >> jtiwh.next(); ) { >> >> On first reading I expect most people would mistake the control >> expression for the iteration-expression due to the next(). I didn't >> even know the control expression could introduce a local variable! I >> find this really hard to read stylistically and far too cute/clever >> for its own good. It also violates the style rules regarding implicit >> boolean checks. >> >> I know Stefan proposed this as he did not like the alternate (in a few >> places): >> >> +? { >> +??? ThreadsListHandle tlh; >> +??? JavaThreadIterator jti(tlh.list()); >> +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { >> ... >> +? } >> >> but it seems to me the iterator code has evolved since then and only a >> couple of places need a distinct TLH separate from the actual iterator >> object. So I'm more in favour of the more classic for-loop, with the >> iterator declared before the loop. Or even convert to while loops, as >> this iterator seems more suited to that. > > Actually both Coleen and Stefan had a problem with how much additional > code was added to support an iterator model. In some cases we went from > 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For > Coleen, she wanted 2 of the new lines compressed into 1 new line by using > an iterator with an embedded ThreadsListHandle. Stefan wanted us to try > and get back to 1-line where possible. > > The advantages of the new style are: > > - 1-line to 1-line in all but a few cases > - automatic limited scope of the embedded ThreadsListHandle so we were > ? able to remove extra braces that were added earlier in most cases > > The disadvantages of the new style are: > > - it is an unusual for-loop style (in HotSpot) > - it breaks HotSpot's style rule about implied booleans > > > Stefan K is pretty adamant that the original iterator version where we > go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC > code. The compiler guys haven't chimed in on this debate so I'm guessing > they don't have a strong opinion either way. One thing that we don't want > to do is have different styles for the different teams. > > So I had to make a judgement call on whether I thought Runtime could live > with Stefan's idea. Originally I wasn't fond of it either and breaking the > implied boolean style rule is a problem for me (I post that comment a lot > in my code reviews). However, I have to say that I'm a lot happier about > the compactness of the code and the fact that limited ThreadsListHandle > scope comes for 'free' in most cases. > > We're planning to keep the new iterator style for the following reasons: > > - smaller change footprint > - consistent style used for all of the teams > - limited ThreadsListHandle scope comes for free in most cases > > We're hoping that you can live with this decision (for now) and maybe > even grow to like it in the future. :-) > > >> Other than that just a couple of comments on variable type choices, >> and a few typos spotted below. > > Replies embedded below. > > >> >> Thanks, >> David >> --- >> >> src/hotspot/share/runtime/globals.hpp >> >> 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, >> ??????? \ >> 2477?????????? "Enable Thread SMR extra validity checks") \ >> 2478 ??????? \ >> 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, ??????? \ >> 2480?????????? "Enable Thread SMR Statistics") ??????? \ >> >> Indent of strings is off by? 3 spaces. > > Fixed. > > >> --- >> >> src/hotspot/share/runtime/handshake.cpp >> >> ?140?????? // There is assumption in code that Threads_lock should be >> lock >> ?200?????? // There is assumption in code that Threads_lock should be >> lock >> >> lock -> locked > > Fixed. > > >> 146???????? // handshake_process_by_vmthread will check the semaphore >> for us again >> >> Needs period at end. > > Fixed. > > >> 148???????? // should be okey since the new thread will not have an >> operation. >> >> okey -> okay > > Fixed. > > >> 151???????? // We can't warn here is since the thread do >> cancel_handshake after it have been removed >> >> I think: >> >> // We can't warn here since the thread does cancel_handshake after it >> has been removed > > Fixed. > > >> 152???????? // from ThreadsList. So we should just keep looping here >> until while below return negative. >> >> from -> from the >> >> Needs period at end. > > Fixed both. > > >> >> ?204???????????? // A new thread on the ThreadsList will not have an >> operation. >> >> Change period to comma, and ... > > Fixed. > > >> 205???????????? // Hence is skipped in handshake_process_by_vmthread. >> >> Hence is -> hence it is > > Fixed. > > >> --- >> >> src/hotspot/share/runtime/thread.cpp >> >> 1482???? // dcubed - Looks like the Threads_lock is for stable access >> 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. >> >> Does this comment need revisiting? > > We've been trying to decide what to do about this comment. We'll be > filing a follow up bug for the Compiler team to take another look at > how _jvmci_old_thread_counters and _jvmci_counters are protected. > Threads_lock is a big hammer and there should be a less expensive > solution. Once that bug gets filed, this comment can go away. > > >> 3451 volatile jint ... >> >> Why are you using jint for field types here? (Surprised Coleen hasn't >> spotted them! ;-) ). > > volatile jint???????? Threads::_smr_delete_notify = 0; > volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; > volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; > volatile jint???????? Threads::_smr_deleted_thread_times = 0; > : > volatile jint???????? Threads::_smr_tlh_cnt = 0; > volatile jint???????? Threads::_smr_tlh_time_max = 0; > volatile jint???????? Threads::_smr_tlh_times = 0; > > _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock > notify() operation is needed. It counts up and down and should be a fairly > small number. > > _smr_deleted_thread_cnt counts the number of Threads that have been > deleted over the life of the VM. It's an always increasing number so > it's size depends on how long the VM has been running. > > _smr_deleted_thread_time_max is the maximum time in millis it has > taken to delete a thread. This field was originally a jlong, but > IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have > just been Atomic::add() that was not happy) > > _smr_deleted_thread_times accumulates the time in millis that it > has taken to delete threads. It's an always increasing number so > it's size depends on how long the VM has been running. This field > was originally a jlong, but IIRC the Atomic::add() code was not > happy on ARM... (might have just been Atomic::cmpxchg() that was > not happy) > > _smr_tlh_cnt counts the number of ThreadsListHandles that have been > deleted over the life of the VM. It's an always increasing number so > it's size depends on how long the VM has been running. > > _smr_tlh_time_max is the maximum time in millis it has taken to > delete a ThreadsListHandle. This field was originally a jlong, but > IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have > just been Atomic::add() that was not happy) > > _smr_tlh_times? accumulates the time in millis that it has taken to > delete ThreadsListHandles. It's an always increasing number so > it's size depends on how long the VM has been running.? This field > was originally a jlong, but IIRC the Atomic::add() code was not > happy on ARM... (might have just been Atomic::cmpxchg() that was > not happy) > > > It just dawned on me that I'm not sure whether you think the 'jint' > fields should be larger or smaller or the same size but a different > type... :-) > > The fact that I had to write so much to explain what these fields > are for and how they're used indicates I need more comments. See > below... > > >> 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; >> 3457 long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; >> >> long? If you really want 64-bit counters here you won't get them on >> Windows with that declaration. If you really want variable sized >> counters I suggest uintptr_t; otherwise uint64_t. > > long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; > long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; > > _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that > have been allocated over the life of the VM. It's an always increasing > number so it's size depends on how long the VM has been running and how > many Threads have been created and destroyed. > > _smr_java_thread_list_free_cnt counts the number of ThreadsLists that > have been freed over the life of the VM. It's an always increasing > number so it's size depends on how long the VM has been running and how > many Threads have been created and destroyed. > > I can't remember why we chose 'long' and I agree it's not a good choice > for Win*. > > > Okay so it dawns on me that we haven't written down a description for > the various new '_cnt', '_max' and '_times" fields so I'm adding these > comments to thread.hpp: > > ?private: > ? // Safe Memory Reclamation (SMR) support: > ? static Monitor*????????????? _smr_delete_lock; > ? // The '_cnt', '_max' and '_times" fields are enabled via > ? // -XX:+EnableThreadSMRStatistics: > ?????????????????????????????? // # of parallel threads in > _smr_delete_lock->wait(). > ? static uint????????????????? _smr_delete_lock_wait_cnt; > ?????????????????????????????? // Max # of parallel threads in > _smr_delete_lock->wait(). > ? static uint????????????????? _smr_delete_lock_wait_max; > ?????????????????????????????? // Flag to indicate when an > _smr_delete_lock->notify() is needed. > ? static volatile uint???????? _smr_delete_notify; > ?????????????????????????????? // # of threads deleted over VM lifetime. > ? static volatile uint???????? _smr_deleted_thread_cnt; > ?????????????????????????????? // Max time in millis to delete a thread. > ? static volatile uint???????? _smr_deleted_thread_time_max; > ?????????????????????????????? // Cumulative time in millis to delete > threads. > ? static volatile uint???????? _smr_deleted_thread_times; > ? static ThreadsList* volatile _smr_java_thread_list; > ? static ThreadsList*????????? get_smr_java_thread_list(); > ? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); > ?????????????????????????????? // # of ThreadsLists allocated over VM > lifetime. > ? static uint64_t????????????? _smr_java_thread_list_alloc_cnt; > ?????????????????????????????? // # of ThreadsLists freed over VM > lifetime. > ? static uint64_t????????????? _smr_java_thread_list_free_cnt; > ?????????????????????????????? // Max size ThreadsList allocated. > ? static uint????????????????? _smr_java_thread_list_max; > ?????????????????????????????? // Max # of nested ThreadsLists for a > thread. > ? static uint????????????????? _smr_nested_thread_list_max; > ?????????????????????????????? // # of ThreadsListHandles deleted over > VM lifetime. > ? static volatile uint???????? _smr_tlh_cnt; > ?????????????????????????????? // Max time in millis to delete a > ThreadsListHandle. > ? static volatile jint???????? _smr_tlh_time_max; > ?????????????????????????????? // Cumulative time in millis to delete > ThreadsListHandles. > ? static volatile jint???????? _smr_tlh_times; > ? static ThreadsList*????????? _smr_to_delete_list; > ?????????????????????????????? // # of parallel ThreadsLists on the > to-delete list. > ? static uint????????????????? _smr_to_delete_list_cnt; > ?????????????????????????????? // Max # of parallel ThreadsLists on the > to-delete list. > ? static uint????????????????? _smr_to_delete_list_max; > > > I've also gone through all the new '_cnt', '_max' and '_times" fields > in thread.cpp and added an impl note to explain the choice of type: > > // Safe Memory Reclamation (SMR) support: > Monitor*????????????? Threads::_smr_delete_lock = > ????????????????????????? new Monitor(Monitor::special, "smr_delete_lock", > ????????????????????????????????????? false /* allow_vm_block */, > Monitor::_safepoint_check_never); > // The '_cnt', '_max' and '_times" fields are enabled via > // -XX:+EnableThreadSMRStatistics: > ????????????????????? // # of parallel threads in > _smr_delete_lock->wait(). > ????????????????????? // Impl note: Hard to imagine > 64K waiting > threads so > ????????????????????? // this could be 16-bit, but there is no nice 16-bit > ????????????????????? // _FORMAT support. > uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; > ????????????????????? // Max # of parallel threads in > _smr_delete_lock->wait(). > ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. > uint????????????????? Threads::_smr_delete_lock_wait_max = 0; > ????????????????????? // Flag to indicate when an > _smr_delete_lock->notify() is needed. > ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. > volatile uint???????? Threads::_smr_delete_notify = 0; > ????????????????????? // # of threads deleted over VM lifetime. > ????????????????????? // Impl note: Atomically incremented over VM > lifetime so > ????????????????????? // use unsigned for more range. Unsigned 64-bit > would > ????????????????????? // be more future proof, but 64-bit atomic inc isn't > ????????????????????? // available everywhere (or is it?). > volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; > ????????????????????? // Max time in millis to delete a thread. > ????????????????????? // Impl note: 16-bit might be too small on an > overloaded > ????????????????????? // machine. Use unsigned since this is a time > value. Set > ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. > volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; > ????????????????????? // Cumulative time in millis to delete threads. > ????????????????????? // Impl note: Atomically added to over VM > lifetime so use > ????????????????????? // unsigned for more range. Unsigned 64-bit would > be more > ????????????????????? // future proof, but 64-bit atomic inc isn't > available > ????????????????????? // everywhere (or is it?). > volatile uint???????? Threads::_smr_deleted_thread_times = 0; > ThreadsList* volatile Threads::_smr_java_thread_list = new ThreadsList(0); > ????????????????????? // # of ThreadsLists allocated over VM lifetime. > ????????????????????? // Impl note: We allocate a new ThreadsList for > every > ????????????????????? // thread create and every thread delete so we > need a > ????????????????????? // bigger type than the _smr_deleted_thread_cnt > field. > uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; > ????????????????????? // # of ThreadsLists freed over VM lifetime. > ????????????????????? // Impl note: See _smr_java_thread_list_alloc_cnt > note. > uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; > ????????????????????? // Max size ThreadsList allocated. > ????????????????????? // Impl note: Max # of threads alive at one time > should > ????????????????????? // fit in unsigned 32-bit. > uint????????????????? Threads::_smr_java_thread_list_max = 0; > ????????????????????? // Max # of nested ThreadsLists for a thread. > ????????????????????? // Impl note: Hard to imagine > 64K nested > ThreadsLists > ????????????????????? // so his could be 16-bit, but there is no nice > 16-bit > ????????????????????? // _FORMAT support. > uint????????????????? Threads::_smr_nested_thread_list_max = 0; > ????????????????????? // # of ThreadsListHandles deleted over VM lifetime. > ????????????????????? // Impl note: Atomically incremented over VM > lifetime so > ????????????????????? // use unsigned for more range. There will be fewer > ????????????????????? // ThreadsListHandles than threads so unsigned > 32-bit > ????????????????????? // should be fine. > volatile uint???????? Threads::_smr_tlh_cnt = 0; > ????????????????????? // Max time in millis to delete a ThreadsListHandle. > ????????????????????? // Impl note: 16-bit might be too small on an > overloaded > ????????????????????? // machine. Use unsigned since this is a time > value. Set > ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. > volatile uint???????? Threads::_smr_tlh_time_max = 0; > ????????????????????? // Cumulative time in millis to delete > ThreadsListHandles. > ????????????????????? // Impl note: Atomically added to over VM > lifetime so use > ????????????????????? // unsigned for more range. Unsigned 64-bit would > be more > ????????????????????? // future proof, but 64-bit atomic inc isn't > available > ????????????????????? // everywhere (or is it?). > volatile uint???????? Threads::_smr_tlh_times = 0; > ThreadsList*????????? Threads::_smr_to_delete_list = NULL; > ????????????????????? // # of parallel ThreadsLists on the to-delete list. > ????????????????????? // Impl note: Hard to imagine > 64K ThreadsLists > needing > ????????????????????? // to be deleted so this could be 16-bit, but > there is no > ????????????????????? // nice 16-bit _FORMAT support. > uint????????????????? Threads::_smr_to_delete_list_cnt = 0; > ????????????????????? // Max # of parallel ThreadsLists on the > to-delete list. > ????????????????????? // Impl note: See _smr_to_delete_list_cnt note. > uint????????????????? Threads::_smr_to_delete_list_max = 0; > > > Yikes! With the additional comments, the additions to thread.hpp and > thread.cpp grew by a bunch... I've done a test build build on my Mac > with these changes and I'm about to kick off a Mach5 tier1 job... > > Dan > > > >> >> ---- >> >> On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up >>>> thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from >>>>> within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>> holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>> closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K and >>>>>>> Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being done >>>>>>> as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to >>>>>>> use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>> describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>> quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>> have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>> tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> > From daniel.daugherty at oracle.com Tue Nov 21 03:51:16 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 20 Nov 2017 22:51:16 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> Message-ID: <0fc655c8-fffe-03cc-0f7b-a34a9893923d@oracle.com> On 11/20/17 9:13 PM, David Holmes wrote: > Hi Dan, > > Just to be clear my comment about use of jint's was not about expected > size but the fact you were using a j-type instead of a C++ type when > the field is not a Java field. (Coleen has been on a crusade to remove > j-types from where they don't belong - and they were removed from the > Atomic API as part of that recent templatization effort.) Thanks for making that clear. I think I've gotten rid of all the jint's at this point... > No further comments. :) Thanks. I'll be sending out more webrevs when I get through all the code review comments... Dan > > Thanks, > David > > On 21/11/2017 11:50 AM, Daniel D. Daugherty wrote: >> On 11/20/17 12:51 AM, David Holmes wrote: >>> Hi Dan, >>> >>> Figured I'd better cast an eye over this again before it gets pushed :) >> >> Thanks for reviewing again!! >> >> >>> Only one thing (repeated many times) jumped out at me and apologies >>> if this has already been debated back and forth. I missed the >>> exchange where the for loop iteration was rewritten into this >>> unusual form: >>> >>> for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = >>> jtiwh.next(); ) { >>> >>> On first reading I expect most people would mistake the control >>> expression for the iteration-expression due to the next(). I didn't >>> even know the control expression could introduce a local variable! I >>> find this really hard to read stylistically and far too cute/clever >>> for its own good. It also violates the style rules regarding >>> implicit boolean checks. >>> >>> I know Stefan proposed this as he did not like the alternate (in a >>> few places): >>> >>> +? { >>> +??? ThreadsListHandle tlh; >>> +??? JavaThreadIterator jti(tlh.list()); >>> +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { >>> ... >>> +? } >>> >>> but it seems to me the iterator code has evolved since then and only >>> a couple of places need a distinct TLH separate from the actual >>> iterator object. So I'm more in favour of the more classic for-loop, >>> with the iterator declared before the loop. Or even convert to while >>> loops, as this iterator seems more suited to that. >> >> Actually both Coleen and Stefan had a problem with how much additional >> code was added to support an iterator model. In some cases we went from >> 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For >> Coleen, she wanted 2 of the new lines compressed into 1 new line by >> using >> an iterator with an embedded ThreadsListHandle. Stefan wanted us to try >> and get back to 1-line where possible. >> >> The advantages of the new style are: >> >> - 1-line to 1-line in all but a few cases >> - automatic limited scope of the embedded ThreadsListHandle so we were >> ?? able to remove extra braces that were added earlier in most cases >> >> The disadvantages of the new style are: >> >> - it is an unusual for-loop style (in HotSpot) >> - it breaks HotSpot's style rule about implied booleans >> >> >> Stefan K is pretty adamant that the original iterator version where we >> go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC >> code. The compiler guys haven't chimed in on this debate so I'm guessing >> they don't have a strong opinion either way. One thing that we don't >> want >> to do is have different styles for the different teams. >> >> So I had to make a judgement call on whether I thought Runtime could >> live >> with Stefan's idea. Originally I wasn't fond of it either and >> breaking the >> implied boolean style rule is a problem for me (I post that comment a >> lot >> in my code reviews). However, I have to say that I'm a lot happier about >> the compactness of the code and the fact that limited ThreadsListHandle >> scope comes for 'free' in most cases. >> >> We're planning to keep the new iterator style for the following reasons: >> >> - smaller change footprint >> - consistent style used for all of the teams >> - limited ThreadsListHandle scope comes for free in most cases >> >> We're hoping that you can live with this decision (for now) and maybe >> even grow to like it in the future. :-) >> >> >>> Other than that just a couple of comments on variable type choices, >>> and a few typos spotted below. >> >> Replies embedded below. >> >> >>> >>> Thanks, >>> David >>> --- >>> >>> src/hotspot/share/runtime/globals.hpp >>> >>> 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, >>> ??????? \ >>> 2477?????????? "Enable Thread SMR extra validity checks") \ >>> 2478 ??????? \ >>> 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, ??????? \ >>> 2480?????????? "Enable Thread SMR Statistics") ??????? \ >>> >>> Indent of strings is off by? 3 spaces. >> >> Fixed. >> >> >>> --- >>> >>> src/hotspot/share/runtime/handshake.cpp >>> >>> ?140?????? // There is assumption in code that Threads_lock should >>> be lock >>> ?200?????? // There is assumption in code that Threads_lock should >>> be lock >>> >>> lock -> locked >> >> Fixed. >> >> >>> 146???????? // handshake_process_by_vmthread will check the >>> semaphore for us again >>> >>> Needs period at end. >> >> Fixed. >> >> >>> 148???????? // should be okey since the new thread will not have an >>> operation. >>> >>> okey -> okay >> >> Fixed. >> >> >>> 151???????? // We can't warn here is since the thread do >>> cancel_handshake after it have been removed >>> >>> I think: >>> >>> // We can't warn here since the thread does cancel_handshake after >>> it has been removed >> >> Fixed. >> >> >>> 152???????? // from ThreadsList. So we should just keep looping here >>> until while below return negative. >>> >>> from -> from the >>> >>> Needs period at end. >> >> Fixed both. >> >> >>> >>> ?204???????????? // A new thread on the ThreadsList will not have an >>> operation. >>> >>> Change period to comma, and ... >> >> Fixed. >> >> >>> 205???????????? // Hence is skipped in handshake_process_by_vmthread. >>> >>> Hence is -> hence it is >> >> Fixed. >> >> >>> --- >>> >>> src/hotspot/share/runtime/thread.cpp >>> >>> 1482???? // dcubed - Looks like the Threads_lock is for stable access >>> 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. >>> >>> Does this comment need revisiting? >> >> We've been trying to decide what to do about this comment. We'll be >> filing a follow up bug for the Compiler team to take another look at >> how _jvmci_old_thread_counters and _jvmci_counters are protected. >> Threads_lock is a big hammer and there should be a less expensive >> solution. Once that bug gets filed, this comment can go away. >> >> >>> 3451 volatile jint ... >>> >>> Why are you using jint for field types here? (Surprised Coleen >>> hasn't spotted them! ;-) ). >> >> volatile jint???????? Threads::_smr_delete_notify = 0; >> volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; >> volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; >> volatile jint???????? Threads::_smr_deleted_thread_times = 0; >> : >> volatile jint???????? Threads::_smr_tlh_cnt = 0; >> volatile jint???????? Threads::_smr_tlh_time_max = 0; >> volatile jint???????? Threads::_smr_tlh_times = 0; >> >> _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock >> notify() operation is needed. It counts up and down and should be a >> fairly >> small number. >> >> _smr_deleted_thread_cnt counts the number of Threads that have been >> deleted over the life of the VM. It's an always increasing number so >> it's size depends on how long the VM has been running. >> >> _smr_deleted_thread_time_max is the maximum time in millis it has >> taken to delete a thread. This field was originally a jlong, but >> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >> just been Atomic::add() that was not happy) >> >> _smr_deleted_thread_times accumulates the time in millis that it >> has taken to delete threads. It's an always increasing number so >> it's size depends on how long the VM has been running. This field >> was originally a jlong, but IIRC the Atomic::add() code was not >> happy on ARM... (might have just been Atomic::cmpxchg() that was >> not happy) >> >> _smr_tlh_cnt counts the number of ThreadsListHandles that have been >> deleted over the life of the VM. It's an always increasing number so >> it's size depends on how long the VM has been running. >> >> _smr_tlh_time_max is the maximum time in millis it has taken to >> delete a ThreadsListHandle. This field was originally a jlong, but >> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >> just been Atomic::add() that was not happy) >> >> _smr_tlh_times? accumulates the time in millis that it has taken to >> delete ThreadsListHandles. It's an always increasing number so >> it's size depends on how long the VM has been running.? This field >> was originally a jlong, but IIRC the Atomic::add() code was not >> happy on ARM... (might have just been Atomic::cmpxchg() that was >> not happy) >> >> >> It just dawned on me that I'm not sure whether you think the 'jint' >> fields should be larger or smaller or the same size but a different >> type... :-) >> >> The fact that I had to write so much to explain what these fields >> are for and how they're used indicates I need more comments. See >> below... >> >> >>> 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; >>> 3457 long Threads::_smr_java_thread_list_free_cnt = 0; >>> >>> long? If you really want 64-bit counters here you won't get them on >>> Windows with that declaration. If you really want variable sized >>> counters I suggest uintptr_t; otherwise uint64_t. >> >> long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >> long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; >> >> _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that >> have been allocated over the life of the VM. It's an always increasing >> number so it's size depends on how long the VM has been running and how >> many Threads have been created and destroyed. >> >> _smr_java_thread_list_free_cnt counts the number of ThreadsLists that >> have been freed over the life of the VM. It's an always increasing >> number so it's size depends on how long the VM has been running and how >> many Threads have been created and destroyed. >> >> I can't remember why we chose 'long' and I agree it's not a good choice >> for Win*. >> >> >> Okay so it dawns on me that we haven't written down a description for >> the various new '_cnt', '_max' and '_times" fields so I'm adding these >> comments to thread.hpp: >> >> ??private: >> ?? // Safe Memory Reclamation (SMR) support: >> ?? static Monitor*????????????? _smr_delete_lock; >> ?? // The '_cnt', '_max' and '_times" fields are enabled via >> ?? // -XX:+EnableThreadSMRStatistics: >> ??????????????????????????????? // # of parallel threads in >> _smr_delete_lock->wait(). >> ?? static uint????????????????? _smr_delete_lock_wait_cnt; >> ??????????????????????????????? // Max # of parallel threads in >> _smr_delete_lock->wait(). >> ?? static uint????????????????? _smr_delete_lock_wait_max; >> ??????????????????????????????? // Flag to indicate when an >> _smr_delete_lock->notify() is needed. >> ?? static volatile uint???????? _smr_delete_notify; >> ??????????????????????????????? // # of threads deleted over VM >> lifetime. >> ?? static volatile uint???????? _smr_deleted_thread_cnt; >> ??????????????????????????????? // Max time in millis to delete a >> thread. >> ?? static volatile uint???????? _smr_deleted_thread_time_max; >> ??????????????????????????????? // Cumulative time in millis to >> delete threads. >> ?? static volatile uint???????? _smr_deleted_thread_times; >> ?? static ThreadsList* volatile _smr_java_thread_list; >> ?? static ThreadsList*????????? get_smr_java_thread_list(); >> ?? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); >> ??????????????????????????????? // # of ThreadsLists allocated over >> VM lifetime. >> ?? static uint64_t????????????? _smr_java_thread_list_alloc_cnt; >> ??????????????????????????????? // # of ThreadsLists freed over VM >> lifetime. >> ?? static uint64_t????????????? _smr_java_thread_list_free_cnt; >> ??????????????????????????????? // Max size ThreadsList allocated. >> ?? static uint????????????????? _smr_java_thread_list_max; >> ??????????????????????????????? // Max # of nested ThreadsLists for a >> thread. >> ?? static uint????????????????? _smr_nested_thread_list_max; >> ??????????????????????????????? // # of ThreadsListHandles deleted >> over VM lifetime. >> ?? static volatile uint???????? _smr_tlh_cnt; >> ??????????????????????????????? // Max time in millis to delete a >> ThreadsListHandle. >> ?? static volatile jint???????? _smr_tlh_time_max; >> ??????????????????????????????? // Cumulative time in millis to >> delete ThreadsListHandles. >> ?? static volatile jint???????? _smr_tlh_times; >> ?? static ThreadsList*????????? _smr_to_delete_list; >> ??????????????????????????????? // # of parallel ThreadsLists on the >> to-delete list. >> ?? static uint????????????????? _smr_to_delete_list_cnt; >> ??????????????????????????????? // Max # of parallel ThreadsLists on >> the to-delete list. >> ?? static uint????????????????? _smr_to_delete_list_max; >> >> >> I've also gone through all the new '_cnt', '_max' and '_times" fields >> in thread.cpp and added an impl note to explain the choice of type: >> >> // Safe Memory Reclamation (SMR) support: >> Monitor*????????????? Threads::_smr_delete_lock = >> ?????????????????????????? new Monitor(Monitor::special, >> "smr_delete_lock", >> ?????????????????????????????????????? false /* allow_vm_block */, >> Monitor::_safepoint_check_never); >> // The '_cnt', '_max' and '_times" fields are enabled via >> // -XX:+EnableThreadSMRStatistics: >> ?????????????????????? // # of parallel threads in >> _smr_delete_lock->wait(). >> ?????????????????????? // Impl note: Hard to imagine > 64K waiting >> threads so >> ?????????????????????? // this could be 16-bit, but there is no nice >> 16-bit >> ?????????????????????? // _FORMAT support. >> uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; >> ?????????????????????? // Max # of parallel threads in >> _smr_delete_lock->wait(). >> ?????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. >> uint????????????????? Threads::_smr_delete_lock_wait_max = 0; >> ?????????????????????? // Flag to indicate when an >> _smr_delete_lock->notify() is needed. >> ?????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. >> volatile uint???????? Threads::_smr_delete_notify = 0; >> ?????????????????????? // # of threads deleted over VM lifetime. >> ?????????????????????? // Impl note: Atomically incremented over VM >> lifetime so >> ?????????????????????? // use unsigned for more range. Unsigned >> 64-bit would >> ?????????????????????? // be more future proof, but 64-bit atomic inc >> isn't >> ?????????????????????? // available everywhere (or is it?). >> volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; >> ?????????????????????? // Max time in millis to delete a thread. >> ?????????????????????? // Impl note: 16-bit might be too small on an >> overloaded >> ?????????????????????? // machine. Use unsigned since this is a time >> value. Set >> ?????????????????????? // via Atomic::cmpxchg() in a loop for >> correctness. >> volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; >> ?????????????????????? // Cumulative time in millis to delete threads. >> ?????????????????????? // Impl note: Atomically added to over VM >> lifetime so use >> ?????????????????????? // unsigned for more range. Unsigned 64-bit >> would be more >> ?????????????????????? // future proof, but 64-bit atomic inc isn't >> available >> ?????????????????????? // everywhere (or is it?). >> volatile uint???????? Threads::_smr_deleted_thread_times = 0; >> ThreadsList* volatile Threads::_smr_java_thread_list = new >> ThreadsList(0); >> ?????????????????????? // # of ThreadsLists allocated over VM lifetime. >> ?????????????????????? // Impl note: We allocate a new ThreadsList >> for every >> ?????????????????????? // thread create and every thread delete so we >> need a >> ?????????????????????? // bigger type than the >> _smr_deleted_thread_cnt field. >> uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >> ?????????????????????? // # of ThreadsLists freed over VM lifetime. >> ?????????????????????? // Impl note: See >> _smr_java_thread_list_alloc_cnt note. >> uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; >> ?????????????????????? // Max size ThreadsList allocated. >> ?????????????????????? // Impl note: Max # of threads alive at one >> time should >> ?????????????????????? // fit in unsigned 32-bit. >> uint????????????????? Threads::_smr_java_thread_list_max = 0; >> ?????????????????????? // Max # of nested ThreadsLists for a thread. >> ?????????????????????? // Impl note: Hard to imagine > 64K nested >> ThreadsLists >> ?????????????????????? // so his could be 16-bit, but there is no >> nice 16-bit >> ?????????????????????? // _FORMAT support. >> uint????????????????? Threads::_smr_nested_thread_list_max = 0; >> ?????????????????????? // # of ThreadsListHandles deleted over VM >> lifetime. >> ?????????????????????? // Impl note: Atomically incremented over VM >> lifetime so >> ?????????????????????? // use unsigned for more range. There will be >> fewer >> ?????????????????????? // ThreadsListHandles than threads so unsigned >> 32-bit >> ?????????????????????? // should be fine. >> volatile uint???????? Threads::_smr_tlh_cnt = 0; >> ?????????????????????? // Max time in millis to delete a >> ThreadsListHandle. >> ?????????????????????? // Impl note: 16-bit might be too small on an >> overloaded >> ?????????????????????? // machine. Use unsigned since this is a time >> value. Set >> ?????????????????????? // via Atomic::cmpxchg() in a loop for >> correctness. >> volatile uint???????? Threads::_smr_tlh_time_max = 0; >> ?????????????????????? // Cumulative time in millis to delete >> ThreadsListHandles. >> ?????????????????????? // Impl note: Atomically added to over VM >> lifetime so use >> ?????????????????????? // unsigned for more range. Unsigned 64-bit >> would be more >> ?????????????????????? // future proof, but 64-bit atomic inc isn't >> available >> ?????????????????????? // everywhere (or is it?). >> volatile uint???????? Threads::_smr_tlh_times = 0; >> ThreadsList*????????? Threads::_smr_to_delete_list = NULL; >> ?????????????????????? // # of parallel ThreadsLists on the to-delete >> list. >> ?????????????????????? // Impl note: Hard to imagine > 64K >> ThreadsLists needing >> ?????????????????????? // to be deleted so this could be 16-bit, but >> there is no >> ?????????????????????? // nice 16-bit _FORMAT support. >> uint????????????????? Threads::_smr_to_delete_list_cnt = 0; >> ?????????????????????? // Max # of parallel ThreadsLists on the >> to-delete list. >> ?????????????????????? // Impl note: See _smr_to_delete_list_cnt note. >> uint????????????????? Threads::_smr_to_delete_list_max = 0; >> >> >> Yikes! With the additional comments, the additions to thread.hpp and >> thread.cpp grew by a bunch... I've done a test build build on my Mac >> with these changes and I'm about to kick off a Mach5 tier1 job... >> >> Dan >> >> >> >>> >>> ---- >>> >>> On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>> test, and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are >>>>> this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final >>>>> review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>> code quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>> server, and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >> From jini.george at oracle.com Tue Nov 21 10:34:55 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 21 Nov 2017 16:04:55 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 Message-ID: Hello, Here's requesting reviews for some SA code cleanup. ID: https://bugs.openjdk.java.net/browse/JDK-8191324 Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html The changes here are primarily to: 1. Remove unused IA64 SA code. 2. Make changes to avoid error-prone redefinition of hotspot constants in SA Java code. Instead read it in through vmStructs and db.lookupIntConstant(). 3. Make variable name changes in SA to align with the hotspot names. The changes are straightforward. Thanks much, Jini. From magnus.ihse.bursie at oracle.com Tue Nov 21 11:13:12 2017 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 21 Nov 2017 12:13:12 +0100 Subject: RFR: JDK-8191203 Remove duplicated jimage.hpp Message-ID: I ran a duplicate name check on the source base. Then I discovered that there is a jimage.hpp in both src/hotspot/share/classfile/jimage.hpp and src/java.base/share/native/libjimage/jimage.hpp. They are identical (apart from copyright headers), and once again, we shouldn't have both. David Holmes suggested removing the hotspot version, so this I've done. I intend to push this to jdk/hs. Bug: https://bugs.openjdk.java.net/browse/JDK-8191203 WebRev: http://cr.openjdk.java.net/~ihse/JDK-8191203-remove-duplicated-jimage-hpp/webrev.01 /Magnus From david.holmes at oracle.com Tue Nov 21 11:44:38 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 21 Nov 2017 21:44:38 +1000 Subject: RFR: JDK-8191203 Remove duplicated jimage.hpp In-Reply-To: References: Message-ID: Hi Magnus, This seems fine to me. Do you know what the story is with the copyright header? Thanks, David On 21/11/2017 9:13 PM, Magnus Ihse Bursie wrote: > I ran a duplicate name check on the source base. Then I discovered that > there is a jimage.hpp in both src/hotspot/share/classfile/jimage.hpp and > src/java.base/share/native/libjimage/jimage.hpp. They are identical > (apart from copyright headers), and once again, we shouldn't have both. > David Holmes suggested removing the hotspot version, so this I've done. > > I intend to push this to jdk/hs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191203 > WebRev: > http://cr.openjdk.java.net/~ihse/JDK-8191203-remove-duplicated-jimage-hpp/webrev.01 > > > /Magnus From magnus.ihse.bursie at oracle.com Tue Nov 21 14:00:26 2017 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 21 Nov 2017 15:00:26 +0100 Subject: RFR: JDK-8191203 Remove duplicated jimage.hpp In-Reply-To: References: Message-ID: <8e65a735-7972-a5eb-94b0-10beee3f42e9@oracle.com> On 2017-11-21 12:44, David Holmes wrote: > Hi Magnus, > > This seems fine to me. > > Do you know what the story is with the copyright header? No, I don't. It was introduced in JDK-8149776, which has just a brief rationale: "Allow the jimage native code to be more easily re-used in JVM implementations which cannot accept code under the GPL." I do not know what prompted this. It seems reasonable to keep the more liberaly licensed version, though. (Which happens to coincide with the JDK version). /Magnus > > Thanks, > David > > On 21/11/2017 9:13 PM, Magnus Ihse Bursie wrote: >> I ran a duplicate name check on the source base. Then I discovered >> that there is a jimage.hpp in both >> src/hotspot/share/classfile/jimage.hpp and >> src/java.base/share/native/libjimage/jimage.hpp. They are identical >> (apart from copyright headers), and once again, we shouldn't have >> both. David Holmes suggested removing the hotspot version, so this >> I've done. >> >> I intend to push this to jdk/hs. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191203 >> WebRev: >> http://cr.openjdk.java.net/~ihse/JDK-8191203-remove-duplicated-jimage-hpp/webrev.01 >> >> >> /Magnus From james.laskey at oracle.com Tue Nov 21 14:11:18 2017 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 21 Nov 2017 10:11:18 -0400 Subject: RFR: JDK-8191203 Remove duplicated jimage.hpp In-Reply-To: <8e65a735-7972-a5eb-94b0-10beee3f42e9@oracle.com> References: <8e65a735-7972-a5eb-94b0-10beee3f42e9@oracle.com> Message-ID: <14AAFF1F-5DF7-4E39-8BBA-CCEC0B091C7C@oracle.com> The file was moved from hs to the jdk to simplify the API, not sure why it?s still there. A bulk merge issue maybe. I don?t recall clearly but I think the BSD license was a request from that other big company. Cheers, ? Jim > On Nov 21, 2017, at 10:00 AM, Magnus Ihse Bursie wrote: > > On 2017-11-21 12:44, David Holmes wrote: >> Hi Magnus, >> >> This seems fine to me. >> >> Do you know what the story is with the copyright header? > No, I don't. It was introduced in JDK-8149776, which has just a brief rationale: > "Allow the jimage native code to be more easily re-used in JVM implementations which cannot accept code under the GPL." I do not know what prompted this. > > It seems reasonable to keep the more liberaly licensed version, though. (Which happens to coincide with the JDK version). > > /Magnus >> >> Thanks, >> David >> >> On 21/11/2017 9:13 PM, Magnus Ihse Bursie wrote: >>> I ran a duplicate name check on the source base. Then I discovered that there is a jimage.hpp in both src/hotspot/share/classfile/jimage.hpp and src/java.base/share/native/libjimage/jimage.hpp. They are identical (apart from copyright headers), and once again, we shouldn't have both. David Holmes suggested removing the hotspot version, so this I've done. >>> >>> I intend to push this to jdk/hs. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191203 >>> WebRev: http://cr.openjdk.java.net/~ihse/JDK-8191203-remove-duplicated-jimage-hpp/webrev.01 >>> >>> /Magnus > From daniel.daugherty at oracle.com Tue Nov 21 14:46:10 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 09:46:10 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: <56ab19ad-58d7-f65a-b29b-8503a8f6f72a@oracle.com> Hi Robin W! Welcome to the Thread-SMR party!! On 11/20/17 4:48 AM, Robin Westberg wrote: > Hi Dan, > > Overall I must say this looks very nice, can?t wait to use it! (Also a disclaimer: not a reviewer, and have not looked at the gc or jmvti specific changes nor the tests (yet)). Code reviews are always welcome (OpenJDK role does not matter). > Didn?t spot any real issues, just a few small efficiency related things: > > src/hotspot/share/runtime/biasedLocking.cpp > > 217 for (JavaThread* cur_thread = Threads::first(); cur_thread != NULL; cur_thread = cur_thread->next()) { > 218 if (cur_thread == biased_thread) { > 219 thread_is_alive = true; > > This loop could be replaced with: > > ThreadsListHandle tlh; > thread_is_alive = tlh.includes(biased_thread); Nice catch! Fixed with your proposed change. The careful reader will notice that in revoke_bias() we are not creating a ThreadsListHandle that is in scope for the entire function and we are doing cached monitor info walks without an obvious ThreadsListHandle in place. revoke_bias() is called at a safepoint from most locations. The one call that is not made at a safepoint is from BiasedLocking::revoke_and_rebias() and it is made by a JavaThread that is revoking a bias on itself which is also safe. We should add an assertion to revoke_bias() that looks something like this: ? assert(requesting_thread == Thread::current() || ???????? SafepointSynchronize::is_at_safepoint(), ???????? "must be operating on self or at a safepoint."); but we'll do that in a separate follow up bug since that will require biased locking focused testing. > src/hotspot/share/runtime/memprofiler.cpp > > 55-56 grabs the Threads_lock, shouldn?t be needed anymore. Nice catch! Deleting lines 55-56. > src/hotspot/share/runtime/thread.inline.hpp > > 198 TerminatedTypes l_terminated = (TerminatedTypes) > 199 OrderAccess::load_acquire((volatile jint *) &_terminated); > 200 return check_is_terminated(_terminated); > > The variable used in the return statement was intended to be l_terminated I guess? Ouch! Looks like JavaThread::is_exiting() is right, but JavaThread::is_terminated() has been inefficient all this time. Fixed. > The following are more minor suggestions / comments: > > src/hotspot/share/runtime/thread.cpp > > 3432 // operations over all threads. It is protected by its own Mutex > 3433 // lock, which is also used in other contexts to protect thread > > Should this comment perhaps be revised to mention SMR? It definitely needs some updating... Here's a stab at it: // The Threads class links together all active threads, and provides // operations over all threads. It is protected by the Threads_lock, // which is also used in other global contexts like safepointing. // ThreadsListHandles are used to safely perform operations on one // or more threads without the risk of the thread exiting during the // operation. // // Note: The Threads_lock is currently more widely used than we // would like. We are actively migrating Threads_lock uses to other // mechanisms in order to reduce Threads_lock contention. > 4745 hash_table_size--; > 4746 hash_table_size |= hash_table_size >> 1; > ... > > This calculation is repeated around line 4922 as well, perhaps put it in a function? The hash_table_size parameter is currently unused. We were using a different hash table before that allowed the size to be set. Unfortunately, that hash table didn't support being freed so we switched to ResourceHashtable. We have a project note to come back and update the underlying hash table to work with dynamic table sizes, but Erik hasn't had the cycles to do it yet. > 4828 // If someone set a handshake on us just as we entered exit path, we simple cancel it. > > Perhaps something like ?.. entered the exit path, we simply cancel it.? I went with this: ? if (ThreadLocalHandshakes) { ??? // The thread is about to be deleted so cancel any handshake. ??? thread->cancel_handshake(); ? } > src/hotspot/share/runtime/thread.hpp > > 1179 bool on_thread_list() { return _on_thread_list; } > 1180 void set_on_thread_list() { _on_thread_list = true; } > 1181 > 1182 // thread has called JavaThread::exit() or is terminated > 1183 bool is_exiting() const; > 1184 // thread is terminated (no longer on the threads list); we compare > 1185 // against the two non-terminated values so that a freed JavaThread > 1186 // will also be considered terminated. > 1187 bool check_is_terminated(TerminatedTypes l_terminated) const { > 1188 return l_terminated != _not_terminated && l_terminated != _thread_exiting; > 1189 } > 1190 bool is_terminated(); > > Use of const is inconsistent here, it?s added to 1183, should it perhaps be added to 1179 and 1190 as well then? Fixed. > src/hotspot/share/runtime/vm_operations.hpp > > 406 DeadlockCycle* result() { return _deadlocks; }; > 407 VMOp_Type type() const { return VMOp_FindDeadlocks; } > > Whitespace only change that seems spurious. Agreed. Restored to the original. Thanks for the review! Dan > > Best regards, > Robin > >> On 19 Nov 2017, at 02:49, Daniel D. Daugherty wrote: >> >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up thru: >>> >>>> Changeset: fa736014cf28 >>>> Author: cjplummer >>>> Date: 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from within hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>>> JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>> >>>> >>> From coleen.phillimore at oracle.com Tue Nov 21 15:56:22 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 21 Nov 2017 10:56:22 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <0fc655c8-fffe-03cc-0f7b-a34a9893923d@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> <0fc655c8-fffe-03cc-0f7b-a34a9893923d@oracle.com> Message-ID: On 11/20/17 10:51 PM, Daniel D. Daugherty wrote: > On 11/20/17 9:13 PM, David Holmes wrote: >> Hi Dan, >> >> Just to be clear my comment about use of jint's was not about >> expected size but the fact you were using a j-type instead of a C++ >> type when the field is not a Java field. (Coleen has been on a >> crusade to remove j-types from where they don't belong - and they >> were removed from the Atomic API as part of that recent >> templatization effort.) > > Thanks for making that clear. I think I've gotten rid of all > the jint's at this point... Oh great, I had missed that.?? Thank you David for commenting on that. Coleen > > >> No further comments. :) > > Thanks. I'll be sending out more webrevs when I get through all > the code review comments... > > Dan > > >> >> Thanks, >> David >> >> On 21/11/2017 11:50 AM, Daniel D. Daugherty wrote: >>> On 11/20/17 12:51 AM, David Holmes wrote: >>>> Hi Dan, >>>> >>>> Figured I'd better cast an eye over this again before it gets >>>> pushed :) >>> >>> Thanks for reviewing again!! >>> >>> >>>> Only one thing (repeated many times) jumped out at me and apologies >>>> if this has already been debated back and forth. I missed the >>>> exchange where the for loop iteration was rewritten into this >>>> unusual form: >>>> >>>> for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = >>>> jtiwh.next(); ) { >>>> >>>> On first reading I expect most people would mistake the control >>>> expression for the iteration-expression due to the next(). I didn't >>>> even know the control expression could introduce a local variable! >>>> I find this really hard to read stylistically and far too >>>> cute/clever for its own good. It also violates the style rules >>>> regarding implicit boolean checks. >>>> >>>> I know Stefan proposed this as he did not like the alternate (in a >>>> few places): >>>> >>>> +? { >>>> +??? ThreadsListHandle tlh; >>>> +??? JavaThreadIterator jti(tlh.list()); >>>> +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { >>>> ... >>>> +? } >>>> >>>> but it seems to me the iterator code has evolved since then and >>>> only a couple of places need a distinct TLH separate from the >>>> actual iterator object. So I'm more in favour of the more classic >>>> for-loop, with the iterator declared before the loop. Or even >>>> convert to while loops, as this iterator seems more suited to that. >>> >>> Actually both Coleen and Stefan had a problem with how much additional >>> code was added to support an iterator model. In some cases we went from >>> 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For >>> Coleen, she wanted 2 of the new lines compressed into 1 new line by >>> using >>> an iterator with an embedded ThreadsListHandle. Stefan wanted us to try >>> and get back to 1-line where possible. >>> >>> The advantages of the new style are: >>> >>> - 1-line to 1-line in all but a few cases >>> - automatic limited scope of the embedded ThreadsListHandle so we were >>> ?? able to remove extra braces that were added earlier in most cases >>> >>> The disadvantages of the new style are: >>> >>> - it is an unusual for-loop style (in HotSpot) >>> - it breaks HotSpot's style rule about implied booleans >>> >>> >>> Stefan K is pretty adamant that the original iterator version where we >>> go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC >>> code. The compiler guys haven't chimed in on this debate so I'm >>> guessing >>> they don't have a strong opinion either way. One thing that we don't >>> want >>> to do is have different styles for the different teams. >>> >>> So I had to make a judgement call on whether I thought Runtime could >>> live >>> with Stefan's idea. Originally I wasn't fond of it either and >>> breaking the >>> implied boolean style rule is a problem for me (I post that comment >>> a lot >>> in my code reviews). However, I have to say that I'm a lot happier >>> about >>> the compactness of the code and the fact that limited ThreadsListHandle >>> scope comes for 'free' in most cases. >>> >>> We're planning to keep the new iterator style for the following >>> reasons: >>> >>> - smaller change footprint >>> - consistent style used for all of the teams >>> - limited ThreadsListHandle scope comes for free in most cases >>> >>> We're hoping that you can live with this decision (for now) and maybe >>> even grow to like it in the future. :-) >>> >>> >>>> Other than that just a couple of comments on variable type choices, >>>> and a few typos spotted below. >>> >>> Replies embedded below. >>> >>> >>>> >>>> Thanks, >>>> David >>>> --- >>>> >>>> src/hotspot/share/runtime/globals.hpp >>>> >>>> 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, >>>> ??????? \ >>>> 2477?????????? "Enable Thread SMR extra validity checks") \ >>>> 2478 ??????? \ >>>> 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, ??????? \ >>>> 2480?????????? "Enable Thread SMR Statistics") ??????? \ >>>> >>>> Indent of strings is off by? 3 spaces. >>> >>> Fixed. >>> >>> >>>> --- >>>> >>>> src/hotspot/share/runtime/handshake.cpp >>>> >>>> ?140?????? // There is assumption in code that Threads_lock should >>>> be lock >>>> ?200?????? // There is assumption in code that Threads_lock should >>>> be lock >>>> >>>> lock -> locked >>> >>> Fixed. >>> >>> >>>> 146???????? // handshake_process_by_vmthread will check the >>>> semaphore for us again >>>> >>>> Needs period at end. >>> >>> Fixed. >>> >>> >>>> 148???????? // should be okey since the new thread will not have an >>>> operation. >>>> >>>> okey -> okay >>> >>> Fixed. >>> >>> >>>> 151???????? // We can't warn here is since the thread do >>>> cancel_handshake after it have been removed >>>> >>>> I think: >>>> >>>> // We can't warn here since the thread does cancel_handshake after >>>> it has been removed >>> >>> Fixed. >>> >>> >>>> 152???????? // from ThreadsList. So we should just keep looping >>>> here until while below return negative. >>>> >>>> from -> from the >>>> >>>> Needs period at end. >>> >>> Fixed both. >>> >>> >>>> >>>> ?204???????????? // A new thread on the ThreadsList will not have >>>> an operation. >>>> >>>> Change period to comma, and ... >>> >>> Fixed. >>> >>> >>>> 205???????????? // Hence is skipped in handshake_process_by_vmthread. >>>> >>>> Hence is -> hence it is >>> >>> Fixed. >>> >>> >>>> --- >>>> >>>> src/hotspot/share/runtime/thread.cpp >>>> >>>> 1482???? // dcubed - Looks like the Threads_lock is for stable access >>>> 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. >>>> >>>> Does this comment need revisiting? >>> >>> We've been trying to decide what to do about this comment. We'll be >>> filing a follow up bug for the Compiler team to take another look at >>> how _jvmci_old_thread_counters and _jvmci_counters are protected. >>> Threads_lock is a big hammer and there should be a less expensive >>> solution. Once that bug gets filed, this comment can go away. >>> >>> >>>> 3451 volatile jint ... >>>> >>>> Why are you using jint for field types here? (Surprised Coleen >>>> hasn't spotted them! ;-) ). >>> >>> volatile jint???????? Threads::_smr_delete_notify = 0; >>> volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; >>> volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; >>> volatile jint???????? Threads::_smr_deleted_thread_times = 0; >>> : >>> volatile jint???????? Threads::_smr_tlh_cnt = 0; >>> volatile jint???????? Threads::_smr_tlh_time_max = 0; >>> volatile jint???????? Threads::_smr_tlh_times = 0; >>> >>> _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock >>> notify() operation is needed. It counts up and down and should be a >>> fairly >>> small number. >>> >>> _smr_deleted_thread_cnt counts the number of Threads that have been >>> deleted over the life of the VM. It's an always increasing number so >>> it's size depends on how long the VM has been running. >>> >>> _smr_deleted_thread_time_max is the maximum time in millis it has >>> taken to delete a thread. This field was originally a jlong, but >>> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >>> just been Atomic::add() that was not happy) >>> >>> _smr_deleted_thread_times accumulates the time in millis that it >>> has taken to delete threads. It's an always increasing number so >>> it's size depends on how long the VM has been running. This field >>> was originally a jlong, but IIRC the Atomic::add() code was not >>> happy on ARM... (might have just been Atomic::cmpxchg() that was >>> not happy) >>> >>> _smr_tlh_cnt counts the number of ThreadsListHandles that have been >>> deleted over the life of the VM. It's an always increasing number so >>> it's size depends on how long the VM has been running. >>> >>> _smr_tlh_time_max is the maximum time in millis it has taken to >>> delete a ThreadsListHandle. This field was originally a jlong, but >>> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >>> just been Atomic::add() that was not happy) >>> >>> _smr_tlh_times? accumulates the time in millis that it has taken to >>> delete ThreadsListHandles. It's an always increasing number so >>> it's size depends on how long the VM has been running.? This field >>> was originally a jlong, but IIRC the Atomic::add() code was not >>> happy on ARM... (might have just been Atomic::cmpxchg() that was >>> not happy) >>> >>> >>> It just dawned on me that I'm not sure whether you think the 'jint' >>> fields should be larger or smaller or the same size but a different >>> type... :-) >>> >>> The fact that I had to write so much to explain what these fields >>> are for and how they're used indicates I need more comments. See >>> below... >>> >>> >>>> 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; >>>> 3457 long Threads::_smr_java_thread_list_free_cnt = 0; >>>> >>>> long? If you really want 64-bit counters here you won't get them on >>>> Windows with that declaration. If you really want variable sized >>>> counters I suggest uintptr_t; otherwise uint64_t. >>> >>> long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >>> long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; >>> >>> _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that >>> have been allocated over the life of the VM. It's an always increasing >>> number so it's size depends on how long the VM has been running and how >>> many Threads have been created and destroyed. >>> >>> _smr_java_thread_list_free_cnt counts the number of ThreadsLists that >>> have been freed over the life of the VM. It's an always increasing >>> number so it's size depends on how long the VM has been running and how >>> many Threads have been created and destroyed. >>> >>> I can't remember why we chose 'long' and I agree it's not a good choice >>> for Win*. >>> >>> >>> Okay so it dawns on me that we haven't written down a description for >>> the various new '_cnt', '_max' and '_times" fields so I'm adding these >>> comments to thread.hpp: >>> >>> ??private: >>> ?? // Safe Memory Reclamation (SMR) support: >>> ?? static Monitor*????????????? _smr_delete_lock; >>> ?? // The '_cnt', '_max' and '_times" fields are enabled via >>> ?? // -XX:+EnableThreadSMRStatistics: >>> ??????????????????????????????? // # of parallel threads in >>> _smr_delete_lock->wait(). >>> ?? static uint????????????????? _smr_delete_lock_wait_cnt; >>> ??????????????????????????????? // Max # of parallel threads in >>> _smr_delete_lock->wait(). >>> ?? static uint????????????????? _smr_delete_lock_wait_max; >>> ??????????????????????????????? // Flag to indicate when an >>> _smr_delete_lock->notify() is needed. >>> ?? static volatile uint???????? _smr_delete_notify; >>> ??????????????????????????????? // # of threads deleted over VM >>> lifetime. >>> ?? static volatile uint???????? _smr_deleted_thread_cnt; >>> ??????????????????????????????? // Max time in millis to delete a >>> thread. >>> ?? static volatile uint???????? _smr_deleted_thread_time_max; >>> ??????????????????????????????? // Cumulative time in millis to >>> delete threads. >>> ?? static volatile uint???????? _smr_deleted_thread_times; >>> ?? static ThreadsList* volatile _smr_java_thread_list; >>> ?? static ThreadsList*????????? get_smr_java_thread_list(); >>> ?? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* >>> new_list); >>> ??????????????????????????????? // # of ThreadsLists allocated over >>> VM lifetime. >>> ?? static uint64_t _smr_java_thread_list_alloc_cnt; >>> ??????????????????????????????? // # of ThreadsLists freed over VM >>> lifetime. >>> ?? static uint64_t _smr_java_thread_list_free_cnt; >>> ??????????????????????????????? // Max size ThreadsList allocated. >>> ?? static uint????????????????? _smr_java_thread_list_max; >>> ??????????????????????????????? // Max # of nested ThreadsLists for >>> a thread. >>> ?? static uint????????????????? _smr_nested_thread_list_max; >>> ??????????????????????????????? // # of ThreadsListHandles deleted >>> over VM lifetime. >>> ?? static volatile uint???????? _smr_tlh_cnt; >>> ??????????????????????????????? // Max time in millis to delete a >>> ThreadsListHandle. >>> ?? static volatile jint???????? _smr_tlh_time_max; >>> ??????????????????????????????? // Cumulative time in millis to >>> delete ThreadsListHandles. >>> ?? static volatile jint???????? _smr_tlh_times; >>> ?? static ThreadsList*????????? _smr_to_delete_list; >>> ??????????????????????????????? // # of parallel ThreadsLists on the >>> to-delete list. >>> ?? static uint????????????????? _smr_to_delete_list_cnt; >>> ??????????????????????????????? // Max # of parallel ThreadsLists on >>> the to-delete list. >>> ?? static uint????????????????? _smr_to_delete_list_max; >>> >>> >>> I've also gone through all the new '_cnt', '_max' and '_times" fields >>> in thread.cpp and added an impl note to explain the choice of type: >>> >>> // Safe Memory Reclamation (SMR) support: >>> Monitor*????????????? Threads::_smr_delete_lock = >>> ?????????????????????????? new Monitor(Monitor::special, >>> "smr_delete_lock", >>> ?????????????????????????????????????? false /* allow_vm_block */, >>> Monitor::_safepoint_check_never); >>> // The '_cnt', '_max' and '_times" fields are enabled via >>> // -XX:+EnableThreadSMRStatistics: >>> ?????????????????????? // # of parallel threads in >>> _smr_delete_lock->wait(). >>> ?????????????????????? // Impl note: Hard to imagine > 64K waiting >>> threads so >>> ?????????????????????? // this could be 16-bit, but there is no nice >>> 16-bit >>> ?????????????????????? // _FORMAT support. >>> uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; >>> ?????????????????????? // Max # of parallel threads in >>> _smr_delete_lock->wait(). >>> ?????????????????????? // Impl note: See _smr_delete_lock_wait_cnt >>> note. >>> uint????????????????? Threads::_smr_delete_lock_wait_max = 0; >>> ?????????????????????? // Flag to indicate when an >>> _smr_delete_lock->notify() is needed. >>> ?????????????????????? // Impl note: See _smr_delete_lock_wait_cnt >>> note. >>> volatile uint???????? Threads::_smr_delete_notify = 0; >>> ?????????????????????? // # of threads deleted over VM lifetime. >>> ?????????????????????? // Impl note: Atomically incremented over VM >>> lifetime so >>> ?????????????????????? // use unsigned for more range. Unsigned >>> 64-bit would >>> ?????????????????????? // be more future proof, but 64-bit atomic >>> inc isn't >>> ?????????????????????? // available everywhere (or is it?). >>> volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; >>> ?????????????????????? // Max time in millis to delete a thread. >>> ?????????????????????? // Impl note: 16-bit might be too small on an >>> overloaded >>> ?????????????????????? // machine. Use unsigned since this is a time >>> value. Set >>> ?????????????????????? // via Atomic::cmpxchg() in a loop for >>> correctness. >>> volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; >>> ?????????????????????? // Cumulative time in millis to delete threads. >>> ?????????????????????? // Impl note: Atomically added to over VM >>> lifetime so use >>> ?????????????????????? // unsigned for more range. Unsigned 64-bit >>> would be more >>> ?????????????????????? // future proof, but 64-bit atomic inc isn't >>> available >>> ?????????????????????? // everywhere (or is it?). >>> volatile uint???????? Threads::_smr_deleted_thread_times = 0; >>> ThreadsList* volatile Threads::_smr_java_thread_list = new >>> ThreadsList(0); >>> ?????????????????????? // # of ThreadsLists allocated over VM lifetime. >>> ?????????????????????? // Impl note: We allocate a new ThreadsList >>> for every >>> ?????????????????????? // thread create and every thread delete so >>> we need a >>> ?????????????????????? // bigger type than the >>> _smr_deleted_thread_cnt field. >>> uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >>> ?????????????????????? // # of ThreadsLists freed over VM lifetime. >>> ?????????????????????? // Impl note: See >>> _smr_java_thread_list_alloc_cnt note. >>> uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; >>> ?????????????????????? // Max size ThreadsList allocated. >>> ?????????????????????? // Impl note: Max # of threads alive at one >>> time should >>> ?????????????????????? // fit in unsigned 32-bit. >>> uint????????????????? Threads::_smr_java_thread_list_max = 0; >>> ?????????????????????? // Max # of nested ThreadsLists for a thread. >>> ?????????????????????? // Impl note: Hard to imagine > 64K nested >>> ThreadsLists >>> ?????????????????????? // so his could be 16-bit, but there is no >>> nice 16-bit >>> ?????????????????????? // _FORMAT support. >>> uint????????????????? Threads::_smr_nested_thread_list_max = 0; >>> ?????????????????????? // # of ThreadsListHandles deleted over VM >>> lifetime. >>> ?????????????????????? // Impl note: Atomically incremented over VM >>> lifetime so >>> ?????????????????????? // use unsigned for more range. There will be >>> fewer >>> ?????????????????????? // ThreadsListHandles than threads so >>> unsigned 32-bit >>> ?????????????????????? // should be fine. >>> volatile uint???????? Threads::_smr_tlh_cnt = 0; >>> ?????????????????????? // Max time in millis to delete a >>> ThreadsListHandle. >>> ?????????????????????? // Impl note: 16-bit might be too small on an >>> overloaded >>> ?????????????????????? // machine. Use unsigned since this is a time >>> value. Set >>> ?????????????????????? // via Atomic::cmpxchg() in a loop for >>> correctness. >>> volatile uint???????? Threads::_smr_tlh_time_max = 0; >>> ?????????????????????? // Cumulative time in millis to delete >>> ThreadsListHandles. >>> ?????????????????????? // Impl note: Atomically added to over VM >>> lifetime so use >>> ?????????????????????? // unsigned for more range. Unsigned 64-bit >>> would be more >>> ?????????????????????? // future proof, but 64-bit atomic inc isn't >>> available >>> ?????????????????????? // everywhere (or is it?). >>> volatile uint???????? Threads::_smr_tlh_times = 0; >>> ThreadsList*????????? Threads::_smr_to_delete_list = NULL; >>> ?????????????????????? // # of parallel ThreadsLists on the >>> to-delete list. >>> ?????????????????????? // Impl note: Hard to imagine > 64K >>> ThreadsLists needing >>> ?????????????????????? // to be deleted so this could be 16-bit, but >>> there is no >>> ?????????????????????? // nice 16-bit _FORMAT support. >>> uint????????????????? Threads::_smr_to_delete_list_cnt = 0; >>> ?????????????????????? // Max # of parallel ThreadsLists on the >>> to-delete list. >>> ?????????????????????? // Impl note: See _smr_to_delete_list_cnt note. >>> uint????????????????? Threads::_smr_to_delete_list_max = 0; >>> >>> >>> Yikes! With the additional comments, the additions to thread.hpp and >>> thread.cpp grew by a bunch... I've done a test build build on my Mac >>> with these changes and I'm about to kick off a Mach5 tier1 job... >>> >>> Dan >>> >>> >>> >>>> >>>> ---- >>>> >>>> On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Testing of the last round of changes revealed a hang in one of the >>>>> new >>>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>>> test, and >>>>> added another TLH test for good measure. >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>>> >>>>> Here is the updated delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>>> >>>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>>> no unexpected failures. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Robbin rebased the project last night/this morning to merge with >>>>>> Thread >>>>>> Local Handshakes (TLH) and also picked up additional changesets >>>>>> up thru: >>>>>> >>>>>>> Changeset: fa736014cf28 >>>>>>> Author:??? cjplummer >>>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>>> >>>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>>> within hotspot source >>>>>>> Summary: added pns2() to debug.cpp >>>>>>> Reviewed-by: stuefe, gthornbr >>>>>> >>>>>> This is the first time we've rebased the project to bits that are >>>>>> this >>>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>>> think >>>>>> we're done with this project and are in the final >>>>>> review-change-retest >>>>>> cycle(s)... In other words, we're not planning on making any more >>>>>> major >>>>>> changes for this project... :-) >>>>>> >>>>>> *** Time for code reviewers to chime in on this thread! *** >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>>> >>>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>>> >>>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>>> (and the new baseline also)... >>>>>> >>>>>> We're expecting this round to be a little noisier than usual because >>>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>>> through Jesper's usual care-and-feeding of integration_blockers, >>>>>> etc. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>>> (Yes, we're playing chase-the-repo...) >>>>>>> >>>>>>> Here is the updated full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>>> >>>>>>> Unlike the previous rebase, there were no changes required to the >>>>>>> open code to get the rebased bits to build so there is no delta >>>>>>> webrev. >>>>>>> >>>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>>> Mach5 tier[1-5] test run... >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>>> >>>>>>>> Here are the updated webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>>> >>>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>>> holiday >>>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>>> closer >>>>>>>> look today. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>>> and Coleen) >>>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>>> done as a >>>>>>>>> standalone patch and corresponding webrevs: >>>>>>>>> >>>>>>>>> Here's the mq comment for the change: >>>>>>>>> >>>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>>> to use >>>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>>> >>>>>>>>> Here is the full webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>>> >>>>>>>>> And here is the delta webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Dan, Erik and Robbin >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>>> Greetings, >>>>>>>>>> >>>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>>> >>>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>>> >>>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>>> >>>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>>> describe >>>>>>>>>> and track the work on this project: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>>> code quotes >>>>>>>>>> in the PDF that are not present in the internal wiki. We >>>>>>>>>> don't have a >>>>>>>>>> solution for that problem yet. >>>>>>>>>> >>>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>>> >>>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>>> tier[2-5] >>>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>>> server, and >>>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>>> >>>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>>> >>>>>>>>>> Daniel Daugherty >>>>>>>>>> Erik Osterlund >>>>>>>>>> Robbin Ehn >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>> > From coleen.phillimore at oracle.com Tue Nov 21 16:14:36 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 21 Nov 2017 11:14:36 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> Message-ID: <9cc8b08b-f373-f493-4429-662d18ff81ad@oracle.com> On 11/20/17 8:50 PM, Daniel D. Daugherty wrote: > On 11/20/17 12:51 AM, David Holmes wrote: >> Hi Dan, >> >> Figured I'd better cast an eye over this again before it gets pushed :) > > Thanks for reviewing again!! > > >> Only one thing (repeated many times) jumped out at me and apologies >> if this has already been debated back and forth. I missed the >> exchange where the for loop iteration was rewritten into this unusual >> form: >> >> for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = >> jtiwh.next(); ) { >> >> On first reading I expect most people would mistake the control >> expression for the iteration-expression due to the next(). I didn't >> even know the control expression could introduce a local variable! I >> find this really hard to read stylistically and far too cute/clever >> for its own good. It also violates the style rules regarding implicit >> boolean checks. >> >> I know Stefan proposed this as he did not like the alternate (in a >> few places): >> >> +? { >> +??? ThreadsListHandle tlh; >> +??? JavaThreadIterator jti(tlh.list()); >> +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { >> ... >> +? } >> >> but it seems to me the iterator code has evolved since then and only >> a couple of places need a distinct TLH separate from the actual >> iterator object. So I'm more in favour of the more classic for-loop, >> with the iterator declared before the loop. Or even convert to while >> loops, as this iterator seems more suited to that. > > Actually both Coleen and Stefan had a problem with how much additional > code was added to support an iterator model. In some cases we went from > 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For > Coleen, she wanted 2 of the new lines compressed into 1 new line by using > an iterator with an embedded ThreadsListHandle. Stefan wanted us to try > and get back to 1-line where possible. > > The advantages of the new style are: > > - 1-line to 1-line in all but a few cases > - automatic limited scope of the embedded ThreadsListHandle so we were > ? able to remove extra braces that were added earlier in most cases > > The disadvantages of the new style are: > > - it is an unusual for-loop style (in HotSpot) > - it breaks HotSpot's style rule about implied booleans > > > Stefan K is pretty adamant that the original iterator version where we > go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC > code. The compiler guys haven't chimed in on this debate so I'm guessing > they don't have a strong opinion either way. One thing that we don't want > to do is have different styles for the different teams. > > So I had to make a judgement call on whether I thought Runtime could live > with Stefan's idea. Originally I wasn't fond of it either and breaking > the > implied boolean style rule is a problem for me (I post that comment a lot > in my code reviews). However, I have to say that I'm a lot happier about > the compactness of the code and the fact that limited ThreadsListHandle > scope comes for 'free' in most cases. > > We're planning to keep the new iterator style for the following reasons: > > - smaller change footprint > - consistent style used for all of the teams > - limited ThreadsListHandle scope comes for free in most cases > > We're hoping that you can live with this decision (for now) and maybe > even grow to like it in the future. :-) The limited ThreadsListHandle scope, since ThreadsListHandle registers and unregisters lists to SMR, seems to be worth the cost of looking at this strange code pattern.?? Thank you for the explanation. > > >> Other than that just a couple of comments on variable type choices, >> and a few typos spotted below. > > Replies embedded below. > > >> >> Thanks, >> David >> --- >> >> src/hotspot/share/runtime/globals.hpp >> >> 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, >> ??????? \ >> 2477?????????? "Enable Thread SMR extra validity checks") \ >> 2478 ??????? \ >> 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, \ >> 2480?????????? "Enable Thread SMR Statistics") ??????? \ >> >> Indent of strings is off by? 3 spaces. > > Fixed. > > >> --- >> >> src/hotspot/share/runtime/handshake.cpp >> >> ?140?????? // There is assumption in code that Threads_lock should be >> lock >> ?200?????? // There is assumption in code that Threads_lock should be >> lock >> >> lock -> locked > > Fixed. > > >> 146???????? // handshake_process_by_vmthread will check the semaphore >> for us again >> >> Needs period at end. > > Fixed. > > >> 148???????? // should be okey since the new thread will not have an >> operation. >> >> okey -> okay > > Fixed. > > >> 151???????? // We can't warn here is since the thread do >> cancel_handshake after it have been removed >> >> I think: >> >> // We can't warn here since the thread does cancel_handshake after it >> has been removed > > Fixed. > > >> 152???????? // from ThreadsList. So we should just keep looping here >> until while below return negative. >> >> from -> from the >> >> Needs period at end. > > Fixed both. > > >> >> ?204???????????? // A new thread on the ThreadsList will not have an >> operation. >> >> Change period to comma, and ... > > Fixed. > > >> 205???????????? // Hence is skipped in handshake_process_by_vmthread. >> >> Hence is -> hence it is > > Fixed. > > >> --- >> >> src/hotspot/share/runtime/thread.cpp >> >> 1482???? // dcubed - Looks like the Threads_lock is for stable access >> 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. >> >> Does this comment need revisiting? > > We've been trying to decide what to do about this comment. We'll be > filing a follow up bug for the Compiler team to take another look at > how _jvmci_old_thread_counters and _jvmci_counters are protected. > Threads_lock is a big hammer and there should be a less expensive > solution. Once that bug gets filed, this comment can go away. > > >> 3451 volatile jint ... >> >> Why are you using jint for field types here? (Surprised Coleen hasn't >> spotted them! ;-) ). Yes, it was the jtypes I would have objected to.? And long isn't a good choice on windows because it's only 32 bits there. > > volatile jint???????? Threads::_smr_delete_notify = 0; > volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; > volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; > volatile jint???????? Threads::_smr_deleted_thread_times = 0; > : > volatile jint???????? Threads::_smr_tlh_cnt = 0; > volatile jint???????? Threads::_smr_tlh_time_max = 0; > volatile jint???????? Threads::_smr_tlh_times = 0; > > _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock > notify() operation is needed. It counts up and down and should be a > fairly > small number. > > _smr_deleted_thread_cnt counts the number of Threads that have been > deleted over the life of the VM. It's an always increasing number so > it's size depends on how long the VM has been running. > > _smr_deleted_thread_time_max is the maximum time in millis it has > taken to delete a thread. This field was originally a jlong, but > IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have > just been Atomic::add() that was not happy) > > _smr_deleted_thread_times accumulates the time in millis that it > has taken to delete threads. It's an always increasing number so > it's size depends on how long the VM has been running. This field > was originally a jlong, but IIRC the Atomic::add() code was not > happy on ARM... (might have just been Atomic::cmpxchg() that was > not happy) > > _smr_tlh_cnt counts the number of ThreadsListHandles that have been > deleted over the life of the VM. It's an always increasing number so > it's size depends on how long the VM has been running. > > _smr_tlh_time_max is the maximum time in millis it has taken to > delete a ThreadsListHandle. This field was originally a jlong, but > IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have > just been Atomic::add() that was not happy) > > _smr_tlh_times? accumulates the time in millis that it has taken to > delete ThreadsListHandles. It's an always increasing number so > it's size depends on how long the VM has been running.? This field > was originally a jlong, but IIRC the Atomic::add() code was not > happy on ARM... (might have just been Atomic::cmpxchg() that was > not happy) > > > It just dawned on me that I'm not sure whether you think the 'jint' > fields should be larger or smaller or the same size but a different > type... :-) > > The fact that I had to write so much to explain what these fields > are for and how they're used indicates I need more comments. See > below... > > >> 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; >> 3457 long Threads::_smr_java_thread_list_free_cnt = 0; >> >> long? If you really want 64-bit counters here you won't get them on >> Windows with that declaration. If you really want variable sized >> counters I suggest uintptr_t; otherwise uint64_t. > > long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; > long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; > > _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that > have been allocated over the life of the VM. It's an always increasing > number so it's size depends on how long the VM has been running and how > many Threads have been created and destroyed. > > _smr_java_thread_list_free_cnt counts the number of ThreadsLists that > have been freed over the life of the VM. It's an always increasing > number so it's size depends on how long the VM has been running and how > many Threads have been created and destroyed. > > I can't remember why we chose 'long' and I agree it's not a good choice > for Win*. > > > Okay so it dawns on me that we haven't written down a description for > the various new '_cnt', '_max' and '_times" fields so I'm adding these > comments to thread.hpp: > With this comment format, it's really hard to pick out the name of the field.? Nobody uses 80 columns anymore.? Can you move the comments over some spaces so the field names are visible? > ?private: > ? // Safe Memory Reclamation (SMR) support: > ? static Monitor*????????????? _smr_delete_lock; > ? // The '_cnt', '_max' and '_times" fields are enabled via > ? // -XX:+EnableThreadSMRStatistics: > ?????????????????????????????? // # of parallel threads in > _smr_delete_lock->wait(). > ? static uint????????????????? _smr_delete_lock_wait_cnt; > ?????????????????????????????? // Max # of parallel threads in > _smr_delete_lock->wait(). > ? static uint????????????????? _smr_delete_lock_wait_max; > ?????????????????????????????? // Flag to indicate when an > _smr_delete_lock->notify() is needed. > ? static volatile uint???????? _smr_delete_notify; > ?????????????????????????????? // # of threads deleted over VM lifetime. > ? static volatile uint???????? _smr_deleted_thread_cnt; > ?????????????????????????????? // Max time in millis to delete a thread. > ? static volatile uint???????? _smr_deleted_thread_time_max; > ?????????????????????????????? // Cumulative time in millis to delete > threads. > ? static volatile uint???????? _smr_deleted_thread_times; > ? static ThreadsList* volatile _smr_java_thread_list; > ? static ThreadsList*????????? get_smr_java_thread_list(); > ? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); > ?????????????????????????????? // # of ThreadsLists allocated over VM > lifetime. > ? static uint64_t????????????? _smr_java_thread_list_alloc_cnt; > ?????????????????????????????? // # of ThreadsLists freed over VM > lifetime. > ? static uint64_t????????????? _smr_java_thread_list_free_cnt; > ?????????????????????????????? // Max size ThreadsList allocated. > ? static uint????????????????? _smr_java_thread_list_max; > ?????????????????????????????? // Max # of nested ThreadsLists for a > thread. > ? static uint????????????????? _smr_nested_thread_list_max; > ?????????????????????????????? // # of ThreadsListHandles deleted over > VM lifetime. > ? static volatile uint???????? _smr_tlh_cnt; > ?????????????????????????????? // Max time in millis to delete a > ThreadsListHandle. > ? static volatile jint???????? _smr_tlh_time_max; > ?????????????????????????????? // Cumulative time in millis to delete > ThreadsListHandles. > ? static volatile jint???????? _smr_tlh_times; > ? static ThreadsList*????????? _smr_to_delete_list; > ?????????????????????????????? // # of parallel ThreadsLists on the > to-delete list. > ? static uint????????????????? _smr_to_delete_list_cnt; > ?????????????????????????????? // Max # of parallel ThreadsLists on > the to-delete list. > ? static uint????????????????? _smr_to_delete_list_max; > > > I've also gone through all the new '_cnt', '_max' and '_times" fields > in thread.cpp and added an impl note to explain the choice of type: > Since this is in the cpp file and there is so much comment text, can you just make it in column 1 and leave a blank line after each field.? The indentation style is really hard to read and only useful if the comment is short. > // Safe Memory Reclamation (SMR) support: > Monitor*????????????? Threads::_smr_delete_lock = > ????????????????????????? new Monitor(Monitor::special, > "smr_delete_lock", > ????????????????????????????????????? false /* allow_vm_block */, > Monitor::_safepoint_check_never); > // The '_cnt', '_max' and '_times" fields are enabled via > // -XX:+EnableThreadSMRStatistics: > ????????????????????? // # of parallel threads in > _smr_delete_lock->wait(). > ????????????????????? // Impl note: Hard to imagine > 64K waiting > threads so > ????????????????????? // this could be 16-bit, but there is no nice > 16-bit > ????????????????????? // _FORMAT support. > uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; > ????????????????????? // Max # of parallel threads in > _smr_delete_lock->wait(). > ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. > uint????????????????? Threads::_smr_delete_lock_wait_max = 0; > ????????????????????? // Flag to indicate when an > _smr_delete_lock->notify() is needed. > ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. > volatile uint???????? Threads::_smr_delete_notify = 0; > ????????????????????? // # of threads deleted over VM lifetime. > ????????????????????? // Impl note: Atomically incremented over VM > lifetime so > ????????????????????? // use unsigned for more range. Unsigned 64-bit > would > ????????????????????? // be more future proof, but 64-bit atomic inc > isn't > ????????????????????? // available everywhere (or is it?). > volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; > ????????????????????? // Max time in millis to delete a thread. > ????????????????????? // Impl note: 16-bit might be too small on an > overloaded > ????????????????????? // machine. Use unsigned since this is a time > value. Set > ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. > volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; > ????????????????????? // Cumulative time in millis to delete threads. > ????????????????????? // Impl note: Atomically added to over VM > lifetime so use > ????????????????????? // unsigned for more range. Unsigned 64-bit > would be more > ????????????????????? // future proof, but 64-bit atomic inc isn't > available > ????????????????????? // everywhere (or is it?). > volatile uint???????? Threads::_smr_deleted_thread_times = 0; > ThreadsList* volatile Threads::_smr_java_thread_list = new > ThreadsList(0); > ????????????????????? // # of ThreadsLists allocated over VM lifetime. > ????????????????????? // Impl note: We allocate a new ThreadsList for > every > ????????????????????? // thread create and every thread delete so we > need a > ????????????????????? // bigger type than the _smr_deleted_thread_cnt > field. > uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; > ????????????????????? // # of ThreadsLists freed over VM lifetime. > ????????????????????? // Impl note: See > _smr_java_thread_list_alloc_cnt note. > uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; > ????????????????????? // Max size ThreadsList allocated. > ????????????????????? // Impl note: Max # of threads alive at one time > should > ????????????????????? // fit in unsigned 32-bit. > uint????????????????? Threads::_smr_java_thread_list_max = 0; > ????????????????????? // Max # of nested ThreadsLists for a thread. > ????????????????????? // Impl note: Hard to imagine > 64K nested > ThreadsLists > ????????????????????? // so his could be 16-bit, but there is no nice > 16-bit > ????????????????????? // _FORMAT support. > uint????????????????? Threads::_smr_nested_thread_list_max = 0; > ????????????????????? // # of ThreadsListHandles deleted over VM > lifetime. > ????????????????????? // Impl note: Atomically incremented over VM > lifetime so > ????????????????????? // use unsigned for more range. There will be fewer > ????????????????????? // ThreadsListHandles than threads so unsigned > 32-bit > ????????????????????? // should be fine. > volatile uint???????? Threads::_smr_tlh_cnt = 0; > ????????????????????? // Max time in millis to delete a > ThreadsListHandle. > ????????????????????? // Impl note: 16-bit might be too small on an > overloaded > ????????????????????? // machine. Use unsigned since this is a time > value. Set > ????????????????????? // via Atomic::cmpxchg() in a loop for correctness. > volatile uint???????? Threads::_smr_tlh_time_max = 0; > ????????????????????? // Cumulative time in millis to delete > ThreadsListHandles. > ????????????????????? // Impl note: Atomically added to over VM > lifetime so use > ????????????????????? // unsigned for more range. Unsigned 64-bit > would be more > ????????????????????? // future proof, but 64-bit atomic inc isn't > available > ????????????????????? // everywhere (or is it?). > volatile uint???????? Threads::_smr_tlh_times = 0; > ThreadsList*????????? Threads::_smr_to_delete_list = NULL; > ????????????????????? // # of parallel ThreadsLists on the to-delete > list. > ????????????????????? // Impl note: Hard to imagine > 64K ThreadsLists > needing > ????????????????????? // to be deleted so this could be 16-bit, but > there is no > ????????????????????? // nice 16-bit _FORMAT support. > uint????????????????? Threads::_smr_to_delete_list_cnt = 0; > ????????????????????? // Max # of parallel ThreadsLists on the > to-delete list. > ????????????????????? // Impl note: See _smr_to_delete_list_cnt note. > uint????????????????? Threads::_smr_to_delete_list_max = 0; > > > Yikes! With the additional comments, the additions to thread.hpp and > thread.cpp grew by a bunch... I've done a test build build on my Mac > with these changes and I'm about to kick off a Mach5 tier1 job... Another reason why I agreed with some earlier comment that this should be in a new file because it's a new thing. I was somewhat surprised that it's not in threadSMR.{hpp,cpp}.?? This refactoring could be deferred but thread.cpp is too big! thanks, Coleen > > Dan > > > >> >> ---- >> >> On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, >>> and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with >>>> Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up >>>> thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from >>>>> within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we >>>> think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more >>>> major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>> holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>> closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>> and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>> done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>> to use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>> describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>> quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>> have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>> tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, >>>>>>>> and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> > From daniel.daugherty at oracle.com Tue Nov 21 16:28:56 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 11:28:56 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> Hi Coleen! Thanks for making time to review the Thread-SMR stuff again!! I have added back the other three OpenJDK aliases... This review is being done on _four_ different OpenJDK aliases. As always, replies are embedded below... On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/os/linux/os_linux.cpp.frames.html > > > I read David's comments about the threads list iterator, and I was > going to say that it can be cleaned up later, as the bulk of the > change is the SMR part but this looks truly bizarre.?? It looks like > it shouldn't compile because 'jt' isn't in scope. > > Why isn't this the syntax: > > JavaThreadIteratorWithHandle jtiwh; > for (JavaThread* jt = jtiwh.first(); jt != NULL; jt = jtiwh.next()) { > } > > Which would do the obvious thing without anyone having to squint at > the code. See my reply to David's review for the more detailed answer. For the above syntax, we would need braces to limit the scope of the 'jtiwh' variable. With Stefan's propsal, you get limited scope on 'jtiwh' for "free". > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.hpp.html > > > As a hater of acronmys, can this file use "Safe Memory Reclaimation" I'm not sure what you mean by this. Do you mean rename the files? ? threadSMR.hpp -> threadSafeMemoryReclaimation.hpp ? threadSMR.cpp -> threadSafeMemoryReclaimation.cpp > and briefly describe the concept in the beginning of the header file, > so one knows why it's called threadSMR.hpp? And then this part of the sentence kind of indicates that you would be okay with the threadSMR.{c,h}pp names if a comment was added to the header file. Please clarify. > It doesn't need to be long and include why Threads list needs this > Sometimes we tell new people that the hotspot documentation is in the > header files. Yup. When I migrated stuff from thread.hpp and thread.cpp to threadSMR.hpp and threadSMR.cpp, I should have written a header comment... I did update a comment in thread.cpp based on Robin W's code review: > > src/hotspot/share/runtime/thread.cpp > > > > 3432 // operations over all threads.? It is protected by its own Mutex > > 3433 // lock, which is also used in other contexts to protect thread > > > > Should this comment perhaps be revised to mention SMR? > > It definitely needs some updating... Here's a stab at it: > > // The Threads class links together all active threads, and provides > // operations over all threads. It is protected by the Threads_lock, > // which is also used in other global contexts like safepointing. > // ThreadsListHandles are used to safely perform operations on one > // or more threads without the risk of the thread exiting during the > // operation. > // > // Note: The Threads_lock is currently more widely used than we > // would like. We are actively migrating Threads_lock uses to other > // mechanisms in order to reduce Threads_lock contention. I'll take a look at adding a header comment to threadSMR.hpp. > 186?? JavaThreadIteratorWithHandle() : _tlh(), _index(0) {} > > This _tlh() call should not be necessary.? The compiler should > generate this for you in the constructor. Deleted. > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html > > > ? 32 ThreadsList::ThreadsList(int entries) : _length(entries), > _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), > _next_list(NULL) { > > Seems like it should be mtThread rather than mtGC. Fixed. Definitely an artifact of Erik's original prototype when he extracted Thread-SMR from his GC work... Thanks for catching it. > Should > > ? 62?? if (EnableThreadSMRStatistics) { > > really be UL, ie: if (log_is_enabled(Info, thread, smr, statistics)) ? EnableThreadSMRStatistics is used in more places than UL code. We use it in Thread::print_*() stuff to control output of Thread-SMR statistics info in thread dumps and hs_err_pid file generation. Currently thread dump and hs_err_pid file output is not generated using UL (and probably can't be?) so we need an option to control the Thread-SMR statistics stuff in all places. > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java.html > > > Can you use for these tests instead (there were a couple): > > *@requires (vm.debug == true)* The test I cloned had this in it: ??? if (!Platform.isDebugBuild()) { ??????? // -XX:ErrorHandlerTest=N option requires debug bits. ??????? return; ??? } and you would like me to switch to the newer mechanism? I have updated the following tests: ? test/hotspot/jtreg/runtime/ErrorHandling/ErrorHandler.java test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java test/hotspot/jtreg/runtime/ErrorHandling/ThreadsListHandleInErrorHandlingTest.java > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/thread.cpp.udiff.html > > > +// thread, has been added the Threads list, the system is not at a > > Has "not" been added to the Threads list?? missing "not"? Nope. If the JavaThread has been added to the Threads list and it is not protected, then it is dangling. In other words, a published JavaThread (on the Threads list) has to be protected by either the system being at a safepoint or the JavaThread has to be on some threads's ThreadsList. > > + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); > > Can you add a comment about where this number came from? I'll have to get that from Erik... > I can't find the caller of threads_do_smr. I'm guessing that's used by the GC code that depends on Thread-SMR... > If these functions xchg_smr_thread_list, get_smr_java_thread_list, > inc_smr_deleted_thread_count are only used by thread.cpp, I think they > should go in that file and not in the .inline.hpp file to be included > and possibly called by other files.? I think they're private to class > Threads. I have a vague memory that some of the compilers don't do inlining when an "inline" function is in a .cpp. I believe we want these functions to be inlined for performance reasons. Erik should probably chime in here. > I don't have any in-depth comments.? This looks really good from my > day of reading it. Thanks for taking the time to review it again! Dan > > Thanks, > Coleen > > On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: > >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up >>> thru: >>> >>>> Changeset: fa736014cf28 >>>> Author:??? cjplummer >>>> Date:????? 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from >>>> within hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>> holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>> closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and >>>>>> Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done >>>>>> as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to >>>>>> use >>>>>> ??? JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>> describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>> quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>> have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>> tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> > From ioi.lam at oracle.com Tue Nov 21 16:36:47 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 21 Nov 2017 08:36:47 -0800 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <5A1335A6.2070509@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> <5A033E72.90800@oracle.com> <5A048F23.9030806@oracle.com> <5A1335A6.2070509@oracle.com> Message-ID: <57af485d-5127-9cd5-316f-c8873003983f@oracle.com> Looks good. Thanks! - Ioi On 11/20/17 12:05 PM, Calvin Cheung wrote: > I've had some off-list discussion with Ioi resulting in another update > to the webrev. > > - added relative path scenario to the test. Currently, this fix > doesn't handle relative path on windows yet. The following RFE has > been filed to cover long relative paths on windows: > https://bugs.openjdk.java.net/browse/JDK-8191521 > - renamed the test LongPath.java to LongBCP.java to better reflect > what is being tested; > - some comments update on os_windows.cpp > > updated webrev: > ??? http://cr.openjdk.java.net/~ccheung/8188122/webrev.05/ > > thanks, > Calvin > > On 11/9/17, 9:23 AM, Calvin Cheung wrote: >> Hi Thomas, >> >> On 11/8/17, 10:40 PM, Thomas St?fe wrote: >>> Hi Calvin, >>> >>> On Wed, Nov 8, 2017 at 6:27 PM, Calvin Cheung >>> > wrote: >>> >>> >>> >>> ??? On 11/7/17, 6:12 AM, Thomas St?fe wrote: >>>> ??? Hi Calvin, >>>> >>>> ??? On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung >>>> ??? > >>>> wrote: >>>> >>>> >>>> >>>> ??????? On 10/27/17, 12:55 AM, Thomas St?fe wrote: >>>>> ??????? Hi Calvin, >>>>> >>>>> ??????? On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >>>>> ??????? >>>> ??????? > wrote: >>>>> >>>>> ??????????? Hi Thomas, >>>>> >>>>> ??????????? On 10/25/17, 11:54 PM, Thomas St?fe wrote: >>>>> >>>>> ??????????????? Hi Calvin, >>>>> >>>>> ??????????????? this is a worthwhile addition, thank you for your >>>>> work! >>>>> >>>>> ??????????? Thanks for your review. >>>>> >>>>> >>>>> ??????? Thanks for your work :) >>>>> >>>>> ??????? First off, there is another issue in >>>>> ??????? file_attribute_data_to_stat(). We actually had this issue >>>>> ??????? before, but your work uncovered it: >>>>> >>>>> ??????? According to POSIX >>>>> (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html >>>>> >>>>> ) >>>>> ??????? and every unix manpage I looked at (solaris, linux, aix..), >>>>> ??????? st_ctime is not the file creation time but the last time >>>>> ??????? file status was changed. Only Microsoft gets this wrong in >>>>> ??????? their posix layer, its stat() function returns >>>>> (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx >>>>> ) >>>>> ??????? creation time in st_ctime. >>>>> >>>>> ??????? But as os::stat() is a platform independent layer, I'd say >>>>> ??????? we should behave the same on all platforms, and that would >>>>> ??????? be the Posix way. >>>>> >>>>> ??????? I did not find any code calling os::stat() and using >>>>> ??????? st_ctime, so this is still save to change for windows. >>>>> ??????? (Note that I found that perfMemory_xxx.cpp uses plain OS >>>>> ??????? ::stat and misuses ctime as "creation time" - I opened a >>>>> ??????? bug for that >>>>> ??????? (https://bugs.openjdk.java.net/browse/JDK-8190260 >>>>> - but >>>>> ??????? that does not affect us, as they do not call os::stat()). >>>>> >>>>> ??????? There is one more complication: in os::stat() we use plain >>>>> ??????? ::stat() in the single byte case.: >>>>> >>>>> ??????? *+ if (strlen(path) < MAX_PATH) {* >>>>> ??????? *+???? ret = ::stat(pathbuf, sbuf);* >>>>> ??????? *+?? } else {* >>>>> ??????? * >>>>> ??????? * >>>>> ??????? But ::stat() on Windows is broken, as I wrote above. We >>>>> ??????? should not use it, especially not if we change the meaing >>>>> ??????? of st_ctime in the double byte path. >>>>> >>>>> ??????? My suggestion would be to just always call >>>>> ??????? GetFileAttributes(), also for the single byte path. A very >>>>> ??????? simple solution would be to just always go the double byte >>>>> ??????? path with UNC translation, regardless of the path >>>>> ??????? length. Or, if you are worried about performance, for paths >>>>> ??????? shorter than MAX_PATH we use GetFileAttributesA(), for >>>>> ??????? longer paths GetFileAttributesW with UNC translation. In >>>>> ??????? both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you >>>>> ??????? can feed tp your? file_attribute_data_to_stat() routine and >>>>> ??????? have the struct stat you return be always consistent. >>>> ??????? I ran into an issue with the dwFileAttributes value for a >>>> ??????? jar file. On Windows Server O/S, the value is set to 0x2020 >>>> ??????? which is (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>>> ??????? FILE_ATTRIBUTE_ARCHIVE) but on a desktop O/S like Windows 7, >>>> ??????? it is set to 0x0020 which is just FILE_ATTRIBUTE_ARCHIVE. >>>> ??????? I've fixed it in file_attribute_data_to_stat(). >>>> >>>> >>>> ??? Oh.. :( good catch! But that opens a new can of worms I did not >>>> ??? see before: >>>> >>>> ??? FILE_ATTRIBUTE_NORMAL is documented as "A file that does not >>>> ??? have other attributes set. This attribute is valid only when >>>> ??? used alone." In addition to this flag, we have a multitude of >>>> ??? things like FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, >>>> ??? FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this >>>> ??? is a normal file in Unix terminology and where we would expect >>>> ??? os::stat to return S_IFREG, but where according to the msdn doc >>>> ??? FILE_ATTRIBUTE_NORMAL is not set. >>>> >>>> ??? Looking at what others do in this scenario (Looked at mingw >>>> ??? sources and at ReactOS - I am not posting any source code here >>>> ??? for legal reasons but feel free to look for yourself), the usual >>>> ??? way to translate WIN32_FILE_ATTRIBUTES to struct stat seems to be: >>>> ??? "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, >>>> ??? no dependency on FILE_ATTRIBUTE_NORMAL). >>> ??? This makes sense but I ran into similar problem as before - the >>> ??? dwFileAttributes has a different value on a windows server O/S vs >>> ??? desktop O/S. So I need to do the check as follows: >>> >>> ??? +?? // A directory has the dwFileAttributes value of 0x2010 >>> which is >>> ??? +?? // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>> FILE_ATTRIBUTE_ARCHIVE) >>> ??? +?? // on Windows Server O/S but the value is 0x0010 on Windows >>> desktop O/S >>> ??? +?? // such as Windows 7. >>> ??? +?? if ((file_data.dwFileAttributes& FILE_ATTRIBUTE_DIRECTORY) >>> != 0) { >>> ??? +???? sbuf->st_mode |= S_IFDIR; >>> ??? +?? } else { >>> ??? +???? sbuf->st_mode |= S_IFREG; >>> ??? +?? } >>> >>> I scratched my head a bit about the comment till I understood that >>> you mean "if FILE_ATTRIBUTE_DIRECTORY bit is set it is a directory >>> regardless of which other flags are set" instead of "if >>> flags==FILE_ATTRIBUTE_DIRECTORY it is a directory". Sure, I guess my >>> comment above was sloppy, but this was what I meant. I am not even >>> sure the comment is needed, this is quite self-explaining. >> I've noticed a typo in the above comment: >> +?? // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) >> >> FILE_ATTRIBUTE_ARCHIVE should be FILE_ATTRIBUTE_DIRECTORY >> >> I'll correct it before push. >> >>> >>> ??? updated webrev: >>> ??? http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ >>> >>> >>> >>> I am fine with all the changes now. Thank you for your work and >>> patience. >> Thanks for your discussions and review. >> >> thanks, >> Calvin >>> >>> Kind Regards, Thomas >>> >>> >>> >>> >>> ??? Diff from webrev.03: >>> >>> ??? < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >>> ??? 08:50:40.170786397 -0800 >>> ??? < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >>> ??? 08:50:39.803751877 -0800 >>> ??? < @@ -4060,41 +4060,119 @@ >>> ??? --- >>> ??? > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >>> ??? 09:40:13.657460425 -0700 >>> ??? > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >>> ??? 09:40:13.261423192 -0700 >>> ??? > @@ -4060,41 +4060,121 @@ >>> ??? 25,29c25 >>> ??? < +? // A directory has the dwFileAttributes value of 0x2010 >>> which is >>> ??? < +? // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>> FILE_ATTRIBUTE_ARCHIVE) >>> ??? < +? // on Windows Server O/S but the value is 0x0010 on Windows >>> ??? desktop O/S >>> ??? < +? // such as Windows 7. >>> ??? < +? if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) >>> ??? != 0) { >>> ??? --- >>> ??? > +? if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { >>> ??? 31c27,33 >>> ??? < +? } else { >>> ??? --- >>> ??? > +? } >>> ??? > +? if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || >>> ??? > +????? // an archive file such as a jar file has the >>> ??? dwFileAttributes value of >>> ??? > +????? // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>> ??? FILE_ATTRIBUTE_ARCHIVE) >>> ??? > +????? // on Windows Server O/S but the value is 0x0020 on >>> ??? > +????? // Windows desktop O/S such as Windows 7. >>> ??? > +????? ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) >>> ??? != 0)) { >>> >>>> >>>> ??? I wonder about other special cases which should work too: >>>> ??? - read-only files should be S_IFREG and !S_IWRITE, >>> ??? For a read-only system file under the user's home dir. >>> >>> ??? st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD >>> ??? dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | >>> ??? FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | >>> ??? FILE_ATTRIBUTE_READONLY) >>>> ??? read-only directories should be S_IFDIR and !S_IWRITE. >>> ??? I've tried the C:\progra~1 dir. >>> >>> ??? st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >>> ??? dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | >>> ??? FILE_ATTRIBUTE_READONLY) >>>> ??? - We should tread the device root ("C:\") as a directory (so, >>>> ??? os::stat("c:\") should return S_IFDIR). Does this work? >>> ??? This one works too. >>> >>> ??? st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >>> ??? dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | >>> ??? FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) >>>> >>>> ??????? I've also used GetTickCounts() to measure the performance of >>>> ??????? ::stat() vs GetFileAttributesA() plus >>>> ??????? file_attribute_data_to_stat(). There's no difference at the >>>> ??????? ms resolution. Using the high-resolution timestamp >>>> (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx) >>>> , >>>> ??????? it doesn't show much difference. >>>> >>>> >>>> ??? stat() seems to be implemented using FindFirstFile() (see crt >>>> ??? sources if you can), and we call GetFileAttributesA(), so I do >>>> ??? not think this differs much. >>> ??? Yes, I saw the same in my Visual Studio installation. >>> >>> ??? thanks, >>> ??? Calvin >>> >>>>> >>>>> ??????? But onward: >>>>> >>>>> >>>>> >>>>> ??????????????? ========================= >>>>> >>>>> ??????????????? src/hotspot/os/windows/os_windows.cpp >>>>> >>>>> ??????????????? Could you please use wchar_t instead of WCHAR? >>>>> >>>>> ??????????? Fixed. >>>>> >>>>> >>>>> ??????????????? --------------- >>>>> ??????????????? in os::stat(): >>>>> >>>>> ??????????????? This cumbersome malloc/strcpy sequence: >>>>> >>>>> ??????????????? !?? size_t path_len = strlen(path) + 1; >>>>> ??????????????? +?? char* pathbuf = (char*)os::malloc(path_len * >>>>> ??????????????? sizeof(char), mtInternal); >>>>> ??????????????? +?? if (pathbuf == NULL) { >>>>> ??????????????? +???? errno = ENOMEM; >>>>> ????????????????????? return -1; >>>>> ??????????????????? } >>>>> ??????????????????? os::native_path(strcpy(pathbuf, path)); >>>>> >>>>> ???????????????? can be reduced to a simple strdup: >>>>> >>>>> ??????????????? char* pathbuf = os::strdup(path, mtInternal); >>>>> ??????????????? if (pathbuf == NULL) { >>>>> ????????????????? errno = ENOMEM; >>>>> ????????????????? return -1; >>>>> ??????????????? } >>>>> ??????????????? os::native_path(pathbuf); >>>>> >>>>> ??????????????? This also would separate strcpy() from >>>>> ??????????????? os::native_path() which is a bit unreadable. >>>>> >>>>> ??????????? I've made the change. >>>>> >>>>> >>>>> ??????????????? -------------------- >>>>> >>>>> ??????????????? The single-byte path (strdup, ::stat()), together >>>>> ??????????????? with its free(), can be moved inside the >>>>> ??????????????? (strlen(path) < MAX_PATH) condition. >>>>> ??????????????? os::native_path will not change the path length (it >>>>> ??????????????? better not, as it operates on the input buffer). >>>>> ??????????????? That avoids? having two allocations on the >>>>> ??????????????? doublebyte path. >>>>> >>>>> >>>>> ??????????? os::native_path() converts a path like C:\\\\somedir to >>>>> ??????????? C:\\somedir >>>>> ??????????? So I'll need the converted path in the wchar_t case >>>>> ??????????? too. The freeing of the pathbuf is being done at the >>>>> ??????????? end of the function or in the middle of the wchar_t >>>>> ??????????? case if there's an error. >>>>> >>>>> >>>>> ??????? Oh, you are right,of course. I missed that it this is >>>>> ??????? needed for both paths. >>>>> >>>>> >>>>> >>>>> ??????????????? ----------------------- >>>>> >>>>> ??????????????? But seeing that translation to UNC paths is >>>>> ??????????????? something we might want more often, I would combine >>>>> ??????????????? allocation and UNC prefix adding to one function >>>>> ??????????????? like this, to avoid the memove and increase >>>>> ??????????????? reusability: >>>>> >>>>> ??????????????? // Creates an unc path from a single byte path. >>>>> ??????????????? Return buffer is allocated in C heap and needs to >>>>> ??????????????? be freed by caller. Returns NULL on error. >>>>> ??????????????? static whchar_t* create_unc_path(const char* s) { >>>>> ????????????????? - does s start with \\?\ ? >>>>> ??????????????? - yes: >>>>> ??????????????????????????? - os::malloc(strlen(s) + 1) and >>>>> mbstowcs_s >>>>> ??????????????????????? - no: >>>>> ??????????????????????????? - os::malloc(strlen(s) + 1 + 4), >>>>> ??????????????? mbstowcs_s to fourth position in string, prefix >>>>> ??????????????? with L"\\?\" >>>>> ??????????????? } >>>>> >>>>> >>>>> ??????????? I also include the case for adding L"\\\\?\\UNC\0" at >>>>> ??????????? the beginning to be consistent with >>>>> ??????????? libjava/canonicalize_md.c. >>>>> >>>>> ??????????????? We also need error checking to mbstowcs_s. >>>>> >>>>> ??????????? I've added assert like the following after the call: >>>>> >>>>> ??????????? assert(converted_chars == path_len, "sanity"); >>>>> >>>>> >>>>> >>>>> ??????? create_unc_path() : >>>>> >>>>> ??????? - could you convert the /**/ to // comments, please? >>>> ??????? Fixed. >>>> >>>> >>>> ??? thanks. >>>> >>>> >>>>> >>>>> ??????? - not sure about the assert after mbstowcs_s: if we happen >>>>> ??????? to encounter an invalid multibyte character, function will >>>>> ??????? fail and return an error: >>>>> >>>>> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >>>>> >>>>> ??????? "If mbstowcs_s encounters an invalid multibyte character, >>>>> ??????? it puts 0 in *``pReturnValue, sets the destination buffer >>>>> ??????? to an empty string, sets errno to EILSEQ, and returns >>>>> EILSEQ." >>>> ??????? I've changed create_unc_path() so that the caller will get >>>> ??????? the errno and removed the assert. >>>> >>>> ??????? static wchar_t* create_unc_path(const char* path, errno_t >>>> &err) >>>> >>>> >>>> ??? Okay, works for me. >>>> >>>> >>>>> >>>>> ???????? As this is dependent on user data, we should not assert, >>>>> ??????? but handle the return code of mbstowcs_s gracefully. Same >>>>> ??????? goes for libjava/canonicalize_md.c. >>>>> >>>>> ??????? - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len >>>>> ??????? + 7, path, path_len); >>>>> ??????? third parameter is wrong, as we hand in an offset into the >>>>> ??????? buffer, we must decrement the buffer size by this offset, >>>>> ??????? so correct would be path_len +7 - 7 or just path_len. >>>>> >>>>> ??????? - Same error below: + ::mbstowcs_s(&converted_chars, >>>>> ??????? &wpath[4], path_len + 4, path, path_len); >>>> ??????? Fixed in both places. >>>> >>>> >>>> ??? Okay. >>>> >>>> >>>>> >>>>> >>>>> ??????????????? Just for cleanliness, I would then wrap >>>>> ??????????????? deallocation into an own function. >>>>> >>>>> ??????????????? static viud destroy_unc_path(whchar_t* s) { >>>>> ??????????????? os::free(s); } >>>>> >>>>> ??????????? I've added the destroy function. >>>>> >>>>> >>>>> ??????????????? These two functions could be candidates of putting >>>>> ??????????????? into os::win32 namespace, as this form of ANSI->UNC >>>>> ??????????????? path translation is quite common - whoever wants to >>>>> ??????????????? use the xxxxW() functions must do this. >>>>> >>>>> ??????????? I'm leaving them in os_windows.cpp since they're being >>>>> ??????????? used only within that file. >>>>> >>>>> >>>>> ??????? Fine by me. >>>>> >>>>> >>>>> >>>>> ??????????????? ----------------------- >>>>> >>>>> ??????????????? FindFirstFileW: >>>>> >>>>> ??????????????? I am pretty sure that you can achieve the same >>>>> ??????????????? result with GetFileAttributesExW(). It also returns >>>>> ??????????????? WIN32_FIND_DATAW. >>>>> >>>>> ??????????? It actually returns WIN32_FILE_ATTRIBUTE_DATA and is >>>>> ??????????? very similar to WIN32_FIND_DATAW. >>>>> >>>>> >>>>> ??????????????? It is way more straightforward to use than >>>>> ??????????????? FindFirstFileW, as it does not require you to write >>>>> ??????????????? a callback hook. >>>>> >>>>> ??????????? I've switched to using GetFileAttributesExW(). >>>>> >>>>> >>>>> ??????? Thank you, this is way more readable. >>>>> ??????? Another issue: do we still need the fix for 6539723 if we >>>>> ??????? switch from stat() to GetFileAttributesExW()? This fix >>>>> ??????? looks important, but the comment >>>>> ??????? indicates that it could break things if the original bug is >>>>> ??????? not present. >>>>> >>>>> ??????? Btw, this is another strong argument for scrapping ::stat() >>>>> ??????? altogether on all code paths, not only for long input >>>>> ??????? paths, because stat() and GetFileAttributesExW() may behave >>>>> ??????? differently. So it would be good to use the same API on all >>>>> ??????? code paths, in order to get the best test coverage. >>>> ??????? For this round of change, I'm using >>>> ??????? GetFileAttributesEx[A|W]() for both code paths. >>>> >>>> ??????? webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ >>>> >>>> >>>> ??????? thanks, >>>> ??????? Calvin >>>> >>>> >>>> ??? Okay, all good apart from the issues mentioned above. Thanks for >>>> ??? your work! >>>> >>>> ??? Best Regards, Thomas >>>> >>>> >>>> >>>>> >>>>> >>>>> >>>>> ??????????????? ------- >>>>> >>>>> ??????????????? eval_find_data(): This is more of a generic helper >>>>> ??????????????? function, could you rename this to something >>>>> ??????????????? clearer, e.g. make_double_word() ? >>>>> >>>>> ??????????? Ok. I've renamed it. >>>>> >>>>> >>>>> ??????????????? Also, setup_stat(), could this be renamed more >>>>> ??????????????? clearly to something like WIN32_FIND_DATA_to_stat? >>>>> ??????????????? or lowercase if this bothers you :) >>>>> >>>>> ??????????? I'm naming the function as >>>>> ??????????? file_attribute_data_to_stat() to match with the data >>>>> ??????????? structure name. >>>>> >>>>> >>>>> ??????? Thanks for taking my suggestions. >>>>> >>>>> >>>>> >>>>> ??????????????? ================================== >>>>> src/hotspot/share/classfile/classLoader.cpp >>>>> >>>>> ??????????????? In ClassPathDirEntry::open_stream(), I would feel >>>>> ??????????????? better if we were asserting _dir and name to be != >>>>> ??????????????? NULL before feeding it to strlen. >>>>> >>>>> ??????????? I've added an assert statement. >>>>> >>>>> >>>>> ??????????????? =================== >>>>> >>>>> ??????????? Here's an updated webrev: >>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >>>>> >>>>> >>>>> >>>>> ??????? Thanks! >>>>> >>>>> ??????? Thomas >>>>> >>>>> ??????????? thanks, >>>>> ??????????? Calvin >>>>> >>>>> >>>>> ??????????????? Thanks! >>>>> >>>>> ??????????????? Thomas >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ??????????????? On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >>>>> ??????????????? >>>> ??????????????? >>>>> ??????????????? >>>> >> wrote: >>>>> >>>>> ??????????????????? I've reworked this fix by using the Unicode >>>>> ??????????????? version of open >>>>> ??????????????????? (wopen) to handle path name longer than max >>>>> ??????????????? path with the path >>>>> ??????????????????? prefixed to indicate an extended length path as >>>>> ??????????????? described in [1]. >>>>> >>>>> ??????????????????? The Unicode version of stat (wstat) doesn't >>>>> ??????????????? work well with long >>>>> ??????????????????? path [2]. So FindFirstFileW will be used.The >>>>> ??????????????? data in >>>>> ??????????????????? WIN32_FIND_DATA returned from FindFirstFileW >>>>> ??????????????? needs to be >>>>> ??????????????????? transferred to the stat structure since the >>>>> ??????????????? caller expects a >>>>> ??????????????????? return stat structure and other platforms >>>>> ??????????????? return a stat structure. >>>>> >>>>> ??????????????????? In classLoader.cpp, calculate the size of >>>>> ??????????????? buffer required instead >>>>> ??????????????????? of limiting it to JVM_MAXPATHLEN. >>>>> ??????????????????? In os_windows.cpp, dynamically allocate buffers >>>>> ??????????????? in os::open and >>>>> ??????????????????? os::stat. >>>>> >>>>> ??????????????????? updated webrev: >>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>>>> >>>>> >>>> > >>>>> >>>>> ??????????????????? It passed hs-tier2 testing using mach5. >>>>> >>>>> ??????????????????? thanks, >>>>> ??????????????????? Calvin >>>>> >>>>> ??????????????????? [1] >>>>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH >>>>> >>>>> >>>> > >>>>> >>>>> ??????????????????? [2] >>>>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>>>> >>>>> >>>> > >>>>> >>>>> >>>>> >>>>> ??????????????????? On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>>>> >>>>> ??????????????????????? JBS: >>>>> https://bugs.openjdk.java.net/browse/JDK-8188122 >>>>> >>>>> >>>> > >>>>> >>>>> ??????????????????????? Adding a warning message if the full path >>>>> ??????????????? or the directory >>>>> ??????????????????????? length exceeds MAX_PATH on windows. >>>>> >>>>> ??????????????????????? Example warning messages. >>>>> >>>>> ??????????????????????? 1) The full path exceeds MAX_PATH: >>>>> >>>>> ??????????????????????? Java HotSpot(TM) 64-Bit Server VM warning: >>>>> ??????????????? construct full path >>>>> ??????????????????????? name failed: total length 270 exceeds max >>>>> ??????????????? length of 260 >>>>> ??????????????????????????? dir >>>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> ??????????????????????? length 259 >>>>> ??????????????????????????? name Hello.class length 11 >>>>> >>>>> ??????????????????????? 2) The directory path exceeds MAX_PATH: >>>>> >>>>> ??????????????????????? Java HotSpot(TM) 64-Bit Server VM warning: >>>>> ??????????????? os::stat failed: >>>>> ??????????????????????? path length 265 exceeds max length 260 >>>>> ??????????????????????????? path >>>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>>>> >>>>> ??????????????????????? webrev: >>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>>>> >>>>> >>>> > >>>>> >>>>> ??????????????????????? Testing: >>>>> ??????????????????????????? JPRT (including the new test) >>>>> >>>>> ??????????????????????? thanks, >>>>> ??????????????????????? Calvin >>>>> >>>>> >>>>> >>>> >>> From calvin.cheung at oracle.com Tue Nov 21 16:41:47 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 21 Nov 2017 08:41:47 -0800 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <57af485d-5127-9cd5-316f-c8873003983f@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> <59FA1B6E.4090000@oracle.com> <5A033E72.90800@oracle.com> <5A048F23.9030806@oracle.com> <5A1335A6.2070509@oracle.com> <57af485d-5127-9cd5-316f-c8873003983f@oracle.com> Message-ID: <5A14574B.8080707@oracle.com> Thanks, Ioi. Calvin On 11/21/17, 8:36 AM, Ioi Lam wrote: > Looks good. Thanks! > > - Ioi > > > On 11/20/17 12:05 PM, Calvin Cheung wrote: >> I've had some off-list discussion with Ioi resulting in another >> update to the webrev. >> >> - added relative path scenario to the test. Currently, this fix >> doesn't handle relative path on windows yet. The following RFE has >> been filed to cover long relative paths on windows: >> https://bugs.openjdk.java.net/browse/JDK-8191521 >> - renamed the test LongPath.java to LongBCP.java to better reflect >> what is being tested; >> - some comments update on os_windows.cpp >> >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.05/ >> >> thanks, >> Calvin >> >> On 11/9/17, 9:23 AM, Calvin Cheung wrote: >>> Hi Thomas, >>> >>> On 11/8/17, 10:40 PM, Thomas St?fe wrote: >>>> Hi Calvin, >>>> >>>> On Wed, Nov 8, 2017 at 6:27 PM, Calvin Cheung >>>> > wrote: >>>> >>>> >>>> >>>> On 11/7/17, 6:12 AM, Thomas St?fe wrote: >>>>> Hi Calvin, >>>>> >>>>> On Wed, Nov 1, 2017 at 8:07 PM, Calvin Cheung >>>>> > wrote: >>>>> >>>>> >>>>> >>>>> On 10/27/17, 12:55 AM, Thomas St?fe wrote: >>>>>> Hi Calvin, >>>>>> >>>>>> On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung >>>>>> >>>>> > wrote: >>>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> On 10/25/17, 11:54 PM, Thomas St?fe wrote: >>>>>> >>>>>> Hi Calvin, >>>>>> >>>>>> this is a worthwhile addition, thank you for your >>>>>> work! >>>>>> >>>>>> Thanks for your review. >>>>>> >>>>>> >>>>>> Thanks for your work :) >>>>>> >>>>>> First off, there is another issue in >>>>>> file_attribute_data_to_stat(). We actually had this issue >>>>>> before, but your work uncovered it: >>>>>> >>>>>> According to POSIX >>>>>> (http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html >>>>>> >>>>>> ) >>>>>> >>>>>> and every unix manpage I looked at (solaris, linux, aix..), >>>>>> st_ctime is not the file creation time but the last time >>>>>> file status was changed. Only Microsoft gets this wrong in >>>>>> their posix layer, its stat() function returns >>>>>> (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx >>>>>> ) >>>>>> creation time in st_ctime. >>>>>> >>>>>> But as os::stat() is a platform independent layer, I'd say >>>>>> we should behave the same on all platforms, and that would >>>>>> be the Posix way. >>>>>> >>>>>> I did not find any code calling os::stat() and using >>>>>> st_ctime, so this is still save to change for windows. >>>>>> (Note that I found that perfMemory_xxx.cpp uses plain OS >>>>>> ::stat and misuses ctime as "creation time" - I opened a >>>>>> bug for that >>>>>> (https://bugs.openjdk.java.net/browse/JDK-8190260 >>>>>> - but >>>>>> that does not affect us, as they do not call os::stat()). >>>>>> >>>>>> There is one more complication: in os::stat() we use plain >>>>>> ::stat() in the single byte case.: >>>>>> >>>>>> *+ if (strlen(path) < MAX_PATH) {* >>>>>> *+ ret = ::stat(pathbuf, sbuf);* >>>>>> *+ } else {* >>>>>> * >>>>>> * >>>>>> But ::stat() on Windows is broken, as I wrote above. We >>>>>> should not use it, especially not if we change the meaing >>>>>> of st_ctime in the double byte path. >>>>>> >>>>>> My suggestion would be to just always call >>>>>> GetFileAttributes(), also for the single byte path. A very >>>>>> simple solution would be to just always go the double byte >>>>>> path with UNC translation, regardless of the path >>>>>> length. Or, if you are worried about performance, for paths >>>>>> shorter than MAX_PATH we use GetFileAttributesA(), for >>>>>> longer paths GetFileAttributesW with UNC translation. In >>>>>> both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you >>>>>> can feed tp your file_attribute_data_to_stat() routine and >>>>>> have the struct stat you return be always consistent. >>>>> I ran into an issue with the dwFileAttributes value for a >>>>> jar file. On Windows Server O/S, the value is set to 0x2020 >>>>> which is (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>>>> FILE_ATTRIBUTE_ARCHIVE) but on a desktop O/S like Windows 7, >>>>> it is set to 0x0020 which is just FILE_ATTRIBUTE_ARCHIVE. >>>>> I've fixed it in file_attribute_data_to_stat(). >>>>> >>>>> >>>>> Oh.. :( good catch! But that opens a new can of worms I did not >>>>> see before: >>>>> >>>>> FILE_ATTRIBUTE_NORMAL is documented as "A file that does not >>>>> have other attributes set. This attribute is valid only when >>>>> used alone." In addition to this flag, we have a multitude of >>>>> things like FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, >>>>> FILE_ATTRIBUTE_READONLY ... etc, all cases where we assume this >>>>> is a normal file in Unix terminology and where we would expect >>>>> os::stat to return S_IFREG, but where according to the msdn doc >>>>> FILE_ATTRIBUTE_NORMAL is not set. >>>>> >>>>> Looking at what others do in this scenario (Looked at mingw >>>>> sources and at ReactOS - I am not posting any source code here >>>>> for legal reasons but feel free to look for yourself), the usual >>>>> way to translate WIN32_FILE_ATTRIBUTES to struct stat seems to >>>>> be: >>>>> "if FILE_ATTRIBUTE_DIRECTORY then set S_IFDIR else S_IFREG" (so, >>>>> no dependency on FILE_ATTRIBUTE_NORMAL). >>>> This makes sense but I ran into similar problem as before - the >>>> dwFileAttributes has a different value on a windows server O/S vs >>>> desktop O/S. So I need to do the check as follows: >>>> >>>> + // A directory has the dwFileAttributes value of 0x2010 >>>> which is >>>> + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>>> FILE_ATTRIBUTE_ARCHIVE) >>>> + // on Windows Server O/S but the value is 0x0010 on Windows >>>> desktop O/S >>>> + // such as Windows 7. >>>> + if ((file_data.dwFileAttributes& FILE_ATTRIBUTE_DIRECTORY) >>>> != 0) { >>>> + sbuf->st_mode |= S_IFDIR; >>>> + } else { >>>> + sbuf->st_mode |= S_IFREG; >>>> + } >>>> >>>> I scratched my head a bit about the comment till I understood that >>>> you mean "if FILE_ATTRIBUTE_DIRECTORY bit is set it is a directory >>>> regardless of which other flags are set" instead of "if >>>> flags==FILE_ATTRIBUTE_DIRECTORY it is a directory". Sure, I guess >>>> my comment above was sloppy, but this was what I meant. I am not >>>> even sure the comment is needed, this is quite self-explaining. >>> I've noticed a typo in the above comment: >>> + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) >>> >>> FILE_ATTRIBUTE_ARCHIVE should be FILE_ATTRIBUTE_DIRECTORY >>> >>> I'll correct it before push. >>> >>>> >>>> updated webrev: >>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.04/ >>>> >>>> >>>> >>>> I am fine with all the changes now. Thank you for your work and >>>> patience. >>> Thanks for your discussions and review. >>> >>> thanks, >>> Calvin >>>> >>>> Kind Regards, Thomas >>>> >>>> >>>> >>>> >>>> Diff from webrev.03: >>>> >>>> < --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >>>> 08:50:40.170786397 -0800 >>>> < +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-08 >>>> 08:50:39.803751877 -0800 >>>> < @@ -4060,41 +4060,119 @@ >>>> --- >>>> > --- old/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >>>> 09:40:13.657460425 -0700 >>>> > +++ new/src/hotspot/os/windows/os_windows.cpp 2017-11-01 >>>> 09:40:13.261423192 -0700 >>>> > @@ -4060,41 +4060,121 @@ >>>> 25,29c25 >>>> < + // A directory has the dwFileAttributes value of 0x2010 which is >>>> < + // (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_ATTRIBUTE_ARCHIVE) >>>> < + // on Windows Server O/S but the value is 0x0010 on Windows >>>> desktop O/S >>>> < + // such as Windows 7. >>>> < + if ((file_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) >>>> != 0) { >>>> --- >>>> > + if (file_data.dwFileAttributes == FILE_ATTRIBUTE_DIRECTORY) { >>>> 31c27,33 >>>> < + } else { >>>> --- >>>> > + } >>>> > + if ((file_data.dwFileAttributes == FILE_ATTRIBUTE_NORMAL) || >>>> > + // an archive file such as a jar file has the >>>> dwFileAttributes value of >>>> > + // 0x2020 (FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | >>>> FILE_ATTRIBUTE_ARCHIVE) >>>> > + // on Windows Server O/S but the value is 0x0020 on >>>> > + // Windows desktop O/S such as Windows 7. >>>> > + ((file_data.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE) >>>> != 0)) { >>>> >>>>> >>>>> I wonder about other special cases which should work too: >>>>> - read-only files should be S_IFREG and !S_IWRITE, >>>> For a read-only system file under the user's home dir. >>>> >>>> st_mode & 0xFF00 = 0x8100 = S_IFREG | S_IREAD >>>> dwFileAttributes = 39 = (FILE_ATTRIBUTE_ARCHIVE | >>>> FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | >>>> FILE_ATTRIBUTE_READONLY) >>>>> read-only directories should be S_IFDIR and !S_IWRITE. >>>> I've tried the C:\progra~1 dir. >>>> >>>> st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >>>> dwFileAttributes = 17 = (FILE_ATTRIBUTE_DIRECTORY | >>>> FILE_ATTRIBUTE_READONLY) >>>>> - We should tread the device root ("C:\") as a directory (so, >>>>> os::stat("c:\") should return S_IFDIR). Does this work? >>>> This one works too. >>>> >>>> st_mode & 0xFF00 = 0x4100 = S_IFDIR | S_IREAD >>>> dwFileAttributes = 22 = (FILE_ATTRIBUTE_DIRECTORY | >>>> FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) >>>>> >>>>> I've also used GetTickCounts() to measure the performance of >>>>> ::stat() vs GetFileAttributesA() plus >>>>> file_attribute_data_to_stat(). There's no difference at the >>>>> ms resolution. Using the high-resolution timestamp >>>>> (https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx) >>>>> >>>>> , >>>>> >>>>> it doesn't show much difference. >>>>> >>>>> >>>>> stat() seems to be implemented using FindFirstFile() (see crt >>>>> sources if you can), and we call GetFileAttributesA(), so I do >>>>> not think this differs much. >>>> Yes, I saw the same in my Visual Studio installation. >>>> >>>> thanks, >>>> Calvin >>>> >>>>>> >>>>>> But onward: >>>>>> >>>>>> >>>>>> >>>>>> ========================= >>>>>> >>>>>> src/hotspot/os/windows/os_windows.cpp >>>>>> >>>>>> Could you please use wchar_t instead of WCHAR? >>>>>> >>>>>> Fixed. >>>>>> >>>>>> >>>>>> --------------- >>>>>> in os::stat(): >>>>>> >>>>>> This cumbersome malloc/strcpy sequence: >>>>>> >>>>>> ! size_t path_len = strlen(path) + 1; >>>>>> + char* pathbuf = (char*)os::malloc(path_len * >>>>>> sizeof(char), mtInternal); >>>>>> + if (pathbuf == NULL) { >>>>>> + errno = ENOMEM; >>>>>> return -1; >>>>>> } >>>>>> os::native_path(strcpy(pathbuf, path)); >>>>>> >>>>>> can be reduced to a simple strdup: >>>>>> >>>>>> char* pathbuf = os::strdup(path, mtInternal); >>>>>> if (pathbuf == NULL) { >>>>>> errno = ENOMEM; >>>>>> return -1; >>>>>> } >>>>>> os::native_path(pathbuf); >>>>>> >>>>>> This also would separate strcpy() from >>>>>> os::native_path() which is a bit unreadable. >>>>>> >>>>>> I've made the change. >>>>>> >>>>>> >>>>>> -------------------- >>>>>> >>>>>> The single-byte path (strdup, ::stat()), together >>>>>> with its free(), can be moved inside the >>>>>> (strlen(path) < MAX_PATH) condition. >>>>>> os::native_path will not change the path length (it >>>>>> better not, as it operates on the input buffer). >>>>>> That avoids having two allocations on the >>>>>> doublebyte path. >>>>>> >>>>>> >>>>>> os::native_path() converts a path like C:\\\\somedir to >>>>>> C:\\somedir >>>>>> So I'll need the converted path in the wchar_t case >>>>>> too. The freeing of the pathbuf is being done at the >>>>>> end of the function or in the middle of the wchar_t >>>>>> case if there's an error. >>>>>> >>>>>> >>>>>> Oh, you are right,of course. I missed that it this is >>>>>> needed for both paths. >>>>>> >>>>>> >>>>>> >>>>>> ----------------------- >>>>>> >>>>>> But seeing that translation to UNC paths is >>>>>> something we might want more often, I would combine >>>>>> allocation and UNC prefix adding to one function >>>>>> like this, to avoid the memove and increase >>>>>> reusability: >>>>>> >>>>>> // Creates an unc path from a single byte path. >>>>>> Return buffer is allocated in C heap and needs to >>>>>> be freed by caller. Returns NULL on error. >>>>>> static whchar_t* create_unc_path(const char* s) { >>>>>> - does s start with \\?\ ? >>>>>> - yes: >>>>>> - os::malloc(strlen(s) + 1) and >>>>>> mbstowcs_s >>>>>> - no: >>>>>> - os::malloc(strlen(s) + 1 + 4), >>>>>> mbstowcs_s to fourth position in string, prefix >>>>>> with L"\\?\" >>>>>> } >>>>>> >>>>>> >>>>>> I also include the case for adding L"\\\\?\\UNC\0" at >>>>>> the beginning to be consistent with >>>>>> libjava/canonicalize_md.c. >>>>>> >>>>>> We also need error checking to mbstowcs_s. >>>>>> >>>>>> I've added assert like the following after the call: >>>>>> >>>>>> assert(converted_chars == path_len, "sanity"); >>>>>> >>>>>> >>>>>> >>>>>> create_unc_path() : >>>>>> >>>>>> - could you convert the /**/ to // comments, please? >>>>> Fixed. >>>>> >>>>> >>>>> thanks. >>>>> >>>>> >>>>>> >>>>>> - not sure about the assert after mbstowcs_s: if we happen >>>>>> to encounter an invalid multibyte character, function will >>>>>> fail and return an error: >>>>>> >>>>>> https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx >>>>>> >>>>>> "If mbstowcs_s encounters an invalid multibyte character, >>>>>> it puts 0 in *``pReturnValue, sets the destination buffer >>>>>> to an empty string, sets errno to EILSEQ, and returns >>>>>> EILSEQ." >>>>> I've changed create_unc_path() so that the caller will get >>>>> the errno and removed the assert. >>>>> >>>>> static wchar_t* create_unc_path(const char* path, errno_t >>>>> &err) >>>>> >>>>> >>>>> Okay, works for me. >>>>> >>>>> >>>>>> >>>>>> As this is dependent on user data, we should not assert, >>>>>> but handle the return code of mbstowcs_s gracefully. Same >>>>>> goes for libjava/canonicalize_md.c. >>>>>> >>>>>> - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len >>>>>> + 7, path, path_len); >>>>>> third parameter is wrong, as we hand in an offset into the >>>>>> buffer, we must decrement the buffer size by this offset, >>>>>> so correct would be path_len +7 - 7 or just path_len. >>>>>> >>>>>> - Same error below: + ::mbstowcs_s(&converted_chars, >>>>>> &wpath[4], path_len + 4, path, path_len); >>>>> Fixed in both places. >>>>> >>>>> >>>>> Okay. >>>>> >>>>> >>>>>> >>>>>> >>>>>> Just for cleanliness, I would then wrap >>>>>> deallocation into an own function. >>>>>> >>>>>> static viud destroy_unc_path(whchar_t* s) { >>>>>> os::free(s); } >>>>>> >>>>>> I've added the destroy function. >>>>>> >>>>>> >>>>>> These two functions could be candidates of putting >>>>>> into os::win32 namespace, as this form of ANSI->UNC >>>>>> path translation is quite common - whoever wants to >>>>>> use the xxxxW() functions must do this. >>>>>> >>>>>> I'm leaving them in os_windows.cpp since they're being >>>>>> used only within that file. >>>>>> >>>>>> >>>>>> Fine by me. >>>>>> >>>>>> >>>>>> >>>>>> ----------------------- >>>>>> >>>>>> FindFirstFileW: >>>>>> >>>>>> I am pretty sure that you can achieve the same >>>>>> result with GetFileAttributesExW(). It also returns >>>>>> WIN32_FIND_DATAW. >>>>>> >>>>>> It actually returns WIN32_FILE_ATTRIBUTE_DATA and is >>>>>> very similar to WIN32_FIND_DATAW. >>>>>> >>>>>> >>>>>> It is way more straightforward to use than >>>>>> FindFirstFileW, as it does not require you to write >>>>>> a callback hook. >>>>>> >>>>>> I've switched to using GetFileAttributesExW(). >>>>>> >>>>>> >>>>>> Thank you, this is way more readable. >>>>>> Another issue: do we still need the fix for 6539723 if we >>>>>> switch from stat() to GetFileAttributesExW()? This fix >>>>>> looks important, but the comment >>>>>> indicates that it could break things if the original bug is >>>>>> not present. >>>>>> >>>>>> Btw, this is another strong argument for scrapping ::stat() >>>>>> altogether on all code paths, not only for long input >>>>>> paths, because stat() and GetFileAttributesExW() may behave >>>>>> differently. So it would be good to use the same API on all >>>>>> code paths, in order to get the best test coverage. >>>>> For this round of change, I'm using >>>>> GetFileAttributesEx[A|W]() for both code paths. >>>>> >>>>> webrev: >>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.03/ >>>>> >>>>> >>>>> thanks, >>>>> Calvin >>>>> >>>>> >>>>> Okay, all good apart from the issues mentioned above. Thanks for >>>>> your work! >>>>> >>>>> Best Regards, Thomas >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------- >>>>>> >>>>>> eval_find_data(): This is more of a generic helper >>>>>> function, could you rename this to something >>>>>> clearer, e.g. make_double_word() ? >>>>>> >>>>>> Ok. I've renamed it. >>>>>> >>>>>> >>>>>> Also, setup_stat(), could this be renamed more >>>>>> clearly to something like WIN32_FIND_DATA_to_stat? >>>>>> or lowercase if this bothers you :) >>>>>> >>>>>> I'm naming the function as >>>>>> file_attribute_data_to_stat() to match with the data >>>>>> structure name. >>>>>> >>>>>> >>>>>> Thanks for taking my suggestions. >>>>>> >>>>>> >>>>>> >>>>>> ================================== >>>>>> src/hotspot/share/classfile/classLoader.cpp >>>>>> >>>>>> In ClassPathDirEntry::open_stream(), I would feel >>>>>> better if we were asserting _dir and name to be != >>>>>> NULL before feeding it to strlen. >>>>>> >>>>>> I've added an assert statement. >>>>>> >>>>>> >>>>>> =================== >>>>>> >>>>>> Here's an updated webrev: >>>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ >>>>>> >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Thomas >>>>>> >>>>>> thanks, >>>>>> Calvin >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Thomas >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung >>>>>> >>>>> >>>>>> >>>>> >> wrote: >>>>>> >>>>>> I've reworked this fix by using the Unicode >>>>>> version of open >>>>>> (wopen) to handle path name longer than max >>>>>> path with the path >>>>>> prefixed to indicate an extended length path as >>>>>> described in [1]. >>>>>> >>>>>> The Unicode version of stat (wstat) doesn't >>>>>> work well with long >>>>>> path [2]. So FindFirstFileW will be used.The >>>>>> data in >>>>>> WIN32_FIND_DATA returned from FindFirstFileW >>>>>> needs to be >>>>>> transferred to the stat structure since the >>>>>> caller expects a >>>>>> return stat structure and other platforms >>>>>> return a stat structure. >>>>>> >>>>>> In classLoader.cpp, calculate the size of >>>>>> buffer required instead >>>>>> of limiting it to JVM_MAXPATHLEN. >>>>>> In os_windows.cpp, dynamically allocate buffers >>>>>> in os::open and >>>>>> os::stat. >>>>>> >>>>>> updated webrev: >>>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >>>>>> >>>>>> >>>>> > >>>>>> >>>>>> It passed hs-tier2 testing using mach5. >>>>>> >>>>>> thanks, >>>>>> Calvin >>>>>> >>>>>> [1] >>>>>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>> > >>>>>> >>>>>> >>>>>> [2] >>>>>> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>> > >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >>>>>> >>>>>> JBS: >>>>>> https://bugs.openjdk.java.net/browse/JDK-8188122 >>>>>> >>>>>> >>>>> > >>>>>> >>>>>> Adding a warning message if the full path >>>>>> or the directory >>>>>> length exceeds MAX_PATH on windows. >>>>>> >>>>>> Example warning messages. >>>>>> >>>>>> 1) The full path exceeds MAX_PATH: >>>>>> >>>>>> Java HotSpot(TM) 64-Bit Server VM warning: >>>>>> construct full path >>>>>> name failed: total length 270 exceeds max >>>>>> length of 260 >>>>>> dir >>>>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>> >>>>>> length 259 >>>>>> name Hello.class length 11 >>>>>> >>>>>> 2) The directory path exceeds MAX_PATH: >>>>>> >>>>>> Java HotSpot(TM) 64-Bit Server VM warning: >>>>>> os::stat failed: >>>>>> path length 265 exceeds max length 260 >>>>>> path >>>>>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>>>>> >>>>>> >>>>>> webrev: >>>>>> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>>>>> >>>>>> >>>>> > >>>>>> >>>>>> Testing: >>>>>> JPRT (including the new test) >>>>>> >>>>>> thanks, >>>>>> Calvin >>>>>> >>>>>> >>>>>> >>>>> >>>> > From coleen.phillimore at oracle.com Tue Nov 21 17:24:11 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 21 Nov 2017 12:24:11 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> Message-ID: <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: > Hi Coleen! > > Thanks for making time to review the Thread-SMR stuff again!! > > I have added back the other three OpenJDK aliases... This review is > being done on _four_ different OpenJDK aliases. > > As always, replies are embedded below... > > > On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/os/linux/os_linux.cpp.frames.html >> >> >> I read David's comments about the threads list iterator, and I was >> going to say that it can be cleaned up later, as the bulk of the >> change is the SMR part but this looks truly bizarre.?? It looks like >> it shouldn't compile because 'jt' isn't in scope. >> >> Why isn't this the syntax: >> >> JavaThreadIteratorWithHandle jtiwh; >> for (JavaThread* jt = jtiwh.first(); jt != NULL; jt = jtiwh.next()) { >> } >> >> Which would do the obvious thing without anyone having to squint at >> the code. > > See my reply to David's review for the more detailed answer. > > For the above syntax, we would need braces to limit the scope of the > 'jtiwh' variable. With Stefan's propsal, you get limited scope on > 'jtiwh' for "free". Yes, thank you, I see that motivation now. > > >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.hpp.html >> >> >> As a hater of acronmys, can this file use "Safe Memory Reclaimation" > > I'm not sure what you mean by this. Do you mean rename the files? > > ? threadSMR.hpp -> threadSafeMemoryReclaimation.hpp > ? threadSMR.cpp -> threadSafeMemoryReclaimation.cpp > No. > >> and briefly describe the concept in the beginning of the header file, >> so one knows why it's called threadSMR.hpp? > > And then this part of the sentence kind of indicates that you would be > okay with the threadSMR.{c,h}pp names if a comment was added to the > header file. > > Please clarify. Yes.? If you added a comment to the beginning of the threadSMR.hpp file that said what SMR stood for and what it was, that would be extremely helpful for future viewers of this file. > > >> It doesn't need to be long and include why Threads list needs this >> Sometimes we tell new people that the hotspot documentation is in the >> header files. > > Yup. When I migrated stuff from thread.hpp and thread.cpp to > threadSMR.hpp > and threadSMR.cpp, I should have written a header comment... > > I did update a comment in thread.cpp based on Robin W's code review: Yes, I think a comment belongs in the SMR file also, or in preference. > >> > src/hotspot/share/runtime/thread.cpp >> > >> > 3432 // operations over all threads.? It is protected by its own Mutex >> > 3433 // lock, which is also used in other contexts to protect thread >> > >> > Should this comment perhaps be revised to mention SMR? >> >> It definitely needs some updating... Here's a stab at it: >> >> // The Threads class links together all active threads, and provides >> // operations over all threads. It is protected by the Threads_lock, >> // which is also used in other global contexts like safepointing. >> // ThreadsListHandles are used to safely perform operations on one >> // or more threads without the risk of the thread exiting during the >> // operation. >> // >> // Note: The Threads_lock is currently more widely used than we >> // would like. We are actively migrating Threads_lock uses to other >> // mechanisms in order to reduce Threads_lock contention. > > I'll take a look at adding a header comment to threadSMR.hpp. > > >> 186?? JavaThreadIteratorWithHandle() : _tlh(), _index(0) {} >> >> This _tlh() call should not be necessary.? The compiler should >> generate this for you in the constructor. > > Deleted. > > >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >> >> >> ? 32 ThreadsList::ThreadsList(int entries) : _length(entries), >> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >> _next_list(NULL) { >> >> Seems like it should be mtThread rather than mtGC. > > Fixed. Definitely an artifact of Erik's original prototype when he > extracted Thread-SMR from his GC work... Thanks for catching it. > > >> Should >> >> ? 62?? if (EnableThreadSMRStatistics) { >> >> really be UL, ie: if (log_is_enabled(Info, thread, smr, statistics)) ? > > EnableThreadSMRStatistics is used in more places than UL code. > We use it in Thread::print_*() stuff to control output of > Thread-SMR statistics info in thread dumps and hs_err_pid file > generation. That sort of thing is also controlled by logging in general. > > Currently thread dump and hs_err_pid file output is not generated > using UL (and probably can't be?) so we need an option to control > the Thread-SMR statistics stuff in all places. > You can use log options in preference to this new diagnostic option in these cases too.?? This option looks a lot like logging to me and would be nice to not have to later convert it. > > >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java.html >> >> >> Can you use for these tests instead (there were a couple): >> >> *@requires (vm.debug == true)* > > The test I cloned had this in it: > > ??? if (!Platform.isDebugBuild()) { > ??????? // -XX:ErrorHandlerTest=N option requires debug bits. > ??????? return; > ??? } > > and you would like me to switch to the newer mechanism? > > I have updated the following tests: > > ? test/hotspot/jtreg/runtime/ErrorHandling/ErrorHandler.java > test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java > > test/hotspot/jtreg/runtime/ErrorHandling/ThreadsListHandleInErrorHandlingTest.java > > > Yes, the newer mechanism makes jtreg not bother to run the test, which makes it faster! >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/thread.cpp.udiff.html >> >> >> +// thread, has been added the Threads list, the system is not at a >> >> Has "not" been added to the Threads list?? missing "not"? > > Nope. If the JavaThread has been added to the Threads list > and it is not protected, then it is dangling. In other words, > a published JavaThread (on the Threads list) has to be protected > by either the system being at a safepoint or the JavaThread has > to be on some threads's ThreadsList. > > >> >> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >> >> Can you add a comment about where this number came from? > > I'll have to get that from Erik... > > >> I can't find the caller of threads_do_smr. > > I'm guessing that's used by the GC code that depends on Thread-SMR... > They? should add this API when the add the use of it then.? I don't see it in any sources. > >> If these functions xchg_smr_thread_list, get_smr_java_thread_list, >> inc_smr_deleted_thread_count are only used by thread.cpp, I think >> they should go in that file and not in the .inline.hpp file to be >> included and possibly called by other files.? I think they're private >> to class Threads. > > I have a vague memory that some of the compilers don't do inlining when > an "inline" function is in a .cpp. I believe we want these functions > to be inlined for performance reasons. Erik should probably chime in > here. I don't see why this should be.? Maybe some (older) compilers require it to be found before the call though, but that can still be accomplished in the .cpp file. > > >> I don't have any in-depth comments.? This looks really good from my >> day of reading it. > > Thanks for taking the time to review it again! > Thanks, Coleen > Dan > > >> >> Thanks, >> Coleen >> >> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >> >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, >>> and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with >>>> Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up >>>> thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from >>>>> within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we >>>> think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more >>>> major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>> holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>> closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>> and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>> done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>> to use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>> describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>> quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>> have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>> tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, >>>>>>>> and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >> > From daniel.daugherty at oracle.com Tue Nov 21 17:30:52 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 12:30:52 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <9cc8b08b-f373-f493-4429-662d18ff81ad@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <43d2f342-3f4b-5c1a-a687-c24c4efedba9@oracle.com> <92d1f572-a745-6f2b-1b5f-e68a1d252bab@oracle.com> <9cc8b08b-f373-f493-4429-662d18ff81ad@oracle.com> Message-ID: Adding back the missing OpenJDK aliases... On 11/21/17 11:14 AM, coleen.phillimore at oracle.com wrote: > > > On 11/20/17 8:50 PM, Daniel D. Daugherty wrote: >> On 11/20/17 12:51 AM, David Holmes wrote: >>> Hi Dan, >>> >>> Figured I'd better cast an eye over this again before it gets pushed :) >> >> Thanks for reviewing again!! >> >> >>> Only one thing (repeated many times) jumped out at me and apologies >>> if this has already been debated back and forth. I missed the >>> exchange where the for loop iteration was rewritten into this >>> unusual form: >>> >>> for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = >>> jtiwh.next(); ) { >>> >>> On first reading I expect most people would mistake the control >>> expression for the iteration-expression due to the next(). I didn't >>> even know the control expression could introduce a local variable! I >>> find this really hard to read stylistically and far too cute/clever >>> for its own good. It also violates the style rules regarding >>> implicit boolean checks. >>> >>> I know Stefan proposed this as he did not like the alternate (in a >>> few places): >>> >>> +? { >>> +??? ThreadsListHandle tlh; >>> +??? JavaThreadIterator jti(tlh.list()); >>> +??? for (JavaThread* t = jti.first(); t != NULL; t = jti.next()) { >>> ... >>> +? } >>> >>> but it seems to me the iterator code has evolved since then and only >>> a couple of places need a distinct TLH separate from the actual >>> iterator object. So I'm more in favour of the more classic for-loop, >>> with the iterator declared before the loop. Or even convert to while >>> loops, as this iterator seems more suited to that. >> >> Actually both Coleen and Stefan had a problem with how much additional >> code was added to support an iterator model. In some cases we went from >> 1-line to 3-lines and in some cases we went from 1-line to 5-lines. For >> Coleen, she wanted 2 of the new lines compressed into 1 new line by >> using >> an iterator with an embedded ThreadsListHandle. Stefan wanted us to try >> and get back to 1-line where possible. >> >> The advantages of the new style are: >> >> - 1-line to 1-line in all but a few cases >> - automatic limited scope of the embedded ThreadsListHandle so we were >> ? able to remove extra braces that were added earlier in most cases >> >> The disadvantages of the new style are: >> >> - it is an unusual for-loop style (in HotSpot) >> - it breaks HotSpot's style rule about implied booleans >> >> >> Stefan K is pretty adamant that the original iterator version where we >> go from 1-line to 3-lines or 1-line to 5-lines is not acceptable for GC >> code. The compiler guys haven't chimed in on this debate so I'm guessing >> they don't have a strong opinion either way. One thing that we don't >> want >> to do is have different styles for the different teams. >> >> So I had to make a judgement call on whether I thought Runtime could >> live >> with Stefan's idea. Originally I wasn't fond of it either and >> breaking the >> implied boolean style rule is a problem for me (I post that comment a >> lot >> in my code reviews). However, I have to say that I'm a lot happier about >> the compactness of the code and the fact that limited ThreadsListHandle >> scope comes for 'free' in most cases. >> >> We're planning to keep the new iterator style for the following reasons: >> >> - smaller change footprint >> - consistent style used for all of the teams >> - limited ThreadsListHandle scope comes for free in most cases >> >> We're hoping that you can live with this decision (for now) and maybe >> even grow to like it in the future. :-) > > The limited ThreadsListHandle scope, since ThreadsListHandle registers > and unregisters lists to SMR, seems to be worth the cost of looking at > this strange code pattern.?? Thank you for the explanation. > >> >> >>> Other than that just a couple of comments on variable type choices, >>> and a few typos spotted below. >> >> Replies embedded below. >> >> >>> >>> Thanks, >>> David >>> --- >>> >>> src/hotspot/share/runtime/globals.hpp >>> >>> 2476?? diagnostic(bool, EnableThreadSMRExtraValidityChecks, true, >>> ??????? \ >>> 2477?????????? "Enable Thread SMR extra validity checks") \ >>> 2478 ??????? \ >>> 2479?? diagnostic(bool, EnableThreadSMRStatistics, true, \ >>> 2480?????????? "Enable Thread SMR Statistics") ??????? \ >>> >>> Indent of strings is off by? 3 spaces. >> >> Fixed. >> >> >>> --- >>> >>> src/hotspot/share/runtime/handshake.cpp >>> >>> ?140?????? // There is assumption in code that Threads_lock should >>> be lock >>> ?200?????? // There is assumption in code that Threads_lock should >>> be lock >>> >>> lock -> locked >> >> Fixed. >> >> >>> 146???????? // handshake_process_by_vmthread will check the >>> semaphore for us again >>> >>> Needs period at end. >> >> Fixed. >> >> >>> 148???????? // should be okey since the new thread will not have an >>> operation. >>> >>> okey -> okay >> >> Fixed. >> >> >>> 151???????? // We can't warn here is since the thread do >>> cancel_handshake after it have been removed >>> >>> I think: >>> >>> // We can't warn here since the thread does cancel_handshake after >>> it has been removed >> >> Fixed. >> >> >>> 152???????? // from ThreadsList. So we should just keep looping here >>> until while below return negative. >>> >>> from -> from the >>> >>> Needs period at end. >> >> Fixed both. >> >> >>> >>> ?204???????????? // A new thread on the ThreadsList will not have an >>> operation. >>> >>> Change period to comma, and ... >> >> Fixed. >> >> >>> 205???????????? // Hence is skipped in handshake_process_by_vmthread. >>> >>> Hence is -> hence it is >> >> Fixed. >> >> >>> --- >>> >>> src/hotspot/share/runtime/thread.cpp >>> >>> 1482???? // dcubed - Looks like the Threads_lock is for stable access >>> 1483???? // to _jvmci_old_thread_counters and _jvmci_counters. >>> >>> Does this comment need revisiting? >> >> We've been trying to decide what to do about this comment. We'll be >> filing a follow up bug for the Compiler team to take another look at >> how _jvmci_old_thread_counters and _jvmci_counters are protected. >> Threads_lock is a big hammer and there should be a less expensive >> solution. Once that bug gets filed, this comment can go away. >> >> >>> 3451 volatile jint ... >>> >>> Why are you using jint for field types here? (Surprised Coleen >>> hasn't spotted them! ;-) ). > > Yes, it was the jtypes I would have objected to.? And long isn't a > good choice on windows because it's only 32 bits there. >> >> volatile jint???????? Threads::_smr_delete_notify = 0; >> volatile jint???????? Threads::_smr_deleted_thread_cnt = 0; >> volatile jint???????? Threads::_smr_deleted_thread_time_max = 0; >> volatile jint???????? Threads::_smr_deleted_thread_times = 0; >> : >> volatile jint???????? Threads::_smr_tlh_cnt = 0; >> volatile jint???????? Threads::_smr_tlh_time_max = 0; >> volatile jint???????? Threads::_smr_tlh_times = 0; >> >> _smr_delete_notify is a "flag" that indicates when an _smr_delete_lock >> notify() operation is needed. It counts up and down and should be a >> fairly >> small number. >> >> _smr_deleted_thread_cnt counts the number of Threads that have been >> deleted over the life of the VM. It's an always increasing number so >> it's size depends on how long the VM has been running. >> >> _smr_deleted_thread_time_max is the maximum time in millis it has >> taken to delete a thread. This field was originally a jlong, but >> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >> just been Atomic::add() that was not happy) >> >> _smr_deleted_thread_times accumulates the time in millis that it >> has taken to delete threads. It's an always increasing number so >> it's size depends on how long the VM has been running. This field >> was originally a jlong, but IIRC the Atomic::add() code was not >> happy on ARM... (might have just been Atomic::cmpxchg() that was >> not happy) >> >> _smr_tlh_cnt counts the number of ThreadsListHandles that have been >> deleted over the life of the VM. It's an always increasing number so >> it's size depends on how long the VM has been running. >> >> _smr_tlh_time_max is the maximum time in millis it has taken to >> delete a ThreadsListHandle. This field was originally a jlong, but >> IIRC the Atomic::cmpxchg() code was not happy on ARM... (might have >> just been Atomic::add() that was not happy) >> >> _smr_tlh_times? accumulates the time in millis that it has taken to >> delete ThreadsListHandles. It's an always increasing number so >> it's size depends on how long the VM has been running.? This field >> was originally a jlong, but IIRC the Atomic::add() code was not >> happy on ARM... (might have just been Atomic::cmpxchg() that was >> not happy) >> >> >> It just dawned on me that I'm not sure whether you think the 'jint' >> fields should be larger or smaller or the same size but a different >> type... :-) >> >> The fact that I had to write so much to explain what these fields >> are for and how they're used indicates I need more comments. See >> below... >> >> >>> 3456 long Threads::_smr_java_thread_list_alloc_cnt = 1; >>> 3457 long Threads::_smr_java_thread_list_free_cnt = 0; >>> >>> long? If you really want 64-bit counters here you won't get them on >>> Windows with that declaration. If you really want variable sized >>> counters I suggest uintptr_t; otherwise uint64_t. >> >> long????????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >> long????????????????? Threads::_smr_java_thread_list_free_cnt = 0; >> >> _smr_java_thread_list_alloc_cnt counts the number of ThreadsLists that >> have been allocated over the life of the VM. It's an always increasing >> number so it's size depends on how long the VM has been running and how >> many Threads have been created and destroyed. >> >> _smr_java_thread_list_free_cnt counts the number of ThreadsLists that >> have been freed over the life of the VM. It's an always increasing >> number so it's size depends on how long the VM has been running and how >> many Threads have been created and destroyed. >> >> I can't remember why we chose 'long' and I agree it's not a good choice >> for Win*. >> >> >> Okay so it dawns on me that we haven't written down a description for >> the various new '_cnt', '_max' and '_times" fields so I'm adding these >> comments to thread.hpp: >> > > With this comment format, it's really hard to pick out the name of the > field.? Nobody uses 80 columns anymore.? Can you move the comments > over some spaces so the field names are visible? Yes, I'll update the patch that I have for dholmes CR resolutions. > >> ?private: >> ? // Safe Memory Reclamation (SMR) support: >> ? static Monitor*????????????? _smr_delete_lock; >> ? // The '_cnt', '_max' and '_times" fields are enabled via >> ? // -XX:+EnableThreadSMRStatistics: >> ?????????????????????????????? // # of parallel threads in >> _smr_delete_lock->wait(). >> ? static uint????????????????? _smr_delete_lock_wait_cnt; >> ?????????????????????????????? // Max # of parallel threads in >> _smr_delete_lock->wait(). >> ? static uint????????????????? _smr_delete_lock_wait_max; >> ?????????????????????????????? // Flag to indicate when an >> _smr_delete_lock->notify() is needed. >> ? static volatile uint???????? _smr_delete_notify; >> ?????????????????????????????? // # of threads deleted over VM lifetime. >> ? static volatile uint???????? _smr_deleted_thread_cnt; >> ?????????????????????????????? // Max time in millis to delete a thread. >> ? static volatile uint???????? _smr_deleted_thread_time_max; >> ?????????????????????????????? // Cumulative time in millis to delete >> threads. >> ? static volatile uint???????? _smr_deleted_thread_times; >> ? static ThreadsList* volatile _smr_java_thread_list; >> ? static ThreadsList*????????? get_smr_java_thread_list(); >> ? static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); >> ?????????????????????????????? // # of ThreadsLists allocated over VM >> lifetime. >> ? static uint64_t????????????? _smr_java_thread_list_alloc_cnt; >> ?????????????????????????????? // # of ThreadsLists freed over VM >> lifetime. >> ? static uint64_t????????????? _smr_java_thread_list_free_cnt; >> ?????????????????????????????? // Max size ThreadsList allocated. >> ? static uint????????????????? _smr_java_thread_list_max; >> ?????????????????????????????? // Max # of nested ThreadsLists for a >> thread. >> ? static uint????????????????? _smr_nested_thread_list_max; >> ?????????????????????????????? // # of ThreadsListHandles deleted >> over VM lifetime. >> ? static volatile uint???????? _smr_tlh_cnt; >> ?????????????????????????????? // Max time in millis to delete a >> ThreadsListHandle. >> ? static volatile jint???????? _smr_tlh_time_max; >> ?????????????????????????????? // Cumulative time in millis to delete >> ThreadsListHandles. >> ? static volatile jint???????? _smr_tlh_times; >> ? static ThreadsList*????????? _smr_to_delete_list; >> ?????????????????????????????? // # of parallel ThreadsLists on the >> to-delete list. >> ? static uint????????????????? _smr_to_delete_list_cnt; >> ?????????????????????????????? // Max # of parallel ThreadsLists on >> the to-delete list. >> ? static uint????????????????? _smr_to_delete_list_max; >> >> >> I've also gone through all the new '_cnt', '_max' and '_times" fields >> in thread.cpp and added an impl note to explain the choice of type: >> > > Since this is in the cpp file and there is so much comment text, can > you just make it in column 1 and leave a blank line after each field.? > The indentation style is really hard to read and only useful if the > comment is short. Yes, I'll update the patch that I have for dholmes CR resolutions. > >> // Safe Memory Reclamation (SMR) support: >> Monitor*????????????? Threads::_smr_delete_lock = >> ????????????????????????? new Monitor(Monitor::special, >> "smr_delete_lock", >> ????????????????????????????????????? false /* allow_vm_block */, >> Monitor::_safepoint_check_never); >> // The '_cnt', '_max' and '_times" fields are enabled via >> // -XX:+EnableThreadSMRStatistics: >> ????????????????????? // # of parallel threads in >> _smr_delete_lock->wait(). >> ????????????????????? // Impl note: Hard to imagine > 64K waiting >> threads so >> ????????????????????? // this could be 16-bit, but there is no nice >> 16-bit >> ????????????????????? // _FORMAT support. >> uint????????????????? Threads::_smr_delete_lock_wait_cnt = 0; >> ????????????????????? // Max # of parallel threads in >> _smr_delete_lock->wait(). >> ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. >> uint????????????????? Threads::_smr_delete_lock_wait_max = 0; >> ????????????????????? // Flag to indicate when an >> _smr_delete_lock->notify() is needed. >> ????????????????????? // Impl note: See _smr_delete_lock_wait_cnt note. >> volatile uint???????? Threads::_smr_delete_notify = 0; >> ????????????????????? // # of threads deleted over VM lifetime. >> ????????????????????? // Impl note: Atomically incremented over VM >> lifetime so >> ????????????????????? // use unsigned for more range. Unsigned 64-bit >> would >> ????????????????????? // be more future proof, but 64-bit atomic inc >> isn't >> ????????????????????? // available everywhere (or is it?). >> volatile uint???????? Threads::_smr_deleted_thread_cnt = 0; >> ????????????????????? // Max time in millis to delete a thread. >> ????????????????????? // Impl note: 16-bit might be too small on an >> overloaded >> ????????????????????? // machine. Use unsigned since this is a time >> value. Set >> ????????????????????? // via Atomic::cmpxchg() in a loop for >> correctness. >> volatile uint???????? Threads::_smr_deleted_thread_time_max = 0; >> ????????????????????? // Cumulative time in millis to delete threads. >> ????????????????????? // Impl note: Atomically added to over VM >> lifetime so use >> ????????????????????? // unsigned for more range. Unsigned 64-bit >> would be more >> ????????????????????? // future proof, but 64-bit atomic inc isn't >> available >> ????????????????????? // everywhere (or is it?). >> volatile uint???????? Threads::_smr_deleted_thread_times = 0; >> ThreadsList* volatile Threads::_smr_java_thread_list = new >> ThreadsList(0); >> ????????????????????? // # of ThreadsLists allocated over VM lifetime. >> ????????????????????? // Impl note: We allocate a new ThreadsList for >> every >> ????????????????????? // thread create and every thread delete so we >> need a >> ????????????????????? // bigger type than the _smr_deleted_thread_cnt >> field. >> uint64_t????????????? Threads::_smr_java_thread_list_alloc_cnt = 1; >> ????????????????????? // # of ThreadsLists freed over VM lifetime. >> ????????????????????? // Impl note: See >> _smr_java_thread_list_alloc_cnt note. >> uint64_t????????????? Threads::_smr_java_thread_list_free_cnt = 0; >> ????????????????????? // Max size ThreadsList allocated. >> ????????????????????? // Impl note: Max # of threads alive at one >> time should >> ????????????????????? // fit in unsigned 32-bit. >> uint????????????????? Threads::_smr_java_thread_list_max = 0; >> ????????????????????? // Max # of nested ThreadsLists for a thread. >> ????????????????????? // Impl note: Hard to imagine > 64K nested >> ThreadsLists >> ????????????????????? // so his could be 16-bit, but there is no nice >> 16-bit >> ????????????????????? // _FORMAT support. >> uint????????????????? Threads::_smr_nested_thread_list_max = 0; >> ????????????????????? // # of ThreadsListHandles deleted over VM >> lifetime. >> ????????????????????? // Impl note: Atomically incremented over VM >> lifetime so >> ????????????????????? // use unsigned for more range. There will be >> fewer >> ????????????????????? // ThreadsListHandles than threads so unsigned >> 32-bit >> ????????????????????? // should be fine. >> volatile uint???????? Threads::_smr_tlh_cnt = 0; >> ????????????????????? // Max time in millis to delete a >> ThreadsListHandle. >> ????????????????????? // Impl note: 16-bit might be too small on an >> overloaded >> ????????????????????? // machine. Use unsigned since this is a time >> value. Set >> ????????????????????? // via Atomic::cmpxchg() in a loop for >> correctness. >> volatile uint???????? Threads::_smr_tlh_time_max = 0; >> ????????????????????? // Cumulative time in millis to delete >> ThreadsListHandles. >> ????????????????????? // Impl note: Atomically added to over VM >> lifetime so use >> ????????????????????? // unsigned for more range. Unsigned 64-bit >> would be more >> ????????????????????? // future proof, but 64-bit atomic inc isn't >> available >> ????????????????????? // everywhere (or is it?). >> volatile uint???????? Threads::_smr_tlh_times = 0; >> ThreadsList*????????? Threads::_smr_to_delete_list = NULL; >> ????????????????????? // # of parallel ThreadsLists on the to-delete >> list. >> ????????????????????? // Impl note: Hard to imagine > 64K >> ThreadsLists needing >> ????????????????????? // to be deleted so this could be 16-bit, but >> there is no >> ????????????????????? // nice 16-bit _FORMAT support. >> uint????????????????? Threads::_smr_to_delete_list_cnt = 0; >> ????????????????????? // Max # of parallel ThreadsLists on the >> to-delete list. >> ????????????????????? // Impl note: See _smr_to_delete_list_cnt note. >> uint????????????????? Threads::_smr_to_delete_list_max = 0; >> >> >> Yikes! With the additional comments, the additions to thread.hpp and >> thread.cpp grew by a bunch... I've done a test build build on my Mac >> with these changes and I'm about to kick off a Mach5 tier1 job... > > Another reason why I agreed with some earlier comment that this should > be in a new file because it's a new thing. I was somewhat surprised > that it's not in threadSMR.{hpp,cpp}.?? This refactoring could be > deferred but thread.cpp is too big! More refactoring is planned for after the initial push of Thread-SMR... Thanks for chiming in on this part of the review thread. Dan > > thanks, > Coleen > >> >> Dan >> >> >> >>> >>> ---- >>> >>> On 19/11/2017 11:49 AM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>> test, and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are >>>>> this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final >>>>> review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>> code quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>> server, and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >> > From daniel.daugherty at oracle.com Tue Nov 21 17:36:31 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 12:36:31 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> Message-ID: <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> Thanks for keeping all the OpenJDK aliases! On 11/21/17 12:24 PM, coleen.phillimore at oracle.com wrote: > > > On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: >> Hi Coleen! >> >> Thanks for making time to review the Thread-SMR stuff again!! >> >> I have added back the other three OpenJDK aliases... This review is >> being done on _four_ different OpenJDK aliases. >> >> As always, replies are embedded below... >> >> >> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/os/linux/os_linux.cpp.frames.html >>> >>> >>> I read David's comments about the threads list iterator, and I was >>> going to say that it can be cleaned up later, as the bulk of the >>> change is the SMR part but this looks truly bizarre. It looks like >>> it shouldn't compile because 'jt' isn't in scope. >>> >>> Why isn't this the syntax: >>> >>> JavaThreadIteratorWithHandle jtiwh; >>> for (JavaThread* jt = jtiwh.first(); jt != NULL; jt = jtiwh.next()) { >>> } >>> >>> Which would do the obvious thing without anyone having to squint at >>> the code. >> >> See my reply to David's review for the more detailed answer. >> >> For the above syntax, we would need braces to limit the scope of the >> 'jtiwh' variable. With Stefan's propsal, you get limited scope on >> 'jtiwh' for "free". > > Yes, thank you, I see that motivation now. >> >> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.hpp.html >>> >>> >>> As a hater of acronmys, can this file use "Safe Memory Reclaimation" >> >> I'm not sure what you mean by this. Do you mean rename the files? >> >> ? threadSMR.hpp -> threadSafeMemoryReclaimation.hpp >> ? threadSMR.cpp -> threadSafeMemoryReclaimation.cpp >> > > No. > >> >>> and briefly describe the concept in the beginning of the header file, >>> so one knows why it's called threadSMR.hpp? >> >> And then this part of the sentence kind of indicates that you would be >> okay with the threadSMR.{c,h}pp names if a comment was added to the >> header file. >> >> Please clarify. > > Yes.? If you added a comment to the beginning of the threadSMR.hpp > file that said what SMR stood for and what it was, that would be > extremely helpful for future viewers of this file. Yup. I just finished with the comment... > >> >> >>> It doesn't need to be long and include why Threads list needs this >>> Sometimes we tell new people that the hotspot documentation is in >>> the header files. >> >> Yup. When I migrated stuff from thread.hpp and thread.cpp to >> threadSMR.hpp >> and threadSMR.cpp, I should have written a header comment... >> >> I did update a comment in thread.cpp based on Robin W's code review: > > Yes, I think a comment belongs in the SMR file also, or in preference. Wasn't trying to say that I thought the update I did for Robin W was sufficient... >> >>> > src/hotspot/share/runtime/thread.cpp >>> > >>> > 3432 // operations over all threads.? It is protected by its own >>> Mutex >>> > 3433 // lock, which is also used in other contexts to protect thread >>> > >>> > Should this comment perhaps be revised to mention SMR? >>> >>> It definitely needs some updating... Here's a stab at it: >>> >>> // The Threads class links together all active threads, and provides >>> // operations over all threads. It is protected by the Threads_lock, >>> // which is also used in other global contexts like safepointing. >>> // ThreadsListHandles are used to safely perform operations on one >>> // or more threads without the risk of the thread exiting during the >>> // operation. >>> // >>> // Note: The Threads_lock is currently more widely used than we >>> // would like. We are actively migrating Threads_lock uses to other >>> // mechanisms in order to reduce Threads_lock contention. >> >> I'll take a look at adding a header comment to threadSMR.hpp. >> >> >>> 186?? JavaThreadIteratorWithHandle() : _tlh(), _index(0) {} >>> >>> This _tlh() call should not be necessary.? The compiler should >>> generate this for you in the constructor. >> >> Deleted. >> >> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >>> >>> >>> ? 32 ThreadsList::ThreadsList(int entries) : _length(entries), >>> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >>> _next_list(NULL) { >>> >>> Seems like it should be mtThread rather than mtGC. >> >> Fixed. Definitely an artifact of Erik's original prototype when he >> extracted Thread-SMR from his GC work... Thanks for catching it. >> >> >>> Should >>> >>> ? 62?? if (EnableThreadSMRStatistics) { >>> >>> really be UL, ie: if (log_is_enabled(Info, thread, smr, statistics)) ? >> >> EnableThreadSMRStatistics is used in more places than UL code. >> We use it in Thread::print_*() stuff to control output of >> Thread-SMR statistics info in thread dumps and hs_err_pid file >> generation. > > That sort of thing is also controlled by logging in general. Don't think I want to do that since logging may be be "up" at the time that I need Thread::print_*() or hs_err_pid generation... Something about chickens and eggs... :-) >> >> Currently thread dump and hs_err_pid file output is not generated >> using UL (and probably can't be?) so we need an option to control >> the Thread-SMR statistics stuff in all places. >> > > You can use log options in preference to this new diagnostic option in > these cases too.?? This option looks a lot like logging to me and > would be nice to not have to later convert it. See above reply... > >> >> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java.html >>> >>> >>> Can you use for these tests instead (there were a couple): >>> >>> *@requires (vm.debug == true)* >> >> The test I cloned had this in it: >> >> ??? if (!Platform.isDebugBuild()) { >> ??????? // -XX:ErrorHandlerTest=N option requires debug bits. >> ??????? return; >> ??? } >> >> and you would like me to switch to the newer mechanism? >> >> I have updated the following tests: >> >> ? test/hotspot/jtreg/runtime/ErrorHandling/ErrorHandler.java >> test/hotspot/jtreg/runtime/ErrorHandling/NestedThreadsListHandleInErrorHandlingTest.java >> >> test/hotspot/jtreg/runtime/ErrorHandling/ThreadsListHandleInErrorHandlingTest.java >> >> >> > Yes, the newer mechanism makes jtreg not bother to run the test, which > makes it faster! More quickly not run the test... Yup... got it... > >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/thread.cpp.udiff.html >>> >>> >>> +// thread, has been added the Threads list, the system is not at a >>> >>> Has "not" been added to the Threads list?? missing "not"? >> >> Nope. If the JavaThread has been added to the Threads list >> and it is not protected, then it is dangling. In other words, >> a published JavaThread (on the Threads list) has to be protected >> by either the system being at a safepoint or the JavaThread has >> to be on some threads's ThreadsList. >> >> >>> >>> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >>> >>> Can you add a comment about where this number came from? >> >> I'll have to get that from Erik... >> >> >>> I can't find the caller of threads_do_smr. >> >> I'm guessing that's used by the GC code that depends on Thread-SMR... >> > > They? should add this API when the add the use of it then.? I don't > see it in any sources. We'll see what Erik wants to do... > >> >>> If these functions xchg_smr_thread_list, get_smr_java_thread_list, >>> inc_smr_deleted_thread_count are only used by thread.cpp, I think >>> they should go in that file and not in the .inline.hpp file to be >>> included and possibly called by other files.? I think they're >>> private to class Threads. >> >> I have a vague memory that some of the compilers don't do inlining when >> an "inline" function is in a .cpp. I believe we want these functions >> to be inlined for performance reasons. Erik should probably chime in >> here. > > I don't see why this should be.? Maybe some (older) compilers require > it to be found before the call though, but that can still be > accomplished in the .cpp file. Again, we'll see what Erik wants to do... >> >> >>> I don't have any in-depth comments. This looks really good from my >>> day of reading it. >> >> Thanks for taking the time to review it again! >> > > Thanks, > Coleen And thanks again... Dan > >> Dan >> >> >>> >>> Thanks, >>> Coleen >>> >>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>> >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>> test, and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are >>>>> this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final >>>>> review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>> code quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>> server, and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> > From erik.joelsson at oracle.com Tue Nov 21 17:49:56 2017 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Tue, 21 Nov 2017 09:49:56 -0800 Subject: RFR: JDK-8191203 Remove duplicated jimage.hpp In-Reply-To: References: Message-ID: Looks good. /Erik On 2017-11-21 03:13, Magnus Ihse Bursie wrote: > I ran a duplicate name check on the source base. Then I discovered > that there is a jimage.hpp in both > src/hotspot/share/classfile/jimage.hpp and > src/java.base/share/native/libjimage/jimage.hpp. They are identical > (apart from copyright headers), and once again, we shouldn't have > both. David Holmes suggested removing the hotspot version, so this > I've done. > > I intend to push this to jdk/hs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191203 > WebRev: > http://cr.openjdk.java.net/~ihse/JDK-8191203-remove-duplicated-jimage-hpp/webrev.01 > > /Magnus From calvin.cheung at oracle.com Tue Nov 21 23:30:08 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 21 Nov 2017 15:30:08 -0800 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 Message-ID: <5A14B700.9050309@oracle.com> The fix is to add the following import: 36 import jdk.test.lib.Platform; I've also re-arranged the import statements in alphabetical order. Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ Tested locally on linux-x64. thanks, Calvin From daniel.daugherty at oracle.com Tue Nov 21 23:33:26 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 18:33:26 -0500 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 In-Reply-To: <5A14B700.9050309@oracle.com> References: <5A14B700.9050309@oracle.com> Message-ID: <1d486c7f-c5c3-80fa-fa4c-d184491b6069@oracle.com> Thumbs up. Dan On 11/21/17 6:30 PM, Calvin Cheung wrote: > The fix is to add the following import: > > 36 import jdk.test.lib.Platform; > > > I've also re-arranged the import statements in alphabetical order. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 > > webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ > > Tested locally on linux-x64. > > thanks, > Calvin > From daniel.daugherty at oracle.com Tue Nov 21 23:35:37 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 18:35:37 -0500 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 In-Reply-To: <1d486c7f-c5c3-80fa-fa4c-d184491b6069@oracle.com> References: <5A14B700.9050309@oracle.com> <1d486c7f-c5c3-80fa-fa4c-d184491b6069@oracle.com> Message-ID: This change qualifies as trivial under HotSpot rules so it only needs one (R)eviewer and does not need to wait for 24 hours. Dan On 11/21/17 6:33 PM, Daniel D. Daugherty wrote: > Thumbs up. > > Dan > > > On 11/21/17 6:30 PM, Calvin Cheung wrote: >> The fix is to add the following import: >> >> 36 import jdk.test.lib.Platform; >> >> >> I've also re-arranged the import statements in alphabetical order. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ >> >> Tested locally on linux-x64. >> >> thanks, >> Calvin >> > > From calvin.cheung at oracle.com Tue Nov 21 23:36:27 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 21 Nov 2017 15:36:27 -0800 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 In-Reply-To: <1d486c7f-c5c3-80fa-fa4c-d184491b6069@oracle.com> References: <5A14B700.9050309@oracle.com> <1d486c7f-c5c3-80fa-fa4c-d184491b6069@oracle.com> Message-ID: <5A14B87B.8090403@oracle.com> Thanks - Dan. Calvin On 11/21/17, 3:33 PM, Daniel D. Daugherty wrote: > Thumbs up. > > Dan > > > On 11/21/17 6:30 PM, Calvin Cheung wrote: >> The fix is to add the following import: >> >> 36 import jdk.test.lib.Platform; >> >> >> I've also re-arranged the import statements in alphabetical order. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ >> >> Tested locally on linux-x64. >> >> thanks, >> Calvin >> > From david.holmes at oracle.com Tue Nov 21 23:40:33 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 22 Nov 2017 09:40:33 +1000 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 In-Reply-To: <5A14B700.9050309@oracle.com> References: <5A14B700.9050309@oracle.com> Message-ID: Looks good. FYI you never need to import anything from java.lang Thanks, David On 22/11/2017 9:30 AM, Calvin Cheung wrote: > The fix is to add the following import: > > 36 import jdk.test.lib.Platform; > > > I've also re-arranged the import statements in alphabetical order. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 > > webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ > > Tested locally on linux-x64. > > thanks, > Calvin > From calvin.cheung at oracle.com Tue Nov 21 23:45:18 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 21 Nov 2017 15:45:18 -0800 Subject: RFR(xxs): 8191739: [TESTBUG] test/hotspot/jtreg/runtime/LoadClass/TestResize.java fails to compile after JDK-8191580 In-Reply-To: References: <5A14B700.9050309@oracle.com> Message-ID: <5A14BA8E.8010003@oracle.com> On 11/21/17, 3:40 PM, David Holmes wrote: > Looks good. Thanks for your review, David. > > FYI you never need to import anything from java.lang Noted. thanks, Calvin > > Thanks, > David > > On 22/11/2017 9:30 AM, Calvin Cheung wrote: >> The fix is to add the following import: >> >> 36 import jdk.test.lib.Platform; >> >> >> I've also re-arranged the import statements in alphabetical order. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191739 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8191739/webrev.00/ >> >> Tested locally on linux-x64. >> >> thanks, >> Calvin >> From daniel.daugherty at oracle.com Wed Nov 22 00:12:49 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 21 Nov 2017 19:12:49 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> Message-ID: <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> Greetings, *** We are wrapping up code review on this project so it is time *** *** for the various code reviewers to chime in one last time. *** In this latest round, we had three different reviewers chime in so we're doing delta webrevs for each of those resolutions: David H's resolutions: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ ? mq comment for dholmes_CR: ??? - Fix indents, trailing spaces and various typos. ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, ????? add impl notes to document the type choices. ? src/hotspot/share/runtime/globals.hpp change is white-space only so it ? is only visible via the file's patch link. Robin W's resolutions: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ ? mq comment for robinw_CR: ??? - Fix some inefficient code, update some comments, fix some indents, ????? and add some 'const' specifiers. ? src/hotspot/share/runtime/vm_operations.hpp change is white-space only so ? it is only visible via the file's patch link. Coleen's resolutions: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ ? mq comment for coleenp_CR: ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', ??? - add header comment to threadSMR.hpp file, ??? - cleanup JavaThreadIteratorWithHandle ctr, ??? - make ErrorHandling more efficient. Some folks prefer to see all of the delta webrevs together so here is a delta webrev relative to jdk10-09-full: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ ? src/hotspot/share/runtime/globals.hpp and ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space only ? so they are only visible via the file's patch link. And, of course, some folks prefer to see everything all at once so here is the latest full webrev: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... The project is currently baselined on Jesper's 2017-11-17 PIT snapshot so there will be some catching up to do before integration... We welcome comments, suggestions and feedback. Dan, Erik and Robbin On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: > Greetings, > > Testing of the last round of changes revealed a hang in one of the new > TLH tests. Robbin has fixed the hang, updated the existing TLH test, and > added another TLH test for good measure. > > Here is the updated full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ > > Here is the updated delta webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ > > Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are > no unexpected failures. > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Robbin rebased the project last night/this morning to merge with Thread >> Local Handshakes (TLH) and also picked up additional changesets up thru: >> >>> Changeset: fa736014cf28 >>> Author:??? cjplummer >>> Date:????? 2017-11-14 18:08 -0800 >>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>> >>> 8191049: Add alternate version of pns() that is callable from within >>> hotspot source >>> Summary: added pns2() to debug.cpp >>> Reviewed-by: stuefe, gthornbr >> >> This is the first time we've rebased the project to bits that are this >> fresh (< 12 hours old at rebase time). We've done this because we think >> we're done with this project and are in the final review-change-retest >> cycle(s)... In other words, we're not planning on making any more major >> changes for this project... :-) >> >> *** Time for code reviewers to chime in on this thread! *** >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >> >> Here's is the delta webrev from the 2017.11.10 rebase: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >> (and the new baseline also)... >> >> We're expecting this round to be a little noisier than usual because >> we did not rebase on a PIT snapshot so the new baseline has not been >> through Jesper's usual care-and-feeding of integration_blockers, etc. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>> (Yes, we're playing chase-the-repo...) >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>> >>> Unlike the previous rebase, there were no changes required to the >>> open code to get the rebased bits to build so there is no delta >>> webrev. >>> >>> These bits have been run through JPRT and I've submitted the usual >>> Mach5 tier[1-5] test run... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>> >>>> Here are the updated webrevs: >>>> >>>> Here's the mq comment for the change: >>>> >>>> ? Rebase to 2017.10.25 PIT snapshot. >>>> >>>> Here is the full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>> >>>> And here is the delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>> >>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>> look today. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Resolving one of the code review comments (from both Stefan K and >>>>> Coleen) >>>>> on jdk10-04-full required quite a few changes so it is being done >>>>> as a >>>>> standalone patch and corresponding webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>> ??? JavaThreadIteratorWithHandle. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> We have a (eXtra Large) fix for the following bug: >>>>>> >>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>> >>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>> >>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>> and track the work on this project: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>> >>>>>> >>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>> quotes >>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>> have a >>>>>> solution for that problem yet. >>>>>> >>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>> >>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>> tier[2-5] >>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>> additional testing on Erik and Robbin's machines. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Daniel Daugherty >>>>>> Erik Osterlund >>>>>> Robbin Ehn >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > From kevin.walls at oracle.com Wed Nov 22 00:38:40 2017 From: kevin.walls at oracle.com (Kevin Walls) Date: Wed, 22 Nov 2017 00:38:40 +0000 Subject: RFR(8u): 8038636, 8055008, 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links ...and 8057570. In-Reply-To: References: Message-ID: <5d040ff1-4a44-7778-a093-22c3826d49dc@oracle.com> Thanks Coleen! On 16/11/2017 23:08, coleen.phillimore at oracle.com wrote: > > Hi Kevin, These changes look good.? It's nice to have the simpler > version of the InstanceKlass::_previous_versions in jdk8.? There was a > bug tail so there may be more changes after 8040237.? It's worth > following through all the bugs to get something close to what we have > in jdk9/10.? But this change looks good so far. > > Thanks, > Coleen > > On 11/13/17 1:08 PM, Kevin Walls wrote: >> Hi, >> >> I'd like to get a hotspot review of these backports from 9 to 8u. >> This is mainly runtime territory but some of the history is from the >> compiler side. >> >> webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ >> >> The one we need is: >> >> 8156137: SIGSEGV in ReceiverTypeData::clean_weak_klass_links >> jbs: https://bugs.openjdk.java.net/browse/JDK-8156137 >> 9 changeset: >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/882e8cda60b3 >> >> The 9 changeset is short, just changing klass.cpp, but not possible >> in 8 without additional work, so... >> >> 1. >> >> 8038636: speculative traps break when classes are redefined >> https://bugs.openjdk.java.net/browse/JDK-8038636 >> >> 9 changeset: >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/a7784ddacbef >> >> 9 review thread: >> http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2014-April/013948.html >> (in 9 since April 2014) >> >> That one imports cleanly into 8u. >> >> >> 2. >> >> With that imported, we need: >> >> 8055008:Clean up code that saves the previous versions of redefined >> classes >> https://bugs.openjdk.java.net/browse/JDK-8055008 >> >> 9 changeset: >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/e3fb51ae8d7d >> >> This is the change to stop using PreviousVersionNode and get an >> InstanceKlass* from previous_versions(). >> >> >> My webrev: http://cr.openjdk.java.net/~kevinw/8055008.8156137/webrev.00/ >> >> ...is 8055008 and 8156137 in 8u. The klass.cpp change here is >> 8156137.? The rest is 8055008, so I can commit them separately. >> >> 8156137 doesn't have its own test, but I have found if you get this >> area wrong, the test runtime/RedefineObject crashes. >> >> >> Notes: >> For 8055008 there was a change in >> src/share/vm/classfile/classLoaderData.cpp which is already in 8u. >> >> src/share/vm/classfile/metadataOnStackMark.cpp: >> +MetadataOnStackMark::MetadataOnStackMark(bool has_redefined_a_class) { >> >> The change is to stop using JvmtiExport::has_redefined_a_class() here >> which is already in 8u at this point, but the param is already there >> with a different name, so renamed as per 8055008. >> >> universe.cpp: had a change in 8055008 but following that there was: >> >> 8057570: RedefineClasses() tests fail >> assert(((Metadata*)obj)->is_valid()) failed: obj is valid >> https://bugs.openjdk.java.net/browse/JDK-8057570 >> 9 changeset: >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/479ed4234a9d >> >> ..this puts back a few lines in nmethod.cpp which 8055008 removed: >> >> + // Call function Method*, not embedded in these other places. >> + if (_method != NULL) f(_method); >> >> ..and in universe.cpp: >> >> + // Make the dependent methods not entrant (in VM_Deoptimize they >> are made zombies) >> + CodeCache::make_marked_nmethods_not_entrant(); >> >> ..and removes: CodeCache::make_marked_nmethods_zombies(); >> >> My webrev takes these into account. >> >> >> Then there are 8038636's two follow-on bugs which I'd like to >> follow-up in a separate thread. >> 8039960 is a test change >> 8040237: nsk/jvmti/RetransformClasses/retransform001 crashed the VM >> on all platforms when run with with -server -Xcomp >> https://bugs.openjdk.java.net/browse/JDK-8040237 >> >> >> Thanks! >> Kevin >> >> >> > From david.holmes at oracle.com Wed Nov 22 04:26:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 22 Nov 2017 14:26:39 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> Message-ID: <4d5d448e-3902-8465-197e-18a5607fbd70@oracle.com> Hi Dan, On 22/11/2017 10:12 AM, Daniel D. Daugherty wrote: > Greetings, > > *** We are wrapping up code review on this project so it is time *** > *** for the various code reviewers to chime in one last time. *** > > In this latest round, we had three different reviewers chime in so we're > doing delta webrevs for each of those resolutions: > > David H's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ > > > ? mq comment for dholmes_CR: > ??? - Fix indents, trailing spaces and various typos. > ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, > ????? add impl notes to document the type choices. src/hotspot/share/runtime/thread.cpp 3466 // range. Unsigned 64-bit would be more future proof, but 64-bit atomic inc 3467 // isn't available everywhere (or is it?). 64-bit atomics should be available on all currently supported platforms (the last time this was an issue was for PPC32 - and the atomic templates should have filled in any previous holes in the allowed types). But if it's always 64-bit then you'd have to use Atomic::load to allow for 32-bit platforms. These counts etc are all just for statistics so we aren't really concerned about eventual overflows in long running VMs - right? --- // # of parallel threads in _smr_delete_lock->wait(). static uint _smr_delete_lock_wait_cnt; // Max # of parallel threads in _smr_delete_lock->wait(). why are the comments indented like that? I thought this is what Coleen previously commented on. Please just start the comments in the first column - no need for any indent. Thanks. > ? src/hotspot/share/runtime/globals.hpp change is white-space only so it > ? is only visible via the file's patch link. > > > Robin W's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ > > > ? mq comment for robinw_CR: > ??? - Fix some inefficient code, update some comments, fix some indents, > ????? and add some 'const' specifiers. > > ? src/hotspot/share/runtime/vm_operations.hpp change is white-space > only so > ? it is only visible via the file's patch link. All looks fine to me. Some good catches there from Robin! > Coleen's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ > > > ? mq comment for coleenp_CR: > ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', > ??? - add header comment to threadSMR.hpp file, > ??? - cleanup JavaThreadIteratorWithHandle ctr, > ??? - make ErrorHandling more efficient. src/hotspot/share/runtime/threadSMR.hpp + // Thread Safe Memory Reclaimation (Thread-SMR) support. Only one 'i' in Reclamation :) Thanks, David ------ > > Some folks prefer to see all of the delta webrevs together so here is > a delta webrev relative to jdk10-09-full: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ > > > ? src/hotspot/share/runtime/globals.hpp and > ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space only > ? so they are only visible via the file's patch link. > > > And, of course, some folks prefer to see everything all at once so here > is the latest full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ > > > Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The > prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... > > The project is currently baselined on Jesper's 2017-11-17 PIT snapshot > so there will be some catching up to do before integration... > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up thru: >>> >>>> Changeset: fa736014cf28 >>>> Author:??? cjplummer >>>> Date:????? 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from within >>>> hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and >>>>>> Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done >>>>>> as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>>> ??? JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>> quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>> have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>> tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From thomas.stuefe at gmail.com Wed Nov 22 08:17:44 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 22 Nov 2017 09:17:44 +0100 Subject: disassembler output in hs-err? Message-ID: Hi all, I keep forgetting, how can I get disassembled output in an hs-err file for the instructions at the crash point? All I have is a hex dump. I'm on Linux x64 (ubuntu 16.4) Thanks! ..Thomas From shade at redhat.com Wed Nov 22 08:21:26 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 22 Nov 2017 09:21:26 +0100 Subject: disassembler output in hs-err? In-Reply-To: References: Message-ID: On 11/22/2017 09:17 AM, Thomas St?fe wrote: > I keep forgetting, how can I get disassembled output in an hs-err file for > the instructions at the crash point? All I have is a hex dump. > > I'm on Linux x64 (ubuntu 16.4) IIRC, you "just" need to have hsdis in your build. I have this handy alias: $ alias hsdis='cp ~/Install/libhsdis-amd64.so jre/lib/amd64/server/libhsdis-amd64.so; cp ~/Install/libhsdis-amd64.so lib/amd64/server/libhsdis-amd64.so; cp ~/Install/libhsdis-amd64.so lib/server/libhsdis-amd64.so; cp ~/Install/libhsdis-i386.so lib/i386/server/libhsdis-i386.so;' If you already have only the hex dump, you can disassemble it. This works for me fine: https://onlinedisassembler.com/odaweb/ Thanks, -Aleksey From thomas.stuefe at gmail.com Wed Nov 22 08:24:23 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 22 Nov 2017 09:24:23 +0100 Subject: disassembler output in hs-err? In-Reply-To: References: Message-ID: Thanks Aleksey! On Wed, Nov 22, 2017 at 9:21 AM, Aleksey Shipilev wrote: > On 11/22/2017 09:17 AM, Thomas St?fe wrote: > > I keep forgetting, how can I get disassembled output in an hs-err file > for > > the instructions at the crash point? All I have is a hex dump. > > > > I'm on Linux x64 (ubuntu 16.4) > > IIRC, you "just" need to have hsdis in your build. I have this handy alias: > > $ alias hsdis='cp ~/Install/libhsdis-amd64.so > jre/lib/amd64/server/libhsdis-amd64.so; cp > ~/Install/libhsdis-amd64.so lib/amd64/server/libhsdis-amd64.so; cp > ~/Install/libhsdis-amd64.so > lib/server/libhsdis-amd64.so; cp ~/Install/libhsdis-i386.so > lib/i386/server/libhsdis-i386.so;' > > If you already have only the hex dump, you can disassemble it. This works > for me fine: > https://onlinedisassembler.com/odaweb/ > > Thanks, > -Aleksey > > From erik.osterlund at oracle.com Wed Nov 22 09:07:30 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 22 Nov 2017 10:07:30 +0100 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> Message-ID: <5A153E52.8000704@oracle.com> Hi, Some replies... On 2017-11-21 17:28, Daniel D. Daugherty wrote: > Hi Coleen! > > Thanks for making time to review the Thread-SMR stuff again!! > > I have added back the other three OpenJDK aliases... This review is > being done on _four_ different OpenJDK aliases. > > As always, replies are embedded below... > > > On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >> >> >> 32 ThreadsList::ThreadsList(int entries) : _length(entries), >> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >> _next_list(NULL) { >> >> Seems like it should be mtThread rather than mtGC. > > Fixed. Definitely an artifact of Erik's original prototype when he > extracted Thread-SMR from his GC work... Thanks for catching it. > Confirmed. At the time I considered the Threads list overheads GC overheads, but I agree mtThread is a better fit today. > >> >> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >> >> Can you add a comment about where this number came from? > > I'll have to get that from Erik... Wow, that looks like code I wrote a *very* long time ago. :) That is a variation of Knuth's multiplicative hash which is outlined in a comment in synchronizer.cpp and referred to in that comment as a phi-based scheme. Basically the magic number is 2^32 * Phi (the golden ratio), which happens to be a useful value for building a reasonably simple yet pretty good hash function. It is not the optimal hash function, but seemed to definitely be good enough for our purposes. Thanks, /Erik From erik.osterlund at oracle.com Wed Nov 22 09:48:32 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 22 Nov 2017 10:48:32 +0100 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> Message-ID: <5A1547F0.7040206@oracle.com> Hi, Some replies... On 2017-11-21 18:36, Daniel D. Daugherty wrote: > Thanks for keeping all the OpenJDK aliases! > > > On 11/21/17 12:24 PM, coleen.phillimore at oracle.com wrote: >> >> >> On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: >>> >>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>> >>>> I can't find the caller of threads_do_smr. >>> >>> I'm guessing that's used by the GC code that depends on Thread-SMR... >>> >> >> They should add this API when the add the use of it then. I don't >> see it in any sources. > > We'll see what Erik wants to do... The new iterators should be more than enough, so that closure-based API is not going to be missed when removed. > >> >>> >>>> If these functions xchg_smr_thread_list, get_smr_java_thread_list, >>>> inc_smr_deleted_thread_count are only used by thread.cpp, I think >>>> they should go in that file and not in the .inline.hpp file to be >>>> included and possibly called by other files. I think they're >>>> private to class Threads. >>> >>> I have a vague memory that some of the compilers don't do inlining when >>> an "inline" function is in a .cpp. I believe we want these functions >>> to be inlined for performance reasons. Erik should probably chime in >>> here. >> >> I don't see why this should be. Maybe some (older) compilers require >> it to be found before the call though, but that can still be >> accomplished in the .cpp file. > > Again, we'll see what Erik wants to do... I don't mind. Either file works for me. For me it's more intuitive if inline member function definitions are in the .inline.hpp files. But if there are strong forces to move this to the cpp file, then sure. Thanks, /Erik From daniel.daugherty at oracle.com Wed Nov 22 12:51:10 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 07:51:10 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <4d5d448e-3902-8465-197e-18a5607fbd70@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <4d5d448e-3902-8465-197e-18a5607fbd70@oracle.com> Message-ID: <4fd06a59-f6b5-33b2-53f5-7bfc96f866a7@oracle.com> On 11/21/17 11:26 PM, David Holmes wrote: > Hi Dan, > > On 22/11/2017 10:12 AM, Daniel D. Daugherty wrote: >> Greetings, >> >> *** We are wrapping up code review on this project so it is time *** >> *** for the various code reviewers to chime in one last time. *** >> >> In this latest round, we had three different reviewers chime in so we're >> doing delta webrevs for each of those resolutions: >> >> David H's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >> >> >> ?? mq comment for dholmes_CR: >> ???? - Fix indents, trailing spaces and various typos. >> ???? - Add descriptions for the '_cnt', '_max' and '_times" fields, >> ?????? add impl notes to document the type choices. > > src/hotspot/share/runtime/thread.cpp > > 3466 // range. Unsigned 64-bit would be more future proof, but 64-bit > atomic inc > 3467 // isn't available everywhere (or is it?). > > 64-bit atomics should be available on all currently supported > platforms (the last time this was an issue was for PPC32 - and the > atomic templates should have filled in any previous holes in the > allowed types). Hmmm... I ran into this when I merged the project with the atomic templates. I got a JPRT build failure on one of the ARM platforms... Unfortunately, I didn't save the log files so I can't quote the error message from the template stuff... > But if it's always 64-bit then you'd have to use Atomic::load to allow > for 32-bit platforms. What's the official story on 32-bit platforms? Is that what bit me in JPRT? i.e., did we still have a 32-bit ARM build config'ed in JPRT a month or two ago??? > These counts etc are all just for statistics so we aren't really > concerned about eventual overflows in long running VMs - right? Yup these are just statistics... overflow is not a killer. > > --- > > ?????????????????????????????????????? // # of parallel threads in > _smr_delete_lock->wait(). > ? static uint????????????????? _smr_delete_lock_wait_cnt; > ?????????????????????????????????????? // Max # of parallel threads in > _smr_delete_lock->wait(). > > > why are the comments indented like that? I thought this is what Coleen > previously commented on. Please just start the comments in the first > column - no need for any indent. Thanks. Coleen asked for the new comments in thread.cpp to be moved over to column 1. She asked for the new comments in thread.hpp to be indented with more spaces. I started with 2 spaces and the variables still didn't stand out. I tried 4 spaces and they still didn't stand out probably because of the leading _smr_... So I went with 8 spaces... > >> ?? src/hotspot/share/runtime/globals.hpp change is white-space only >> so it >> ?? is only visible via the file's patch link. >> >> >> Robin W's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >> >> >> ?? mq comment for robinw_CR: >> ???? - Fix some inefficient code, update some comments, fix some >> indents, >> ?????? and add some 'const' specifiers. >> >> ?? src/hotspot/share/runtime/vm_operations.hpp change is white-space >> only so >> ?? it is only visible via the file's patch link. > > All looks fine to me. Some good catches there from Robin! Yes, a new pair of eyes on this code is always useful. > >> Coleen's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >> >> >> ?? mq comment for coleenp_CR: >> ???? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >> ???? - add header comment to threadSMR.hpp file, >> ???? - cleanup JavaThreadIteratorWithHandle ctr, >> ???? - make ErrorHandling more efficient. > > src/hotspot/share/runtime/threadSMR.hpp > > + // Thread Safe Memory Reclaimation (Thread-SMR) support. > > Only one 'i' in Reclamation :) Will fix. I tried typing both and neither looked right to me. (I might be getting blurry eyed with this stuff)... So I went with the spelling from Coleen's comment... :-) Thanks for taking another look!! Dan > > Thanks, > David > ------ > >> >> Some folks prefer to see all of the delta webrevs together so here is >> a delta webrev relative to jdk10-09-full: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >> >> >> ?? src/hotspot/share/runtime/globals.hpp and >> ?? src/hotspot/share/runtime/vm_operations.hpp changes are >> white-space only >> ?? so they are only visible via the file's patch link. >> >> >> And, of course, some folks prefer to see everything all at once so here >> is the latest full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >> >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >> >> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >> so there will be some catching up to do before integration... >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, >>> and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with >>>> Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up >>>> thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from >>>>> within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we >>>> think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more >>>> major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>> holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>> closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>> and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>> done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>> to use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>> describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>> quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>> have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>> tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, >>>>>>>> and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> From daniel.daugherty at oracle.com Wed Nov 22 12:53:16 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 07:53:16 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5A153E52.8000704@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <5A153E52.8000704@oracle.com> Message-ID: On 11/22/17 4:07 AM, Erik ?sterlund wrote: > Hi, > > Some replies... > > On 2017-11-21 17:28, Daniel D. Daugherty wrote: >> Hi Coleen! >> >> Thanks for making time to review the Thread-SMR stuff again!! >> >> I have added back the other three OpenJDK aliases... This review is >> being done on _four_ different OpenJDK aliases. >> >> As always, replies are embedded below... >> >> >> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >>> >>> >>> ? 32 ThreadsList::ThreadsList(int entries) : _length(entries), >>> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >>> _next_list(NULL) { >>> >>> Seems like it should be mtThread rather than mtGC. >> >> Fixed. Definitely an artifact of Erik's original prototype when he >> extracted Thread-SMR from his GC work... Thanks for catching it. >> > > Confirmed. At the time I considered the Threads list overheads GC > overheads, but I agree mtThread is a better fit today. > >> >>> >>> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >>> >>> Can you add a comment about where this number came from? >> >> I'll have to get that from Erik... > > Wow, that looks like code I wrote a *very* long time ago. :) That is a > variation of Knuth's multiplicative hash which is outlined in a > comment in synchronizer.cpp and referred to in that comment as a > phi-based scheme. Basically the magic number is 2^32 * Phi (the golden > ratio), which happens to be a useful value for building a reasonably > simple yet pretty good hash function. It is not the optimal hash > function, but seemed to definitely be good enough for our purposes. So a reasonable comment would be: // The literal value is 2^32 * Phi (the golden ratio). If so, then I'll add it in my wrap up round... Dan > > Thanks, > /Erik From daniel.daugherty at oracle.com Wed Nov 22 12:54:51 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 07:54:51 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5A1547F0.7040206@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> <5A1547F0.7040206@oracle.com> Message-ID: <6906e0e0-e5cc-a415-6db2-13eff8d76ca8@oracle.com> On 11/22/17 4:48 AM, Erik ?sterlund wrote: > Hi, > > Some replies... > > On 2017-11-21 18:36, Daniel D. Daugherty wrote: >> Thanks for keeping all the OpenJDK aliases! >> >> >> On 11/21/17 12:24 PM, coleen.phillimore at oracle.com wrote: >>> >>> >>> On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: >>>> >>>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>>> >>>>> I can't find the caller of threads_do_smr. >>>> >>>> I'm guessing that's used by the GC code that depends on Thread-SMR... >>>> >>> >>> They? should add this API when the add the use of it then.? I don't >>> see it in any sources. >> >> We'll see what Erik wants to do... > > The new iterators should be more than enough, so that closure-based > API is not going to be missed when removed. I will delete it in the wrap up round. > >> >>> >>>> >>>>> If these functions xchg_smr_thread_list, get_smr_java_thread_list, >>>>> inc_smr_deleted_thread_count are only used by thread.cpp, I think >>>>> they should go in that file and not in the .inline.hpp file to be >>>>> included and possibly called by other files.? I think they're >>>>> private to class Threads. >>>> >>>> I have a vague memory that some of the compilers don't do inlining >>>> when >>>> an "inline" function is in a .cpp. I believe we want these functions >>>> to be inlined for performance reasons. Erik should probably chime in >>>> here. >>> >>> I don't see why this should be.? Maybe some (older) compilers >>> require it to be found before the call though, but that can still be >>> accomplished in the .cpp file. >> >> Again, we'll see what Erik wants to do... > > I don't mind. Either file works for me. For me it's more intuitive if > inline member function definitions are in the .inline.hpp files. But > if there are strong forces to move this to the cpp file, then sure. I prefer inline member function definitions in the .inline.hpp files. (There might even be a style guide note about this...) Coleen, are you okay if we leave them there? Dan > > Thanks, > /Erik > From coleen.phillimore at oracle.com Wed Nov 22 13:01:27 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 22 Nov 2017 08:01:27 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <4fd06a59-f6b5-33b2-53f5-7bfc96f866a7@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <4d5d448e-3902-8465-197e-18a5607fbd70@oracle.com> <4fd06a59-f6b5-33b2-53f5-7bfc96f866a7@oracle.com> Message-ID: On 11/22/17 7:51 AM, Daniel D. Daugherty wrote: > On 11/21/17 11:26 PM, David Holmes wrote: >> Hi Dan, >> >> On 22/11/2017 10:12 AM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> *** We are wrapping up code review on this project so it is time *** >>> *** for the various code reviewers to chime in one last time. *** >>> >>> In this latest round, we had three different reviewers chime in so >>> we're >>> doing delta webrevs for each of those resolutions: >>> >>> David H's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >>> >>> >>> ?? mq comment for dholmes_CR: >>> ???? - Fix indents, trailing spaces and various typos. >>> ???? - Add descriptions for the '_cnt', '_max' and '_times" fields, >>> ?????? add impl notes to document the type choices. >> >> src/hotspot/share/runtime/thread.cpp >> >> 3466 // range. Unsigned 64-bit would be more future proof, but 64-bit >> atomic inc >> 3467 // isn't available everywhere (or is it?). >> >> 64-bit atomics should be available on all currently supported >> platforms (the last time this was an issue was for PPC32 - and the >> atomic templates should have filled in any previous holes in the >> allowed types). > > Hmmm... I ran into this when I merged the project with the atomic > templates. I got a JPRT build failure on one of the ARM platforms... > Unfortunately, I didn't save the log files so I can't quote the > error message from the template stuff... > > >> But if it's always 64-bit then you'd have to use Atomic::load to >> allow for 32-bit platforms. > > What's the official story on 32-bit platforms? Is that what > bit me in JPRT? i.e., did we still have a 32-bit ARM build > config'ed in JPRT a month or two ago??? > > >> These counts etc are all just for statistics so we aren't really >> concerned about eventual overflows in long running VMs - right? > > Yup these are just statistics... overflow is not a killer. > > >> >> --- >> >> ?????????????????????????????????????? // # of parallel threads in >> _smr_delete_lock->wait(). >> ? static uint????????????????? _smr_delete_lock_wait_cnt; >> ?????????????????????????????????????? // Max # of parallel threads >> in _smr_delete_lock->wait(). >> >> >> why are the comments indented like that? I thought this is what >> Coleen previously commented on. Please just start the comments in the >> first column - no need for any indent. Thanks. > > Coleen asked for the new comments in thread.cpp to be moved > over to column 1. She asked for the new comments in thread.hpp > to be indented with more spaces. I started with 2 spaces and > the variables still didn't stand out. I tried 4 spaces and > they still didn't stand out probably because of the leading > _smr_... So I went with 8 spaces... Since you have the same but more indepth comments in the .cpp file, and at least for my sampling the comments in the .hpp file look redundant, I think you should not have the same comments in the .hpp file.?? They're really values that the implementation uses, not part of the interface.? My vote is to remove the comments from the .hpp file. Coleen > > >> >>> ?? src/hotspot/share/runtime/globals.hpp change is white-space only >>> so it >>> ?? is only visible via the file's patch link. >>> >>> >>> Robin W's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >>> >>> >>> ?? mq comment for robinw_CR: >>> ???? - Fix some inefficient code, update some comments, fix some >>> indents, >>> ?????? and add some 'const' specifiers. >>> >>> ?? src/hotspot/share/runtime/vm_operations.hpp change is white-space >>> only so >>> ?? it is only visible via the file's patch link. >> >> All looks fine to me. Some good catches there from Robin! > > Yes, a new pair of eyes on this code is always useful. > > >> >>> Coleen's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >>> >>> >>> ?? mq comment for coleenp_CR: >>> ???? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >>> ???? - add header comment to threadSMR.hpp file, >>> ???? - cleanup JavaThreadIteratorWithHandle ctr, >>> ???? - make ErrorHandling more efficient. >> >> src/hotspot/share/runtime/threadSMR.hpp >> >> + // Thread Safe Memory Reclaimation (Thread-SMR) support. >> >> Only one 'i' in Reclamation :) > > Will fix. I tried typing both and neither looked right to me. > (I might be getting blurry eyed with this stuff)... > So I went with the spelling from Coleen's comment... :-) > > Thanks for taking another look!! > > Dan > > >> >> Thanks, >> David >> ------ >> >>> >>> Some folks prefer to see all of the delta webrevs together so here is >>> a delta webrev relative to jdk10-09-full: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >>> >>> >>> ?? src/hotspot/share/runtime/globals.hpp and >>> ?? src/hotspot/share/runtime/vm_operations.hpp changes are >>> white-space only >>> ?? so they are only visible via the file's patch link. >>> >>> >>> And, of course, some folks prefer to see everything all at once so here >>> is the latest full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >>> >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >>> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >>> >>> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >>> so there will be some catching up to do before integration... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>> test, and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are >>>>> this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final >>>>> review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>> code quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>> server, and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> > From coleen.phillimore at oracle.com Wed Nov 22 13:02:32 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 22 Nov 2017 08:02:32 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <6906e0e0-e5cc-a415-6db2-13eff8d76ca8@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> <5A1547F0.7040206@oracle.com> <6906e0e0-e5cc-a415-6db2-13eff8d76ca8@oracle.com> Message-ID: <50fdb219-4aa2-bdcc-cf69-8f60c935208f@oracle.com> On 11/22/17 7:54 AM, Daniel D. Daugherty wrote: > On 11/22/17 4:48 AM, Erik ?sterlund wrote: >> Hi, >> >> Some replies... >> >> On 2017-11-21 18:36, Daniel D. Daugherty wrote: >>> Thanks for keeping all the OpenJDK aliases! >>> >>> >>> On 11/21/17 12:24 PM, coleen.phillimore at oracle.com wrote: >>>> >>>> >>>> On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: >>>>> >>>>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>>>> >>>>>> I can't find the caller of threads_do_smr. >>>>> >>>>> I'm guessing that's used by the GC code that depends on Thread-SMR... >>>>> >>>> >>>> They? should add this API when the add the use of it then. I don't >>>> see it in any sources. >>> >>> We'll see what Erik wants to do... >> >> The new iterators should be more than enough, so that closure-based >> API is not going to be missed when removed. > > I will delete it in the wrap up round. > > Thank you. >> >>> >>>> >>>>> >>>>>> If these functions xchg_smr_thread_list, >>>>>> get_smr_java_thread_list, inc_smr_deleted_thread_count are only >>>>>> used by thread.cpp, I think they should go in that file and not >>>>>> in the .inline.hpp file to be included and possibly called by >>>>>> other files.? I think they're private to class Threads. >>>>> >>>>> I have a vague memory that some of the compilers don't do inlining >>>>> when >>>>> an "inline" function is in a .cpp. I believe we want these functions >>>>> to be inlined for performance reasons. Erik should probably chime in >>>>> here. >>>> >>>> I don't see why this should be.? Maybe some (older) compilers >>>> require it to be found before the call though, but that can still >>>> be accomplished in the .cpp file. >>> >>> Again, we'll see what Erik wants to do... >> >> I don't mind. Either file works for me. For me it's more intuitive if >> inline member function definitions are in the .inline.hpp files. But >> if there are strong forces to move this to the cpp file, then sure. > > I prefer inline member function definitions in the .inline.hpp files. > (There might even be a style guide note about this...) > > Coleen, are you okay if we leave them there? Yes, that's fine. Coleen > > Dan > > >> >> Thanks, >> /Erik >> > From erik.osterlund at oracle.com Wed Nov 22 13:06:50 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 22 Nov 2017 14:06:50 +0100 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <5A153E52.8000704@oracle.com> Message-ID: <5A15766A.1090701@oracle.com> Hi Dan, On 2017-11-22 13:53, Daniel D. Daugherty wrote: > On 11/22/17 4:07 AM, Erik ?sterlund wrote: >> Hi, >> >> Some replies... >> >> On 2017-11-21 17:28, Daniel D. Daugherty wrote: >>> Hi Coleen! >>> >>> Thanks for making time to review the Thread-SMR stuff again!! >>> >>> I have added back the other three OpenJDK aliases... This review is >>> being done on _four_ different OpenJDK aliases. >>> >>> As always, replies are embedded below... >>> >>> >>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >>>> >>>> >>>> 32 ThreadsList::ThreadsList(int entries) : _length(entries), >>>> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >>>> _next_list(NULL) { >>>> >>>> Seems like it should be mtThread rather than mtGC. >>> >>> Fixed. Definitely an artifact of Erik's original prototype when he >>> extracted Thread-SMR from his GC work... Thanks for catching it. >>> >> >> Confirmed. At the time I considered the Threads list overheads GC >> overheads, but I agree mtThread is a better fit today. >> >>> >>>> >>>> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >>>> >>>> Can you add a comment about where this number came from? >>> >>> I'll have to get that from Erik... >> >> Wow, that looks like code I wrote a *very* long time ago. :) That is >> a variation of Knuth's multiplicative hash which is outlined in a >> comment in synchronizer.cpp and referred to in that comment as a >> phi-based scheme. Basically the magic number is 2^32 * Phi (the >> golden ratio), which happens to be a useful value for building a >> reasonably simple yet pretty good hash function. It is not the >> optimal hash function, but seemed to definitely be good enough for >> our purposes. > > So a reasonable comment would be: > > // The literal value is 2^32 * Phi (the golden ratio). > Yes. > If so, then I'll add it in my wrap up round... Excellent, thanks Dan. /Erik > Dan > >> >> Thanks, >> /Erik > From daniel.daugherty at oracle.com Wed Nov 22 13:09:16 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 08:09:16 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <4d5d448e-3902-8465-197e-18a5607fbd70@oracle.com> <4fd06a59-f6b5-33b2-53f5-7bfc96f866a7@oracle.com> Message-ID: <1e70bbf5-6f52-315d-85ba-fd53a01b66ca@oracle.com> Adding back the other OpenJDK aliases... On 11/22/17 8:01 AM, coleen.phillimore at oracle.com wrote: > > > On 11/22/17 7:51 AM, Daniel D. Daugherty wrote: >> On 11/21/17 11:26 PM, David Holmes wrote: >>> Hi Dan, >>> >>> On 22/11/2017 10:12 AM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> *** We are wrapping up code review on this project so it is time *** >>>> *** for the various code reviewers to chime in one last time. *** >>>> >>>> In this latest round, we had three different reviewers chime in so >>>> we're >>>> doing delta webrevs for each of those resolutions: >>>> >>>> David H's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >>>> >>>> >>>> ?? mq comment for dholmes_CR: >>>> ???? - Fix indents, trailing spaces and various typos. >>>> ???? - Add descriptions for the '_cnt', '_max' and '_times" fields, >>>> ?????? add impl notes to document the type choices. >>> >>> src/hotspot/share/runtime/thread.cpp >>> >>> 3466 // range. Unsigned 64-bit would be more future proof, but >>> 64-bit atomic inc >>> 3467 // isn't available everywhere (or is it?). >>> >>> 64-bit atomics should be available on all currently supported >>> platforms (the last time this was an issue was for PPC32 - and the >>> atomic templates should have filled in any previous holes in the >>> allowed types). >> >> Hmmm... I ran into this when I merged the project with the atomic >> templates. I got a JPRT build failure on one of the ARM platforms... >> Unfortunately, I didn't save the log files so I can't quote the >> error message from the template stuff... >> >> >>> But if it's always 64-bit then you'd have to use Atomic::load to >>> allow for 32-bit platforms. >> >> What's the official story on 32-bit platforms? Is that what >> bit me in JPRT? i.e., did we still have a 32-bit ARM build >> config'ed in JPRT a month or two ago??? >> >> >>> These counts etc are all just for statistics so we aren't really >>> concerned about eventual overflows in long running VMs - right? >> >> Yup these are just statistics... overflow is not a killer. >> >> >>> >>> --- >>> >>> ?????????????????????????????????????? // # of parallel threads in >>> _smr_delete_lock->wait(). >>> ? static uint????????????????? _smr_delete_lock_wait_cnt; >>> ?????????????????????????????????????? // Max # of parallel threads >>> in _smr_delete_lock->wait(). >>> >>> >>> why are the comments indented like that? I thought this is what >>> Coleen previously commented on. Please just start the comments in >>> the first column - no need for any indent. Thanks. >> >> Coleen asked for the new comments in thread.cpp to be moved >> over to column 1. She asked for the new comments in thread.hpp >> to be indented with more spaces. I started with 2 spaces and >> the variables still didn't stand out. I tried 4 spaces and >> they still didn't stand out probably because of the leading >> _smr_... So I went with 8 spaces... > > Since you have the same but more indepth comments in the .cpp file, > and at least for my sampling the comments in the .hpp file look > redundant, I think you should not have the same comments in the .hpp > file.?? They're really values that the implementation uses, not part > of the interface.? My vote is to remove the comments from the .hpp file. The first line is intended to be a 1-line summary and it is identical between thread.cpp and thread.hpp. My thinking was that someone would look in thread.hpp first to see what the field was for and if they wanted the gory details about the field they could look at the impl notes in thread.cpp... Of course, that creates a maintenance problem in keeping the 1-liners in sync between thread.hpp and thread.cpp. I'm okay with deleting the 1-liners from thread.hpp if folks think they should go... Dan > > Coleen >> >> >>> >>>> src/hotspot/share/runtime/globals.hpp change is white-space only so it >>>> ?? is only visible via the file's patch link. >>>> >>>> >>>> Robin W's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >>>> >>>> >>>> ?? mq comment for robinw_CR: >>>> ???? - Fix some inefficient code, update some comments, fix some >>>> indents, >>>> ?????? and add some 'const' specifiers. >>>> >>>> ?? src/hotspot/share/runtime/vm_operations.hpp change is >>>> white-space only so >>>> ?? it is only visible via the file's patch link. >>> >>> All looks fine to me. Some good catches there from Robin! >> >> Yes, a new pair of eyes on this code is always useful. >> >> >>> >>>> Coleen's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >>>> >>>> >>>> ?? mq comment for coleenp_CR: >>>> ???? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >>>> ???? - add header comment to threadSMR.hpp file, >>>> ???? - cleanup JavaThreadIteratorWithHandle ctr, >>>> ???? - make ErrorHandling more efficient. >>> >>> src/hotspot/share/runtime/threadSMR.hpp >>> >>> + // Thread Safe Memory Reclaimation (Thread-SMR) support. >>> >>> Only one 'i' in Reclamation :) >> >> Will fix. I tried typing both and neither looked right to me. >> (I might be getting blurry eyed with this stuff)... >> So I went with the spelling from Coleen's comment... :-) >> >> Thanks for taking another look!! >> >> Dan >> >> >>> >>> Thanks, >>> David >>> ------ >>> >>>> >>>> Some folks prefer to see all of the delta webrevs together so here is >>>> a delta webrev relative to jdk10-09-full: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >>>> >>>> >>>> ?? src/hotspot/share/runtime/globals.hpp and >>>> ?? src/hotspot/share/runtime/vm_operations.hpp changes are >>>> white-space only >>>> ?? so they are only visible via the file's patch link. >>>> >>>> >>>> And, of course, some folks prefer to see everything all at once so >>>> here >>>> is the latest full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >>>> >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >>>> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >>>> >>>> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >>>> so there will be some catching up to do before integration... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Testing of the last round of changes revealed a hang in one of the >>>>> new >>>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>>> test, and >>>>> added another TLH test for good measure. >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>>> >>>>> Here is the updated delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>>> >>>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>>> no unexpected failures. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Robbin rebased the project last night/this morning to merge with >>>>>> Thread >>>>>> Local Handshakes (TLH) and also picked up additional changesets >>>>>> up thru: >>>>>> >>>>>>> Changeset: fa736014cf28 >>>>>>> Author:??? cjplummer >>>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>>> >>>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>>> within hotspot source >>>>>>> Summary: added pns2() to debug.cpp >>>>>>> Reviewed-by: stuefe, gthornbr >>>>>> >>>>>> This is the first time we've rebased the project to bits that are >>>>>> this >>>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>>> think >>>>>> we're done with this project and are in the final >>>>>> review-change-retest >>>>>> cycle(s)... In other words, we're not planning on making any more >>>>>> major >>>>>> changes for this project... :-) >>>>>> >>>>>> *** Time for code reviewers to chime in on this thread! *** >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>>> >>>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>>> >>>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>>> (and the new baseline also)... >>>>>> >>>>>> We're expecting this round to be a little noisier than usual because >>>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>>> through Jesper's usual care-and-feeding of integration_blockers, >>>>>> etc. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>>> (Yes, we're playing chase-the-repo...) >>>>>>> >>>>>>> Here is the updated full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>>> >>>>>>> Unlike the previous rebase, there were no changes required to the >>>>>>> open code to get the rebased bits to build so there is no delta >>>>>>> webrev. >>>>>>> >>>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>>> Mach5 tier[1-5] test run... >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>>> >>>>>>>> Here are the updated webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>>> >>>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>>> holiday >>>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>>> closer >>>>>>>> look today. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>>> and Coleen) >>>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>>> done as a >>>>>>>>> standalone patch and corresponding webrevs: >>>>>>>>> >>>>>>>>> Here's the mq comment for the change: >>>>>>>>> >>>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>>> to use >>>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>>> >>>>>>>>> Here is the full webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>>> >>>>>>>>> And here is the delta webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Dan, Erik and Robbin >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>>> Greetings, >>>>>>>>>> >>>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>>> >>>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>>> >>>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>>> >>>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>>> describe >>>>>>>>>> and track the work on this project: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>>> code quotes >>>>>>>>>> in the PDF that are not present in the internal wiki. We >>>>>>>>>> don't have a >>>>>>>>>> solution for that problem yet. >>>>>>>>>> >>>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>>> >>>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>>> tier[2-5] >>>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>>> server, and >>>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>>> >>>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>>> >>>>>>>>>> Daniel Daugherty >>>>>>>>>> Erik Osterlund >>>>>>>>>> Robbin Ehn >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >> > From daniel.daugherty at oracle.com Wed Nov 22 13:10:34 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 08:10:34 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <50fdb219-4aa2-bdcc-cf69-8f60c935208f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <68b0b2bc-2d26-5442-bafb-5ce21420767b@oracle.com> <5e154ff2-65d4-bad4-db5e-88cf4c733c7d@oracle.com> <5A1547F0.7040206@oracle.com> <6906e0e0-e5cc-a415-6db2-13eff8d76ca8@oracle.com> <50fdb219-4aa2-bdcc-cf69-8f60c935208f@oracle.com> Message-ID: <6fa0d4d9-4acd-99eb-9769-613735561e93@oracle.com> On 11/22/17 8:02 AM, coleen.phillimore at oracle.com wrote: > > > On 11/22/17 7:54 AM, Daniel D. Daugherty wrote: >> On 11/22/17 4:48 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> Some replies... >>> >>> On 2017-11-21 18:36, Daniel D. Daugherty wrote: >>>> Thanks for keeping all the OpenJDK aliases! >>>> >>>> >>>> On 11/21/17 12:24 PM, coleen.phillimore at oracle.com wrote: >>>>> >>>>> >>>>> On 11/21/17 11:28 AM, Daniel D. Daugherty wrote: >>>>>> >>>>>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>>>>> >>>>>>> I can't find the caller of threads_do_smr. >>>>>> >>>>>> I'm guessing that's used by the GC code that depends on >>>>>> Thread-SMR... >>>>>> >>>>> >>>>> They? should add this API when the add the use of it then. I don't >>>>> see it in any sources. >>>> >>>> We'll see what Erik wants to do... >>> >>> The new iterators should be more than enough, so that closure-based >>> API is not going to be missed when removed. >> >> I will delete it in the wrap up round. >> >> > > Thank you. > >>> >>>> >>>>> >>>>>> >>>>>>> If these functions xchg_smr_thread_list, >>>>>>> get_smr_java_thread_list, inc_smr_deleted_thread_count are only >>>>>>> used by thread.cpp, I think they should go in that file and not >>>>>>> in the .inline.hpp file to be included and possibly called by >>>>>>> other files.? I think they're private to class Threads. >>>>>> >>>>>> I have a vague memory that some of the compilers don't do >>>>>> inlining when >>>>>> an "inline" function is in a .cpp. I believe we want these functions >>>>>> to be inlined for performance reasons. Erik should probably chime in >>>>>> here. >>>>> >>>>> I don't see why this should be.? Maybe some (older) compilers >>>>> require it to be found before the call though, but that can still >>>>> be accomplished in the .cpp file. >>>> >>>> Again, we'll see what Erik wants to do... >>> >>> I don't mind. Either file works for me. For me it's more intuitive >>> if inline member function definitions are in the .inline.hpp files. >>> But if there are strong forces to move this to the cpp file, then sure. >> >> I prefer inline member function definitions in the .inline.hpp files. >> (There might even be a style guide note about this...) >> >> Coleen, are you okay if we leave them there? > > Yes, that's fine. Thanks for confirmation! Dan > > Coleen > >> >> Dan >> >> >>> >>> Thanks, >>> /Erik >>> >> > From daniel.daugherty at oracle.com Wed Nov 22 13:10:58 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 08:10:58 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5A15766A.1090701@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <70cc1311-de55-5502-159b-cd0a3522cfd6@oracle.com> <5A153E52.8000704@oracle.com> <5A15766A.1090701@oracle.com> Message-ID: <40b27a38-d7f2-5a3b-0022-92d3678e5971@oracle.com> On 11/22/17 8:06 AM, Erik ?sterlund wrote: > Hi Dan, > > On 2017-11-22 13:53, Daniel D. Daugherty wrote: >> On 11/22/17 4:07 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> Some replies... >>> >>> On 2017-11-21 17:28, Daniel D. Daugherty wrote: >>>> Hi Coleen! >>>> >>>> Thanks for making time to review the Thread-SMR stuff again!! >>>> >>>> I have added back the other three OpenJDK aliases... This review is >>>> being done on _four_ different OpenJDK aliases. >>>> >>>> As always, replies are embedded below... >>>> >>>> >>>> On 11/20/17 3:12 PM, coleen.phillimore at oracle.com wrote: >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/src/hotspot/share/runtime/threadSMR.cpp.html >>>>> >>>>> >>>>> ? 32 ThreadsList::ThreadsList(int entries) : _length(entries), >>>>> _threads(NEW_C_HEAP_ARRAY(JavaThread*, entries + 1, mtGC)), >>>>> _next_list(NULL) { >>>>> >>>>> Seems like it should be mtThread rather than mtGC. >>>> >>>> Fixed. Definitely an artifact of Erik's original prototype when he >>>> extracted Thread-SMR from his GC work... Thanks for catching it. >>>> >>> >>> Confirmed. At the time I considered the Threads list overheads GC >>> overheads, but I agree mtThread is a better fit today. >>> >>>> >>>>> >>>>> + return (unsigned int)(((uint32_t)(uintptr_t)s1) * 2654435761u); >>>>> >>>>> Can you add a comment about where this number came from? >>>> >>>> I'll have to get that from Erik... >>> >>> Wow, that looks like code I wrote a *very* long time ago. :) That is >>> a variation of Knuth's multiplicative hash which is outlined in a >>> comment in synchronizer.cpp and referred to in that comment as a >>> phi-based scheme. Basically the magic number is 2^32 * Phi (the >>> golden ratio), which happens to be a useful value for building a >>> reasonably simple yet pretty good hash function. It is not the >>> optimal hash function, but seemed to definitely be good enough for >>> our purposes. >> >> So a reasonable comment would be: >> >> // The literal value is 2^32 * Phi (the golden ratio). >> > > Yes. > >> If so, then I'll add it in my wrap up round... > > Excellent, thanks Dan. Thanks for confirmation! Dan > > /Erik > >> Dan >> >>> >>> Thanks, >>> /Erik >> > From calvin.cheung at oracle.com Wed Nov 22 17:05:50 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 22 Nov 2017 09:05:50 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module Message-ID: <5A15AE6E.1000004@oracle.com> The jdk.internal.vm.compiler module is not available on Solaris/Sparc. A simple fix is to detect if the platform is Solaris and not adding the module to the --limit-modules option. Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. thanks, Calvin From thomas.stuefe at gmail.com Wed Nov 22 18:01:06 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 22 Nov 2017 19:01:06 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert Message-ID: Hi all, may I please have reviews for the following enhancement: issue: https://bugs.openjdk.java.net/browse/JDK-8191101 webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show-registers-on-assert/webrev.00/webrev/ (Patch looks big but lot of it is os_cpu fluff.) Prior email discussion at: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/025018.html ---- Basically, this adds the ability to show register values in assert situations into the error report. This can be useful in certain corner cases, e.g. if you want to know the value of some local variables or how deep your stack currently runs. This works by triggering a fault right when an assert happens and squirreling the context away for later error reporting. When an assert happens, we touch a poison page. To preserve the context as best as possible, we want to avoid running too much code after the assert condition has been evaluated, so this is done in a very simple way: directly in the assert macro, right after the condition is evaluated, we dereference the content of a global poison page pointer. In my tests, even with slowdebug, this only spoils one register, rax on x64. In the signal handler, we recognize the assertion poison fault by the faulting address. We disable the poison page and store the ucontext away. Then we just return from signal handling. Poison page is now disarmed, the load from it is retried and now goes through. Normal assertion handling is then resumed - so, things like -XX:SuppressAt are unaffected and work fine. Then, when an error report is generated due to this assert, we now also use the stored context. So now we get registers and instructions at the assert point. Right now, this is implemented on (non-zero) Linux, though other posix platforms should be no problem. Have not yet thought deeply about windows. It is tested and - in debug - switched on by default on linux x86, ppc, s390. If implemented, it can be switched on and off with -XX:+ShowRegistersOnAssert. This is a failsafe, in case the mechanism does not work and we want to have clean asserts. To test this, do a java -XX:ErrorHandlerTest=1 -XX:+ShowRegistersOnAssert with a not-product VM. On Linux x64, ppc and s390 we should now see the register output in the hs-err file: 4 # Internal Error (/shared/projects/openjdk/jdk-hs/source/src/hotspot/share/utilities/vmError.cpp:1660), pid=4022, tid=4023 5 # assert(str == NULL) failed: expected null 6 # 7 # ... 59 Registers: 60 RAX=0x00007f736c8f7000, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0xffffffffffb5ded4 61 RSP=0x00007f736c8d8ce0, RBP=0x00007f736c8d8d30, RSI=0x0000000000000001, RDI=0x0000000000000001 62 R8 =0x0000000000000040, R9 =0x0000000000000001, R10=0x0000000000efb028, R11=0x00040788a244f42a 63 R12=0x0000000000000000, R13=0x00007fff3de46dbf, R14=0x00007f736c8d99c0, R15=0x0000000000000000 64 RIP=0x00007f736aec5529, EFLAGS=0x0000000000010202, CSGSFS=0x002b000000000033, ERR=0x0000000000000006 65 TRAPNO=0x000000000000000e ... 72 73 Instructions: (pc=0x00007f736aec5529) 74 0x00007f736aec5509: 8d 05 31 21 4a 00 48 01 d0 ff e0 48 83 7d c8 00 75 0x00007f736aec5519: 0f 84 7f 03 00 00 48 8d 05 82 fc b1 00 48 8b 00 76 0x00007f736aec5529: c6 00 58 e8 cb 17 62 ff 84 c0 74 11 48 8d 3d 2f 77 0x00007f736aec5539: 1f 4a 00 b8 00 00 00 00 e8 ed 17 62 ff 48 8d 0d 78 --- Implementation notes: - when handling the poison fault, we need to copy the context away from the signal handler stack. For posix, this means copying the ucontext_t. This is undefined territory. On most platforms, this simply means copying the ucontex_t as a flat structure. On some platforms more is needed, e.g. on linux ppc, we need to patch up the context after copying (the context is not position independent), and on MacOS, the context is not self-contained but contains pointers to sub structures which need to be copied too and whose size is unknown at compile time. Because of these platform dependencies, I factored out the copying of ucontext_t to os::Posix::copy_ucontext and its implementations are os_cpu specific. - As an added precaution, when copying the context, we use a safe version of memcpy (os::safe_memcpy) which I added to copy from potentially invalid memory regions. The reason is that we have seen on some Unices - e.g. hpux - that the size of the ucontext_t structure at runtime may be different from the build machine, so we tread carefully. os::safe_memcpy() uses SafeFetch to copy a range of memory. Risk: If this does not work, asserts will become segfaults, which can be confusing. But the feature can be disabled with -XX:+ShowRegistersOnAssert and for now on most platforms is disabled by default. --- Limitations: - as it is now implemented, this is a one-shot mechanism and only works for the first assert. - -XX:SuppressAt=... is not affected and works fine. However, if the first assert is suppressed, follow-up asserts will not show register values. - When multiple threads run into an assert, we may or may not see register values depending on which thread is the first of finishing the poison page handler. I do not think these limitations are severe. They can be solved, but at the cost of added complexity, which I preferred not to add. Thank you, Thomas From ioi.lam at oracle.com Wed Nov 22 18:37:04 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 22 Nov 2017 10:37:04 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <5A15AE6E.1000004@oracle.com> References: <5A15AE6E.1000004@oracle.com> Message-ID: <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> Hi Calvin, Instead of hard coding for Solaris, is it possible to check if the VM has the graal module programmatically? Thanks Ioi > On Nov 22, 2017, at 9:05 AM, Calvin Cheung wrote: > > The jdk.internal.vm.compiler module is not available on Solaris/Sparc. A simple fix is to detect if the platform is Solaris and not adding the module to the --limit-modules option. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 > > webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ > > Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. > > thanks, > Calvin From zgu at redhat.com Wed Nov 22 19:25:01 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 22 Nov 2017 14:25:01 -0500 Subject: [8u-backport] 8139873: NMT stack traces in output should show mt component Message-ID: <628dceaa-89de-2754-fbc8-9d46e6c349ec@redhat.com> Can I get a review for 8u backport of this fix? The original patch did not apply cleanly, with a few minor merge conflicts. Bug: https://bugs.openjdk.java.net/browse/JDK-8139673 Changeset: http://hg.openjdk.java.net/jdk10/jdk10/hotspot/rev/82d4003d20b2 Review: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-May/023516.html Webrev: http://cr.openjdk.java.net/~zgu/8139673/webrev.8u_backport/ Thanks, -Zhengyu From vladimir.kozlov at oracle.com Wed Nov 22 19:29:37 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 22 Nov 2017 11:29:37 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> Message-ID: <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> On 11/22/17 10:37 AM, Ioi Lam wrote: > Hi Calvin, > > Instead of hard coding for Solaris, is it possible to check if the VM has the graal module programmatically? Yes, it would be better. We do plan extend support of Graal (jdk.internal.vm.compiler) to other platforms in a future. For example, we do check presence of jaotc file for setting jtreg key vm.aot: http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 Would be nice to add vm.graal jtreg key and use with @require. Thanks, Vladimir > > Thanks > Ioi > >> On Nov 22, 2017, at 9:05 AM, Calvin Cheung wrote: >> >> The jdk.internal.vm.compiler module is not available on Solaris/Sparc. A simple fix is to detect if the platform is Solaris and not adding the module to the --limit-modules option. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >> >> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >> >> thanks, >> Calvin > From vladimir.kozlov at oracle.com Wed Nov 22 19:32:06 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 22 Nov 2017 11:32:06 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> Message-ID: Note, we can't use vm.graal.enabled key because it is set only when Graal is used as JIT with VM flags: http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 Vladimir On 11/22/17 11:29 AM, Vladimir Kozlov wrote: > On 11/22/17 10:37 AM, Ioi Lam wrote: >> Hi Calvin, >> >> Instead of hard coding for Solaris, is it possible to check if the VM >> has the graal module programmatically? > > Yes, it would be better. We do plan extend support of Graal > (jdk.internal.vm.compiler) to other platforms in a future. > > For example, we do check presence of jaotc file for setting jtreg key > vm.aot: > > http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 > > > Would be nice to add vm.graal jtreg key and use with @require. > > Thanks, > Vladimir > >> >> Thanks >> Ioi >> >>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>> wrote: >>> >>> The jdk.internal.vm.compiler module is not available on >>> Solaris/Sparc. A simple fix is to detect if the platform is Solaris >>> and not adding the module to the --limit-modules option. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>> >>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>> >>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>> >>> thanks, >>> Calvin >> From shade at redhat.com Wed Nov 22 19:37:23 2017 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 22 Nov 2017 20:37:23 +0100 Subject: [8u-backport] 8139873: NMT stack traces in output should show mt component In-Reply-To: <628dceaa-89de-2754-fbc8-9d46e6c349ec@redhat.com> References: <628dceaa-89de-2754-fbc8-9d46e6c349ec@redhat.com> Message-ID: <0ae1f7ff-ab7c-a8f0-4b44-ac4da2bedfad@redhat.com> On 11/22/2017 08:25 PM, Zhengyu Gu wrote: > Can I get a review for 8u backport of this fix? > > The original patch did not apply cleanly, with a few minor merge conflicts. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8139673 > Changeset: http://hg.openjdk.java.net/jdk10/jdk10/hotspot/rev/82d4003d20b2 > Review: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-May/023516.html > > > Webrev: http://cr.openjdk.java.net/~zgu/8139673/webrev.8u_backport/ *) Indenting is off here: mallocSiteTable.cpp: 137 size_t* pos_idx, MEMFLAGS flags) { *) Excess whitespace after "const": mallocSiteTable.hpp: 58 MEMFLAGS flags() const { return (MEMFLAGS)_flags; } Otherwise still looks good to me. Thanks, -Aleksey From gerald.thornbrugh at oracle.com Wed Nov 22 20:21:43 2017 From: gerald.thornbrugh at oracle.com (Gerald Thornbrugh) Date: Wed, 22 Nov 2017 13:21:43 -0700 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> Message-ID: Hi Dan, Your changes look good. Jerry > On Nov 21, 2017, at 5:12 PM, Daniel D. Daugherty wrote: > > Greetings, > > *** We are wrapping up code review on this project so it is time *** > *** for the various code reviewers to chime in one last time. *** > > In this latest round, we had three different reviewers chime in so we're > doing delta webrevs for each of those resolutions: > > David H's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ > > mq comment for dholmes_CR: > - Fix indents, trailing spaces and various typos. > - Add descriptions for the '_cnt', '_max' and '_times" fields, > add impl notes to document the type choices. > > src/hotspot/share/runtime/globals.hpp change is white-space only so it > is only visible via the file's patch link. > > > Robin W's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ > > mq comment for robinw_CR: > - Fix some inefficient code, update some comments, fix some indents, > and add some 'const' specifiers. > > src/hotspot/share/runtime/vm_operations.hpp change is white-space only so > it is only visible via the file's patch link. > > > Coleen's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ > > mq comment for coleenp_CR: > - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', > - add header comment to threadSMR.hpp file, > - cleanup JavaThreadIteratorWithHandle ctr, > - make ErrorHandling more efficient. > > > Some folks prefer to see all of the delta webrevs together so here is > a delta webrev relative to jdk10-09-full: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ > > src/hotspot/share/runtime/globals.hpp and > src/hotspot/share/runtime/vm_operations.hpp changes are white-space only > so they are only visible via the file's patch link. > > > And, of course, some folks prefer to see everything all at once so here > is the latest full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ > > > Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The > prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... > > The project is currently baselined on Jesper's 2017-11-17 PIT snapshot > so there will be some catching up to do before integration... > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up thru: >>> >>>> Changeset: fa736014cf28 >>>> Author: cjplummer >>>> Date: 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from within hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>>> JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From robbin.ehn at oracle.com Wed Nov 22 20:24:54 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 22 Nov 2017 21:24:54 +0100 Subject: RFR(one-liner): 8191707: Options with invalid values are incorrectly treated as obsolete and ignored Message-ID: <80a99246-28f9-3035-2469-7287eadfb31f@oracle.com> Hi all, please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8191707 Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. Contributed by David Holmes. Looks good, thanks for fixing! /Robbin diff -r 2cd1c2b03782 src/hotspot/share/runtime/arguments.cpp --- a/src/hotspot/share/runtime/arguments.cpp Wed Nov 22 01:12:23 2017 -0800 +++ b/src/hotspot/share/runtime/arguments.cpp Wed Nov 22 21:20:42 2017 +0100 @@ -493,15 +493,15 @@ } bool Arguments::is_obsolete_flag(const char *flag_name, JDK_Version* version) { assert(version != NULL, "Must provide a version buffer"); SpecialFlag flag; if (lookup_special_flag(flag_name, flag)) { if (!flag.obsolete_in.is_undefined()) { - if (version_less_than(JDK_Version::current(), flag.expired_in)) { + if (!version_less_than(JDK_Version::current(), flag.obsolete_in)) { *version = flag.obsolete_in; return true; } } } return false; } From robbin.ehn at oracle.com Wed Nov 22 20:37:25 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 22 Nov 2017 21:37:25 +0100 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java Message-ID: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Hi all, please review. Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. Thanks, Robbin! diff -r 2cd1c2b03782 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 01:12:23 2017 -0800 +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 21:28:36 2017 +0100 @@ -46,6 +46,11 @@ {"MinRAMFraction", "2"}, {"InitialRAMFraction", "64"}, {"AssumeMP", "false"}, + {"UseMembar", "true"}, + {"FastTLABRefill", "false"}, + {"DeferPollingPageLoopCount", "-1"}, + {"SafepointSpinBeforeYield", "2000"}, + {"DeferPollingPageLoopCount", "4000"}, // deprecated alias flags (see also aliased_jvm_flags): {"DefaultMaxRAMFraction", "4"}, From ioi.lam at oracle.com Wed Nov 22 20:38:05 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 22 Nov 2017 12:38:05 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> Message-ID: <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> Adding hotspot-compiler-dev at openjdk.java.net I am not sure if @require vm.graal is the right fix for this bug. In fact, this test has nothing to do with graal. The reason ??? --limit-modules=java.base,jdk.internal.vm.compiler was added was because of JDK-8190975 [Graal] Tests which run with "--limit-modules java.base" could fail when Graal is used as JIT ??? https://bugs.openjdk.java.net/browse/JDK-8190975 Unfortunately, JDK-8190975 leaks dependencies of jdk.internal.vm.compiler into tests unrelated to graal. I think it's better to backout JDK-8190975, and then fix the VM such that if "-Djvmci.Compiler=graal" is added in the command-line, jdk.internal.vm.compiler is automatically added to the list of available modules, regardless of any --limit-modules flag. Thanks - Ioi On 11/22/17 11:32 AM, Vladimir Kozlov wrote: > Note, we can't use vm.graal.enabled key because it is set only when > Graal is used as JIT with VM flags: > > http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 > > > Vladimir > > On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >> On 11/22/17 10:37 AM, Ioi Lam wrote: >>> Hi Calvin, >>> >>> Instead of hard coding for Solaris, is it possible to check if the >>> VM has the graal module programmatically? >> >> Yes, it would be better. We do plan extend support of Graal >> (jdk.internal.vm.compiler) to other platforms in a future. >> >> For example, we do check presence of jaotc file for setting jtreg key >> vm.aot: >> >> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >> >> >> Would be nice to add vm.graal jtreg key and use with @require. >> >> Thanks, >> Vladimir >> >>> >>> Thanks >>> Ioi >>> >>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>>> wrote: >>>> >>>> The jdk.internal.vm.compiler module is not available on >>>> Solaris/Sparc. A simple fix is to detect if the platform is Solaris >>>> and not adding the module to the --limit-modules option. >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>> >>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>> >>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>> >>>> thanks, >>>> Calvin >>> From vladimir.kozlov at oracle.com Wed Nov 22 21:01:06 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 22 Nov 2017 13:01:06 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> Message-ID: <89d9fb75-5ed7-8995-7111-d06be67f997a@oracle.com> On 11/22/17 12:38 PM, Ioi Lam wrote: > Adding hotspot-compiler-dev at openjdk.java.net > > I am not sure if @require vm.graal is the right fix for this bug. In > fact, this test has nothing to do with graal. The reason > > ??? --limit-modules=java.base,jdk.internal.vm.compiler > > was added was because of JDK-8190975 [Graal] Tests which run with > "--limit-modules java.base" could fail when Graal is used as JIT > > ??? https://bugs.openjdk.java.net/browse/JDK-8190975 > > Unfortunately, JDK-8190975 leaks dependencies of > jdk.internal.vm.compiler into tests unrelated to graal. > > I think it's better to backout JDK-8190975, and then fix the VM such > that if "-Djvmci.Compiler=graal" is added in the command-line, > jdk.internal.vm.compiler is automatically added to the list of available > modules, regardless of any --limit-modules flag. Agree. Thanks, Vladimir > > Thanks > - Ioi > > > On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >> Note, we can't use vm.graal.enabled key because it is set only when >> Graal is used as JIT with VM flags: >> >> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >> >> >> Vladimir >> >> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>> Hi Calvin, >>>> >>>> Instead of hard coding for Solaris, is it possible to check if the >>>> VM has the graal module programmatically? >>> >>> Yes, it would be better. We do plan extend support of Graal >>> (jdk.internal.vm.compiler) to other platforms in a future. >>> >>> For example, we do check presence of jaotc file for setting jtreg key >>> vm.aot: >>> >>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>> >>> >>> Would be nice to add vm.graal jtreg key and use with @require. >>> >>> Thanks, >>> Vladimir >>> >>>> >>>> Thanks >>>> Ioi >>>> >>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>>>> wrote: >>>>> >>>>> The jdk.internal.vm.compiler module is not available on >>>>> Solaris/Sparc. A simple fix is to detect if the platform is Solaris >>>>> and not adding the module to the --limit-modules option. >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>> >>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>> >>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>> >>>>> thanks, >>>>> Calvin >>>> > From zgu at redhat.com Wed Nov 22 21:22:36 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 22 Nov 2017 16:22:36 -0500 Subject: [8u-backport] 8139873: NMT stack traces in output should show mt component In-Reply-To: <0ae1f7ff-ab7c-a8f0-4b44-ac4da2bedfad@redhat.com> References: <628dceaa-89de-2754-fbc8-9d46e6c349ec@redhat.com> <0ae1f7ff-ab7c-a8f0-4b44-ac4da2bedfad@redhat.com> Message-ID: <6fba1c24-373c-f6c1-06c9-f5d804826764@redhat.com> Thanks for the quick review, Aleksey. On 11/22/2017 02:37 PM, Aleksey Shipilev wrote: > On 11/22/2017 08:25 PM, Zhengyu Gu wrote: >> Can I get a review for 8u backport of this fix? >> >> The original patch did not apply cleanly, with a few minor merge conflicts. >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8139673 >> Changeset: http://hg.openjdk.java.net/jdk10/jdk10/hotspot/rev/82d4003d20b2 >> Review: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-May/023516.html >> >> >> Webrev: http://cr.openjdk.java.net/~zgu/8139673/webrev.8u_backport/ > > *) Indenting is off here: > > mallocSiteTable.cpp: > 137 size_t* pos_idx, MEMFLAGS flags) { > > *) Excess whitespace after "const": > > mallocSiteTable.hpp: > > 58 MEMFLAGS flags() const { return (MEMFLAGS)_flags; } Updated: http://cr.openjdk.java.net/~zgu/8139673/8139673/webrev.8u_backport.01/ -Zhengyu > > Otherwise still looks good to me. > > Thanks, > -Aleksey > From daniel.daugherty at oracle.com Wed Nov 22 21:48:50 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 16:48:50 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> Message-ID: <06e3a59e-9225-ee56-eefe-2614e29bd8f9@oracle.com> Thanks Jerry! Dan On 11/22/17 3:21 PM, Gerald Thornbrugh wrote: > Hi Dan, > > Your changes look good. > > Jerry > >> On Nov 21, 2017, at 5:12 PM, Daniel D. Daugherty wrote: >> >> Greetings, >> >> *** We are wrapping up code review on this project so it is time *** >> *** for the various code reviewers to chime in one last time. *** >> >> In this latest round, we had three different reviewers chime in so we're >> doing delta webrevs for each of those resolutions: >> >> David H's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >> >> mq comment for dholmes_CR: >> - Fix indents, trailing spaces and various typos. >> - Add descriptions for the '_cnt', '_max' and '_times" fields, >> add impl notes to document the type choices. >> >> src/hotspot/share/runtime/globals.hpp change is white-space only so it >> is only visible via the file's patch link. >> >> >> Robin W's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >> >> mq comment for robinw_CR: >> - Fix some inefficient code, update some comments, fix some indents, >> and add some 'const' specifiers. >> >> src/hotspot/share/runtime/vm_operations.hpp change is white-space only so >> it is only visible via the file's patch link. >> >> >> Coleen's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >> >> mq comment for coleenp_CR: >> - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >> - add header comment to threadSMR.hpp file, >> - cleanup JavaThreadIteratorWithHandle ctr, >> - make ErrorHandling more efficient. >> >> >> Some folks prefer to see all of the delta webrevs together so here is >> a delta webrev relative to jdk10-09-full: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >> >> src/hotspot/share/runtime/globals.hpp and >> src/hotspot/share/runtime/vm_operations.hpp changes are white-space only >> so they are only visible via the file's patch link. >> >> >> And, of course, some folks prefer to see everything all at once so here >> is the latest full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >> >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >> >> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >> so there will be some catching up to do before integration... >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author: cjplummer >>>>> Date: 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>>>> JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> From thomas.stuefe at gmail.com Wed Nov 22 22:42:10 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 22 Nov 2017 22:42:10 +0000 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: Small correction, see below... On Wed 22. Nov 2017 at 19:01, Thomas St?fe wrote: > Hi all, > > may I please have reviews for the following enhancement: > > issue: https://bugs.openjdk.java.net/browse/JDK-8191101 > webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show-registers-on-assert/webrev.00/webrev/ > > > (Patch looks big but lot of it is os_cpu fluff.) > > Prior email discussion at: > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/025018.html > > ---- > > Basically, this adds the ability to show register values in assert > situations into the error report. This can be useful in certain corner > cases, e.g. if you want to know the value of some local variables or how > deep your stack currently runs. > > This works by triggering a fault right when an assert happens and > squirreling the context away for later error reporting. > > When an assert happens, we touch a poison page. To preserve the context as > best as possible, we want to avoid running too much code after the assert > condition has been evaluated, so this is done in a very simple way: > directly in the assert macro, right after the condition is evaluated, we > dereference the content of a global poison page pointer. In my tests, even > with slowdebug, this only spoils one register, rax on x64. > > In the signal handler, we recognize the assertion poison fault by the > faulting address. We disable the poison page and store the ucontext away. > Then we just return from signal handling. Poison page is now disarmed, the > load from it is retried and now goes through. Normal assertion handling is > then resumed - so, things like -XX:SuppressAt are unaffected and work fine. > > Then, when an error report is generated due to this assert, we now also > use the stored context. So now we get registers and instructions at the > assert point. > > Right now, this is implemented on (non-zero) Linux, though other posix > platforms should be no problem. Have not yet thought deeply about windows. > It is tested and - in debug - switched on by default on linux x86, ppc, > s390. > > If implemented, it can be switched on and off with > -XX:+ShowRegistersOnAssert. This is a failsafe, in case the mechanism does > not work and we want to have clean asserts. > > To test this, do a java -XX:ErrorHandlerTest=1 -XX:+ShowRegistersOnAssert > with a not-product VM. On Linux x64, ppc and s390 we should now see the > register output in the hs-err file: > > 4 # Internal Error > (/shared/projects/openjdk/jdk-hs/source/src/hotspot/share/utilities/vmError.cpp:1660), > pid=4022, tid=4023 > 5 # assert(str == NULL) failed: expected null > 6 # > 7 # > ... > 59 Registers: > 60 RAX=0x00007f736c8f7000, RBX=0x0000000000000000, > RCX=0x0000000000000000, RDX=0xffffffffffb5ded4 > 61 RSP=0x00007f736c8d8ce0, RBP=0x00007f736c8d8d30, > RSI=0x0000000000000001, RDI=0x0000000000000001 > 62 R8 =0x0000000000000040, R9 =0x0000000000000001, > R10=0x0000000000efb028, R11=0x00040788a244f42a > 63 R12=0x0000000000000000, R13=0x00007fff3de46dbf, > R14=0x00007f736c8d99c0, R15=0x0000000000000000 > 64 RIP=0x00007f736aec5529, EFLAGS=0x0000000000010202, > CSGSFS=0x002b000000000033, ERR=0x0000000000000006 > 65 TRAPNO=0x000000000000000e > ... > 72 > 73 Instructions: (pc=0x00007f736aec5529) > 74 0x00007f736aec5509: 8d 05 31 21 4a 00 48 01 d0 ff e0 48 83 7d c8 00 > 75 0x00007f736aec5519: 0f 84 7f 03 00 00 48 8d 05 82 fc b1 00 48 8b 00 > 76 0x00007f736aec5529: c6 00 58 e8 cb 17 62 ff 84 c0 74 11 48 8d 3d 2f > 77 0x00007f736aec5539: 1f 4a 00 b8 00 00 00 00 e8 ed 17 62 ff 48 8d 0d > 78 > > --- > > Implementation notes: > > - when handling the poison fault, we need to copy the context away from > the signal handler stack. For posix, this means copying the ucontext_t. > This is undefined territory. On most platforms, this simply means copying > the ucontex_t as a flat structure. On some platforms more is needed, e.g. > on linux ppc, we need to patch up the context after copying (the context is > not position independent), and on MacOS, the context is not self-contained > but contains pointers to sub structures which need to be copied too and > whose size is unknown at compile time. Because of these platform > dependencies, I factored out the copying of ucontext_t to > os::Posix::copy_ucontext and its implementations are os_cpu specific. > > - As an added precaution, when copying the context, we use a safe version > of memcpy (os::safe_memcpy) which I added to copy from potentially invalid > memory regions. The reason is that we have seen on some Unices - e.g. hpux > - that the size of the ucontext_t structure at runtime may be different > from the build machine, so we tread carefully. os::safe_memcpy() uses > SafeFetch to copy a range of memory. > > Risk: > > If this does not work, asserts will become segfaults, which can be > confusing. But the feature can be disabled with -XX:+ShowRegistersOnAssert > and for now on most platforms is disabled by default. > Feature is disabled with -XX:-ShowRegistersOnAssert, of course. Sorry for the typo. > --- > > Limitations: > - as it is now implemented, this is a one-shot mechanism and only works > for the first assert. > - -XX:SuppressAt=... is not affected and works fine. However, if the first > assert is suppressed, follow-up asserts will not show register values. > - When multiple threads run into an assert, we may or may not see register > values depending on which thread is the first of finishing the poison page > handler. > > I do not think these limitations are severe. They can be solved, but at > the cost of added complexity, which I preferred not to add. > > Thank you, > > Thomas > > > > From calvin.cheung at oracle.com Wed Nov 22 22:47:55 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 22 Nov 2017 14:47:55 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> Message-ID: <5A15FE9B.1020304@oracle.com> On 11/22/17, 12:38 PM, Ioi Lam wrote: > Adding hotspot-compiler-dev at openjdk.java.net > > I am not sure if @require vm.graal is the right fix for this bug. In > fact, this test has nothing to do with graal. The reason > > --limit-modules=java.base,jdk.internal.vm.compiler > > was added was because of JDK-8190975 [Graal] Tests which run with > "--limit-modules java.base" could fail when Graal is used as JIT > > https://bugs.openjdk.java.net/browse/JDK-8190975 > > Unfortunately, JDK-8190975 leaks dependencies of > jdk.internal.vm.compiler into tests unrelated to graal. > > I think it's better to backout JDK-8190975, Here's an updated webrev for backing out fix for JDK-8190975: http://cr.openjdk.java.net/~ccheung/8191653/webrev.01/ Tested locally on linux-x64. Testing underway on other platforms. > and then fix the VM such that if "-Djvmci.Compiler=graal" is added in > the command-line, jdk.internal.vm.compiler is automatically added to > the list of available modules, regardless of any --limit-modules flag. I've filed the following RFE: https://bugs.openjdk.java.net/browse/JDK-8191788 thanks, Calvin > > Thanks > - Ioi > > > On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >> Note, we can't use vm.graal.enabled key because it is set only when >> Graal is used as JIT with VM flags: >> >> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >> >> >> Vladimir >> >> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>> Hi Calvin, >>>> >>>> Instead of hard coding for Solaris, is it possible to check if the >>>> VM has the graal module programmatically? >>> >>> Yes, it would be better. We do plan extend support of Graal >>> (jdk.internal.vm.compiler) to other platforms in a future. >>> >>> For example, we do check presence of jaotc file for setting jtreg >>> key vm.aot: >>> >>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>> >>> >>> Would be nice to add vm.graal jtreg key and use with @require. >>> >>> Thanks, >>> Vladimir >>> >>>> >>>> Thanks >>>> Ioi >>>> >>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>>>> wrote: >>>>> >>>>> The jdk.internal.vm.compiler module is not available on >>>>> Solaris/Sparc. A simple fix is to detect if the platform is >>>>> Solaris and not adding the module to the --limit-modules option. >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>> >>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>> >>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>> >>>>> thanks, >>>>> Calvin >>>> > From vladimir.kozlov at oracle.com Wed Nov 22 23:01:49 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 22 Nov 2017 15:01:49 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <5A15FE9B.1020304@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> <5A15FE9B.1020304@oracle.com> Message-ID: <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> On 11/22/17 2:47 PM, Calvin Cheung wrote: > > > On 11/22/17, 12:38 PM, Ioi Lam wrote: >> Adding hotspot-compiler-dev at openjdk.java.net >> >> I am not sure if @require vm.graal is the right fix for this bug. In >> fact, this test has nothing to do with graal. The reason >> >> ??? --limit-modules=java.base,jdk.internal.vm.compiler >> >> was added was because of JDK-8190975 [Graal] Tests which run with >> "--limit-modules java.base" could fail when Graal is used as JIT >> >> ??? https://bugs.openjdk.java.net/browse/JDK-8190975 >> >> Unfortunately, JDK-8190975 leaks dependencies of >> jdk.internal.vm.compiler into tests unrelated to graal. >> >> I think it's better to backout JDK-8190975, > Here's an updated webrev for backing out fix for JDK-8190975: > ??? http://cr.openjdk.java.net/~ccheung/8191653/webrev.01/ Good. > > Tested locally on linux-x64. Testing underway on other platforms. >> and then fix the VM such that if "-Djvmci.Compiler=graal" is added in >> the command-line, jdk.internal.vm.compiler is automatically added to >> the list of available modules, regardless of any --limit-modules flag. > I've filed the following RFE: > ??? https://bugs.openjdk.java.net/browse/JDK-8191788 Thanks, Vladimir > > thanks, > Calvin >> >> Thanks >> - Ioi >> >> >> On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >>> Note, we can't use vm.graal.enabled key because it is set only when >>> Graal is used as JIT with VM flags: >>> >>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >>> >>> >>> Vladimir >>> >>> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>>> Hi Calvin, >>>>> >>>>> Instead of hard coding for Solaris, is it possible to check if the >>>>> VM has the graal module programmatically? >>>> >>>> Yes, it would be better. We do plan extend support of Graal >>>> (jdk.internal.vm.compiler) to other platforms in a future. >>>> >>>> For example, we do check presence of jaotc file for setting jtreg >>>> key vm.aot: >>>> >>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>>> >>>> >>>> Would be nice to add vm.graal jtreg key and use with @require. >>>> >>>> Thanks, >>>> Vladimir >>>> >>>>> >>>>> Thanks >>>>> Ioi >>>>> >>>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>>>>> wrote: >>>>>> >>>>>> The jdk.internal.vm.compiler module is not available on >>>>>> Solaris/Sparc. A simple fix is to detect if the platform is >>>>>> Solaris and not adding the module to the --limit-modules option. >>>>>> >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>>> >>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>>> >>>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>>> >>>>>> thanks, >>>>>> Calvin >>>>> >> From calvin.cheung at oracle.com Wed Nov 22 23:37:53 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 22 Nov 2017 15:37:53 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> <5A15FE9B.1020304@oracle.com> <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> Message-ID: <5A160A51.1030907@oracle.com> Thanks - Vladimir. Calvin On 11/22/17, 3:01 PM, Vladimir Kozlov wrote: > On 11/22/17 2:47 PM, Calvin Cheung wrote: >> >> >> On 11/22/17, 12:38 PM, Ioi Lam wrote: >>> Adding hotspot-compiler-dev at openjdk.java.net >>> >>> I am not sure if @require vm.graal is the right fix for this bug. In >>> fact, this test has nothing to do with graal. The reason >>> >>> --limit-modules=java.base,jdk.internal.vm.compiler >>> >>> was added was because of JDK-8190975 [Graal] Tests which run with >>> "--limit-modules java.base" could fail when Graal is used as JIT >>> >>> https://bugs.openjdk.java.net/browse/JDK-8190975 >>> >>> Unfortunately, JDK-8190975 leaks dependencies of >>> jdk.internal.vm.compiler into tests unrelated to graal. >>> >>> I think it's better to backout JDK-8190975, >> Here's an updated webrev for backing out fix for JDK-8190975: >> http://cr.openjdk.java.net/~ccheung/8191653/webrev.01/ > > Good. > >> >> Tested locally on linux-x64. Testing underway on other platforms. >>> and then fix the VM such that if "-Djvmci.Compiler=graal" is added >>> in the command-line, jdk.internal.vm.compiler is automatically added >>> to the list of available modules, regardless of any --limit-modules >>> flag. >> I've filed the following RFE: >> https://bugs.openjdk.java.net/browse/JDK-8191788 > > Thanks, > Vladimir > >> >> thanks, >> Calvin >>> >>> Thanks >>> - Ioi >>> >>> >>> On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >>>> Note, we can't use vm.graal.enabled key because it is set only when >>>> Graal is used as JIT with VM flags: >>>> >>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >>>> >>>> >>>> Vladimir >>>> >>>> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>>>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>>>> Hi Calvin, >>>>>> >>>>>> Instead of hard coding for Solaris, is it possible to check if >>>>>> the VM has the graal module programmatically? >>>>> >>>>> Yes, it would be better. We do plan extend support of Graal >>>>> (jdk.internal.vm.compiler) to other platforms in a future. >>>>> >>>>> For example, we do check presence of jaotc file for setting jtreg >>>>> key vm.aot: >>>>> >>>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>>>> >>>>> >>>>> Would be nice to add vm.graal jtreg key and use with @require. >>>>> >>>>> Thanks, >>>>> Vladimir >>>>> >>>>>> >>>>>> Thanks >>>>>> Ioi >>>>>> >>>>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung >>>>>>> wrote: >>>>>>> >>>>>>> The jdk.internal.vm.compiler module is not available on >>>>>>> Solaris/Sparc. A simple fix is to detect if the platform is >>>>>>> Solaris and not adding the module to the --limit-modules option. >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>>>> >>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>>>> >>>>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>> >>> From ioi.lam at oracle.com Wed Nov 22 23:44:40 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 22 Nov 2017 15:44:40 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> <5A15FE9B.1020304@oracle.com> <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> Message-ID: <3601DB7A-653F-4DF6-9964-40E3E40969D4@oracle.com> +1 Thanks Ioi > On Nov 22, 2017, at 3:01 PM, Vladimir Kozlov wrote: > >> On 11/22/17 2:47 PM, Calvin Cheung wrote: >>> On 11/22/17, 12:38 PM, Ioi Lam wrote: >>> Adding hotspot-compiler-dev at openjdk.java.net >>> >>> I am not sure if @require vm.graal is the right fix for this bug. In fact, this test has nothing to do with graal. The reason >>> >>> --limit-modules=java.base,jdk.internal.vm.compiler >>> >>> was added was because of JDK-8190975 [Graal] Tests which run with "--limit-modules java.base" could fail when Graal is used as JIT >>> >>> https://bugs.openjdk.java.net/browse/JDK-8190975 >>> >>> Unfortunately, JDK-8190975 leaks dependencies of jdk.internal.vm.compiler into tests unrelated to graal. >>> >>> I think it's better to backout JDK-8190975, >> Here's an updated webrev for backing out fix for JDK-8190975: >> http://cr.openjdk.java.net/~ccheung/8191653/webrev.01/ > > Good. > >> Tested locally on linux-x64. Testing underway on other platforms. >>> and then fix the VM such that if "-Djvmci.Compiler=graal" is added in the command-line, jdk.internal.vm.compiler is automatically added to the list of available modules, regardless of any --limit-modules flag. >> I've filed the following RFE: >> https://bugs.openjdk.java.net/browse/JDK-8191788 > > Thanks, > Vladimir > >> thanks, >> Calvin >>> >>> Thanks >>> - Ioi >>> >>> >>>> On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >>>> Note, we can't use vm.graal.enabled key because it is set only when Graal is used as JIT with VM flags: >>>> >>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >>>> >>>> Vladimir >>>> >>>>> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>>>>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>>>> Hi Calvin, >>>>>> >>>>>> Instead of hard coding for Solaris, is it possible to check if the VM has the graal module programmatically? >>>>> >>>>> Yes, it would be better. We do plan extend support of Graal (jdk.internal.vm.compiler) to other platforms in a future. >>>>> >>>>> For example, we do check presence of jaotc file for setting jtreg key vm.aot: >>>>> >>>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>>>> >>>>> Would be nice to add vm.graal jtreg key and use with @require. >>>>> >>>>> Thanks, >>>>> Vladimir >>>>> >>>>>> >>>>>> Thanks >>>>>> Ioi >>>>>> >>>>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung wrote: >>>>>>> >>>>>>> The jdk.internal.vm.compiler module is not available on Solaris/Sparc. A simple fix is to detect if the platform is Solaris and not adding the module to the --limit-modules option. >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>>>> >>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>>>> >>>>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>> >>> From calvin.cheung at oracle.com Wed Nov 22 23:47:22 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 22 Nov 2017 15:47:22 -0800 Subject: RFR(XS): 8191653: Test failures in BootAppendTests - missing jdk.internal.vm.compiler module In-Reply-To: <3601DB7A-653F-4DF6-9964-40E3E40969D4@oracle.com> References: <5A15AE6E.1000004@oracle.com> <350B4F2D-8CD7-40C1-938C-F9A7317D1C59@oracle.com> <7ca98e81-13d2-fa58-4176-ba172d409966@oracle.com> <969574dd-8ac7-2cdf-e137-c38d6e28da64@oracle.com> <5A15FE9B.1020304@oracle.com> <85f0cde7-1dc9-6b4b-77c7-249ddf06b9ee@oracle.com> <3601DB7A-653F-4DF6-9964-40E3E40969D4@oracle.com> Message-ID: <5A160C8A.2000200@oracle.com> Thanks - Ioi. Calvin On 11/22/17, 3:44 PM, Ioi Lam wrote: > +1 > > Thanks > Ioi > >> On Nov 22, 2017, at 3:01 PM, Vladimir Kozlov wrote: >> >>> On 11/22/17 2:47 PM, Calvin Cheung wrote: >>>> On 11/22/17, 12:38 PM, Ioi Lam wrote: >>>> Adding hotspot-compiler-dev at openjdk.java.net >>>> >>>> I am not sure if @require vm.graal is the right fix for this bug. In fact, this test has nothing to do with graal. The reason >>>> >>>> --limit-modules=java.base,jdk.internal.vm.compiler >>>> >>>> was added was because of JDK-8190975 [Graal] Tests which run with "--limit-modules java.base" could fail when Graal is used as JIT >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8190975 >>>> >>>> Unfortunately, JDK-8190975 leaks dependencies of jdk.internal.vm.compiler into tests unrelated to graal. >>>> >>>> I think it's better to backout JDK-8190975, >>> Here's an updated webrev for backing out fix for JDK-8190975: >>> http://cr.openjdk.java.net/~ccheung/8191653/webrev.01/ >> Good. >> >>> Tested locally on linux-x64. Testing underway on other platforms. >>>> and then fix the VM such that if "-Djvmci.Compiler=graal" is added in the command-line, jdk.internal.vm.compiler is automatically added to the list of available modules, regardless of any --limit-modules flag. >>> I've filed the following RFE: >>> https://bugs.openjdk.java.net/browse/JDK-8191788 >> Thanks, >> Vladimir >> >>> thanks, >>> Calvin >>>> Thanks >>>> - Ioi >>>> >>>> >>>>> On 11/22/17 11:32 AM, Vladimir Kozlov wrote: >>>>> Note, we can't use vm.graal.enabled key because it is set only when Graal is used as JIT with VM flags: >>>>> >>>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l299 >>>>> >>>>> Vladimir >>>>> >>>>>> On 11/22/17 11:29 AM, Vladimir Kozlov wrote: >>>>>>> On 11/22/17 10:37 AM, Ioi Lam wrote: >>>>>>> Hi Calvin, >>>>>>> >>>>>>> Instead of hard coding for Solaris, is it possible to check if the VM has the graal module programmatically? >>>>>> Yes, it would be better. We do plan extend support of Graal (jdk.internal.vm.compiler) to other platforms in a future. >>>>>> >>>>>> For example, we do check presence of jaotc file for setting jtreg key vm.aot: >>>>>> >>>>>> http://hg.openjdk.java.net/jdk10/hs/file/d85284ccd1bd/test/jtreg-ext/requires/VMProps.java#l270 >>>>>> >>>>>> Would be nice to add vm.graal jtreg key and use with @require. >>>>>> >>>>>> Thanks, >>>>>> Vladimir >>>>>> >>>>>>> Thanks >>>>>>> Ioi >>>>>>> >>>>>>>> On Nov 22, 2017, at 9:05 AM, Calvin Cheung wrote: >>>>>>>> >>>>>>>> The jdk.internal.vm.compiler module is not available on Solaris/Sparc. A simple fix is to detect if the platform is Solaris and not adding the module to the --limit-modules option. >>>>>>>> >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8191653 >>>>>>>> >>>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8191653/webrev.00/ >>>>>>>> >>>>>>>> Tested on 64-bit platforms: linux, solaris/sparc, macosx, windows. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin From daniel.daugherty at oracle.com Thu Nov 23 00:19:46 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 19:19:46 -0500 Subject: RFR(one-liner): 8191707: Options with invalid values are incorrectly treated as obsolete and ignored In-Reply-To: <80a99246-28f9-3035-2469-7287eadfb31f@oracle.com> References: <80a99246-28f9-3035-2469-7287eadfb31f@oracle.com> Message-ID: <54ab4e67-abe2-9801-d237-e4052f9f74a4@oracle.com> On 11/22/17 3:24 PM, Robbin Ehn wrote: > Hi all, please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191707 > > Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no > unexpected failure. > > Contributed by David Holmes. > > Looks good, thanks for fixing! Thumbs up on the fix. What I don't understand is why this logic did not fall over before now. Dan > > /Robbin > > > diff -r 2cd1c2b03782 src/hotspot/share/runtime/arguments.cpp > --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 01:12:23 > 2017 -0800 > +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 21:20:42 > 2017 +0100 > @@ -493,15 +493,15 @@ > ?} > > ?bool Arguments::is_obsolete_flag(const char *flag_name, JDK_Version* > version) { > ?? assert(version != NULL, "Must provide a version buffer"); > ?? SpecialFlag flag; > ?? if (lookup_special_flag(flag_name, flag)) { > ???? if (!flag.obsolete_in.is_undefined()) { > -????? if (version_less_than(JDK_Version::current(), flag.expired_in)) { > +????? if (!version_less_than(JDK_Version::current(), > flag.obsolete_in)) { > ???????? *version = flag.obsolete_in; > ???????? return true; > ?????? } > ???? } > ?? } > ?? return false; > ?} From daniel.daugherty at oracle.com Thu Nov 23 00:24:31 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 19:24:31 -0500 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Message-ID: On 11/22/17 3:37 PM, Robbin Ehn wrote: > Hi all, please review. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 > > Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no > unexpected failure. > > Thanks, Robbin! Thumbs up on the change. We should put a comment in the code where deprecated options are put to remind folks to update this test. Dan > > diff -r 2cd1c2b03782 > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 01:12:23 2017 -0800 > +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 21:28:36 2017 +0100 > @@ -46,6 +46,11 @@ > ???????? {"MinRAMFraction",??????????? "2"}, > ???????? {"InitialRAMFraction",??????? "64"}, > ???????? {"AssumeMP",????????????????? "false"}, > +??????? {"UseMembar",???????????????? "true"}, > +??????? {"FastTLABRefill",??????????? "false"}, > +??????? {"DeferPollingPageLoopCount", "-1"}, > +??????? {"SafepointSpinBeforeYield",? "2000"}, > +??????? {"DeferPollingPageLoopCount", "4000"}, > > ???????? // deprecated alias flags (see also aliased_jvm_flags): > ???????? {"DefaultMaxRAMFraction", "4"}, From daniel.daugherty at oracle.com Thu Nov 23 02:16:35 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 22 Nov 2017 21:16:35 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> Message-ID: <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> Greetings, I've made minor tweaks to the Thread-SMR project based on the remaining code review comments over the last couple of days. I've also rebased the project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] testing... Here's a delta webrev for the latest round of tweaks: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta Here's the proposed commit message: $ cat SMR_prototype_10/open/commit.txt 8167108: inconsistent handling of SR_lock can lead to crashes Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, kbarrett, rehn, sspitsyn, stefank Contributed-by: daniel.daugherty at oracle.com, erik.osterlund at oracle.com, robbin.ehn at oracle.com I've gone through 880+ emails in my archive for this project and added anyone that made a code review comment. Reviewers are not in my usual by-comment-posting order; it was way too hard to figure that out... So reviewers and contributors are simply in alpha-sort order... Here's the collection of follow-up bugs that I filed based on sifting through the email archive: 8191784 JavaThread::collect_counters() should stop using Threads_lock https://bugs.openjdk.java.net/browse/JDK-8191784 8191785 revoke_bias() needs a new assertion https://bugs.openjdk.java.net/browse/JDK-8191785 8191786 Thread-SMR hash table size should be dynamic https://bugs.openjdk.java.net/browse/JDK-8191786 8191787 move private inline functions from thread.inline.hpp -> thread.cpp https://bugs.openjdk.java.net/browse/JDK-8191787 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> threadSMR.[ch]pp https://bugs.openjdk.java.net/browse/JDK-8191789 8191798 redo nested ThreadsListHandle to drop Threads_lock https://bugs.openjdk.java.net/browse/JDK-8191789 I have undoubtedly missed at least one future idea that someone had and for that my apologies... Thanks again to everyone who contributed to the Thread-SMR project!! Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for their rigorous review of the design that we documented on the wiki. Thanks for keeping us honest! I also have to thank my co-contributor Erik ?sterlund for starting the ball rolling on this crazy project with his presentation on Safe Memory Reclamation, for the initial prototype and for all your patches. Erik, hopefully we got far enough in your crazy vision... :-) Thanks to my co-contributor Robbin Ehn for proposing that we do early code reviews in the form of patches. Don't like something? Fix it with a patch and a new test if appropriate!! A pretty cool way to work that was new to me. Finally, many thanks to Jerry Thornbrugh for all the early code reviews, whiteboard chats and phone calls. Thanks for asking the right questions and pointing out some places where our logic was faulty. Also thanks for keeping me sane when I was literally on my own in Monument, CO. Dan On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: > Greetings, > > *** We are wrapping up code review on this project so it is time *** > *** for the various code reviewers to chime in one last time. *** > > In this latest round, we had three different reviewers chime in so we're > doing delta webrevs for each of those resolutions: > > David H's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ > > > ? mq comment for dholmes_CR: > ??? - Fix indents, trailing spaces and various typos. > ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, > ????? add impl notes to document the type choices. > > ? src/hotspot/share/runtime/globals.hpp change is white-space only so it > ? is only visible via the file's patch link. > > > Robin W's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ > > > ? mq comment for robinw_CR: > ??? - Fix some inefficient code, update some comments, fix some indents, > ????? and add some 'const' specifiers. > > ? src/hotspot/share/runtime/vm_operations.hpp change is white-space > only so > ? it is only visible via the file's patch link. > > > Coleen's resolutions: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ > > > ? mq comment for coleenp_CR: > ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', > ??? - add header comment to threadSMR.hpp file, > ??? - cleanup JavaThreadIteratorWithHandle ctr, > ??? - make ErrorHandling more efficient. > > > Some folks prefer to see all of the delta webrevs together so here is > a delta webrev relative to jdk10-09-full: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ > > > ? src/hotspot/share/runtime/globals.hpp and > ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space > only > ? so they are only visible via the file's patch link. > > > And, of course, some folks prefer to see everything all at once so here > is the latest full webrev: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ > > > Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The > prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... > > The project is currently baselined on Jesper's 2017-11-17 PIT snapshot > so there will be some catching up to do before integration... > > We welcome comments, suggestions and feedback. > > Dan, Erik and Robbin > > > On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Testing of the last round of changes revealed a hang in one of the new >> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >> added another TLH test for good measure. >> >> Here is the updated full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >> >> Here is the updated delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >> >> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >> no unexpected failures. >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Robbin rebased the project last night/this morning to merge with Thread >>> Local Handshakes (TLH) and also picked up additional changesets up >>> thru: >>> >>>> Changeset: fa736014cf28 >>>> Author:??? cjplummer >>>> Date:????? 2017-11-14 18:08 -0800 >>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>> >>>> 8191049: Add alternate version of pns() that is callable from >>>> within hotspot source >>>> Summary: added pns2() to debug.cpp >>>> Reviewed-by: stuefe, gthornbr >>> >>> This is the first time we've rebased the project to bits that are this >>> fresh (< 12 hours old at rebase time). We've done this because we think >>> we're done with this project and are in the final review-change-retest >>> cycle(s)... In other words, we're not planning on making any more major >>> changes for this project... :-) >>> >>> *** Time for code reviewers to chime in on this thread! *** >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>> >>> Here's is the delta webrev from the 2017.11.10 rebase: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>> (and the new baseline also)... >>> >>> We're expecting this round to be a little noisier than usual because >>> we did not rebase on a PIT snapshot so the new baseline has not been >>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>> (Yes, we're playing chase-the-repo...) >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>> >>>> Unlike the previous rebase, there were no changes required to the >>>> open code to get the rebased bits to build so there is no delta >>>> webrev. >>>> >>>> These bits have been run through JPRT and I've submitted the usual >>>> Mach5 tier[1-5] test run... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>> >>>>> Here are the updated webrevs: >>>>> >>>>> Here's the mq comment for the change: >>>>> >>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>> >>>>> Here is the full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>> >>>>> And here is the delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>> >>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>> holiday >>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>> closer >>>>> look today. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Resolving one of the code review comments (from both Stefan K and >>>>>> Coleen) >>>>>> on jdk10-04-full required quite a few changes so it is being done >>>>>> as a >>>>>> standalone patch and corresponding webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to >>>>>> use >>>>>> ??? JavaThreadIteratorWithHandle. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>> >>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>> >>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>> >>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>> describe >>>>>>> and track the work on this project: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>> >>>>>>> >>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>> quotes >>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>> have a >>>>>>> solution for that problem yet. >>>>>>> >>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>> >>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>> tier[2-5] >>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>> additional testing on Erik and Robbin's machines. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Daniel Daugherty >>>>>>> Erik Osterlund >>>>>>> Robbin Ehn >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > From robbin.ehn at oracle.com Thu Nov 23 08:20:01 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 23 Nov 2017 09:20:01 +0100 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> Message-ID: <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> Thanks Dan for dragging this freight train to the docks, it's time to ship it! Created follow-up bug: 8191809: Threads::number_of_threads() is not 'MT-safe' https://bugs.openjdk.java.net/browse/JDK-8191809 /Robbin On 2017-11-23 03:16, Daniel D. Daugherty wrote: > Greetings, > > I've made minor tweaks to the Thread-SMR project based on the remaining > code review comments over the last couple of days. I've also rebased the > project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm > running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] > testing... > > Here's a delta webrev for the latest round of tweaks: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta > > > Here's the proposed commit message: > > $ cat SMR_prototype_10/open/commit.txt > 8167108: inconsistent handling of SR_lock can lead to crashes > Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. > Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, kbarrett, rehn, sspitsyn, stefank > Contributed-by: daniel.daugherty at oracle.com, erik.osterlund at oracle.com, robbin.ehn at oracle.com > > I've gone through 880+ emails in my archive for this project and added > anyone that made a code review comment. Reviewers are not in my usual > by-comment-posting order; it was way too hard to figure that out... So > reviewers and contributors are simply in alpha-sort order... > > Here's the collection of follow-up bugs that I filed based on sifting > through the email archive: > > 8191784 JavaThread::collect_counters() should stop using Threads_lock > https://bugs.openjdk.java.net/browse/JDK-8191784 > > 8191785 revoke_bias() needs a new assertion > https://bugs.openjdk.java.net/browse/JDK-8191785 > > 8191786 Thread-SMR hash table size should be dynamic > https://bugs.openjdk.java.net/browse/JDK-8191786 > > 8191787 move private inline functions from thread.inline.hpp -> thread.cpp > https://bugs.openjdk.java.net/browse/JDK-8191787 > > 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> threadSMR.[ch]pp > https://bugs.openjdk.java.net/browse/JDK-8191789 > > 8191798 redo nested ThreadsListHandle to drop Threads_lock > https://bugs.openjdk.java.net/browse/JDK-8191789 > > I have undoubtedly missed at least one future idea that someone had > and for that my apologies... > > Thanks again to everyone who contributed to the Thread-SMR project!! > > Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for > their rigorous review of the design that we documented on the wiki. > Thanks for keeping us honest! > > I also have to thank my co-contributor Erik ?sterlund for starting the > ball rolling on this crazy project with his presentation on Safe Memory > Reclamation, for the initial prototype and for all your patches. Erik, > hopefully we got far enough in your crazy vision... :-) > > Thanks to my co-contributor Robbin Ehn for proposing that we do early > code reviews in the form of patches. Don't like something? Fix it with > a patch and a new test if appropriate!! A pretty cool way to work that > was new to me. > > Finally, many thanks to Jerry Thornbrugh for all the early code reviews, > whiteboard chats and phone calls. Thanks for asking the right questions > and pointing out some places where our logic was faulty. Also thanks for > keeping me sane when I was literally on my own in Monument, CO. > > Dan > > > On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> *** We are wrapping up code review on this project so it is time *** >> *** for the various code reviewers to chime in one last time. *** >> >> In this latest round, we had three different reviewers chime in so we're >> doing delta webrevs for each of those resolutions: >> >> David H's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >> >> ? mq comment for dholmes_CR: >> ??? - Fix indents, trailing spaces and various typos. >> ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, >> ????? add impl notes to document the type choices. >> >> ? src/hotspot/share/runtime/globals.hpp change is white-space only so it >> ? is only visible via the file's patch link. >> >> >> Robin W's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >> >> ? mq comment for robinw_CR: >> ??? - Fix some inefficient code, update some comments, fix some indents, >> ????? and add some 'const' specifiers. >> >> ? src/hotspot/share/runtime/vm_operations.hpp change is white-space only so >> ? it is only visible via the file's patch link. >> >> >> Coleen's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >> >> ? mq comment for coleenp_CR: >> ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >> ??? - add header comment to threadSMR.hpp file, >> ??? - cleanup JavaThreadIteratorWithHandle ctr, >> ??? - make ErrorHandling more efficient. >> >> >> Some folks prefer to see all of the delta webrevs together so here is >> a delta webrev relative to jdk10-09-full: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >> >> ? src/hotspot/share/runtime/globals.hpp and >> ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space only >> ? so they are only visible via the file's patch link. >> >> >> And, of course, some folks prefer to see everything all at once so here >> is the latest full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >> >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >> >> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >> so there will be some catching up to do before integration... >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage to use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > From robbin.ehn at oracle.com Thu Nov 23 08:42:09 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 23 Nov 2017 09:42:09 +0100 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Message-ID: <44d10d22-a801-235b-dc8e-cc3062352e10@oracle.com> Thanks Dan. /Robbin On 2017-11-23 01:24, Daniel D. Daugherty wrote: > On 11/22/17 3:37 PM, Robbin Ehn wrote: >> Hi all, please review. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 >> >> Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. >> >> Thanks, Robbin! > > Thumbs up on the change. We should put a comment in the code where > deprecated options are put to remind folks to update this test. > > Dan > > >> >> diff -r 2cd1c2b03782 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 01:12:23 2017 -0800 >> +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 21:28:36 2017 +0100 >> @@ -46,6 +46,11 @@ >> ???????? {"MinRAMFraction",??????????? "2"}, >> ???????? {"InitialRAMFraction",??????? "64"}, >> ???????? {"AssumeMP",????????????????? "false"}, >> +??????? {"UseMembar",???????????????? "true"}, >> +??????? {"FastTLABRefill",??????????? "false"}, >> +??????? {"DeferPollingPageLoopCount", "-1"}, >> +??????? {"SafepointSpinBeforeYield",? "2000"}, >> +??????? {"DeferPollingPageLoopCount", "4000"}, >> >> ???????? // deprecated alias flags (see also aliased_jvm_flags): >> ???????? {"DefaultMaxRAMFraction", "4"}, > From robbin.ehn at oracle.com Thu Nov 23 08:43:34 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 23 Nov 2017 09:43:34 +0100 Subject: RFR(one-liner): 8191707: Options with invalid values are incorrectly treated as obsolete and ignored In-Reply-To: <54ab4e67-abe2-9801-d237-e4052f9f74a4@oracle.com> References: <80a99246-28f9-3035-2469-7287eadfb31f@oracle.com> <54ab4e67-abe2-9801-d237-e4052f9f74a4@oracle.com> Message-ID: <046340cd-7cbb-2840-d0eb-03a83c61aee7@oracle.com> Thanks Dan. On 2017-11-23 01:19, Daniel D. Daugherty wrote: > On 11/22/17 3:24 PM, Robbin Ehn wrote: >> Hi all, please review: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191707 >> >> Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. >> >> Contributed by David Holmes. >> >> Looks good, thanks for fixing! > > Thumbs up on the fix. > > What I don't understand is why this logic did not fall over before now. Same here, Robbin > > Dan > >> >> /Robbin >> >> >> diff -r 2cd1c2b03782 src/hotspot/share/runtime/arguments.cpp >> --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 01:12:23 2017 -0800 >> +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 21:20:42 2017 +0100 >> @@ -493,15 +493,15 @@ >> ?} >> >> ?bool Arguments::is_obsolete_flag(const char *flag_name, JDK_Version* version) { >> ?? assert(version != NULL, "Must provide a version buffer"); >> ?? SpecialFlag flag; >> ?? if (lookup_special_flag(flag_name, flag)) { >> ???? if (!flag.obsolete_in.is_undefined()) { >> -????? if (version_less_than(JDK_Version::current(), flag.expired_in)) { >> +????? if (!version_less_than(JDK_Version::current(), flag.obsolete_in)) { >> ???????? *version = flag.obsolete_in; >> ???????? return true; >> ?????? } >> ???? } >> ?? } >> ?? return false; >> ?} > From rahul.v.raghavan at oracle.com Thu Nov 23 08:56:37 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Thu, 23 Nov 2017 14:26:37 +0530 Subject: [10] RFR: 8191227: issues with unsafe handle resolution Message-ID: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Hi, Please review the following fix proposal. webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ Please check jbs comments - jbs - https://bugs.openjdk.java.net/browse/JDK-8191227 - related - https://bugs.openjdk.java.net/browse/JDK-8160166 Fix tried - -- ConstantOopWriteValue::write_on() Used ThreadInVMfromUnknown and added required comment. -- ConstantOopWriteValue::print_on() found no issue in testing with usage of ThreadInVMfromNative here. -- LIR_Assembler::jobject2reg() Similarly used ThreadInVMfromNative, before the assert. No new issues with testing - tier1, tier2, tier3, compiler pre-checkin and also jprt (-testset hotspot). Thanks, Rahul From adinn at redhat.com Thu Nov 23 10:52:46 2017 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 23 Nov 2017 10:52:46 +0000 Subject: [8u-backport] 8139873: NMT stack traces in output should show mt component In-Reply-To: <6fba1c24-373c-f6c1-06c9-f5d804826764@redhat.com> References: <628dceaa-89de-2754-fbc8-9d46e6c349ec@redhat.com> <0ae1f7ff-ab7c-a8f0-4b44-ac4da2bedfad@redhat.com> <6fba1c24-373c-f6c1-06c9-f5d804826764@redhat.com> Message-ID: <5bc44c7c-f12b-612f-9550-ecfbacda8c8b@redhat.com> On 22/11/17 21:22, Zhengyu Gu wrote: > Thanks for the quick review, Aleksey. > . . . > Updated: > http://cr.openjdk.java.net/~zgu/8139673/8139673/webrev.8u_backport.01/ Strictly, I'm not a jdk8 reviewer. However, I checked this and the patch looks good and builds/runs as expected. regards, Andrew Dinn ----------- From vladimir.x.ivanov at oracle.com Thu Nov 23 11:49:05 2017 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Thu, 23 Nov 2017 14:49:05 +0300 Subject: [10] RFR: 8191227: issues with unsafe handle resolution In-Reply-To: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> References: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Message-ID: > webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ Please, use ThreadInVMfromUnknown in ConstantOopWriteValue::print_on() as well. Otherwise, looks good. Best regards, Vladimir Ivanov > > > Please check jbs comments - > jbs - https://bugs.openjdk.java.net/browse/JDK-8191227 > ??? - related - https://bugs.openjdk.java.net/browse/JDK-8160166 > > Fix tried - > -- ConstantOopWriteValue::write_on() > ??? Used ThreadInVMfromUnknown and added required comment. > -- ConstantOopWriteValue::print_on() > ??? found no issue in testing with usage of ThreadInVMfromNative here. > -- LIR_Assembler::jobject2reg() > ??? Similarly used ThreadInVMfromNative, before the assert. > > No new issues with testing - tier1, tier2, tier3, compiler pre-checkin > ?and also jprt (-testset hotspot). > > > Thanks, > Rahul From daniel.daugherty at oracle.com Thu Nov 23 13:31:30 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 23 Nov 2017 08:31:30 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> Message-ID: <28edcbec-4e10-6c3c-df46-5dba25a93c58@oracle.com> On 11/23/17 3:20 AM, Robbin Ehn wrote: > Thanks Dan for dragging this freight train to the docks, it's time to > ship it! Working on that very thing today... will likely push late today (my TZ)... > Created follow-up bug: > 8191809: Threads::number_of_threads() is not 'MT-safe' > https://bugs.openjdk.java.net/browse/JDK-8191809 Nice follow-up! Thanks for filing it. Dan > > /Robbin > > On 2017-11-23 03:16, Daniel D. Daugherty wrote: >> Greetings, >> >> I've made minor tweaks to the Thread-SMR project based on the remaining >> code review comments over the last couple of days. I've also rebased the >> project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm >> running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] >> testing... >> >> Here's a delta webrev for the latest round of tweaks: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta >> >> >> Here's the proposed commit message: >> >> $ cat SMR_prototype_10/open/commit.txt >> 8167108: inconsistent handling of SR_lock can lead to crashes >> Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. >> Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, >> kbarrett, rehn, sspitsyn, stefank >> Contributed-by: daniel.daugherty at oracle.com, >> erik.osterlund at oracle.com, robbin.ehn at oracle.com >> >> I've gone through 880+ emails in my archive for this project and added >> anyone that made a code review comment. Reviewers are not in my usual >> by-comment-posting order; it was way too hard to figure that out... So >> reviewers and contributors are simply in alpha-sort order... >> >> Here's the collection of follow-up bugs that I filed based on sifting >> through the email archive: >> >> 8191784 JavaThread::collect_counters() should stop using Threads_lock >> https://bugs.openjdk.java.net/browse/JDK-8191784 >> >> 8191785 revoke_bias() needs a new assertion >> https://bugs.openjdk.java.net/browse/JDK-8191785 >> >> 8191786 Thread-SMR hash table size should be dynamic >> https://bugs.openjdk.java.net/browse/JDK-8191786 >> >> 8191787 move private inline functions from thread.inline.hpp -> >> thread.cpp >> https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> >> threadSMR.[ch]pp >> https://bugs.openjdk.java.net/browse/JDK-8191789 >> >> 8191798 redo nested ThreadsListHandle to drop Threads_lock >> https://bugs.openjdk.java.net/browse/JDK-8191789 >> >> I have undoubtedly missed at least one future idea that someone had >> and for that my apologies... >> >> Thanks again to everyone who contributed to the Thread-SMR project!! >> >> Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for >> their rigorous review of the design that we documented on the wiki. >> Thanks for keeping us honest! >> >> I also have to thank my co-contributor Erik ?sterlund for starting the >> ball rolling on this crazy project with his presentation on Safe Memory >> Reclamation, for the initial prototype and for all your patches. Erik, >> hopefully we got far enough in your crazy vision... :-) >> >> Thanks to my co-contributor Robbin Ehn for proposing that we do early >> code reviews in the form of patches. Don't like something? Fix it with >> a patch and a new test if appropriate!! A pretty cool way to work that >> was new to me. >> >> Finally, many thanks to Jerry Thornbrugh for all the early code reviews, >> whiteboard chats and phone calls. Thanks for asking the right questions >> and pointing out some places where our logic was faulty. Also thanks for >> keeping me sane when I was literally on my own in Monument, CO. >> >> Dan >> >> >> On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> *** We are wrapping up code review on this project so it is time *** >>> *** for the various code reviewers to chime in one last time. *** >>> >>> In this latest round, we had three different reviewers chime in so >>> we're >>> doing delta webrevs for each of those resolutions: >>> >>> David H's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >>> >>> >>> ? mq comment for dholmes_CR: >>> ??? - Fix indents, trailing spaces and various typos. >>> ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, >>> ????? add impl notes to document the type choices. >>> >>> ? src/hotspot/share/runtime/globals.hpp change is white-space only >>> so it >>> ? is only visible via the file's patch link. >>> >>> >>> Robin W's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >>> >>> >>> ? mq comment for robinw_CR: >>> ??? - Fix some inefficient code, update some comments, fix some >>> indents, >>> ????? and add some 'const' specifiers. >>> >>> ? src/hotspot/share/runtime/vm_operations.hpp change is white-space >>> only so >>> ? it is only visible via the file's patch link. >>> >>> >>> Coleen's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >>> >>> >>> ? mq comment for coleenp_CR: >>> ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >>> ??? - add header comment to threadSMR.hpp file, >>> ??? - cleanup JavaThreadIteratorWithHandle ctr, >>> ??? - make ErrorHandling more efficient. >>> >>> >>> Some folks prefer to see all of the delta webrevs together so here is >>> a delta webrev relative to jdk10-09-full: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >>> >>> >>> ? src/hotspot/share/runtime/globals.hpp and >>> ? src/hotspot/share/runtime/vm_operations.hpp changes are >>> white-space only >>> ? so they are only visible via the file's patch link. >>> >>> >>> And, of course, some folks prefer to see everything all at once so here >>> is the latest full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >>> >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >>> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >>> >>> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >>> so there will be some catching up to do before integration... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>> test, and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are >>>>> this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final >>>>> review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>> code quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>> server, and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> From marcus.larsson at oracle.com Fri Nov 24 10:34:37 2017 From: marcus.larsson at oracle.com (Marcus Larsson) Date: Fri, 24 Nov 2017 11:34:37 +0100 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Message-ID: <11a0b86d-9a7b-3650-58bf-b746323c9d8d@oracle.com> Hi, Looks good! Thanks, Marcus On 2017-11-22 21:37, Robbin Ehn wrote: > Hi all, please review. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 > > Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no > unexpected failure. > > Thanks, Robbin! > > diff -r 2cd1c2b03782 > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 01:12:23 2017 -0800 > +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 21:28:36 2017 +0100 > @@ -46,6 +46,11 @@ > ???????? {"MinRAMFraction",??????????? "2"}, > ???????? {"InitialRAMFraction",??????? "64"}, > ???????? {"AssumeMP",????????????????? "false"}, > +??????? {"UseMembar",???????????????? "true"}, > +??????? {"FastTLABRefill",??????????? "false"}, > +??????? {"DeferPollingPageLoopCount", "-1"}, > +??????? {"SafepointSpinBeforeYield",? "2000"}, > +??????? {"DeferPollingPageLoopCount", "4000"}, > > ???????? // deprecated alias flags (see also aliased_jvm_flags): > ???????? {"DefaultMaxRAMFraction", "4"}, From rahul.v.raghavan at oracle.com Fri Nov 24 11:16:23 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Fri, 24 Nov 2017 16:46:23 +0530 Subject: [10] RFR: 8191813: compiler/runtime/SpreadNullArg.java fails in tier1 Message-ID: Hi, Please review the following fix proposal for 8191813. webrev - http://cr.openjdk.java.net/~rraghavan/8191813/webrev.00/ jbs - https://bugs.openjdk.java.net/browse/JDK-8191813 -- reported issue - After the latest sync from jdk/jdk the test SpreadNullArg started to fail in tier1 on all platforms. -- Found this SpreadNullArg.java failure Caused after JDK-8157246 fix. jbs - https://bugs.openjdk.java.net/browse/JDK-8157246. 'MHs.arrayLength, arrayElementGetter/Setter, arrayConstructor need to specify invocation-time behavior'. changeset - http://hg.openjdk.java.net/jdk/jdk/rev/76519338df34 - So now with 8157246 fix, if method handle is invoked with a null array reference, NullPointerException will be thrown. (Not IllegalArgumentException) - So above fix proposal in SpreadNullArg.java to expect NullPointerException after 8157246 changes. Thanks, Rahul From robbin.ehn at oracle.com Fri Nov 24 12:56:16 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 24 Nov 2017 13:56:16 +0100 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: <11a0b86d-9a7b-3650-58bf-b746323c9d8d@oracle.com> References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> <11a0b86d-9a7b-3650-58bf-b746323c9d8d@oracle.com> Message-ID: <3a04accd-bc6d-230c-0ab8-2aa06dbd2f7e@oracle.com> Thanks Robbin! On 2017-11-24 11:34, Marcus Larsson wrote: > Hi, > > Looks good! > > Thanks, > Marcus > > On 2017-11-22 21:37, Robbin Ehn wrote: >> Hi all, please review. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 >> >> Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. >> >> Thanks, Robbin! >> >> diff -r 2cd1c2b03782 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 01:12:23 2017 -0800 >> +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 21:28:36 2017 +0100 >> @@ -46,6 +46,11 @@ >> ???????? {"MinRAMFraction",??????????? "2"}, >> ???????? {"InitialRAMFraction",??????? "64"}, >> ???????? {"AssumeMP",????????????????? "false"}, >> +??????? {"UseMembar",???????????????? "true"}, >> +??????? {"FastTLABRefill",??????????? "false"}, >> +??????? {"DeferPollingPageLoopCount", "-1"}, >> +??????? {"SafepointSpinBeforeYield",? "2000"}, >> +??????? {"DeferPollingPageLoopCount", "4000"}, >> >> ???????? // deprecated alias flags (see also aliased_jvm_flags): >> ???????? {"DefaultMaxRAMFraction", "4"}, > From erik.osterlund at oracle.com Fri Nov 24 14:45:36 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 24 Nov 2017 15:45:36 +0100 Subject: RFR (S): 8191681: Unsafe accesses should use field_offset_to_byte_offset() Message-ID: <5A183090.5000705@oracle.com> Hi, Before the Access API, there was a no-op conversion function in unsafe.cpp for getting the byte offset of a field offset from unsafe as a jlong. Now, the field offsets *are* byte offsets, so the conversion did not do anything. But I guess it could be useful nevertheless for abstraction purposes. In this patch I have a new function called field_offset_to_access_offset that gets the offset to be used by Access (ptrdiff_t) from a field offset, and uses that for the access calls. Bug: https://bugs.openjdk.java.net/browse/JDK-8191681 Webrev: http://cr.openjdk.java.net/~eosterlund/8191681/webrev.00/ Thanks, /Erik From volker.simonis at gmail.com Fri Nov 24 14:51:02 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 24 Nov 2017 15:51:02 +0100 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> Message-ID: Hi Ioi, you patch is not applying any more after "8187118: Remove appending -cp path to the boot class path at AppCDS dump time" was pushed to jdk/hs because of conflicts in classLoaderExt.hpp. Could you please rebase your change upon the newest version of jdk/hs ? By the way, on ppc64 all the tests passed after I found and fixed a trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite invokefinal if DumpSharedSpaces" for it). Can you please explain why AppCDS with custom class loaders is currently restricted to Linux/x86_64 and 64-bit Solaris? With the following small patch: diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 15:37:19 2017 +0100 +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 15:38:21 2017 +0100 @@ -272,6 +272,7 @@ // during archive dumping. InstanceKlass* ClassListParser::load_class_from_source(Symbol* class_name, TRAPS) { #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ + (defined(PPC) && defined(_LP64)) || \ (defined(SOLARIS) && defined(_LP64))) // The only supported platforms are: (1) Linux/AMD64; (2) Solaris/64-bit error("AppCDS custom class loaders not supported on this platform"); diff -r 23deffd3a9c4 test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java --- a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java Fri Nov 24 15:37:19 2017 +0100 +++ b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java Fri Nov 24 15:38:21 2017 +0100 @@ -56,6 +56,7 @@ OutputAnalyzer out = TestCommon.dump(appJar, classlist); if ((Platform.isSolaris() && Platform.is64bit()) || + (Platform.isLinux() && Platform.isPPC()) || (Platform.isLinux() && Platform.isX64())) { out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); out.shouldHaveExitValue(0); I could at least pass all the relevant jtreg tests on Linux/ppc64. So if there are no other hidden reasons I'd kindly ask you to add this patch to you change. I'm currently testing on AIX and I hope the tests will also succeed. If this will be the case, the list in the aforementioned patch could also be extended by AIX. Finally I've also did some small fixes on s390 (opened "8191863: [s390] Fix CDS: some bytecode rewriting doesn't depend on RewriteControl" for it). Currently I'm building and running some tests and I hope that until Monday we may even extend the list of supported platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if that works as well, the check could be simplified to allow all the 64-bit Linux platforms. Thank you and best regards, Volker On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: > Hi Volker, > > Thanks for trying the patch out. > > > On 11/17/17 5:42 AM, Volker Simonis wrote: >> >> Hi Ioi, >> >> I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX >> ports. >> >> I applied your patch to the current jdk-hs repo as of today and first >> of all I see the same compilation problems already reported by Dmitry >> before. Can you please update you webrev with the corresponding >> changes or create a new one. That will help other who wish to look at >> this change. > > > I've updated the patch inside the webrev. You can download it from > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch > > > The previous patch (which you had the problems with) is still available as > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old > > I will post a new webrev, but it's big and I might be running out of quota > from cr.openjdk.java.net, so I need to figure out how to handle that. > > The jdk/hs changeset corresponding to the new patch is the following: > > changeset: 47860:564882d918d4 > user: zgu > date: Thu Nov 16 20:21:11 2017 -0500 > files: src/hotspot/share/services/memTracker.cpp > description: > 8190357: NMT: Include metadata information in NMT final report when > PrintNMTStatistics is on > Summary: Include metadata information in NMT final report > Reviewed-by: adinn, stuefe > >> After building, I ran the appcds jtreg tests (i.e. >> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see >> the same four failures like Dmitry. > > Those are also due to merge issues. With the updated patch, I passed all > tests under on Linux/x64: > > test/hotspot/jtreg/runtime/SharedArchiveFile > test/hotspot/jtreg/runtime/CDSCompressedKPtrs > test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java > test/hotspot/jtreg/runtime/appcds > > $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 > -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk > -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 > -retain -agentvm > -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt > -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt > -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native > -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ > open/test/hotspot/jtreg/runtime/SharedArchiveFile > open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs > open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java > open/test/hotspot/jtreg/runtime/appcds > [...] > Test results: passed: 128 > Results written to /jdk/tmp/jtreg/work > > >> Is that expected? I see a lot more errors on Linux/ppc64le but before >> going into more detail I'd like to know what is the correct baseline >> on which we can rely. >> >> Finally, I saw that with a product build, I get two additional test >> failures: >> >> runtime/appcds/ProhibitedPackage.java >> runtime/appcds/DumpClassList.java >> >> because of: >> >> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >> available only in debug version of VM. >> >> So these two tests should probably be adjusted to only run for >> slow/fastdebug builds. > > > You're correct. I will exclude those tests from product builds, and file an > RFE to fix them later. > > Thanks > - Ioi > >> Thank you and best regards, >> Volker >> >> >> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >> wrote: >>> >>> Ioi, >>> >>> Thank you! >>> >>> I did some additional testing and find that four tests >>> fails the same way on both x86_64 and AARCH64 (see below). >>> >>> runtime/appcds/VerifierTest_0.java >>> runtime/appcds/cacheObject/GCStressTest.java >>> runtime/appcds/cacheObject/RedefineClassTest.java >>> runtime/appcds/sharedStrings/InternSharedString.java >>> >>> Is it expected? >>> >>> test result: Failed. Execution failed: `main' threw exception: >>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>> asses' missing from stdout/stderr >>> >>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>> GCStressApp_nonshared_string should not be shared >>> at GCStressApp.main(GCStressApp.java:73) >>> >>> >>> FAILED. buzzshould not be shared >>> >>> Exception in thread "main" java.lang.RuntimeException: Failed: >>> unexpected shared string. >>> at InternStringTest.main(InternStringTest.java:63) >>> >>> -Dmitry >>> >>> On 16.11.2017 03:10, Ioi Lam wrote: >>>> >>>> I filed: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>>> VMProps to filter out CDS tests >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>>> when CDS is not supported >>>> >>>> Thanks >>>> >>>> - Ioi >>>> >>>> >>>> >>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>> >>>>> Hi Dmitry, >>>>> >>>>> Thanks for the review. >>>>> >>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>> >>>>>> Ioi, >>>>>> >>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>> below[1]. >>>>> >>>>> I've fixed it. Thanks for the patch. >>>>>> >>>>>> Also tests doesn't work with exploded image - is it possible to check >>>>>> for this case and produce appropriate error message? >>>>> >>>>> CDS is not supported on exploded images. I think the best thing to do >>>>> is to add something like "@requires vm.cds" into the test cases >>>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>>> none of the CDS tests will be executed for >>>>> >>>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>>> it in the next release. >>>>> >>>>> I'll also file another RFE to print a better message. Today if you use >>>>> an exploded build the error message is kind of confusing: >>>>> >>>>> $ ./jdk/bin/java -Xshare:dump >>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>> Error occurred during initialization of VM >>>>> CDS allows only empty directories in archived classpaths >>>>> >>>>> It should say something like "CDS is not supported on exploded images". >>>>> >>>>> Thanks >>>>> - Ioi >>>>> >>>>>> 1. >>>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>> 09:45:23 2017 +0000 >>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>> 15:02:51 2017 +0000 >>>>>> @@ -31,7 +31,6 @@ >>>>>> -#include "prims/jvm.h" >>>>>> >>>>>> diff -r dbf9cec6a568 >>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>> 06 09:45:23 2017 +0000 >>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>> 06 15:02:51 2017 +0000 >>>>>> @@ -518,7 +518,7 @@ >>>>>> >>>>>> { >>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>>> if (check != NULL) { >>>>>> return InstanceKlass::cast(check); >>>>>> } >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>>> During internal review of the JEP, we have decided to integrate both >>>>>>> implementation and tests at the same time. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>> >>>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>>> result of copying the closed source files on top of files of the same >>>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>>> "changed", please focus on the entire content of the new version of >>>>>>> each >>>>>>> file. >>>>>>> >>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>> sources) and all the new appcds tests passed. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> - Ioi >>>>>>> >>>>>>> >>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>> >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>> >>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> - Ioi >>>>>>>> >>>>>>>> >>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>> >>>>>>>>> Ioi, >>>>>>>>> >>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>> the compilation fails with: >>>>>>>>> >>>>>>>>> >>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>> >>>>>>>>> >>>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>>> >>>>>>>>> Did I miss something? >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Please review this change set. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>> >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>> >>>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>>> moves >>>>>>>>>> AppCDS from >>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>> >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>> >>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>>> repo. >>>>>>>>>> As part >>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>>> code >>>>>>>>>> into the >>>>>>>>>> open repo. >>>>>>>>>> >>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>> possible. The >>>>>>>>>> intention is >>>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>>> and not >>>>>>>>>> to do any sort of refactoring. >>>>>>>>>> >>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>> copying the >>>>>>>>>> closed source >>>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>>> reviewing, instead of >>>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>>> content >>>>>>>>>> of the new >>>>>>>>>> version of each file. >>>>>>>>>> >>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>> flag is >>>>>>>>>> changed from >>>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>>> "commercial" >>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>> >>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>> open/closed >>>>>>>>>> source >>>>>>>>>> code structure had introduced some intermediary APIs to connect >>>>>>>>>> code >>>>>>>>>> between >>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>> >>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>>> tests >>>>>>>>>> will be >>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>> >>>>>>>>>> All the AppCDS tests (currently still in closed sources) passed >>>>>>>>>> with >>>>>>>>>> both Oracle JDK >>>>>>>>>> and OpenJDK. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> - Ioi >>> >>> > From daniel.daugherty at oracle.com Fri Nov 24 15:26:08 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 24 Nov 2017 10:26:08 -0500 Subject: [10] RFR: 8191813: compiler/runtime/SpreadNullArg.java fails in tier1 In-Reply-To: References: Message-ID: <752a4e80-9141-6115-dee6-2bbd0ec99e92@oracle.com> On 11/24/17 6:16 AM, Rahul Raghavan wrote: > Hi, > > Please review the following fix proposal for 8191813. > > webrev - http://cr.openjdk.java.net/~rraghavan/8191813/webrev.00/ test/hotspot/jtreg/compiler/runtime/SpreadNullArg.java ??? No comments. So this style of call: ??? mh_spreadInvoker.invokeExact(mh_spread_target, (Object[]) null); was changed from throwing IllegalArgumentException due to the "null" parameter to throwing NullPointerException. You've updated the test cleanly to switch from one exception to the other. Thumbs up! I believe this patch qualifies as trivial and does not need to wait for 24 hours. I leave it up to you whether you wait for a Compiler team member review also. Dan > > > jbs - https://bugs.openjdk.java.net/browse/JDK-8191813 > > -- reported issue - After the latest sync from jdk/jdk the test > SpreadNullArg started to fail in tier1 on all platforms. > > -- Found this SpreadNullArg.java failure Caused after JDK-8157246 fix. > jbs - https://bugs.openjdk.java.net/browse/JDK-8157246. > 'MHs.arrayLength, arrayElementGetter/Setter, arrayConstructor need to > specify invocation-time behavior'. > changeset - http://hg.openjdk.java.net/jdk/jdk/rev/76519338df34 > > - So now with 8157246 fix, if method handle is invoked with a null > array reference, NullPointerException will be thrown. (Not > IllegalArgumentException) > > - So above fix proposal in SpreadNullArg.java to expect > NullPointerException after 8157246 changes. > > > Thanks, > Rahul From dms at samersoff.net Fri Nov 24 16:08:03 2017 From: dms at samersoff.net (Dmitry Samersoff) Date: Fri, 24 Nov 2017 19:08:03 +0300 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> Message-ID: Volker, > Currently I'm building and running some tests > and I hope that until Monday we may even extend the list of supported > platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if > that works as well, the check could be simplified to allow all the > 64-bit Linux platforms. Yes, I added similar patch for Linux/AARCH64 and it works. So we should simplify the check. -Dmitry On 11/24/2017 05:51 PM, Volker Simonis wrote: > Hi Ioi, > > you patch is not applying any more after "8187118: Remove appending > -cp path to the boot class path at AppCDS dump time" was pushed to > jdk/hs because of conflicts in classLoaderExt.hpp. > > Could you please rebase your change upon the newest version of jdk/hs ? > > By the way, on ppc64 all the tests passed after I found and fixed a > trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite > invokefinal if DumpSharedSpaces" for it). > > Can you please explain why AppCDS with custom class loaders is > currently restricted to Linux/x86_64 and 64-bit Solaris? > > With the following small patch: > > diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp > --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 > 15:37:19 2017 +0100 > +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 > 15:38:21 2017 +0100 > @@ -272,6 +272,7 @@ > // during archive dumping. > InstanceKlass* ClassListParser::load_class_from_source(Symbol* > class_name, TRAPS) { > #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ > + (defined(PPC) && defined(_LP64)) || \ > (defined(SOLARIS) && defined(_LP64))) > // The only supported platforms are: (1) Linux/AMD64; (2) Solaris/64-bit > error("AppCDS custom class loaders not supported on this platform"); > diff -r 23deffd3a9c4 > test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java > --- a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java > Fri Nov 24 15:37:19 2017 +0100 > +++ b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java > Fri Nov 24 15:38:21 2017 +0100 > @@ -56,6 +56,7 @@ > OutputAnalyzer out = TestCommon.dump(appJar, classlist); > > if ((Platform.isSolaris() && Platform.is64bit()) || > + (Platform.isLinux() && Platform.isPPC()) || > (Platform.isLinux() && Platform.isX64())) { > out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); > out.shouldHaveExitValue(0); > > > I could at least pass all the relevant jtreg tests on Linux/ppc64. So > if there are no other hidden reasons I'd kindly ask you to add this > patch to you change. I'm currently testing on AIX and I hope the tests > will also succeed. If this will be the case, the list in the > aforementioned patch could also be extended by AIX. > > Finally I've also did some small fixes on s390 (opened "8191863: > [s390] Fix CDS: some bytecode rewriting doesn't depend on > RewriteControl" for it). Currently I'm building and running some tests > and I hope that until Monday we may even extend the list of supported > platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if > that works as well, the check could be simplified to allow all the > 64-bit Linux platforms. > > Thank you and best regards, > Volker > > On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: >> Hi Volker, >> >> Thanks for trying the patch out. >> >> >> On 11/17/17 5:42 AM, Volker Simonis wrote: >>> >>> Hi Ioi, >>> >>> I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX >>> ports. >>> >>> I applied your patch to the current jdk-hs repo as of today and first >>> of all I see the same compilation problems already reported by Dmitry >>> before. Can you please update you webrev with the corresponding >>> changes or create a new one. That will help other who wish to look at >>> this change. >> >> >> I've updated the patch inside the webrev. You can download it from >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch >> >> >> The previous patch (which you had the problems with) is still available as >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old >> >> I will post a new webrev, but it's big and I might be running out of quota >> from cr.openjdk.java.net, so I need to figure out how to handle that. >> >> The jdk/hs changeset corresponding to the new patch is the following: >> >> changeset: 47860:564882d918d4 >> user: zgu >> date: Thu Nov 16 20:21:11 2017 -0500 >> files: src/hotspot/share/services/memTracker.cpp >> description: >> 8190357: NMT: Include metadata information in NMT final report when >> PrintNMTStatistics is on >> Summary: Include metadata information in NMT final report >> Reviewed-by: adinn, stuefe >> >>> After building, I ran the appcds jtreg tests (i.e. >>> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see >>> the same four failures like Dmitry. >> >> Those are also due to merge issues. With the updated patch, I passed all >> tests under on Linux/x64: >> >> test/hotspot/jtreg/runtime/SharedArchiveFile >> test/hotspot/jtreg/runtime/CDSCompressedKPtrs >> test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >> test/hotspot/jtreg/runtime/appcds >> >> $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 >> -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk >> -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 >> -retain -agentvm >> -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt >> -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt >> -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native >> -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ >> open/test/hotspot/jtreg/runtime/SharedArchiveFile >> open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs >> open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >> open/test/hotspot/jtreg/runtime/appcds >> [...] >> Test results: passed: 128 >> Results written to /jdk/tmp/jtreg/work >> >> >>> Is that expected? I see a lot more errors on Linux/ppc64le but before >>> going into more detail I'd like to know what is the correct baseline >>> on which we can rely. >>> >>> Finally, I saw that with a product build, I get two additional test >>> failures: >>> >>> runtime/appcds/ProhibitedPackage.java >>> runtime/appcds/DumpClassList.java >>> >>> because of: >>> >>> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >>> available only in debug version of VM. >>> >>> So these two tests should probably be adjusted to only run for >>> slow/fastdebug builds. >> >> >> You're correct. I will exclude those tests from product builds, and file an >> RFE to fix them later. >> >> Thanks >> - Ioi >> >>> Thank you and best regards, >>> Volker >>> >>> >>> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >>> wrote: >>>> >>>> Ioi, >>>> >>>> Thank you! >>>> >>>> I did some additional testing and find that four tests >>>> fails the same way on both x86_64 and AARCH64 (see below). >>>> >>>> runtime/appcds/VerifierTest_0.java >>>> runtime/appcds/cacheObject/GCStressTest.java >>>> runtime/appcds/cacheObject/RedefineClassTest.java >>>> runtime/appcds/sharedStrings/InternSharedString.java >>>> >>>> Is it expected? >>>> >>>> test result: Failed. Execution failed: `main' threw exception: >>>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>>> asses' missing from stdout/stderr >>>> >>>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>>> GCStressApp_nonshared_string should not be shared >>>> at GCStressApp.main(GCStressApp.java:73) >>>> >>>> >>>> FAILED. buzzshould not be shared >>>> >>>> Exception in thread "main" java.lang.RuntimeException: Failed: >>>> unexpected shared string. >>>> at InternStringTest.main(InternStringTest.java:63) >>>> >>>> -Dmitry >>>> >>>> On 16.11.2017 03:10, Ioi Lam wrote: >>>>> >>>>> I filed: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>>>> VMProps to filter out CDS tests >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>>>> when CDS is not supported >>>>> >>>>> Thanks >>>>> >>>>> - Ioi >>>>> >>>>> >>>>> >>>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>>> >>>>>> Hi Dmitry, >>>>>> >>>>>> Thanks for the review. >>>>>> >>>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>>> >>>>>>> Ioi, >>>>>>> >>>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>>> below[1]. >>>>>> >>>>>> I've fixed it. Thanks for the patch. >>>>>>> >>>>>>> Also tests doesn't work with exploded image - is it possible to check >>>>>>> for this case and produce appropriate error message? >>>>>> >>>>>> CDS is not supported on exploded images. I think the best thing to do >>>>>> is to add something like "@requires vm.cds" into the test cases >>>>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>>>> none of the CDS tests will be executed for >>>>>> >>>>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>>>> it in the next release. >>>>>> >>>>>> I'll also file another RFE to print a better message. Today if you use >>>>>> an exploded build the error message is kind of confusing: >>>>>> >>>>>> $ ./jdk/bin/java -Xshare:dump >>>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>>> Error occurred during initialization of VM >>>>>> CDS allows only empty directories in archived classpaths >>>>>> >>>>>> It should say something like "CDS is not supported on exploded images". >>>>>> >>>>>> Thanks >>>>>> - Ioi >>>>>> >>>>>>> 1. >>>>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>> 09:45:23 2017 +0000 >>>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>> 15:02:51 2017 +0000 >>>>>>> @@ -31,7 +31,6 @@ >>>>>>> -#include "prims/jvm.h" >>>>>>> >>>>>>> diff -r dbf9cec6a568 >>>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>> 06 09:45:23 2017 +0000 >>>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>> 06 15:02:51 2017 +0000 >>>>>>> @@ -518,7 +518,7 @@ >>>>>>> >>>>>>> { >>>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>>>> if (check != NULL) { >>>>>>> return InstanceKlass::cast(check); >>>>>>> } >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>>>> During internal review of the JEP, we have decided to integrate both >>>>>>>> implementation and tests at the same time. >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>>> >>>>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>>>> result of copying the closed source files on top of files of the same >>>>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>>>> "changed", please focus on the entire content of the new version of >>>>>>>> each >>>>>>>> file. >>>>>>>> >>>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>>> sources) and all the new appcds tests passed. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> - Ioi >>>>>>>> >>>>>>>> >>>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>>> >>>>>>>>> Hi Dmitry, >>>>>>>>> >>>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>>> >>>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> - Ioi >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>>> >>>>>>>>>> Ioi, >>>>>>>>>> >>>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>>> the compilation fails with: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>>>> >>>>>>>>>> Did I miss something? >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Please review this change set. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>>> >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>>> >>>>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>>>> moves >>>>>>>>>>> AppCDS from >>>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>>> >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>>> >>>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>>>> repo. >>>>>>>>>>> As part >>>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>>>> code >>>>>>>>>>> into the >>>>>>>>>>> open repo. >>>>>>>>>>> >>>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>>> possible. The >>>>>>>>>>> intention is >>>>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>>>> and not >>>>>>>>>>> to do any sort of refactoring. >>>>>>>>>>> >>>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>>> copying the >>>>>>>>>>> closed source >>>>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>>>> reviewing, instead of >>>>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>>>> content >>>>>>>>>>> of the new >>>>>>>>>>> version of each file. >>>>>>>>>>> >>>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>>> flag is >>>>>>>>>>> changed from >>>>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>>>> "commercial" >>>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>>> >>>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>>> open/closed >>>>>>>>>>> source >>>>>>>>>>> code structure had introduced some intermediary APIs to connect >>>>>>>>>>> code >>>>>>>>>>> between >>>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>>> >>>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>>>> tests >>>>>>>>>>> will be >>>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>>> >>>>>>>>>>> All the AppCDS tests (currently still in closed sources) passed >>>>>>>>>>> with >>>>>>>>>>> both Oracle JDK >>>>>>>>>>> and OpenJDK. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> - Ioi >>>> >>>> >> From mandy.chung at oracle.com Fri Nov 24 17:41:34 2017 From: mandy.chung at oracle.com (mandy chung) Date: Fri, 24 Nov 2017 09:41:34 -0800 Subject: [10] RFR: 8191813: compiler/runtime/SpreadNullArg.java fails in tier1 In-Reply-To: References: Message-ID: <792dadfc-89d3-054c-d3f2-303a69346be6@oracle.com> Looks good.? Thanks for fixing the test, Rahul. Mandy On 11/24/17 3:16 AM, Rahul Raghavan wrote: > Hi, > > Please review the following fix proposal for 8191813. > > webrev - http://cr.openjdk.java.net/~rraghavan/8191813/webrev.00/ > > > jbs - https://bugs.openjdk.java.net/browse/JDK-8191813 > > -- reported issue - After the latest sync from jdk/jdk the test > SpreadNullArg started to fail in tier1 on all platforms. > > -- Found this SpreadNullArg.java failure Caused after JDK-8157246 fix. > jbs - https://bugs.openjdk.java.net/browse/JDK-8157246. > 'MHs.arrayLength, arrayElementGetter/Setter, arrayConstructor need to > specify invocation-time behavior'. > changeset - http://hg.openjdk.java.net/jdk/jdk/rev/76519338df34 > > - So now with 8157246 fix, if method handle is invoked with a null > array reference, NullPointerException will be thrown. (Not > IllegalArgumentException) > > - So above fix proposal in SpreadNullArg.java to expect > NullPointerException after 8157246 changes. > > > Thanks, > Rahul From aph at redhat.com Fri Nov 24 17:44:22 2017 From: aph at redhat.com (Andrew Haley) Date: Fri, 24 Nov 2017 17:44:22 +0000 Subject: RFR (S): 8191681: Unsafe accesses should use field_offset_to_byte_offset() In-Reply-To: <5A183090.5000705@oracle.com> References: <5A183090.5000705@oracle.com> Message-ID: On 24/11/17 14:45, Erik ?sterlund wrote: > > Before the Access API, there was a no-op conversion function in > unsafe.cpp for getting the byte offset of a field offset from unsafe as > a jlong. Now, the field offsets *are* byte offsets, so the conversion > did not do anything. But I guess it could be useful nevertheless for > abstraction purposes. This came up when I was working on Unsafe, and I think it's really a bit of nonsense. These days the byte offsets might even be absolute addresses, and the speculation that one day we might change our minds is rather pointless. I could find the discussion from that thread if you're interested. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From daniel.daugherty at oracle.com Fri Nov 24 18:13:20 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 24 Nov 2017 13:13:20 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> Message-ID: Greetings, The Thread-SMR bits were pushed to jdk/hs late last night! Mach5 Tier[1-5] on the exact bits that were pushed showed no unexpected failures. I've looked at the jdk/hs CI pipeline test results for my push and for the push before and after mine and I don't see anything that worries me there either. Obviously I'll be keeping an eye on testing for the next several days, but so far everything looks good!! Dan On 11/22/17 9:16 PM, Daniel D. Daugherty wrote: > Greetings, > > I've made minor tweaks to the Thread-SMR project based on the remaining > code review comments over the last couple of days. I've also rebased the > project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm > running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] > testing... > > Here's a delta webrev for the latest round of tweaks: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta > > > Here's the proposed commit message: > > $ cat SMR_prototype_10/open/commit.txt > 8167108: inconsistent handling of SR_lock can lead to crashes > Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. > Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, kbarrett, > rehn, sspitsyn, stefank > Contributed-by: daniel.daugherty at oracle.com, > erik.osterlund at oracle.com, robbin.ehn at oracle.com > > I've gone through 880+ emails in my archive for this project and added > anyone that made a code review comment. Reviewers are not in my usual > by-comment-posting order; it was way too hard to figure that out... So > reviewers and contributors are simply in alpha-sort order... > > Here's the collection of follow-up bugs that I filed based on sifting > through the email archive: > > 8191784 JavaThread::collect_counters() should stop using Threads_lock > https://bugs.openjdk.java.net/browse/JDK-8191784 > > 8191785 revoke_bias() needs a new assertion > https://bugs.openjdk.java.net/browse/JDK-8191785 > > 8191786 Thread-SMR hash table size should be dynamic > https://bugs.openjdk.java.net/browse/JDK-8191786 > > 8191787 move private inline functions from thread.inline.hpp -> > thread.cpp > https://bugs.openjdk.java.net/browse/JDK-8191787 > > 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> > threadSMR.[ch]pp > https://bugs.openjdk.java.net/browse/JDK-8191789 > > 8191798 redo nested ThreadsListHandle to drop Threads_lock > https://bugs.openjdk.java.net/browse/JDK-8191789 > > I have undoubtedly missed at least one future idea that someone had > and for that my apologies... > > Thanks again to everyone who contributed to the Thread-SMR project!! > > Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for > their rigorous review of the design that we documented on the wiki. > Thanks for keeping us honest! > > I also have to thank my co-contributor Erik ?sterlund for starting the > ball rolling on this crazy project with his presentation on Safe Memory > Reclamation, for the initial prototype and for all your patches. Erik, > hopefully we got far enough in your crazy vision... :-) > > Thanks to my co-contributor Robbin Ehn for proposing that we do early > code reviews in the form of patches. Don't like something? Fix it with > a patch and a new test if appropriate!! A pretty cool way to work that > was new to me. > > Finally, many thanks to Jerry Thornbrugh for all the early code reviews, > whiteboard chats and phone calls. Thanks for asking the right questions > and pointing out some places where our logic was faulty. Also thanks for > keeping me sane when I was literally on my own in Monument, CO. > > Dan > > > On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> *** We are wrapping up code review on this project so it is time *** >> *** for the various code reviewers to chime in one last time. *** >> >> In this latest round, we had three different reviewers chime in so we're >> doing delta webrevs for each of those resolutions: >> >> David H's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >> >> >> ? mq comment for dholmes_CR: >> ??? - Fix indents, trailing spaces and various typos. >> ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, >> ????? add impl notes to document the type choices. >> >> ? src/hotspot/share/runtime/globals.hpp change is white-space only so it >> ? is only visible via the file's patch link. >> >> >> Robin W's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >> >> >> ? mq comment for robinw_CR: >> ??? - Fix some inefficient code, update some comments, fix some indents, >> ????? and add some 'const' specifiers. >> >> ? src/hotspot/share/runtime/vm_operations.hpp change is white-space >> only so >> ? it is only visible via the file's patch link. >> >> >> Coleen's resolutions: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >> >> >> ? mq comment for coleenp_CR: >> ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >> ??? - add header comment to threadSMR.hpp file, >> ??? - cleanup JavaThreadIteratorWithHandle ctr, >> ??? - make ErrorHandling more efficient. >> >> >> Some folks prefer to see all of the delta webrevs together so here is >> a delta webrev relative to jdk10-09-full: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >> >> >> ? src/hotspot/share/runtime/globals.hpp and >> ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space >> only >> ? so they are only visible via the file's patch link. >> >> >> And, of course, some folks prefer to see everything all at once so here >> is the latest full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >> >> >> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >> >> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >> so there will be some catching up to do before integration... >> >> We welcome comments, suggestions and feedback. >> >> Dan, Erik and Robbin >> >> >> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Testing of the last round of changes revealed a hang in one of the new >>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, >>> and >>> added another TLH test for good measure. >>> >>> Here is the updated full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>> >>> Here is the updated delta webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>> >>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>> no unexpected failures. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Robbin rebased the project last night/this morning to merge with >>>> Thread >>>> Local Handshakes (TLH) and also picked up additional changesets up >>>> thru: >>>> >>>>> Changeset: fa736014cf28 >>>>> Author:??? cjplummer >>>>> Date:????? 2017-11-14 18:08 -0800 >>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>> >>>>> 8191049: Add alternate version of pns() that is callable from >>>>> within hotspot source >>>>> Summary: added pns2() to debug.cpp >>>>> Reviewed-by: stuefe, gthornbr >>>> >>>> This is the first time we've rebased the project to bits that are this >>>> fresh (< 12 hours old at rebase time). We've done this because we >>>> think >>>> we're done with this project and are in the final review-change-retest >>>> cycle(s)... In other words, we're not planning on making any more >>>> major >>>> changes for this project... :-) >>>> >>>> *** Time for code reviewers to chime in on this thread! *** >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>> >>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>> (and the new baseline also)... >>>> >>>> We're expecting this round to be a little noisier than usual because >>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>> (Yes, we're playing chase-the-repo...) >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>> >>>>> Unlike the previous rebase, there were no changes required to the >>>>> open code to get the rebased bits to build so there is no delta >>>>> webrev. >>>>> >>>>> These bits have been run through JPRT and I've submitted the usual >>>>> Mach5 tier[1-5] test run... >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>> >>>>>> Here are the updated webrevs: >>>>>> >>>>>> Here's the mq comment for the change: >>>>>> >>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>> >>>>>> Here is the full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>> >>>>>> And here is the delta webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>> >>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>> holiday >>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>> closer >>>>>> look today. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>> and Coleen) >>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>> done as a >>>>>>> standalone patch and corresponding webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>> to use >>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>> >>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>> >>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>> >>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>> describe >>>>>>>> and track the work on this project: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>> >>>>>>>> >>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>> quotes >>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>> have a >>>>>>>> solution for that problem yet. >>>>>>>> >>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>> >>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>> tier[2-5] >>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, >>>>>>>> and >>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Daniel Daugherty >>>>>>>> Erik Osterlund >>>>>>>> Robbin Ehn >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > From rahul.v.raghavan at oracle.com Sun Nov 26 17:09:36 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Sun, 26 Nov 2017 22:39:36 +0530 Subject: [10] RFR: 8191813: compiler/runtime/SpreadNullArg.java fails in tier1 In-Reply-To: <752a4e80-9141-6115-dee6-2bbd0ec99e92@oracle.com> References: <752a4e80-9141-6115-dee6-2bbd0ec99e92@oracle.com> Message-ID: <9e8de2ad-8ea2-3e66-7ea1-bd5d90acf936@oracle.com> Thank you Dan for the review. Rahul On Friday 24 November 2017 08:56 PM, Daniel D. Daugherty wrote: > On 11/24/17 6:16 AM, Rahul Raghavan wrote: >> Hi, >> >> Please review the following fix proposal for 8191813. >> >> webrev - http://cr.openjdk.java.net/~rraghavan/8191813/webrev.00/ > > test/hotspot/jtreg/compiler/runtime/SpreadNullArg.java > ??? No comments. > > So this style of call: > > ??? mh_spreadInvoker.invokeExact(mh_spread_target, (Object[]) null); > > was changed from throwing IllegalArgumentException due to the "null" > parameter to throwing NullPointerException. You've updated the test > cleanly to switch from one exception to the other. > > Thumbs up! I believe this patch qualifies as trivial and does not > need to wait for 24 hours. I leave it up to you whether you wait > for a Compiler team member review also. > > Dan > > >> >> >> jbs - https://bugs.openjdk.java.net/browse/JDK-8191813 >> >> -- reported issue - After the latest sync from jdk/jdk the test >> SpreadNullArg started to fail in tier1 on all platforms. >> >> -- Found this SpreadNullArg.java failure Caused after JDK-8157246 fix. >> jbs - https://bugs.openjdk.java.net/browse/JDK-8157246. >> 'MHs.arrayLength, arrayElementGetter/Setter, arrayConstructor need to >> specify invocation-time behavior'. >> changeset - http://hg.openjdk.java.net/jdk/jdk/rev/76519338df34 >> >> - So now with 8157246 fix, if method handle is invoked with a null >> array reference, NullPointerException will be thrown. (Not >> IllegalArgumentException) >> >> - So above fix proposal in SpreadNullArg.java to expect >> NullPointerException after 8157246 changes. >> >> >> Thanks, >> Rahul > From rahul.v.raghavan at oracle.com Sun Nov 26 17:10:21 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Sun, 26 Nov 2017 22:40:21 +0530 Subject: [10] RFR: 8191813: compiler/runtime/SpreadNullArg.java fails in tier1 In-Reply-To: <792dadfc-89d3-054c-d3f2-303a69346be6@oracle.com> References: <792dadfc-89d3-054c-d3f2-303a69346be6@oracle.com> Message-ID: <011150aa-9a29-2786-522f-c755469f0335@oracle.com> Thank you Mandy for review. On Friday 24 November 2017 11:11 PM, mandy chung wrote: > Looks good.? Thanks for fixing the test, Rahul. > > Mandy > > On 11/24/17 3:16 AM, Rahul Raghavan wrote: >> Hi, >> >> Please review the following fix proposal for 8191813. >> >> webrev - http://cr.openjdk.java.net/~rraghavan/8191813/webrev.00/ >> >> >> jbs - https://bugs.openjdk.java.net/browse/JDK-8191813 >> >> -- reported issue - After the latest sync from jdk/jdk the test >> SpreadNullArg started to fail in tier1 on all platforms. >> >> -- Found this SpreadNullArg.java failure Caused after JDK-8157246 fix. >> jbs - https://bugs.openjdk.java.net/browse/JDK-8157246. >> 'MHs.arrayLength, arrayElementGetter/Setter, arrayConstructor need to >> specify invocation-time behavior'. >> changeset - http://hg.openjdk.java.net/jdk/jdk/rev/76519338df34 >> >> - So now with 8157246 fix, if method handle is invoked with a null >> array reference, NullPointerException will be thrown. (Not >> IllegalArgumentException) >> >> - So above fix proposal in SpreadNullArg.java to expect >> NullPointerException after 8157246 changes. >> >> >> Thanks, >> Rahul > From erik.osterlund at oracle.com Mon Nov 27 08:21:59 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 27 Nov 2017 09:21:59 +0100 Subject: RFR (S): 8191681: Unsafe accesses should use field_offset_to_byte_offset() In-Reply-To: References: <5A183090.5000705@oracle.com> Message-ID: <5A1BCB27.1000300@oracle.com> Hi Andrew, On 2017-11-24 18:44, Andrew Haley wrote: > On 24/11/17 14:45, Erik ?sterlund wrote: >> Before the Access API, there was a no-op conversion function in >> unsafe.cpp for getting the byte offset of a field offset from unsafe as >> a jlong. Now, the field offsets *are* byte offsets, so the conversion >> did not do anything. But I guess it could be useful nevertheless for >> abstraction purposes. > This came up when I was working on Unsafe, and I think it's really a > bit of nonsense. These days the byte offsets might even be absolute > addresses, and the speculation that one day we might change our minds > is rather pointless. I could find the discussion from that thread if > you're interested. > Thank you for your feedback. I would not mind dropping these changes in favour of simply assuming that Unsafe works with byte offsets only. My Access API changes introduced that assumption in this file by dodging the no-op conversion. Then I was asked to remove the assumption by re-introducing code performing the no-op conversion of field offsets to byte offsets. Personally I do not have a preference whether to perform or not perform this no-op conversion. Thanks, /Erik From erik.osterlund at oracle.com Mon Nov 27 09:06:59 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 27 Nov 2017 10:06:59 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API Message-ID: <5A1BD5B3.7030808@oracle.com> Hi, The ClassLoaderData::remove_handle() member function currently uses explicit G1 SATB barriers to remove an oop from the root set, as these handles are not necessarily walked by the GC in a safepoint. Therefore G1 needs pre-write barriers. This should now be modeled as a RootAccess::oop_store instead. This maps to performing a pre-write SATB barrier with G1, but other GCs are free to do other things as necessary. Webrev: http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8191888 Thanks, /Erik From rahul.v.raghavan at oracle.com Mon Nov 27 09:54:27 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Mon, 27 Nov 2017 15:24:27 +0530 Subject: [10] RFR: 8191227: issues with unsafe handle resolution In-Reply-To: References: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Message-ID: Hi, On Thursday 23 November 2017 05:19 PM, Vladimir Ivanov wrote: > >> webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ > > Please, use ThreadInVMfromUnknown in ConstantOopWriteValue::print_on() > as well. Done. http://cr.openjdk.java.net/~rraghavan/8191227/webrev.03/ Thank you Vladimir for review. > > Otherwise, looks good. > > Best regards, > Vladimir Ivanov > Thanks, Rahul From tobias.hartmann at oracle.com Mon Nov 27 09:59:49 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Mon, 27 Nov 2017 10:59:49 +0100 Subject: [10] RFR: 8191227: issues with unsafe handle resolution In-Reply-To: References: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Message-ID: Hi Rahul, On 27.11.2017 10:54, Rahul Raghavan wrote: >>> webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ Looks good to me! Best regards, Tobias From thomas.stuefe at gmail.com Mon Nov 27 10:33:46 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 27 Nov 2017 11:33:46 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: Ping... Could I please have reviews? Thank you! ..Thomas On Wed, Nov 22, 2017 at 7:01 PM, Thomas St?fe wrote: > Hi all, > > may I please have reviews for the following enhancement: > > issue: https://bugs.openjdk.java.net/browse/JDK-8191101 > webrev: http://cr.openjdk.java.net/~stuefe/webrevs/ > 8191101-show-registers-on-assert/webrev.00/webrev/ > > (Patch looks big but lot of it is os_cpu fluff.) > > Prior email discussion at: http://mail.openjdk.java.net/ > pipermail/hotspot-runtime-dev/2017-October/025018.html > > ---- > > Basically, this adds the ability to show register values in assert > situations into the error report. This can be useful in certain corner > cases, e.g. if you want to know the value of some local variables or how > deep your stack currently runs. > > This works by triggering a fault right when an assert happens and > squirreling the context away for later error reporting. > > When an assert happens, we touch a poison page. To preserve the context as > best as possible, we want to avoid running too much code after the assert > condition has been evaluated, so this is done in a very simple way: > directly in the assert macro, right after the condition is evaluated, we > dereference the content of a global poison page pointer. In my tests, even > with slowdebug, this only spoils one register, rax on x64. > > In the signal handler, we recognize the assertion poison fault by the > faulting address. We disable the poison page and store the ucontext away. > Then we just return from signal handling. Poison page is now disarmed, the > load from it is retried and now goes through. Normal assertion handling is > then resumed - so, things like -XX:SuppressAt are unaffected and work fine. > > Then, when an error report is generated due to this assert, we now also > use the stored context. So now we get registers and instructions at the > assert point. > > Right now, this is implemented on (non-zero) Linux, though other posix > platforms should be no problem. Have not yet thought deeply about windows. > It is tested and - in debug - switched on by default on linux x86, ppc, > s390. > > If implemented, it can be switched on and off with > -XX:+ShowRegistersOnAssert. This is a failsafe, in case the mechanism does > not work and we want to have clean asserts. > > To test this, do a java -XX:ErrorHandlerTest=1 -XX:+ShowRegistersOnAssert > with a not-product VM. On Linux x64, ppc and s390 we should now see the > register output in the hs-err file: > > 4 # Internal Error (/shared/projects/openjdk/jdk- > hs/source/src/hotspot/share/utilities/vmError.cpp:1660), pid=4022, > tid=4023 > 5 # assert(str == NULL) failed: expected null > 6 # > 7 # > ... > 59 Registers: > 60 RAX=0x00007f736c8f7000, RBX=0x0000000000000000, > RCX=0x0000000000000000, RDX=0xffffffffffb5ded4 > 61 RSP=0x00007f736c8d8ce0, RBP=0x00007f736c8d8d30, > RSI=0x0000000000000001, RDI=0x0000000000000001 > 62 R8 =0x0000000000000040, R9 =0x0000000000000001, > R10=0x0000000000efb028, R11=0x00040788a244f42a > 63 R12=0x0000000000000000, R13=0x00007fff3de46dbf, > R14=0x00007f736c8d99c0, R15=0x0000000000000000 > 64 RIP=0x00007f736aec5529, EFLAGS=0x0000000000010202, > CSGSFS=0x002b000000000033, ERR=0x0000000000000006 > 65 TRAPNO=0x000000000000000e > ... > 72 > 73 Instructions: (pc=0x00007f736aec5529) > 74 0x00007f736aec5509: 8d 05 31 21 4a 00 48 01 d0 ff e0 48 83 7d c8 00 > 75 0x00007f736aec5519: 0f 84 7f 03 00 00 48 8d 05 82 fc b1 00 48 8b 00 > 76 0x00007f736aec5529: c6 00 58 e8 cb 17 62 ff 84 c0 74 11 48 8d 3d 2f > 77 0x00007f736aec5539: 1f 4a 00 b8 00 00 00 00 e8 ed 17 62 ff 48 8d 0d > 78 > > --- > > Implementation notes: > > - when handling the poison fault, we need to copy the context away from > the signal handler stack. For posix, this means copying the ucontext_t. > This is undefined territory. On most platforms, this simply means copying > the ucontex_t as a flat structure. On some platforms more is needed, e.g. > on linux ppc, we need to patch up the context after copying (the context is > not position independent), and on MacOS, the context is not self-contained > but contains pointers to sub structures which need to be copied too and > whose size is unknown at compile time. Because of these platform > dependencies, I factored out the copying of ucontext_t to > os::Posix::copy_ucontext and its implementations are os_cpu specific. > > - As an added precaution, when copying the context, we use a safe version > of memcpy (os::safe_memcpy) which I added to copy from potentially invalid > memory regions. The reason is that we have seen on some Unices - e.g. hpux > - that the size of the ucontext_t structure at runtime may be different > from the build machine, so we tread carefully. os::safe_memcpy() uses > SafeFetch to copy a range of memory. > > Risk: > > If this does not work, asserts will become segfaults, which can be > confusing. But the feature can be disabled with -XX:+ShowRegistersOnAssert > and for now on most platforms is disabled by default. > > --- > > Limitations: > - as it is now implemented, this is a one-shot mechanism and only works > for the first assert. > - -XX:SuppressAt=... is not affected and works fine. However, if the first > assert is suppressed, follow-up asserts will not show register values. > - When multiple threads run into an assert, we may or may not see register > values depending on which thread is the first of finishing the poison page > handler. > > I do not think these limitations are severe. They can be solved, but at > the cost of added complexity, which I preferred not to add. > > Thank you, > > Thomas > > > > From rahul.v.raghavan at oracle.com Mon Nov 27 10:45:19 2017 From: rahul.v.raghavan at oracle.com (Rahul Raghavan) Date: Mon, 27 Nov 2017 16:15:19 +0530 Subject: [10] RFR: 8191227: issues with unsafe handle resolution In-Reply-To: References: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Message-ID: Thank you Tobias. On Monday 27 November 2017 03:29 PM, Tobias Hartmann wrote: > Hi Rahul, > > On 27.11.2017 10:54, Rahul Raghavan wrote: >>>> webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ > > Looks good to me! > > Best regards, > Tobias > From erik.osterlund at oracle.com Mon Nov 27 11:28:48 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 27 Nov 2017 12:28:48 +0100 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API Message-ID: <5A1BF6F0.5030903@oracle.com> Hi, The JVMTI tag hashmap has weak oop references that are handled using raw oop accesses and a G1-specific SATB enqueue call when leaking out objects from the tag map, requiring them to be marked as live by G1. This should now be refactored to use the Access API to annotate that these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. Webrev: http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8191894 Thanks, /Erik From vladimir.x.ivanov at oracle.com Mon Nov 27 12:29:56 2017 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Mon, 27 Nov 2017 15:29:56 +0300 Subject: [10] RFR: 8191227: issues with unsafe handle resolution In-Reply-To: References: <2bebdde2-b579-3049-4610-4e3949c88d01@oracle.com> Message-ID: <50761764-2293-788a-142b-c479eebce77b@oracle.com> Looks good. Best regards, Vladimir Ivanov On 11/27/17 12:54 PM, Rahul Raghavan wrote: > Hi, > > On Thursday 23 November 2017 05:19 PM, Vladimir Ivanov wrote: >> >>> webrev - http://cr.openjdk.java.net/~rraghavan/8191227/webrev.02/ >> >> Please, use ThreadInVMfromUnknown in ConstantOopWriteValue::print_on() >> as well. > > Done. > http://cr.openjdk.java.net/~rraghavan/8191227/webrev.03/ > Thank you Vladimir for review. > >> >> Otherwise, looks good. >> >> Best regards, >> Vladimir Ivanov >> > > Thanks, > Rahul From erik.osterlund at oracle.com Mon Nov 27 12:36:40 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 27 Nov 2017 13:36:40 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte Message-ID: <5A1C06D8.1040405@oracle.com> Hi, There is currently a bug when using unsafe accesses off-heap: 1) We write into a thread that we enable crash protection (using GuardUnsafeAccess): 2) We perform the access 3) We write into a thread that we disable crash protection (using ~GuardUnsafeAccess) The problem is that the crash protection stores are volatile, but the actual access is non-volatile. Compilers have different interpretation whether volatile - non-volatile accesses are allowed to reorder. MSVC is known to interpret such interactions as-if the volatile accesses have acquire/release semantics from the compiler point of view, and others such as GCC are known to reorder away freely. To prevent any issues, the accesses involved when using GuardUnsafeAccess should be at least volatile. This change makes the few remaining ones volatile. The JMM-volatile (SEQ_CST) accesses with crash protection already have stronger ordering than volatile and hence do not need changing. By making the address passed in to the Access API volatile, the MO_VOLATILE decorator is automatically set, which not surprisingly makes the access volatile. Therefore, the solution is to simply make the address passed in to Access volatile in this case. Bug: https://bugs.openjdk.java.net/browse/JDK-8186787 Webrev: http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ Thanks, /Erik From adinn at redhat.com Mon Nov 27 12:43:53 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 27 Nov 2017 12:43:53 +0000 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: <38111282-dff9-576e-9b62-21c34c238cff@redhat.com> On 27/11/17 10:33, Thomas St?fe wrote: > Ping... Could I please have reviews? Thank you! > ..Thomas This looks ok except that I'm a bit unsure about the changes to VMError.cpp, specifically 1) the message "Life goes on..." printed at the bottom of the loop 2) the fact that this replaces the call to ShouldNotReachHere(); I understand that the message is there to give a positive for the test. Also that it only gets printed if you suppress error reporting in this code. However, to someone who doesn't know that the idea that one might continue through to this point is rather misleading and, perhaps, confusing. Could this be commented out? Or could the positive confirmation just be omitted? Anyway, I built and tested on AArch64 and it passed the test (for what that indicates about register printing -- as you say a more stringent test is needed that allows for different behaviour on different platforms.) regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From thomas.stuefe at gmail.com Mon Nov 27 13:14:13 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 27 Nov 2017 14:14:13 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: <38111282-dff9-576e-9b62-21c34c238cff@redhat.com> References: <38111282-dff9-576e-9b62-21c34c238cff@redhat.com> Message-ID: Hi Andrew, thanks a lot for the quick review! Pls find comments inline. On Mon, Nov 27, 2017 at 1:43 PM, Andrew Dinn wrote: > On 27/11/17 10:33, Thomas St?fe wrote: > > Ping... Could I please have reviews? Thank you! > > ..Thomas > > This looks ok except that I'm a bit unsure about the changes to > VMError.cpp, specifically > > 1) the message "Life goes on..." printed at the bottom of the loop > 2) the fact that this replaces the call to ShouldNotReachHere(); > > I understand that the message is there to give a positive for the test. > Also that it only gets printed if you suppress error reporting in this > code. However, to someone who doesn't know that the idea that one might > continue through to this point is rather misleading and, perhaps, > confusing. Could this be commented out? Or could the positive > confirmation just be omitted? > > Okay, I see your point. I will revert back to ShouldNotReachHere() and change the jtreg test to recognize that as a positive test. > Anyway, I built and tested on AArch64 and it passed the test (for what > that indicates about register printing -- as you say a more stringent > test is needed that allows for different behaviour on different platforms.) > > Could you please quickly run java -XX:+ShowRegistersOnAssert -XX:ErrorHandlerTest=1 on aarch64 and take a quick glance at the hs-err file, whether it contains register output? regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > Thanks again, Best Regards, Thomas From daniel.daugherty at oracle.com Mon Nov 27 13:22:07 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 27 Nov 2017 08:22:07 -0500 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <5A1BF6F0.5030903@oracle.com> References: <5A1BF6F0.5030903@oracle.com> Message-ID: <88659101-1693-d5a4-a605-8f7b5ab53299@oracle.com> Adding serviceability-dev at ... since this is JVM/TI related... Dan On 11/27/17 6:28 AM, Erik ?sterlund wrote: > Hi, > > The JVMTI tag hashmap has weak oop references that are handled using > raw oop accesses and a G1-specific SATB enqueue call when leaking out > objects from the tag map, requiring them to be marked as live by G1. > > This should now be refactored to use the Access API to annotate that > these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use > ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191894 > > Thanks, > /Erik From adinn at redhat.com Mon Nov 27 16:18:34 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 27 Nov 2017 16:18:34 +0000 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: <38111282-dff9-576e-9b62-21c34c238cff@redhat.com> Message-ID: Hi Thomas, On 27/11/17 13:14, Thomas St?fe wrote: > thanks a lot for the quick review! Thanks for the fix :-) > Okay, I see your point. I will revert back to ShouldNotReachHere() and > change the jtreg test to recognize that as a positive test. Ok, thanks. > Could you please quickly run? > > java -XX:+ShowRegistersOnAssert -XX:ErrorHandlerTest=1 > ? > on aarch64 and take a quick glance at the hs-err file, whether it > contains register output? Oh my! The relevant (beautiful) output is appended below. Very nice. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/home/adinn/openjdk/jdkdev/hs/src/hotspot/share/utilities/vmError.cpp:1660), pid=29190, tid=29191 . . . Registers: R0 =0x000003ff83340000 is an unknown value R1 =0x0000000000000058 is an unknown value R2 =0x000003ff826da55c: in /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal-server-slowdebug/jdk/lib/server/libjvm.so at 0x000003ff81510000 R3 =0x000003ff826da55c: in /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal-server-slowdebug/jdk/lib/server/libjvm.so at 0x000003ff81510000 R4 =0x000003ff812fe748 is pointing into the stack for thread: 0x000003ff7c018000 R5 =0x000003ff7c016a28 is an unknown value R6 =0x000003ff83360000 is an unknown value R7 =0x0000000000000000 is an unknown value R8 =0x00000000000c4c96 is an unknown value R9 =0x00000000000c4c96 is an unknown value R10=0x0000000000004438 is an unknown value R11=0x000000000a0751f8 is an unknown value R12=0x0000000000000018 is an unknown value R13=0xffffffffa5e40b06 is an unknown value R14=0x001b901602000000 is an unknown value R15=0x003b9aca00000000 is an unknown value R16=0x000003ff82fc6bc0: in /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal-server-slowdebug/jdk/lib/server/libjvm.so at 0x000003ff81510000 R17=0x000003ff831548ac: memset+0x0000000000000000 in /lib64/libc.so.6 at 0x000003ff830d0000 R18=0x00000006cf9f754a is pointing into object: 0x00000006cf9f7538 [B {0x00000006cf9f7538} - klass: {type array byte} - length: 21 - 0: 73 s - 1: 75 u - 2: 6e n - 3: 2e . - 4: 62 b - 5: 6f o - 6: 6f o - 7: 74 t - 8: 2e . - 9: 6c l - 10: 69 i - 11: 62 b - 12: 72 r - 13: 61 a - 14: 72 r - 15: 79 y - 16: 2e . - 17: 70 p - 18: 61 a - 19: 74 t - 20: 68 h R19=0x0000000000000000 is an unknown value R20=0x000003ffe7d21468 is an unknown value R21=0x000003ff832f0000: in /lib64/libpthread.so.0 at 0x000003ff832c0000 R22=0x000003ff812feb10 is pointing into the stack for thread: 0x000003ff7c018000 R23=0x000003ffe7d21468 is an unknown value R24=0x0000000000000000 is an unknown value R25=0x000003ffe7d21468 is an unknown value R26=0x000003ff832f4000: in /lib64/libpthread.so.0 at 0x000003ff832c0000 R27=0x000003ff812ff2c0 is pointing into the stack for thread: 0x000003ff7c018000 R28=0x000003ff812ff8f0 is pointing into the stack for thread: 0x000003ff7c018000 R29=0x000003ff812fe750 is pointing into the stack for thread: 0x000003ff7c018000 R30=0x000003ff826da518: in /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal-server-slowdebug/jdk/lib/server/libjvm.so at 0x000003ff81510000 Top of Stack: (sp=0x000003ff812fe690) . . . From thomas.stuefe at gmail.com Mon Nov 27 16:48:36 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 27 Nov 2017 17:48:36 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: <38111282-dff9-576e-9b62-21c34c238cff@redhat.com> Message-ID: On Mon, Nov 27, 2017 at 5:18 PM, Andrew Dinn wrote: > Hi Thomas, > > On 27/11/17 13:14, Thomas St?fe wrote: > > thanks a lot for the quick review! > > Thanks for the fix :-) > > > Okay, I see your point. I will revert back to ShouldNotReachHere() and > > change the jtreg test to recognize that as a positive test. > > Ok, thanks. > > > Could you please quickly run > > > > java -XX:+ShowRegistersOnAssert -XX:ErrorHandlerTest=1 > > > > on aarch64 and take a quick glance at the hs-err file, whether it > > contains register output? > Oh my! The relevant (beautiful) output is appended below. Very nice. > > :) Cool! Thanks for testing! I'll post an update with the corrected vmError.cpp soon ..Thomas > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # Internal Error > (/home/adinn/openjdk/jdkdev/hs/src/hotspot/share/ > utilities/vmError.cpp:1660), > pid=29190, tid=29191 > > . . . > > Registers: > R0 =0x000003ff83340000 is an unknown value > R1 =0x0000000000000058 is an unknown value > R2 =0x000003ff826da55c: in > /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal- > server-slowdebug/jdk/lib/server/libjvm.so > at 0x000003ff81510000 > R3 =0x000003ff826da55c: in > /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal- > server-slowdebug/jdk/lib/server/libjvm.so > at 0x000003ff81510000 > R4 =0x000003ff812fe748 is pointing into the stack for thread: > 0x000003ff7c018000 > R5 =0x000003ff7c016a28 is an unknown value > R6 =0x000003ff83360000 is an unknown value > R7 =0x0000000000000000 is an unknown value > R8 =0x00000000000c4c96 is an unknown value > R9 =0x00000000000c4c96 is an unknown value > R10=0x0000000000004438 is an unknown value > R11=0x000000000a0751f8 is an unknown value > R12=0x0000000000000018 is an unknown value > R13=0xffffffffa5e40b06 is an unknown value > R14=0x001b901602000000 is an unknown value > R15=0x003b9aca00000000 is an unknown value > R16=0x000003ff82fc6bc0: in > /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal- > server-slowdebug/jdk/lib/server/libjvm.so > at 0x000003ff81510000 > R17=0x000003ff831548ac: memset+0x0000000000000000 in /lib64/libc.so.6 at > 0x000003ff830d0000 > R18=0x00000006cf9f754a is pointing into object: 0x00000006cf9f7538 > [B > {0x00000006cf9f7538} - klass: {type array byte} > - length: 21 > - 0: 73 s > - 1: 75 u > - 2: 6e n > - 3: 2e . > - 4: 62 b > - 5: 6f o > - 6: 6f o > - 7: 74 t > - 8: 2e . > - 9: 6c l > - 10: 69 i > - 11: 62 b > - 12: 72 r > - 13: 61 a > - 14: 72 r > - 15: 79 y > - 16: 2e . > - 17: 70 p > - 18: 61 a > - 19: 74 t > - 20: 68 h > R19=0x0000000000000000 is an unknown value > R20=0x000003ffe7d21468 is an unknown value > R21=0x000003ff832f0000: in > /lib64/libpthread.so.0 at 0x000003ff832c0000 > R22=0x000003ff812feb10 is pointing into the stack for thread: > 0x000003ff7c018000 > R23=0x000003ffe7d21468 is an unknown value > R24=0x0000000000000000 is an unknown value > R25=0x000003ffe7d21468 is an unknown value > R26=0x000003ff832f4000: in > /lib64/libpthread.so.0 at 0x000003ff832c0000 > R27=0x000003ff812ff2c0 is pointing into the stack for thread: > 0x000003ff7c018000 > R28=0x000003ff812ff8f0 is pointing into the stack for thread: > 0x000003ff7c018000 > R29=0x000003ff812fe750 is pointing into the stack for thread: > 0x000003ff7c018000 > R30=0x000003ff826da518: in > /home/adinn/openjdk/jdkdev/hs/build/linux-aarch64-normal- > server-slowdebug/jdk/lib/server/libjvm.so > at 0x000003ff81510000 > > Top of Stack: (sp=0x000003ff812fe690) > . . . > From coleen.phillimore at oracle.com Mon Nov 27 18:08:47 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 27 Nov 2017 13:08:47 -0500 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <5A1BD5B3.7030808@oracle.com> References: <5A1BD5B3.7030808@oracle.com> Message-ID: <4b890e2e-8cf7-2d2d-5373-d9144245b0ce@oracle.com> This looks good! Coleen On 11/27/17 4:06 AM, Erik ?sterlund wrote: > Hi, > > The ClassLoaderData::remove_handle() member function currently uses > explicit G1 SATB barriers to remove an oop from the root set, as these > handles are not necessarily walked by the GC in a safepoint. Therefore > G1 needs pre-write barriers. > > This should now be modeled as a > RootAccess::oop_store instead. This maps to > performing a pre-write SATB barrier with G1, but other GCs are free to > do other things as necessary. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191888 > > Thanks, > /Erik From coleen.phillimore at oracle.com Mon Nov 27 18:21:34 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 27 Nov 2017 13:21:34 -0500 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <5A1BF6F0.5030903@oracle.com> References: <5A1BF6F0.5030903@oracle.com> Message-ID: http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/src/hotspot/share/prims/jvmtiTagMap.cpp.udiff.html + return RootAccess::oop_load(object_addr()); Why is this not access ON_ROOT{_CONCURRENT} ?? The thing holding the object that you are peeking at is not in the Java Heap? thanks, Coleen On 11/27/17 6:28 AM, Erik ?sterlund wrote: > Hi, > > The JVMTI tag hashmap has weak oop references that are handled using > raw oop accesses and a G1-specific SATB enqueue call when leaking out > objects from the tag map, requiring them to be marked as live by G1. > > This should now be refactored to use the Access API to annotate that > these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use > ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191894 > > Thanks, > /Erik From coleen.phillimore at oracle.com Mon Nov 27 18:27:07 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 27 Nov 2017 13:27:07 -0500 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <5A1C06D8.1040405@oracle.com> References: <5A1C06D8.1040405@oracle.com> Message-ID: <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> I think this is a nice simple fix and should be pushed for JDK10. Thanks, Coleen On 11/27/17 7:36 AM, Erik ?sterlund wrote: > Hi, > > There is currently a bug when using unsafe accesses off-heap: > > 1) We write into a thread that we enable crash protection (using > GuardUnsafeAccess): > 2) We perform the access > 3) We write into a thread that we disable crash protection (using > ~GuardUnsafeAccess) > > The problem is that the crash protection stores are volatile, but the > actual access is non-volatile. Compilers have different interpretation > whether volatile - non-volatile accesses are allowed to reorder. MSVC > is known to interpret such interactions as-if the volatile accesses > have acquire/release semantics from the compiler point of view, and > others such as GCC are known to reorder away freely. > > To prevent any issues, the accesses involved when using > GuardUnsafeAccess should be at least volatile. > This change makes the few remaining ones volatile. The JMM-volatile > (SEQ_CST) accesses with crash protection already have stronger > ordering than volatile and hence do not need changing. > > By making the address passed in to the Access API volatile, the > MO_VOLATILE decorator is automatically set, which not surprisingly > makes the access volatile. Therefore, the solution is to simply make > the address passed in to Access volatile in this case. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8186787 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ > > Thanks, > /Erik From coleen.phillimore at oracle.com Mon Nov 27 18:28:57 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 27 Nov 2017 13:28:57 -0500 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: References: <5A1BF6F0.5030903@oracle.com> Message-ID: <3b2f7a9a-8abf-5a2a-9381-9046ffc21ee9@oracle.com> On 11/27/17 1:21 PM, coleen.phillimore at oracle.com wrote: > http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/src/hotspot/share/prims/jvmtiTagMap.cpp.udiff.html > > > + return RootAccess AS_NO_KEEPALIVE>::oop_load(object_addr()); > Sorry I have my Access API dimensions mixed up.? RootAccess is IN_ROOT not ON_ROOT (and not concurrent). Coleen > > Why is this not access ON_ROOT{_CONCURRENT} ?? The thing holding the > object that you are peeking at is not in the Java Heap? > > thanks, > Coleen > > On 11/27/17 6:28 AM, Erik ?sterlund wrote: >> Hi, >> >> The JVMTI tag hashmap has weak oop references that are handled using >> raw oop accesses and a G1-specific SATB enqueue call when leaking out >> objects from the tag map, requiring them to be marked as live by G1. >> >> This should now be refactored to use the Access API to annotate that >> these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use >> ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191894 >> >> Thanks, >> /Erik > From volker.simonis at gmail.com Mon Nov 27 18:34:49 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Mon, 27 Nov 2017 19:34:49 +0100 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> Message-ID: Hi Ioi, Dmitry, I've finished my testing on Linux/ppc64, Linux/ppc64le, Linux/s390 and AIX. All the tests passed successfully (with the corresponding extra ppc/s390 patches currently under review). I did run the tests with the following add-on to your change, which enables AppCDS with custom class loaders on all 64-bit Linux, Solaris and AIX platforms: http://cr.openjdk.java.net/~simonis/webrevs/2017/8188791-open-appcds.v02.tests-addon/ The only change required in HotSpot is in "classListParser.cpp" to enable the feature on the named platforms. The other changes just enable the additional tests on all 64-bit Linux, Solaris and AIX platfroms. @Ioi: can you please append this add-on to your change? @Dmitry: can you please re-build and re-run the tests on 64-bit ARM just to make sure everything is working? Notice, that with my add-ons about 20 more tests than before will be executed, but that was no problem an the other Linux platforms I've tested. Thank you and best regards, Volker On Fri, Nov 24, 2017 at 5:08 PM, Dmitry Samersoff wrote: > Volker, > >> Currently I'm building and running some tests >> and I hope that until Monday we may even extend the list of supported >> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >> that works as well, the check could be simplified to allow all the >> 64-bit Linux platforms. > > Yes, I added similar patch for Linux/AARCH64 and it works. > > So we should simplify the check. > > -Dmitry > > On 11/24/2017 05:51 PM, Volker Simonis wrote: >> Hi Ioi, >> >> you patch is not applying any more after "8187118: Remove appending >> -cp path to the boot class path at AppCDS dump time" was pushed to >> jdk/hs because of conflicts in classLoaderExt.hpp. >> >> Could you please rebase your change upon the newest version of jdk/hs ? >> >> By the way, on ppc64 all the tests passed after I found and fixed a >> trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite >> invokefinal if DumpSharedSpaces" for it). >> >> Can you please explain why AppCDS with custom class loaders is >> currently restricted to Linux/x86_64 and 64-bit Solaris? >> >> With the following small patch: >> >> diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp >> --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >> 15:37:19 2017 +0100 >> +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >> 15:38:21 2017 +0100 >> @@ -272,6 +272,7 @@ >> // during archive dumping. >> InstanceKlass* ClassListParser::load_class_from_source(Symbol* >> class_name, TRAPS) { >> #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ >> + (defined(PPC) && defined(_LP64)) || \ >> (defined(SOLARIS) && defined(_LP64))) >> // The only supported platforms are: (1) Linux/AMD64; (2) Solaris/64-bit >> error("AppCDS custom class loaders not supported on this platform"); >> diff -r 23deffd3a9c4 >> test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >> --- a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >> Fri Nov 24 15:37:19 2017 +0100 >> +++ b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >> Fri Nov 24 15:38:21 2017 +0100 >> @@ -56,6 +56,7 @@ >> OutputAnalyzer out = TestCommon.dump(appJar, classlist); >> >> if ((Platform.isSolaris() && Platform.is64bit()) || >> + (Platform.isLinux() && Platform.isPPC()) || >> (Platform.isLinux() && Platform.isX64())) { >> out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); >> out.shouldHaveExitValue(0); >> >> >> I could at least pass all the relevant jtreg tests on Linux/ppc64. So >> if there are no other hidden reasons I'd kindly ask you to add this >> patch to you change. I'm currently testing on AIX and I hope the tests >> will also succeed. If this will be the case, the list in the >> aforementioned patch could also be extended by AIX. >> >> Finally I've also did some small fixes on s390 (opened "8191863: >> [s390] Fix CDS: some bytecode rewriting doesn't depend on >> RewriteControl" for it). Currently I'm building and running some tests >> and I hope that until Monday we may even extend the list of supported >> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >> that works as well, the check could be simplified to allow all the >> 64-bit Linux platforms. >> >> Thank you and best regards, >> Volker >> >> On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: >>> Hi Volker, >>> >>> Thanks for trying the patch out. >>> >>> >>> On 11/17/17 5:42 AM, Volker Simonis wrote: >>>> >>>> Hi Ioi, >>>> >>>> I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX >>>> ports. >>>> >>>> I applied your patch to the current jdk-hs repo as of today and first >>>> of all I see the same compilation problems already reported by Dmitry >>>> before. Can you please update you webrev with the corresponding >>>> changes or create a new one. That will help other who wish to look at >>>> this change. >>> >>> >>> I've updated the patch inside the webrev. You can download it from >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch >>> >>> >>> The previous patch (which you had the problems with) is still available as >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old >>> >>> I will post a new webrev, but it's big and I might be running out of quota >>> from cr.openjdk.java.net, so I need to figure out how to handle that. >>> >>> The jdk/hs changeset corresponding to the new patch is the following: >>> >>> changeset: 47860:564882d918d4 >>> user: zgu >>> date: Thu Nov 16 20:21:11 2017 -0500 >>> files: src/hotspot/share/services/memTracker.cpp >>> description: >>> 8190357: NMT: Include metadata information in NMT final report when >>> PrintNMTStatistics is on >>> Summary: Include metadata information in NMT final report >>> Reviewed-by: adinn, stuefe >>> >>>> After building, I ran the appcds jtreg tests (i.e. >>>> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see >>>> the same four failures like Dmitry. >>> >>> Those are also due to merge issues. With the updated patch, I passed all >>> tests under on Linux/x64: >>> >>> test/hotspot/jtreg/runtime/SharedArchiveFile >>> test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>> test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>> test/hotspot/jtreg/runtime/appcds >>> >>> $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 >>> -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk >>> -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 >>> -retain -agentvm >>> -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt >>> -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt >>> -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native >>> -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ >>> open/test/hotspot/jtreg/runtime/SharedArchiveFile >>> open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>> open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>> open/test/hotspot/jtreg/runtime/appcds >>> [...] >>> Test results: passed: 128 >>> Results written to /jdk/tmp/jtreg/work >>> >>> >>>> Is that expected? I see a lot more errors on Linux/ppc64le but before >>>> going into more detail I'd like to know what is the correct baseline >>>> on which we can rely. >>>> >>>> Finally, I saw that with a product build, I get two additional test >>>> failures: >>>> >>>> runtime/appcds/ProhibitedPackage.java >>>> runtime/appcds/DumpClassList.java >>>> >>>> because of: >>>> >>>> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >>>> available only in debug version of VM. >>>> >>>> So these two tests should probably be adjusted to only run for >>>> slow/fastdebug builds. >>> >>> >>> You're correct. I will exclude those tests from product builds, and file an >>> RFE to fix them later. >>> >>> Thanks >>> - Ioi >>> >>>> Thank you and best regards, >>>> Volker >>>> >>>> >>>> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >>>> wrote: >>>>> >>>>> Ioi, >>>>> >>>>> Thank you! >>>>> >>>>> I did some additional testing and find that four tests >>>>> fails the same way on both x86_64 and AARCH64 (see below). >>>>> >>>>> runtime/appcds/VerifierTest_0.java >>>>> runtime/appcds/cacheObject/GCStressTest.java >>>>> runtime/appcds/cacheObject/RedefineClassTest.java >>>>> runtime/appcds/sharedStrings/InternSharedString.java >>>>> >>>>> Is it expected? >>>>> >>>>> test result: Failed. Execution failed: `main' threw exception: >>>>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>>>> asses' missing from stdout/stderr >>>>> >>>>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>>>> GCStressApp_nonshared_string should not be shared >>>>> at GCStressApp.main(GCStressApp.java:73) >>>>> >>>>> >>>>> FAILED. buzzshould not be shared >>>>> >>>>> Exception in thread "main" java.lang.RuntimeException: Failed: >>>>> unexpected shared string. >>>>> at InternStringTest.main(InternStringTest.java:63) >>>>> >>>>> -Dmitry >>>>> >>>>> On 16.11.2017 03:10, Ioi Lam wrote: >>>>>> >>>>>> I filed: >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>>>>> VMProps to filter out CDS tests >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>>>>> when CDS is not supported >>>>>> >>>>>> Thanks >>>>>> >>>>>> - Ioi >>>>>> >>>>>> >>>>>> >>>>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>>>> >>>>>>> Hi Dmitry, >>>>>>> >>>>>>> Thanks for the review. >>>>>>> >>>>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>>>> >>>>>>>> Ioi, >>>>>>>> >>>>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>>>> below[1]. >>>>>>> >>>>>>> I've fixed it. Thanks for the patch. >>>>>>>> >>>>>>>> Also tests doesn't work with exploded image - is it possible to check >>>>>>>> for this case and produce appropriate error message? >>>>>>> >>>>>>> CDS is not supported on exploded images. I think the best thing to do >>>>>>> is to add something like "@requires vm.cds" into the test cases >>>>>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>>>>> none of the CDS tests will be executed for >>>>>>> >>>>>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>>>>> it in the next release. >>>>>>> >>>>>>> I'll also file another RFE to print a better message. Today if you use >>>>>>> an exploded build the error message is kind of confusing: >>>>>>> >>>>>>> $ ./jdk/bin/java -Xshare:dump >>>>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>>>> Error occurred during initialization of VM >>>>>>> CDS allows only empty directories in archived classpaths >>>>>>> >>>>>>> It should say something like "CDS is not supported on exploded images". >>>>>>> >>>>>>> Thanks >>>>>>> - Ioi >>>>>>> >>>>>>>> 1. >>>>>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>> 09:45:23 2017 +0000 >>>>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>> 15:02:51 2017 +0000 >>>>>>>> @@ -31,7 +31,6 @@ >>>>>>>> -#include "prims/jvm.h" >>>>>>>> >>>>>>>> diff -r dbf9cec6a568 >>>>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>> 06 09:45:23 2017 +0000 >>>>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>> 06 15:02:51 2017 +0000 >>>>>>>> @@ -518,7 +518,7 @@ >>>>>>>> >>>>>>>> { >>>>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>>>>> if (check != NULL) { >>>>>>>> return InstanceKlass::cast(check); >>>>>>>> } >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>>>>> During internal review of the JEP, we have decided to integrate both >>>>>>>>> implementation and tests at the same time. >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>>>> >>>>>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>>>>> result of copying the closed source files on top of files of the same >>>>>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>>>>> "changed", please focus on the entire content of the new version of >>>>>>>>> each >>>>>>>>> file. >>>>>>>>> >>>>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>>>> sources) and all the new appcds tests passed. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> - Ioi >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>>>> >>>>>>>>>> Hi Dmitry, >>>>>>>>>> >>>>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>>>> >>>>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> - Ioi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>>>> >>>>>>>>>>> Ioi, >>>>>>>>>>> >>>>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>>>> the compilation fails with: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>>>>> >>>>>>>>>>> Did I miss something? >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Please review this change set. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>>>> >>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>>>> >>>>>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>>>>> moves >>>>>>>>>>>> AppCDS from >>>>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>>>> >>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>>>> >>>>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>>>>> repo. >>>>>>>>>>>> As part >>>>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>>>>> code >>>>>>>>>>>> into the >>>>>>>>>>>> open repo. >>>>>>>>>>>> >>>>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>>>> possible. The >>>>>>>>>>>> intention is >>>>>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>>>>> and not >>>>>>>>>>>> to do any sort of refactoring. >>>>>>>>>>>> >>>>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>>>> copying the >>>>>>>>>>>> closed source >>>>>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>>>>> reviewing, instead of >>>>>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>>>>> content >>>>>>>>>>>> of the new >>>>>>>>>>>> version of each file. >>>>>>>>>>>> >>>>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>>>> flag is >>>>>>>>>>>> changed from >>>>>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>>>>> "commercial" >>>>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>>>> >>>>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>>>> open/closed >>>>>>>>>>>> source >>>>>>>>>>>> code structure had introduced some intermediary APIs to connect >>>>>>>>>>>> code >>>>>>>>>>>> between >>>>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>>>> >>>>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>>>>> tests >>>>>>>>>>>> will be >>>>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>>>> >>>>>>>>>>>> All the AppCDS tests (currently still in closed sources) passed >>>>>>>>>>>> with >>>>>>>>>>>> both Oracle JDK >>>>>>>>>>>> and OpenJDK. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> - Ioi >>>>> >>>>> >>> > > From dmitry.samersoff at bell-sw.com Mon Nov 27 19:15:16 2017 From: dmitry.samersoff at bell-sw.com (Dmitry Samersoff) Date: Mon, 27 Nov 2017 22:15:16 +0300 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> Message-ID: <6060657c-36fc-1126-ea17-4cbe0a479892@bell-sw.com> Volker, > @Dmitry: can you please re-build and re-run the tests on 64-bit ARM > just to make sure everything is working? Notice, that with my add-ons > about 20 more tests than before will be executed, but that was no > problem an the other Linux platforms I've tested. I'll do it. -Dmitry On 11/27/2017 09:34 PM, Volker Simonis wrote: > Hi Ioi, Dmitry, > > I've finished my testing on Linux/ppc64, Linux/ppc64le, Linux/s390 and > AIX. All the tests passed successfully (with the corresponding extra > ppc/s390 patches currently under review). > > I did run the tests with the following add-on to your change, which > enables AppCDS with custom class loaders on all 64-bit Linux, Solaris > and AIX platforms: > > http://cr.openjdk.java.net/~simonis/webrevs/2017/8188791-open-appcds.v02.tests-addon/ > > The only change required in HotSpot is in "classListParser.cpp" to > enable the feature on the named platforms. The other changes just > enable the additional tests on all 64-bit Linux, Solaris and AIX > platfroms. > > @Ioi: can you please append this add-on to your change? > > @Dmitry: can you please re-build and re-run the tests on 64-bit ARM > just to make sure everything is working? Notice, that with my add-ons > about 20 more tests than before will be executed, but that was no > problem an the other Linux platforms I've tested. > > Thank you and best regards, > Volker > > > On Fri, Nov 24, 2017 at 5:08 PM, Dmitry Samersoff wrote: >> Volker, >> >>> Currently I'm building and running some tests >>> and I hope that until Monday we may even extend the list of supported >>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>> that works as well, the check could be simplified to allow all the >>> 64-bit Linux platforms. >> >> Yes, I added similar patch for Linux/AARCH64 and it works. >> >> So we should simplify the check. >> >> -Dmitry >> >> On 11/24/2017 05:51 PM, Volker Simonis wrote: >>> Hi Ioi, >>> >>> you patch is not applying any more after "8187118: Remove appending >>> -cp path to the boot class path at AppCDS dump time" was pushed to >>> jdk/hs because of conflicts in classLoaderExt.hpp. >>> >>> Could you please rebase your change upon the newest version of jdk/hs ? >>> >>> By the way, on ppc64 all the tests passed after I found and fixed a >>> trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite >>> invokefinal if DumpSharedSpaces" for it). >>> >>> Can you please explain why AppCDS with custom class loaders is >>> currently restricted to Linux/x86_64 and 64-bit Solaris? >>> >>> With the following small patch: >>> >>> diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp >>> --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>> 15:37:19 2017 +0100 >>> +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>> 15:38:21 2017 +0100 >>> @@ -272,6 +272,7 @@ >>> // during archive dumping. >>> InstanceKlass* ClassListParser::load_class_from_source(Symbol* >>> class_name, TRAPS) { >>> #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ >>> + (defined(PPC) && defined(_LP64)) || \ >>> (defined(SOLARIS) && defined(_LP64))) >>> // The only supported platforms are: (1) Linux/AMD64; (2) Solaris/64-bit >>> error("AppCDS custom class loaders not supported on this platform"); >>> diff -r 23deffd3a9c4 >>> test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>> --- a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>> Fri Nov 24 15:37:19 2017 +0100 >>> +++ b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>> Fri Nov 24 15:38:21 2017 +0100 >>> @@ -56,6 +56,7 @@ >>> OutputAnalyzer out = TestCommon.dump(appJar, classlist); >>> >>> if ((Platform.isSolaris() && Platform.is64bit()) || >>> + (Platform.isLinux() && Platform.isPPC()) || >>> (Platform.isLinux() && Platform.isX64())) { >>> out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); >>> out.shouldHaveExitValue(0); >>> >>> >>> I could at least pass all the relevant jtreg tests on Linux/ppc64. So >>> if there are no other hidden reasons I'd kindly ask you to add this >>> patch to you change. I'm currently testing on AIX and I hope the tests >>> will also succeed. If this will be the case, the list in the >>> aforementioned patch could also be extended by AIX. >>> >>> Finally I've also did some small fixes on s390 (opened "8191863: >>> [s390] Fix CDS: some bytecode rewriting doesn't depend on >>> RewriteControl" for it). Currently I'm building and running some tests >>> and I hope that until Monday we may even extend the list of supported >>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>> that works as well, the check could be simplified to allow all the >>> 64-bit Linux platforms. >>> >>> Thank you and best regards, >>> Volker >>> >>> On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: >>>> Hi Volker, >>>> >>>> Thanks for trying the patch out. >>>> >>>> >>>> On 11/17/17 5:42 AM, Volker Simonis wrote: >>>>> >>>>> Hi Ioi, >>>>> >>>>> I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX >>>>> ports. >>>>> >>>>> I applied your patch to the current jdk-hs repo as of today and first >>>>> of all I see the same compilation problems already reported by Dmitry >>>>> before. Can you please update you webrev with the corresponding >>>>> changes or create a new one. That will help other who wish to look at >>>>> this change. >>>> >>>> >>>> I've updated the patch inside the webrev. You can download it from >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch >>>> >>>> >>>> The previous patch (which you had the problems with) is still available as >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old >>>> >>>> I will post a new webrev, but it's big and I might be running out of quota >>>> from cr.openjdk.java.net, so I need to figure out how to handle that. >>>> >>>> The jdk/hs changeset corresponding to the new patch is the following: >>>> >>>> changeset: 47860:564882d918d4 >>>> user: zgu >>>> date: Thu Nov 16 20:21:11 2017 -0500 >>>> files: src/hotspot/share/services/memTracker.cpp >>>> description: >>>> 8190357: NMT: Include metadata information in NMT final report when >>>> PrintNMTStatistics is on >>>> Summary: Include metadata information in NMT final report >>>> Reviewed-by: adinn, stuefe >>>> >>>>> After building, I ran the appcds jtreg tests (i.e. >>>>> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see >>>>> the same four failures like Dmitry. >>>> >>>> Those are also due to merge issues. With the updated patch, I passed all >>>> tests under on Linux/x64: >>>> >>>> test/hotspot/jtreg/runtime/SharedArchiveFile >>>> test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>> test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>> test/hotspot/jtreg/runtime/appcds >>>> >>>> $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 >>>> -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk >>>> -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 >>>> -retain -agentvm >>>> -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt >>>> -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt >>>> -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native >>>> -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ >>>> open/test/hotspot/jtreg/runtime/SharedArchiveFile >>>> open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>> open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>> open/test/hotspot/jtreg/runtime/appcds >>>> [...] >>>> Test results: passed: 128 >>>> Results written to /jdk/tmp/jtreg/work >>>> >>>> >>>>> Is that expected? I see a lot more errors on Linux/ppc64le but before >>>>> going into more detail I'd like to know what is the correct baseline >>>>> on which we can rely. >>>>> >>>>> Finally, I saw that with a product build, I get two additional test >>>>> failures: >>>>> >>>>> runtime/appcds/ProhibitedPackage.java >>>>> runtime/appcds/DumpClassList.java >>>>> >>>>> because of: >>>>> >>>>> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >>>>> available only in debug version of VM. >>>>> >>>>> So these two tests should probably be adjusted to only run for >>>>> slow/fastdebug builds. >>>> >>>> >>>> You're correct. I will exclude those tests from product builds, and file an >>>> RFE to fix them later. >>>> >>>> Thanks >>>> - Ioi >>>> >>>>> Thank you and best regards, >>>>> Volker >>>>> >>>>> >>>>> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >>>>> wrote: >>>>>> >>>>>> Ioi, >>>>>> >>>>>> Thank you! >>>>>> >>>>>> I did some additional testing and find that four tests >>>>>> fails the same way on both x86_64 and AARCH64 (see below). >>>>>> >>>>>> runtime/appcds/VerifierTest_0.java >>>>>> runtime/appcds/cacheObject/GCStressTest.java >>>>>> runtime/appcds/cacheObject/RedefineClassTest.java >>>>>> runtime/appcds/sharedStrings/InternSharedString.java >>>>>> >>>>>> Is it expected? >>>>>> >>>>>> test result: Failed. Execution failed: `main' threw exception: >>>>>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>>>>> asses' missing from stdout/stderr >>>>>> >>>>>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>>>>> GCStressApp_nonshared_string should not be shared >>>>>> at GCStressApp.main(GCStressApp.java:73) >>>>>> >>>>>> >>>>>> FAILED. buzzshould not be shared >>>>>> >>>>>> Exception in thread "main" java.lang.RuntimeException: Failed: >>>>>> unexpected shared string. >>>>>> at InternStringTest.main(InternStringTest.java:63) >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> On 16.11.2017 03:10, Ioi Lam wrote: >>>>>>> >>>>>>> I filed: >>>>>>> >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>>>>>> VMProps to filter out CDS tests >>>>>>> >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>>>>>> when CDS is not supported >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> - Ioi >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>>>>> >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> Thanks for the review. >>>>>>>> >>>>>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>>>>> >>>>>>>>> Ioi, >>>>>>>>> >>>>>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>>>>> below[1]. >>>>>>>> >>>>>>>> I've fixed it. Thanks for the patch. >>>>>>>>> >>>>>>>>> Also tests doesn't work with exploded image - is it possible to check >>>>>>>>> for this case and produce appropriate error message? >>>>>>>> >>>>>>>> CDS is not supported on exploded images. I think the best thing to do >>>>>>>> is to add something like "@requires vm.cds" into the test cases >>>>>>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>>>>>> none of the CDS tests will be executed for >>>>>>>> >>>>>>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>>>>>> it in the next release. >>>>>>>> >>>>>>>> I'll also file another RFE to print a better message. Today if you use >>>>>>>> an exploded build the error message is kind of confusing: >>>>>>>> >>>>>>>> $ ./jdk/bin/java -Xshare:dump >>>>>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>>>>> Error occurred during initialization of VM >>>>>>>> CDS allows only empty directories in archived classpaths >>>>>>>> >>>>>>>> It should say something like "CDS is not supported on exploded images". >>>>>>>> >>>>>>>> Thanks >>>>>>>> - Ioi >>>>>>>> >>>>>>>>> 1. >>>>>>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>>> 09:45:23 2017 +0000 >>>>>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>>> 15:02:51 2017 +0000 >>>>>>>>> @@ -31,7 +31,6 @@ >>>>>>>>> -#include "prims/jvm.h" >>>>>>>>> >>>>>>>>> diff -r dbf9cec6a568 >>>>>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>>> 06 09:45:23 2017 +0000 >>>>>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>>> 06 15:02:51 2017 +0000 >>>>>>>>> @@ -518,7 +518,7 @@ >>>>>>>>> >>>>>>>>> { >>>>>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>>>>>> if (check != NULL) { >>>>>>>>> return InstanceKlass::cast(check); >>>>>>>>> } >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>>>>>> During internal review of the JEP, we have decided to integrate both >>>>>>>>>> implementation and tests at the same time. >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>>>>> >>>>>>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>>>>>> result of copying the closed source files on top of files of the same >>>>>>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>>>>>> "changed", please focus on the entire content of the new version of >>>>>>>>>> each >>>>>>>>>> file. >>>>>>>>>> >>>>>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>>>>> sources) and all the new appcds tests passed. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> - Ioi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Dmitry, >>>>>>>>>>> >>>>>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>>>>> >>>>>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> - Ioi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>>>>> >>>>>>>>>>>> Ioi, >>>>>>>>>>>> >>>>>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>>>>> the compilation fails with: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>>>>>> >>>>>>>>>>>> Did I miss something? >>>>>>>>>>>> >>>>>>>>>>>> -Dmitry >>>>>>>>>>>> >>>>>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Please review this change set. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>>>>> >>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>>>>> >>>>>>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>>>>>> moves >>>>>>>>>>>>> AppCDS from >>>>>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>>>>> >>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>>>>> >>>>>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>>>>>> repo. >>>>>>>>>>>>> As part >>>>>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>>>>>> code >>>>>>>>>>>>> into the >>>>>>>>>>>>> open repo. >>>>>>>>>>>>> >>>>>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>>>>> possible. The >>>>>>>>>>>>> intention is >>>>>>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>>>>>> and not >>>>>>>>>>>>> to do any sort of refactoring. >>>>>>>>>>>>> >>>>>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>>>>> copying the >>>>>>>>>>>>> closed source >>>>>>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>>>>>> reviewing, instead of >>>>>>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>>>>>> content >>>>>>>>>>>>> of the new >>>>>>>>>>>>> version of each file. >>>>>>>>>>>>> >>>>>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>>>>> flag is >>>>>>>>>>>>> changed from >>>>>>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>>>>>> "commercial" >>>>>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>>>>> >>>>>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>>>>> open/closed >>>>>>>>>>>>> source >>>>>>>>>>>>> code structure had introduced some intermediary APIs to connect >>>>>>>>>>>>> code >>>>>>>>>>>>> between >>>>>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>>>>> >>>>>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>>>>>> tests >>>>>>>>>>>>> will be >>>>>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>>>>> >>>>>>>>>>>>> All the AppCDS tests (currently still in closed sources) passed >>>>>>>>>>>>> with >>>>>>>>>>>>> both Oracle JDK >>>>>>>>>>>>> and OpenJDK. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> - Ioi >>>>>> >>>>>> >>>> >> >> From ioi.lam at oracle.com Mon Nov 27 20:37:19 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 27 Nov 2017 12:37:19 -0800 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <6060657c-36fc-1126-ea17-4cbe0a479892@bell-sw.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> <6060657c-36fc-1126-ea17-4cbe0a479892@bell-sw.com> Message-ID: <4c4c3846-908c-5ba8-07f5-a283025c3dd8@oracle.com> Hi Volker & Dmitry, Thanks for testing on the additional platforms. It really is great help! We had a very limited set of supported platforms for the custom loaders due to lack of testing resources. I think now we can expand the set of platforms. However, I would like to keep JDK-8188791 as simple as possible -- just moving the code, but don't introduce other changes. So I've created ??? https://bugs.openjdk.java.net/browse/JDK-8191927 ??? Enable AppCDS for custom loaders on all 64-bit Linux and AIX We can try to push that after 8191927 is pushed (which should happen very soon once the JEP is targeted to JDK 10, hopefully today). I've posted the latest webrev at http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ It's based on this changeset ??? changeset:?? 48006:235a18d659fc ??? user:??????? roland ??? date:??????? Mon Nov 27 10:44:19 2017 -0800 ??? files:?????? src/hotspot/share/opto/split_if.cpp test/hotspot/jtreg/compiler/loopopts/TestSplitIfPinnedCMove.java ??? description: ??? 8191153: assert(u_ctrl != blk1 && u_ctrl != blk2) failed: won't converge ??? Summary: relax assert ??? Reviewed-by: kvn This version also fixes the issue with -XX:+PrintSystemDictionaryAtExit on product builds with the two tests DumpClassList.java and ProhibitedPackage.java. Thanks - Ioi On 11/27/17 11:15 AM, Dmitry Samersoff wrote: > Volker, > >> @Dmitry: can you please re-build and re-run the tests on 64-bit ARM >> just to make sure everything is working? Notice, that with my add-ons >> about 20 more tests than before will be executed, but that was no >> problem an the other Linux platforms I've tested. > > I'll do it. > > -Dmitry > > > On 11/27/2017 09:34 PM, Volker Simonis wrote: >> Hi Ioi, Dmitry, >> >> I've finished my testing on Linux/ppc64, Linux/ppc64le, Linux/s390 and >> AIX. All the tests passed successfully (with the corresponding extra >> ppc/s390 patches currently under review). >> >> I did run the tests with the following add-on to your change, which >> enables AppCDS with custom class loaders on all 64-bit Linux, Solaris >> and AIX platforms: >> >> http://cr.openjdk.java.net/~simonis/webrevs/2017/8188791-open-appcds.v02.tests-addon/ >> >> The only change required in HotSpot is in "classListParser.cpp" to >> enable the feature on the named platforms. The other changes just >> enable the additional tests on all 64-bit Linux, Solaris and AIX >> platfroms. >> >> @Ioi: can you please append this add-on to your change? >> >> @Dmitry: can you please re-build and re-run the tests on 64-bit ARM >> just to make sure everything is working? Notice, that with my add-ons >> about 20 more tests than before will be executed, but that was no >> problem an the other Linux platforms I've tested. >> >> Thank you and best regards, >> Volker >> >> >> On Fri, Nov 24, 2017 at 5:08 PM, Dmitry Samersoff wrote: >>> Volker, >>> >>>> Currently I'm building and running some tests >>>> and I hope that until Monday we may even extend the list of supported >>>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>>> that works as well, the check could be simplified to allow all the >>>> 64-bit Linux platforms. >>> Yes, I added similar patch for Linux/AARCH64 and it works. >>> >>> So we should simplify the check. >>> >>> -Dmitry >>> >>> On 11/24/2017 05:51 PM, Volker Simonis wrote: >>>> Hi Ioi, >>>> >>>> you patch is not applying any more after "8187118: Remove appending >>>> -cp path to the boot class path at AppCDS dump time" was pushed to >>>> jdk/hs because of conflicts in classLoaderExt.hpp. >>>> >>>> Could you please rebase your change upon the newest version of jdk/hs ? >>>> >>>> By the way, on ppc64 all the tests passed after I found and fixed a >>>> trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite >>>> invokefinal if DumpSharedSpaces" for it). >>>> >>>> Can you please explain why AppCDS with custom class loaders is >>>> currently restricted to Linux/x86_64 and 64-bit Solaris? >>>> >>>> With the following small patch: >>>> >>>> diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp >>>> --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>>> 15:37:19 2017 +0100 >>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>>> 15:38:21 2017 +0100 >>>> @@ -272,6 +272,7 @@ >>>> // during archive dumping. >>>> InstanceKlass* ClassListParser::load_class_from_source(Symbol* >>>> class_name, TRAPS) { >>>> #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ >>>> + (defined(PPC) && defined(_LP64)) || \ >>>> (defined(SOLARIS) && defined(_LP64))) >>>> // The only supported platforms are: (1) Linux/AMD64; (2) Solaris/64-bit >>>> error("AppCDS custom class loaders not supported on this platform"); >>>> diff -r 23deffd3a9c4 >>>> test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>> --- a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>> Fri Nov 24 15:37:19 2017 +0100 >>>> +++ b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>> Fri Nov 24 15:38:21 2017 +0100 >>>> @@ -56,6 +56,7 @@ >>>> OutputAnalyzer out = TestCommon.dump(appJar, classlist); >>>> >>>> if ((Platform.isSolaris() && Platform.is64bit()) || >>>> + (Platform.isLinux() && Platform.isPPC()) || >>>> (Platform.isLinux() && Platform.isX64())) { >>>> out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); >>>> out.shouldHaveExitValue(0); >>>> >>>> >>>> I could at least pass all the relevant jtreg tests on Linux/ppc64. So >>>> if there are no other hidden reasons I'd kindly ask you to add this >>>> patch to you change. I'm currently testing on AIX and I hope the tests >>>> will also succeed. If this will be the case, the list in the >>>> aforementioned patch could also be extended by AIX. >>>> >>>> Finally I've also did some small fixes on s390 (opened "8191863: >>>> [s390] Fix CDS: some bytecode rewriting doesn't depend on >>>> RewriteControl" for it). Currently I'm building and running some tests >>>> and I hope that until Monday we may even extend the list of supported >>>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>>> that works as well, the check could be simplified to allow all the >>>> 64-bit Linux platforms. >>>> >>>> Thank you and best regards, >>>> Volker >>>> >>>> On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: >>>>> Hi Volker, >>>>> >>>>> Thanks for trying the patch out. >>>>> >>>>> >>>>> On 11/17/17 5:42 AM, Volker Simonis wrote: >>>>>> Hi Ioi, >>>>>> >>>>>> I'm just trying to verify if AppCDS will be running on our ppc/s390/AIX >>>>>> ports. >>>>>> >>>>>> I applied your patch to the current jdk-hs repo as of today and first >>>>>> of all I see the same compilation problems already reported by Dmitry >>>>>> before. Can you please update you webrev with the corresponding >>>>>> changes or create a new one. That will help other who wish to look at >>>>>> this change. >>>>> >>>>> I've updated the patch inside the webrev. You can download it from >>>>> >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch >>>>> >>>>> >>>>> The previous patch (which you had the problems with) is still available as >>>>> >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old >>>>> >>>>> I will post a new webrev, but it's big and I might be running out of quota >>>>> from cr.openjdk.java.net, so I need to figure out how to handle that. >>>>> >>>>> The jdk/hs changeset corresponding to the new patch is the following: >>>>> >>>>> changeset: 47860:564882d918d4 >>>>> user: zgu >>>>> date: Thu Nov 16 20:21:11 2017 -0500 >>>>> files: src/hotspot/share/services/memTracker.cpp >>>>> description: >>>>> 8190357: NMT: Include metadata information in NMT final report when >>>>> PrintNMTStatistics is on >>>>> Summary: Include metadata information in NMT final report >>>>> Reviewed-by: adinn, stuefe >>>>> >>>>>> After building, I ran the appcds jtreg tests (i.e. >>>>>> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I see >>>>>> the same four failures like Dmitry. >>>>> Those are also due to merge issues. With the updated patch, I passed all >>>>> tests under on Linux/x64: >>>>> >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile >>>>> test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>>> test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>>> test/hotspot/jtreg/runtime/appcds >>>>> >>>>> $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 >>>>> -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk >>>>> -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 -timeout:4.0 >>>>> -retain -agentvm >>>>> -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt >>>>> -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt >>>>> -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native >>>>> -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ >>>>> open/test/hotspot/jtreg/runtime/SharedArchiveFile >>>>> open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>>> open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>>> open/test/hotspot/jtreg/runtime/appcds >>>>> [...] >>>>> Test results: passed: 128 >>>>> Results written to /jdk/tmp/jtreg/work >>>>> >>>>> >>>>>> Is that expected? I see a lot more errors on Linux/ppc64le but before >>>>>> going into more detail I'd like to know what is the correct baseline >>>>>> on which we can rely. >>>>>> >>>>>> Finally, I saw that with a product build, I get two additional test >>>>>> failures: >>>>>> >>>>>> runtime/appcds/ProhibitedPackage.java >>>>>> runtime/appcds/DumpClassList.java >>>>>> >>>>>> because of: >>>>>> >>>>>> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >>>>>> available only in debug version of VM. >>>>>> >>>>>> So these two tests should probably be adjusted to only run for >>>>>> slow/fastdebug builds. >>>>> >>>>> You're correct. I will exclude those tests from product builds, and file an >>>>> RFE to fix them later. >>>>> >>>>> Thanks >>>>> - Ioi >>>>> >>>>>> Thank you and best regards, >>>>>> Volker >>>>>> >>>>>> >>>>>> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >>>>>> wrote: >>>>>>> Ioi, >>>>>>> >>>>>>> Thank you! >>>>>>> >>>>>>> I did some additional testing and find that four tests >>>>>>> fails the same way on both x86_64 and AARCH64 (see below). >>>>>>> >>>>>>> runtime/appcds/VerifierTest_0.java >>>>>>> runtime/appcds/cacheObject/GCStressTest.java >>>>>>> runtime/appcds/cacheObject/RedefineClassTest.java >>>>>>> runtime/appcds/sharedStrings/InternSharedString.java >>>>>>> >>>>>>> Is it expected? >>>>>>> >>>>>>> test result: Failed. Execution failed: `main' threw exception: >>>>>>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>>>>>> asses' missing from stdout/stderr >>>>>>> >>>>>>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>>>>>> GCStressApp_nonshared_string should not be shared >>>>>>> at GCStressApp.main(GCStressApp.java:73) >>>>>>> >>>>>>> >>>>>>> FAILED. buzzshould not be shared >>>>>>> >>>>>>> Exception in thread "main" java.lang.RuntimeException: Failed: >>>>>>> unexpected shared string. >>>>>>> at InternStringTest.main(InternStringTest.java:63) >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 16.11.2017 03:10, Ioi Lam wrote: >>>>>>>> I filed: >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level jtreg >>>>>>>> VMProps to filter out CDS tests >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error message >>>>>>>> when CDS is not supported >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> - Ioi >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>>>>>> Hi Dmitry, >>>>>>>>> >>>>>>>>> Thanks for the review. >>>>>>>>> >>>>>>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>>>>>> >>>>>>>>>> Ioi, >>>>>>>>>> >>>>>>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>>>>>> below[1]. >>>>>>>>> I've fixed it. Thanks for the patch. >>>>>>>>>> Also tests doesn't work with exploded image - is it possible to check >>>>>>>>>> for this case and produce appropriate error message? >>>>>>>>> CDS is not supported on exploded images. I think the best thing to do >>>>>>>>> is to add something like "@requires vm.cds" into the test cases >>>>>>>>> (similar to the use of "@require vm.aot" in other tests). That way, >>>>>>>>> none of the CDS tests will be executed for >>>>>>>>> >>>>>>>>> It's too late to do this in JDK 10 now, but I will file an RFE to do >>>>>>>>> it in the next release. >>>>>>>>> >>>>>>>>> I'll also file another RFE to print a better message. Today if you use >>>>>>>>> an exploded build the error message is kind of confusing: >>>>>>>>> >>>>>>>>> $ ./jdk/bin/java -Xshare:dump >>>>>>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>>>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>>>>>> Error occurred during initialization of VM >>>>>>>>> CDS allows only empty directories in archived classpaths >>>>>>>>> >>>>>>>>> It should say something like "CDS is not supported on exploded images". >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> - Ioi >>>>>>>>> >>>>>>>>>> 1. >>>>>>>>>> diff -r dbf9cec6a568 src/hotspot/share/classfile/classListParser.cpp >>>>>>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>>>> 09:45:23 2017 +0000 >>>>>>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov 06 >>>>>>>>>> 15:02:51 2017 +0000 >>>>>>>>>> @@ -31,7 +31,6 @@ >>>>>>>>>> -#include "prims/jvm.h" >>>>>>>>>> >>>>>>>>>> diff -r dbf9cec6a568 >>>>>>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>>>> 06 09:45:23 2017 +0000 >>>>>>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon Nov >>>>>>>>>> 06 15:02:51 2017 +0000 >>>>>>>>>> @@ -518,7 +518,7 @@ >>>>>>>>>> >>>>>>>>>> { >>>>>>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>>>>>> - Klass* check = find_class(d_index, d_hash, name, dictionary); >>>>>>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, name); >>>>>>>>>> if (check != NULL) { >>>>>>>>>> return InstanceKlass::cast(check); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> -Dmitry >>>>>>>>>> >>>>>>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Here's the new webrev for both the AppCDS implementation and tests. >>>>>>>>>>> During internal review of the JEP, we have decided to integrate both >>>>>>>>>>> implementation and tests at the same time. >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>>>>>> >>>>>>>>>>> As mentioned before, most of the "diffs" shown in this webrev are the >>>>>>>>>>> result of copying the closed source files on top of files of the same >>>>>>>>>>> name in the open repo. So in reviewing, instead of focusing on what's >>>>>>>>>>> "changed", please focus on the entire content of the new version of >>>>>>>>>>> each >>>>>>>>>>> file. >>>>>>>>>>> >>>>>>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>>>>>> sources) and all the new appcds tests passed. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> - Ioi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>>>>>> Hi Dmitry, >>>>>>>>>>>> >>>>>>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>>>>>> >>>>>>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> - Ioi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>>>>>> Ioi, >>>>>>>>>>>>> >>>>>>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>>>>>> the compilation fails with: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >>>>>>>>>>>>> >>>>>>>>>>>>> Did I miss something? >>>>>>>>>>>>> >>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>> >>>>>>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please review this change set. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is the first step of implementing the following JEP, which >>>>>>>>>>>>>> moves >>>>>>>>>>>>>> AppCDS from >>>>>>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>>>>>> >>>>>>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the closed >>>>>>>>>>>>>> repo. >>>>>>>>>>>>>> As part >>>>>>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the source >>>>>>>>>>>>>> code >>>>>>>>>>>>>> into the >>>>>>>>>>>>>> open repo. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>>>>>> possible. The >>>>>>>>>>>>>> intention is >>>>>>>>>>>>>> only to relocate the sources, not to changing existing behaviors, >>>>>>>>>>>>>> and not >>>>>>>>>>>>>> to do any sort of refactoring. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>>>>>> copying the >>>>>>>>>>>>>> closed source >>>>>>>>>>>>>> files on top of files of the same name in the open repo. So in >>>>>>>>>>>>>> reviewing, instead of >>>>>>>>>>>>>> focusing on what's "changed", it's better to focus on the entire >>>>>>>>>>>>>> content >>>>>>>>>>>>>> of the new >>>>>>>>>>>>>> version of each file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>>>>>> flag is >>>>>>>>>>>>>> changed from >>>>>>>>>>>>>> a "commercial" flag to a regular "product" flag. This is because >>>>>>>>>>>>>> "commercial" >>>>>>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>>>>>> open/closed >>>>>>>>>>>>>> source >>>>>>>>>>>>>> code structure had introduced some intermediary APIs to connect >>>>>>>>>>>>>> code >>>>>>>>>>>>>> between >>>>>>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. These >>>>>>>>>>>>>> tests >>>>>>>>>>>>>> will be >>>>>>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All the AppCDS tests (currently still in closed sources) passed >>>>>>>>>>>>>> with >>>>>>>>>>>>>> both Oracle JDK >>>>>>>>>>>>>> and OpenJDK. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> - Ioi >>>>>>> >>> From ioi.lam at oracle.com Mon Nov 27 21:42:05 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 27 Nov 2017 13:42:05 -0800 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX Message-ID: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Hi, Please review this change to support AppCDS on all 64-bit Linux platforms as well as AIX. ??? https://bugs.openjdk.java.net/browse/JDK-8191927 http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ The patch must be applied on top of the AppCDS patch: http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ The patch was initially contributed by Volker Simonis. I modified it a little to avoid repeating the platform enumerations in all the test cases. Instead, I added a new test property, so the tests that require AppCDS support for custom loaders can be written as: ??? * @requires vm.cds.custom.loaders I've tested on Linux manually, and I am now running on all of the Oracle-supported platforms using our internal test harness. Thanks - Ioi From jiangli.zhou at Oracle.COM Mon Nov 27 23:25:08 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Mon, 27 Nov 2017 15:25:08 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure Message-ID: Hi, Please review following small test fix. CDS has multiple archived memory regions, which are mapped individually at runtime. CDSTestUtils.isUnableToMap() checks for mapping failures, but missing the ?OptionalData? memory region. The fix in the webrev adds the check for ?OptionalData? region. It also updated the error checking for archived heap memory region. webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 Tested with tier1, trer2, tier3, tier4, and tier5. Thanks, Jiangli From ioi.lam at oracle.com Mon Nov 27 23:33:45 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 27 Nov 2017 15:33:45 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: References: Message-ID: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> Looks good. Thanks! - Ioi On 11/27/17 3:25 PM, Jiangli Zhou wrote: > Hi, > > Please review following small test fix. CDS has multiple archived memory regions, which are mapped individually at runtime. CDSTestUtils.isUnableToMap() checks for mapping failures, but missing the ?OptionalData? memory region. The fix in the webrev adds the check for ?OptionalData? region. It also updated the error checking for archived heap memory region. > > webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 > > Tested with tier1, trer2, tier3, tier4, and tier5. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Mon Nov 27 23:46:16 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 27 Nov 2017 15:46:16 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> References: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> Message-ID: <9AD5E5CB-7F27-49BD-A8E5-9DC644E82BE3@oracle.com> Thanks for the quick review! Thanks, Jiangli > On Nov 27, 2017, at 3:33 PM, Ioi Lam wrote: > > Looks good. Thanks! > > - Ioi > > >> On 11/27/17 3:25 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review following small test fix. CDS has multiple archived memory regions, which are mapped individually at runtime. CDSTestUtils.isUnableToMap() checks for mapping failures, but missing the ?OptionalData? memory region. The fix in the webrev adds the check for ?OptionalData? region. It also updated the error checking for archived heap memory region. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 >> >> Tested with tier1, trer2, tier3, tier4, and tier5. >> >> Thanks, >> Jiangli > From calvin.cheung at oracle.com Mon Nov 27 23:53:01 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 27 Nov 2017 15:53:01 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> References: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> Message-ID: <5A1CA55D.5050009@oracle.com> +1 thanks, Calvin On 11/27/17, 3:33 PM, Ioi Lam wrote: > Looks good. Thanks! > > - Ioi > > > On 11/27/17 3:25 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review following small test fix. CDS has multiple archived >> memory regions, which are mapped individually at runtime. >> CDSTestUtils.isUnableToMap() checks for mapping failures, but missing >> the ?OptionalData? memory region. The fix in the webrev adds the >> check for ?OptionalData? region. It also updated the error checking >> for archived heap memory region. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 >> >> >> Tested with tier1, trer2, tier3, tier4, and tier5. >> >> Thanks, >> Jiangli > From jiangli.zhou at oracle.com Tue Nov 28 00:03:24 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 27 Nov 2017 16:03:24 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: <5A1CA55D.5050009@oracle.com> References: <6274f3b8-401c-6f3b-4ad0-7f7d357a6648@oracle.com> <5A1CA55D.5050009@oracle.com> Message-ID: Thanks, Calvin! Thanks, Jiangli > On Nov 27, 2017, at 3:53 PM, Calvin Cheung wrote: > > +1 > > thanks, > Calvin > >> On 11/27/17, 3:33 PM, Ioi Lam wrote: >> Looks good. Thanks! >> >> - Ioi >> >> >>> On 11/27/17 3:25 PM, Jiangli Zhou wrote: >>> Hi, >>> >>> Please review following small test fix. CDS has multiple archived >>> memory regions, which are mapped individually at runtime. >>> CDSTestUtils.isUnableToMap() checks for mapping failures, but missing >>> the ?OptionalData? memory region. The fix in the webrev adds the >>> check for ?OptionalData? region. It also updated the error checking >>> for archived heap memory region. >>> >>> webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 >>> >>> >>> Tested with tier1, trer2, tier3, tier4, and tier5. >>> >>> Thanks, >>> Jiangli >> From mikhailo.seledtsov at oracle.com Tue Nov 28 00:08:35 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Mon, 27 Nov 2017 16:08:35 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: References: Message-ID: Looks good, Misha On 11/27/2017 03:25 PM, Jiangli Zhou wrote: > Hi, > > Please review following small test fix. CDS has multiple archived memory regions, which are mapped individually at runtime. CDSTestUtils.isUnableToMap() checks for mapping failures, but missing the ?OptionalData? memory region. The fix in the webrev adds the check for ?OptionalData? region. It also updated the error checking for archived heap memory region. > > webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 > > Tested with tier1, trer2, tier3, tier4, and tier5. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Tue Nov 28 00:17:38 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 27 Nov 2017 16:17:38 -0800 Subject: RFR(XS): 8191504: CDSTestUtils.isUnableToMap() should check OptionalData region mapping failure In-Reply-To: References: Message-ID: <5A7750DB-4511-4E90-AB87-E67D68D7E580@oracle.com> Thanks for the review. Thanks, Jiangli > On Nov 27, 2017, at 4:08 PM, mikhailo wrote: > > Looks good, > > Misha > > >> On 11/27/2017 03:25 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review following small test fix. CDS has multiple archived memory regions, which are mapped individually at runtime. CDSTestUtils.isUnableToMap() checks for mapping failures, but missing the ?OptionalData? memory region. The fix in the webrev adds the check for ?OptionalData? region. It also updated the error checking for archived heap memory region. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8191504/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8191504?filter=14921 >> >> Tested with tier1, trer2, tier3, tier4, and tier5. >> >> Thanks, >> Jiangli > From jini.george at oracle.com Tue Nov 28 03:44:14 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 28 Nov 2017 09:14:14 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: References: Message-ID: Gentle reminder. Thank you, Jini. On 11/21/2017 4:04 PM, Jini George wrote: > Hello, > > Here's requesting reviews for some SA code cleanup. > > ID: https://bugs.openjdk.java.net/browse/JDK-8191324 > Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html > > The changes here are primarily to: > > 1. Remove unused IA64 SA code. > 2. Make changes to avoid error-prone redefinition of hotspot constants > in SA Java code. Instead read it in through vmStructs and > db.lookupIntConstant(). > 3. Make variable name changes in SA to align with the hotspot names. > > The changes are straightforward. > > Thanks much, > Jini. > From serguei.spitsyn at oracle.com Tue Nov 28 05:43:50 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 27 Nov 2017 21:43:50 -0800 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <88659101-1693-d5a4-a605-8f7b5ab53299@oracle.com> References: <5A1BF6F0.5030903@oracle.com> <88659101-1693-d5a4-a605-8f7b5ab53299@oracle.com> Message-ID: <2b566339-d3f8-27ab-c5c9-670aa278c9da@oracle.com> Hi Erik, This looks good to me. Thanks, Serguei On 11/27/17 05:22, Daniel D. Daugherty wrote: > Adding serviceability-dev at ... since this is JVM/TI related... > > Dan > > > On 11/27/17 6:28 AM, Erik ?sterlund wrote: >> Hi, >> >> The JVMTI tag hashmap has weak oop references that are handled using >> raw oop accesses and a G1-specific SATB enqueue call when leaking out >> objects from the tag map, requiring them to be marked as live by G1. >> >> This should now be refactored to use the Access API to annotate that >> these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use >> ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191894 >> >> Thanks, >> /Erik > From david.holmes at oracle.com Tue Nov 28 06:41:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 28 Nov 2017 16:41:06 +1000 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> References: <5A1C06D8.1040405@oracle.com> <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> Message-ID: +1 David On 28/11/2017 4:27 AM, coleen.phillimore at oracle.com wrote: > > I think this is a nice simple fix and should be pushed for JDK10. > Thanks, > Coleen > > On 11/27/17 7:36 AM, Erik ?sterlund wrote: >> Hi, >> >> There is currently a bug when using unsafe accesses off-heap: >> >> 1) We write into a thread that we enable crash protection (using >> GuardUnsafeAccess): >> 2) We perform the access >> 3) We write into a thread that we disable crash protection (using >> ~GuardUnsafeAccess) >> >> The problem is that the crash protection stores are volatile, but the >> actual access is non-volatile. Compilers have different interpretation >> whether volatile - non-volatile accesses are allowed to reorder. MSVC >> is known to interpret such interactions as-if the volatile accesses >> have acquire/release semantics from the compiler point of view, and >> others such as GCC are known to reorder away freely. >> >> To prevent any issues, the accesses involved when using >> GuardUnsafeAccess should be at least volatile. >> This change makes the few remaining ones volatile. The JMM-volatile >> (SEQ_CST) accesses with crash protection already have stronger >> ordering than volatile and hence do not need changing. >> >> By making the address passed in to the Access API volatile, the >> MO_VOLATILE decorator is automatically set, which not surprisingly >> makes the access volatile. Therefore, the solution is to simply make >> the address passed in to Access volatile in this case. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8186787 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >> >> Thanks, >> /Erik > From david.holmes at oracle.com Tue Nov 28 06:59:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 28 Nov 2017 16:59:15 +1000 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <5A1BD5B3.7030808@oracle.com> References: <5A1BD5B3.7030808@oracle.com> Message-ID: Hi Erik, Is there a non-GC person's guide to what the different forms of RootAccess mean and when to use them? Or do we expect a GC person to always jump in to show where such things are needed? ;-) Thanks, David On 27/11/2017 7:06 PM, Erik ?sterlund wrote: > Hi, > > The ClassLoaderData::remove_handle() member function currently uses > explicit G1 SATB barriers to remove an oop from the root set, as these > handles are not necessarily walked by the GC in a safepoint. Therefore > G1 needs pre-write barriers. > > This should now be modeled as a > RootAccess::oop_store instead. This maps to > performing a pre-write SATB barrier with G1, but other GCs are free to > do other things as necessary. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191888 > > Thanks, > /Erik From david.holmes at oracle.com Tue Nov 28 07:35:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 28 Nov 2017 17:35:55 +1000 Subject: RFR(one-liner): 8191707: Options with invalid values are incorrectly treated as obsolete and ignored In-Reply-To: <54ab4e67-abe2-9801-d237-e4052f9f74a4@oracle.com> References: <80a99246-28f9-3035-2469-7287eadfb31f@oracle.com> <54ab4e67-abe2-9801-d237-e4052f9f74a4@oracle.com> Message-ID: <90437262-53cd-4d49-2a9b-ed893999e261@oracle.com> On 23/11/2017 10:19 AM, Daniel D. Daugherty wrote: > On 11/22/17 3:24 PM, Robbin Ehn wrote: >> Hi all, please review: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191707 >> >> Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no >> unexpected failure. >> >> Contributed by David Holmes. >> >> Looks good, thanks for fixing! > > Thumbs up on the fix. > > What I don't understand is why this logic did not fall over before now. I think we previously always hit the "undefined" case for the obsolete version, when dealing with a flag that could have an invalid value. All of the other recently added deprecated flags were booleans. Of course that means we are missing some test coverage here. David > Dan > >> >> /Robbin >> >> >> diff -r 2cd1c2b03782 src/hotspot/share/runtime/arguments.cpp >> --- a/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 01:12:23 >> 2017 -0800 >> +++ b/src/hotspot/share/runtime/arguments.cpp??? Wed Nov 22 21:20:42 >> 2017 +0100 >> @@ -493,15 +493,15 @@ >> ?} >> >> ?bool Arguments::is_obsolete_flag(const char *flag_name, JDK_Version* >> version) { >> ?? assert(version != NULL, "Must provide a version buffer"); >> ?? SpecialFlag flag; >> ?? if (lookup_special_flag(flag_name, flag)) { >> ???? if (!flag.obsolete_in.is_undefined()) { >> -????? if (version_less_than(JDK_Version::current(), flag.expired_in)) { >> +????? if (!version_less_than(JDK_Version::current(), >> flag.obsolete_in)) { >> ???????? *version = flag.obsolete_in; >> ???????? return true; >> ?????? } >> ???? } >> ?? } >> ?? return false; >> ?} > From david.holmes at oracle.com Tue Nov 28 08:01:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 28 Nov 2017 18:01:27 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> Message-ID: <157a90ee-35c9-1adf-490a-9910f3ee646f@oracle.com> Just for the record ... On 23/11/2017 6:20 PM, Robbin Ehn wrote: > Thanks Dan for dragging this freight train to the docks, it's time to > ship it! I agree. The latest delta seems fine to me. > Created follow-up bug: > 8191809: Threads::number_of_threads() is not 'MT-safe' > https://bugs.openjdk.java.net/browse/JDK-8191809 Just updated that bug - I don't see any MT issues there. :) Cheers, David PS. Dan: yes JPRT was still doing 32-bit ARM a "month or two back". 64-bit atomics should not be a concern. That said I thought all the atomic updates were done for ARM etc. > /Robbin > > On 2017-11-23 03:16, Daniel D. Daugherty wrote: >> Greetings, >> >> I've made minor tweaks to the Thread-SMR project based on the remaining >> code review comments over the last couple of days. I've also rebased the >> project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm >> running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] >> testing... >> >> Here's a delta webrev for the latest round of tweaks: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta >> >> >> Here's the proposed commit message: >> >> $ cat SMR_prototype_10/open/commit.txt >> 8167108: inconsistent handling of SR_lock can lead to crashes >> Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. >> Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, kbarrett, >> rehn, sspitsyn, stefank >> Contributed-by: daniel.daugherty at oracle.com, >> erik.osterlund at oracle.com, robbin.ehn at oracle.com >> >> I've gone through 880+ emails in my archive for this project and added >> anyone that made a code review comment. Reviewers are not in my usual >> by-comment-posting order; it was way too hard to figure that out... So >> reviewers and contributors are simply in alpha-sort order... >> >> Here's the collection of follow-up bugs that I filed based on sifting >> through the email archive: >> >> 8191784 JavaThread::collect_counters() should stop using Threads_lock >> https://bugs.openjdk.java.net/browse/JDK-8191784 >> >> 8191785 revoke_bias() needs a new assertion >> https://bugs.openjdk.java.net/browse/JDK-8191785 >> >> 8191786 Thread-SMR hash table size should be dynamic >> https://bugs.openjdk.java.net/browse/JDK-8191786 >> >> 8191787 move private inline functions from thread.inline.hpp -> >> thread.cpp >> https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> >> threadSMR.[ch]pp >> https://bugs.openjdk.java.net/browse/JDK-8191789 >> >> 8191798 redo nested ThreadsListHandle to drop Threads_lock >> https://bugs.openjdk.java.net/browse/JDK-8191789 >> >> I have undoubtedly missed at least one future idea that someone had >> and for that my apologies... >> >> Thanks again to everyone who contributed to the Thread-SMR project!! >> >> Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for >> their rigorous review of the design that we documented on the wiki. >> Thanks for keeping us honest! >> >> I also have to thank my co-contributor Erik ?sterlund for starting the >> ball rolling on this crazy project with his presentation on Safe Memory >> Reclamation, for the initial prototype and for all your patches. Erik, >> hopefully we got far enough in your crazy vision... :-) >> >> Thanks to my co-contributor Robbin Ehn for proposing that we do early >> code reviews in the form of patches. Don't like something? Fix it with >> a patch and a new test if appropriate!! A pretty cool way to work that >> was new to me. >> >> Finally, many thanks to Jerry Thornbrugh for all the early code reviews, >> whiteboard chats and phone calls. Thanks for asking the right questions >> and pointing out some places where our logic was faulty. Also thanks for >> keeping me sane when I was literally on my own in Monument, CO. >> >> Dan >> >> >> On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> *** We are wrapping up code review on this project so it is time *** >>> *** for the various code reviewers to chime in one last time. *** >>> >>> In this latest round, we had three different reviewers chime in so we're >>> doing delta webrevs for each of those resolutions: >>> >>> David H's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >>> >>> >>> ? mq comment for dholmes_CR: >>> ??? - Fix indents, trailing spaces and various typos. >>> ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, >>> ????? add impl notes to document the type choices. >>> >>> ? src/hotspot/share/runtime/globals.hpp change is white-space only so it >>> ? is only visible via the file's patch link. >>> >>> >>> Robin W's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >>> >>> >>> ? mq comment for robinw_CR: >>> ??? - Fix some inefficient code, update some comments, fix some indents, >>> ????? and add some 'const' specifiers. >>> >>> ? src/hotspot/share/runtime/vm_operations.hpp change is white-space >>> only so >>> ? it is only visible via the file's patch link. >>> >>> >>> Coleen's resolutions: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >>> >>> >>> ? mq comment for coleenp_CR: >>> ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >>> ??? - add header comment to threadSMR.hpp file, >>> ??? - cleanup JavaThreadIteratorWithHandle ctr, >>> ??? - make ErrorHandling more efficient. >>> >>> >>> Some folks prefer to see all of the delta webrevs together so here is >>> a delta webrev relative to jdk10-09-full: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >>> >>> >>> ? src/hotspot/share/runtime/globals.hpp and >>> ? src/hotspot/share/runtime/vm_operations.hpp changes are white-space >>> only >>> ? so they are only visible via the file's patch link. >>> >>> >>> And, of course, some folks prefer to see everything all at once so here >>> is the latest full webrev: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >>> >>> >>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >>> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >>> >>> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >>> so there will be some catching up to do before integration... >>> >>> We welcome comments, suggestions and feedback. >>> >>> Dan, Erik and Robbin >>> >>> >>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Testing of the last round of changes revealed a hang in one of the new >>>> TLH tests. Robbin has fixed the hang, updated the existing TLH test, >>>> and >>>> added another TLH test for good measure. >>>> >>>> Here is the updated full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>> >>>> Here is the updated delta webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>> >>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>> no unexpected failures. >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Robbin rebased the project last night/this morning to merge with >>>>> Thread >>>>> Local Handshakes (TLH) and also picked up additional changesets up >>>>> thru: >>>>> >>>>>> Changeset: fa736014cf28 >>>>>> Author:??? cjplummer >>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>> >>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>> within hotspot source >>>>>> Summary: added pns2() to debug.cpp >>>>>> Reviewed-by: stuefe, gthornbr >>>>> >>>>> This is the first time we've rebased the project to bits that are this >>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>> think >>>>> we're done with this project and are in the final review-change-retest >>>>> cycle(s)... In other words, we're not planning on making any more >>>>> major >>>>> changes for this project... :-) >>>>> >>>>> *** Time for code reviewers to chime in on this thread! *** >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>> >>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>> >>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>> (and the new baseline also)... >>>>> >>>>> We're expecting this round to be a little noisier than usual because >>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>> through Jesper's usual care-and-feeding of integration_blockers, etc. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>> (Yes, we're playing chase-the-repo...) >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>> >>>>>> Unlike the previous rebase, there were no changes required to the >>>>>> open code to get the rebased bits to build so there is no delta >>>>>> webrev. >>>>>> >>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>> Mach5 tier[1-5] test run... >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>> >>>>>>> Here are the updated webrevs: >>>>>>> >>>>>>> Here's the mq comment for the change: >>>>>>> >>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>> >>>>>>> Here is the full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>> >>>>>>> And here is the delta webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>> >>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>> holiday >>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>> closer >>>>>>> look today. >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>> and Coleen) >>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>> done as a >>>>>>>> standalone patch and corresponding webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>> to use >>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>> >>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>> >>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>> >>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>> describe >>>>>>>>> and track the work on this project: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan has noticed that the indenting is wrong in some of the code >>>>>>>>> quotes >>>>>>>>> in the PDF that are not present in the internal wiki. We don't >>>>>>>>> have a >>>>>>>>> solution for that problem yet. >>>>>>>>> >>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>> >>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>> tier[2-5] >>>>>>>>> testing, additional stress testing on Dan's Solaris X64 server, >>>>>>>>> and >>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Daniel Daugherty >>>>>>>>> Erik Osterlund >>>>>>>>> Robbin Ehn >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> From thomas.stuefe at gmail.com Tue Nov 28 08:08:41 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 28 Nov 2017 09:08:41 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: Updated webrev with Andrews request worked in: http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show-registers-on-assert/webrev.01/webrev/ The only difference to before is that I restored the ShouldNotReachHere() at the end of VMError::controlled_crash(). Before that assert, I added a hopefully clearer message than before to enable my test to scan for this message. I also changed the jtreg test to scan for this new message. Note: in the jtreg test, I test that assert supression works with -XX:SupressErrorAt. But I just give it the file (-XX:SuppressErrorAt=/vmError.cpp) and omit the line number, because I do not want the test to stop working when someone adds lines in front of VMError::controlled_crash(). But that means that the ShouldNotReachHere() at the end of controlled_crash() does not fire either, because now all asserts in vmError.cpp are disabled. So, I need this new message. Thanks, Thomas On Mon, Nov 27, 2017 at 11:33 AM, Thomas St?fe wrote: > Ping... Could I please have reviews? Thank you! > ..Thomas > > On Wed, Nov 22, 2017 at 7:01 PM, Thomas St?fe > wrote: > >> Hi all, >> >> may I please have reviews for the following enhancement: >> >> issue: https://bugs.openjdk.java.net/browse/JDK-8191101 >> webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8191101- >> show-registers-on-assert/webrev.00/webrev/ >> >> (Patch looks big but lot of it is os_cpu fluff.) >> >> Prior email discussion at: http://mail.openjdk.java.net/p >> ipermail/hotspot-runtime-dev/2017-October/025018.html >> >> ---- >> >> Basically, this adds the ability to show register values in assert >> situations into the error report. This can be useful in certain corner >> cases, e.g. if you want to know the value of some local variables or how >> deep your stack currently runs. >> >> This works by triggering a fault right when an assert happens and >> squirreling the context away for later error reporting. >> >> When an assert happens, we touch a poison page. To preserve the context >> as best as possible, we want to avoid running too much code after the >> assert condition has been evaluated, so this is done in a very simple way: >> directly in the assert macro, right after the condition is evaluated, we >> dereference the content of a global poison page pointer. In my tests, even >> with slowdebug, this only spoils one register, rax on x64. >> >> In the signal handler, we recognize the assertion poison fault by the >> faulting address. We disable the poison page and store the ucontext away. >> Then we just return from signal handling. Poison page is now disarmed, the >> load from it is retried and now goes through. Normal assertion handling is >> then resumed - so, things like -XX:SuppressAt are unaffected and work fine. >> >> Then, when an error report is generated due to this assert, we now also >> use the stored context. So now we get registers and instructions at the >> assert point. >> >> Right now, this is implemented on (non-zero) Linux, though other posix >> platforms should be no problem. Have not yet thought deeply about windows. >> It is tested and - in debug - switched on by default on linux x86, ppc, >> s390. >> >> If implemented, it can be switched on and off with >> -XX:+ShowRegistersOnAssert. This is a failsafe, in case the mechanism does >> not work and we want to have clean asserts. >> >> To test this, do a java -XX:ErrorHandlerTest=1 -XX:+ShowRegistersOnAssert >> with a not-product VM. On Linux x64, ppc and s390 we should now see the >> register output in the hs-err file: >> >> 4 # Internal Error (/shared/projects/openjdk/jdk- >> hs/source/src/hotspot/share/utilities/vmError.cpp:1660), pid=4022, >> tid=4023 >> 5 # assert(str == NULL) failed: expected null >> 6 # >> 7 # >> ... >> 59 Registers: >> 60 RAX=0x00007f736c8f7000, RBX=0x0000000000000000, >> RCX=0x0000000000000000, RDX=0xffffffffffb5ded4 >> 61 RSP=0x00007f736c8d8ce0, RBP=0x00007f736c8d8d30, >> RSI=0x0000000000000001, RDI=0x0000000000000001 >> 62 R8 =0x0000000000000040, R9 =0x0000000000000001, >> R10=0x0000000000efb028, R11=0x00040788a244f42a >> 63 R12=0x0000000000000000, R13=0x00007fff3de46dbf, >> R14=0x00007f736c8d99c0, R15=0x0000000000000000 >> 64 RIP=0x00007f736aec5529, EFLAGS=0x0000000000010202, >> CSGSFS=0x002b000000000033, ERR=0x0000000000000006 >> 65 TRAPNO=0x000000000000000e >> ... >> 72 >> 73 Instructions: (pc=0x00007f736aec5529) >> 74 0x00007f736aec5509: 8d 05 31 21 4a 00 48 01 d0 ff e0 48 83 7d c8 00 >> 75 0x00007f736aec5519: 0f 84 7f 03 00 00 48 8d 05 82 fc b1 00 48 8b 00 >> 76 0x00007f736aec5529: c6 00 58 e8 cb 17 62 ff 84 c0 74 11 48 8d 3d 2f >> 77 0x00007f736aec5539: 1f 4a 00 b8 00 00 00 00 e8 ed 17 62 ff 48 8d 0d >> 78 >> >> --- >> >> Implementation notes: >> >> - when handling the poison fault, we need to copy the context away from >> the signal handler stack. For posix, this means copying the ucontext_t. >> This is undefined territory. On most platforms, this simply means copying >> the ucontex_t as a flat structure. On some platforms more is needed, e.g. >> on linux ppc, we need to patch up the context after copying (the context is >> not position independent), and on MacOS, the context is not self-contained >> but contains pointers to sub structures which need to be copied too and >> whose size is unknown at compile time. Because of these platform >> dependencies, I factored out the copying of ucontext_t to >> os::Posix::copy_ucontext and its implementations are os_cpu specific. >> >> - As an added precaution, when copying the context, we use a safe version >> of memcpy (os::safe_memcpy) which I added to copy from potentially invalid >> memory regions. The reason is that we have seen on some Unices - e.g. hpux >> - that the size of the ucontext_t structure at runtime may be different >> from the build machine, so we tread carefully. os::safe_memcpy() uses >> SafeFetch to copy a range of memory. >> >> Risk: >> >> If this does not work, asserts will become segfaults, which can be >> confusing. But the feature can be disabled with -XX:+ShowRegistersOnAssert >> and for now on most platforms is disabled by default. >> >> --- >> >> Limitations: >> - as it is now implemented, this is a one-shot mechanism and only works >> for the first assert. >> - -XX:SuppressAt=... is not affected and works fine. However, if the >> first assert is suppressed, follow-up asserts will not show register values. >> - When multiple threads run into an assert, we may or may not see >> register values depending on which thread is the first of finishing the >> poison page handler. >> >> I do not think these limitations are severe. They can be solved, but at >> the cost of added complexity, which I preferred not to add. >> >> Thank you, >> >> Thomas >> >> >> >> > From volker.simonis at gmail.com Tue Nov 28 09:42:41 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Tue, 28 Nov 2017 10:42:41 +0100 Subject: RFR(L) JDK-8188791 Move AppCDS from closed repo to open repo In-Reply-To: <4c4c3846-908c-5ba8-07f5-a283025c3dd8@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> <3de1ce17-6ade-0bcd-bc97-850c348bbeca@oracle.com> <02a094d0-037b-6b04-d14a-b743ad3c06d3@bell-sw.com> <0773ab98-096e-75e8-0c7f-f3a2b59a1541@samersoff.net> <4c993be4-2b4d-7b1e-1463-caa923bf9986@oracle.com> <6060657c-36fc-1126-ea17-4cbe0a479892@bell-sw.com> <4c4c3846-908c-5ba8-07f5-a283025c3dd8@oracle.com> Message-ID: On Mon, Nov 27, 2017 at 9:37 PM, Ioi Lam wrote: > Hi Volker & Dmitry, > > Thanks for testing on the additional platforms. It really is great help! > > We had a very limited set of supported platforms for the custom loaders due > to lack of testing resources. I think now we can expand the set of > platforms. However, I would like to keep JDK-8188791 as simple as possible > -- just moving the code, but don't introduce other changes. So I've created > > https://bugs.openjdk.java.net/browse/JDK-8191927 > > Enable AppCDS for custom loaders on all 64-bit Linux and AIX > > We can try to push that after 8191927 is pushed (which should happen very > soon once the JEP is targeted to JDK 10, hopefully today). > Thanks for opening the bug. I like your changes to it which introduce "@requires vm.cds.custom.loaders" to simplify the logic: http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ After you've pushed 8188791 please send out a RFR for 8191927. I can review it so we can push it in time before jdk10 will be branched off. Regards, Volker > I've posted the latest webrev at > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ > > It's based on this changeset > > changeset: 48006:235a18d659fc > user: roland > date: Mon Nov 27 10:44:19 2017 -0800 > files: src/hotspot/share/opto/split_if.cpp > test/hotspot/jtreg/compiler/loopopts/TestSplitIfPinnedCMove.java > description: > 8191153: assert(u_ctrl != blk1 && u_ctrl != blk2) failed: won't converge > Summary: relax assert > Reviewed-by: kvn > > This version also fixes the issue with -XX:+PrintSystemDictionaryAtExit on > product builds with > the two tests DumpClassList.java and ProhibitedPackage.java. > > Thanks > > - Ioi > > > > > On 11/27/17 11:15 AM, Dmitry Samersoff wrote: >> >> Volker, >> >>> @Dmitry: can you please re-build and re-run the tests on 64-bit ARM >>> just to make sure everything is working? Notice, that with my add-ons >>> about 20 more tests than before will be executed, but that was no >>> problem an the other Linux platforms I've tested. >> >> >> I'll do it. >> >> -Dmitry >> >> >> On 11/27/2017 09:34 PM, Volker Simonis wrote: >>> >>> Hi Ioi, Dmitry, >>> >>> I've finished my testing on Linux/ppc64, Linux/ppc64le, Linux/s390 and >>> AIX. All the tests passed successfully (with the corresponding extra >>> ppc/s390 patches currently under review). >>> >>> I did run the tests with the following add-on to your change, which >>> enables AppCDS with custom class loaders on all 64-bit Linux, Solaris >>> and AIX platforms: >>> >>> >>> http://cr.openjdk.java.net/~simonis/webrevs/2017/8188791-open-appcds.v02.tests-addon/ >>> >>> The only change required in HotSpot is in "classListParser.cpp" to >>> enable the feature on the named platforms. The other changes just >>> enable the additional tests on all 64-bit Linux, Solaris and AIX >>> platfroms. >>> >>> @Ioi: can you please append this add-on to your change? >>> >>> @Dmitry: can you please re-build and re-run the tests on 64-bit ARM >>> just to make sure everything is working? Notice, that with my add-ons >>> about 20 more tests than before will be executed, but that was no >>> problem an the other Linux platforms I've tested. >>> >>> Thank you and best regards, >>> Volker >>> >>> >>> On Fri, Nov 24, 2017 at 5:08 PM, Dmitry Samersoff >>> wrote: >>>> >>>> Volker, >>>> >>>>> Currently I'm building and running some tests >>>>> and I hope that until Monday we may even extend the list of supported >>>>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>>>> that works as well, the check could be simplified to allow all the >>>>> 64-bit Linux platforms. >>>> >>>> Yes, I added similar patch for Linux/AARCH64 and it works. >>>> >>>> So we should simplify the check. >>>> >>>> -Dmitry >>>> >>>> On 11/24/2017 05:51 PM, Volker Simonis wrote: >>>>> >>>>> Hi Ioi, >>>>> >>>>> you patch is not applying any more after "8187118: Remove appending >>>>> -cp path to the boot class path at AppCDS dump time" was pushed to >>>>> jdk/hs because of conflicts in classLoaderExt.hpp. >>>>> >>>>> Could you please rebase your change upon the newest version of jdk/hs ? >>>>> >>>>> By the way, on ppc64 all the tests passed after I found and fixed a >>>>> trivial bug (I've opened "JDK-8191770: [ppc64] Fix CDS: don't rewrite >>>>> invokefinal if DumpSharedSpaces" for it). >>>>> >>>>> Can you please explain why AppCDS with custom class loaders is >>>>> currently restricted to Linux/x86_64 and 64-bit Solaris? >>>>> >>>>> With the following small patch: >>>>> >>>>> diff -r 23deffd3a9c4 src/hotspot/share/classfile/classListParser.cpp >>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>>>> 15:37:19 2017 +0100 >>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Fri Nov 24 >>>>> 15:38:21 2017 +0100 >>>>> @@ -272,6 +272,7 @@ >>>>> // during archive dumping. >>>>> InstanceKlass* ClassListParser::load_class_from_source(Symbol* >>>>> class_name, TRAPS) { >>>>> #if !((defined(LINUX) && defined(X86) && defined(_LP64)) || \ >>>>> + (defined(PPC) && defined(_LP64)) || \ >>>>> (defined(SOLARIS) && defined(_LP64))) >>>>> // The only supported platforms are: (1) Linux/AMD64; (2) >>>>> Solaris/64-bit >>>>> error("AppCDS custom class loaders not supported on this >>>>> platform"); >>>>> diff -r 23deffd3a9c4 >>>>> >>>>> test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>>> --- >>>>> a/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>>> Fri Nov 24 15:37:19 2017 +0100 >>>>> +++ >>>>> b/test/hotspot/jtreg/runtime/appcds/customLoader/UnsupportedPlatforms.java >>>>> Fri Nov 24 15:38:21 2017 +0100 >>>>> @@ -56,6 +56,7 @@ >>>>> OutputAnalyzer out = TestCommon.dump(appJar, classlist); >>>>> >>>>> if ((Platform.isSolaris() && Platform.is64bit()) || >>>>> + (Platform.isLinux() && Platform.isPPC()) || >>>>> (Platform.isLinux() && Platform.isX64())) { >>>>> out.shouldNotContain(PLATFORM_NOT_SUPPORTED_WARNING); >>>>> out.shouldHaveExitValue(0); >>>>> >>>>> >>>>> I could at least pass all the relevant jtreg tests on Linux/ppc64. So >>>>> if there are no other hidden reasons I'd kindly ask you to add this >>>>> patch to you change. I'm currently testing on AIX and I hope the tests >>>>> will also succeed. If this will be the case, the list in the >>>>> aforementioned patch could also be extended by AIX. >>>>> >>>>> Finally I've also did some small fixes on s390 (opened "8191863: >>>>> [s390] Fix CDS: some bytecode rewriting doesn't depend on >>>>> RewriteControl" for it). Currently I'm building and running some tests >>>>> and I hope that until Monday we may even extend the list of supported >>>>> platforms by Linux/s390. I'm not sure about AArch (Dmitry?), but if >>>>> that works as well, the check could be simplified to allow all the >>>>> 64-bit Linux platforms. >>>>> >>>>> Thank you and best regards, >>>>> Volker >>>>> >>>>> On Fri, Nov 17, 2017 at 4:51 PM, Ioi Lam wrote: >>>>>> >>>>>> Hi Volker, >>>>>> >>>>>> Thanks for trying the patch out. >>>>>> >>>>>> >>>>>> On 11/17/17 5:42 AM, Volker Simonis wrote: >>>>>>> >>>>>>> Hi Ioi, >>>>>>> >>>>>>> I'm just trying to verify if AppCDS will be running on our >>>>>>> ppc/s390/AIX >>>>>>> ports. >>>>>>> >>>>>>> I applied your patch to the current jdk-hs repo as of today and first >>>>>>> of all I see the same compilation problems already reported by Dmitry >>>>>>> before. Can you please update you webrev with the corresponding >>>>>>> changes or create a new one. That will help other who wish to look at >>>>>>> this change. >>>>>> >>>>>> >>>>>> I've updated the patch inside the webrev. You can download it from >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch >>>>>> >>>>>> >>>>>> The previous patch (which you had the problems with) is still >>>>>> available as >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/open.patch.old >>>>>> >>>>>> I will post a new webrev, but it's big and I might be running out of >>>>>> quota >>>>>> from cr.openjdk.java.net, so I need to figure out how to handle that. >>>>>> >>>>>> The jdk/hs changeset corresponding to the new patch is the following: >>>>>> >>>>>> changeset: 47860:564882d918d4 >>>>>> user: zgu >>>>>> date: Thu Nov 16 20:21:11 2017 -0500 >>>>>> files: src/hotspot/share/services/memTracker.cpp >>>>>> description: >>>>>> 8190357: NMT: Include metadata information in NMT final report when >>>>>> PrintNMTStatistics is on >>>>>> Summary: Include metadata information in NMT final report >>>>>> Reviewed-by: adinn, stuefe >>>>>> >>>>>>> After building, I ran the appcds jtreg tests (i.e. >>>>>>> test/hotspot/jtreg/runtime/appcds/) under Linux/x86_64 and again I >>>>>>> see >>>>>>> the same four failures like Dmitry. >>>>>> >>>>>> Those are also due to merge issues. With the updated patch, I passed >>>>>> all >>>>>> tests under on Linux/x64: >>>>>> >>>>>> test/hotspot/jtreg/runtime/SharedArchiveFile >>>>>> test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>>>> test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>>>> test/hotspot/jtreg/runtime/appcds >>>>>> >>>>>> $ jtreg -J-Djavatest.maxOutputSize=1000000 -conc:28 >>>>>> -testjdk:/home/iklam/jdk/bld/opencdso-fastdebug/images/jdk >>>>>> -compilejdk:/home/iklam/jdk/bld/opencds/images/jdk -verbose:2 >>>>>> -timeout:4.0 >>>>>> -retain -agentvm >>>>>> >>>>>> -exclude:/home/iklam/jdk/opencds/closed/test/hotspot/jtreg/ProblemList.txt >>>>>> >>>>>> -exclude:/home/iklam/jdk/opencds/open/test/hotspot/jtreg/ProblemList.txt >>>>>> >>>>>> -nativepath:/home/iklam/jdk/bld/opencdso/images/jdk/../../images/test/hotspot/jtreg/native >>>>>> -nr -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ >>>>>> open/test/hotspot/jtreg/runtime/SharedArchiveFile >>>>>> open/test/hotspot/jtreg/runtime/CDSCompressedKPtrs >>>>>> >>>>>> open/test/hotspot/jtreg/runtime/modules/PatchModule/PatchModuleCDS.java >>>>>> open/test/hotspot/jtreg/runtime/appcds >>>>>> [...] >>>>>> Test results: passed: 128 >>>>>> Results written to /jdk/tmp/jtreg/work >>>>>> >>>>>> >>>>>>> Is that expected? I see a lot more errors on Linux/ppc64le but before >>>>>>> going into more detail I'd like to know what is the correct baseline >>>>>>> on which we can rely. >>>>>>> >>>>>>> Finally, I saw that with a product build, I get two additional test >>>>>>> failures: >>>>>>> >>>>>>> runtime/appcds/ProhibitedPackage.java >>>>>>> runtime/appcds/DumpClassList.java >>>>>>> >>>>>>> because of: >>>>>>> >>>>>>> Error: VM option 'PrintSystemDictionaryAtExit' is notproduct and is >>>>>>> available only in debug version of VM. >>>>>>> >>>>>>> So these two tests should probably be adjusted to only run for >>>>>>> slow/fastdebug builds. >>>>>> >>>>>> >>>>>> You're correct. I will exclude those tests from product builds, and >>>>>> file an >>>>>> RFE to fix them later. >>>>>> >>>>>> Thanks >>>>>> - Ioi >>>>>> >>>>>>> Thank you and best regards, >>>>>>> Volker >>>>>>> >>>>>>> >>>>>>> On Thu, Nov 16, 2017 at 12:22 PM, Dmitry Samersoff >>>>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> Ioi, >>>>>>>> >>>>>>>> Thank you! >>>>>>>> >>>>>>>> I did some additional testing and find that four tests >>>>>>>> fails the same way on both x86_64 and AARCH64 (see below). >>>>>>>> >>>>>>>> runtime/appcds/VerifierTest_0.java >>>>>>>> runtime/appcds/cacheObject/GCStressTest.java >>>>>>>> runtime/appcds/cacheObject/RedefineClassTest.java >>>>>>>> runtime/appcds/sharedStrings/InternSharedString.java >>>>>>>> >>>>>>>> Is it expected? >>>>>>>> >>>>>>>> test result: Failed. Execution failed: `main' threw exception: >>>>>>>> java.lang.RuntimeException: 'Please remove the unverifiable cl >>>>>>>> asses' missing from stdout/stderr >>>>>>>> >>>>>>>> Exception in thread "main" java.lang.RuntimeException: FAILED. >>>>>>>> GCStressApp_nonshared_string should not be shared >>>>>>>> at GCStressApp.main(GCStressApp.java:73) >>>>>>>> >>>>>>>> >>>>>>>> FAILED. buzzshould not be shared >>>>>>>> >>>>>>>> Exception in thread "main" java.lang.RuntimeException: Failed: >>>>>>>> unexpected shared string. >>>>>>>> at InternStringTest.main(InternStringTest.java:63) >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >>>>>>>> On 16.11.2017 03:10, Ioi Lam wrote: >>>>>>>>> >>>>>>>>> I filed: >>>>>>>>> >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191375 Add high-level >>>>>>>>> jtreg >>>>>>>>> VMProps to filter out CDS tests >>>>>>>>> >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8191374 Improve error >>>>>>>>> message >>>>>>>>> when CDS is not supported >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> - Ioi >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/15/17 4:01 PM, Ioi Lam wrote: >>>>>>>>>> >>>>>>>>>> Hi Dmitry, >>>>>>>>>> >>>>>>>>>> Thanks for the review. >>>>>>>>>> >>>>>>>>>> On 11/6/17 7:07 AM, Dmitry Samersoff wrote: >>>>>>>>>> >>>>>>>>>>> Ioi, >>>>>>>>>>> >>>>>>>>>>> I tested new patch on aarch64 and it works fine with the changes >>>>>>>>>>> below[1]. >>>>>>>>>> >>>>>>>>>> I've fixed it. Thanks for the patch. >>>>>>>>>>> >>>>>>>>>>> Also tests doesn't work with exploded image - is it possible to >>>>>>>>>>> check >>>>>>>>>>> for this case and produce appropriate error message? >>>>>>>>>> >>>>>>>>>> CDS is not supported on exploded images. I think the best thing to >>>>>>>>>> do >>>>>>>>>> is to add something like "@requires vm.cds" into the test cases >>>>>>>>>> (similar to the use of "@require vm.aot" in other tests). That >>>>>>>>>> way, >>>>>>>>>> none of the CDS tests will be executed for >>>>>>>>>> >>>>>>>>>> It's too late to do this in JDK 10 now, but I will file an RFE to >>>>>>>>>> do >>>>>>>>>> it in the next release. >>>>>>>>>> >>>>>>>>>> I'll also file another RFE to print a better message. Today if you >>>>>>>>>> use >>>>>>>>>> an exploded build the error message is kind of confusing: >>>>>>>>>> >>>>>>>>>> $ ./jdk/bin/java -Xshare:dump >>>>>>>>>> Error: non-empty directory '/jdk/bld/cons/jdk/modules/java.base' >>>>>>>>>> Hint: enable -Xlog:class+path=info to diagnose the failure >>>>>>>>>> Error occurred during initialization of VM >>>>>>>>>> CDS allows only empty directories in archived classpaths >>>>>>>>>> >>>>>>>>>> It should say something like "CDS is not supported on exploded >>>>>>>>>> images". >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> - Ioi >>>>>>>>>> >>>>>>>>>>> 1. >>>>>>>>>>> diff -r dbf9cec6a568 >>>>>>>>>>> src/hotspot/share/classfile/classListParser.cpp >>>>>>>>>>> --- a/src/hotspot/share/classfile/classListParser.cpp Mon Nov >>>>>>>>>>> 06 >>>>>>>>>>> 09:45:23 2017 +0000 >>>>>>>>>>> +++ b/src/hotspot/share/classfile/classListParser.cpp Mon Nov >>>>>>>>>>> 06 >>>>>>>>>>> 15:02:51 2017 +0000 >>>>>>>>>>> @@ -31,7 +31,6 @@ >>>>>>>>>>> -#include "prims/jvm.h" >>>>>>>>>>> >>>>>>>>>>> diff -r dbf9cec6a568 >>>>>>>>>>> src/hotspot/share/classfile/systemDictionaryShared.cpp >>>>>>>>>>> --- a/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon >>>>>>>>>>> Nov >>>>>>>>>>> 06 09:45:23 2017 +0000 >>>>>>>>>>> +++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp Mon >>>>>>>>>>> Nov >>>>>>>>>>> 06 15:02:51 2017 +0000 >>>>>>>>>>> @@ -518,7 +518,7 @@ >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> MutexLocker mu(SystemDictionary_lock, THREAD); >>>>>>>>>>> - Klass* check = find_class(d_index, d_hash, name, >>>>>>>>>>> dictionary); >>>>>>>>>>> + Klass* check = dictionary->find_class(d_index, d_hash, >>>>>>>>>>> name); >>>>>>>>>>> if (check != NULL) { >>>>>>>>>>> return InstanceKlass::cast(check); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>>>>>>>>>> On 11/01/2017 05:43 AM, Ioi Lam wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Here's the new webrev for both the AppCDS implementation and >>>>>>>>>>>> tests. >>>>>>>>>>>> During internal review of the JEP, we have decided to integrate >>>>>>>>>>>> both >>>>>>>>>>>> implementation and tests at the same time. >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v02/ >>>>>>>>>>>> >>>>>>>>>>>> As mentioned before, most of the "diffs" shown in this webrev >>>>>>>>>>>> are the >>>>>>>>>>>> result of copying the closed source files on top of files of the >>>>>>>>>>>> same >>>>>>>>>>>> name in the open repo. So in reviewing, instead of focusing on >>>>>>>>>>>> what's >>>>>>>>>>>> "changed", please focus on the entire content of the new version >>>>>>>>>>>> of >>>>>>>>>>>> each >>>>>>>>>>>> file. >>>>>>>>>>>> >>>>>>>>>>>> Testing: I did an OpenJDK linux-x64 build (without Oracle closed >>>>>>>>>>>> sources) and all the new appcds tests passed. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> - Ioi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 10/30/17 8:52 AM, Ioi Lam wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Dmitry, >>>>>>>>>>>>> >>>>>>>>>>>>> In the latest JDK 10 repo, is_jrt has been renamed to >>>>>>>>>>>>> is_modules_image. Please change the code accordingly. >>>>>>>>>>>>> >>>>>>>>>>>>> I will post my latest diff soon, with some test cases as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> - Ioi >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ioi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd tried to apply your patch to latest open JDK10 and >>>>>>>>>>>>>> the compilation fails with: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> error: ?class SharedClassPathEntry? has no member named >>>>>>>>>>>>>> ?is_jrt? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Did I miss something? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Dmitry >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 13.10.2017 02:48, Ioi Lam wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please review this change set. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8188791 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is the first step of implementing the following JEP, >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>> moves >>>>>>>>>>>>>>> AppCDS from >>>>>>>>>>>>>>> closed repos into the openjdk repo: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185996 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In JDK 9, significant portion of AppCDS code resided in the >>>>>>>>>>>>>>> closed >>>>>>>>>>>>>>> repo. >>>>>>>>>>>>>>> As part >>>>>>>>>>>>>>> of the open-sourcing effort of JDK 18.3, we will move the >>>>>>>>>>>>>>> source >>>>>>>>>>>>>>> code >>>>>>>>>>>>>>> into the >>>>>>>>>>>>>>> open repo. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In this changeset, the code is moved verbatim as much as >>>>>>>>>>>>>>> possible. The >>>>>>>>>>>>>>> intention is >>>>>>>>>>>>>>> only to relocate the sources, not to changing existing >>>>>>>>>>>>>>> behaviors, >>>>>>>>>>>>>>> and not >>>>>>>>>>>>>>> to do any sort of refactoring. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Most of the "diffs" shown in this webrev are the result of >>>>>>>>>>>>>>> copying the >>>>>>>>>>>>>>> closed source >>>>>>>>>>>>>>> files on top of files of the same name in the open repo. So >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>> reviewing, instead of >>>>>>>>>>>>>>> focusing on what's "changed", it's better to focus on the >>>>>>>>>>>>>>> entire >>>>>>>>>>>>>>> content >>>>>>>>>>>>>>> of the new >>>>>>>>>>>>>>> version of each file. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The only functional change in this task is that the UseAppCDS >>>>>>>>>>>>>>> flag is >>>>>>>>>>>>>>> changed from >>>>>>>>>>>>>>> a "commercial" flag to a regular "product" flag. This is >>>>>>>>>>>>>>> because >>>>>>>>>>>>>>> "commercial" >>>>>>>>>>>>>>> flags are not supported by the OpenJDK build. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Source code refactoring may be desirable, because the old >>>>>>>>>>>>>>> open/closed >>>>>>>>>>>>>>> source >>>>>>>>>>>>>>> code structure had introduced some intermediary APIs to >>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>> code >>>>>>>>>>>>>>> between >>>>>>>>>>>>>>> the two repos. Such API should be removed in a separate RFE. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also, some AppCDS tests are currently in the closed repo. >>>>>>>>>>>>>>> These >>>>>>>>>>>>>>> tests >>>>>>>>>>>>>>> will be >>>>>>>>>>>>>>> moved in a separate task. See JDK-8188792 for details. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> All the AppCDS tests (currently still in closed sources) >>>>>>>>>>>>>>> passed >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>> both Oracle JDK >>>>>>>>>>>>>>> and OpenJDK. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> - Ioi >>>>>>>> >>>>>>>> >>>> > From volker.simonis at gmail.com Tue Nov 28 10:03:04 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Tue, 28 Nov 2017 11:03:04 +0100 Subject: Future plans for AppCDS? In-Reply-To: References: Message-ID: Hi Mike, what you are actually asking for is support for AOT (Ahead Of Time compilation) [1]. It is already available in Java 9. Although the support for user code is still experimental you should definitely give it a try. Regards, Volker [1] http://openjdk.java.net/jeps/295 On Tue, Aug 4, 2015 at 8:23 PM, Mike Hearn wrote: > Hi there, > > I'm wondering if there are any plans to extend AppCDS in future to include > serializing the code cache to disk? > > I ask because I have a (not very complicated) desktop JavaFX app. It starts > in about three seconds, which isn't terrible, but I'd like it to be faster. > > Unfortunately I am frequently frustrated in this goal. Some testing shows > that one step of the initialisation sequence takes about a second normally. > If I put it in a loop to let it fully compile, that drops to more like a > quarter of a second. > > Likewise, my app loads a series of items to display them on the screen. The > first one can take a solid 700-800 msec. The rest are more like 10% of that. > > I tried parallelising some of the things done during startup, but it made > no difference. My theory is that the compile threads are taking up the > spare cores that could be doing startup tasks (it's a bit hard to profile > this though as the whole sequence only lasts a few seconds). > > AppCDS already knows how to serialise lots of HotSpot state to disk for > unchanging JARs. If it could store compiled method code as well, then I > could probably shave a second or two off app startup. > > I do understand that this situation is a bit rare for Java developers and > that due to extra disk IO etc, loading compiled code from disk might not > always be faster. But for devices that have an SSD it probably can be, > especially if HotSpot were to do the same trick Microsoft does and lay out > compiled methods on disk in the order they will be used. From adinn at redhat.com Tue Nov 28 10:29:28 2017 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 28 Nov 2017 10:29:28 +0000 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: <4e87dfe2-3438-4efa-5dfe-dbeb7ed9c4cb@redhat.com> On 28/11/17 08:08, Thomas St?fe wrote: > Updated webrev with Andrews request worked in: > > http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show-registers-on-assert/webrev.01/webrev/ > > The only difference to before is that I restored the > ShouldNotReachHere() at the end of VMError::controlled_crash(). Before > that assert, I added a hopefully clearer message than before to enable > my test to scan for this message. I also changed the jtreg test to scan > for this new message. Ok, thanks. That makes it clear that this point is only arrived at artificially. > Note: in the jtreg test, I test that assert supression works with > -XX:SupressErrorAt. But I just give it the file > (-XX:SuppressErrorAt=/vmError.cpp) and omit the line number,?because I > do not want the test to stop working when someone adds lines in front of > VMError::controlled_crash(). But that means that the > ShouldNotReachHere() at the end of controlled_crash() does not fire > either, because now all asserts in vmError.cpp are disabled. So, I need > this new message. Yes, and strictly this means the call to ShouldNotReachHere() is never actually to execute. However, it still serves in a declarative role so is good to keep it there. This version looks good. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From thomas.stuefe at gmail.com Tue Nov 28 10:32:46 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 28 Nov 2017 11:32:46 +0100 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: <4e87dfe2-3438-4efa-5dfe-dbeb7ed9c4cb@redhat.com> References: <4e87dfe2-3438-4efa-5dfe-dbeb7ed9c4cb@redhat.com> Message-ID: On Tue, Nov 28, 2017 at 11:29 AM, Andrew Dinn wrote: > On 28/11/17 08:08, Thomas St?fe wrote: > > Updated webrev with Andrews request worked in: > > > > http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show- > registers-on-assert/webrev.01/webrev/ > > > > The only difference to before is that I restored the > > ShouldNotReachHere() at the end of VMError::controlled_crash(). Before > > that assert, I added a hopefully clearer message than before to enable > > my test to scan for this message. I also changed the jtreg test to scan > > for this new message. > > Ok, thanks. That makes it clear that this point is only arrived at > artificially. > > > Note: in the jtreg test, I test that assert supression works with > > -XX:SupressErrorAt. But I just give it the file > > (-XX:SuppressErrorAt=/vmError.cpp) and omit the line number, because I > > do not want the test to stop working when someone adds lines in front of > > VMError::controlled_crash(). But that means that the > > ShouldNotReachHere() at the end of controlled_crash() does not fire > > either, because now all asserts in vmError.cpp are disabled. So, I need > > this new message. > Yes, and strictly this means the call to ShouldNotReachHere() is never > actually to execute. However, it still serves in a declarative role so > is good to keep it there. This version looks good. > > regards, > > > Andrew Dinn > Thanks, Andrew > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From erik.osterlund at oracle.com Tue Nov 28 11:22:12 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 12:22:12 +0100 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <2b566339-d3f8-27ab-c5c9-670aa278c9da@oracle.com> References: <5A1BF6F0.5030903@oracle.com> <88659101-1693-d5a4-a605-8f7b5ab53299@oracle.com> <2b566339-d3f8-27ab-c5c9-670aa278c9da@oracle.com> Message-ID: <5A1D46E4.7020801@oracle.com> Hi Serguei, Thanks for the review. /Erik On 2017-11-28 06:43, serguei.spitsyn at oracle.com wrote: > Hi Erik, > > This looks good to me. > > Thanks, > Serguei > > > On 11/27/17 05:22, Daniel D. Daugherty wrote: >> Adding serviceability-dev at ... since this is JVM/TI related... >> >> Dan >> >> >> On 11/27/17 6:28 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> The JVMTI tag hashmap has weak oop references that are handled using >>> raw oop accesses and a G1-specific SATB enqueue call when leaking >>> out objects from the tag map, requiring them to be marked as live by >>> G1. >>> >>> This should now be refactored to use the Access API to annotate that >>> these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use >>> ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8191894 >>> >>> Thanks, >>> /Erik >> > From erik.osterlund at oracle.com Tue Nov 28 12:19:54 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:19:54 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: References: <5A1BD5B3.7030808@oracle.com> Message-ID: <5A1D546A.1040601@oracle.com> Hi David, That is a good question. There is a description (written by a GC person, but still) for each decorator in the access.hpp file where the decorators are declared. I try to go in to some details there when you should and should not use a decorator. If we find that the text might have been biased from a GC perspective, we can update the comments. The previous situation has been to use naked accesses, and then suddenly a GC person jumps in moaning about the lack of explicit #if INCLUDE_ALL_GCS if (UseG1) SATB barrier goo, or manual insertion of post-write barriers if a CAS was successful. There was no good way of knowing you had to manually insert these barriers, what barriers were offered to shared code, or what their semantics were. Now at least you have a set of decorators to choose from, with some automatic verification if you select nonsense decorators, and documentation in one place what they are used for and what they mean. So if you do not know what you need, at least you can read the descriptions and hopefully figure it out, which was impossible before. Having said that, I believe things like Coleen's OopHandle and PhantomOopHandle will build a layer on top of Access with possibly a few less details exposed that are even simpler to use for the conservative non-GC person that just wants the thing to work as simple as possible and wants to stay as far away from GC land as possible, which is sane. And if you feel uncertain, you can always loop me in any time, and I will probably say "there's a decorator for that". I hope this answers your question. Thanks, /Erik On 2017-11-28 07:59, David Holmes wrote: > Hi Erik, > > Is there a non-GC person's guide to what the different forms of > RootAccess mean and when to use them? Or do we expect a GC person > to always jump in to show where such things are needed? ;-) > > Thanks, > David > > On 27/11/2017 7:06 PM, Erik ?sterlund wrote: >> Hi, >> >> The ClassLoaderData::remove_handle() member function currently uses >> explicit G1 SATB barriers to remove an oop from the root set, as >> these handles are not necessarily walked by the GC in a safepoint. >> Therefore G1 needs pre-write barriers. >> >> This should now be modeled as a >> RootAccess::oop_store instead. This maps to >> performing a pre-write SATB barrier with G1, but other GCs are free >> to do other things as necessary. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191888 >> >> Thanks, >> /Erik From erik.osterlund at oracle.com Tue Nov 28 12:27:28 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:27:28 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> References: <5A1C06D8.1040405@oracle.com> <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> Message-ID: <5A1D5630.8040201@oracle.com> Hi Coleen, On 2017-11-27 19:27, coleen.phillimore at oracle.com wrote: > > I think this is a nice simple fix and should be pushed for JDK10. Thank you for the review. /Erik > Thanks, > Coleen > > On 11/27/17 7:36 AM, Erik ?sterlund wrote: >> Hi, >> >> There is currently a bug when using unsafe accesses off-heap: >> >> 1) We write into a thread that we enable crash protection (using >> GuardUnsafeAccess): >> 2) We perform the access >> 3) We write into a thread that we disable crash protection (using >> ~GuardUnsafeAccess) >> >> The problem is that the crash protection stores are volatile, but the >> actual access is non-volatile. Compilers have different >> interpretation whether volatile - non-volatile accesses are allowed >> to reorder. MSVC is known to interpret such interactions as-if the >> volatile accesses have acquire/release semantics from the compiler >> point of view, and others such as GCC are known to reorder away freely. >> >> To prevent any issues, the accesses involved when using >> GuardUnsafeAccess should be at least volatile. >> This change makes the few remaining ones volatile. The JMM-volatile >> (SEQ_CST) accesses with crash protection already have stronger >> ordering than volatile and hence do not need changing. >> >> By making the address passed in to the Access API volatile, the >> MO_VOLATILE decorator is automatically set, which not surprisingly >> makes the access volatile. Therefore, the solution is to simply make >> the address passed in to Access volatile in this case. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8186787 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >> >> Thanks, >> /Erik > From erik.osterlund at oracle.com Tue Nov 28 12:28:16 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:28:16 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <4b890e2e-8cf7-2d2d-5373-d9144245b0ce@oracle.com> References: <5A1BD5B3.7030808@oracle.com> <4b890e2e-8cf7-2d2d-5373-d9144245b0ce@oracle.com> Message-ID: <5A1D5660.20907@oracle.com> Hi Coleen, Thank you for the review. /Erik On 2017-11-27 19:08, coleen.phillimore at oracle.com wrote: > > This looks good! > Coleen > > On 11/27/17 4:06 AM, Erik ?sterlund wrote: >> Hi, >> >> The ClassLoaderData::remove_handle() member function currently uses >> explicit G1 SATB barriers to remove an oop from the root set, as >> these handles are not necessarily walked by the GC in a safepoint. >> Therefore G1 needs pre-write barriers. >> >> This should now be modeled as a >> RootAccess::oop_store instead. This maps to >> performing a pre-write SATB barrier with G1, but other GCs are free >> to do other things as necessary. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191888 >> >> Thanks, >> /Erik > From daniel.daugherty at oracle.com Tue Nov 28 12:31:02 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 28 Nov 2017 07:31:02 -0500 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <157a90ee-35c9-1adf-490a-9910f3ee646f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <836b5c6c-5eef-9245-c16a-e71372093d9a@oracle.com> <82d0e473-a24f-9feb-82e9-7d04c4d7a91d@oracle.com> <354f7ece-66bc-c065-9ee8-f55abcc1c23e@oracle.com> <0ff8df8e-e93c-02d2-cb97-b13b1ac0f53f@oracle.com> <5365606e-4036-b5d7-9bac-e91e42f38d01@oracle.com> <567339a6-e36a-5416-5c34-2a87cbfa6cdd@oracle.com> <5976a6bf-c9b0-c0d5-d5f9-fcdf7dc9a1a8@oracle.com> <157a90ee-35c9-1adf-490a-9910f3ee646f@oracle.com> Message-ID: On 11/28/17 3:01 AM, David Holmes wrote: > Just for the record ... > > On 23/11/2017 6:20 PM, Robbin Ehn wrote: >> Thanks Dan for dragging this freight train to the docks, it's time to >> ship it! > > I agree. The latest delta seems fine to me. Thanks! > >> Created follow-up bug: >> 8191809: Threads::number_of_threads() is not 'MT-safe' >> https://bugs.openjdk.java.net/browse/JDK-8191809 > > Just updated that bug - I don't see any MT issues there. :) > > Cheers, > David > > PS. Dan: yes JPRT was still doing 32-bit ARM a "month or two back". > 64-bit atomics should not be a concern. That said I thought all the > atomic updates were done for ARM etc. The atomic template updates were done for ARM. It was the new template stuff that gave me the error about a match not being available for one of the operations on a 64-bit field. I'm going to guess that pre-template, we would have gotten a runtime error or something... I really wish I had saved that JPRT build log... :-( Dan > >> /Robbin >> >> On 2017-11-23 03:16, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> I've made minor tweaks to the Thread-SMR project based on the remaining >>> code review comments over the last couple of days. I've also rebased >>> the >>> project to jdk/hs bits as of mid-afternoon (my TZ) on 2017.11.22. I'm >>> running baseline Mach5 Tier[1-5] testing and prototype Mach5 Tier[1-5] >>> testing... >>> >>> Here's a delta webrev for the latest round of tweaks: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-11-delta >>> >>> >>> Here's the proposed commit message: >>> >>> $ cat SMR_prototype_10/open/commit.txt >>> 8167108: inconsistent handling of SR_lock can lead to crashes >>> Summary: Add Thread Safe Memory Reclamation (Thread-SMR) mechanism. >>> Reviewed-by: coleenp, dcubed, dholmes, eosterlund, gthornbr, >>> kbarrett, rehn, sspitsyn, stefank >>> Contributed-by: daniel.daugherty at oracle.com, >>> erik.osterlund at oracle.com, robbin.ehn at oracle.com >>> >>> I've gone through 880+ emails in my archive for this project and added >>> anyone that made a code review comment. Reviewers are not in my usual >>> by-comment-posting order; it was way too hard to figure that out... So >>> reviewers and contributors are simply in alpha-sort order... >>> >>> Here's the collection of follow-up bugs that I filed based on sifting >>> through the email archive: >>> >>> 8191784 JavaThread::collect_counters() should stop using Threads_lock >>> https://bugs.openjdk.java.net/browse/JDK-8191784 >>> >>> 8191785 revoke_bias() needs a new assertion >>> https://bugs.openjdk.java.net/browse/JDK-8191785 >>> >>> 8191786 Thread-SMR hash table size should be dynamic >>> https://bugs.openjdk.java.net/browse/JDK-8191786 >>> >>> 8191787 move private inline functions from thread.inline.hpp -> >>> thread.cpp >>> https://bugs.openjdk.java.net/browse/JDK-8191787 >>> >>> 8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> >>> threadSMR.[ch]pp >>> https://bugs.openjdk.java.net/browse/JDK-8191789 >>> >>> 8191798 redo nested ThreadsListHandle to drop Threads_lock >>> https://bugs.openjdk.java.net/browse/JDK-8191789 >>> >>> I have undoubtedly missed at least one future idea that someone had >>> and for that my apologies... >>> >>> Thanks again to everyone who contributed to the Thread-SMR project!! >>> >>> Special thanks goes to Karen Kinnear, Kim Barrett and David Holmes for >>> their rigorous review of the design that we documented on the wiki. >>> Thanks for keeping us honest! >>> >>> I also have to thank my co-contributor Erik ?sterlund for starting the >>> ball rolling on this crazy project with his presentation on Safe Memory >>> Reclamation, for the initial prototype and for all your patches. Erik, >>> hopefully we got far enough in your crazy vision... :-) >>> >>> Thanks to my co-contributor Robbin Ehn for proposing that we do early >>> code reviews in the form of patches. Don't like something? Fix it with >>> a patch and a new test if appropriate!! A pretty cool way to work that >>> was new to me. >>> >>> Finally, many thanks to Jerry Thornbrugh for all the early code >>> reviews, >>> whiteboard chats and phone calls. Thanks for asking the right questions >>> and pointing out some places where our logic was faulty. Also thanks >>> for >>> keeping me sane when I was literally on my own in Monument, CO. >>> >>> Dan >>> >>> >>> On 11/21/17 7:12 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> *** We are wrapping up code review on this project so it is time *** >>>> *** for the various code reviewers to chime in one last time. *** >>>> >>>> In this latest round, we had three different reviewers chime in so >>>> we're >>>> doing delta webrevs for each of those resolutions: >>>> >>>> David H's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.0-delta/ >>>> >>>> >>>> ? mq comment for dholmes_CR: >>>> ??? - Fix indents, trailing spaces and various typos. >>>> ??? - Add descriptions for the '_cnt', '_max' and '_times" fields, >>>> ????? add impl notes to document the type choices. >>>> >>>> ? src/hotspot/share/runtime/globals.hpp change is white-space only >>>> so it >>>> ? is only visible via the file's patch link. >>>> >>>> >>>> Robin W's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.0,10.10.1-delta/ >>>> >>>> >>>> ? mq comment for robinw_CR: >>>> ??? - Fix some inefficient code, update some comments, fix some >>>> indents, >>>> ????? and add some 'const' specifiers. >>>> >>>> ? src/hotspot/share/runtime/vm_operations.hpp change is white-space >>>> only so >>>> ? it is only visible via the file's patch link. >>>> >>>> >>>> Coleen's resolutions: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10.1,10.10.2-delta/ >>>> >>>> >>>> ? mq comment for coleenp_CR: >>>> ??? - Change ThreadsList::_threads from 'mtGC' -> 'mtThread', >>>> ??? - add header comment to threadSMR.hpp file, >>>> ??? - cleanup JavaThreadIteratorWithHandle ctr, >>>> ??? - make ErrorHandling more efficient. >>>> >>>> >>>> Some folks prefer to see all of the delta webrevs together so here is >>>> a delta webrev relative to jdk10-09-full: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.09,10.10.2-delta/ >>>> >>>> >>>> ? src/hotspot/share/runtime/globals.hpp and >>>> ? src/hotspot/share/runtime/vm_operations.hpp changes are >>>> white-space only >>>> ? so they are only visible via the file's patch link. >>>> >>>> >>>> And, of course, some folks prefer to see everything all at once so >>>> here >>>> is the latest full webrev: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-10.10-full/ >>>> >>>> >>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing. The >>>> prelim Mach5 tier1 (JPRT equivalent) on these bits passed just fine... >>>> >>>> The project is currently baselined on Jesper's 2017-11-17 PIT snapshot >>>> so there will be some catching up to do before integration... >>>> >>>> We welcome comments, suggestions and feedback. >>>> >>>> Dan, Erik and Robbin >>>> >>>> >>>> On 11/18/17 8:49 PM, Daniel D. Daugherty wrote: >>>>> Greetings, >>>>> >>>>> Testing of the last round of changes revealed a hang in one of the >>>>> new >>>>> TLH tests. Robbin has fixed the hang, updated the existing TLH >>>>> test, and >>>>> added another TLH test for good measure. >>>>> >>>>> Here is the updated full webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-full/ >>>>> >>>>> Here is the updated delta webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-09-delta/ >>>>> >>>>> Dan ran the bits thru the usual Mach5 tier[1-5] testing and there are >>>>> no unexpected failures. >>>>> >>>>> We welcome comments, suggestions and feedback. >>>>> >>>>> Dan, Erik and Robbin >>>>> >>>>> >>>>> On 11/15/17 3:32 PM, Daniel D. Daugherty wrote: >>>>>> Greetings, >>>>>> >>>>>> Robbin rebased the project last night/this morning to merge with >>>>>> Thread >>>>>> Local Handshakes (TLH) and also picked up additional changesets >>>>>> up thru: >>>>>> >>>>>>> Changeset: fa736014cf28 >>>>>>> Author:??? cjplummer >>>>>>> Date:????? 2017-11-14 18:08 -0800 >>>>>>> URL:http://hg.openjdk.java.net/jdk/hs/rev/fa736014cf28 >>>>>>> >>>>>>> 8191049: Add alternate version of pns() that is callable from >>>>>>> within hotspot source >>>>>>> Summary: added pns2() to debug.cpp >>>>>>> Reviewed-by: stuefe, gthornbr >>>>>> >>>>>> This is the first time we've rebased the project to bits that are >>>>>> this >>>>>> fresh (< 12 hours old at rebase time). We've done this because we >>>>>> think >>>>>> we're done with this project and are in the final >>>>>> review-change-retest >>>>>> cycle(s)... In other words, we're not planning on making any more >>>>>> major >>>>>> changes for this project... :-) >>>>>> >>>>>> *** Time for code reviewers to chime in on this thread! *** >>>>>> >>>>>> Here is the updated full webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-full/ >>>>>> >>>>>> Here's is the delta webrev from the 2017.11.10 rebase: >>>>>> >>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-08-delta/ >>>>>> >>>>>> Dan has submitted the bits for the usual Mach5 tier[1-5] testing >>>>>> (and the new baseline also)... >>>>>> >>>>>> We're expecting this round to be a little noisier than usual because >>>>>> we did not rebase on a PIT snapshot so the new baseline has not been >>>>>> through Jesper's usual care-and-feeding of integration_blockers, >>>>>> etc. >>>>>> >>>>>> We welcome comments, suggestions and feedback. >>>>>> >>>>>> Dan, Erik and Robbin >>>>>> >>>>>> >>>>>> On 11/14/17 4:48 PM, Daniel D. Daugherty wrote: >>>>>>> Greetings, >>>>>>> >>>>>>> I rebased the project to the 2017.11.10 jdk/hs PIT snapshot. >>>>>>> (Yes, we're playing chase-the-repo...) >>>>>>> >>>>>>> Here is the updated full webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-07-full/ >>>>>>> >>>>>>> Unlike the previous rebase, there were no changes required to the >>>>>>> open code to get the rebased bits to build so there is no delta >>>>>>> webrev. >>>>>>> >>>>>>> These bits have been run through JPRT and I've submitted the usual >>>>>>> Mach5 tier[1-5] test run... >>>>>>> >>>>>>> We welcome comments, suggestions and feedback. >>>>>>> >>>>>>> Dan, Erik and Robbin >>>>>>> >>>>>>> >>>>>>> On 11/13/17 12:30 PM, Daniel D. Daugherty wrote: >>>>>>>> Greetings, >>>>>>>> >>>>>>>> I rebased the project to the 2017.10.26 jdk10/hs PIT snapshot. >>>>>>>> >>>>>>>> Here are the updated webrevs: >>>>>>>> >>>>>>>> Here's the mq comment for the change: >>>>>>>> >>>>>>>> ? Rebase to 2017.10.25 PIT snapshot. >>>>>>>> >>>>>>>> Here is the full webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-full/ >>>>>>>> >>>>>>>> And here is the delta webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-06-delta/ >>>>>>>> >>>>>>>> I ran the above bits throught Mach5 tier[1-5] testing over the >>>>>>>> holiday >>>>>>>> weekend. Didn't see any issues in a quick look. Going to take a >>>>>>>> closer >>>>>>>> look today. >>>>>>>> >>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>> >>>>>>>> Dan, Erik and Robbin >>>>>>>> >>>>>>>> >>>>>>>> On 11/8/17 1:05 PM, Daniel D. Daugherty wrote: >>>>>>>>> Greetings, >>>>>>>>> >>>>>>>>> Resolving one of the code review comments (from both Stefan K >>>>>>>>> and Coleen) >>>>>>>>> on jdk10-04-full required quite a few changes so it is being >>>>>>>>> done as a >>>>>>>>> standalone patch and corresponding webrevs: >>>>>>>>> >>>>>>>>> Here's the mq comment for the change: >>>>>>>>> >>>>>>>>> ? stefank, coleenp CR - refactor most JavaThreadIterator usage >>>>>>>>> to use >>>>>>>>> ??? JavaThreadIteratorWithHandle. >>>>>>>>> >>>>>>>>> Here is the full webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-full/ >>>>>>>>> >>>>>>>>> And here is the delta webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-05-delta/ >>>>>>>>> >>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>> >>>>>>>>> Dan, Erik and Robbin >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>>>>>>>>> Greetings, >>>>>>>>>> >>>>>>>>>> We have a (eXtra Large) fix for the following bug: >>>>>>>>>> >>>>>>>>>> 8167108 inconsistent handling of SR_lock can lead to crashes >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>>>>>>>>> >>>>>>>>>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>>>>>>>>> Hazard Pointers to manage JavaThread lifecycle. >>>>>>>>>> >>>>>>>>>> Here's a PDF for the internal wiki that we've been using to >>>>>>>>>> describe >>>>>>>>>> and track the work on this project: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dan has noticed that the indenting is wrong in some of the >>>>>>>>>> code quotes >>>>>>>>>> in the PDF that are not present in the internal wiki. We >>>>>>>>>> don't have a >>>>>>>>>> solution for that problem yet. >>>>>>>>>> >>>>>>>>>> Here's the webrev for current JDK10 version of this fix: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>>>>>>>>> >>>>>>>>>> This fix has been run through many rounds of JPRT and Mach5 >>>>>>>>>> tier[2-5] >>>>>>>>>> testing, additional stress testing on Dan's Solaris X64 >>>>>>>>>> server, and >>>>>>>>>> additional testing on Erik and Robbin's machines. >>>>>>>>>> >>>>>>>>>> We welcome comments, suggestions and feedback. >>>>>>>>>> >>>>>>>>>> Daniel Daugherty >>>>>>>>>> Erik Osterlund >>>>>>>>>> Robbin Ehn >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> From erik.osterlund at oracle.com Tue Nov 28 12:35:57 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:35:57 +0100 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: References: <5A1BF6F0.5030903@oracle.com> Message-ID: <5A1D582D.3090505@oracle.com> Hi Coleen, On 2017-11-27 19:21, coleen.phillimore at oracle.com wrote: > http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/src/hotspot/share/prims/jvmtiTagMap.cpp.udiff.html > > > + return RootAccess AS_NO_KEEPALIVE>::oop_load(object_addr()); > > > Why is this not access ON_ROOT{_CONCURRENT} ? The thing holding the > object that you are peeking at is not in the Java Heap? The reason is that all GC root processing of these roots (off-heap oop*) are performed completely in a safepoints as opposed to concurrently. Thanks, /Erik > > thanks, > Coleen > > On 11/27/17 6:28 AM, Erik ?sterlund wrote: >> Hi, >> >> The JVMTI tag hashmap has weak oop references that are handled using >> raw oop accesses and a G1-specific SATB enqueue call when leaking out >> objects from the tag map, requiring them to be marked as live by G1. >> >> This should now be refactored to use the Access API to annotate that >> these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use >> ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191894 >> >> Thanks, >> /Erik > From erik.osterlund at oracle.com Tue Nov 28 12:37:08 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:37:08 +0100 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <3b2f7a9a-8abf-5a2a-9381-9046ffc21ee9@oracle.com> References: <5A1BF6F0.5030903@oracle.com> <3b2f7a9a-8abf-5a2a-9381-9046ffc21ee9@oracle.com> Message-ID: <5A1D5874.4050200@oracle.com> Hi Coleen, That's it, yes. Hope I answered the question. Thanks, /Erik On 2017-11-27 19:28, coleen.phillimore at oracle.com wrote: > > > On 11/27/17 1:21 PM, coleen.phillimore at oracle.com wrote: >> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/src/hotspot/share/prims/jvmtiTagMap.cpp.udiff.html >> >> >> + return RootAccess> AS_NO_KEEPALIVE>::oop_load(object_addr()); >> > > Sorry I have my Access API dimensions mixed up. RootAccess is IN_ROOT > not ON_ROOT (and not concurrent). > > Coleen >> >> Why is this not access ON_ROOT{_CONCURRENT} ? The thing holding the >> object that you are peeking at is not in the Java Heap? >> >> thanks, >> Coleen >> >> On 11/27/17 6:28 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> The JVMTI tag hashmap has weak oop references that are handled using >>> raw oop accesses and a G1-specific SATB enqueue call when leaking >>> out objects from the tag map, requiring them to be marked as live by >>> G1. >>> >>> This should now be refactored to use the Access API to annotate that >>> these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads to use >>> ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8191894 >>> >>> Thanks, >>> /Erik >> > From erik.osterlund at oracle.com Tue Nov 28 12:37:46 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 13:37:46 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: References: <5A1C06D8.1040405@oracle.com> <31206de6-f8c2-7dda-b811-ffc163291675@oracle.com> Message-ID: <5A1D589A.6000204@oracle.com> Hi David, Thanks for the review. /Erik On 2017-11-28 07:41, David Holmes wrote: > +1 > > David > > On 28/11/2017 4:27 AM, coleen.phillimore at oracle.com wrote: >> >> I think this is a nice simple fix and should be pushed for JDK10. >> Thanks, >> Coleen >> >> On 11/27/17 7:36 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> There is currently a bug when using unsafe accesses off-heap: >>> >>> 1) We write into a thread that we enable crash protection (using >>> GuardUnsafeAccess): >>> 2) We perform the access >>> 3) We write into a thread that we disable crash protection (using >>> ~GuardUnsafeAccess) >>> >>> The problem is that the crash protection stores are volatile, but >>> the actual access is non-volatile. Compilers have different >>> interpretation whether volatile - non-volatile accesses are allowed >>> to reorder. MSVC is known to interpret such interactions as-if the >>> volatile accesses have acquire/release semantics from the compiler >>> point of view, and others such as GCC are known to reorder away freely. >>> >>> To prevent any issues, the accesses involved when using >>> GuardUnsafeAccess should be at least volatile. >>> This change makes the few remaining ones volatile. The JMM-volatile >>> (SEQ_CST) accesses with crash protection already have stronger >>> ordering than volatile and hence do not need changing. >>> >>> By making the address passed in to the Access API volatile, the >>> MO_VOLATILE decorator is automatically set, which not surprisingly >>> makes the access volatile. Therefore, the solution is to simply make >>> the address passed in to Access volatile in this case. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>> >>> Thanks, >>> /Erik >> From thomas.schatzl at oracle.com Tue Nov 28 13:07:13 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 28 Nov 2017 14:07:13 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <5A1BD5B3.7030808@oracle.com> References: <5A1BD5B3.7030808@oracle.com> Message-ID: <1511874433.2867.0.camel@oracle.com> Hi, On Mon, 2017-11-27 at 10:06 +0100, Erik ?sterlund wrote: > Hi, > > The ClassLoaderData::remove_handle() member function currently uses > explicit G1 SATB barriers to remove an oop from the root set, as > these handles are not necessarily walked by the GC in a safepoint. > Therefore G1 needs pre-write barriers. > > This should now be modeled as a > RootAccess::oop_store instead. This maps to > performing a pre-write SATB barrier with G1, but other GCs are free > to do other things as necessary. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191888 looks good. Thomas From erik.osterlund at oracle.com Tue Nov 28 14:09:49 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 15:09:49 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <1511874433.2867.0.camel@oracle.com> References: <5A1BD5B3.7030808@oracle.com> <1511874433.2867.0.camel@oracle.com> Message-ID: <5A1D6E2D.70405@oracle.com> Hi Thomas, Thanks for the review. /Erik On 2017-11-28 14:07, Thomas Schatzl wrote: > Hi, > > On Mon, 2017-11-27 at 10:06 +0100, Erik ?sterlund wrote: >> Hi, >> >> The ClassLoaderData::remove_handle() member function currently uses >> explicit G1 SATB barriers to remove an oop from the root set, as >> these handles are not necessarily walked by the GC in a safepoint. >> Therefore G1 needs pre-write barriers. >> >> This should now be modeled as a >> RootAccess::oop_store instead. This maps to >> performing a pre-write SATB barrier with G1, but other GCs are free >> to do other things as necessary. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191888 > looks good. > > Thomas From per.liden at oracle.com Tue Nov 28 15:07:43 2017 From: per.liden at oracle.com (Per Liden) Date: Tue, 28 Nov 2017 16:07:43 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <5A1BD5B3.7030808@oracle.com> References: <5A1BD5B3.7030808@oracle.com> Message-ID: Looks good! /Per On 2017-11-27 10:06, Erik ?sterlund wrote: > Hi, > > The ClassLoaderData::remove_handle() member function currently uses > explicit G1 SATB barriers to remove an oop from the root set, as these > handles are not necessarily walked by the GC in a safepoint. Therefore > G1 needs pre-write barriers. > > This should now be modeled as a > RootAccess::oop_store instead. This maps to > performing a pre-write SATB barrier with G1, but other GCs are free to > do other things as necessary. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8191888 > > Thanks, > /Erik From erik.osterlund at oracle.com Tue Nov 28 15:32:25 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 16:32:25 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: References: <5A1BD5B3.7030808@oracle.com> Message-ID: <5A1D8189.10603@oracle.com> Hi Per, Thanks for the review. /Erik On 2017-11-28 16:07, Per Liden wrote: > Looks good! > > /Per > > On 2017-11-27 10:06, Erik ?sterlund wrote: >> Hi, >> >> The ClassLoaderData::remove_handle() member function currently uses >> explicit G1 SATB barriers to remove an oop from the root set, as these >> handles are not necessarily walked by the GC in a safepoint. Therefore >> G1 needs pre-write barriers. >> >> This should now be modeled as a >> RootAccess::oop_store instead. This maps to >> performing a pre-write SATB barrier with G1, but other GCs are free to >> do other things as necessary. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8191888 >> >> Thanks, >> /Erik From volker.simonis at gmail.com Tue Nov 28 16:46:50 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Tue, 28 Nov 2017 17:46:50 +0100 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Message-ID: Hi Ioi, the change and your refactorings to it look good :) I specially like the introduction of "@requires vm.cds.custom.loaders" to simplify the logic. I've only found two little problems: - the patch for the test "cacheObject/CheckCachedResolvedReferences.java" is empty. That's probably because it is not related to AppCDS. But the test can and should still run on all 64-bit Linux and AIX platfroms as well. - the following two tests still get executed in the product build where they fail because they use non-product flags: runtime/appcds/DumpClassList.java runtime/appcds/ProhibitedPackage.java I think you fixed that in one of your previous patches but it probably got lost again before pushing. The attached "addon.patch" fixes these two problems. Thank you and bet regards, Volker On Mon, Nov 27, 2017 at 10:42 PM, Ioi Lam wrote: > Hi, > > Please review this change to support AppCDS on all 64-bit Linux platforms as > well as AIX. > > https://bugs.openjdk.java.net/browse/JDK-8191927 > http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ > > The patch must be applied on top of the AppCDS patch: > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ > > The patch was initially contributed by Volker Simonis. I modified it a > little to avoid repeating the platform enumerations in all the test cases. > Instead, I added a new test property, so the tests that require AppCDS > support for custom loaders can be written as: > > * @requires vm.cds.custom.loaders > > I've tested on Linux manually, and I am now running on all of the > Oracle-supported platforms using our internal test harness. > > Thanks > - Ioi -------------- next part -------------- A non-text attachment was scrubbed... Name: addon.patch Type: text/x-patch Size: 2034 bytes Desc: not available URL: From erik.osterlund at oracle.com Tue Nov 28 16:50:09 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 28 Nov 2017 17:50:09 +0100 Subject: RFR (S): 8192003: Refactor weak references in StringTable to use the Access API Message-ID: <5A1D93C1.3020605@oracle.com> Hi, The StringTable contains weak references to oops. Today the weak semantics is managed using explicit calls to G1 SATB enqueue barriers. Now that the Access API is available, it should be used instead as it is more modular. This change fixes that by making these oops ON_PHANTOM_OOP_REF with a corresponding AS_NO_KEEPALIVE accessor when we do not want to keep it alive, much like my previous changes to other weak tables. Webrev: http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8192003 Thanks, /Erik From vladimir.kozlov at oracle.com Tue Nov 28 16:52:21 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 28 Nov 2017 08:52:21 -0800 Subject: Future plans for AppCDS? In-Reply-To: References: Message-ID: Here is very good and detailed blog about what you can do with AppCDS and AOT: https://mjg123.github.io/2017/10/04/AppCDS-and-Clojure.html Regards, Vladimir On 11/28/17 2:03 AM, Volker Simonis wrote: > Hi Mike, > > what you are actually asking for is support for AOT (Ahead Of Time > compilation) [1]. > > It is already available in Java 9. Although the support for user code > is still experimental you should definitely give it a try. > > Regards, > Volker > > [1] http://openjdk.java.net/jeps/295 > > > On Tue, Aug 4, 2015 at 8:23 PM, Mike Hearn wrote: >> Hi there, >> >> I'm wondering if there are any plans to extend AppCDS in future to include >> serializing the code cache to disk? >> >> I ask because I have a (not very complicated) desktop JavaFX app. It starts >> in about three seconds, which isn't terrible, but I'd like it to be faster. >> >> Unfortunately I am frequently frustrated in this goal. Some testing shows >> that one step of the initialisation sequence takes about a second normally. >> If I put it in a loop to let it fully compile, that drops to more like a >> quarter of a second. >> >> Likewise, my app loads a series of items to display them on the screen. The >> first one can take a solid 700-800 msec. The rest are more like 10% of that. >> >> I tried parallelising some of the things done during startup, but it made >> no difference. My theory is that the compile threads are taking up the >> spare cores that could be doing startup tasks (it's a bit hard to profile >> this though as the whole sequence only lasts a few seconds). >> >> AppCDS already knows how to serialise lots of HotSpot state to disk for >> unchanging JARs. If it could store compiled method code as well, then I >> could probably shave a second or two off app startup. >> >> I do understand that this situation is a bit rare for Java developers and >> that due to extra disk IO etc, loading compiled code from disk might not >> always be faster. But for devices that have an SSD it probably can be, >> especially if HotSpot were to do the same trick Microsoft does and lay out >> compiled methods on disk in the order they will be used. From ioi.lam at oracle.com Tue Nov 28 17:41:26 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 28 Nov 2017 09:41:26 -0800 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Message-ID: <99afe888-ecb9-ff36-a627-d7d31176025b@oracle.com> Hi Volker, Thanks for the review On 11/28/17 8:46 AM, Volker Simonis wrote: > Hi Ioi, > > the change and your refactorings to it look good :) I specially like > the introduction of "@requires vm.cds.custom.loaders" to simplify the > logic. > > I've only found two little problems: > > - the patch for the test > "cacheObject/CheckCachedResolvedReferences.java" is empty. That's > probably because it is not related to AppCDS. But the test can and > should still run on all 64-bit Linux and AIX platfroms as well. I've fixed this. I thought it's unrelated to custom loaders, but it actually does. I've added a comment to reflect that: --- a/CheckCachedResolvedReferences.java??? Tue Nov 28 21:04:42 2017 +0530 +++ b/CheckCachedResolvedReferences.java??? Tue Nov 28 09:33:46 2017 -0800 @@ -26,8 +26,7 @@ ? * @test ? * @summary Test resolved_references ? * @requires (vm.opt.UseCompressedOops == null) | (vm.opt.UseCompressedOops == true) - * @requires (sun.arch.data.model == "64") - * @requires ((os.family == "linux") & (os.arch=="amd64")) | (os.family == "solaris") + * @requires vm.cds.custom.loaders ? * @requires (vm.gc=="null") ? * @library /test/lib /test/hotspot/jtreg/runtime/appcds ? * @modules java.base/jdk.internal.misc @@ -53,9 +52,9 @@ ???????? String helloJarPath = ClassFileInstaller.getJarPath("hello.jar"); ???????? String classlist[] = new String[] { -??????????? "CheckCachedResolvedReferencesApp", -??????????? "java/lang/Object id: 1", -??????????? "Hello id: 2 super: 1 source: " + helloJarPath +??????????? "CheckCachedResolvedReferencesApp",??????????? // built-in app loader +??????????? "java/lang/Object id: 1",????????????????????? // boot loader +??????????? "Hello id: 2 super: 1 source: " + helloJarPath // custom loader ???????? }; ???????? TestCommon.testDump(appJar, classlist, use_whitebox_jar); > - the following two tests still get executed in the product build > where they fail because they use non-product flags: > runtime/appcds/DumpClassList.java > runtime/appcds/ProhibitedPackage.java > > I think you fixed that in one of your previous patches but it probably > got lost again before pushing. I wanted to fix them in a separate changeset, so that the "move appcds to open" changeset is a more verbatim move. It's the following bug, and I will post RFR shortly. https://bugs.openjdk.java.net/browse/JDK-8191747 "[TESTBUG] runtime/AppCDS/DumpClassList.java and ProhibitedPackage.java fail on product bits" Instead of "@require vm.debug", I'll try to rewrite the test to remove the dependency on -XX:+PrintSystemDictionaryAtExit Thanks - Ioi > The attached "addon.patch" fixes these two problems. > > Thank you and bet regards, > Volker > > > On Mon, Nov 27, 2017 at 10:42 PM, Ioi Lam wrote: >> Hi, >> >> Please review this change to support AppCDS on all 64-bit Linux platforms as >> well as AIX. >> >> https://bugs.openjdk.java.net/browse/JDK-8191927 >> http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ >> >> The patch must be applied on top of the AppCDS patch: >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ >> >> The patch was initially contributed by Volker Simonis. I modified it a >> little to avoid repeating the platform enumerations in all the test cases. >> Instead, I added a new test property, so the tests that require AppCDS >> support for custom loaders can be written as: >> >> * @requires vm.cds.custom.loaders >> >> I've tested on Linux manually, and I am now running on all of the >> Oracle-supported platforms using our internal test harness. >> >> Thanks >> - Ioi From ioi.lam at oracle.com Tue Nov 28 17:58:53 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 28 Nov 2017 09:58:53 -0800 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits Message-ID: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Hi, Here's a fix for these two tests failing in product builds. The fix is to replace the use of debug flag PrintSystemDictionaryAtExit flag with -Xlog:class+load, which is available in product builds: https://bugs.openjdk.java.net/browse/JDK-8191747 http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ Thanks - Ioi From volker.simonis at gmail.com Tue Nov 28 18:00:10 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Tue, 28 Nov 2017 19:00:10 +0100 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: <99afe888-ecb9-ff36-a627-d7d31176025b@oracle.com> References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> <99afe888-ecb9-ff36-a627-d7d31176025b@oracle.com> Message-ID: On Tue, Nov 28, 2017 at 6:41 PM, Ioi Lam wrote: > Hi Volker, > > Thanks for the review > > > On 11/28/17 8:46 AM, Volker Simonis wrote: >> >> Hi Ioi, >> >> the change and your refactorings to it look good :) I specially like >> the introduction of "@requires vm.cds.custom.loaders" to simplify the >> logic. >> >> I've only found two little problems: >> >> - the patch for the test >> "cacheObject/CheckCachedResolvedReferences.java" is empty. That's >> probably because it is not related to AppCDS. But the test can and >> should still run on all 64-bit Linux and AIX platfroms as well. > > > I've fixed this. I thought it's unrelated to custom loaders, but it actually > does. I've added a comment to reflect that: > > > --- a/CheckCachedResolvedReferences.java Tue Nov 28 21:04:42 2017 +0530 > +++ b/CheckCachedResolvedReferences.java Tue Nov 28 09:33:46 2017 -0800 > @@ -26,8 +26,7 @@ > * @test > * @summary Test resolved_references > * @requires (vm.opt.UseCompressedOops == null) | (vm.opt.UseCompressedOops > == true) > - * @requires (sun.arch.data.model == "64") > - * @requires ((os.family == "linux") & (os.arch=="amd64")) | (os.family == > "solaris") > + * @requires vm.cds.custom.loaders > * @requires (vm.gc=="null") > * @library /test/lib /test/hotspot/jtreg/runtime/appcds > * @modules java.base/jdk.internal.misc > @@ -53,9 +52,9 @@ > String helloJarPath = ClassFileInstaller.getJarPath("hello.jar"); > > String classlist[] = new String[] { > - "CheckCachedResolvedReferencesApp", > - "java/lang/Object id: 1", > - "Hello id: 2 super: 1 source: " + helloJarPath > + "CheckCachedResolvedReferencesApp", // built-in app > loader > + "java/lang/Object id: 1", // boot loader > + "Hello id: 2 super: 1 source: " + helloJarPath // custom loader > }; > > TestCommon.testDump(appJar, classlist, use_whitebox_jar); > OK > > >> - the following two tests still get executed in the product build >> where they fail because they use non-product flags: >> runtime/appcds/DumpClassList.java >> runtime/appcds/ProhibitedPackage.java >> >> I think you fixed that in one of your previous patches but it probably >> got lost again before pushing. > > > I wanted to fix them in a separate changeset, so that the "move appcds to > open" > changeset is a more verbatim move. It's the following bug, and I will post > RFR shortly. > > https://bugs.openjdk.java.net/browse/JDK-8191747 > "[TESTBUG] runtime/AppCDS/DumpClassList.java and ProhibitedPackage.java fail > on product bits" > > Instead of "@require vm.debug", I'll try to rewrite the test to remove the > dependency on -XX:+PrintSystemDictionaryAtExit > OK, that's fine for me. Thumbs up then for the change! Volker > Thanks > - Ioi > > >> The attached "addon.patch" fixes these two problems. >> >> Thank you and bet regards, >> Volker >> >> >> On Mon, Nov 27, 2017 at 10:42 PM, Ioi Lam wrote: >>> >>> Hi, >>> >>> Please review this change to support AppCDS on all 64-bit Linux platforms >>> as >>> well as AIX. >>> >>> https://bugs.openjdk.java.net/browse/JDK-8191927 >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ >>> >>> The patch must be applied on top of the AppCDS patch: >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ >>> >>> The patch was initially contributed by Volker Simonis. I modified it a >>> little to avoid repeating the platform enumerations in all the test >>> cases. >>> Instead, I added a new test property, so the tests that require AppCDS >>> support for custom loaders can be written as: >>> >>> * @requires vm.cds.custom.loaders >>> >>> I've tested on Linux manually, and I am now running on all of the >>> Oracle-supported platforms using our internal test harness. >>> >>> Thanks >>> - Ioi > > From george.triantafillou at oracle.com Tue Nov 28 18:17:30 2017 From: george.triantafillou at oracle.com (George Triantafillou) Date: Tue, 28 Nov 2017 13:17:30 -0500 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: Hi Ioi, Looks good. -George On 11/28/2017 12:58 PM, Ioi Lam wrote: > Hi, > > Here's a fix for these two tests failing in product builds. The fix is > to replace the use of debug flag PrintSystemDictionaryAtExit flag with > -Xlog:class+load, which is available in product builds: > > https://bugs.openjdk.java.net/browse/JDK-8191747 > http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ > > > Thanks > - Ioi > From jiangli.zhou at oracle.com Tue Nov 28 18:19:09 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 28 Nov 2017 10:19:09 -0800 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: <1E36AD86-581D-4942-B7D8-75EE29C588A5@oracle.com> Hi Ioi, For the following check at line 72 in ProhibitedPackage.java, can you include more complete logging output "[info][class,load] java.lang.Prohibited?? So other output message for java.lang.Prohibited would not be mistakenly matched for this check. No new webrev needed for me with the shouldNotContain() content change. .shouldNotContain("java.lang.Prohibited") I noticed you didn't have the tests mentioned in the email. Could you please list the tests that you run? Thanks, Jiangli > On Nov 28, 2017, at 9:58 AM, Ioi Lam wrote: > > Hi, > > Here's a fix for these two tests failing in product builds. The fix is to replace the use of debug flag PrintSystemDictionaryAtExit flag with -Xlog:class+load, which is available in product builds: > > https://bugs.openjdk.java.net/browse/JDK-8191747 > http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ > > Thanks > - Ioi > From volker.simonis at gmail.com Tue Nov 28 18:21:38 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Tue, 28 Nov 2017 19:21:38 +0100 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: Hi Ioi, looks good! I was a little confused first that "-Xlog:cds" was not set as argument in ProhibitedPackage.java But I saw now that it is set by default in TestCommon for every invocation of "dump()". Maybe you could mention that in ProhibitedPackage.java in the comment before the execution of the dump where you parse the "[cds]" logs? Regards, Volker On Tue, Nov 28, 2017 at 6:58 PM, Ioi Lam wrote: > Hi, > > Here's a fix for these two tests failing in product builds. The fix is to > replace the use of debug flag PrintSystemDictionaryAtExit flag with > -Xlog:class+load, which is available in product builds: > > https://bugs.openjdk.java.net/browse/JDK-8191747 > http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ > > Thanks > - Ioi > From ioi.lam at oracle.com Tue Nov 28 18:46:22 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 28 Nov 2017 10:46:22 -0800 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: <1E36AD86-581D-4942-B7D8-75EE29C588A5@oracle.com> References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> <1E36AD86-581D-4942-B7D8-75EE29C588A5@oracle.com> Message-ID: <66e276e5-759f-ddaa-49df-d00fba622bfb@oracle.com> Hi Jiangli, Thanks for the review. On 11/28/17 10:19 AM, Jiangli Zhou wrote: > Hi Ioi, > > For the following check at line 72 in?ProhibitedPackage.java, can you > include more complete logging output "[info][class,load] > java.lang.Prohibited?? So other output message for > java.lang.Prohibited would not be mistakenly matched for this check. > No new webrev needed for me with the shouldNotContain() content change. > > .shouldNotContain("java.lang.Prohibited") Good idea! I changed it to ??????????? .shouldNotContain("[info][class,load] java.lang.Prohibited source: ") so it won't be confused with other messages that could possibly have the string "java.lang.Prohibited" in it. > I noticed you didn't have the tests mentioned in the email. Could you > please list the tests that you run? > I ran all the appcds tests on Linux/x64 in both product and debug builds. The names of the two tests are mentioned in the subject of this email :-) Thanks - Ioi > Thanks, > Jiangli > >> On Nov 28, 2017, at 9:58 AM, Ioi Lam > > wrote: >> >> Hi, >> >> Here's a fix for these two tests failing in product builds. The fix >> is to replace the use of debug flag PrintSystemDictionaryAtExit flag >> with -Xlog:class+load, which is available in product builds: >> >> https://bugs.openjdk.java.net/browse/JDK-8191747 >> http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ >> >> Thanks >> - Ioi >> > From ioi.lam at oracle.com Tue Nov 28 18:51:10 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 28 Nov 2017 10:51:10 -0800 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: On 11/28/17 10:21 AM, Volker Simonis wrote: > Hi Ioi, > > looks good! > > I was a little confused first that "-Xlog:cds" was not set as argument > in ProhibitedPackage.java > But I saw now that it is set by default in TestCommon for every > invocation of "dump()". Maybe you could mention that in > ProhibitedPackage.java in the comment before the execution of the dump > where you parse the "[cds]" logs? Hi Volker, Thanks for noticing this. I added -Xlog:cds unconditionally so if TestCommon is changed in the future, this test will still work: ???????????? // Make sure a class in a prohibited package for a custom loader ???????????? // will be ignored during dumping. -??????????? TestCommon.dump(appJar, -??????????????????????????? classlist, - "-XX:+PrintSystemDictionaryAtExit") +??????????? TestCommon.dump(appJar,? classlist, "-Xlog:cds") ???????????????? .shouldContain("Dumping") -??????????????? .shouldNotContain("java.lang.Prohibited") +??????????????? .shouldContain("[cds] Prohibited package for non-bootstrap classes: java/lang/Prohibited.class") ???????????????? .shouldHaveExitValue(0); Thanks - Ioi > Regards, > Volker > > > On Tue, Nov 28, 2017 at 6:58 PM, Ioi Lam wrote: >> Hi, >> >> Here's a fix for these two tests failing in product builds. The fix is to >> replace the use of debug flag PrintSystemDictionaryAtExit flag with >> -Xlog:class+load, which is available in product builds: >> >> https://bugs.openjdk.java.net/browse/JDK-8191747 >> http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ >> >> Thanks >> - Ioi >> From coleen.phillimore at oracle.com Tue Nov 28 20:20:59 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 28 Nov 2017 15:20:59 -0500 Subject: RFR (S): 8191894: Refactor weak references in JvmtiTagHashmap to use the Access API In-Reply-To: <5A1D5874.4050200@oracle.com> References: <5A1BF6F0.5030903@oracle.com> <3b2f7a9a-8abf-5a2a-9381-9046ffc21ee9@oracle.com> <5A1D5874.4050200@oracle.com> Message-ID: <59c3d40e-6493-7ca2-3ff3-9cc15fa461c3@oracle.com> On 11/28/17 7:37 AM, Erik ?sterlund wrote: > Hi Coleen, > > That's it, yes. Hope I answered the question. Yes, thanks. Coleen > > Thanks, > /Erik > > On 2017-11-27 19:28, coleen.phillimore at oracle.com wrote: >> >> >> On 11/27/17 1:21 PM, coleen.phillimore at oracle.com wrote: >>> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/src/hotspot/share/prims/jvmtiTagMap.cpp.udiff.html >>> >>> >>> + return RootAccess>> AS_NO_KEEPALIVE>::oop_load(object_addr()); >>> >> >> Sorry I have my Access API dimensions mixed up.? RootAccess is >> IN_ROOT not ON_ROOT (and not concurrent). >> >> Coleen >>> >>> Why is this not access ON_ROOT{_CONCURRENT} ?? The thing holding the >>> object that you are peeking at is not in the Java Heap? >>> >>> thanks, >>> Coleen >>> >>> On 11/27/17 6:28 AM, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> The JVMTI tag hashmap has weak oop references that are handled >>>> using raw oop accesses and a G1-specific SATB enqueue call when >>>> leaking out objects from the tag map, requiring them to be marked >>>> as live by G1. >>>> >>>> This should now be refactored to use the Access API to annotate >>>> that these are ON_PHANTOM_OOP_REF, and refactor the raw oop loads >>>> to use ON_PHANTOM_OOP_REF | AS_NO_KEEPALIVE. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8191894/webrev.00/ >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8191894 >>>> >>>> Thanks, >>>> /Erik >>> >> > From kim.barrett at oracle.com Tue Nov 28 20:30:11 2017 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 28 Nov 2017 15:30:11 -0500 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <5A1C06D8.1040405@oracle.com> References: <5A1C06D8.1040405@oracle.com> Message-ID: > On Nov 27, 2017, at 7:36 AM, Erik ?sterlund wrote: > > Hi, > > There is currently a bug when using unsafe accesses off-heap: > > 1) We write into a thread that we enable crash protection (using GuardUnsafeAccess): > 2) We perform the access > 3) We write into a thread that we disable crash protection (using ~GuardUnsafeAccess) > > The problem is that the crash protection stores are volatile, but the actual access is non-volatile. Compilers have different interpretation whether volatile - non-volatile accesses are allowed to reorder. MSVC is known to interpret such interactions as-if the volatile accesses have acquire/release semantics from the compiler point of view, and others such as GCC are known to reorder away freely. > > To prevent any issues, the accesses involved when using GuardUnsafeAccess should be at least volatile. > This change makes the few remaining ones volatile. The JMM-volatile (SEQ_CST) accesses with crash protection already have stronger ordering than volatile and hence do not need changing. > > By making the address passed in to the Access API volatile, the MO_VOLATILE decorator is automatically set, which not surprisingly makes the access volatile. Therefore, the solution is to simply make the address passed in to Access volatile in this case. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8186787 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ > > Thanks, > /Erik Not my first choice for a fix (you know how I feel about casts), but it works, and is currently much simpler than my preferred solution. Should there be a comment justifying the cast to volatile? Perhaps something like "volatile to keep access within guarded scope." I don't need a new webrev if you add such a comment. Looks good. From david.holmes at oracle.com Tue Nov 28 21:28:42 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 29 Nov 2017 07:28:42 +1000 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <5A1D546A.1040601@oracle.com> References: <5A1BD5B3.7030808@oracle.com> <5A1D546A.1040601@oracle.com> Message-ID: <1b1473cd-2235-8c98-617e-fac4c96c5b04@oracle.com> Hi Erik, On 28/11/2017 10:19 PM, Erik ?sterlund wrote: > Hi David, > > That is a good question. > There is a description (written by a GC person, but still) for each > decorator in the access.hpp file where the decorators are declared. I > try to go in to some details there when you should and should not use a > decorator. If we find that the text might have been biased from a GC > perspective, we can update the comments. The comments serve as a reference, but I think we may lack a "tutorial". :) As I think someone commented earlier re the Access API, how do I know that, for example, IN_CONCURRENT_ROOT applies when I have no idea what may or may not be scanned during safepoint ?? But that's OT. Thanks, David > The previous situation has been to use naked accesses, and then suddenly > a GC person jumps in moaning about the lack of explicit #if > INCLUDE_ALL_GCS if (UseG1) SATB barrier goo, or manual insertion of > post-write barriers if a CAS was successful. There was no good way of > knowing you had to manually insert these barriers, what barriers were > offered to shared code, or what their semantics were. > > Now at least you have a set of decorators to choose from, with some > automatic verification if you select nonsense decorators, and > documentation in one place what they are used for and what they mean. So > if you do not know what you need, at least you can read the descriptions > and hopefully figure it out, which was impossible before. > > Having said that, I believe things like Coleen's OopHandle and > PhantomOopHandle will build a layer on top of Access with possibly a few > less details exposed that are even simpler to use for the conservative > non-GC person that just wants the thing to work as simple as possible > and wants to stay as far away from GC land as possible, which is sane. > > And if you feel uncertain, you can always loop me in any time, and I > will probably say "there's a decorator for that". > > I hope this answers your question. > > Thanks, > /Erik > > On 2017-11-28 07:59, David Holmes wrote: >> Hi Erik, >> >> Is there a non-GC person's guide to what the different forms of >> RootAccess mean and when to use them? Or do we expect a GC person >> to always jump in to show where such things are needed? ;-) >> >> Thanks, >> David >> >> On 27/11/2017 7:06 PM, Erik ?sterlund wrote: >>> Hi, >>> >>> The ClassLoaderData::remove_handle() member function currently uses >>> explicit G1 SATB barriers to remove an oop from the root set, as >>> these handles are not necessarily walked by the GC in a safepoint. >>> Therefore G1 needs pre-write barriers. >>> >>> This should now be modeled as a >>> RootAccess::oop_store instead. This maps to >>> performing a pre-write SATB barrier with G1, but other GCs are free >>> to do other things as necessary. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8191888 >>> >>> Thanks, >>> /Erik > From mikhailo.seledtsov at oracle.com Wed Nov 29 00:50:14 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Tue, 28 Nov 2017 16:50:14 -0800 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs Message-ID: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java based on the number of processors available. ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 ??? Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ ??? Testing: ??????? 1. Locally: exercised the affected test locally on Linux-x64 ??????? 2. Exercised the affected test on 2 processor machine ??????? 3. Exercise the affected test via automated distributed test system ?????????? In progress Misha From david.holmes at oracle.com Wed Nov 29 04:46:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 29 Nov 2017 14:46:49 +1000 Subject: RFR(s-ish): 8191101: Show register content in hs-err file on assert In-Reply-To: References: Message-ID: Hi Thomas, I took a look at this and have a few concerns ... But first, this is currently targeted to JDK 11 - is that what you are aiming for? The JDK 10 gate closes on Friday for hotspot changes. On 28/11/2017 6:08 PM, Thomas St?fe wrote: > Updated webrev with Andrews request worked in: > > http://cr.openjdk.java.net/~stuefe/webrevs/8191101-show-registers-on-assert/webrev.01/webrev/ I don't understand why a number of platforms define os::Posix::copy_context as a no-op instead of using os::Posix::default_copy_ucontext - this is either common posix code or it is not. ?? Is it just a case of you not being able to test these other "posix" platforms? But if the actual ucontext_t layout is OS specific and not Posix specific then this should not be defined in the os::Posix class. (os::Posix is supposed to be a place for standard Posix conforming functionality, not a convenient place to share stuff on non-windows OS.) Any why the combination of build-time and run-time checking? This seems overly complicated. Can't you just rely on runtime checks? Your bug synopsis and preamble only mentions assert so I was going to comment that: ! diagnostic_pd(bool, ShowRegistersOnAssert, should be a develop flag not diagnostic but then I saw you do this for guarantees and all the other abort points too! In which case use of "assert" in the flag name and CAN_SHOW_REGISTERS_ON_ASSERT, is misleading - how about s/assert/abort/ ? --- I can't really comment on the safe* functionality - I'm not sure what that is all about. ?? --- src/hotspot/os/posix/os_posix.cpp Style nit: if (copy) -> if (copy != NULL) + ::free(uc); // [sic] see os::Posix::copy_context(): the ucontext should have been copied with raw ::malloc, I don't understand the comment - should it be referring to os::Posix::default_copy_ucontext? Shouldn't the issue with os::malloc use be documented in os::Posix::default_copy_ucontext? And the comment here just needs to say: ::free(uc); // Use raw free to match the raw malloc --- src/hotspot/os/posix/os_posix.hpp + // Copy an ucontext. an -> a + // (which is most of them).Do Space before Do --- src/hotspot/os_cpu/linux_ppc/os_linux_ppc.cpp Style nit: if (uc) -> if (uc != NULL) --- src/hotspot/share/utilities/debug.hpp 38 #endif 39 40 #ifdef CAN_SHOW_REGISTERS_ON_ASSERT You can delete these three lines as you know you can show registers on assert. 42 #define TOUCH_ASSERT_POISON (*(char*)g_assert_poison = 'X'); Why the casting? Why not declare g_assert_poison as intptr_t* and write an int into it? Nit: usually macros don't include the trailing ; so that in the code they look like statements. (Though I think the use of macro and conditional compilation is overkill here.) --- test/hotspot/jtreg/runtime/ErrorHandling/ShowRegistersOnAssertTest.java I think we need to be able to test a guarantee failure in a product build, else how do we know this works okay in that scenario? Do we have something in place to allow that? --- Thanks, David ----- > The only difference to before is that I restored the ShouldNotReachHere() > at the end of VMError::controlled_crash(). Before that assert, I added a > hopefully clearer message than before to enable my test to scan for this > message. I also changed the jtreg test to scan for this new message. > > Note: in the jtreg test, I test that assert supression works with > -XX:SupressErrorAt. But I just give it the file > (-XX:SuppressErrorAt=/vmError.cpp) > and omit the line number, because I do not want the test to stop working > when someone adds lines in front of VMError::controlled_crash(). But that > means that the ShouldNotReachHere() at the end of controlled_crash() does > not fire either, because now all asserts in vmError.cpp are disabled. So, I > need this new message. > > Thanks, Thomas > > On Mon, Nov 27, 2017 at 11:33 AM, Thomas St?fe > wrote: > >> Ping... Could I please have reviews? Thank you! >> ..Thomas >> >> On Wed, Nov 22, 2017 at 7:01 PM, Thomas St?fe >> wrote: >> >>> Hi all, >>> >>> may I please have reviews for the following enhancement: >>> >>> issue: https://bugs.openjdk.java.net/browse/JDK-8191101 >>> webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8191101- >>> show-registers-on-assert/webrev.00/webrev/ >>> >>> (Patch looks big but lot of it is os_cpu fluff.) >>> >>> Prior email discussion at: http://mail.openjdk.java.net/p >>> ipermail/hotspot-runtime-dev/2017-October/025018.html >>> >>> ---- >>> >>> Basically, this adds the ability to show register values in assert >>> situations into the error report. This can be useful in certain corner >>> cases, e.g. if you want to know the value of some local variables or how >>> deep your stack currently runs. >>> >>> This works by triggering a fault right when an assert happens and >>> squirreling the context away for later error reporting. >>> >>> When an assert happens, we touch a poison page. To preserve the context >>> as best as possible, we want to avoid running too much code after the >>> assert condition has been evaluated, so this is done in a very simple way: >>> directly in the assert macro, right after the condition is evaluated, we >>> dereference the content of a global poison page pointer. In my tests, even >>> with slowdebug, this only spoils one register, rax on x64. >>> >>> In the signal handler, we recognize the assertion poison fault by the >>> faulting address. We disable the poison page and store the ucontext away. >>> Then we just return from signal handling. Poison page is now disarmed, the >>> load from it is retried and now goes through. Normal assertion handling is >>> then resumed - so, things like -XX:SuppressAt are unaffected and work fine. >>> >>> Then, when an error report is generated due to this assert, we now also >>> use the stored context. So now we get registers and instructions at the >>> assert point. >>> >>> Right now, this is implemented on (non-zero) Linux, though other posix >>> platforms should be no problem. Have not yet thought deeply about windows. >>> It is tested and - in debug - switched on by default on linux x86, ppc, >>> s390. >>> >>> If implemented, it can be switched on and off with >>> -XX:+ShowRegistersOnAssert. This is a failsafe, in case the mechanism does >>> not work and we want to have clean asserts. >>> >>> To test this, do a java -XX:ErrorHandlerTest=1 -XX:+ShowRegistersOnAssert >>> with a not-product VM. On Linux x64, ppc and s390 we should now see the >>> register output in the hs-err file: >>> >>> 4 # Internal Error (/shared/projects/openjdk/jdk- >>> hs/source/src/hotspot/share/utilities/vmError.cpp:1660), pid=4022, >>> tid=4023 >>> 5 # assert(str == NULL) failed: expected null >>> 6 # >>> 7 # >>> ... >>> 59 Registers: >>> 60 RAX=0x00007f736c8f7000, RBX=0x0000000000000000, >>> RCX=0x0000000000000000, RDX=0xffffffffffb5ded4 >>> 61 RSP=0x00007f736c8d8ce0, RBP=0x00007f736c8d8d30, >>> RSI=0x0000000000000001, RDI=0x0000000000000001 >>> 62 R8 =0x0000000000000040, R9 =0x0000000000000001, >>> R10=0x0000000000efb028, R11=0x00040788a244f42a >>> 63 R12=0x0000000000000000, R13=0x00007fff3de46dbf, >>> R14=0x00007f736c8d99c0, R15=0x0000000000000000 >>> 64 RIP=0x00007f736aec5529, EFLAGS=0x0000000000010202, >>> CSGSFS=0x002b000000000033, ERR=0x0000000000000006 >>> 65 TRAPNO=0x000000000000000e >>> ... >>> 72 >>> 73 Instructions: (pc=0x00007f736aec5529) >>> 74 0x00007f736aec5509: 8d 05 31 21 4a 00 48 01 d0 ff e0 48 83 7d c8 00 >>> 75 0x00007f736aec5519: 0f 84 7f 03 00 00 48 8d 05 82 fc b1 00 48 8b 00 >>> 76 0x00007f736aec5529: c6 00 58 e8 cb 17 62 ff 84 c0 74 11 48 8d 3d 2f >>> 77 0x00007f736aec5539: 1f 4a 00 b8 00 00 00 00 e8 ed 17 62 ff 48 8d 0d >>> 78 >>> >>> --- >>> >>> Implementation notes: >>> >>> - when handling the poison fault, we need to copy the context away from >>> the signal handler stack. For posix, this means copying the ucontext_t. >>> This is undefined territory. On most platforms, this simply means copying >>> the ucontex_t as a flat structure. On some platforms more is needed, e.g. >>> on linux ppc, we need to patch up the context after copying (the context is >>> not position independent), and on MacOS, the context is not self-contained >>> but contains pointers to sub structures which need to be copied too and >>> whose size is unknown at compile time. Because of these platform >>> dependencies, I factored out the copying of ucontext_t to >>> os::Posix::copy_ucontext and its implementations are os_cpu specific. >>> >>> - As an added precaution, when copying the context, we use a safe version >>> of memcpy (os::safe_memcpy) which I added to copy from potentially invalid >>> memory regions. The reason is that we have seen on some Unices - e.g. hpux >>> - that the size of the ucontext_t structure at runtime may be different >>> from the build machine, so we tread carefully. os::safe_memcpy() uses >>> SafeFetch to copy a range of memory. >>> >>> Risk: >>> >>> If this does not work, asserts will become segfaults, which can be >>> confusing. But the feature can be disabled with -XX:+ShowRegistersOnAssert >>> and for now on most platforms is disabled by default. >>> >>> --- >>> >>> Limitations: >>> - as it is now implemented, this is a one-shot mechanism and only works >>> for the first assert. >>> - -XX:SuppressAt=... is not affected and works fine. However, if the >>> first assert is suppressed, follow-up asserts will not show register values. >>> - When multiple threads run into an assert, we may or may not see >>> register values depending on which thread is the first of finishing the >>> poison page handler. >>> >>> I do not think these limitations are severe. They can be solved, but at >>> the cost of added complexity, which I preferred not to add. >>> >>> Thank you, >>> >>> Thomas >>> >>> >>> >>> >> From david.holmes at oracle.com Wed Nov 29 05:23:20 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 29 Nov 2017 15:23:20 +1000 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> Message-ID: Seems okay. Thanks, David ----- On 29/11/2017 10:50 AM, mikhailo wrote: > Could you, please, review this simple fix? It limits test cases in > TestCPUAwareness.java > based on the number of processors available. > > ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 > ??? Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ > ??? Testing: > ??????? 1. Locally: exercised the affected test locally on Linux-x64 > ??????? 2. Exercised the affected test on 2 processor machine > ??????? 3. Exercise the affected test via automated distributed test > system > ?????????? In progress > > Misha > From volker.simonis at gmail.com Wed Nov 29 08:02:11 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Wed, 29 Nov 2017 09:02:11 +0100 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: On Tue, Nov 28, 2017 at 7:51 PM, Ioi Lam wrote: > > > On 11/28/17 10:21 AM, Volker Simonis wrote: >> >> Hi Ioi, >> >> looks good! >> >> I was a little confused first that "-Xlog:cds" was not set as argument >> in ProhibitedPackage.java >> But I saw now that it is set by default in TestCommon for every >> invocation of "dump()". Maybe you could mention that in >> ProhibitedPackage.java in the comment before the execution of the dump >> where you parse the "[cds]" logs? > > Hi Volker, > > Thanks for noticing this. I added -Xlog:cds unconditionally so if TestCommon > is changed in the future, this test will still work: > > // Make sure a class in a prohibited package for a custom > loader > // will be ignored during dumping. > - TestCommon.dump(appJar, > - classlist, > - "-XX:+PrintSystemDictionaryAtExit") > + TestCommon.dump(appJar, classlist, "-Xlog:cds") > .shouldContain("Dumping") > - .shouldNotContain("java.lang.Prohibited") > + .shouldContain("[cds] Prohibited package for non-bootstrap > classes: java/lang/Prohibited.class") > .shouldHaveExitValue(0); > > Looks good. Thumbs up! I've just realized that the JBS entry for this bug is not publicly visible. Maybe you can also change that? Regards, Volker > Thanks > - Ioi > > >> Regards, >> Volker >> >> >> On Tue, Nov 28, 2017 at 6:58 PM, Ioi Lam wrote: >>> >>> Hi, >>> >>> Here's a fix for these two tests failing in product builds. The fix is to >>> replace the use of debug flag PrintSystemDictionaryAtExit flag with >>> -Xlog:class+load, which is available in product builds: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8191747 >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ >>> >>> Thanks >>> - Ioi >>> > From erik.osterlund at oracle.com Wed Nov 29 08:41:08 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 29 Nov 2017 09:41:08 +0100 Subject: RFR (S): 8191888: Refactor ClassLoaderData::remove_handle to use the Access API In-Reply-To: <1b1473cd-2235-8c98-617e-fac4c96c5b04@oracle.com> References: <5A1BD5B3.7030808@oracle.com> <5A1D546A.1040601@oracle.com> <1b1473cd-2235-8c98-617e-fac4c96c5b04@oracle.com> Message-ID: <09eb9609-b87c-a10d-fdba-192de4e535e3@oracle.com> Hi David, On 2017-11-28 22:28, David Holmes wrote: > Hi Erik, > > On 28/11/2017 10:19 PM, Erik ?sterlund wrote: >> Hi David, >> >> That is a good question. >> There is a description (written by a GC person, but still) for each >> decorator in the access.hpp file where the decorators are declared. I >> try to go in to some details there when you should and should not use >> a decorator. If we find that the text might have been biased from a >> GC perspective, we can update the comments. > > The comments serve as a reference, but I think we may lack a > "tutorial". :) As I think someone commented earlier re the Access API, > how do I know that, for example, IN_CONCURRENT_ROOT applies when I > have no idea what may or may not be scanned during safepoint ?? Would it be helpful if I wrote a little tutorial in a comment section in access.hpp? Hopefully, as the code gets more populated with calls to this thing, it starts to look more familiar as well. As for whether to use IN_CONCURRENT_ROOT or not, the basic idea is that if you do not know whether your root may be processed outside of safepoints, then use IN_CONCURRENT_ROOT conservatively. False positives are fine and only come with a small performance loss for some collectors (e.g. cause unnecessary G1 pre-write SATB barrier). False negatives on the other hand can be fatal (e.g. not performing necessary G1 pre-write SATB barrier). It is a little bit like using MO_SEQ_CST vs MO_ACQUIRE. Using MO_SEQ_CST when you are not sure is strictly safer but comes with a performance penalty. If you for example build thread-local values in value types, you might not want to take that performance penalty when you know your values are conceptually part of your stack and are processed in safepoints. As we move forward, we might change whether assuming roots may be processed concurrently, outside of safepoints, should be default or not. AFAIK, it is currently the rare exception rather than the rule. But perhaps things will change over time. Thanks, /Erik > But that's OT. > > Thanks, > David > > > >> The previous situation has been to use naked accesses, and then >> suddenly a GC person jumps in moaning about the lack of explicit #if >> INCLUDE_ALL_GCS if (UseG1) SATB barrier goo, or manual insertion of >> post-write barriers if a CAS was successful. There was no good way of >> knowing you had to manually insert these barriers, what barriers were >> offered to shared code, or what their semantics were. >> >> Now at least you have a set of decorators to choose from, with some >> automatic verification if you select nonsense decorators, and >> documentation in one place what they are used for and what they mean. >> So if you do not know what you need, at least you can read the >> descriptions and hopefully figure it out, which was impossible before. >> >> Having said that, I believe things like Coleen's OopHandle and >> PhantomOopHandle will build a layer on top of Access with possibly a >> few less details exposed that are even simpler to use for the >> conservative non-GC person that just wants the thing to work as >> simple as possible and wants to stay as far away from GC land as >> possible, which is sane. >> >> And if you feel uncertain, you can always loop me in any time, and I >> will probably say "there's a decorator for that". >> >> I hope this answers your question. >> >> Thanks, >> /Erik >> >> On 2017-11-28 07:59, David Holmes wrote: >>> Hi Erik, >>> >>> Is there a non-GC person's guide to what the different forms of >>> RootAccess mean and when to use them? Or do we expect a GC >>> person to always jump in to show where such things are needed? ;-) >>> >>> Thanks, >>> David >>> >>> On 27/11/2017 7:06 PM, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> The ClassLoaderData::remove_handle() member function currently uses >>>> explicit G1 SATB barriers to remove an oop from the root set, as >>>> these handles are not necessarily walked by the GC in a safepoint. >>>> Therefore G1 needs pre-write barriers. >>>> >>>> This should now be modeled as a >>>> RootAccess::oop_store instead. This maps to >>>> performing a pre-write SATB barrier with G1, but other GCs are free >>>> to do other things as necessary. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8191888/webrev.00/ >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8191888 >>>> >>>> Thanks, >>>> /Erik >> From david.holmes at oracle.com Wed Nov 29 10:00:56 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 29 Nov 2017 20:00:56 +1000 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Message-ID: Late to the party but ... On 23/11/2017 6:37 AM, Robbin Ehn wrote: > Hi all, please review. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 > > Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no > unexpected failure. > > Thanks, Robbin! > > diff -r 2cd1c2b03782 > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 01:12:23 2017 -0800 > +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 22 21:28:36 2017 +0100 > @@ -46,6 +46,11 @@ > ???????? {"MinRAMFraction",??????????? "2"}, > ???????? {"InitialRAMFraction",??????? "64"}, > ???????? {"AssumeMP",????????????????? "false"}, > +??????? {"UseMembar",???????????????? "true"}, > +??????? {"FastTLABRefill",??????????? "false"}, > +??????? {"DeferPollingPageLoopCount", "-1"}, > +??????? {"SafepointSpinBeforeYield",? "2000"}, > +??????? {"DeferPollingPageLoopCount", "4000"}, Why is DeferPollingPageLoopCount in there twice?? Cheers, David > ???????? // deprecated alias flags (see also aliased_jvm_flags): > ???????? {"DefaultMaxRAMFraction", "4"}, From robbin.ehn at oracle.com Wed Nov 29 10:10:10 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 29 Nov 2017 11:10:10 +0100 Subject: RFR: 8191782: Missing deprecated options in VMDeprecatedOptions.java In-Reply-To: References: <43f11705-acc4-7b00-5bc0-647719cdf44b@oracle.com> Message-ID: Thanks, David. I'll fix that! /Robbin On 2017-11-29 11:00, David Holmes wrote: > Late to the party but ... > > On 23/11/2017 6:37 AM, Robbin Ehn wrote: >> Hi all, please review. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8191782 >> >> Test: test/hotspot/jtreg/runtime/CommandLine/ and tier 1-5 with no unexpected failure. >> >> Thanks, Robbin! >> >> diff -r 2cd1c2b03782 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 01:12:23 2017 -0800 >> +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 22 21:28:36 2017 +0100 >> @@ -46,6 +46,11 @@ >> ????????? {"MinRAMFraction",??????????? "2"}, >> ????????? {"InitialRAMFraction",??????? "64"}, >> ????????? {"AssumeMP",????????????????? "false"}, >> +??????? {"UseMembar",???????????????? "true"}, >> +??????? {"FastTLABRefill",??????????? "false"}, >> +??????? {"DeferPollingPageLoopCount", "-1"}, >> +??????? {"SafepointSpinBeforeYield",? "2000"}, >> +??????? {"DeferPollingPageLoopCount", "4000"}, > > Why is DeferPollingPageLoopCount in there twice?? > > Cheers, > David > > >> ????????? // deprecated alias flags (see also aliased_jvm_flags): >> ????????? {"DefaultMaxRAMFraction", "4"}, From aph at redhat.com Wed Nov 29 10:16:44 2017 From: aph at redhat.com (Andrew Haley) Date: Wed, 29 Nov 2017 10:16:44 +0000 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: References: <5A1C06D8.1040405@oracle.com> Message-ID: <991db09f-64e8-4c4d-9b40-57cd0f028198@redhat.com> On 28/11/17 20:30, Kim Barrett wrote: > Should there be a comment justifying the cast to volatile? Perhaps > something like "volatile to keep access within guarded scope." I > don't need a new webrev if you add such a comment. Better: "This raw memory access may fault, so make sure it happens while within guarded scope." You really do have to spell this out. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From jini.george at oracle.com Wed Nov 29 11:51:05 2017 From: jini.george at oracle.com (Jini George) Date: Wed, 29 Nov 2017 17:21:05 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: <06a024a5-a1ff-5627-7d7d-1c84fac45f79@oracle.com> References: <06a024a5-a1ff-5627-7d7d-1c84fac45f79@oracle.com> Message-ID: <38e83c23-7a65-c664-c092-94fc131f9c2c@oracle.com> Thank you very much for the review, Serguei. Continuing to keep these constants in an interface would mean that they would need to have the 'final' qualifier. And that would mean that we would not be able to populate the values of these constants at runtime by reading those in from vmStructs using db.lookupIntConstant(). I have instead made these as inner classes in this new webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.01/ As David had pointed out wrt the review of 8190837: BasicType and BasicTypeSize should refer to HotSpot values, I realize that removal of the 'final' qualifier could cause parfait warnings, but since we would need to do that for the rest of the constants in the file also, I would prefer taking it up as a separate exercise. I had removed the PerfDataVariability constants since these were not used, and we are trying to remove unused code in SA to reduce the probability of bugs creeping in. Guess we can always put it back if we start checking the values against these constants. Let me know if you don't agree with this. Thanks, Jini. On 11/28/2017 12:38 PM, serguei.spitsyn at oracle.com wrote: > Hi Jini, > > It looks good to me. > > A couple of questions on the: > http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/PerfDataEntry.java.udiff.html > > + private static int U_None; > + private static int U_Bytes; > + private static int U_Ticks; > + private static int U_Events; > + private static int U_String; > + private static int U_Hertz; > +. . . + > + // PerfData Units enum > + U_None = db.lookupIntConstant("PerfData::U_None"); > + U_Bytes = db.lookupIntConstant("PerfData::U_Bytes"); > + U_Ticks = db.lookupIntConstant("PerfData::U_Ticks"); > + U_Events = db.lookupIntConstant("PerfData::U_Events"); > + U_String = db.lookupIntConstant("PerfData::U_String"); > + U_Hertz = db.lookupIntConstant("PerfData::U_Hertz");Before your fix > these private static fields were defined in the interface which I like: > > - public interface PerfDataUnits { > - public static final int U_None = 1; > - public static final int U_Bytes = 2; > - public static final int U_Ticks = 3; > - public static final int U_Events = 4; > - public static final int U_String = 5; > - public static final int U_Hertz = 6; > - } > > > I think, it'd still make sense to keep them in an interface or a small > class, > so they are not separate constants but a part of a common outer type. > > - public interface PerfDataVariability { > - public static final int V_Constant = 1; > - public static final int V_Monotonic = 2; > - public static final int V_Variable = 3; > - } I wonder it these constants can still be useful as the following > method returns one of them as a result which may need to be checked for > an exact value.Thanks, Serguei > > > > > On 11/21/17 02:34, Jini George wrote: >> Hello, >> >> Here's requesting reviews for some SA code cleanup. >> >> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >> Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >> >> The changes here are primarily to: >> >> 1. Remove unused IA64 SA code. >> 2. Make changes to avoid error-prone redefinition of hotspot constants >> in SA Java code. Instead read it in through vmStructs and >> db.lookupIntConstant(). >> 3. Make variable name changes in SA to align with the hotspot names. >> >> The changes are straightforward. >> >> Thanks much, >> Jini. >> > From robbin.ehn at oracle.com Wed Nov 29 12:19:40 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 29 Nov 2017 13:19:40 +0100 Subject: 8192072(one-liner): 8191782 fix for VMDeprecatedOptions.java missed DeferThrSuspendLoopCount and duplicated DeferPollingPageLoopCount Message-ID: Hi all, please review. Tested VMDeprecatedOptions which still passes. Issue: https://bugs.openjdk.java.net/browse/JDK-8192072 Old issue with bad fix: https://bugs.openjdk.java.net/browse/JDK-8191782 Thanks, Robbin diff -r bc1cffa26561 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 29 09:26:58 2017 +0900 +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 29 13:15:46 2017 +0100 @@ -51,5 +51,5 @@ {"DeferPollingPageLoopCount", "-1"}, {"SafepointSpinBeforeYield", "2000"}, - {"DeferPollingPageLoopCount", "4000"}, + {"DeferThrSuspendLoopCount", "4000"}, // deprecated alias flags (see also aliased_jvm_flags): From coleen.phillimore at oracle.com Wed Nov 29 12:24:30 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 29 Nov 2017 07:24:30 -0500 Subject: 8192072(one-liner): 8191782 fix for VMDeprecatedOptions.java missed DeferThrSuspendLoopCount and duplicated DeferPollingPageLoopCount In-Reply-To: References: Message-ID: Looks good, and trivial. thanks, Coleen On 11/29/17 7:19 AM, Robbin Ehn wrote: > Hi all, please review. > > Tested VMDeprecatedOptions which still passes. > Issue: > https://bugs.openjdk.java.net/browse/JDK-8192072 > Old issue with bad fix: > https://bugs.openjdk.java.net/browse/JDK-8191782 > > Thanks, Robbin > > diff -r bc1cffa26561 > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 29 09:26:58 2017 +0900 > +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > Wed Nov 29 13:15:46 2017 +0100 > @@ -51,5 +51,5 @@ > ???????? {"DeferPollingPageLoopCount", "-1"}, > ???????? {"SafepointSpinBeforeYield",? "2000"}, > -??????? {"DeferPollingPageLoopCount", "4000"}, > +??????? {"DeferThrSuspendLoopCount",? "4000"}, > > ???????? // deprecated alias flags (see also aliased_jvm_flags): From robbin.ehn at oracle.com Wed Nov 29 13:38:40 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 29 Nov 2017 14:38:40 +0100 Subject: 8192072(one-liner): 8191782 fix for VMDeprecatedOptions.java missed DeferThrSuspendLoopCount and duplicated DeferPollingPageLoopCount In-Reply-To: References: Message-ID: <3350904c-c762-4430-86e9-c8bb97aebe29@oracle.com> Thanks Coleen, Robbin. On 2017-11-29 13:24, coleen.phillimore at oracle.com wrote: > Looks good, and trivial. > thanks, > Coleen > > On 11/29/17 7:19 AM, Robbin Ehn wrote: >> Hi all, please review. >> >> Tested VMDeprecatedOptions which still passes. >> Issue: >> https://bugs.openjdk.java.net/browse/JDK-8192072 >> Old issue with bad fix: >> https://bugs.openjdk.java.net/browse/JDK-8191782 >> >> Thanks, Robbin >> >> diff -r bc1cffa26561 test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> --- a/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 29 09:26:58 2017 +0900 >> +++ b/test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java Wed Nov 29 13:15:46 2017 +0100 >> @@ -51,5 +51,5 @@ >> ???????? {"DeferPollingPageLoopCount", "-1"}, >> ???????? {"SafepointSpinBeforeYield",? "2000"}, >> -??????? {"DeferPollingPageLoopCount", "4000"}, >> +??????? {"DeferThrSuspendLoopCount",? "4000"}, >> >> ???????? // deprecated alias flags (see also aliased_jvm_flags): > From harold.seigel at oracle.com Wed Nov 29 13:40:18 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 29 Nov 2017 08:40:18 -0500 Subject: RFR 8167372: Add code to check for getting oops while thread is in native In-Reply-To: <96631f21-5266-59a7-bd70-cfbb2d7dfa37@redhat.com> References: <96631f21-5266-59a7-bd70-cfbb2d7dfa37@redhat.com> Message-ID: Please review this updated webrev that adds the assert suggested below by Aleksey. http://cr.openjdk.java.net/~hseigel/bug_8167372.2/webrev/index.html This update was re-tested with the same tests as the original webrev. Note that this change will be pushed post JDK-10. Thanks, Harold On 11/16/2017 10:06 AM, Aleksey Shipilev wrote: > On 11/16/2017 02:38 PM, harold seigel wrote: >> Hi, >> >> Please review this JDK-10 enhancement for bug JDK-8167372.? As described in the bug, this fix helps >> check for naked oops.? The fix was tested with JCK lang and VM tests, JTReg hotspot and many JTReg >> JDK tests, M5 tier1 - tier5 tests, and JPRT. >> >> Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8167372/webrev/index.html > This looks good. > > It probably makes sense to assert that JNIHandle::resolve path that unpacks the naked oops also has > thread in proper state? This would expose failures like in JDK-8191227 [1] better. > > Thanks, > -Aleksey > > [1] https://bugs.openjdk.java.net/browse/JDK-8191227 > From erik.osterlund at oracle.com Wed Nov 29 13:48:14 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 29 Nov 2017 14:48:14 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: References: <5A1C06D8.1040405@oracle.com> Message-ID: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Hi Kim, I know how you feel about casts, and I also saw how Andrew Haley wanted a more explicit comment about why it needs to be volatile. To make both of you happy, I thought I'd try to make the address (addr()) in MemoryAccess volatile T*. That way I can write a more explicit comment about why it is volatile in one single place, and make you happier to not cast the address to something else than it was declared. I hope this makes both of you happy. If not, I am okay with the old variant too, and write a comment that I copy around instead. Full webrev: http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ Incremental webrev: http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ Thanks, /Erik On 2017-11-28 21:30, Kim Barrett wrote: >> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund wrote: >> >> Hi, >> >> There is currently a bug when using unsafe accesses off-heap: >> >> 1) We write into a thread that we enable crash protection (using GuardUnsafeAccess): >> 2) We perform the access >> 3) We write into a thread that we disable crash protection (using ~GuardUnsafeAccess) >> >> The problem is that the crash protection stores are volatile, but the actual access is non-volatile. Compilers have different interpretation whether volatile - non-volatile accesses are allowed to reorder. MSVC is known to interpret such interactions as-if the volatile accesses have acquire/release semantics from the compiler point of view, and others such as GCC are known to reorder away freely. >> >> To prevent any issues, the accesses involved when using GuardUnsafeAccess should be at least volatile. >> This change makes the few remaining ones volatile. The JMM-volatile (SEQ_CST) accesses with crash protection already have stronger ordering than volatile and hence do not need changing. >> >> By making the address passed in to the Access API volatile, the MO_VOLATILE decorator is automatically set, which not surprisingly makes the access volatile. Therefore, the solution is to simply make the address passed in to Access volatile in this case. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8186787 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >> >> Thanks, >> /Erik > Not my first choice for a fix (you know how I feel about casts), but > it works, and is currently much simpler than my preferred solution. > > Should there be a comment justifying the cast to volatile? Perhaps > something like "volatile to keep access within guarded scope." I > don't need a new webrev if you add such a comment. > > Looks good. > From bob.vandette at oracle.com Wed Nov 29 14:02:56 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 29 Nov 2017 09:02:56 -0500 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> Message-ID: Misha, Which specific subtest failed on a 2 cpu system? I don?t see any subtests that should have failed. You should be able to pass any quota, period and share value to docker independent of the number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. I?m ok with the change but just wondering why the guards are necessary. Bob. > On Nov 28, 2017, at 7:50 PM, mikhailo wrote: > > Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java > based on the number of processors available. > > JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 > Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ > Testing: > 1. Locally: exercised the affected test locally on Linux-x64 > 2. Exercised the affected test on 2 processor machine > 3. Exercise the affected test via automated distributed test system > In progress > > Misha > From aph at redhat.com Wed Nov 29 14:25:28 2017 From: aph at redhat.com (Andrew Haley) Date: Wed, 29 Nov 2017 14:25:28 +0000 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: <58ba68ff-c6f9-e7b7-55c1-10a56d309839@redhat.com> On 29/11/17 13:48, Erik ?sterlund wrote: > I hope this makes both of you happy. :-) :-) :-) -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From dms at samersoff.net Wed Nov 29 14:43:01 2017 From: dms at samersoff.net (Dmitry Samersoff) Date: Wed, 29 Nov 2017 17:43:01 +0300 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: Erik, I like this variant too :) -Dmitry On 29.11.2017 16:48, Erik ?sterlund wrote: > Hi Kim, > > I know how you feel about casts, and I also saw how Andrew Haley wanted > a more explicit comment about why it needs to be volatile. > To make both of you happy, I thought I'd try to make the address > (addr()) in MemoryAccess volatile T*. That way I can write a more > explicit comment about why it is volatile in one single place, and make > you happier to not cast the address to something else than it was declared. > > I hope this makes both of you happy. If not, I am okay with the old > variant too, and write a comment that I copy around instead. > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ > > Incremental webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ > > Thanks, > /Erik > > On 2017-11-28 21:30, Kim Barrett wrote: >>> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund >>> wrote: >>> >>> Hi, >>> >>> There is currently a bug when using unsafe accesses off-heap: >>> >>> 1) We write into a thread that we enable crash protection (using >>> GuardUnsafeAccess): >>> 2) We perform the access >>> 3) We write into a thread that we disable crash protection (using >>> ~GuardUnsafeAccess) >>> >>> The problem is that the crash protection stores are volatile, but the >>> actual access is non-volatile. Compilers have different >>> interpretation whether volatile - non-volatile accesses are allowed >>> to reorder. MSVC is known to interpret such interactions as-if the >>> volatile accesses have acquire/release semantics from the compiler >>> point of view, and others such as GCC are known to reorder away freely. >>> >>> To prevent any issues, the accesses involved when using >>> GuardUnsafeAccess should be at least volatile. >>> This change makes the few remaining ones volatile. The JMM-volatile >>> (SEQ_CST) accesses with crash protection already have stronger >>> ordering than volatile and hence do not need changing. >>> >>> By making the address passed in to the Access API volatile, the >>> MO_VOLATILE decorator is automatically set, which not surprisingly >>> makes the access volatile. Therefore, the solution is to simply make >>> the address passed in to Access volatile in this case. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>> >>> Thanks, >>> /Erik >> Not my first choice for a fix (you know how I feel about casts), but >> it works, and is currently much simpler than my preferred solution. >> >> Should there be a comment justifying the cast to volatile?? Perhaps >> something like "volatile to keep access within guarded scope."? I >> don't need a new webrev if you add such a comment. >> >> Looks good. >> > From coleen.phillimore at oracle.com Wed Nov 29 15:03:34 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 29 Nov 2017 10:03:34 -0500 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: I like this version. Coleen On 11/29/17 8:48 AM, Erik ?sterlund wrote: > Hi Kim, > > I know how you feel about casts, and I also saw how Andrew Haley > wanted a more explicit comment about why it needs to be volatile. > To make both of you happy, I thought I'd try to make the address > (addr()) in MemoryAccess volatile T*. That way I can write a more > explicit comment about why it is volatile in one single place, and > make you happier to not cast the address to something else than it was > declared. > > I hope this makes both of you happy. If not, I am okay with the old > variant too, and write a comment that I copy around instead. > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ > > Incremental webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ > > Thanks, > /Erik > > On 2017-11-28 21:30, Kim Barrett wrote: >>> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund >>> wrote: >>> >>> Hi, >>> >>> There is currently a bug when using unsafe accesses off-heap: >>> >>> 1) We write into a thread that we enable crash protection (using >>> GuardUnsafeAccess): >>> 2) We perform the access >>> 3) We write into a thread that we disable crash protection (using >>> ~GuardUnsafeAccess) >>> >>> The problem is that the crash protection stores are volatile, but >>> the actual access is non-volatile. Compilers have different >>> interpretation whether volatile - non-volatile accesses are allowed >>> to reorder. MSVC is known to interpret such interactions as-if the >>> volatile accesses have acquire/release semantics from the compiler >>> point of view, and others such as GCC are known to reorder away freely. >>> >>> To prevent any issues, the accesses involved when using >>> GuardUnsafeAccess should be at least volatile. >>> This change makes the few remaining ones volatile. The JMM-volatile >>> (SEQ_CST) accesses with crash protection already have stronger >>> ordering than volatile and hence do not need changing. >>> >>> By making the address passed in to the Access API volatile, the >>> MO_VOLATILE decorator is automatically set, which not surprisingly >>> makes the access volatile. Therefore, the solution is to simply make >>> the address passed in to Access volatile in this case. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>> >>> Thanks, >>> /Erik >> Not my first choice for a fix (you know how I feel about casts), but >> it works, and is currently much simpler than my preferred solution. >> >> Should there be a comment justifying the cast to volatile? Perhaps >> something like "volatile to keep access within guarded scope." I >> don't need a new webrev if you add such a comment. >> >> Looks good. >> > From rkennke at redhat.com Wed Nov 29 15:38:49 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 29 Nov 2017 16:38:49 +0100 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> Message-ID: <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> Including hotspot-runtime-dev... I need one more review (esp. for the src/hotspot/share/services part): http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ Thanks, Roman > On 11/29/2017 09:39 AM, Roman Kennke wrote: >> Hi Erik, >> >> thanks for the review! >> >> I think this requires another reviewer from serviceability-dev. Who >> can I ping about this? > > You could try the hotspot-runtime-dev email list, although I suspect > most of the runtime developers are on serviceability-dev list as well... > > Thanks, > Erik > >> Roman >> >> >>> Hi Roman, >>> >>> Looks good now. Thanks for doing this. >>> >>> Thanks, >>> /Erik >>> >>> On 2017-11-28 22:26, Roman Kennke wrote: >>>> Hi Erik, >>>> >>>> thank you for reviewing! >>>> >>>> You are right. I think this is a leftover from when we tried to pass >>>> the GCMemoryManager* into the Generation constructor. The way it is >>>> done now (installing the GCMmemoryManager* later through >>>> set_memory_manager()) we can group all serviceability related set-up >>>> into initialize_serviceability(): >>>> >>>> Differential: >>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>> Full: >>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>> >>>> Notice that I hooked this up into CollectedHeap::post_initialize() >>>> and in doing so made that method concrete, and changed all subclass >>>> post_initialize() methods to call the super-class. >>>> >>>> Good now? >>>> >>>> Thanks, Roman >>>> >>>> >>>>> Hi Roman, >>>>> >>>>> This looks better now. Nice job. >>>>> I wonder though, is it possible to extract the creation of managers >>>>> and pools to a separate function for each collected heap? >>>>> Now I see managers are created in e.g. CMS constructor, and the >>>>> pools are created in CMSHeap::initialize(). Perhaps initialize >>>>> could call initialize_serviceability() that sets up those things, >>>>> so that they are nicely separated. Unless of course there is a good >>>>> reason why the presence of managers is needed earlier than that in >>>>> the bootstrapping. >>>>> >>>>> Otherwise I think this looks good! >>>>> >>>>> Thanks, >>>>> /Erik >>>>> >>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>> Hi Erik, >>>>>> >>>>>> Thanks for your review! >>>>>> >>>>>> All of the things that you mentioned should be addressed in the >>>>>> following changes: >>>>>> >>>>>> Differential: >>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>> Full: >>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>> >>>>>> Also, Erik D (H) was so kind to contribute an additional testcase, >>>>>> which is also included in the above patch. >>>>>> >>>>>> Ok now? >>>>>> >>>>>> Also, I need a review from serviceability-dev too! >>>>>> >>>>>> Thanks, >>>>>> Roman >>>>>> >>>>>> >>>>>>> 1) The code uses the "mgr" short name for "manager" in a bunch of >>>>>>> names. I would be happy if we could write out the whole thing >>>>>>> instead of the abbreviation. >>>>>>> 2) It would be great if a generation would have a pointer to its >>>>>>> manager, so we do not have to pass around the manager where we >>>>>>> already pass around the generation (such as >>>>>>> GenCollectedHeap::collect_generation). >>>>>>> The generation could be created first, then the pools, then the >>>>>>> managers, then do something like generation->set_memory_manager(x). >>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use camel case. >>>>>>> >>>>>>> Otherwise I think it looks good. >>>>>>> >>>>>>> Thanks, >>>>>>> /Erik >>>>>>> >>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>> Erik implemented a few more refactorings and touch-ups, and here >>>>>>>> is our final (pending reviews) webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>> >>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>> gc-end-message, avoids dragging the GCMemoryManager through >>>>>>>> Generation and a few little related refactorings. >>>>>>>> >>>>>>>> Ok to push now? >>>>>>>> >>>>>>>> Roman >>>>>>>> >>>>>>>>> After a few more discussions with Erik I made some more changes. >>>>>>>>> >>>>>>>>> MemoryService is now unaware of the number and meaning of GC >>>>>>>>> memory managers (minor vs major). This should be better for GCs >>>>>>>>> that don't make that distinction and/or use more different GCs >>>>>>>>> (e.g. minor, intermediate, full). >>>>>>>>> >>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>> - GCMemoryManager now has gc_end_message() which is used by >>>>>>>>> GCNotifier::pushNotification(). >>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now accept a >>>>>>>>> GCMemoryManager* instead of bull full_gc >>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>> - Generation now knows about the corresponding GCMemoryManager >>>>>>>>> >>>>>>>>> Please review the full change: >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>> >>>>>>>>> Thanks, Roman >>>>>>>>> >>>>>>>>> >>>>>>>>>> I had some off-band discussions with Erik Helin and re-did >>>>>>>>>> most of the changeset: >>>>>>>>>> - The GC interface now resides in CollectedHeap, specifically >>>>>>>>>> the two methods memory_managers() and memory_pools(), and is >>>>>>>>>> implemented in the various concrete subclasses. >>>>>>>>>> - Both methods return (by value) a >>>>>>>>>> GrowableArray and GrowableArray >>>>>>>>>> respectively. Returning a stack-allocated GrowableArray seemed >>>>>>>>>> least complicated (avoid explicit cleanup of short-lived array >>>>>>>>>> object), and most future-proof, e.g. currently there is an >>>>>>>>>> implicit expectation to get 2 GCMemoryManagers, even though >>>>>>>>>> some GCs don't necessarily have two. The API allows for easy >>>>>>>>>> extension of the situation. >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>> >>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>> Serviceability team. >>>>>>>>>> >>>>>>>>>> Roman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Currently, there's lots of GC specific code sprinkled over >>>>>>>>>>> src/hotspot/share/services. This change introduces a GC >>>>>>>>>>> interface for that, and moves all GC specific code to their >>>>>>>>>>> respective src/hotspot/share/gc directory. >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none showed >>>>>>>>>>> regressions >>>>>>>>>>> >>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>> >>>>>>>>>>> What do you think? >>>>>>>>>>> >>>>>>>>>>> Roman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> From ioi.lam at oracle.com Wed Nov 29 19:07:15 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 29 Nov 2017 11:07:15 -0800 Subject: RFR[XS] 8191747 [TESTBUG] runtime/appcds/DumpClassList.java and ProhibitedPackage.java fail on product bits In-Reply-To: References: <40c4d08f-669e-ddd2-958e-a03b32e43bd3@oracle.com> Message-ID: <13d990fd-6c8c-e5f0-70a1-9d859f6efe48@oracle.com> On 11/29/17 12:02 AM, Volker Simonis wrote: > On Tue, Nov 28, 2017 at 7:51 PM, Ioi Lam wrote: >> >> On 11/28/17 10:21 AM, Volker Simonis wrote: >>> Hi Ioi, >>> >>> looks good! >>> >>> I was a little confused first that "-Xlog:cds" was not set as argument >>> in ProhibitedPackage.java >>> But I saw now that it is set by default in TestCommon for every >>> invocation of "dump()". Maybe you could mention that in >>> ProhibitedPackage.java in the comment before the execution of the dump >>> where you parse the "[cds]" logs? >> Hi Volker, >> >> Thanks for noticing this. I added -Xlog:cds unconditionally so if TestCommon >> is changed in the future, this test will still work: >> >> // Make sure a class in a prohibited package for a custom >> loader >> // will be ignored during dumping. >> - TestCommon.dump(appJar, >> - classlist, >> - "-XX:+PrintSystemDictionaryAtExit") >> + TestCommon.dump(appJar, classlist, "-Xlog:cds") >> .shouldContain("Dumping") >> - .shouldNotContain("java.lang.Prohibited") >> + .shouldContain("[cds] Prohibited package for non-bootstrap >> classes: java/lang/Prohibited.class") >> .shouldHaveExitValue(0); >> >> > Looks good. Thumbs up! > > I've just realized that the JBS entry for this bug is not publicly > visible. Maybe you can also change that? Done. I've changed the bug to be publicly visible. Thanks - Ioi > Regards, > Volker > >> Thanks >> - Ioi >> >> >>> Regards, >>> Volker >>> >>> >>> On Tue, Nov 28, 2017 at 6:58 PM, Ioi Lam wrote: >>>> Hi, >>>> >>>> Here's a fix for these two tests failing in product builds. The fix is to >>>> replace the use of debug flag PrintSystemDictionaryAtExit flag with >>>> -Xlog:class+load, which is available in product builds: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8191747 >>>> >>>> http://cr.openjdk.java.net/~iklam/jdk10/8191747-avoid-PrintSystemDictionaryAtExit-in-tests.v01/ >>>> >>>> Thanks >>>> - Ioi >>>> From mikhailo.seledtsov at oracle.com Wed Nov 29 21:10:04 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 29 Nov 2017 13:10:04 -0800 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> Message-ID: <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> Hi Bob, The failure was caused by invoking this test case: ??? testCpuShares(4096, 4); ??? The test case sets --cpu-shares to 4096, expects active processor count to be 4; ==> actual APC is 2. Command: ??? docker run --tty=true --rm --cpu-shares=4096 jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version Detailed log is attached at: https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt Once I saw this issue, I decided to put limits on other test cases based on the number of processors available on a machine, just to be on a safe side. Perhaps a better approach here is to change the expected APC inside the testCpuShares() to be no greater than max number of available processors? Misha On 11/29/2017 06:02 AM, Bob Vandette wrote: > Misha, > > Which specific subtest failed on a 2 cpu system? > > I don?t see any subtests that should have failed. You should be able > to pass any quota, period and share value to docker independent of the > number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. > > I?m ok with the change but just wondering why the guards are necessary. > > Bob. > >> On Nov 28, 2017, at 7:50 PM, mikhailo wrote: >> >> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java >> based on the number of processors available. >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >> Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >> Testing: >> 1. Locally: exercised the affected test locally on Linux-x64 >> 2. Exercised the affected test on 2 processor machine >> 3. Exercise the affected test via automated distributed test system >> In progress >> >> Misha >> From daniel.daugherty at oracle.com Wed Nov 29 21:16:22 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 29 Nov 2017 16:16:22 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp Message-ID: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> Greetings, Coleen, this is one of your Thread-SMR follow-up suggestions so I need to hear from you on this thread. Thanks! I have a simple cleanup fix for Thread-SMR. The bug is: ??? JDK-8191787 move private inline functions from thread.inline.hpp -> thread.cpp ??? https://bugs.openjdk.java.net/browse/JDK-8191787 This fix is pure code motion: - moving inline functions from thread.inline.hpp -> thread.cpp - making a few functions in thread.hpp private instead of public Here is the webrev URL: http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 This fix was (over) tested with a Mach5 tier[1-5] run. There were no unexpected test failures. Thanks, in advance, for any comments, questions or suggestions. Dan From bob.vandette at oracle.com Wed Nov 29 21:29:28 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 29 Nov 2017 16:29:28 -0500 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> Message-ID: <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> > On Nov 29, 2017, at 4:10 PM, mikhailo wrote: > > Hi Bob, > > The failure was caused by invoking this test case: > > testCpuShares(4096, 4); > The test case sets --cpu-shares to 4096, expects active processor count to be 4; ==> actual APC is 2. Ahh, that makes sense. I thought docker was complaining due to the arguments being passed going beyond the available cpus. > > Command: > > docker run --tty=true --rm --cpu-shares=4096 jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version > > Detailed log is attached at: > > https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt > > > Once I saw this issue, I decided to put limits on other test cases based on the number of processors available on a machine, just to be on a safe side. > > Perhaps a better approach here is to change the expected APC inside the testCpuShares() to be no greater than max number of available processors? Yes, the selection algorithm returns the smallest number of computed shares, quotas or active processors. Using this approach, you?d actually be validating the algorithm more precisely. Bob. > > > Misha > > > > > > > On 11/29/2017 06:02 AM, Bob Vandette wrote: >> Misha, >> >> Which specific subtest failed on a 2 cpu system? >> >> I don?t see any subtests that should have failed. You should be able >> to pass any quota, period and share value to docker independent of the >> number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. >> >> I?m ok with the change but just wondering why the guards are necessary. >> >> Bob. >> >>> On Nov 28, 2017, at 7:50 PM, mikhailo wrote: >>> >>> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java >>> based on the number of processors available. >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>> Testing: >>> 1. Locally: exercised the affected test locally on Linux-x64 >>> 2. Exercised the affected test on 2 processor machine >>> 3. Exercise the affected test via automated distributed test system >>> In progress >>> >>> Misha >>> > From mikhailo.seledtsov at oracle.com Wed Nov 29 21:32:20 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 29 Nov 2017 13:32:20 -0800 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> Message-ID: <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> On 11/29/2017 01:29 PM, Bob Vandette wrote: >> On Nov 29, 2017, at 4:10 PM, mikhailo wrote: >> >> Hi Bob, >> >> The failure was caused by invoking this test case: >> >> testCpuShares(4096, 4); >> The test case sets --cpu-shares to 4096, expects active processor count to be 4; ==> actual APC is 2. > Ahh, that makes sense. I thought docker was complaining due to the arguments being passed going > beyond the available cpus. > >> Command: >> >> docker run --tty=true --rm --cpu-shares=4096 jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >> >> Detailed log is attached at: >> >> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >> >> >> Once I saw this issue, I decided to put limits on other test cases based on the number of processors available on a machine, just to be on a safe side. >> >> Perhaps a better approach here is to change the expected APC inside the testCpuShares() to be no greater than max number of available processors? > Yes, the selection algorithm returns the smallest number of computed shares, quotas or active processors. Using this approach, you?d actually be validating the algorithm more precisely. OK. I will update the tests to do this kind of validation, and post a new webrev. Misha > > Bob. > > >> >> Misha >> >> >> >> >> >> >> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>> Misha, >>> >>> Which specific subtest failed on a 2 cpu system? >>> >>> I don?t see any subtests that should have failed. You should be able >>> to pass any quota, period and share value to docker independent of the >>> number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. >>> >>> I?m ok with the change but just wondering why the guards are necessary. >>> >>> Bob. >>> >>>> On Nov 28, 2017, at 7:50 PM, mikhailo wrote: >>>> >>>> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java >>>> based on the number of processors available. >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>> Testing: >>>> 1. Locally: exercised the affected test locally on Linux-x64 >>>> 2. Exercised the affected test on 2 processor machine >>>> 3. Exercise the affected test via automated distributed test system >>>> In progress >>>> >>>> Misha >>>> From coleen.phillimore at oracle.com Wed Nov 29 22:04:49 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 29 Nov 2017 17:04:49 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> Message-ID: <444b5c64-85e7-43c3-8d18-c25140806aac@oracle.com> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0/src/hotspot/share/runtime/thread.hpp.udiff.html Can you put a section with the static data member declarations, then a blank line and then have a section with the functions??? We don't usually mix them like that (maybe in thread.hpp because it's the kitchen sink but not everywhere else).? It makes it hard to read. Otherwise, looks fine.? Thank you for moving these! Coleen On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: > Greetings, > > Coleen, this is one of your Thread-SMR follow-up suggestions so I need > to hear from you on this thread. Thanks! > > I have a simple cleanup fix for Thread-SMR. The bug is: > > ??? JDK-8191787 move private inline functions from thread.inline.hpp > -> thread.cpp > ??? https://bugs.openjdk.java.net/browse/JDK-8191787 > > This fix is pure code motion: > > - moving inline functions from thread.inline.hpp -> thread.cpp > - making a few functions in thread.hpp private instead of public > > Here is the webrev URL: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 > > This fix was (over) tested with a Mach5 tier[1-5] run. There were no > unexpected test failures. > > Thanks, in advance, for any comments, questions or suggestions. > > Dan From daniel.daugherty at oracle.com Wed Nov 29 22:11:55 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 29 Nov 2017 17:11:55 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <444b5c64-85e7-43c3-8d18-c25140806aac@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <444b5c64-85e7-43c3-8d18-c25140806aac@oracle.com> Message-ID: <011f4d27-2c1e-361c-4776-8aa7cba3bca5@oracle.com> On 11/29/17 5:04 PM, coleen.phillimore at oracle.com wrote: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0/src/hotspot/share/runtime/thread.hpp.udiff.html > > > Can you put a section with the static data member declarations, then a > blank line and then have a section with the functions??? We don't > usually mix them like that (maybe in thread.hpp because it's the > kitchen sink but not everywhere else).? It makes it hard to read. We're trying to keep all the "stuff" associated with a field together. It's definitely not a style typically in use in HotSpot, but we're experimenting with it because thread.hpp is currently a kitchen sink collection. Some folks would say it is a mess... :-) I'm going to take a look this cleanup bug also: JDK-8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> threadSMR.[ch]pp https://bugs.openjdk.java.net/browse/JDK-8191789 since it will primarily be "code motion" fix also. Keeping "stuff" together will make that cleanup bug easier... Would you be okay with this fix (8191787) as it is? > Otherwise, looks fine.? Thank you for moving these! Thanks! Dan > > Coleen > > On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >> to hear from you on this thread. Thanks! >> >> I have a simple cleanup fix for Thread-SMR. The bug is: >> >> ??? JDK-8191787 move private inline functions from thread.inline.hpp >> -> thread.cpp >> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> This fix is pure code motion: >> >> - moving inline functions from thread.inline.hpp -> thread.cpp >> - making a few functions in thread.hpp private instead of public >> >> Here is the webrev URL: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >> >> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >> unexpected test failures. >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan > From mikhailo.seledtsov at oracle.com Wed Nov 29 22:15:23 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 29 Nov 2017 14:15:23 -0800 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Message-ID: Hi Ioi, ? Overall the change looks good. I have just one comment: 1. In test/lib/Platform.java you note in the comment that the condition should match one in ClassListParser::load_class_from_source(). However, there is not reverse reference. That is, there is no note/comment in ClassListParser::load_class_from_source() to update test.lib.Platform.areCustomLoadersSupportedForCDS() if the condition changes in load_class_from_source(). I can think of couple of ways to address this: A. Simple quick way, just add the comment in classListParser.cpp that test library should be updated if this condition is updated. B. Add a white box API similar to WhiteBox.isCDSIncludedInVmBuild(); and have a method in VM that would compute the result based on the platform conditionals. The option B is more robust in my view, but I will not have strong objection to option A either. Also, I understand that option B can result in some sort of complex dependencies. Thank you, Misha On 11/27/2017 01:42 PM, Ioi Lam wrote: > Hi, > > Please review this change to support AppCDS on all 64-bit Linux > platforms as well as AIX. > > ??? https://bugs.openjdk.java.net/browse/JDK-8191927 > http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ > > > The patch must be applied on top of the AppCDS patch: > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ > > The patch was initially contributed by Volker Simonis. I modified it a > little to avoid repeating the platform enumerations in all the test > cases. Instead, I added a new test property, so the tests that require > AppCDS support for custom loaders can be written as: > > ??? * @requires vm.cds.custom.loaders > > I've tested on Linux manually, and I am now running on all of the > Oracle-supported platforms using our internal test harness. > > Thanks > - Ioi From manc at google.com Wed Nov 29 23:21:05 2017 From: manc at google.com (Man Cao) Date: Wed, 29 Nov 2017 15:21:05 -0800 Subject: Make reserved_size for compressed class space and metaspace respect the ergo-initialized CompressedClassSpaceSize flag value Message-ID: Hello, This patch is a follow-up fix for https://bugs.openjdk.java.net/browse/JDK-8087291 This patch moves the call to set_compressed_class_space_size() after the flag value for CompressedClassSpaceSize is ergo-initialized, fixing the issue that the reserved size for compressed class space and metaspace is excessively large when MaxMetaspaceSize is set to a small value. More discussion about it is available here: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-November/025200.html This code patch is attached. Could anyone review and/or sponsor this patch? Best, Man -------------- next part -------------- A non-text attachment was scrubbed... Name: jdk10hs.patch Type: text/x-patch Size: 970 bytes Desc: not available URL: From ioi.lam at oracle.com Wed Nov 29 23:29:29 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 29 Nov 2017 15:29:29 -0800 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Message-ID: On 11/29/17 2:15 PM, mikhailo wrote: > Hi Ioi, > > ? Overall the change looks good. I have just one comment: > > 1. In test/lib/Platform.java you note in the comment that the > condition should match one in > ClassListParser::load_class_from_source(). However, there is not > reverse reference. That is, there is no note/comment in > ClassListParser::load_class_from_source() to update > test.lib.Platform.areCustomLoadersSupportedForCDS() if the condition > changes in load_class_from_source(). > > I can think of couple of ways to address this: > > A. Simple quick way, just add the comment in classListParser.cpp that > test library should be updated if this condition is updated. > > B. Add a white box API similar to WhiteBox.isCDSIncludedInVmBuild(); > and have a method in VM that would compute the result based on the > platform conditionals. > > The option B is more robust in my view, but I will not have strong > objection to option A either. Also, I understand that option B can > result in some sort of complex dependencies. > Hi Misha, Option B will require that the VM has -XX:+WhiteBoxAPI and the whitebox.jar in the classpath, but the Platform class can be used without these conditions. So I don't think B is doable. I'll add the comments as you suggested in A. Thanks - Ioi > > Thank you, > > Misha > > > > > On 11/27/2017 01:42 PM, Ioi Lam wrote: >> Hi, >> >> Please review this change to support AppCDS on all 64-bit Linux >> platforms as well as AIX. >> >> ??? https://bugs.openjdk.java.net/browse/JDK-8191927 >> http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ >> >> >> The patch must be applied on top of the AppCDS patch: >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ >> >> The patch was initially contributed by Volker Simonis. I modified it >> a little to avoid repeating the platform enumerations in all the test >> cases. Instead, I added a new test property, so the tests that >> require AppCDS support for custom loaders can be written as: >> >> ??? * @requires vm.cds.custom.loaders >> >> I've tested on Linux manually, and I am now running on all of the >> Oracle-supported platforms using our internal test harness. >> >> Thanks >> - Ioi > From mikhailo.seledtsov at oracle.com Wed Nov 29 23:31:39 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 29 Nov 2017 15:31:39 -0800 Subject: RFR[S] 8191927 Enable AppCDS for custom loaders on all 64-bit Linux and AIX In-Reply-To: References: <4987e05a-45aa-7e3a-4aaf-bbe4319d1465@oracle.com> Message-ID: <9d435a6b-46ed-6268-7b04-b29bfdc31f5e@oracle.com> Thank you, Consider it reviewed for my part; no need to post a new webrev for the added comment. Misha On 11/29/2017 03:29 PM, Ioi Lam wrote: > > > On 11/29/17 2:15 PM, mikhailo wrote: >> Hi Ioi, >> >> ? Overall the change looks good. I have just one comment: >> >> 1. In test/lib/Platform.java you note in the comment that the >> condition should match one in >> ClassListParser::load_class_from_source(). However, there is not >> reverse reference. That is, there is no note/comment in >> ClassListParser::load_class_from_source() to update >> test.lib.Platform.areCustomLoadersSupportedForCDS() if the condition >> changes in load_class_from_source(). >> >> I can think of couple of ways to address this: >> >> A. Simple quick way, just add the comment in classListParser.cpp that >> test library should be updated if this condition is updated. >> >> B. Add a white box API similar to WhiteBox.isCDSIncludedInVmBuild(); >> and have a method in VM that would compute the result based on the >> platform conditionals. >> >> The option B is more robust in my view, but I will not have strong >> objection to option A either. Also, I understand that option B can >> result in some sort of complex dependencies. >> > > Hi Misha, > > Option B will require that the VM has -XX:+WhiteBoxAPI and the > whitebox.jar in the classpath, but the Platform class can be used > without these conditions. So I don't think B is doable. I'll add the > comments as you suggested in A. > > Thanks > - Ioi > >> >> Thank you, >> >> Misha >> >> >> >> >> On 11/27/2017 01:42 PM, Ioi Lam wrote: >>> Hi, >>> >>> Please review this change to support AppCDS on all 64-bit Linux >>> platforms as well as AIX. >>> >>> ??? https://bugs.openjdk.java.net/browse/JDK-8191927 >>> http://cr.openjdk.java.net/~iklam/jdk10/8191927-appcds-custom-loader-for-more-platforms.v01/ >>> >>> >>> The patch must be applied on top of the AppCDS patch: >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds.v03/ >>> >>> The patch was initially contributed by Volker Simonis. I modified it >>> a little to avoid repeating the platform enumerations in all the >>> test cases. Instead, I added a new test property, so the tests that >>> require AppCDS support for custom loaders can be written as: >>> >>> ??? * @requires vm.cds.custom.loaders >>> >>> I've tested on Linux manually, and I am now running on all of the >>> Oracle-supported platforms using our internal test harness. >>> >>> Thanks >>> - Ioi >> > From mikhailo.seledtsov at oracle.com Thu Nov 30 00:08:13 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 29 Nov 2017 16:08:13 -0800 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> Message-ID: <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> Here is an updated webrev: ??? http://cr.openjdk.java.net/~mseledtsov/8191943.01/ Misha On 11/29/2017 01:32 PM, mikhailo wrote: > > On 11/29/2017 01:29 PM, Bob Vandette wrote: >>> On Nov 29, 2017, at 4:10 PM, mikhailo >>> wrote: >>> >>> Hi Bob, >>> >>> The failure was caused by invoking this test case: >>> >>> ???? testCpuShares(4096, 4); >>> ???? The test case sets --cpu-shares to 4096, expects active >>> processor count to be 4; ==> actual APC is 2. >> Ahh, that makes sense.? I thought docker was complaining due to the >> arguments being passed going >> beyond the available cpus. >> >>> Command: >>> >>> ???? docker run --tty=true --rm --cpu-shares=4096 >>> jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >>> >>> Detailed log is attached at: >>> >>> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >>> >>> >>> >>> Once I saw this issue, I decided to put limits on other test cases >>> based on the number of processors available on a machine, just to be >>> on a safe side. >>> >>> Perhaps a better approach here is to change the expected APC inside >>> the testCpuShares() to be no greater than max number of available >>> processors? >> Yes, the selection algorithm returns the smallest number of computed >> shares, quotas or active processors.? Using this approach, you?d >> actually be validating the algorithm more precisely. > OK. I will update the tests to do this kind of validation, and post a > new webrev. > > > Misha >> >> Bob. >> >> >>> >>> Misha >>> >>> >>> >>> >>> >>> >>> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>>> Misha, >>>> >>>> Which specific subtest failed on a 2 cpu system? >>>> >>>> I don?t see any subtests that should have failed.? You should be able >>>> to pass any quota, period and share value to docker independent of the >>>> number of CPUs.? testCpus and testAPCCombo don?t try to test more >>>> than 2 cpus. >>>> >>>> I?m ok with the change but just wondering why the guards are >>>> necessary. >>>> >>>> Bob. >>>> >>>>> On Nov 28, 2017, at 7:50 PM, mikhailo >>>>> wrote: >>>>> >>>>> Could you, please, review this simple fix? It limits test cases in >>>>> TestCPUAwareness.java >>>>> based on the number of processors available. >>>>> >>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>>> ???? Testing: >>>>> ???????? 1. Locally: exercised the affected test locally on Linux-x64 >>>>> ???????? 2. Exercised the affected test on 2 processor machine >>>>> ???????? 3. Exercise the affected test via automated distributed >>>>> test system >>>>> ??????????? In progress >>>>> >>>>> Misha >>>>> > From bob.vandette at oracle.com Thu Nov 30 00:38:21 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 29 Nov 2017 19:38:21 -0500 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> Message-ID: <0311B749-680B-4812-AD8F-B3512CA98C94@oracle.com> I think you?ll have problems on a single CPU box since the ?cpuset-cpus argument to docker will most likely fail when you try to set it to ?0,1?. 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2); Bob. > On Nov 29, 2017, at 7:08 PM, mikhailo wrote: > > Here is an updated webrev: > > http://cr.openjdk.java.net/~mseledtsov/8191943.01/ > > > Misha > > > On 11/29/2017 01:32 PM, mikhailo wrote: >> >> On 11/29/2017 01:29 PM, Bob Vandette wrote: >>>> On Nov 29, 2017, at 4:10 PM, mikhailo wrote: >>>> >>>> Hi Bob, >>>> >>>> The failure was caused by invoking this test case: >>>> >>>> testCpuShares(4096, 4); >>>> The test case sets --cpu-shares to 4096, expects active processor count to be 4; ==> actual APC is 2. >>> Ahh, that makes sense. I thought docker was complaining due to the arguments being passed going >>> beyond the available cpus. >>> >>>> Command: >>>> >>>> docker run --tty=true --rm --cpu-shares=4096 jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >>>> >>>> Detailed log is attached at: >>>> >>>> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >>>> >>>> >>>> Once I saw this issue, I decided to put limits on other test cases based on the number of processors available on a machine, just to be on a safe side. >>>> >>>> Perhaps a better approach here is to change the expected APC inside the testCpuShares() to be no greater than max number of available processors? >>> Yes, the selection algorithm returns the smallest number of computed shares, quotas or active processors. Using this approach, you?d actually be validating the algorithm more precisely. >> OK. I will update the tests to do this kind of validation, and post a new webrev. >> >> >> Misha >>> >>> Bob. >>> >>> >>>> >>>> Misha >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>>>> Misha, >>>>> >>>>> Which specific subtest failed on a 2 cpu system? >>>>> >>>>> I don?t see any subtests that should have failed. You should be able >>>>> to pass any quota, period and share value to docker independent of the >>>>> number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. >>>>> >>>>> I?m ok with the change but just wondering why the guards are necessary. >>>>> >>>>> Bob. >>>>> >>>>>> On Nov 28, 2017, at 7:50 PM, mikhailo wrote: >>>>>> >>>>>> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java >>>>>> based on the number of processors available. >>>>>> >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>>>> Testing: >>>>>> 1. Locally: exercised the affected test locally on Linux-x64 >>>>>> 2. Exercised the affected test on 2 processor machine >>>>>> 3. Exercise the affected test via automated distributed test system >>>>>> In progress >>>>>> >>>>>> Misha >>>>>> >> > From coleen.phillimore at oracle.com Thu Nov 30 03:46:45 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 29 Nov 2017 22:46:45 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <011f4d27-2c1e-361c-4776-8aa7cba3bca5@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <444b5c64-85e7-43c3-8d18-c25140806aac@oracle.com> <011f4d27-2c1e-361c-4776-8aa7cba3bca5@oracle.com> Message-ID: Yes, definitely the "kitchen sink collection".??? I meant instead of : // Safe Memory Reclamation (SMR) support: + // The coordination between Threads::release_stable_list() and + // Threads::smr_delete() uses the smr_delete_lock in order to + // reduce the traffic on the Threads_lock. static Monitor* _smr_delete_lock; + static Monitor* smr_delete_lock() { return _smr_delete_lock; } // The '_cnt', '_max' and '_times" fields are enabled via // -XX:+EnableThreadSMRStatistics (see thread.cpp for a // description about each field): static uint _smr_delete_lock_wait_cnt; static uint _smr_delete_lock_wait_max; + // The smr_delete_notify flag is used for proper double-check + // locking in order to reduce the traffic on the smr_delete_lock. static volatile uint _smr_delete_notify; + static bool smr_delete_notify(); + static void set_smr_delete_notify(); + static void clear_smr_delete_notify(); static volatile uint _smr_deleted_thread_cnt; + static void inc_smr_deleted_thread_cnt(); static volatile uint _smr_deleted_thread_time_max; + static void update_smr_deleted_thread_time_max(uint new_value); static volatile uint _smr_deleted_thread_times; + static void add_smr_deleted_thread_times(uint add_value); static ThreadsList* volatile _smr_java_thread_list; static ThreadsList* get_smr_java_thread_list(); static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); static uint64_t _smr_java_thread_list_alloc_cnt; static uint64_t _smr_java_thread_list_free_cnt; To have the below.? They're all together but fields and functions have been separated. // Safe Memory Reclamation (SMR) support: + // The coordination between Threads::release_stable_list() and + // Threads::smr_delete() uses the smr_delete_lock in order to + // reduce the traffic on the Threads_lock. static Monitor* _smr_delete_lock; // The '_cnt', '_max' and '_times" fields are enabled via // -XX:+EnableThreadSMRStatistics (see thread.cpp for a // description about each field): static uint _smr_delete_lock_wait_cnt; static uint _smr_delete_lock_wait_max; + // The smr_delete_notify flag is used for proper double-check + // locking in order to reduce the traffic on the smr_delete_lock. static volatile uint _smr_delete_notify; static volatile uint _smr_deleted_thread_cnt; static volatile uint _smr_deleted_thread_time_max; static volatile uint _smr_deleted_thread_times; static ThreadsList* volatile _smr_java_thread_list; static uint64_t _smr_java_thread_list_alloc_cnt; static uint64_t _smr_java_thread_list_free_cnt; + static Monitor* smr_delete_lock() { return _smr_delete_lock; } + static bool smr_delete_notify(); + static void set_smr_delete_notify(); + static void clear_smr_delete_notify(); + static void inc_smr_deleted_thread_cnt(); + static void update_smr_deleted_thread_time_max(uint new_value); + static void add_smr_deleted_thread_times(uint add_value); static ThreadsList* get_smr_java_thread_list(); static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); Thanks, Coleen On 11/29/17 5:11 PM, Daniel D. Daugherty wrote: > On 11/29/17 5:04 PM, coleen.phillimore at oracle.com wrote: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0/src/hotspot/share/runtime/thread.hpp.udiff.html >> >> >> Can you put a section with the static data member declarations, then >> a blank line and then have a section with the functions? We don't >> usually mix them like that (maybe in thread.hpp because it's the >> kitchen sink but not everywhere else).? It makes it hard to read. > > We're trying to keep all the "stuff" associated with a field together. > It's definitely not a style typically in use in HotSpot, but we're > experimenting with it because thread.hpp is currently a kitchen sink > collection. Some folks would say it is a mess... :-) > > I'm going to take a look this cleanup bug also: > > JDK-8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> > threadSMR.[ch]pp > https://bugs.openjdk.java.net/browse/JDK-8191789 > > since it will primarily be "code motion" fix also. Keeping "stuff" > together will make that cleanup bug easier... > > Would you be okay with this fix (8191787) as it is? > > >> Otherwise, looks fine.? Thank you for moving these! > > Thanks! > > Dan > > >> >> Coleen >> >> On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >>> to hear from you on this thread. Thanks! >>> >>> I have a simple cleanup fix for Thread-SMR. The bug is: >>> >>> ??? JDK-8191787 move private inline functions from thread.inline.hpp >>> -> thread.cpp >>> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >>> >>> This fix is pure code motion: >>> >>> - moving inline functions from thread.inline.hpp -> thread.cpp >>> - making a few functions in thread.hpp private instead of public >>> >>> Here is the webrev URL: >>> >>> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >>> >>> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >>> unexpected test failures. >>> >>> Thanks, in advance, for any comments, questions or suggestions. >>> >>> Dan >> > From david.holmes at oracle.com Thu Nov 30 04:44:48 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 14:44:48 +1000 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: <77db1e4a-77ad-5fe3-afe0-65e46e38c1f5@oracle.com> +1 :) David On 29/11/2017 11:48 PM, Erik ?sterlund wrote: > Hi Kim, > > I know how you feel about casts, and I also saw how Andrew Haley wanted > a more explicit comment about why it needs to be volatile. > To make both of you happy, I thought I'd try to make the address > (addr()) in MemoryAccess volatile T*. That way I can write a more > explicit comment about why it is volatile in one single place, and make > you happier to not cast the address to something else than it was declared. > > I hope this makes both of you happy. If not, I am okay with the old > variant too, and write a comment that I copy around instead. > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ > > Incremental webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ > > Thanks, > /Erik > > On 2017-11-28 21:30, Kim Barrett wrote: >>> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund >>> wrote: >>> >>> Hi, >>> >>> There is currently a bug when using unsafe accesses off-heap: >>> >>> 1) We write into a thread that we enable crash protection (using >>> GuardUnsafeAccess): >>> 2) We perform the access >>> 3) We write into a thread that we disable crash protection (using >>> ~GuardUnsafeAccess) >>> >>> The problem is that the crash protection stores are volatile, but the >>> actual access is non-volatile. Compilers have different >>> interpretation whether volatile - non-volatile accesses are allowed >>> to reorder. MSVC is known to interpret such interactions as-if the >>> volatile accesses have acquire/release semantics from the compiler >>> point of view, and others such as GCC are known to reorder away freely. >>> >>> To prevent any issues, the accesses involved when using >>> GuardUnsafeAccess should be at least volatile. >>> This change makes the few remaining ones volatile. The JMM-volatile >>> (SEQ_CST) accesses with crash protection already have stronger >>> ordering than volatile and hence do not need changing. >>> >>> By making the address passed in to the Access API volatile, the >>> MO_VOLATILE decorator is automatically set, which not surprisingly >>> makes the access volatile. Therefore, the solution is to simply make >>> the address passed in to Access volatile in this case. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>> >>> Thanks, >>> /Erik >> Not my first choice for a fix (you know how I feel about casts), but >> it works, and is currently much simpler than my preferred solution. >> >> Should there be a comment justifying the cast to volatile?? Perhaps >> something like "volatile to keep access within guarded scope."? I >> don't need a new webrev if you add such a comment. >> >> Looks good. >> > From david.holmes at oracle.com Thu Nov 30 05:07:44 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 15:07:44 +1000 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> Message-ID: Hi Dan, On 30/11/2017 7:16 AM, Daniel D. Daugherty wrote: > Greetings, > > Coleen, this is one of your Thread-SMR follow-up suggestions so I need > to hear from you on this thread. Thanks! > > I have a simple cleanup fix for Thread-SMR. The bug is: > > ??? JDK-8191787 move private inline functions from thread.inline.hpp -> > thread.cpp > ??? https://bugs.openjdk.java.net/browse/JDK-8191787 > > This fix is pure code motion: > > - moving inline functions from thread.inline.hpp -> thread.cpp > - making a few functions in thread.hpp private instead of public > > Here is the webrev URL: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 3779 inline ThreadsList* Threads::get_smr_java_thread_list() { 3780 return (ThreadsList*)OrderAccess::load_acquire(&_smr_java_thread_list); 3781 } Don't these inline definitions need to come before any use of them elsewhere in the source file, to actually get the inlining benefit? Otherwise no comments from me on the code motion. I'm abstaining from the debate on whether to list the functions immediately after the associated fields, or whether to group fields and functions separately (normally different access means they are separate, but not in this case). :) Thanks, David ----- > > This fix was (over) tested with a Mach5 tier[1-5] run. There were no > unexpected test failures. > > Thanks, in advance, for any comments, questions or suggestions. > > Dan From david.holmes at oracle.com Thu Nov 30 05:28:47 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 15:28:47 +1000 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> Message-ID: On 30/11/2017 10:08 AM, mikhailo wrote: > Here is an updated webrev: > > ??? http://cr.openjdk.java.net/~mseledtsov/8191943.01/ As Bob noted you may still need the guard here: 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2); though perhaps docker ignores invalid cpu-set values? 161 Common.logNewTestCase("expectedAPC = " + expectedAPC); That should still be a System.out.println - you already set the test case prior: 155 Common.logNewTestCase("test cpu shares, shares = " + shares); Having the same check duplicated three times is a bit messy, but not horrendously so. :) Thanks, David > > Misha > > > On 11/29/2017 01:32 PM, mikhailo wrote: >> >> On 11/29/2017 01:29 PM, Bob Vandette wrote: >>>> On Nov 29, 2017, at 4:10 PM, mikhailo >>>> wrote: >>>> >>>> Hi Bob, >>>> >>>> The failure was caused by invoking this test case: >>>> >>>> ???? testCpuShares(4096, 4); >>>> ???? The test case sets --cpu-shares to 4096, expects active >>>> processor count to be 4; ==> actual APC is 2. >>> Ahh, that makes sense.? I thought docker was complaining due to the >>> arguments being passed going >>> beyond the available cpus. >>> >>>> Command: >>>> >>>> ???? docker run --tty=true --rm --cpu-shares=4096 >>>> jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >>>> >>>> Detailed log is attached at: >>>> >>>> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >>>> >>>> >>>> >>>> Once I saw this issue, I decided to put limits on other test cases >>>> based on the number of processors available on a machine, just to be >>>> on a safe side. >>>> >>>> Perhaps a better approach here is to change the expected APC inside >>>> the testCpuShares() to be no greater than max number of available >>>> processors? >>> Yes, the selection algorithm returns the smallest number of computed >>> shares, quotas or active processors.? Using this approach, you?d >>> actually be validating the algorithm more precisely. >> OK. I will update the tests to do this kind of validation, and post a >> new webrev. >> >> >> Misha >>> >>> Bob. >>> >>> >>>> >>>> Misha >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>>>> Misha, >>>>> >>>>> Which specific subtest failed on a 2 cpu system? >>>>> >>>>> I don?t see any subtests that should have failed.? You should be able >>>>> to pass any quota, period and share value to docker independent of the >>>>> number of CPUs.? testCpus and testAPCCombo don?t try to test more >>>>> than 2 cpus. >>>>> >>>>> I?m ok with the change but just wondering why the guards are >>>>> necessary. >>>>> >>>>> Bob. >>>>> >>>>>> On Nov 28, 2017, at 7:50 PM, mikhailo >>>>>> wrote: >>>>>> >>>>>> Could you, please, review this simple fix? It limits test cases in >>>>>> TestCPUAwareness.java >>>>>> based on the number of processors available. >>>>>> >>>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>>>> ???? Testing: >>>>>> ???????? 1. Locally: exercised the affected test locally on Linux-x64 >>>>>> ???????? 2. Exercised the affected test on 2 processor machine >>>>>> ???????? 3. Exercise the affected test via automated distributed >>>>>> test system >>>>>> ??????????? In progress >>>>>> >>>>>> Misha >>>>>> >> > From david.holmes at oracle.com Thu Nov 30 05:32:17 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 15:32:17 +1000 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> Message-ID: Hi Roman, Just glancing at this but did you use "hg rename" to move the files? The webrev suggests not. Thanks, David On 30/11/2017 1:38 AM, Roman Kennke wrote: > Including hotspot-runtime-dev... > > I need one more review (esp. for the src/hotspot/share/services part): > > http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ > > Thanks, Roman > > >> On 11/29/2017 09:39 AM, Roman Kennke wrote: >>> Hi Erik, >>> >>> thanks for the review! >>> >>> I think this requires another reviewer from serviceability-dev. Who >>> can I ping about this? >> >> You could try the hotspot-runtime-dev email list, although I suspect >> most of the runtime developers are on serviceability-dev list as well... >> >> Thanks, >> Erik >> >>> Roman >>> >>> >>>> Hi Roman, >>>> >>>> Looks good now. Thanks for doing this. >>>> >>>> Thanks, >>>> /Erik >>>> >>>> On 2017-11-28 22:26, Roman Kennke wrote: >>>>> Hi Erik, >>>>> >>>>> thank you for reviewing! >>>>> >>>>> You are right. I think this is a leftover from when we tried to >>>>> pass the GCMemoryManager* into the Generation constructor. The way >>>>> it is done now (installing the GCMmemoryManager* later through >>>>> set_memory_manager()) we can group all serviceability related >>>>> set-up into initialize_serviceability(): >>>>> >>>>> Differential: >>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>>> Full: >>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>> >>>>> Notice that I hooked this up into CollectedHeap::post_initialize() >>>>> and in doing so made that method concrete, and changed all subclass >>>>> post_initialize() methods to call the super-class. >>>>> >>>>> Good now? >>>>> >>>>> Thanks, Roman >>>>> >>>>> >>>>>> Hi Roman, >>>>>> >>>>>> This looks better now. Nice job. >>>>>> I wonder though, is it possible to extract the creation of >>>>>> managers and pools to a separate function for each collected heap? >>>>>> Now I see managers are created in e.g. CMS constructor, and the >>>>>> pools are created in CMSHeap::initialize(). Perhaps initialize >>>>>> could call initialize_serviceability() that sets up those things, >>>>>> so that they are nicely separated. Unless of course there is a >>>>>> good reason why the presence of managers is needed earlier than >>>>>> that in the bootstrapping. >>>>>> >>>>>> Otherwise I think this looks good! >>>>>> >>>>>> Thanks, >>>>>> /Erik >>>>>> >>>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>>> Hi Erik, >>>>>>> >>>>>>> Thanks for your review! >>>>>>> >>>>>>> All of the things that you mentioned should be addressed in the >>>>>>> following changes: >>>>>>> >>>>>>> Differential: >>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>>> Full: >>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>>> >>>>>>> Also, Erik D (H) was so kind to contribute an additional >>>>>>> testcase, which is also included in the above patch. >>>>>>> >>>>>>> Ok now? >>>>>>> >>>>>>> Also, I need a review from serviceability-dev too! >>>>>>> >>>>>>> Thanks, >>>>>>> Roman >>>>>>> >>>>>>> >>>>>>>> 1) The code uses the "mgr" short name for "manager" in a bunch >>>>>>>> of names. I would be happy if we could write out the whole thing >>>>>>>> instead of the abbreviation. >>>>>>>> 2) It would be great if a generation would have a pointer to its >>>>>>>> manager, so we do not have to pass around the manager where we >>>>>>>> already pass around the generation (such as >>>>>>>> GenCollectedHeap::collect_generation). >>>>>>>> The generation could be created first, then the pools, then the >>>>>>>> managers, then do something like generation->set_memory_manager(x). >>>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use camel case. >>>>>>>> >>>>>>>> Otherwise I think it looks good. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> /Erik >>>>>>>> >>>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>>> Erik implemented a few more refactorings and touch-ups, and >>>>>>>>> here is our final (pending reviews) webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>>> >>>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>>> gc-end-message, avoids dragging the GCMemoryManager through >>>>>>>>> Generation and a few little related refactorings. >>>>>>>>> >>>>>>>>> Ok to push now? >>>>>>>>> >>>>>>>>> Roman >>>>>>>>> >>>>>>>>>> After a few more discussions with Erik I made some more changes. >>>>>>>>>> >>>>>>>>>> MemoryService is now unaware of the number and meaning of GC >>>>>>>>>> memory managers (minor vs major). This should be better for >>>>>>>>>> GCs that don't make that distinction and/or use more different >>>>>>>>>> GCs (e.g. minor, intermediate, full). >>>>>>>>>> >>>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>>> - GCMemoryManager now has gc_end_message() which is used by >>>>>>>>>> GCNotifier::pushNotification(). >>>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now accept >>>>>>>>>> a GCMemoryManager* instead of bull full_gc >>>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>>> - Generation now knows about the corresponding GCMemoryManager >>>>>>>>>> >>>>>>>>>> Please review the full change: >>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>>> >>>>>>>>>> Thanks, Roman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I had some off-band discussions with Erik Helin and re-did >>>>>>>>>>> most of the changeset: >>>>>>>>>>> - The GC interface now resides in CollectedHeap, specifically >>>>>>>>>>> the two methods memory_managers() and memory_pools(), and is >>>>>>>>>>> implemented in the various concrete subclasses. >>>>>>>>>>> - Both methods return (by value) a >>>>>>>>>>> GrowableArray and GrowableArray >>>>>>>>>>> respectively. Returning a stack-allocated GrowableArray >>>>>>>>>>> seemed least complicated (avoid explicit cleanup of >>>>>>>>>>> short-lived array object), and most future-proof, e.g. >>>>>>>>>>> currently there is an implicit expectation to get 2 >>>>>>>>>>> GCMemoryManagers, even though some GCs don't necessarily have >>>>>>>>>>> two. The API allows for easy extension of the situation. >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>>> Serviceability team. >>>>>>>>>>> >>>>>>>>>>> Roman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Currently, there's lots of GC specific code sprinkled over >>>>>>>>>>>> src/hotspot/share/services. This change introduces a GC >>>>>>>>>>>> interface for that, and moves all GC specific code to their >>>>>>>>>>>> respective src/hotspot/share/gc directory. >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none showed >>>>>>>>>>>> regressions >>>>>>>>>>>> >>>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>>> >>>>>>>>>>>> What do you think? >>>>>>>>>>>> >>>>>>>>>>>> Roman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> > From david.holmes at oracle.com Thu Nov 30 07:47:45 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 17:47:45 +1000 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: <38e83c23-7a65-c664-c092-94fc131f9c2c@oracle.com> References: <06a024a5-a1ff-5627-7d7d-1c84fac45f79@oracle.com> <38e83c23-7a65-c664-c092-94fc131f9c2c@oracle.com> Message-ID: <39fe335c-84aa-a6d9-2f9d-7754d9a57735@oracle.com> Hi Jini, On 29/11/2017 9:51 PM, Jini George wrote: > Thank you very much for the review, Serguei. Continuing to keep these > constants in an interface would mean that they would need to have the > 'final' qualifier. And that would mean that we would not be able to > populate the values of these constants at runtime by reading those in > from vmStructs using db.lookupIntConstant(). I have instead made these > as inner classes in this new webrev: > > http://cr.openjdk.java.net/~jgeorge/8191324/webrev.01/ I had a look at this and it all seems fine. Good to see the ia64 code gone! > As David had pointed out wrt the review of 8190837: BasicType and > BasicTypeSize should refer to HotSpot values, I realize that removal of > the 'final' qualifier could cause parfait warnings, but since we would > need to do that for the rest of the constants in the file also, I would > prefer taking it up as a separate exercise. The fact it is a private inner class will probably be enough to stop parfait from flagging the non-final static fields. You should also be able to declare them private (or at least package-access) rather than public, which further removes them from the kind of construct parfait is looking for. Thanks, David ----- > I had removed the PerfDataVariability constants since these were not > used, and we are trying to remove unused code in SA to reduce the > probability of bugs creeping in. Guess we can always put it back if we > start checking the values against these constants. Let me know if you > don't agree with this. > > Thanks, > Jini. > > > > On 11/28/2017 12:38 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jini, >> >> It looks good to me. >> >> A couple of questions on the: >> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/PerfDataEntry.java.udiff.html >> >> >> + private static int U_None; >> + private static int U_Bytes; >> + private static int U_Ticks; >> + private static int U_Events; >> + private static int U_String; >> + private static int U_Hertz; >> +. . . + >> + // PerfData Units enum >> + U_None = db.lookupIntConstant("PerfData::U_None"); >> + U_Bytes = db.lookupIntConstant("PerfData::U_Bytes"); >> + U_Ticks = db.lookupIntConstant("PerfData::U_Ticks"); >> + U_Events = db.lookupIntConstant("PerfData::U_Events"); >> + U_String = db.lookupIntConstant("PerfData::U_String"); >> + U_Hertz = db.lookupIntConstant("PerfData::U_Hertz");Before your fix >> these private static fields were defined in the interface which I like: >> >> - public interface PerfDataUnits { >> - public static final int U_None = 1; >> - public static final int U_Bytes = 2; >> - public static final int U_Ticks = 3; >> - public static final int U_Events = 4; >> - public static final int U_String = 5; >> - public static final int U_Hertz = 6; >> - } >> >> >> I think, it'd still make sense to keep them in an interface or a small >> class, >> so they are not separate constants but a part of a common outer type. >> >> - public interface PerfDataVariability { >> - public static final int V_Constant = 1; >> - public static final int V_Monotonic = 2; >> - public static final int V_Variable = 3; >> - } I wonder it these constants can still be useful as the following >> method returns one of them as a result which may need to be checked >> for an exact value.Thanks, Serguei >> >> >> >> >> On 11/21/17 02:34, Jini George wrote: >>> Hello, >>> >>> Here's requesting reviews for some SA code cleanup. >>> >>> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >>> Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >>> >>> The changes here are primarily to: >>> >>> 1. Remove unused IA64 SA code. >>> 2. Make changes to avoid error-prone redefinition of hotspot >>> constants in SA Java code. Instead read it in through vmStructs and >>> db.lookupIntConstant(). >>> 3. Make variable name changes in SA to align with the hotspot names. >>> >>> The changes are straightforward. >>> >>> Thanks much, >>> Jini. >>> >> From jini.george at oracle.com Thu Nov 30 07:53:09 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 30 Nov 2017 13:23:09 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: <39fe335c-84aa-a6d9-2f9d-7754d9a57735@oracle.com> References: <06a024a5-a1ff-5627-7d7d-1c84fac45f79@oracle.com> <38e83c23-7a65-c664-c092-94fc131f9c2c@oracle.com> <39fe335c-84aa-a6d9-2f9d-7754d9a57735@oracle.com> Message-ID: <0e69bdb3-1b0a-8407-d91c-00a3d294996a@oracle.com> Thanks much, David! - Jini. On 11/30/2017 1:17 PM, David Holmes wrote: > Hi Jini, > > On 29/11/2017 9:51 PM, Jini George wrote: >> Thank you very much for the review, Serguei. Continuing to keep these >> constants in an interface would mean that they would need to have the >> 'final' qualifier. And that would mean that we would not be able to >> populate the values of these constants at runtime by reading those in >> from vmStructs using db.lookupIntConstant(). I have instead made these >> as inner classes in this new webrev: >> >> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.01/ > > I had a look at this and it all seems fine. Good to see the ia64 code gone! > >> As David had pointed out wrt the review of 8190837: BasicType and >> BasicTypeSize should refer to HotSpot values, I realize that removal >> of the 'final' qualifier could cause parfait warnings, but since we >> would need to do that for the rest of the constants in the file also, >> I would prefer taking it up as a separate exercise. > > The fact it is a private inner class will probably be enough to stop > parfait from flagging the non-final static fields. You should also be > able to declare them private (or at least package-access) rather than > public, which further removes them from the kind of construct parfait is > looking for. > > Thanks, > David > ----- > >> I had removed the PerfDataVariability constants since these were not >> used, and we are trying to remove unused code in SA to reduce the >> probability of bugs creeping in. Guess we can always put it back if we >> start checking the values against these constants. Let me know if you >> don't agree with this. >> >> Thanks, >> Jini. >> >> >> >> On 11/28/2017 12:38 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Jini, >>> >>> It looks good to me. >>> >>> A couple of questions on the: >>> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/PerfDataEntry.java.udiff.html >>> >>> >>> + private static int U_None; >>> + private static int U_Bytes; >>> + private static int U_Ticks; >>> + private static int U_Events; >>> + private static int U_String; >>> + private static int U_Hertz; >>> +. . . + >>> + // PerfData Units enum >>> + U_None = db.lookupIntConstant("PerfData::U_None"); >>> + U_Bytes = db.lookupIntConstant("PerfData::U_Bytes"); >>> + U_Ticks = db.lookupIntConstant("PerfData::U_Ticks"); >>> + U_Events = db.lookupIntConstant("PerfData::U_Events"); >>> + U_String = db.lookupIntConstant("PerfData::U_String"); >>> + U_Hertz = db.lookupIntConstant("PerfData::U_Hertz");Before your fix >>> these private static fields were defined in the interface which I like: >>> >>> - public interface PerfDataUnits { >>> - public static final int U_None = 1; >>> - public static final int U_Bytes = 2; >>> - public static final int U_Ticks = 3; >>> - public static final int U_Events = 4; >>> - public static final int U_String = 5; >>> - public static final int U_Hertz = 6; >>> - } >>> >>> >>> I think, it'd still make sense to keep them in an interface or a >>> small class, >>> so they are not separate constants but a part of a common outer type. >>> >>> - public interface PerfDataVariability { >>> - public static final int V_Constant = 1; >>> - public static final int V_Monotonic = 2; >>> - public static final int V_Variable = 3; >>> - } I wonder it these constants can still be useful as the following >>> method returns one of them as a result which may need to be checked >>> for an exact value.Thanks, Serguei >>> >>> >>> >>> >>> On 11/21/17 02:34, Jini George wrote: >>>> Hello, >>>> >>>> Here's requesting reviews for some SA code cleanup. >>>> >>>> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >>>> Webrev: >>>> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >>>> >>>> The changes here are primarily to: >>>> >>>> 1. Remove unused IA64 SA code. >>>> 2. Make changes to avoid error-prone redefinition of hotspot >>>> constants in SA Java code. Instead read it in through vmStructs and >>>> db.lookupIntConstant(). >>>> 3. Make variable name changes in SA to align with the hotspot names. >>>> >>>> The changes are straightforward. >>>> >>>> Thanks much, >>>> Jini. >>>> >>> From jini.george at oracle.com Thu Nov 30 10:10:15 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 30 Nov 2017 15:40:15 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: <528de53b-1a66-1e14-7422-4097c6136f86@oracle.com> References: <06a024a5-a1ff-5627-7d7d-1c84fac45f79@oracle.com> <38e83c23-7a65-c664-c092-94fc131f9c2c@oracle.com> <39fe335c-84aa-a6d9-2f9d-7754d9a57735@oracle.com> <528de53b-1a66-1e14-7422-4097c6136f86@oracle.com> Message-ID: <74835c6f-d780-eee1-6c49-8850bb6563bb@oracle.com> Thank you very much, Serguei! - Jini. On 11/30/2017 2:16 PM, serguei.spitsyn at oracle.com wrote: > Hi Jini, > > It looks good to me. > Thank you for the update! > > A minor tip: > ?It'd match the Hotspot naming better if the PerfDataUnits is replaced > with PerfData. > > But I leave it up to you. > > Thanks, > Serguei > > > On 11/29/17 23:47, David Holmes wrote: >> Hi Jini, >> >> On 29/11/2017 9:51 PM, Jini George wrote: >>> Thank you very much for the review, Serguei. Continuing to keep these >>> constants in an interface would mean that they would need to have the >>> 'final' qualifier. And that would mean that we would not be able to >>> populate the values of these constants at runtime by reading those in >>> from vmStructs using db.lookupIntConstant(). I have instead made >>> these as inner classes in this new webrev: >>> >>> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.01/ >> >> I had a look at this and it all seems fine. Good to see the ia64 code >> gone! >> >>> As David had pointed out wrt the review of 8190837: BasicType and >>> BasicTypeSize should refer to HotSpot values, I realize that removal >>> of the 'final' qualifier could cause parfait warnings, but since we >>> would need to do that for the rest of the constants in the file also, >>> I would prefer taking it up as a separate exercise. >> >> The fact it is a private inner class will probably be enough to stop >> parfait from flagging the non-final static fields. You should also be >> able to declare them private (or at least package-access) rather than >> public, which further removes them from the kind of construct parfait >> is looking for. >> >> Thanks, >> David >> ----- >> >>> I had removed the PerfDataVariability constants since these were not >>> used, and we are trying to remove unused code in SA to reduce the >>> probability of bugs creeping in. Guess we can always put it back if >>> we start checking the values against these constants. Let me know if >>> you don't agree with this. >>> >>> Thanks, >>> Jini. >>> >>> >>> >>> On 11/28/2017 12:38 PM, serguei.spitsyn at oracle.com wrote: >>>> Hi Jini, >>>> >>>> It looks good to me. >>>> >>>> A couple of questions on the: >>>> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/PerfDataEntry.java.udiff.html >>>> >>>> >>>> + private static int U_None; >>>> + private static int U_Bytes; >>>> + private static int U_Ticks; >>>> + private static int U_Events; >>>> + private static int U_String; >>>> + private static int U_Hertz; >>>> +. . . + >>>> + // PerfData Units enum >>>> + U_None = db.lookupIntConstant("PerfData::U_None"); >>>> + U_Bytes = db.lookupIntConstant("PerfData::U_Bytes"); >>>> + U_Ticks = db.lookupIntConstant("PerfData::U_Ticks"); >>>> + U_Events = db.lookupIntConstant("PerfData::U_Events"); >>>> + U_String = db.lookupIntConstant("PerfData::U_String"); >>>> + U_Hertz = db.lookupIntConstant("PerfData::U_Hertz");Before your >>>> fix these private static fields were defined in the interface which >>>> I like: >>>> >>>> - public interface PerfDataUnits { >>>> - public static final int U_None = 1; >>>> - public static final int U_Bytes = 2; >>>> - public static final int U_Ticks = 3; >>>> - public static final int U_Events = 4; >>>> - public static final int U_String = 5; >>>> - public static final int U_Hertz = 6; >>>> - } >>>> >>>> >>>> I think, it'd still make sense to keep them in an interface or a >>>> small class, >>>> so they are not separate constants but a part of a common outer type. >>>> >>>> - public interface PerfDataVariability { >>>> - public static final int V_Constant = 1; >>>> - public static final int V_Monotonic = 2; >>>> - public static final int V_Variable = 3; >>>> - } I wonder it these constants can still be useful as the following >>>> method returns one of them as a result which may need to be checked >>>> for an exact value.Thanks, Serguei >>>> >>>> >>>> >>>> >>>> On 11/21/17 02:34, Jini George wrote: >>>>> Hello, >>>>> >>>>> Here's requesting reviews for some SA code cleanup. >>>>> >>>>> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >>>>> >>>>> The changes here are primarily to: >>>>> >>>>> 1. Remove unused IA64 SA code. >>>>> 2. Make changes to avoid error-prone redefinition of hotspot >>>>> constants in SA Java code. Instead read it in through vmStructs and >>>>> db.lookupIntConstant(). >>>>> 3. Make variable name changes in SA to align with the hotspot names. >>>>> >>>>> The changes are straightforward. >>>>> >>>>> Thanks much, >>>>> Jini. >>>>> >>>> > From erik.osterlund at oracle.com Thu Nov 30 10:19:37 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 30 Nov 2017 11:19:37 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <77db1e4a-77ad-5fe3-afe0-65e46e38c1f5@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> <77db1e4a-77ad-5fe3-afe0-65e46e38c1f5@oracle.com> Message-ID: <5A1FDB39.1000104@oracle.com> Hi David, Thanks for the review. /Erik On 2017-11-30 05:44, David Holmes wrote: > +1 :) > > David > > On 29/11/2017 11:48 PM, Erik ?sterlund wrote: >> Hi Kim, >> >> I know how you feel about casts, and I also saw how Andrew Haley >> wanted a more explicit comment about why it needs to be volatile. >> To make both of you happy, I thought I'd try to make the address >> (addr()) in MemoryAccess volatile T*. That way I can write a more >> explicit comment about why it is volatile in one single place, and >> make you happier to not cast the address to something else than it >> was declared. >> >> I hope this makes both of you happy. If not, I am okay with the old >> variant too, and write a comment that I copy around instead. >> >> Full webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ >> >> Incremental webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ >> >> Thanks, >> /Erik >> >> On 2017-11-28 21:30, Kim Barrett wrote: >>>> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund >>>> wrote: >>>> >>>> Hi, >>>> >>>> There is currently a bug when using unsafe accesses off-heap: >>>> >>>> 1) We write into a thread that we enable crash protection (using >>>> GuardUnsafeAccess): >>>> 2) We perform the access >>>> 3) We write into a thread that we disable crash protection (using >>>> ~GuardUnsafeAccess) >>>> >>>> The problem is that the crash protection stores are volatile, but >>>> the actual access is non-volatile. Compilers have different >>>> interpretation whether volatile - non-volatile accesses are allowed >>>> to reorder. MSVC is known to interpret such interactions as-if the >>>> volatile accesses have acquire/release semantics from the compiler >>>> point of view, and others such as GCC are known to reorder away >>>> freely. >>>> >>>> To prevent any issues, the accesses involved when using >>>> GuardUnsafeAccess should be at least volatile. >>>> This change makes the few remaining ones volatile. The JMM-volatile >>>> (SEQ_CST) accesses with crash protection already have stronger >>>> ordering than volatile and hence do not need changing. >>>> >>>> By making the address passed in to the Access API volatile, the >>>> MO_VOLATILE decorator is automatically set, which not surprisingly >>>> makes the access volatile. Therefore, the solution is to simply >>>> make the address passed in to Access volatile in this case. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>>> >>>> Thanks, >>>> /Erik >>> Not my first choice for a fix (you know how I feel about casts), but >>> it works, and is currently much simpler than my preferred solution. >>> >>> Should there be a comment justifying the cast to volatile? Perhaps >>> something like "volatile to keep access within guarded scope." I >>> don't need a new webrev if you add such a comment. >>> >>> Looks good. >>> >> From erik.osterlund at oracle.com Thu Nov 30 10:21:03 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 30 Nov 2017 11:21:03 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <58ba68ff-c6f9-e7b7-55c1-10a56d309839@redhat.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> <58ba68ff-c6f9-e7b7-55c1-10a56d309839@redhat.com> Message-ID: <5A1FDB8F.5090907@oracle.com> Hi Andrew, Thanks for the review! /Erik On 2017-11-29 15:25, Andrew Haley wrote: > On 29/11/17 13:48, Erik ?sterlund wrote: >> I hope this makes both of you happy. > :-) :-) :-) > From erik.osterlund at oracle.com Thu Nov 30 10:21:26 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 30 Nov 2017 11:21:26 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: <5A1FDBA6.8070308@oracle.com> Hi Dmitry, Thank you for the review. /Erik On 2017-11-29 15:43, Dmitry Samersoff wrote: > Erik, > > I like this variant too :) > > -Dmitry > > On 29.11.2017 16:48, Erik ?sterlund wrote: >> Hi Kim, >> >> I know how you feel about casts, and I also saw how Andrew Haley wanted >> a more explicit comment about why it needs to be volatile. >> To make both of you happy, I thought I'd try to make the address >> (addr()) in MemoryAccess volatile T*. That way I can write a more >> explicit comment about why it is volatile in one single place, and make >> you happier to not cast the address to something else than it was declared. >> >> I hope this makes both of you happy. If not, I am okay with the old >> variant too, and write a comment that I copy around instead. >> >> Full webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ >> >> Incremental webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ >> >> Thanks, >> /Erik >> >> On 2017-11-28 21:30, Kim Barrett wrote: >>>> On Nov 27, 2017, at 7:36 AM, Erik ?sterlund >>>> wrote: >>>> >>>> Hi, >>>> >>>> There is currently a bug when using unsafe accesses off-heap: >>>> >>>> 1) We write into a thread that we enable crash protection (using >>>> GuardUnsafeAccess): >>>> 2) We perform the access >>>> 3) We write into a thread that we disable crash protection (using >>>> ~GuardUnsafeAccess) >>>> >>>> The problem is that the crash protection stores are volatile, but the >>>> actual access is non-volatile. Compilers have different >>>> interpretation whether volatile - non-volatile accesses are allowed >>>> to reorder. MSVC is known to interpret such interactions as-if the >>>> volatile accesses have acquire/release semantics from the compiler >>>> point of view, and others such as GCC are known to reorder away freely. >>>> >>>> To prevent any issues, the accesses involved when using >>>> GuardUnsafeAccess should be at least volatile. >>>> This change makes the few remaining ones volatile. The JMM-volatile >>>> (SEQ_CST) accesses with crash protection already have stronger >>>> ordering than volatile and hence do not need changing. >>>> >>>> By making the address passed in to the Access API volatile, the >>>> MO_VOLATILE decorator is automatically set, which not surprisingly >>>> makes the access volatile. Therefore, the solution is to simply make >>>> the address passed in to Access volatile in this case. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8186787 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00/ >>>> >>>> Thanks, >>>> /Erik >>> Not my first choice for a fix (you know how I feel about casts), but >>> it works, and is currently much simpler than my preferred solution. >>> >>> Should there be a comment justifying the cast to volatile? Perhaps >>> something like "volatile to keep access within guarded scope." I >>> don't need a new webrev if you add such a comment. >>> >>> Looks good. >>> > From per.liden at oracle.com Thu Nov 30 10:32:33 2017 From: per.liden at oracle.com (Per Liden) Date: Thu, 30 Nov 2017 11:32:33 +0100 Subject: RFR (S): 8192003: Refactor weak references in StringTable to use the Access API In-Reply-To: <5A1D93C1.3020605@oracle.com> References: <5A1D93C1.3020605@oracle.com> Message-ID: <81e2ffcb-4912-9fc7-55d9-0553b737a695@oracle.com> Hi Erik, On 2017-11-28 17:50, Erik ?sterlund wrote: > Hi, > > The StringTable contains weak references to oops. Today the weak > semantics is managed using explicit calls to G1 SATB enqueue barriers. > > Now that the Access API is available, it should be used instead as it is > more modular. > > This change fixes that by making these oops ON_PHANTOM_OOP_REF with a > corresponding AS_NO_KEEPALIVE accessor when we do not want to keep it > alive, much like my previous changes to other weak tables. > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00/ share/classfile/javaClasses.inline.hpp -------------------------------------- 57 typeArrayOop java_lang_String::value_no_keepalive(oop java_string) { 58 assert(initialized && (value_offset > 0), "Must be initialized"); 59 assert(is_instance(java_string), "must be java_string"); 60 oop value = HeapAccess::oop_load_at(java_string, value_offset); 61 return (typeArrayOop)value; 62 } How about pushing this barrier down to oopDesc, with a oopDesc::obj_field_special_access(...) function? 76 typeArrayOop value = java_lang_String::value_no_keepalive(java_string); 77 assert(initialized, "Must be initialized"); 78 assert(is_instance(java_string), "must be java_string"); Looks like you accidentally moved the value_no_keepalive() call above the asserts? share/classfile/stringTable.cpp ------------------------------- 155 oop string = string_object_no_keepalive(l); 156 if (java_lang_String::equals(string, name, len)) { 157 return string_object(l); 158 } Can we please add a comment here, saying that returning "string" would be very bad, or maybe even fold this a bit, so that no one will be tempted to ever return that value, something like: if (java_lang_String::equals(string_object_no_keepalive(l), name, len)) { // Comment saying why we must call string_object() here... return string_object(l); } cheers, Per > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8192003 > > Thanks, > /Erik From rkennke at redhat.com Thu Nov 30 11:30:24 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 30 Nov 2017 12:30:24 +0100 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> Message-ID: <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> Hi David, yes I did, but it's probably got lost somewhere, while I was bouncing the patch between me and Erik D. (I noticed that the msg also got lost). Both reinstated: http://cr.openjdk.java.net/~rkennke/8191564/webrev.07/ Roman > Hi Roman, > > Just glancing at this but did you use "hg rename" to move the files? The > webrev suggests not. > > Thanks, > David > > On 30/11/2017 1:38 AM, Roman Kennke wrote: >> Including hotspot-runtime-dev... >> >> I need one more review (esp. for the src/hotspot/share/services part): >> >> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >> >> Thanks, Roman >> >> >>> On 11/29/2017 09:39 AM, Roman Kennke wrote: >>>> Hi Erik, >>>> >>>> thanks for the review! >>>> >>>> I think this requires another reviewer from serviceability-dev. Who >>>> can I ping about this? >>> >>> You could try the hotspot-runtime-dev email list, although I suspect >>> most of the runtime developers are on serviceability-dev list as well... >>> >>> Thanks, >>> Erik >>> >>>> Roman >>>> >>>> >>>>> Hi Roman, >>>>> >>>>> Looks good now. Thanks for doing this. >>>>> >>>>> Thanks, >>>>> /Erik >>>>> >>>>> On 2017-11-28 22:26, Roman Kennke wrote: >>>>>> Hi Erik, >>>>>> >>>>>> thank you for reviewing! >>>>>> >>>>>> You are right. I think this is a leftover from when we tried to >>>>>> pass the GCMemoryManager* into the Generation constructor. The way >>>>>> it is done now (installing the GCMmemoryManager* later through >>>>>> set_memory_manager()) we can group all serviceability related >>>>>> set-up into initialize_serviceability(): >>>>>> >>>>>> Differential: >>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>>>> Full: >>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>>> >>>>>> Notice that I hooked this up into CollectedHeap::post_initialize() >>>>>> and in doing so made that method concrete, and changed all >>>>>> subclass post_initialize() methods to call the super-class. >>>>>> >>>>>> Good now? >>>>>> >>>>>> Thanks, Roman >>>>>> >>>>>> >>>>>>> Hi Roman, >>>>>>> >>>>>>> This looks better now. Nice job. >>>>>>> I wonder though, is it possible to extract the creation of >>>>>>> managers and pools to a separate function for each collected heap? >>>>>>> Now I see managers are created in e.g. CMS constructor, and the >>>>>>> pools are created in CMSHeap::initialize(). Perhaps initialize >>>>>>> could call initialize_serviceability() that sets up those things, >>>>>>> so that they are nicely separated. Unless of course there is a >>>>>>> good reason why the presence of managers is needed earlier than >>>>>>> that in the bootstrapping. >>>>>>> >>>>>>> Otherwise I think this looks good! >>>>>>> >>>>>>> Thanks, >>>>>>> /Erik >>>>>>> >>>>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>>>> Hi Erik, >>>>>>>> >>>>>>>> Thanks for your review! >>>>>>>> >>>>>>>> All of the things that you mentioned should be addressed in the >>>>>>>> following changes: >>>>>>>> >>>>>>>> Differential: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>>>> Full: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>>>> >>>>>>>> Also, Erik D (H) was so kind to contribute an additional >>>>>>>> testcase, which is also included in the above patch. >>>>>>>> >>>>>>>> Ok now? >>>>>>>> >>>>>>>> Also, I need a review from serviceability-dev too! >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Roman >>>>>>>> >>>>>>>> >>>>>>>>> 1) The code uses the "mgr" short name for "manager" in a bunch >>>>>>>>> of names. I would be happy if we could write out the whole >>>>>>>>> thing instead of the abbreviation. >>>>>>>>> 2) It would be great if a generation would have a pointer to >>>>>>>>> its manager, so we do not have to pass around the manager where >>>>>>>>> we already pass around the generation (such as >>>>>>>>> GenCollectedHeap::collect_generation). >>>>>>>>> The generation could be created first, then the pools, then the >>>>>>>>> managers, then do something like >>>>>>>>> generation->set_memory_manager(x). >>>>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use camel >>>>>>>>> case. >>>>>>>>> >>>>>>>>> Otherwise I think it looks good. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> /Erik >>>>>>>>> >>>>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>>>> Erik implemented a few more refactorings and touch-ups, and >>>>>>>>>> here is our final (pending reviews) webrev: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>>>> >>>>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>>>> gc-end-message, avoids dragging the GCMemoryManager through >>>>>>>>>> Generation and a few little related refactorings. >>>>>>>>>> >>>>>>>>>> Ok to push now? >>>>>>>>>> >>>>>>>>>> Roman >>>>>>>>>> >>>>>>>>>>> After a few more discussions with Erik I made some more changes. >>>>>>>>>>> >>>>>>>>>>> MemoryService is now unaware of the number and meaning of GC >>>>>>>>>>> memory managers (minor vs major). This should be better for >>>>>>>>>>> GCs that don't make that distinction and/or use more >>>>>>>>>>> different GCs (e.g. minor, intermediate, full). >>>>>>>>>>> >>>>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>>>> - GCMemoryManager now has gc_end_message() which is used by >>>>>>>>>>> GCNotifier::pushNotification(). >>>>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now accept >>>>>>>>>>> a GCMemoryManager* instead of bull full_gc >>>>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>>>> - Generation now knows about the corresponding GCMemoryManager >>>>>>>>>>> >>>>>>>>>>> Please review the full change: >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>>>> >>>>>>>>>>> Thanks, Roman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I had some off-band discussions with Erik Helin and re-did >>>>>>>>>>>> most of the changeset: >>>>>>>>>>>> - The GC interface now resides in CollectedHeap, >>>>>>>>>>>> specifically the two methods memory_managers() and >>>>>>>>>>>> memory_pools(), and is implemented in the various concrete >>>>>>>>>>>> subclasses. >>>>>>>>>>>> - Both methods return (by value) a >>>>>>>>>>>> GrowableArray and >>>>>>>>>>>> GrowableArray respectively. Returning a >>>>>>>>>>>> stack-allocated GrowableArray seemed least complicated >>>>>>>>>>>> (avoid explicit cleanup of short-lived array object), and >>>>>>>>>>>> most future-proof, e.g. currently there is an implicit >>>>>>>>>>>> expectation to get 2 GCMemoryManagers, even though some GCs >>>>>>>>>>>> don't necessarily have two. The API allows for easy >>>>>>>>>>>> extension of the situation. >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>>>> >>>>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>>>> Serviceability team. >>>>>>>>>>>> >>>>>>>>>>>> Roman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Currently, there's lots of GC specific code sprinkled over >>>>>>>>>>>>> src/hotspot/share/services. This change introduces a GC >>>>>>>>>>>>> interface for that, and moves all GC specific code to their >>>>>>>>>>>>> respective src/hotspot/share/gc directory. >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none showed >>>>>>>>>>>>> regressions >>>>>>>>>>>>> >>>>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>>>> >>>>>>>>>>>>> What do you think? >>>>>>>>>>>>> >>>>>>>>>>>>> Roman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> From david.holmes at oracle.com Thu Nov 30 12:21:25 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 30 Nov 2017 22:21:25 +1000 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> Message-ID: <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> On 30/11/2017 9:30 PM, Roman Kennke wrote: > Hi David, > > yes I did, but it's probably got lost somewhere, while I was bouncing > the patch between me and Erik D. (I noticed that the msg also got lost). > Both reinstated: > > http://cr.openjdk.java.net/~rkennke/8191564/webrev.07/ Thanks - that's much simpler to follow. IIRC in the test I think you need "@requires vm.gc == null" to skip the test if the framework is trying to run it with an explicit GC configuration - else it may conflict with your hardwired GC selection. The overall refactoring seems reasonable, but I can't really vouch for its accuracy. I'm not clear how/if these service APIs are accesses from the Java-level serviceability code. David > Roman > >> Hi Roman, >> >> Just glancing at this but did you use "hg rename" to move the files? >> The webrev suggests not. >> >> Thanks, >> David >> >> On 30/11/2017 1:38 AM, Roman Kennke wrote: >>> Including hotspot-runtime-dev... >>> >>> I need one more review (esp. for the src/hotspot/share/services part): >>> >>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>> >>> Thanks, Roman >>> >>> >>>> On 11/29/2017 09:39 AM, Roman Kennke wrote: >>>>> Hi Erik, >>>>> >>>>> thanks for the review! >>>>> >>>>> I think this requires another reviewer from serviceability-dev. Who >>>>> can I ping about this? >>>> >>>> You could try the hotspot-runtime-dev email list, although I suspect >>>> most of the runtime developers are on serviceability-dev list as >>>> well... >>>> >>>> Thanks, >>>> Erik >>>> >>>>> Roman >>>>> >>>>> >>>>>> Hi Roman, >>>>>> >>>>>> Looks good now. Thanks for doing this. >>>>>> >>>>>> Thanks, >>>>>> /Erik >>>>>> >>>>>> On 2017-11-28 22:26, Roman Kennke wrote: >>>>>>> Hi Erik, >>>>>>> >>>>>>> thank you for reviewing! >>>>>>> >>>>>>> You are right. I think this is a leftover from when we tried to >>>>>>> pass the GCMemoryManager* into the Generation constructor. The >>>>>>> way it is done now (installing the GCMmemoryManager* later >>>>>>> through set_memory_manager()) we can group all serviceability >>>>>>> related set-up into initialize_serviceability(): >>>>>>> >>>>>>> Differential: >>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>>>>> Full: >>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>>>> >>>>>>> Notice that I hooked this up into >>>>>>> CollectedHeap::post_initialize() and in doing so made that method >>>>>>> concrete, and changed all subclass post_initialize() methods to >>>>>>> call the super-class. >>>>>>> >>>>>>> Good now? >>>>>>> >>>>>>> Thanks, Roman >>>>>>> >>>>>>> >>>>>>>> Hi Roman, >>>>>>>> >>>>>>>> This looks better now. Nice job. >>>>>>>> I wonder though, is it possible to extract the creation of >>>>>>>> managers and pools to a separate function for each collected heap? >>>>>>>> Now I see managers are created in e.g. CMS constructor, and the >>>>>>>> pools are created in CMSHeap::initialize(). Perhaps initialize >>>>>>>> could call initialize_serviceability() that sets up those >>>>>>>> things, so that they are nicely separated. Unless of course >>>>>>>> there is a good reason why the presence of managers is needed >>>>>>>> earlier than that in the bootstrapping. >>>>>>>> >>>>>>>> Otherwise I think this looks good! >>>>>>>> >>>>>>>> Thanks, >>>>>>>> /Erik >>>>>>>> >>>>>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>>>>> Hi Erik, >>>>>>>>> >>>>>>>>> Thanks for your review! >>>>>>>>> >>>>>>>>> All of the things that you mentioned should be addressed in the >>>>>>>>> following changes: >>>>>>>>> >>>>>>>>> Differential: >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>>>>> Full: >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>>>>> >>>>>>>>> Also, Erik D (H) was so kind to contribute an additional >>>>>>>>> testcase, which is also included in the above patch. >>>>>>>>> >>>>>>>>> Ok now? >>>>>>>>> >>>>>>>>> Also, I need a review from serviceability-dev too! >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Roman >>>>>>>>> >>>>>>>>> >>>>>>>>>> 1) The code uses the "mgr" short name for "manager" in a bunch >>>>>>>>>> of names. I would be happy if we could write out the whole >>>>>>>>>> thing instead of the abbreviation. >>>>>>>>>> 2) It would be great if a generation would have a pointer to >>>>>>>>>> its manager, so we do not have to pass around the manager >>>>>>>>>> where we already pass around the generation (such as >>>>>>>>>> GenCollectedHeap::collect_generation). >>>>>>>>>> The generation could be created first, then the pools, then >>>>>>>>>> the managers, then do something like >>>>>>>>>> generation->set_memory_manager(x). >>>>>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use camel >>>>>>>>>> case. >>>>>>>>>> >>>>>>>>>> Otherwise I think it looks good. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> /Erik >>>>>>>>>> >>>>>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>>>>> Erik implemented a few more refactorings and touch-ups, and >>>>>>>>>>> here is our final (pending reviews) webrev: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>>>>> >>>>>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>>>>> gc-end-message, avoids dragging the GCMemoryManager through >>>>>>>>>>> Generation and a few little related refactorings. >>>>>>>>>>> >>>>>>>>>>> Ok to push now? >>>>>>>>>>> >>>>>>>>>>> Roman >>>>>>>>>>> >>>>>>>>>>>> After a few more discussions with Erik I made some more >>>>>>>>>>>> changes. >>>>>>>>>>>> >>>>>>>>>>>> MemoryService is now unaware of the number and meaning of GC >>>>>>>>>>>> memory managers (minor vs major). This should be better for >>>>>>>>>>>> GCs that don't make that distinction and/or use more >>>>>>>>>>>> different GCs (e.g. minor, intermediate, full). >>>>>>>>>>>> >>>>>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>>>>> - GCMemoryManager now has gc_end_message() which is used by >>>>>>>>>>>> GCNotifier::pushNotification(). >>>>>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now >>>>>>>>>>>> accept a GCMemoryManager* instead of bull full_gc >>>>>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>>>>> - Generation now knows about the corresponding GCMemoryManager >>>>>>>>>>>> >>>>>>>>>>>> Please review the full change: >>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>>>>> >>>>>>>>>>>> Thanks, Roman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I had some off-band discussions with Erik Helin and re-did >>>>>>>>>>>>> most of the changeset: >>>>>>>>>>>>> - The GC interface now resides in CollectedHeap, >>>>>>>>>>>>> specifically the two methods memory_managers() and >>>>>>>>>>>>> memory_pools(), and is implemented in the various concrete >>>>>>>>>>>>> subclasses. >>>>>>>>>>>>> - Both methods return (by value) a >>>>>>>>>>>>> GrowableArray and >>>>>>>>>>>>> GrowableArray respectively. Returning a >>>>>>>>>>>>> stack-allocated GrowableArray seemed least complicated >>>>>>>>>>>>> (avoid explicit cleanup of short-lived array object), and >>>>>>>>>>>>> most future-proof, e.g. currently there is an implicit >>>>>>>>>>>>> expectation to get 2 GCMemoryManagers, even though some GCs >>>>>>>>>>>>> don't necessarily have two. The API allows for easy >>>>>>>>>>>>> extension of the situation. >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>>>>> >>>>>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>>>>> Serviceability team. >>>>>>>>>>>>> >>>>>>>>>>>>> Roman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Currently, there's lots of GC specific code sprinkled over >>>>>>>>>>>>>> src/hotspot/share/services. This change introduces a GC >>>>>>>>>>>>>> interface for that, and moves all GC specific code to >>>>>>>>>>>>>> their respective src/hotspot/share/gc directory. >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none >>>>>>>>>>>>>> showed regressions >>>>>>>>>>>>>> >>>>>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Roman >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> > From coleen.phillimore at oracle.com Thu Nov 30 12:47:24 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 30 Nov 2017 07:47:24 -0500 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: References: Message-ID: <4dcfe382-488a-9e88-dbfa-7cf4d45aeb44@oracle.com> Hi Jini, Sorry for the delay.? I've looked at it and the cleanup looks great. Thanks, Coleen On 11/27/17 10:44 PM, Jini George wrote: > Gentle reminder. > > Thank you, > Jini. > > On 11/21/2017 4:04 PM, Jini George wrote: >> Hello, >> >> Here's requesting reviews for some SA code cleanup. >> >> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >> Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >> >> The changes here are primarily to: >> >> 1. Remove unused IA64 SA code. >> 2. Make changes to avoid error-prone redefinition of hotspot >> constants in SA Java code. Instead read it in through vmStructs and >> db.lookupIntConstant(). >> 3. Make variable name changes in SA to align with the hotspot names. >> >> The changes are straightforward. >> >> Thanks much, >> Jini. >> From daniel.daugherty at oracle.com Thu Nov 30 13:02:45 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 30 Nov 2017 08:02:45 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <444b5c64-85e7-43c3-8d18-c25140806aac@oracle.com> <011f4d27-2c1e-361c-4776-8aa7cba3bca5@oracle.com> Message-ID: <46464a78-fe70-5ff4-7a7c-ee5d3b7d9ad5@oracle.com> On 11/29/17 10:46 PM, coleen.phillimore at oracle.com wrote: > > Yes, definitely the "kitchen sink collection".??? I meant instead of : > > // Safe Memory Reclamation (SMR) support: > + // The coordination between Threads::release_stable_list() and > + // Threads::smr_delete() uses the smr_delete_lock in order to > + // reduce the traffic on the Threads_lock. > static Monitor* _smr_delete_lock; > + static Monitor* smr_delete_lock() { return _smr_delete_lock; } > // The '_cnt', '_max' and '_times" fields are enabled via > // -XX:+EnableThreadSMRStatistics (see thread.cpp for a > // description about each field): > static uint _smr_delete_lock_wait_cnt; > static uint _smr_delete_lock_wait_max; > + // The smr_delete_notify flag is used for proper double-check > + // locking in order to reduce the traffic on the smr_delete_lock. > static volatile uint _smr_delete_notify; > + static bool smr_delete_notify(); > + static void set_smr_delete_notify(); > + static void clear_smr_delete_notify(); > static volatile uint _smr_deleted_thread_cnt; > + static void inc_smr_deleted_thread_cnt(); > static volatile uint _smr_deleted_thread_time_max; > + static void update_smr_deleted_thread_time_max(uint new_value); > static volatile uint _smr_deleted_thread_times; > + static void add_smr_deleted_thread_times(uint add_value); > static ThreadsList* volatile _smr_java_thread_list; > static ThreadsList* get_smr_java_thread_list(); > static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); > static uint64_t _smr_java_thread_list_alloc_cnt; > static uint64_t _smr_java_thread_list_free_cnt; > > To have the below.? They're all together but fields and functions have > been separated. > > // Safe Memory Reclamation (SMR) support: > + // The coordination between Threads::release_stable_list() and > + // Threads::smr_delete() uses the smr_delete_lock in order to > + // reduce the traffic on the Threads_lock. > static Monitor* _smr_delete_lock; > // The '_cnt', '_max' and '_times" fields are enabled via > // -XX:+EnableThreadSMRStatistics (see thread.cpp for a > // description about each field): > static uint _smr_delete_lock_wait_cnt; > static uint _smr_delete_lock_wait_max; > + // The smr_delete_notify flag is used for proper double-check > + // locking in order to reduce the traffic on the smr_delete_lock. > static volatile uint _smr_delete_notify; > static volatile uint _smr_deleted_thread_cnt; > static volatile uint _smr_deleted_thread_time_max; > static volatile uint _smr_deleted_thread_times; > static ThreadsList* volatile _smr_java_thread_list; > static uint64_t _smr_java_thread_list_alloc_cnt; > static uint64_t _smr_java_thread_list_free_cnt; > + static Monitor* smr_delete_lock() { return _smr_delete_lock; } + > static bool smr_delete_notify(); > + static void set_smr_delete_notify(); > + static void clear_smr_delete_notify(); > + static void inc_smr_deleted_thread_cnt(); > + static void update_smr_deleted_thread_time_max(uint new_value); > + static void add_smr_deleted_thread_times(uint add_value); > static ThreadsList* get_smr_java_thread_list(); > static ThreadsList* xchg_smr_java_thread_list(ThreadsList* new_list); I can do this. I will sort that list of functions though... :-) Dan > > Thanks, > Coleen > > On 11/29/17 5:11 PM, Daniel D. Daugherty wrote: >> On 11/29/17 5:04 PM, coleen.phillimore at oracle.com wrote: >>> >>> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0/src/hotspot/share/runtime/thread.hpp.udiff.html >>> >>> >>> Can you put a section with the static data member declarations, then >>> a blank line and then have a section with the functions??? We don't >>> usually mix them like that (maybe in thread.hpp because it's the >>> kitchen sink but not everywhere else).? It makes it hard to read. >> >> We're trying to keep all the "stuff" associated with a field together. >> It's definitely not a style typically in use in HotSpot, but we're >> experimenting with it because thread.hpp is currently a kitchen sink >> collection. Some folks would say it is a mess... :-) >> >> I'm going to take a look this cleanup bug also: >> >> JDK-8191789 migrate more Thread-SMR stuff from thread.[ch]pp -> >> threadSMR.[ch]pp >> https://bugs.openjdk.java.net/browse/JDK-8191789 >> >> since it will primarily be "code motion" fix also. Keeping "stuff" >> together will make that cleanup bug easier... >> >> Would you be okay with this fix (8191787) as it is? >> >> >>> Otherwise, looks fine.? Thank you for moving these! >> >> Thanks! >> >> Dan >> >> >>> >>> Coleen >>> >>> On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >>>> Greetings, >>>> >>>> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >>>> to hear from you on this thread. Thanks! >>>> >>>> I have a simple cleanup fix for Thread-SMR. The bug is: >>>> >>>> ??? JDK-8191787 move private inline functions from >>>> thread.inline.hpp -> thread.cpp >>>> https://bugs.openjdk.java.net/browse/JDK-8191787 >>>> >>>> This fix is pure code motion: >>>> >>>> - moving inline functions from thread.inline.hpp -> thread.cpp >>>> - making a few functions in thread.hpp private instead of public >>>> >>>> Here is the webrev URL: >>>> >>>> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >>>> >>>> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >>>> unexpected test failures. >>>> >>>> Thanks, in advance, for any comments, questions or suggestions. >>>> >>>> Dan >>> >> > From jini.george at oracle.com Thu Nov 30 13:03:14 2017 From: jini.george at oracle.com (Jini George) Date: Thu, 30 Nov 2017 18:33:14 +0530 Subject: RFR: JDK-8191324: SA cleanup -- part 2 In-Reply-To: <4dcfe382-488a-9e88-dbfa-7cf4d45aeb44@oracle.com> References: <4dcfe382-488a-9e88-dbfa-7cf4d45aeb44@oracle.com> Message-ID: Thank you very much, Coleen! - Jini. On 11/30/2017 6:17 PM, coleen.phillimore at oracle.com wrote: > Hi Jini, > Sorry for the delay.? I've looked at it and the cleanup looks great. > Thanks, > Coleen > > On 11/27/17 10:44 PM, Jini George wrote: >> Gentle reminder. >> >> Thank you, >> Jini. >> >> On 11/21/2017 4:04 PM, Jini George wrote: >>> Hello, >>> >>> Here's requesting reviews for some SA code cleanup. >>> >>> ID: https://bugs.openjdk.java.net/browse/JDK-8191324 >>> Webrev: http://cr.openjdk.java.net/~jgeorge/8191324/webrev.00/index.html >>> >>> The changes here are primarily to: >>> >>> 1. Remove unused IA64 SA code. >>> 2. Make changes to avoid error-prone redefinition of hotspot >>> constants in SA Java code. Instead read it in through vmStructs and >>> db.lookupIntConstant(). >>> 3. Make variable name changes in SA to align with the hotspot names. >>> >>> The changes are straightforward. >>> >>> Thanks much, >>> Jini. >>> > From daniel.daugherty at oracle.com Thu Nov 30 13:09:44 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 30 Nov 2017 08:09:44 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> Message-ID: <057d52b1-d04d-e083-3717-6faaae1e9f18@oracle.com> On 11/30/17 12:07 AM, David Holmes wrote: > Hi Dan, > > On 30/11/2017 7:16 AM, Daniel D. Daugherty wrote: >> Greetings, >> >> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >> to hear from you on this thread. Thanks! >> >> I have a simple cleanup fix for Thread-SMR. The bug is: >> >> ???? JDK-8191787 move private inline functions from thread.inline.hpp >> -> thread.cpp >> ???? https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> This fix is pure code motion: >> >> - moving inline functions from thread.inline.hpp -> thread.cpp >> - making a few functions in thread.hpp private instead of public >> >> Here is the webrev URL: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 > > 3779 inline ThreadsList* Threads::get_smr_java_thread_list() { > 3780?? return > (ThreadsList*)OrderAccess::load_acquire(&_smr_java_thread_list); > 3781 } > > Don't these inline definitions need to come before any use of them > elsewhere in the source file, to actually get the inlining benefit? I don't know. I was under the impression that some of the compilers do not properly do inlining of functions in .cpp files, but I was told during the 8167108 review that that is no longer the case with the current compilers. Hopefully one of the C++ compiler cognizant folks will chime in... > Otherwise no comments from me on the code motion. I'm abstaining from > the debate on whether to list the functions immediately after the > associated fields, or whether to group fields and functions separately > (normally different access means they are separate, but not in this > case). :) Thanks for the review! Dan > > Thanks, > David > ----- > >> >> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >> unexpected test failures. >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan From erik.osterlund at oracle.com Thu Nov 30 13:44:32 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 30 Nov 2017 14:44:32 +0100 Subject: RFR (S): 8192003: Refactor weak references in StringTable to use the Access API In-Reply-To: <81e2ffcb-4912-9fc7-55d9-0553b737a695@oracle.com> References: <5A1D93C1.3020605@oracle.com> <81e2ffcb-4912-9fc7-55d9-0553b737a695@oracle.com> Message-ID: <5A200B40.9060206@oracle.com> Hi Per, Thank you for reviewing this. New full webrev: http://cr.openjdk.java.net/~eosterlund/8192003/webrev.01/ New incremental webrev: http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00_01/ On 2017-11-30 11:32, Per Liden wrote: > Hi Erik, > > On 2017-11-28 17:50, Erik ?sterlund wrote: >> Hi, >> >> The StringTable contains weak references to oops. Today the weak >> semantics is managed using explicit calls to G1 SATB enqueue barriers. >> >> Now that the Access API is available, it should be used instead as it is >> more modular. >> >> This change fixes that by making these oops ON_PHANTOM_OOP_REF with a >> corresponding AS_NO_KEEPALIVE accessor when we do not want to keep it >> alive, much like my previous changes to other weak tables. >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00/ > > share/classfile/javaClasses.inline.hpp > -------------------------------------- > > 57 typeArrayOop java_lang_String::value_no_keepalive(oop java_string) { > 58 assert(initialized && (value_offset > 0), "Must be initialized"); > 59 assert(is_instance(java_string), "must be java_string"); > 60 oop value = > HeapAccess::oop_load_at(java_string, value_offset); > 61 return (typeArrayOop)value; > 62 } > > How about pushing this barrier down to oopDesc, with a > oopDesc::obj_field_special_access(...) function? Sounds doable. Fixed. (Although I called the method just "obj_field_special", hope nobody minds...) > > 76 typeArrayOop value = > java_lang_String::value_no_keepalive(java_string); > 77 assert(initialized, "Must be initialized"); > 78 assert(is_instance(java_string), "must be java_string"); > > Looks like you accidentally moved the value_no_keepalive() call above > the asserts? Fixed. > > share/classfile/stringTable.cpp > ------------------------------- > > 155 oop string = string_object_no_keepalive(l); > 156 if (java_lang_String::equals(string, name, len)) { > 157 return string_object(l); > 158 } > > Can we please add a comment here, saying that returning "string" would > be very bad, or maybe even fold this a bit, so that no one will be > tempted to ever return that value, something like: > > if (java_lang_String::equals(string_object_no_keepalive(l), name, len)) { > // Comment saying why we must call string_object() here... > return string_object(l); > } Fixed. Thanks, /Erik > cheers, > Per > >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8192003 >> >> Thanks, >> /Erik From rkennke at redhat.com Thu Nov 30 14:22:09 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 30 Nov 2017 15:22:09 +0100 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> Message-ID: <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> Hi David, I added the tag as you proposed: Differential: http://cr.openjdk.java.net/~rkennke/8191564/webrev.08.diff/ Full: http://cr.openjdk.java.net/~rkennke/8191564/webrev.08/ Re-run tests without regressions. Roman > On 30/11/2017 9:30 PM, Roman Kennke wrote: >> Hi David, >> >> yes I did, but it's probably got lost somewhere, while I was bouncing >> the patch between me and Erik D. (I noticed that the msg also got >> lost). Both reinstated: >> >> http://cr.openjdk.java.net/~rkennke/8191564/webrev.07/ > > Thanks - that's much simpler to follow. > > IIRC in the test I think you need "@requires vm.gc == null" to skip the > test if the framework is trying to run it with an explicit GC > configuration - else it may conflict with your hardwired GC selection. > > The overall refactoring seems reasonable, but I can't really vouch for > its accuracy. I'm not clear how/if these service APIs are accesses from > the Java-level serviceability code. > > David > >> Roman >> >>> Hi Roman, >>> >>> Just glancing at this but did you use "hg rename" to move the files? >>> The webrev suggests not. >>> >>> Thanks, >>> David >>> >>> On 30/11/2017 1:38 AM, Roman Kennke wrote: >>>> Including hotspot-runtime-dev... >>>> >>>> I need one more review (esp. for the src/hotspot/share/services part): >>>> >>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>> >>>> Thanks, Roman >>>> >>>> >>>>> On 11/29/2017 09:39 AM, Roman Kennke wrote: >>>>>> Hi Erik, >>>>>> >>>>>> thanks for the review! >>>>>> >>>>>> I think this requires another reviewer from serviceability-dev. >>>>>> Who can I ping about this? >>>>> >>>>> You could try the hotspot-runtime-dev email list, although I >>>>> suspect most of the runtime developers are on serviceability-dev >>>>> list as well... >>>>> >>>>> Thanks, >>>>> Erik >>>>> >>>>>> Roman >>>>>> >>>>>> >>>>>>> Hi Roman, >>>>>>> >>>>>>> Looks good now. Thanks for doing this. >>>>>>> >>>>>>> Thanks, >>>>>>> /Erik >>>>>>> >>>>>>> On 2017-11-28 22:26, Roman Kennke wrote: >>>>>>>> Hi Erik, >>>>>>>> >>>>>>>> thank you for reviewing! >>>>>>>> >>>>>>>> You are right. I think this is a leftover from when we tried to >>>>>>>> pass the GCMemoryManager* into the Generation constructor. The >>>>>>>> way it is done now (installing the GCMmemoryManager* later >>>>>>>> through set_memory_manager()) we can group all serviceability >>>>>>>> related set-up into initialize_serviceability(): >>>>>>>> >>>>>>>> Differential: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>>>>>> Full: >>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>>>>> >>>>>>>> Notice that I hooked this up into >>>>>>>> CollectedHeap::post_initialize() and in doing so made that >>>>>>>> method concrete, and changed all subclass post_initialize() >>>>>>>> methods to call the super-class. >>>>>>>> >>>>>>>> Good now? >>>>>>>> >>>>>>>> Thanks, Roman >>>>>>>> >>>>>>>> >>>>>>>>> Hi Roman, >>>>>>>>> >>>>>>>>> This looks better now. Nice job. >>>>>>>>> I wonder though, is it possible to extract the creation of >>>>>>>>> managers and pools to a separate function for each collected heap? >>>>>>>>> Now I see managers are created in e.g. CMS constructor, and the >>>>>>>>> pools are created in CMSHeap::initialize(). Perhaps initialize >>>>>>>>> could call initialize_serviceability() that sets up those >>>>>>>>> things, so that they are nicely separated. Unless of course >>>>>>>>> there is a good reason why the presence of managers is needed >>>>>>>>> earlier than that in the bootstrapping. >>>>>>>>> >>>>>>>>> Otherwise I think this looks good! >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> /Erik >>>>>>>>> >>>>>>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>>>>>> Hi Erik, >>>>>>>>>> >>>>>>>>>> Thanks for your review! >>>>>>>>>> >>>>>>>>>> All of the things that you mentioned should be addressed in >>>>>>>>>> the following changes: >>>>>>>>>> >>>>>>>>>> Differential: >>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>>>>>> Full: >>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>>>>>> >>>>>>>>>> Also, Erik D (H) was so kind to contribute an additional >>>>>>>>>> testcase, which is also included in the above patch. >>>>>>>>>> >>>>>>>>>> Ok now? >>>>>>>>>> >>>>>>>>>> Also, I need a review from serviceability-dev too! >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Roman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 1) The code uses the "mgr" short name for "manager" in a >>>>>>>>>>> bunch of names. I would be happy if we could write out the >>>>>>>>>>> whole thing instead of the abbreviation. >>>>>>>>>>> 2) It would be great if a generation would have a pointer to >>>>>>>>>>> its manager, so we do not have to pass around the manager >>>>>>>>>>> where we already pass around the generation (such as >>>>>>>>>>> GenCollectedHeap::collect_generation). >>>>>>>>>>> The generation could be created first, then the pools, then >>>>>>>>>>> the managers, then do something like >>>>>>>>>>> generation->set_memory_manager(x). >>>>>>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use camel >>>>>>>>>>> case. >>>>>>>>>>> >>>>>>>>>>> Otherwise I think it looks good. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> /Erik >>>>>>>>>>> >>>>>>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>>>>>> Erik implemented a few more refactorings and touch-ups, and >>>>>>>>>>>> here is our final (pending reviews) webrev: >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>>>>>> >>>>>>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>>>>>> gc-end-message, avoids dragging the GCMemoryManager through >>>>>>>>>>>> Generation and a few little related refactorings. >>>>>>>>>>>> >>>>>>>>>>>> Ok to push now? >>>>>>>>>>>> >>>>>>>>>>>> Roman >>>>>>>>>>>> >>>>>>>>>>>>> After a few more discussions with Erik I made some more >>>>>>>>>>>>> changes. >>>>>>>>>>>>> >>>>>>>>>>>>> MemoryService is now unaware of the number and meaning of >>>>>>>>>>>>> GC memory managers (minor vs major). This should be better >>>>>>>>>>>>> for GCs that don't make that distinction and/or use more >>>>>>>>>>>>> different GCs (e.g. minor, intermediate, full). >>>>>>>>>>>>> >>>>>>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>>>>>> - GCMemoryManager now has gc_end_message() which is used by >>>>>>>>>>>>> GCNotifier::pushNotification(). >>>>>>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now >>>>>>>>>>>>> accept a GCMemoryManager* instead of bull full_gc >>>>>>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>>>>>> - Generation now knows about the corresponding GCMemoryManager >>>>>>>>>>>>> >>>>>>>>>>>>> Please review the full change: >>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, Roman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> I had some off-band discussions with Erik Helin and re-did >>>>>>>>>>>>>> most of the changeset: >>>>>>>>>>>>>> - The GC interface now resides in CollectedHeap, >>>>>>>>>>>>>> specifically the two methods memory_managers() and >>>>>>>>>>>>>> memory_pools(), and is implemented in the various concrete >>>>>>>>>>>>>> subclasses. >>>>>>>>>>>>>> - Both methods return (by value) a >>>>>>>>>>>>>> GrowableArray and >>>>>>>>>>>>>> GrowableArray respectively. Returning a >>>>>>>>>>>>>> stack-allocated GrowableArray seemed least complicated >>>>>>>>>>>>>> (avoid explicit cleanup of short-lived array object), and >>>>>>>>>>>>>> most future-proof, e.g. currently there is an implicit >>>>>>>>>>>>>> expectation to get 2 GCMemoryManagers, even though some >>>>>>>>>>>>>> GCs don't necessarily have two. The API allows for easy >>>>>>>>>>>>>> extension of the situation. >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>>>>>> Serviceability team. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Roman >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Currently, there's lots of GC specific code sprinkled >>>>>>>>>>>>>>> over src/hotspot/share/services. This change introduces a >>>>>>>>>>>>>>> GC interface for that, and moves all GC specific code to >>>>>>>>>>>>>>> their respective src/hotspot/share/gc directory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none >>>>>>>>>>>>>>> showed regressions >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Roman >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >> From volker.simonis at gmail.com Thu Nov 30 14:46:04 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Thu, 30 Nov 2017 15:46:04 +0100 Subject: Several questions regarding AppCDS Message-ID: Hi Jiangli, Ioi, after AppCDS has been pushed to the open repos, I started looking at it. However, I found very little in-deep documentation of the feature. All I could find so far is basically the Oracle docs, blogs and videos I've listed below (in the hope that they may be useful for others as well). However this still doesn't answer the following questions: 1. Is there a specification and explanation of the extended class list syntax which is used for AppCDS with custom class loaders? I only found the documentation in systemDictionaryShared.hpp. But that for example mentions the "loader" keyword (as do some of the JTreg tests) without further explanation. 2. Is there a way or tool to create this extended class list automatically? As far as I can see, all the Jtreg tests which use the extended syntax, generate the class list manually. I did some simple tests where I load some classes into my own URL class loader but they don't seem to appear in the class list generated by -XX:DumpLoadedClassList. Any further references are highly welcome. Thank you and best regards, Volker PS: I really liked your JavaOne 2016 presentation :) Class Data Sharing ==================== https://docs.oracle.com/javase/9/vm/class-data-sharing.htm#JSJVM-GUID-7EAA3411-8CF0-4D19-BD05-DF5E1780AA91 Application Class Data Sharing ============================== https://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 JEP 310: Application Class-Data Sharing ======================================= http://openjdk.java.net/jeps/310 Matthew Gilliard's blog ----------------------- Fast JVM startup with JDK 9 https://mjg123.github.io/2017/10/02/JVM-startup.html Quicker Clojure startup with AppCDS and AOT https://mjg123.github.io/2017/10/04/AppCDS-and-Clojure.html Leveraging AppCDS to Optimize Application Startup and Memory Footprint in the Cloud ----------------------------------------------------------------------------------- https://www.youtube.com/watch?v=dRw77QDSL-A From erik.osterlund at oracle.com Thu Nov 30 14:55:11 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 30 Nov 2017 15:55:11 +0100 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> Message-ID: <5A201BCF.8030307@oracle.com> Hi Roman, Looks good. You do need a class GCMemoryManager; declaration in generation.hpp to build without precompiled headers though. I do not need a new webrev for that change. Thanks, /Erik On 2017-11-30 15:22, Roman Kennke wrote: > Hi David, > > I added the tag as you proposed: > > Differential: > http://cr.openjdk.java.net/~rkennke/8191564/webrev.08.diff/ > Full: > http://cr.openjdk.java.net/~rkennke/8191564/webrev.08/ > > Re-run tests without regressions. > > Roman > >> On 30/11/2017 9:30 PM, Roman Kennke wrote: >>> Hi David, >>> >>> yes I did, but it's probably got lost somewhere, while I was >>> bouncing the patch between me and Erik D. (I noticed that the msg >>> also got lost). Both reinstated: >>> >>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.07/ >> >> Thanks - that's much simpler to follow. >> >> IIRC in the test I think you need "@requires vm.gc == null" to skip >> the test if the framework is trying to run it with an explicit GC >> configuration - else it may conflict with your hardwired GC selection. >> >> The overall refactoring seems reasonable, but I can't really vouch >> for its accuracy. I'm not clear how/if these service APIs are >> accesses from the Java-level serviceability code. >> >> David >> >>> Roman >>> >>>> Hi Roman, >>>> >>>> Just glancing at this but did you use "hg rename" to move the >>>> files? The webrev suggests not. >>>> >>>> Thanks, >>>> David >>>> >>>> On 30/11/2017 1:38 AM, Roman Kennke wrote: >>>>> Including hotspot-runtime-dev... >>>>> >>>>> I need one more review (esp. for the src/hotspot/share/services >>>>> part): >>>>> >>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>> >>>>> Thanks, Roman >>>>> >>>>> >>>>>> On 11/29/2017 09:39 AM, Roman Kennke wrote: >>>>>>> Hi Erik, >>>>>>> >>>>>>> thanks for the review! >>>>>>> >>>>>>> I think this requires another reviewer from serviceability-dev. >>>>>>> Who can I ping about this? >>>>>> >>>>>> You could try the hotspot-runtime-dev email list, although I >>>>>> suspect most of the runtime developers are on serviceability-dev >>>>>> list as well... >>>>>> >>>>>> Thanks, >>>>>> Erik >>>>>> >>>>>>> Roman >>>>>>> >>>>>>> >>>>>>>> Hi Roman, >>>>>>>> >>>>>>>> Looks good now. Thanks for doing this. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> /Erik >>>>>>>> >>>>>>>> On 2017-11-28 22:26, Roman Kennke wrote: >>>>>>>>> Hi Erik, >>>>>>>>> >>>>>>>>> thank you for reviewing! >>>>>>>>> >>>>>>>>> You are right. I think this is a leftover from when we tried >>>>>>>>> to pass the GCMemoryManager* into the Generation constructor. >>>>>>>>> The way it is done now (installing the GCMmemoryManager* later >>>>>>>>> through set_memory_manager()) we can group all serviceability >>>>>>>>> related set-up into initialize_serviceability(): >>>>>>>>> >>>>>>>>> Differential: >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06.diff/ >>>>>>>>> Full: >>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.06/ >>>>>>>>> >>>>>>>>> Notice that I hooked this up into >>>>>>>>> CollectedHeap::post_initialize() and in doing so made that >>>>>>>>> method concrete, and changed all subclass post_initialize() >>>>>>>>> methods to call the super-class. >>>>>>>>> >>>>>>>>> Good now? >>>>>>>>> >>>>>>>>> Thanks, Roman >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi Roman, >>>>>>>>>> >>>>>>>>>> This looks better now. Nice job. >>>>>>>>>> I wonder though, is it possible to extract the creation of >>>>>>>>>> managers and pools to a separate function for each collected >>>>>>>>>> heap? >>>>>>>>>> Now I see managers are created in e.g. CMS constructor, and >>>>>>>>>> the pools are created in CMSHeap::initialize(). Perhaps >>>>>>>>>> initialize could call initialize_serviceability() that sets >>>>>>>>>> up those things, so that they are nicely separated. Unless of >>>>>>>>>> course there is a good reason why the presence of managers is >>>>>>>>>> needed earlier than that in the bootstrapping. >>>>>>>>>> >>>>>>>>>> Otherwise I think this looks good! >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> /Erik >>>>>>>>>> >>>>>>>>>> On 2017-11-28 12:22, Roman Kennke wrote: >>>>>>>>>>> Hi Erik, >>>>>>>>>>> >>>>>>>>>>> Thanks for your review! >>>>>>>>>>> >>>>>>>>>>> All of the things that you mentioned should be addressed in >>>>>>>>>>> the following changes: >>>>>>>>>>> >>>>>>>>>>> Differential: >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05.diff/ >>>>>>>>>>> Full: >>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.05/ >>>>>>>>>>> >>>>>>>>>>> Also, Erik D (H) was so kind to contribute an additional >>>>>>>>>>> testcase, which is also included in the above patch. >>>>>>>>>>> >>>>>>>>>>> Ok now? >>>>>>>>>>> >>>>>>>>>>> Also, I need a review from serviceability-dev too! >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Roman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 1) The code uses the "mgr" short name for "manager" in a >>>>>>>>>>>> bunch of names. I would be happy if we could write out the >>>>>>>>>>>> whole thing instead of the abbreviation. >>>>>>>>>>>> 2) It would be great if a generation would have a pointer >>>>>>>>>>>> to its manager, so we do not have to pass around the >>>>>>>>>>>> manager where we already pass around the generation (such >>>>>>>>>>>> as GenCollectedHeap::collect_generation). >>>>>>>>>>>> The generation could be created first, then the pools, then >>>>>>>>>>>> the managers, then do something like >>>>>>>>>>>> generation->set_memory_manager(x). >>>>>>>>>>>> 3) In cmsHeap.cpp:54: maxSize should preferably not use >>>>>>>>>>>> camel case. >>>>>>>>>>>> >>>>>>>>>>>> Otherwise I think it looks good. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> /Erik >>>>>>>>>>>> >>>>>>>>>>>> On 2017-11-27 10:30, Roman Kennke wrote: >>>>>>>>>>>>> Erik implemented a few more refactorings and touch-ups, >>>>>>>>>>>>> and here is our final (pending reviews) webrev: >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.04/ >>>>>>>>>>>>> >>>>>>>>>>>>> Compared to webrev.02, it improves the handling of >>>>>>>>>>>>> gc-end-message, avoids dragging the GCMemoryManager >>>>>>>>>>>>> through Generation and a few little related refactorings. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok to push now? >>>>>>>>>>>>> >>>>>>>>>>>>> Roman >>>>>>>>>>>>> >>>>>>>>>>>>>> After a few more discussions with Erik I made some more >>>>>>>>>>>>>> changes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> MemoryService is now unaware of the number and meaning of >>>>>>>>>>>>>> GC memory managers (minor vs major). This should be >>>>>>>>>>>>>> better for GCs that don't make that distinction and/or >>>>>>>>>>>>>> use more different GCs (e.g. minor, intermediate, full). >>>>>>>>>>>>>> >>>>>>>>>>>>>> This means that I needed to add some abstractions: >>>>>>>>>>>>>> - GCMemoryManager now has gc_end_message() which is used >>>>>>>>>>>>>> by GCNotifier::pushNotification(). >>>>>>>>>>>>>> - gc_begin() and gc_end() methods in MemoryService now >>>>>>>>>>>>>> accept a GCMemoryManager* instead of bull full_gc >>>>>>>>>>>>>> - Same for TraceMemoryManagerStats >>>>>>>>>>>>>> - Generation now knows about the corresponding >>>>>>>>>>>>>> GCMemoryManager >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please review the full change: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.02/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Roman >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I had some off-band discussions with Erik Helin and >>>>>>>>>>>>>>> re-did most of the changeset: >>>>>>>>>>>>>>> - The GC interface now resides in CollectedHeap, >>>>>>>>>>>>>>> specifically the two methods memory_managers() and >>>>>>>>>>>>>>> memory_pools(), and is implemented in the various >>>>>>>>>>>>>>> concrete subclasses. >>>>>>>>>>>>>>> - Both methods return (by value) a >>>>>>>>>>>>>>> GrowableArray and >>>>>>>>>>>>>>> GrowableArray respectively. Returning a >>>>>>>>>>>>>>> stack-allocated GrowableArray seemed least complicated >>>>>>>>>>>>>>> (avoid explicit cleanup of short-lived array object), >>>>>>>>>>>>>>> and most future-proof, e.g. currently there is an >>>>>>>>>>>>>>> implicit expectation to get 2 GCMemoryManagers, even >>>>>>>>>>>>>>> though some GCs don't necessarily have two. The API >>>>>>>>>>>>>>> allows for easy extension of the situation. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think this requires reviews from both the GC and >>>>>>>>>>>>>>> Serviceability team. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Roman >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Currently, there's lots of GC specific code sprinkled >>>>>>>>>>>>>>>> over src/hotspot/share/services. This change introduces >>>>>>>>>>>>>>>> a GC interface for that, and moves all GC specific code >>>>>>>>>>>>>>>> to their respective src/hotspot/share/gc directory. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~rkennke/8191564/webrev.00/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Testing: hotspot_gc and hotspot_serviceability, none >>>>>>>>>>>>>>>> showed regressions >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Built minimal and server without regressions >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Roman >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> > From per.liden at oracle.com Thu Nov 30 15:43:20 2017 From: per.liden at oracle.com (=?utf-8?Q?Per_Lid=C3=A9n?=) Date: Thu, 30 Nov 2017 16:43:20 +0100 Subject: RFR (S): 8192003: Refactor weak references in StringTable to use the Access API In-Reply-To: <5A200B40.9060206@oracle.com> References: <5A1D93C1.3020605@oracle.com> <81e2ffcb-4912-9fc7-55d9-0553b737a695@oracle.com> <5A200B40.9060206@oracle.com> Message-ID: Looks good! /Per > On 30 Nov 2017, at 14:44, Erik ?sterlund wrote: > > Hi Per, > > Thank you for reviewing this. > > New full webrev: > http://cr.openjdk.java.net/~eosterlund/8192003/webrev.01/ > > New incremental webrev: > http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00_01/ > >> On 2017-11-30 11:32, Per Liden wrote: >> Hi Erik, >> >>> On 2017-11-28 17:50, Erik ?sterlund wrote: >>> Hi, >>> >>> The StringTable contains weak references to oops. Today the weak >>> semantics is managed using explicit calls to G1 SATB enqueue barriers. >>> >>> Now that the Access API is available, it should be used instead as it is >>> more modular. >>> >>> This change fixes that by making these oops ON_PHANTOM_OOP_REF with a >>> corresponding AS_NO_KEEPALIVE accessor when we do not want to keep it >>> alive, much like my previous changes to other weak tables. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00/ >> >> share/classfile/javaClasses.inline.hpp >> -------------------------------------- >> >> 57 typeArrayOop java_lang_String::value_no_keepalive(oop java_string) { >> 58 assert(initialized && (value_offset > 0), "Must be initialized"); >> 59 assert(is_instance(java_string), "must be java_string"); >> 60 oop value = HeapAccess::oop_load_at(java_string, value_offset); >> 61 return (typeArrayOop)value; >> 62 } >> >> How about pushing this barrier down to oopDesc, with a oopDesc::obj_field_special_access(...) function? > > Sounds doable. Fixed. (Although I called the method just "obj_field_special", hope nobody minds...) > >> >> 76 typeArrayOop value = java_lang_String::value_no_keepalive(java_string); >> 77 assert(initialized, "Must be initialized"); >> 78 assert(is_instance(java_string), "must be java_string"); >> >> Looks like you accidentally moved the value_no_keepalive() call above the asserts? > > Fixed. > >> >> share/classfile/stringTable.cpp >> ------------------------------- >> >> 155 oop string = string_object_no_keepalive(l); >> 156 if (java_lang_String::equals(string, name, len)) { >> 157 return string_object(l); >> 158 } >> >> Can we please add a comment here, saying that returning "string" would be very bad, or maybe even fold this a bit, so that no one will be tempted to ever return that value, something like: >> >> if (java_lang_String::equals(string_object_no_keepalive(l), name, len)) { >> // Comment saying why we must call string_object() here... >> return string_object(l); >> } > > Fixed. > > Thanks, > /Erik > >> cheers, >> Per >> >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8192003 >>> >>> Thanks, >>> /Erik > From daniel.daugherty at oracle.com Thu Nov 30 15:49:43 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 30 Nov 2017 10:49:43 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> Message-ID: <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> Greetings, I have updated the fix based on Coleen's and David H's code reviews. Delta webrev: http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-delta/ Full webrev: http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-full/ Thanks, in advance, for any comments, questions or suggestions. Dan On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: > Greetings, > > Coleen, this is one of your Thread-SMR follow-up suggestions so I need > to hear from you on this thread. Thanks! > > I have a simple cleanup fix for Thread-SMR. The bug is: > > ??? JDK-8191787 move private inline functions from thread.inline.hpp > -> thread.cpp > ??? https://bugs.openjdk.java.net/browse/JDK-8191787 > > This fix is pure code motion: > > - moving inline functions from thread.inline.hpp -> thread.cpp > - making a few functions in thread.hpp private instead of public > > Here is the webrev URL: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 > > This fix was (over) tested with a Mach5 tier[1-5] run. There were no > unexpected test failures. > > Thanks, in advance, for any comments, questions or suggestions. > > Dan > From harold.seigel at oracle.com Thu Nov 30 16:15:56 2017 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 30 Nov 2017 11:15:56 -0500 Subject: RFR 8154587: Resolution fails for default method named 'clone' Message-ID: <22c41169-4c42-bff3-5090-a27ad9019bff@oracle.com> Hi, Please review this fix for JDK-8154587.? The fix adds additional special casing to skip over non-public methods in class java.lang.Object during default method and itable processing for interfaces.? These methods need to be skipped over because of the interface method resolution rules in JVM Spec 9, section 5.4.3.4 : 3. Otherwise, if the class|Object|declares a method with the name and descriptor specified by the interface method reference, which has its|ACC_PUBLIC|flag set and does not have its|ACC_STATIC|flag set, method lookup succeeds. Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8154587/webrev/ JBS Bug:? https://bugs.openjdk.java.net/browse/JDK-8154587 The fix was tested with JCK lang and VM tests, JTReg hotspot and many JTReg JDK tests, M5 tier1 - tier5 tests, and JPRT. Thanks, Harold From erik.osterlund at oracle.com Thu Nov 30 16:17:44 2017 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Thu, 30 Nov 2017 17:17:44 +0100 Subject: RFR (S): 8192003: Refactor weak references in StringTable to use the Access API In-Reply-To: References: <5A1D93C1.3020605@oracle.com> <81e2ffcb-4912-9fc7-55d9-0553b737a695@oracle.com> <5A200B40.9060206@oracle.com> Message-ID: <39525A98-9025-417F-8EFE-B5FF6EE3180A@oracle.com> Hi Per, Thanks for the review. /Erik > On 30 Nov 2017, at 16:43, Per Lid?n wrote: > > Looks good! > > /Per > >> On 30 Nov 2017, at 14:44, Erik ?sterlund wrote: >> >> Hi Per, >> >> Thank you for reviewing this. >> >> New full webrev: >> http://cr.openjdk.java.net/~eosterlund/8192003/webrev.01/ >> >> New incremental webrev: >> http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00_01/ >> >>> On 2017-11-30 11:32, Per Liden wrote: >>> Hi Erik, >>> >>>> On 2017-11-28 17:50, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> The StringTable contains weak references to oops. Today the weak >>>> semantics is managed using explicit calls to G1 SATB enqueue barriers. >>>> >>>> Now that the Access API is available, it should be used instead as it is >>>> more modular. >>>> >>>> This change fixes that by making these oops ON_PHANTOM_OOP_REF with a >>>> corresponding AS_NO_KEEPALIVE accessor when we do not want to keep it >>>> alive, much like my previous changes to other weak tables. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8192003/webrev.00/ >>> >>> share/classfile/javaClasses.inline.hpp >>> -------------------------------------- >>> >>> 57 typeArrayOop java_lang_String::value_no_keepalive(oop java_string) { >>> 58 assert(initialized && (value_offset > 0), "Must be initialized"); >>> 59 assert(is_instance(java_string), "must be java_string"); >>> 60 oop value = HeapAccess::oop_load_at(java_string, value_offset); >>> 61 return (typeArrayOop)value; >>> 62 } >>> >>> How about pushing this barrier down to oopDesc, with a oopDesc::obj_field_special_access(...) function? >> >> Sounds doable. Fixed. (Although I called the method just "obj_field_special", hope nobody minds...) >> >>> >>> 76 typeArrayOop value = java_lang_String::value_no_keepalive(java_string); >>> 77 assert(initialized, "Must be initialized"); >>> 78 assert(is_instance(java_string), "must be java_string"); >>> >>> Looks like you accidentally moved the value_no_keepalive() call above the asserts? >> >> Fixed. >> >>> >>> share/classfile/stringTable.cpp >>> ------------------------------- >>> >>> 155 oop string = string_object_no_keepalive(l); >>> 156 if (java_lang_String::equals(string, name, len)) { >>> 157 return string_object(l); >>> 158 } >>> >>> Can we please add a comment here, saying that returning "string" would be very bad, or maybe even fold this a bit, so that no one will be tempted to ever return that value, something like: >>> >>> if (java_lang_String::equals(string_object_no_keepalive(l), name, len)) { >>> // Comment saying why we must call string_object() here... >>> return string_object(l); >>> } >> >> Fixed. >> >> Thanks, >> /Erik >> >>> cheers, >>> Per >>> >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8192003 >>>> >>>> Thanks, >>>> /Erik >> > From calvin.cheung at oracle.com Thu Nov 30 17:49:32 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 30 Nov 2017 09:49:32 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive Message-ID: <5A2044AC.5040204@oracle.com> For the regular CDS case (UseAppCDS=0), only boot classes are loaded from the archive. Therefore, we should not invalidate the archive when the bootclasspath is appended. bug: https://bugs.openjdk.java.net/browse/JDK-8174101 webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, windows-x64. thanks, Calvin From martin.doerr at sap.com Thu Nov 30 18:08:57 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Thu, 30 Nov 2017 18:08:57 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review In-Reply-To: References: <1509719477.3207.10.camel@oracle.com> <9b259cdd-111a-0634-80a1-0c6ba1a8d260@oracle.com> <4171dc3d-3a43-a455-0efb-ee5ff5640b93@oracle.com> <1510605610.7132.3.camel@oracle.com> <3e6131cd-bac3-c8db-971f-297bb13b087d@oracle.com> Message-ID: Hi, I just noticed that "MAP_NORESERVE" is not defined on AIX (os_posix.cpp reserve_mmapped_memory). I guess it's not POSIX, so this looks like a bug. Would it make sense to replace it by "NOT_AIX( | MAP_NORESERVE )"? Or is there a better idea? Best regards, Martin -----Original Message----- From: hotspot-gc-dev [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Kharbas, Kishor Sent: Montag, 13. November 2017 22:33 To: sangheon.kim ; Thomas Schatzl ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Cc: Kharbas, Kishor Subject: RE: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Thanks Sangheon and Thomas! -----Original Message----- From: sangheon.kim [mailto:sangheon.kim at oracle.com] Sent: Monday, November 13, 2017 12:41 PM To: Thomas Schatzl ; Kharbas, Kishor ; 'hotspot-gc-dev at openjdk.java.net' ; hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Hi Kishor, On 11/13/2017 12:40 PM, Thomas Schatzl wrote: > Hi Kishor, > > On Mon, 2017-11-13 at 19:40 +0000, Kharbas, Kishor wrote: >> Greetings, >> >> I have an updated webrev to remove compilation warning on Windows 32- >> bit. >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.15_to_14/ >> >> Sorry missed this earlier. I request for a review on this update. > looks good. The other changes from webrev.13 on also look good. +1 Thanks, Sangheon > > Thanks, > Thomas > >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 4:07 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> >> On 11/03/2017 02:59 PM, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Here is link to the updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.14_to_13/ >> Looks good to me. >> >> Thanks, >> Sangheon >> >> >> >> >> Thanks >> Kishor >> >> From: sangheon.kim [mailto:sangheon.kim at oracle.com] >> Sent: Friday, November 3, 2017 2:38 PM >> To: Kharbas, Kishor ; Thomas Schatzl > s.schatzl at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' > dev at openjdk.java.net>; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi Kishor, >> >> On 11/03/2017 11:39 AM, Kharbas, Kishor wrote: >> Thanks a lot! >> >> Link to updated webrevs - >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13/ >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.13_to_12/ >> Thank you for fixing all. >> Looks good to me except below. >> >> Could you update the copyright format in TestAllocateHeapAt.java? >> 2? * Copyright (c) 2017 Oracle and/or its affiliates. All rights >> reserved. >> - Missing comma: * Copyright (c) 2017, Oracle and/or its affiliates. >> All rights reserved. >> >> Thanks, >> Sangheon >> >> >> >> >> >> >> @Sangheon: Please let me know if you see any corrections needed. >> >> -Kishor >> >> -----Original Message----- >> From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] >> Sent: Friday, November 3, 2017 7:31 AM >> To: Kharbas, Kishor ; sangheon.kim >> ; 'hotspot-gc-dev at openjdk.java.net' >> ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8190308: Supporting heap allocation on >> alternative memory devices and CSR review >> >> Hi, >> >> On Fri, 2017-11-03 at 08:55 +0000, Kharbas, Kishor wrote: >> Hi Sangheon, >> >> Thanks for the review and comments. Here is an updated webrev- >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12 >> http://cr.openjdk.java.net/~kkharbas/8190308/webrev.12_to_11 >> >> In addition to your suggested corrections, I added code to set Linux >> core dump filter ensuring Heap is dumped correctly when this feature >> is used. This is follow-up to Jini George?s comment >> (http://openjdk.5641.n7.nabble.com/RFR-M-8171181-Supporting-heap- >> allocation-on-alternative-memory-devices-td300109.html#a300450). >> >> Some minor nits: >> >> ?- os_posix.cpp:300: please move the else next to the brace >> ?- arguments.cpp:4624: please add a space between "if" and the >> bracket >> >> I do not need to see a new webrev for these changes. Looks good. >> >> Thanks, >> ? Thomas >> >> >> From mandy.chung at oracle.com Thu Nov 30 18:12:01 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 30 Nov 2017 10:12:01 -0800 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> Message-ID: On 11/30/17 6:22 AM, Roman Kennke wrote: > Hi David, > > I added the tag as you proposed: > > Differential: > http://cr.openjdk.java.net/~rkennke/8191564/webrev.08.diff/ > Full: > http://cr.openjdk.java.net/~rkennke/8191564/webrev.08/ > This looks really good.? This version looks much better than an early version I looked at (thanks for regenerating webrev with hg rename). A minor comment that GCMemoryManager could take an enum to indicate the type of this? GC action (major vs minor).?? This can be a future cleanup. thanks Mandy From rkennke at redhat.com Thu Nov 30 18:26:57 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 30 Nov 2017 19:26:57 +0100 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> Message-ID: <2da42a09-73d7-6f3e-b027-337ff6ff29a2@redhat.com> Hi Mandy, thanks for reviewing! I think Erik ?. already pushed it for me. > A minor comment that GCMemoryManager could take an enum to indicate the > type of this? GC action (major vs minor).?? This can be a future cleanup. This may be overly restrictive: some GCs may not necessarily provide a major/minor distinction, but have only one GC manager (e.g. epsilon), or may provide 3 or more GC managers (e.g. 'minor' partial (or young) GC, 'intermediate' concurrent GC, 'intermediate or major' concurrent full GC, 'major' STW last-ditch full GC...). Roman From ioi.lam at oracle.com Thu Nov 30 18:28:25 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 30 Nov 2017 10:28:25 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive In-Reply-To: <5A2044AC.5040204@oracle.com> References: <5A2044AC.5040204@oracle.com> Message-ID: Hi Calvin, Looks good. Thanks for fixing this. - Ioi On 11/30/17 9:49 AM, Calvin Cheung wrote: > For the regular CDS case (UseAppCDS=0), only boot classes are loaded > from the archive. Therefore, we should not invalidate the archive when > the bootclasspath is appended. > > bug: https://bugs.openjdk.java.net/browse/JDK-8174101 > > webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ > > Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, > windows-x64. > > thanks, > Calvin From jiangli.zhou at oracle.com Thu Nov 30 18:40:51 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 30 Nov 2017 10:40:51 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive In-Reply-To: <5A2044AC.5040204@oracle.com> References: <5A2044AC.5040204@oracle.com> Message-ID: <63664BE3-647C-4BB8-A6EC-B20906263463@oracle.com> Hi Calvin, The fix looks ok to me. Could you please add a comment/note above the line130 change: ?In the future we should perform the check based on the content of the mapped archive.? Thanks, Jiangli > On Nov 30, 2017, at 9:49 AM, Calvin Cheung wrote: > > For the regular CDS case (UseAppCDS=0), only boot classes are loaded from the archive. Therefore, we should not invalidate the archive when the bootclasspath is appended. > > bug: https://bugs.openjdk.java.net/browse/JDK-8174101 > > webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ > > Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, windows-x64. > > thanks, > Calvin From mandy.chung at oracle.com Thu Nov 30 18:41:58 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 30 Nov 2017 10:41:58 -0800 Subject: RFR: 8191564: Refactor GC related servicability code into GC specific subclasses In-Reply-To: <2da42a09-73d7-6f3e-b027-337ff6ff29a2@redhat.com> References: <387d2b93-cb8b-51be-a6e0-3b5f9d6cbb1c@redhat.com> <713325a4-3ed8-cd7d-c66d-a1c1767b6d3a@redhat.com> <2a7e6307-8c0d-ad4b-e50c-a7e1c2311080@redhat.com> <56567db1-caf6-ed05-ed12-336edfca0c75@redhat.com> <5A1C3C07.30800@oracle.com> <5A1D8BE7.4070008@oracle.com> <4b899587-f7b4-82d7-4917-41d202e26ef4@oracle.com> <6f47a8ab-8b7e-c19c-2e49-10664ca90dd9@redhat.com> <2a0f294a-b5d8-b2a8-971d-b1aa6146db2d@redhat.com> <5c51ba9c-f64c-b8a9-7e46-1edeffb414ba@oracle.com> <243153d8-1836-9d3e-b242-52e17b2e2134@redhat.com> <2da42a09-73d7-6f3e-b027-337ff6ff29a2@redhat.com> Message-ID: <67b97a9b-ccd0-3232-ae4c-2a24dc9d605f@oracle.com> On 11/30/17 10:26 AM, Roman Kennke wrote: > Hi Mandy, > > thanks for reviewing! I think Erik ?. already pushed it for me. > >> A minor comment that GCMemoryManager could take an enum to indicate >> the type of this? GC action (major vs minor).?? This can be a future >> cleanup. > > This may be overly restrictive: some GCs may not necessarily provide a > major/minor distinction, but have only one GC manager (e.g. epsilon), > or may provide 3 or more GC managers (e.g. 'minor' partial (or young) > GC, 'intermediate' concurrent GC, 'intermediate or major' concurrent > full GC, 'major' STW last-ditch full GC...). Right - there will be more than 2 enums and major and major are just examples that could let GC notififer to determine the message based on the type.? This is minor and just a thought that can be considered in the future. Thanks. Mandy From coleen.phillimore at oracle.com Thu Nov 30 18:57:40 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 30 Nov 2017 13:57:40 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> Message-ID: Looks great! Coleen On 11/30/17 10:49 AM, Daniel D. Daugherty wrote: > Greetings, > > I have updated the fix based on Coleen's and David H's code reviews. > > Delta webrev: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-delta/ > > Full webrev: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-full/ > > Thanks, in advance, for any comments, questions or suggestions. > > Dan > > On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >> to hear from you on this thread. Thanks! >> >> I have a simple cleanup fix for Thread-SMR. The bug is: >> >> ??? JDK-8191787 move private inline functions from thread.inline.hpp >> -> thread.cpp >> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> This fix is pure code motion: >> >> - moving inline functions from thread.inline.hpp -> thread.cpp >> - making a few functions in thread.hpp private instead of public >> >> Here is the webrev URL: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >> >> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >> unexpected test failures. >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan >> > From daniel.daugherty at oracle.com Thu Nov 30 19:06:10 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 30 Nov 2017 14:06:10 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> Message-ID: Thanks for taking another look! Dan On 11/30/17 1:57 PM, coleen.phillimore at oracle.com wrote: > > Looks great! > Coleen > > On 11/30/17 10:49 AM, Daniel D. Daugherty wrote: >> Greetings, >> >> I have updated the fix based on Coleen's and David H's code reviews. >> >> Delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-delta/ >> >> Full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-full/ >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan >> >> On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >>> to hear from you on this thread. Thanks! >>> >>> I have a simple cleanup fix for Thread-SMR. The bug is: >>> >>> ??? JDK-8191787 move private inline functions from thread.inline.hpp >>> -> thread.cpp >>> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >>> >>> This fix is pure code motion: >>> >>> - moving inline functions from thread.inline.hpp -> thread.cpp >>> - making a few functions in thread.hpp private instead of public >>> >>> Here is the webrev URL: >>> >>> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >>> >>> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >>> unexpected test failures. >>> >>> Thanks, in advance, for any comments, questions or suggestions. >>> >>> Dan >>> >> > From jiangli.zhou at oracle.com Thu Nov 30 19:27:30 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 30 Nov 2017 11:27:30 -0800 Subject: Several questions regarding AppCDS In-Reply-To: References: Message-ID: <08A854C2-BC11-43E3-9FB9-98E6050D098A@oracle.com> Hi Volker, Just some high level information for your questions. We are continually investigating the usage/requirement of the CDS/AppCDS support for custom class loaders and the current implementation may change when the investigation is maturing. That?s part of reason why there is no extended public documentation about the custom class loader support and external tool for generating the new class list available at the time. Ioi probably will fill in more details on this topic. Thanks, Jiangli > On Nov 30, 2017, at 6:46 AM, Volker Simonis wrote: > > Hi Jiangli, Ioi, > > after AppCDS has been pushed to the open repos, I started looking at > it. However, I found very little in-deep documentation of the feature. > All I could find so far is basically the Oracle docs, blogs and videos > I've listed below (in the hope that they may be useful for others as > well). > > However this still doesn't answer the following questions: > > 1. Is there a specification and explanation of the extended class list > syntax which is used for AppCDS with custom class loaders? > > I only found the documentation in systemDictionaryShared.hpp. But that > for example mentions the "loader" keyword (as do some of the JTreg > tests) without further explanation. > > 2. Is there a way or tool to create this extended class list automatically? > > As far as I can see, all the Jtreg tests which use the extended > syntax, generate the class list manually. I did some simple tests > where I load some classes into my own URL class loader but they don't > seem to appear in the class list generated by -XX:DumpLoadedClassList. > > Any further references are highly welcome. > > Thank you and best regards, > Volker > > PS: I really liked your JavaOne 2016 presentation :) > > Class Data Sharing > ==================== > https://docs.oracle.com/javase/9/vm/class-data-sharing.htm#JSJVM-GUID-7EAA3411-8CF0-4D19-BD05-DF5E1780AA91 > > Application Class Data Sharing > ============================== > https://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > JEP 310: Application Class-Data Sharing > ======================================= > http://openjdk.java.net/jeps/310 > > Matthew Gilliard's blog > ----------------------- > Fast JVM startup with JDK 9 > https://mjg123.github.io/2017/10/02/JVM-startup.html > > Quicker Clojure startup with AppCDS and AOT > https://mjg123.github.io/2017/10/04/AppCDS-and-Clojure.html > > Leveraging AppCDS to Optimize Application Startup and Memory Footprint > in the Cloud > ----------------------------------------------------------------------------------- > https://www.youtube.com/watch?v=dRw77QDSL-A From calvin.cheung at oracle.com Thu Nov 30 19:34:31 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 30 Nov 2017 11:34:31 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive In-Reply-To: References: <5A2044AC.5040204@oracle.com> Message-ID: <5A205D47.7030203@oracle.com> Thanks, Ioi. Calvin On 11/30/17, 10:28 AM, Ioi Lam wrote: > Hi Calvin, > > Looks good. Thanks for fixing this. > > - Ioi > > > On 11/30/17 9:49 AM, Calvin Cheung wrote: >> For the regular CDS case (UseAppCDS=0), only boot classes are loaded >> from the archive. Therefore, we should not invalidate the archive >> when the bootclasspath is appended. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8174101 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ >> >> Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, >> windows-x64. >> >> thanks, >> Calvin > From kim.barrett at oracle.com Thu Nov 30 19:34:43 2017 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 30 Nov 2017 14:34:43 -0500 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> Message-ID: <5E2E2948-7E5A-4227-903C-A7A0A5B57D20@oracle.com> > On Nov 29, 2017, at 8:48 AM, Erik ?sterlund wrote: > > Hi Kim, > > I know how you feel about casts, and I also saw how Andrew Haley wanted a more explicit comment about why it needs to be volatile. > To make both of you happy, I thought I'd try to make the address (addr()) in MemoryAccess volatile T*. That way I can write a more explicit comment about why it is volatile in one single place, and make you happier to not cast the address to something else than it was declared. > > I hope this makes both of you happy. If not, I am okay with the old variant too, and write a comment that I copy around instead. > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ > > Incremental webrev: > http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ Sorry to be late; I hadn't noticed there was followup and a new version yesterday until Coleen pointed it out to me. This is okay, though I think better would be to make addr() a function template (with a non-deducable type parameter), rather than making MemoryAccess a class template. The point of that difference is the principle of minimizing the dependencies between members and type parameters of a generic class (in this case by not making the class generic at all). From calvin.cheung at oracle.com Thu Nov 30 19:35:46 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 30 Nov 2017 11:35:46 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive In-Reply-To: <63664BE3-647C-4BB8-A6EC-B20906263463@oracle.com> References: <5A2044AC.5040204@oracle.com> <63664BE3-647C-4BB8-A6EC-B20906263463@oracle.com> Message-ID: <5A205D92.5070802@oracle.com> Thanks, Jiangli. I'll add the comment before push. Calvin On 11/30/17, 10:40 AM, Jiangli Zhou wrote: > Hi Calvin, > > The fix looks ok to me. Could you please add a comment/note above the line130 change: > > ?In the future we should perform the check based on the content of the mapped archive.? > > Thanks, > Jiangli > >> On Nov 30, 2017, at 9:49 AM, Calvin Cheung wrote: >> >> For the regular CDS case (UseAppCDS=0), only boot classes are loaded from the archive. Therefore, we should not invalidate the archive when the bootclasspath is appended. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8174101 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ >> >> Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, windows-x64. >> >> thanks, >> Calvin From jiangli.zhou at oracle.com Thu Nov 30 19:41:56 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 30 Nov 2017 11:41:56 -0800 Subject: [10] RFR(XS): 8174101: Bootclasspath append should not invalidate CDS archive In-Reply-To: <5A205D92.5070802@oracle.com> References: <5A2044AC.5040204@oracle.com> <63664BE3-647C-4BB8-A6EC-B20906263463@oracle.com> <5A205D92.5070802@oracle.com> Message-ID: <9F26CB3C-CEFD-417C-87F4-63C525CF9EAD@oracle.com> Thanks! Jiangli > On Nov 30, 2017, at 11:35 AM, Calvin Cheung wrote: > > Thanks, Jiangli. > I'll add the comment before push. > > Calvin > > On 11/30/17, 10:40 AM, Jiangli Zhou wrote: >> Hi Calvin, >> >> The fix looks ok to me. Could you please add a comment/note above the line130 change: >> >> ?In the future we should perform the check based on the content of the mapped archive.? >> >> Thanks, >> Jiangli >> >>> On Nov 30, 2017, at 9:49 AM, Calvin Cheung wrote: >>> >>> For the regular CDS case (UseAppCDS=0), only boot classes are loaded from the archive. Therefore, we should not invalidate the archive when the bootclasspath is appended. >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8174101 >>> >>> webrev: http://cr.openjdk.java.net/~ccheung/8174101/webrev.00/ >>> >>> Testing: hs-tier1 and hs-tier2 on linux-x64, solaris-sparc, macosx, windows-x64. >>> >>> thanks, >>> Calvin From erik.osterlund at oracle.com Thu Nov 30 19:50:02 2017 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Thu, 30 Nov 2017 20:50:02 +0100 Subject: RFR (S): 8186787: clang-4.0 SIGSEGV in Unsafe_PutByte In-Reply-To: <5E2E2948-7E5A-4227-903C-A7A0A5B57D20@oracle.com> References: <5A1C06D8.1040405@oracle.com> <7cf70296-585f-7c61-2a9c-14f39084e8c6@oracle.com> <5E2E2948-7E5A-4227-903C-A7A0A5B57D20@oracle.com> Message-ID: Hi Kim, Thanks for the review. /Erik On 30 Nov 2017, at 20:34, Kim Barrett wrote: >> On Nov 29, 2017, at 8:48 AM, Erik ?sterlund wrote: >> >> Hi Kim, >> >> I know how you feel about casts, and I also saw how Andrew Haley wanted a more explicit comment about why it needs to be volatile. >> To make both of you happy, I thought I'd try to make the address (addr()) in MemoryAccess volatile T*. That way I can write a more explicit comment about why it is volatile in one single place, and make you happier to not cast the address to something else than it was declared. >> >> I hope this makes both of you happy. If not, I am okay with the old variant too, and write a comment that I copy around instead. >> >> Full webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.01/ >> >> Incremental webrev: >> http://cr.openjdk.java.net/~eosterlund/8186787/webrev.00_01/ > > Sorry to be late; I hadn't noticed there was followup and a new > version yesterday until Coleen pointed it out to me. > > This is okay, though I think better would be to make addr() a function > template (with a non-deducable type parameter), rather than making > MemoryAccess a class template. The point of that difference is the > principle of minimizing the dependencies between members and type > parameters of a generic class (in this case by not making the class > generic at all). > From mikhailo.seledtsov at oracle.com Thu Nov 30 20:01:28 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Thu, 30 Nov 2017 12:01:28 -0800 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> Message-ID: <804ff854-2010-cc6a-851c-ee20c5de1c96@oracle.com> Bob, David, ? Thank you for review. ?? Here is an updated webrev addressing your comments: ??? http://cr.openjdk.java.net/~mseledtsov/8191943.02/ Misha On 11/29/2017 09:28 PM, David Holmes wrote: > On 30/11/2017 10:08 AM, mikhailo wrote: >> Here is an updated webrev: >> >> ???? http://cr.openjdk.java.net/~mseledtsov/8191943.01/ > > As Bob noted you may still need the guard here: > > ? 55???????????? testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); > ? 56???????????? testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2); I introduced a proper check for this. > > though perhaps docker ignores invalid cpu-set values? > > 161???????? Common.logNewTestCase("expectedAPC = " + expectedAPC); > > That should still be a System.out.println - you already set the test > case prior: I changed code to output both the original expected APC and the adjusted APC. > > 155???????? Common.logNewTestCase("test cpu shares, shares = " + shares); > > Having the same check duplicated three times is a bit messy, but not > horrendously so. :) Introduced a common method to conditionally adjust APC. Thank you, Misha > > Thanks, > David > >> >> Misha >> >> >> On 11/29/2017 01:32 PM, mikhailo wrote: >>> >>> On 11/29/2017 01:29 PM, Bob Vandette wrote: >>>>> On Nov 29, 2017, at 4:10 PM, mikhailo >>>>> wrote: >>>>> >>>>> Hi Bob, >>>>> >>>>> The failure was caused by invoking this test case: >>>>> >>>>> ???? testCpuShares(4096, 4); >>>>> ???? The test case sets --cpu-shares to 4096, expects active >>>>> processor count to be 4; ==> actual APC is 2. >>>> Ahh, that makes sense.? I thought docker was complaining due to the >>>> arguments being passed going >>>> beyond the available cpus. >>>> >>>>> Command: >>>>> >>>>> ???? docker run --tty=true --rm --cpu-shares=4096 >>>>> jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >>>>> >>>>> Detailed log is attached at: >>>>> >>>>> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >>>>> >>>>> >>>>> >>>>> Once I saw this issue, I decided to put limits on other test cases >>>>> based on the number of processors available on a machine, just to >>>>> be on a safe side. >>>>> >>>>> Perhaps a better approach here is to change the expected APC >>>>> inside the testCpuShares() to be no greater than max number of >>>>> available processors? >>>> Yes, the selection algorithm returns the smallest number of >>>> computed shares, quotas or active processors.? Using this approach, >>>> you?d actually be validating the algorithm more precisely. >>> OK. I will update the tests to do this kind of validation, and post >>> a new webrev. >>> >>> >>> Misha >>>> >>>> Bob. >>>> >>>> >>>>> >>>>> Misha >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>>>>> Misha, >>>>>> >>>>>> Which specific subtest failed on a 2 cpu system? >>>>>> >>>>>> I don?t see any subtests that should have failed.? You should be >>>>>> able >>>>>> to pass any quota, period and share value to docker independent >>>>>> of the >>>>>> number of CPUs.? testCpus and testAPCCombo don?t try to test more >>>>>> than 2 cpus. >>>>>> >>>>>> I?m ok with the change but just wondering why the guards are >>>>>> necessary. >>>>>> >>>>>> Bob. >>>>>> >>>>>>> On Nov 28, 2017, at 7:50 PM, mikhailo >>>>>>> wrote: >>>>>>> >>>>>>> Could you, please, review this simple fix? It limits test cases >>>>>>> in TestCPUAwareness.java >>>>>>> based on the number of processors available. >>>>>>> >>>>>>> ???? JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>>>>> ???? Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>>>>> ???? Testing: >>>>>>> ???????? 1. Locally: exercised the affected test locally on >>>>>>> Linux-x64 >>>>>>> ???????? 2. Exercised the affected test on 2 processor machine >>>>>>> ???????? 3. Exercise the affected test via automated distributed >>>>>>> test system >>>>>>> ??????????? In progress >>>>>>> >>>>>>> Misha >>>>>>> >>> >> From bob.vandette at oracle.com Thu Nov 30 21:14:58 2017 From: bob.vandette at oracle.com (Bob Vandette) Date: Thu, 30 Nov 2017 16:14:58 -0500 Subject: RFR(XS): 8191943: [TESTBUG] docker/TestCPUAwareness fails on machine with 2 CPUs In-Reply-To: <804ff854-2010-cc6a-851c-ee20c5de1c96@oracle.com> References: <3f7beb17-21ad-73ef-96ec-d0d620d6afae@oracle.com> <4a95a8ce-44a6-340c-c3bd-268bdc0b713e@oracle.com> <077EB0DB-2E88-4EE1-9467-B4839FFA7E45@oracle.com> <9e0bfd91-16a9-2082-db66-8caf4fe3f5c0@oracle.com> <59e6d7f7-5343-2513-cf74-423c35e0158d@oracle.com> <804ff854-2010-cc6a-851c-ee20c5de1c96@oracle.com> Message-ID: The Cpus_allowed_list need Linux 2.6.24 but that shouldn?t be a big problem. We?ll only skip some of the tests on older systems. Not sure docker would run well on older kernels anyway. Looks good, Bob. > On Nov 30, 2017, at 3:01 PM, mikhailo wrote: > > Bob, David, > > Thank you for review. > > Here is an updated webrev addressing your comments: > > http://cr.openjdk.java.net/~mseledtsov/8191943.02/ > > > Misha > > > On 11/29/2017 09:28 PM, David Holmes wrote: >> On 30/11/2017 10:08 AM, mikhailo wrote: >>> Here is an updated webrev: >>> >>> http://cr.openjdk.java.net/~mseledtsov/8191943.01/ >> >> As Bob noted you may still need the guard here: >> >> 55 testAPCCombo("0,1", 200*1000, 100*1000, 4*1024, 2); >> 56 testAPCCombo("0,1", 200*1000, 100*1000, 1*1024, 2); > I introduced a proper check for this. >> >> though perhaps docker ignores invalid cpu-set values? >> >> 161 Common.logNewTestCase("expectedAPC = " + expectedAPC); >> >> That should still be a System.out.println - you already set the test case prior: > I changed code to output both the original expected APC and the adjusted APC. >> >> 155 Common.logNewTestCase("test cpu shares, shares = " + shares); >> >> Having the same check duplicated three times is a bit messy, but not horrendously so. :) > Introduced a common method to conditionally adjust APC. > > Thank you, > Misha >> >> Thanks, >> David >> >>> >>> Misha >>> >>> >>> On 11/29/2017 01:32 PM, mikhailo wrote: >>>> >>>> On 11/29/2017 01:29 PM, Bob Vandette wrote: >>>>>> On Nov 29, 2017, at 4:10 PM, mikhailo wrote: >>>>>> >>>>>> Hi Bob, >>>>>> >>>>>> The failure was caused by invoking this test case: >>>>>> >>>>>> testCpuShares(4096, 4); >>>>>> The test case sets --cpu-shares to 4096, expects active processor count to be 4; ==> actual APC is 2. >>>>> Ahh, that makes sense. I thought docker was complaining due to the arguments being passed going >>>>> beyond the available cpus. >>>>> >>>>>> Command: >>>>>> >>>>>> docker run --tty=true --rm --cpu-shares=4096 jdk-internal:test-cpu /jdk/bin/java -Xlog:os+container=trace -version >>>>>> >>>>>> Detailed log is attached at: >>>>>> >>>>>> https://bugs.openjdk.java.net/secure/attachment/73820/test-case-failure-cpu-shares-4096.txt >>>>>> >>>>>> >>>>>> Once I saw this issue, I decided to put limits on other test cases based on the number of processors available on a machine, just to be on a safe side. >>>>>> >>>>>> Perhaps a better approach here is to change the expected APC inside the testCpuShares() to be no greater than max number of available processors? >>>>> Yes, the selection algorithm returns the smallest number of computed shares, quotas or active processors. Using this approach, you?d actually be validating the algorithm more precisely. >>>> OK. I will update the tests to do this kind of validation, and post a new webrev. >>>> >>>> >>>> Misha >>>>> >>>>> Bob. >>>>> >>>>> >>>>>> >>>>>> Misha >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 11/29/2017 06:02 AM, Bob Vandette wrote: >>>>>>> Misha, >>>>>>> >>>>>>> Which specific subtest failed on a 2 cpu system? >>>>>>> >>>>>>> I don?t see any subtests that should have failed. You should be able >>>>>>> to pass any quota, period and share value to docker independent of the >>>>>>> number of CPUs. testCpus and testAPCCombo don?t try to test more than 2 cpus. >>>>>>> >>>>>>> I?m ok with the change but just wondering why the guards are necessary. >>>>>>> >>>>>>> Bob. >>>>>>> >>>>>>>> On Nov 28, 2017, at 7:50 PM, mikhailo wrote: >>>>>>>> >>>>>>>> Could you, please, review this simple fix? It limits test cases in TestCPUAwareness.java >>>>>>>> based on the number of processors available. >>>>>>>> >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8191943 >>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8191943.00/ >>>>>>>> Testing: >>>>>>>> 1. Locally: exercised the affected test locally on Linux-x64 >>>>>>>> 2. Exercised the affected test on 2 processor machine >>>>>>>> 3. Exercise the affected test via automated distributed test system >>>>>>>> In progress >>>>>>>> >>>>>>>> Misha >>>>>>>> >>>> >>> > From david.holmes at oracle.com Thu Nov 30 21:24:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 1 Dec 2017 07:24:06 +1000 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> Message-ID: <3e685be3-4f8a-e328-7bd9-cbb2f72b6b65@oracle.com> Hi Dan, May I suggest simply moving all of the inline smr functions to the same position, after all the field initializations, so that it is hopefully more evident that they appear before any use. My layperson understanding - perhaps out of date in 2017 - is that to inline a function the compiler has to have already seen the definition. Thanks, David On 1/12/2017 1:49 AM, Daniel D. Daugherty wrote: > Greetings, > > I have updated the fix based on Coleen's and David H's code reviews. > > Delta webrev: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-delta/ > > Full webrev: > > http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-full/ > > Thanks, in advance, for any comments, questions or suggestions. > > Dan > > On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >> to hear from you on this thread. Thanks! >> >> I have a simple cleanup fix for Thread-SMR. The bug is: >> >> ??? JDK-8191787 move private inline functions from thread.inline.hpp >> -> thread.cpp >> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >> >> This fix is pure code motion: >> >> - moving inline functions from thread.inline.hpp -> thread.cpp >> - making a few functions in thread.hpp private instead of public >> >> Here is the webrev URL: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >> >> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >> unexpected test failures. >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan >> > From daniel.daugherty at oracle.com Thu Nov 30 21:33:34 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 30 Nov 2017 16:33:34 -0500 Subject: RFR(XS): 8191787 - move private inline functions from thread.inline.hpp -> thread.cpp In-Reply-To: <3e685be3-4f8a-e328-7bd9-cbb2f72b6b65@oracle.com> References: <695072b6-fcb2-0dd7-fe55-42a7b464dca2@oracle.com> <05cd0495-9635-2a3d-2fc9-30afc0880dc1@oracle.com> <3e685be3-4f8a-e328-7bd9-cbb2f72b6b65@oracle.com> Message-ID: Hi David, Thanks for looking at this again! On 11/30/17 4:24 PM, David Holmes wrote: > Hi Dan, > > May I suggest simply moving all of the inline smr functions to the > same position, after all the field initializations, so that it is > hopefully more evident that they appear before any use. I'm in my pre-integration Mach5 tier[12] test cycle so I wasn't planning on any more tweaks for this bug. Can I pick up this change in my next code motion fix (8191789)? This fix (8191787) will likely make Jesper's snapshot on 12.01. My next fix will not... > My layperson understanding - perhaps out of date in 2017 - is that to > inline a function the compiler has to have already seen the definition. I was hoping for someone else to jump in to confirm or deny, but that hasn't happened. That's why I went ahead and moved the one inline definition before its first use... All other inline defs are before first use (but it is not obvious). Dan > > Thanks, > David > > On 1/12/2017 1:49 AM, Daniel D. Daugherty wrote: >> Greetings, >> >> I have updated the fix based on Coleen's and David H's code reviews. >> >> Delta webrev: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-delta/ >> >> Full webrev: >> >> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-1-full/ >> >> Thanks, in advance, for any comments, questions or suggestions. >> >> Dan >> >> On 11/29/17 4:16 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> Coleen, this is one of your Thread-SMR follow-up suggestions so I need >>> to hear from you on this thread. Thanks! >>> >>> I have a simple cleanup fix for Thread-SMR. The bug is: >>> >>> ??? JDK-8191787 move private inline functions from thread.inline.hpp >>> -> thread.cpp >>> ??? https://bugs.openjdk.java.net/browse/JDK-8191787 >>> >>> This fix is pure code motion: >>> >>> - moving inline functions from thread.inline.hpp -> thread.cpp >>> - making a few functions in thread.hpp private instead of public >>> >>> Here is the webrev URL: >>> >>> http://cr.openjdk.java.net/~dcubed/8191787-webrev/jdk10-0 >>> >>> This fix was (over) tested with a Mach5 tier[1-5] run. There were no >>> unexpected test failures. >>> >>> Thanks, in advance, for any comments, questions or suggestions. >>> >>> Dan >>> >> From manc at google.com Thu Nov 30 22:03:59 2017 From: manc at google.com (Man Cao) Date: Thu, 30 Nov 2017 14:03:59 -0800 Subject: Make reserved_size for compressed class space and metaspace respect the ergo-initialized CompressedClassSpaceSize flag value In-Reply-To: References: Message-ID: I realized that the email attachment is probably dropped by the mailing list, so below is the inlined patch. --- old/src/hotspot/share/memory/metaspace.cpp 2017-11-29 14:56:59.017118444 -0800 +++ new/src/hotspot/share/memory/metaspace.cpp 2017-11-29 14:56:58.657121375 -0800 @@ -3321,9 +3321,6 @@ MinMetaspaceExpansion = align_down_bounded(MinMetaspaceExpansion, _commit_alignment); MaxMetaspaceExpansion = align_down_bounded(MaxMetaspaceExpansion, _commit_alignment); - CompressedClassSpaceSize = align_down_bounded(CompressedClassSpaceSize, _reserve_alignment); - set_compressed_class_space_size(CompressedClassSpaceSize); - // Initial virtual space size will be calculated at global_initialize() size_t min_metaspace_sz = VIRTUALSPACEMULTIPLIER * InitialBootClassLoaderMetaspaceSize; @@ -3341,6 +3338,8 @@ min_metaspace_sz); } + CompressedClassSpaceSize = align_down_bounded(CompressedClassSpaceSize, _reserve_alignment); + set_compressed_class_space_size(CompressedClassSpaceSize); } void Metaspace::global_initialize() { Best, Man On Wed, Nov 29, 2017 at 3:21 PM, Man Cao wrote: > Hello, > > This patch is a follow-up fix for https://bugs.openjdk.java. > net/browse/JDK-8087291 > > This patch moves the call to set_compressed_class_space_size() after the > flag value for CompressedClassSpaceSize is ergo-initialized, fixing the > issue that the reserved size for compressed class space and metaspace is > excessively large when MaxMetaspaceSize is set to a small value. More > discussion about it is available here: > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/ > 2017-November/025200.html > > This code patch is attached. Could anyone review and/or sponsor this patch? > > Best, > Man > From volker.simonis at gmail.com Thu Nov 30 22:56:56 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Thu, 30 Nov 2017 22:56:56 +0000 Subject: Several questions regarding AppCDS In-Reply-To: <08A854C2-BC11-43E3-9FB9-98E6050D098A@oracle.com> References: <08A854C2-BC11-43E3-9FB9-98E6050D098A@oracle.com> Message-ID: Jiangli Zhou schrieb am Do. 30. Nov. 2017 um 20:27: > Hi Volker, > > Just some high level information for your questions. We are continually > investigating the usage/requirement of the CDS/AppCDS support for custom > class loaders and the current implementation may change when the > investigation is maturing. That?s part of reason why there is no extended > public documentation about the custom class loader support and external > tool for generating the new class list available at the time. Ioi probably > will fill in more details on this topic. > Hi Jiangli, thanks a lot for your answer. Without the possibility to create the extended class list entries for classes loaded by custom class loaders automatically it will be probably quite hard to use this feature productively. There's one thing I forgot to ask. Is there a way (e.g. a tool) to dump the contents of an archive in readable form. E.g. something like SA agent to create class files from an archive? That could be quite helpful for debugging. Thank's a lot and best regards, Volker > Thanks, > > Jiangli > > > On Nov 30, 2017, at 6:46 AM, Volker Simonis > wrote: > > > > Hi Jiangli, Ioi, > > > > after AppCDS has been pushed to the open repos, I started looking at > > it. However, I found very little in-deep documentation of the feature. > > All I could find so far is basically the Oracle docs, blogs and videos > > I've listed below (in the hope that they may be useful for others as > > well). > > > > However this still doesn't answer the following questions: > > > > 1. Is there a specification and explanation of the extended class list > > syntax which is used for AppCDS with custom class loaders? > > > > I only found the documentation in systemDictionaryShared.hpp. But that > > for example mentions the "loader" keyword (as do some of the JTreg > > tests) without further explanation. > > > > 2. Is there a way or tool to create this extended class list > automatically? > > > > As far as I can see, all the Jtreg tests which use the extended > > syntax, generate the class list manually. I did some simple tests > > where I load some classes into my own URL class loader but they don't > > seem to appear in the class list generated by -XX:DumpLoadedClassList. > > > > Any further references are highly welcome. > > > > Thank you and best regards, > > Volker > > > > PS: I really liked your JavaOne 2016 presentation :) > > > > Class Data Sharing > > ==================== > > > https://docs.oracle.com/javase/9/vm/class-data-sharing.htm#JSJVM-GUID-7EAA3411-8CF0-4D19-BD05-DF5E1780AA91 > > > > Application Class Data Sharing > > ============================== > > > https://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > > > JEP 310: Application Class-Data Sharing > > ======================================= > > http://openjdk.java.net/jeps/310 > > > > Matthew Gilliard's blog > > ----------------------- > > Fast JVM startup with JDK 9 > > https://mjg123.github.io/2017/10/02/JVM-startup.html > > > > Quicker Clojure startup with AppCDS and AOT > > https://mjg123.github.io/2017/10/04/AppCDS-and-Clojure.html > > > > Leveraging AppCDS to Optimize Application Startup and Memory Footprint > > in the Cloud > > > ----------------------------------------------------------------------------------- > > https://www.youtube.com/watch?v=dRw77QDSL-A > >