From hohensee at amazon.com Sun Sep 1 00:05:24 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Sun, 1 Sep 2019 00:05:24 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> Message-ID: <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> Thanks, Mandy. I?ve finalized the CSR. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.04/. In management.cpp, I now have if (THREAD->is_Java_thread()) { return ((JavaThread*)THREAD)->cooked_allocated_bytes(); } In ThreadImpl.java, using requireNonNull would produce a different and less informative message, so I?d like to leave it as is. I changed throwIfNullThreadIds to ensureNonNullThreadIds, and throwIfThreadAllocatedMemoryNotSupported to ensureThreadAllocatedMemorySupported. I dropped the ?java.lang.? prefix from all uses of UnsupportedOperationException in both c.s.m.ThreadMXBean.java and j.l.m.ThreadMXBean.java, and did the same with SecurityException. ?@since 14? added to c.s.m.ThreadMXBean.java and the CSR. Do I need another reviewer? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 4:26 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread CSR reviewed. management.cpp 2083 java_thread = (JavaThread*)THREAD; 2084 if (java_thread->is_Java_thread()) { 2085 return java_thread->cooked_allocated_bytes(); 2086 } The cast should be done after is_Java_thread() test. ThreadImpl.java 162 private void throwIfNullThreadIds(long[] ids) { Even better: simply use Objects::requiresNonNull and this method can be removed. This suggests positive naming alternative to throwIfThreadAllocatedMemoryNotSupported - "ensureThreadAllocatedMemorySupported" (sorry I should have suggested that) ThreadMXBean.java 130 * @throws java.lang.UnsupportedOperationException if the Java virtual Nit: "java.lang." can be dropped. @since 14 is missing. Mandy On 8/30/19 3:33 PM, Hohensee, Paul wrote: Thanks for your review, Mandy. Revised webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.02/. I updated the CSR with your suggested javadoc for getCurrentThreadAllocatedBytes. It now matches that for getCurrentThreadUserTime and getCurrentThreadCputime. I also fixed the ?convenient? -> ?convenience? typos in j.l.m.ThreadMXBean.java. I meant GetOneThreads to be the possessive, but don?t feel strongly either way so I?m fine with GetOneThread. I updated ThreadImpl.java as you suggested, though in getThreadAllocatedBytes(long[] ids) I had to add a redundant-in-the-not-length-1-case check for a null ids reference. Would someone take a look at the Hotspot side and the test please? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 10:22 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread OK. That's better. Some review comments: The javadoc of getCurrentThreadAllocatedBytes() can simply say: "Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the current thread. This is a convenient method for local management use and is equivalent to calling getThreadAllocatedBytes(Thread.currentThread().getId()). src/hotspot/share/include/jmm.h GetOneThreadsAllocatedMemory: s/OneThreads/OneThread/ sun/management/ThreadImpl.java 43 private static final String THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED = 44 "Thread allocated memory measurement is not supported."; if (!isThreadAllocatedMemorySupported()) { throw new UnsupportedOperationException(THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED); } Perhaps the above can be refactored as throwIfAllocatedMemoryUnsupported() method. 391 if (ids.length == 1) { 392 sizes[0] = -1; : 398 if (ids.length == 1) { 399 long id = ids[0]; 400 sizes[0] = getThreadAllocatedMemory0( 401 Thread.currentThread().getId() == id ? 0 : id); 402 } else { It seems cleaner to handle the 1-element array case at the beginning of this method: if (ids.length == 1) { long size = getThreadAllocatedBytes(ids[0]); return new long[] { size }; } I didn't review the hotspot implementation and the test. Mandy On 8/29/19 10:01 AM, Hohensee, Paul wrote: My bad, Mandy. The webrev puts getCurrentThreadAllocatedBytes in com.sun.management.ThreadMXBean along with the current two getThreadAllocatedBytes methods for the reasons you list. I?ve updated the CSR to specify com.sun.management and added a rationale. AllocatedBytes is currently enabled by Hotspot by default because the overhead of recording TLAB occupancy is negligible. There?s no new GC code, nor will there be, so imo we don?t have to involve the GC folks. I.e., the new JMM method GetOneThreadsAllocatedBytes uses the existing cooked_allocated_bytes JavaThread method, and getCurrentThreadAllocatedBytes is the same as getThreadAllocatedBytes: it just bypasses the thread lookup code. I hadn?t tracked down what happens when getCurrentThreadUserTime and getCurrentThreadCpuTime are called before, but if I?m not mistaken, it the code in jcmd() in attachListener.cpp will call GetThreadCpuTimeWithKind in management.cpp, and it will ultimately use Thread::current() as the subject of the call, see os::current_thread_cpu_time in os_linux.cpp. That means that the CurrentThread methods should work remotely the same way they do locally. GetOneThreadsAllocatedBytes in management.cpp uses THREAD as its subject when called on behalf of getCurrentThreadAllocatedBytes, so it will also uses the current remote Java thread. Even if these methods only worked locally, there are many setups where apps are self-monitoring that could use the performance improvement. Thanks, Paul From: Mandy Chung Date: Wednesday, August 28, 2019 at 3:59 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, The CSR proposes this method in java.lang.management.ThreadMXBean as a Java SE feature. Has this been discussed with the GC team to commit measuring current thread's allocated bytes as Java SE feature? Can this be supported by all JVM implementation? What is the overhead if this is enabled by default? Does it need to be disabled? This metric is from TLAB that might be okay. This needs advice/discussion with GC experts. I see that CSR mentions it can be disabled and link to isThreadAllocatedMemoryEnabled() and setThreadAllocatedMemoryEnabled() methods but these methods are defined in com.sun.management.ThreadMXBean. As Alan points out, current thread makes sense only in local VM management. When this is monitored from a JMX client (e.g. jconsole to connect to a running JVM, "currentThreadAllowcatedBytes" attribute is the current thread in jconsole process which invoking Thread::currentThread? Mandy On 8/28/19 12:22 PM, Hohensee, Paul wrote: Please review a performance improvement for ThreadMXBean.getThreadAllocatedBytes and the addition of getCurrentThreadAllocatedBytes. JBS issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Webrev: http://cr.openjdk.java.net/~phh/8207266/webrev.00/ CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 Previous email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-July/024441.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-August/024763.html The CSR is for adding ThreadMXBean.getCurrentThreadAllocatedBytes. I?d be great for someone to review it. I took Mandy?s advice and put the fast paths in the library code. I added a new JMM method GetOneThreadsAllocatedBytes that works the same as GetThreadCpuTime: it uses a thread_id value of zero to distinguish the current thread. On my Mac laptop, the result runs 47x faster for the current thread than the old implementation. The 3 tests in test/jdk/com/sun/management/ThreadMXBean all pass. I added code to ThreadAllocatedMemory.java to test getCurrentThreadAllocatedBytes as well as variations on getThreadAllocatedBytes(id). A submit repo job is in progress. Thanks, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Tue Sep 3 09:14:40 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 3 Sep 2019 19:14:40 +1000 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: Message-ID: Hi Matthias, Re-directing to serviceability-dev. David On 3/09/2019 5:42 pm, Baesken, Matthias wrote: > Hello, please review the following small fix . > > In jdk.hotspot.agent native code (linux / macosx) we miss to check the result of malloc/calloc a few times . > This should be adjusted. > Additionally I added initialization to the symtab array in symtab.c (by calling memset to make sure we have a defined state ) . > > > > One question (was not really sure about this one so I did not change it so far) : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot.agent/macosx/native/libsaproc/symtab.c.frames.html > > 359 void destroy_symtab(symtab_t* symtab) { > 360 if (!symtab) return; > 361 free(symtab->strs); > 362 free(symtab->symbols); > 363 free(symtab); > 364 } > > > > Here we miss to close symtab->hash_table (opened by dbopen) , is it needed (haven't used dbopen much - maybe someone can comment on this)? > > > bug/webrev : > > https://bugs.openjdk.java.net/browse/JDK-8230466 > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ > > > Thanks and best regards, Matthias > From hohensee at amazon.com Tue Sep 3 19:38:08 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 3 Sep 2019 19:38:08 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> Message-ID: Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I removed ensureNonNullThreadIds() in favor of Objects.requireNonNull(ids). Thanks, Mandy, for your through reviews. May I get another reviewer to weigh in? Paul ?On 8/31/19, 5:06 PM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: Thanks, Mandy. I?ve finalized the CSR. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.04/. In management.cpp, I now have if (THREAD->is_Java_thread()) { return ((JavaThread*)THREAD)->cooked_allocated_bytes(); } In ThreadImpl.java, using requireNonNull would produce a different and less informative message, so I?d like to leave it as is. I changed throwIfNullThreadIds to ensureNonNullThreadIds, and throwIfThreadAllocatedMemoryNotSupported to ensureThreadAllocatedMemorySupported. I dropped the ?java.lang.? prefix from all uses of UnsupportedOperationException in both c.s.m.ThreadMXBean.java and j.l.m.ThreadMXBean.java, and did the same with SecurityException. ?@since 14? added to c.s.m.ThreadMXBean.java and the CSR. Do I need another reviewer? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 4:26 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread CSR reviewed. management.cpp 2083 java_thread = (JavaThread*)THREAD; 2084 if (java_thread->is_Java_thread()) { 2085 return java_thread->cooked_allocated_bytes(); 2086 } The cast should be done after is_Java_thread() test. ThreadImpl.java 162 private void throwIfNullThreadIds(long[] ids) { Even better: simply use Objects::requiresNonNull and this method can be removed. This suggests positive naming alternative to throwIfThreadAllocatedMemoryNotSupported - "ensureThreadAllocatedMemorySupported" (sorry I should have suggested that) ThreadMXBean.java 130 * @throws java.lang.UnsupportedOperationException if the Java virtual Nit: "java.lang." can be dropped. @since 14 is missing. Mandy On 8/30/19 3:33 PM, Hohensee, Paul wrote: Thanks for your review, Mandy. Revised webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.02/. I updated the CSR with your suggested javadoc for getCurrentThreadAllocatedBytes. It now matches that for getCurrentThreadUserTime and getCurrentThreadCputime. I also fixed the ?convenient? -> ?convenience? typos in j.l.m.ThreadMXBean.java. I meant GetOneThreads to be the possessive, but don?t feel strongly either way so I?m fine with GetOneThread. I updated ThreadImpl.java as you suggested, though in getThreadAllocatedBytes(long[] ids) I had to add a redundant-in-the-not-length-1-case check for a null ids reference. Would someone take a look at the Hotspot side and the test please? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 10:22 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread OK. That's better. Some review comments: The javadoc of getCurrentThreadAllocatedBytes() can simply say: "Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the current thread. This is a convenient method for local management use and is equivalent to calling getThreadAllocatedBytes(Thread.currentThread().getId()). src/hotspot/share/include/jmm.h GetOneThreadsAllocatedMemory: s/OneThreads/OneThread/ sun/management/ThreadImpl.java 43 private static final String THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED = 44 "Thread allocated memory measurement is not supported."; if (!isThreadAllocatedMemorySupported()) { throw new UnsupportedOperationException(THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED); } Perhaps the above can be refactored as throwIfAllocatedMemoryUnsupported() method. 391 if (ids.length == 1) { 392 sizes[0] = -1; : 398 if (ids.length == 1) { 399 long id = ids[0]; 400 sizes[0] = getThreadAllocatedMemory0( 401 Thread.currentThread().getId() == id ? 0 : id); 402 } else { It seems cleaner to handle the 1-element array case at the beginning of this method: if (ids.length == 1) { long size = getThreadAllocatedBytes(ids[0]); return new long[] { size }; } I didn't review the hotspot implementation and the test. Mandy On 8/29/19 10:01 AM, Hohensee, Paul wrote: My bad, Mandy. The webrev puts getCurrentThreadAllocatedBytes in com.sun.management.ThreadMXBean along with the current two getThreadAllocatedBytes methods for the reasons you list. I?ve updated the CSR to specify com.sun.management and added a rationale. AllocatedBytes is currently enabled by Hotspot by default because the overhead of recording TLAB occupancy is negligible. There?s no new GC code, nor will there be, so imo we don?t have to involve the GC folks. I.e., the new JMM method GetOneThreadsAllocatedBytes uses the existing cooked_allocated_bytes JavaThread method, and getCurrentThreadAllocatedBytes is the same as getThreadAllocatedBytes: it just bypasses the thread lookup code. I hadn?t tracked down what happens when getCurrentThreadUserTime and getCurrentThreadCpuTime are called before, but if I?m not mistaken, it the code in jcmd() in attachListener.cpp will call GetThreadCpuTimeWithKind in management.cpp, and it will ultimately use Thread::current() as the subject of the call, see os::current_thread_cpu_time in os_linux.cpp. That means that the CurrentThread methods should work remotely the same way they do locally. GetOneThreadsAllocatedBytes in management.cpp uses THREAD as its subject when called on behalf of getCurrentThreadAllocatedBytes, so it will also uses the current remote Java thread. Even if these methods only worked locally, there are many setups where apps are self-monitoring that could use the performance improvement. Thanks, Paul From: Mandy Chung Date: Wednesday, August 28, 2019 at 3:59 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, The CSR proposes this method in java.lang.management.ThreadMXBean as a Java SE feature. Has this been discussed with the GC team to commit measuring current thread's allocated bytes as Java SE feature? Can this be supported by all JVM implementation? What is the overhead if this is enabled by default? Does it need to be disabled? This metric is from TLAB that might be okay. This needs advice/discussion with GC experts. I see that CSR mentions it can be disabled and link to isThreadAllocatedMemoryEnabled() and setThreadAllocatedMemoryEnabled() methods but these methods are defined in com.sun.management.ThreadMXBean. As Alan points out, current thread makes sense only in local VM management. When this is monitored from a JMX client (e.g. jconsole to connect to a running JVM, "currentThreadAllowcatedBytes" attribute is the current thread in jconsole process which invoking Thread::currentThread? Mandy On 8/28/19 12:22 PM, Hohensee, Paul wrote: Please review a performance improvement for ThreadMXBean.getThreadAllocatedBytes and the addition of getCurrentThreadAllocatedBytes. JBS issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Webrev: http://cr.openjdk.java.net/~phh/8207266/webrev.00/ CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 Previous email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-July/024441.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-August/024763.html The CSR is for adding ThreadMXBean.getCurrentThreadAllocatedBytes. I?d be great for someone to review it. I took Mandy?s advice and put the fast paths in the library code. I added a new JMM method GetOneThreadsAllocatedBytes that works the same as GetThreadCpuTime: it uses a thread_id value of zero to distinguish the current thread. On my Mac laptop, the result runs 47x faster for the current thread than the old implementation. The 3 tests in test/jdk/com/sun/management/ThreadMXBean all pass. I added code to ThreadAllocatedMemory.java to test getCurrentThreadAllocatedBytes as well as variations on getThreadAllocatedBytes(id). A submit repo job is in progress. Thanks, Paul From chris.plummer at oracle.com Wed Sep 4 00:12:02 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 3 Sep 2019 17:12:02 -0700 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: Message-ID: Sorry, I don't have an answer to your symtab->hash_table question. But the rest of the changes look good to me. thanks, Chris On 9/3/19 2:14 AM, David Holmes wrote: > Hi Matthias, > > Re-directing to serviceability-dev. > > David > > On 3/09/2019 5:42 pm, Baesken, Matthias wrote: >> Hello, please review the following small fix . >> >> In?? jdk.hotspot.agent? native code (linux / macosx)?? we miss to >> check the? result of malloc/calloc a few times . >> This should be? adjusted. >> Additionally? I added initialization? to the symtab? array? in >> symtab.c?? (by calling memset? to make sure we have a defined state )? . >> >> >> >> One question (was not really sure about this one so I did not change >> it so far) : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot.agent/macosx/native/libsaproc/symtab.c.frames.html >> >> >> 359 void destroy_symtab(symtab_t* symtab) { >> 360?? if (!symtab) return; >> 361?? free(symtab->strs); >> 362?? free(symtab->symbols); >> 363?? free(symtab); >> 364 } >> >> >> >> Here we miss to close?? symtab->hash_table?? (opened by dbopen) ,? is >> it needed? (haven't? used dbopen much - maybe someone can comment on >> this)? >> >> >> bug/webrev : >> >> https://bugs.openjdk.java.net/browse/JDK-8230466 >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ >> >> >> Thanks and best regards, Matthias >> From yasuenag at gmail.com Wed Sep 4 05:59:24 2019 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Wed, 4 Sep 2019 14:59:24 +0900 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: Message-ID: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> Hi Matthias, src/jdk.hotspot.agent/linux/native/libsaproc/symtab.c: ``` 405 // guarantee(symtab == NULL, "multiple symtab"); 406 symtab = (struct symtab*)calloc(1, sizeof(struct symtab)); 407 if (symtab == NULL) { 408 goto quit; 409 } 410 memset(symtab, 0, sizeof(struct symtab)); ``` Why do you call memset() to clear symtab in L410? symtab is allocated via calloc() in L406, so symtab would already cleared. Thanks, Yasumasa (ysuenaga) On 2019/09/03 18:14, David Holmes wrote: > Hi Matthias, > > Re-directing to serviceability-dev. > > David > > On 3/09/2019 5:42 pm, Baesken, Matthias wrote: >> Hello, please review the following small fix . >> >> In jdk.hotspot.agent native code (linux / macosx) we miss to check the result of malloc/calloc a few times . >> This should be adjusted. >> Additionally I added initialization to the symtab array in symtab.c (by calling memset to make sure we have a defined state ) . >> >> >> >> One question (was not really sure about this one so I did not change it so far) : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot.agent/macosx/native/libsaproc/symtab.c.frames.html >> >> 359 void destroy_symtab(symtab_t* symtab) { >> 360 if (!symtab) return; >> 361 free(symtab->strs); >> 362 free(symtab->symbols); >> 363 free(symtab); >> 364 } >> >> >> >> Here we miss to close symtab->hash_table (opened by dbopen) , is it needed (haven't used dbopen much - maybe someone can comment on this)? >> >> >> bug/webrev : >> >> https://bugs.openjdk.java.net/browse/JDK-8230466 >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ >> >> >> Thanks and best regards, Matthias >> From matthias.baesken at sap.com Wed Sep 4 07:28:26 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 4 Sep 2019 07:28:26 +0000 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> References: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> Message-ID: Hello Yasumasa and Chris, thanks for your input . Here is a new webrev , without the unneeded memset-calls after calloc . http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/ Hope everyone is happy with this now ?? ! Best regards, Matthias > Hi Matthias, > > src/jdk.hotspot.agent/linux/native/libsaproc/symtab.c: > ``` > 405 // guarantee(symtab == NULL, "multiple symtab"); > 406 symtab = (struct symtab*)calloc(1, sizeof(struct symtab)); > 407 if (symtab == NULL) { > 408 goto quit; > 409 } > 410 memset(symtab, 0, sizeof(struct symtab)); > ``` > > Why do you call memset() to clear symtab in L410? > symtab is allocated via calloc() in L406, so symtab would already cleared. > > > Thanks, > > Yasumasa (ysuenaga) > > > On 2019/09/03 18:14, David Holmes wrote: > > Hi Matthias, > > > > Re-directing to serviceability-dev. > > > > David > > > > On 3/09/2019 5:42 pm, Baesken, Matthias wrote: > >> Hello, please review the following small fix . > >> > >> In jdk.hotspot.agent native code (linux / macosx) we miss to check the > result of malloc/calloc a few times . > >> This should be adjusted. > >> Additionally I added initialization to the symtab array in symtab.c (by > calling memset to make sure we have a defined state ) . > >> > >> > >> > >> One question (was not really sure about this one so I did not change it so > far) : > >> > >> > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot. > agent/macosx/native/libsaproc/symtab.c.frames.html > >> > >> 359 void destroy_symtab(symtab_t* symtab) { > >> 360 if (!symtab) return; > >> 361 free(symtab->strs); > >> 362 free(symtab->symbols); > >> 363 free(symtab); > >> 364 } > >> > >> > >> > >> Here we miss to close symtab->hash_table (opened by dbopen) , is it > needed (haven't used dbopen much - maybe someone can comment on > this)? > >> > >> > >> bug/webrev : > >> > >> https://bugs.openjdk.java.net/browse/JDK-8230466 > >> > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ > >> > >> > >> Thanks and best regards, Matthias > >> From yasuenag at gmail.com Wed Sep 4 07:37:48 2019 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Wed, 4 Sep 2019 16:37:48 +0900 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> Message-ID: Looks good! Yasumasa (ysuenaga) On 2019/09/04 16:28, Baesken, Matthias wrote: > Hello Yasumasa and Chris, thanks for your input . > Here is a new webrev , without the unneeded memset-calls after calloc . > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/ > > Hope everyone is happy with this now ?? ! > > Best regards, Matthias > > >> Hi Matthias, >> >> src/jdk.hotspot.agent/linux/native/libsaproc/symtab.c: >> ``` >> 405 // guarantee(symtab == NULL, "multiple symtab"); >> 406 symtab = (struct symtab*)calloc(1, sizeof(struct symtab)); >> 407 if (symtab == NULL) { >> 408 goto quit; >> 409 } >> 410 memset(symtab, 0, sizeof(struct symtab)); >> ``` >> >> Why do you call memset() to clear symtab in L410? >> symtab is allocated via calloc() in L406, so symtab would already cleared. >> >> >> Thanks, >> >> Yasumasa (ysuenaga) >> >> >> On 2019/09/03 18:14, David Holmes wrote: >>> Hi Matthias, >>> >>> Re-directing to serviceability-dev. >>> >>> David >>> >>> On 3/09/2019 5:42 pm, Baesken, Matthias wrote: >>>> Hello, please review the following small fix . >>>> >>>> In jdk.hotspot.agent native code (linux / macosx) we miss to check the >> result of malloc/calloc a few times . >>>> This should be adjusted. >>>> Additionally I added initialization to the symtab array in symtab.c (by >> calling memset to make sure we have a defined state ) . >>>> >>>> >>>> >>>> One question (was not really sure about this one so I did not change it so >> far) : >>>> >>>> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot. >> agent/macosx/native/libsaproc/symtab.c.frames.html >>>> >>>> 359 void destroy_symtab(symtab_t* symtab) { >>>> 360 if (!symtab) return; >>>> 361 free(symtab->strs); >>>> 362 free(symtab->symbols); >>>> 363 free(symtab); >>>> 364 } >>>> >>>> >>>> >>>> Here we miss to close symtab->hash_table (opened by dbopen) , is it >> needed (haven't used dbopen much - maybe someone can comment on >> this)? >>>> >>>> >>>> bug/webrev : >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8230466 >>>> >>>> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ >>>> >>>> >>>> Thanks and best regards, Matthias >>>> From alexey.menkov at oracle.com Wed Sep 4 19:19:29 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Wed, 4 Sep 2019 12:19:29 -0700 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException Message-ID: Hi all, Please review the fix for BadHandshakeTest test. The problem is the test connects to the server twice and if debuggee hasn't yet handled disconnection, the next connect gets "connection refused" error. Instead of adding delay before 2nd connect (we never know "good" value for the delay and big delay can cause "accept timeout"), the test re-tries connect in case of ConnectException. Also improved/simplified the test slightly - debuggee is now run with auto port assignment (used lib.jdb.Debuggee test class which implements required functionality). jira: https://bugs.openjdk.java.net/browse/JDK-8192057 webrev: http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ --alex From serguei.spitsyn at oracle.com Wed Sep 4 19:45:27 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 4 Sep 2019 12:45:27 -0700 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> Message-ID: <796065c6-a4ee-92b2-0881-451537f2a4b9@oracle.com> Hi Matthias, It looks good in general but I have some minor comments below. http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot.agent/linux/native/libsaproc/symtab.c.frames.html ?279 build_id_to_debug_filename (size_t size, unsigned char *data) ?280 { ?. . . ?283?? filename = malloc(strlen (debug_file_directory) + (sizeof "/.build-id/" - 1) + 1 ?284???????????????????? + 2 * size + (sizeof ".debug" - 1) + 1); ?285?? if (filename == NULL) { ?286???? return NULL; ?287?? } ?. . . ?312?? char *filename ?313???? = (build_id_to_debug_filename (note->n_descsz, bytes)); ?314?? if (filename == NULL) { ?315???? return NULL; ?316?? } There is no need to check filename for NULL at the line 314 as the function build_id_to_debug_filename with new check at the line 285 never returns NULL. http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot.agent/macosx/native/libsaproc/MacosxDebuggerLocal.m.frames.html ?354?? array = (*env)->NewByteArray(env, numBytes); ?. . . ?376?? if (pages == NULL) { ?377???? return NULL; ?378?? } ?379?? mapped = calloc(pageCount, sizeof(int)); ?380?? if (mapped == NULL) { ?381???? free(pages); ?382???? return NULL; ?383?? } Just a question: ? We do not release the array allocated at line 354 because this local reference ? will be auto-released when returning to Java. Is this correct? http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot.agent/macosx/native/libsaproc/symtab.c.frames.html ? 69???? if (is_debug()) { ? 70?????? DBT rkey, rvalue;? 71?????? char* tmp = (char *)malloc(strlen(symtab->symbols[i].name) + 1); ? 72?????? if (tmp != NULL) { ? 73???????? strcpy(tmp, symtab->symbols[i].name); ? 74???????? rkey.data = tmp; ? 75???????? rkey.size = strlen(tmp) + 1; ? 76 (*symtab->hash_table->get)(symtab->hash_table, &rkey, &rvalue, 0); ? 77???????? // we may get a copy back so compare contents ? 78???????? symtab_symbol *res = (symtab_symbol *)rvalue.data; ? 79???????? if (strcmp(res->name, symtab->symbols[i].name) || ? 80?????????? res->offset != symtab->symbols[i].offset || ? 81?????????? res->size != symtab->symbols[i].size) { ? 82???????????? print_debug("error to get hash_table value!\n"); ? 83???????? } ? 84???????? free(tmp); ? 85?????? } If malloc returns NULL then this debugging part will be we silently skipped. In other such cases there is an attempt to print a debug message. For instance: ?140?? symtab = (symtab_t *)malloc(sizeof(symtab_t)); ?141?? if (symtab == NULL) { ?142???? print_debug("out of memory: allocating symtab\n"); ?143???? return NULL; ?144?? } I understand that print_debug can fail with out of memory as well. But it depends on its implementation. Thanks, Serguei On 9/4/19 00:28, Baesken, Matthias wrote: > Hello Yasumasa and Chris, thanks for your input . > Here is a new webrev , without the unneeded memset-calls after calloc . > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/ > > Hope everyone is happy with this now ?? ! > > Best regards, Matthias > > >> Hi Matthias, >> >> src/jdk.hotspot.agent/linux/native/libsaproc/symtab.c: >> ``` >> 405 // guarantee(symtab == NULL, "multiple symtab"); >> 406 symtab = (struct symtab*)calloc(1, sizeof(struct symtab)); >> 407 if (symtab == NULL) { >> 408 goto quit; >> 409 } >> 410 memset(symtab, 0, sizeof(struct symtab)); >> ``` >> >> Why do you call memset() to clear symtab in L410? >> symtab is allocated via calloc() in L406, so symtab would already cleared. >> >> >> Thanks, >> >> Yasumasa (ysuenaga) >> >> >> On 2019/09/03 18:14, David Holmes wrote: >>> Hi Matthias, >>> >>> Re-directing to serviceability-dev. >>> >>> David >>> >>> On 3/09/2019 5:42 pm, Baesken, Matthias wrote: >>>> Hello, please review the following small fix . >>>> >>>> In jdk.hotspot.agent native code (linux / macosx) we miss to check the >> result of malloc/calloc a few times . >>>> This should be adjusted. >>>> Additionally I added initialization to the symtab array in symtab.c (by >> calling memset to make sure we have a defined state ) . >>>> >>>> >>>> One question (was not really sure about this one so I did not change it so >> far) : >>>> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/src/jdk.hotspot. >> agent/macosx/native/libsaproc/symtab.c.frames.html >>>> 359 void destroy_symtab(symtab_t* symtab) { >>>> 360 if (!symtab) return; >>>> 361 free(symtab->strs); >>>> 362 free(symtab->symbols); >>>> 363 free(symtab); >>>> 364 } >>>> >>>> >>>> >>>> Here we miss to close symtab->hash_table (opened by dbopen) , is it >> needed (haven't used dbopen much - maybe someone can comment on >> this)? >>>> >>>> bug/webrev : >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8230466 >>>> >>>> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.0/ >>>> >>>> >>>> Thanks and best regards, Matthias >>>> From serguei.spitsyn at oracle.com Wed Sep 4 20:11:07 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 4 Sep 2019 13:11:07 -0700 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: Message-ID: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Hi Alex, The fix looks good. Good simplification! Thanks, Serguei On 9/4/19 12:19, Alex Menkov wrote: > Hi all, > > Please review the fix for BadHandshakeTest test. > The problem is the test connects to the server twice and if debuggee > hasn't yet handled disconnection, the next connect gets "connection > refused" error. > Instead of adding delay before 2nd connect (we never know "good" value > for the delay and big delay can cause "accept timeout"), the test > re-tries connect in case of ConnectException. > Also improved/simplified the test slightly - debuggee is now run with > auto port assignment (used lib.jdb.Debuggee test class which > implements required functionality). > > jira: > ? https://bugs.openjdk.java.net/browse/JDK-8192057 > webrev: > http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ > > --alex From david.holmes at oracle.com Thu Sep 5 06:43:16 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Sep 2019 16:43:16 +1000 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... Message-ID: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ See bug report for gory details. Basically on Windows thread-cpu-time may only get updated at the resolution of the timer interrupt, which may be as long as 16ms. The test checks that the elapsed cpu time is < 10ms and so would fail if the first and second calls to get the time straddled a timer interrupt (and normally it passes because both calls return zero). Fix is to bump the allowed time to 16ms on Windows. I also corrected a comment as to why part of the test is already disabled on Windows, and enabled verbose logging so that if this test fails again it will be much easier to see why. Testing: itself, on Windows numerous times Thanks, David From matthias.baesken at sap.com Thu Sep 5 08:19:05 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Thu, 5 Sep 2019 08:19:05 +0000 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: <796065c6-a4ee-92b2-0881-451537f2a4b9@oracle.com> References: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> <796065c6-a4ee-92b2-0881-451537f2a4b9@oracle.com> Message-ID: Hello Serguei, thanks for the comments . > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. > agent/linux/native/libsaproc/symtab.c.frames.html > > ?279 build_id_to_debug_filename (size_t size, unsigned char *data) > ?280 { > ?. . . > ?283?? filename = malloc(strlen (debug_file_directory) + (sizeof > "/.build-id/" - 1) + 1 > ?284???????????????????? + 2 * size + (sizeof ".debug" - 1) + 1); > ?285?? if (filename == NULL) { > ?286???? return NULL; > ?287?? } > ?. . . > ?312?? char *filename > ?313???? = (build_id_to_debug_filename (note->n_descsz, bytes)); > ?314?? if (filename == NULL) { > ?315???? return NULL; > ?316?? } > > There is no need to check filename for NULL at the line 314 as the function > build_id_to_debug_filename with new check at the line 285 never returns > NULL. > At line 286 of build_id_to_debug_filename we now return NULL (in case malloc cannot alloc memory). So we should check this also at line 314 , or do I miss something ? > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. > agent/macosx/native/libsaproc/MacosxDebuggerLocal.m.frames.html > > ?354?? array = (*env)->NewByteArray(env, numBytes); > ?. . . > ?376?? if (pages == NULL) { > ?377???? return NULL; > ?378?? } > ?379?? mapped = calloc(pageCount, sizeof(int)); > ?380?? if (mapped == NULL) { > ?381???? free(pages); > ?382???? return NULL; > ?383?? } > > Just a question: > ? We do not release the array allocated at line 354 because this local > reference > ? will be auto-released when returning to Java. Is this correct? > Good point, I think we better add DeleteLocalRef here in case of the new early returns . > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. > agent/macosx/native/libsaproc/symtab.c.frames.html > > ? 69???? if (is_debug()) { > ? 70?????? DBT rkey, rvalue;? 71?????? char* tmp = (char > *)malloc(strlen(symtab->symbols[i].name) + 1); > ? 72?????? if (tmp != NULL) { > ? 73???????? strcpy(tmp, symtab->symbols[i].name); > ? 74???????? rkey.data = tmp; > ? 75???????? rkey.size = strlen(tmp) + 1; > ? 76 (*symtab->hash_table->get)(symtab->hash_table, &rkey, &rvalue, 0); > ? 77???????? // we may get a copy back so compare contents > ? 78???????? symtab_symbol *res = (symtab_symbol *)rvalue.data; > ? 79???????? if (strcmp(res->name, symtab->symbols[i].name) || > ? 80?????????? res->offset != symtab->symbols[i].offset || > ? 81?????????? res->size != symtab->symbols[i].size) { > ? 82???????????? print_debug("error to get hash_table value!\n"); > ? 83???????? } > ? 84???????? free(tmp); > ? 85?????? } > > If malloc returns NULL then this debugging part will be we silently skipped. > In other such cases there is an attempt to print a debug message. > For instance: > > ?140?? symtab = (symtab_t *)malloc(sizeof(symtab_t)); > ?141?? if (symtab == NULL) { > ?142???? print_debug("out of memory: allocating symtab\n"); > ?143???? return NULL; > ?144?? } > > I understand that print_debug can fail with out of memory as well. > But it depends on its implementation. > > Thanks, > Serguei > > That's a good idea . I added a message . See new webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.2/ Thanks, Matthias From serguei.spitsyn at oracle.com Thu Sep 5 08:33:29 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 5 Sep 2019 01:33:29 -0700 Subject: RFR: 8230466: check malloc/calloc results in jdk.hotspot.agent In-Reply-To: References: <6b6000a6-ca58-dc65-8996-5862a6e3763a@gmail.com> <796065c6-a4ee-92b2-0881-451537f2a4b9@oracle.com> Message-ID: Hi Matthias, On 9/5/19 01:19, Baesken, Matthias wrote: > Hello Serguei, thanks for the comments . > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. >> agent/linux/native/libsaproc/symtab.c.frames.html >> >> ?279 build_id_to_debug_filename (size_t size, unsigned char *data) >> ?280 { >> ?. . . >> ?283?? filename = malloc(strlen (debug_file_directory) + (sizeof >> "/.build-id/" - 1) + 1 >> ?284???????????????????? + 2 * size + (sizeof ".debug" - 1) + 1); >> ?285?? if (filename == NULL) { >> ?286???? return NULL; >> ?287?? } >> ?. . . >> ?312?? char *filename >> ?313???? = (build_id_to_debug_filename (note->n_descsz, bytes)); >> ?314?? if (filename == NULL) { >> ?315???? return NULL; >> ?316?? } >> >> There is no need to check filename for NULL at the line 314 as the function >> build_id_to_debug_filename with new check at the line 285 never returns >> NULL. >> > At line 286 of build_id_to_debug_filename we now return NULL (in case malloc cannot alloc memory). So we should check this also > at line 314 , or do I miss something ? Oh, sorry. You are right, thanks! >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. >> agent/macosx/native/libsaproc/MacosxDebuggerLocal.m.frames.html >> >> ?354?? array = (*env)->NewByteArray(env, numBytes); >> ?. . . >> ?376?? if (pages == NULL) { >> ?377???? return NULL; >> ?378?? } >> ?379?? mapped = calloc(pageCount, sizeof(int)); >> ?380?? if (mapped == NULL) { >> ?381???? free(pages); >> ?382???? return NULL; >> ?383?? } >> >> Just a question: >> ? We do not release the array allocated at line 354 because this local >> reference >> ? will be auto-released when returning to Java. Is this correct? >> > Good point, I think we better add DeleteLocalRef here in case of the new early returns . > > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.1/src/jdk.hotspot. >> agent/macosx/native/libsaproc/symtab.c.frames.html >> >> ? 69???? if (is_debug()) { >> ? 70?????? DBT rkey, rvalue;? 71?????? char* tmp = (char >> *)malloc(strlen(symtab->symbols[i].name) + 1); >> ? 72?????? if (tmp != NULL) { >> ? 73???????? strcpy(tmp, symtab->symbols[i].name); >> ? 74???????? rkey.data = tmp; >> ? 75???????? rkey.size = strlen(tmp) + 1; >> ? 76 (*symtab->hash_table->get)(symtab->hash_table, &rkey, &rvalue, 0); >> ? 77???????? // we may get a copy back so compare contents >> ? 78???????? symtab_symbol *res = (symtab_symbol *)rvalue.data; >> ? 79???????? if (strcmp(res->name, symtab->symbols[i].name) || >> ? 80?????????? res->offset != symtab->symbols[i].offset || >> ? 81?????????? res->size != symtab->symbols[i].size) { >> ? 82???????????? print_debug("error to get hash_table value!\n"); >> ? 83???????? } >> ? 84???????? free(tmp); >> ? 85?????? } >> >> If malloc returns NULL then this debugging part will be we silently skipped. >> In other such cases there is an attempt to print a debug message. >> For instance: >> >> ?140?? symtab = (symtab_t *)malloc(sizeof(symtab_t)); >> ?141?? if (symtab == NULL) { >> ?142???? print_debug("out of memory: allocating symtab\n"); >> ?143???? return NULL; >> ?144?? } >> >> I understand that print_debug can fail with out of memory as well. >> But it depends on its implementation. >> >> Thanks, >> Serguei >> >> > That's a good idea . I added a message . > > See new webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8230466.2/ Thank you for the update! It looks good to me. Thanks, Serguei > > > Thanks, Matthias > From christoph.langer at sap.com Thu Sep 5 15:01:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 5 Sep 2019 15:01:16 +0000 Subject: RFR(S): 8230666: Exclude serviceability/sa/TestInstanceKlassSize.java on linuxppc64 and linuxppc64le Message-ID: Hi, please review exclusion of serviceability/sa/TestInstanceKlassSize.java for linux on the ppc platforms. The test was probably ever failing on these platforms. Martin has done some initial analysis of the problem. I've opened https://bugs.openjdk.java.net/browse/JDK-8230664 to track resolution (@Martin: can you please add some more technical detail to the bug? Thanks.) The resolution will probably take some time and the platforms are not tier1, so we shall exempt the test from being executed for the time being. This is the bug: https://bugs.openjdk.java.net/browse/JDK-8230666 This is the patch: diff -r 397b97fb989c test/hotspot/jtreg/ProblemList.txt --- a/test/hotspot/jtreg/ProblemList.txt Thu Sep 05 16:50:18 2019 +0200 +++ b/test/hotspot/jtreg/ProblemList.txt Thu Sep 05 16:54:36 2019 +0200 @@ -128,7 +128,7 @@ serviceability/sa/TestG1HeapRegion.java 8193639 solaris-all serviceability/sa/TestHeapDumpForInvokeDynamic.java 8193639 solaris-all serviceability/sa/TestHeapDumpForLargeArray.java 8193639 solaris-all -serviceability/sa/TestInstanceKlassSize.java 8193639 solaris-all +serviceability/sa/TestInstanceKlassSize.java 8193639,8230664 solaris-all,linux-ppc64le,linux-ppc64 serviceability/sa/TestInstanceKlassSizeForInterface.java 8193639 solaris-all serviceability/sa/TestIntConstant.java 8193639,8211767 solaris-all,linux-ppc64le,linux-ppc64 serviceability/sa/TestJhsdbJstackLock.java 8193639 solaris-all Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Thu Sep 5 15:49:21 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 5 Sep 2019 11:49:21 -0400 Subject: RFR(S): 8230666: Exclude serviceability/sa/TestInstanceKlassSize.java on linuxppc64 and linuxppc64le In-Reply-To: References: Message-ID: Thumbs up. This is a trivial fix so only requires a single (R)eviewer. Dan On 9/5/19 11:01 AM, Langer, Christoph wrote: > > Hi, > > please review exclusion of > serviceability/sa/TestInstanceKlassSize.java for linux on the ppc > platforms. > > The test was probably ever failing on these platforms. Martin has done > some initial analysis of the problem. I?ve opened > https://bugs.openjdk.java.net/browse/JDK-8230664 to track resolution > (@Martin: can you please add some more technical detail to the bug? > Thanks.) > > The resolution will probably take some time and the platforms are not > tier1, so we shall exempt the test from being executed for the time being. > > This is the bug: https://bugs.openjdk.java.net/browse/JDK-8230666 > > > This is the patch: > > diff -r 397b97fb989c test/hotspot/jtreg/ProblemList.txt > > --- a/test/hotspot/jtreg/ProblemList.txt??????? Thu Sep 05 16:50:18 > 2019 +0200 > > +++ b/test/hotspot/jtreg/ProblemList.txt??????? Thu Sep 05 16:54:36 > 2019 +0200 > > @@ -128,7 +128,7 @@ > > serviceability/sa/TestG1HeapRegion.java 8193639 solaris-all > > serviceability/sa/TestHeapDumpForInvokeDynamic.java 8193639 solaris-all > > serviceability/sa/TestHeapDumpForLargeArray.java 8193639 solaris-all > > -serviceability/sa/TestInstanceKlassSize.java 8193639 solaris-all > > +serviceability/sa/TestInstanceKlassSize.java 8193639,8230664 > solaris-all,linux-ppc64le,linux-ppc64 > > serviceability/sa/TestInstanceKlassSizeForInterface.java 8193639 > solaris-all > > serviceability/sa/TestIntConstant.java 8193639,8211767 > solaris-all,linux-ppc64le,linux-ppc64 > > serviceability/sa/TestJhsdbJstackLock.java 8193639 solaris-all > > Thanks > > Christoph > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.menkov at oracle.com Thu Sep 5 17:36:29 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Thu, 5 Sep 2019 10:36:29 -0700 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> Message-ID: <879069b3-5fa5-0264-d1b3-f21291e40aa9@oracle.com> Looks reasonable. --alex On 09/04/2019 23:43, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 > webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ > > See bug report for gory details. Basically on Windows thread-cpu-time > may only get updated at the resolution of the timer interrupt, which may > be as long as 16ms. The test checks that the elapsed cpu time is < 10ms > and so would fail if the first and second calls to get the time > straddled a timer interrupt (and normally it passes because both calls > return zero). > > Fix is to bump the allowed time to 16ms on Windows. > > I also corrected a comment as to why part of the test is already > disabled on Windows, and enabled verbose logging so that if this test > fails again it will be much easier to see why. > > Testing: itself, on Windows numerous times > > Thanks, > David From alexey.menkov at oracle.com Thu Sep 5 18:20:19 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Thu, 5 Sep 2019 11:20:19 -0700 Subject: RFR: JDK-8186825: some memory leak issues in the transport_startTransport Message-ID: Hi all, Please review the fix for https://bugs.openjdk.java.net/browse/JDK-8186825 webrev: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp_memory_leak/webrev/ TransportInfo structure is used to pass data from main thread (transport_startTransport function) to acceptThread/attachThread and should be released by acceptThread/attachThread. --alex From serguei.spitsyn at oracle.com Thu Sep 5 18:48:38 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 5 Sep 2019 11:48:38 -0700 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> Message-ID: Hi David, It looks good. Thank you for taking care about this! Thanks, Serguei On 9/4/19 23:43, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 > webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ > > See bug report for gory details. Basically on Windows thread-cpu-time > may only get updated at the resolution of the timer interrupt, which > may be as long as 16ms. The test checks that the elapsed cpu time is < > 10ms and so would fail if the first and second calls to get the time > straddled a timer interrupt (and normally it passes because both calls > return zero). > > Fix is to bump the allowed time to 16ms on Windows. > > I also corrected a comment as to why part of the test is already > disabled on Windows, and enabled verbose logging so that if this test > fails again it will be much easier to see why. > > Testing: itself, on Windows numerous times > > Thanks, > David From serguei.spitsyn at oracle.com Thu Sep 5 19:08:19 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 5 Sep 2019 12:08:19 -0700 Subject: RFR: JDK-8186825: some memory leak issues in the transport_startTransport In-Reply-To: References: Message-ID: <9531fb93-e9e0-5897-a466-8422244477ca@oracle.com> Hi Alex, Looks good to me. Thanks, Serguei On 9/5/19 11:20, Alex Menkov wrote: > Hi all, > > Please review the fix for > ? https://bugs.openjdk.java.net/browse/JDK-8186825 > webrev: > http://cr.openjdk.java.net/~amenkov/jdk14/jdwp_memory_leak/webrev/ > > TransportInfo structure is used to pass data from main thread > (transport_startTransport function) to acceptThread/attachThread and > should be released by acceptThread/attachThread. > > --alex From chris.plummer at oracle.com Thu Sep 5 20:53:10 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 5 Sep 2019 13:53:10 -0700 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> Message-ID: <5a6c9757-166c-bf9e-d125-6782d5a863f1@oracle.com> +1 Chris On 9/5/19 11:48 AM, serguei.spitsyn at oracle.com wrote: > Hi David, > > It looks good. > Thank you for taking care about this! > > Thanks, > Serguei > > > On 9/4/19 23:43, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 >> webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ >> >> See bug report for gory details. Basically on Windows thread-cpu-time >> may only get updated at the resolution of the timer interrupt, which >> may be as long as 16ms. The test checks that the elapsed cpu time is >> < 10ms and so would fail if the first and second calls to get the >> time straddled a timer interrupt (and normally it passes because both >> calls return zero). >> >> Fix is to bump the allowed time to 16ms on Windows. >> >> I also corrected a comment as to why part of the test is already >> disabled on Windows, and enabled verbose logging so that if this test >> fails again it will be much easier to see why. >> >> Testing: itself, on Windows numerous times >> >> Thanks, >> David > From david.holmes at oracle.com Thu Sep 5 22:00:57 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Sep 2019 08:00:57 +1000 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: <879069b3-5fa5-0264-d1b3-f21291e40aa9@oracle.com> References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> <879069b3-5fa5-0264-d1b3-f21291e40aa9@oracle.com> Message-ID: Thanks Alex. david On 6/09/2019 3:36 am, Alex Menkov wrote: > Looks reasonable. > > --alex > > On 09/04/2019 23:43, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 >> webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ >> >> See bug report for gory details. Basically on Windows thread-cpu-time >> may only get updated at the resolution of the timer interrupt, which >> may be as long as 16ms. The test checks that the elapsed cpu time is < >> 10ms and so would fail if the first and second calls to get the time >> straddled a timer interrupt (and normally it passes because both calls >> return zero). >> >> Fix is to bump the allowed time to 16ms on Windows. >> >> I also corrected a comment as to why part of the test is already >> disabled on Windows, and enabled verbose logging so that if this test >> fails again it will be much easier to see why. >> >> Testing: itself, on Windows numerous times >> >> Thanks, >> David From david.holmes at oracle.com Thu Sep 5 22:02:30 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Sep 2019 08:02:30 +1000 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> Message-ID: <59729749-dafb-4f2a-c76d-2e4acf260f7b@oracle.com> On 6/09/2019 4:48 am, serguei.spitsyn at oracle.com wrote: > Hi David, > > It looks good. Thanks serguei. > Thank you for taking care about this! It's going to fail more often after my changes in JDK-6313903 (not really certain why though). Cheers, David > Thanks, > Serguei > > > On 9/4/19 23:43, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 >> webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ >> >> See bug report for gory details. Basically on Windows thread-cpu-time >> may only get updated at the resolution of the timer interrupt, which >> may be as long as 16ms. The test checks that the elapsed cpu time is < >> 10ms and so would fail if the first and second calls to get the time >> straddled a timer interrupt (and normally it passes because both calls >> return zero). >> >> Fix is to bump the allowed time to 16ms on Windows. >> >> I also corrected a comment as to why part of the test is already >> disabled on Windows, and enabled verbose logging so that if this test >> fails again it will be much easier to see why. >> >> Testing: itself, on Windows numerous times >> >> Thanks, >> David > From david.holmes at oracle.com Thu Sep 5 22:02:53 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Sep 2019 08:02:53 +1000 Subject: RFR (S): 8227563: jvmti/scenarios/contention/TC05/tc05t001 fails ... In-Reply-To: <5a6c9757-166c-bf9e-d125-6782d5a863f1@oracle.com> References: <57caaba9-3d9d-6db7-d4af-79f48dd8db1c@oracle.com> <5a6c9757-166c-bf9e-d125-6782d5a863f1@oracle.com> Message-ID: <3384cc69-19d0-137c-bb5c-b97adbb85d87@oracle.com> Thanks Chris! David On 6/09/2019 6:53 am, Chris Plummer wrote: > +1 > > Chris > > On 9/5/19 11:48 AM, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> It looks good. >> Thank you for taking care about this! >> >> Thanks, >> Serguei >> >> >> On 9/4/19 23:43, David Holmes wrote: >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8227563 >>> webrev: http://cr.openjdk.java.net/~dholmes/8227563/webrev/ >>> >>> See bug report for gory details. Basically on Windows thread-cpu-time >>> may only get updated at the resolution of the timer interrupt, which >>> may be as long as 16ms. The test checks that the elapsed cpu time is >>> < 10ms and so would fail if the first and second calls to get the >>> time straddled a timer interrupt (and normally it passes because both >>> calls return zero). >>> >>> Fix is to bump the allowed time to 16ms on Windows. >>> >>> I also corrected a comment as to why part of the test is already >>> disabled on Windows, and enabled verbose logging so that if this test >>> fails again it will be much easier to see why. >>> >>> Testing: itself, on Windows numerous times >>> >>> Thanks, >>> David >> > From ioi.lam at oracle.com Fri Sep 6 02:27:38 2019 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 5 Sep 2019 19:27:38 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects Message-ID: https://bugs.openjdk.java.net/browse/JDK-8230674 http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 Please review this small fix: When CDS is in use, archived objects are memory-mapped into the heap (currently G1GC only). These objects are partitioned into "subgraphs". Some of these subgraphs may not be loaded (e.g., those related to jdk.internal.math.FDBigInteger) at the time a heap dump is requested. When a subgraph is not loaded, some of the objects in this subgraph may belong to a class that's not yet loaded. The bug happens when such an "dormant" object is dumped, but its class is not dumped because the class is not in the system dictionary. There is already code in DumperSupport::dump_instance() that tries to handle dormant objects, but it needs to be extended to cover arrays, as well as and references from non-dormant object/arrays to dormant ones. Thanks - Ioi From david.holmes at oracle.com Fri Sep 6 03:18:52 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Sep 2019 13:18:52 +1000 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects In-Reply-To: References: Message-ID: Hi Ioi, On 6/09/2019 12:27 pm, Ioi Lam wrote: > https://bugs.openjdk.java.net/browse/JDK-8230674 > http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 > > > Please review this small fix: > > When CDS is in use, archived objects are memory-mapped into the heap > (currently G1GC only). These objects are partitioned into > "subgraphs". Some of these subgraphs may not be loaded (e.g., those > related to jdk.internal.math.FDBigInteger) at the time a heap dump is > requested. > > When a subgraph is not loaded, some of the objects in this subgraph may > belong to a class that's not yet loaded. > > The bug happens when such an "dormant" object is dumped, but its class > is not dumped because the class is not in the system dictionary. > > There is already code in DumperSupport::dump_instance() that tries to > handle dormant objects, but it needs to be extended to cover arrays, as well as > and references from non-dormant object/arrays to dormant ones. I have to confess I did not pay any attention to the CDS archived objects work, so I don't have a firm grasp of how you have implemented things. But I'm wondering how can you have a reference to a dormant object from a non-dormant one? Shouldn't the act of becoming non-dormant automatically cause the subgraph from that object to also become non-dormant? Or do you have "read barriers" to perform the changes on demand? That aside the code changes seem reasonable, you moved the check out of DumperSupport::dump_instance and into the higher-level HeapObjectDumper::do_object so that it catches instances and arrays, plus you added a check for array elements. Thanks, David > > > Thanks > - Ioi From ioi.lam at oracle.com Fri Sep 6 03:39:51 2019 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 5 Sep 2019 20:39:51 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects In-Reply-To: References: Message-ID: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> On 9/5/19 8:18 PM, David Holmes wrote: > Hi Ioi, > > On 6/09/2019 12:27 pm, Ioi Lam wrote: >> https://bugs.openjdk.java.net/browse/JDK-8230674 >> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >> >> >> Please review this small fix: >> >> When CDS is in use, archived objects are memory-mapped into the heap >> (currently G1GC only). These objects are partitioned into >> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >> requested. > >> When a subgraph is not loaded, some of the objects in this subgraph >> may belong to a class that's not yet loaded. >> >> The bug happens when such an "dormant" object is dumped, but its class >> is not dumped because the class is not in the system dictionary. >> >> There is already code in DumperSupport::dump_instance() that tries to >> handle dormant objects, but it needs to be extended to cover arrays, >> as well as and references from non-dormant object/arrays to dormant >> ones. > > I have to confess I did not pay any attention to the CDS archived > objects work, so I don't have a firm grasp of how you have implemented > things. But I'm wondering how can you have a reference to a dormant > object from a non-dormant one? Shouldn't the act of becoming > non-dormant automatically cause the subgraph from that object to also > become non-dormant? Or do you have "read barriers" to perform the > changes on demand? > Hi David, Thanks for the review. The dormant objects are not reachable via the GC roots. They become non-dormant via explicit calls to JVM_InitializeFromArchive, after which they become reachable via the static fields of loaded classes. The only issue here is heap dump is done by scanning all objects in the heap, including unreachable ones ? HeapObjectDumper obj_dumper(this, writer()); ? Universe::heap()->safe_object_iterate(&obj_dumper); that's how these dormant objects are discovered during heap dump. > That aside the code changes seem reasonable, you moved the check out > of DumperSupport::dump_instance and into the higher-level > HeapObjectDumper::do_object so that it catches instances and arrays, > plus you added a check for array elements. > I am debating whether I should put the masking code in here: void DumpWriter::write_objectID(oop o) { ? o = mask_dormant_archived_object(o);? /// <---- add ? address a = (address)o; #ifdef _LP64 ? write_u8((u8)a); #else ? write_u4((u4)a); #endif } That way, even if a dormant object (unintentionally) becomes reachable via the GC roots, we won't write an invalid reference to it (the object "body" will not be written, so the ID will not point to anything valid). But this seems a little too aggressive to me. What do you think? Thanks - Ioi From david.holmes at oracle.com Fri Sep 6 06:11:19 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Sep 2019 16:11:19 +1000 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects In-Reply-To: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> Message-ID: On 6/09/2019 1:39 pm, Ioi Lam wrote: > On 9/5/19 8:18 PM, David Holmes wrote: >> Hi Ioi, >> >> On 6/09/2019 12:27 pm, Ioi Lam wrote: >>> https://bugs.openjdk.java.net/browse/JDK-8230674 >>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >>> >>> >>> Please review this small fix: >>> >>> When CDS is in use, archived objects are memory-mapped into the heap >>> (currently G1GC only). These objects are partitioned into >>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >>> requested. > >>> When a subgraph is not loaded, some of the objects in this subgraph >>> may belong to a class that's not yet loaded. >>> >>> The bug happens when such an "dormant" object is dumped, but its class >>> is not dumped because the class is not in the system dictionary. >>> >>> There is already code in DumperSupport::dump_instance() that tries to >>> handle dormant objects, but it needs to be extended to cover arrays, >>> as well as and references from non-dormant object/arrays to dormant >>> ones. >> >> I have to confess I did not pay any attention to the CDS archived >> objects work, so I don't have a firm grasp of how you have implemented >> things. But I'm wondering how can you have a reference to a dormant >> object from a non-dormant one? Shouldn't the act of becoming >> non-dormant automatically cause the subgraph from that object to also >> become non-dormant? Or do you have "read barriers" to perform the >> changes on demand? >> > > Hi David, > > Thanks for the review. > > The dormant objects are not reachable via the GC roots. They become > non-dormant via explicit calls to JVM_InitializeFromArchive, after which > they become reachable via the static fields of loaded classes. Right, so is there a distinction between non-dormant and reachable at the time an object becomes non-dormant? I'm still unclear how a drmant array becomes non-dormant but still contains elements that refer to dormant objects. > The only issue here is heap dump is done by scanning all objects in the > heap, including unreachable ones > > ? HeapObjectDumper obj_dumper(this, writer()); > ? Universe::heap()->safe_object_iterate(&obj_dumper); > > that's how these dormant objects are discovered during heap dump. > >> That aside the code changes seem reasonable, you moved the check out >> of DumperSupport::dump_instance and into the higher-level >> HeapObjectDumper::do_object so that it catches instances and arrays, >> plus you added a check for array elements. >> > > I am debating whether I should put the masking code in here: > > void DumpWriter::write_objectID(oop o) { > ? o = mask_dormant_archived_object(o);? /// <---- add > ? address a = (address)o; > #ifdef _LP64 > ? write_u8((u8)a); > #else > ? write_u4((u4)a); > #endif > } > > > That way, even if a dormant object (unintentionally) becomes reachable > via the GC roots, we won't write an invalid reference to it (the object > "body" will not be written, so the ID will not point to anything valid). > > But this seems a little too aggressive to me. What do you think? It does seem a little aggressive as it seems to introduce the dormancy check into a lot of places that don't need it. But as I said I don't know this code so I'm really not the right person to ask. Cheers, David ----- > Thanks > - Ioi > From christoph.langer at sap.com Fri Sep 6 13:15:26 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 6 Sep 2019 13:15:26 +0000 Subject: RFR(S): 8230666: Exclude serviceability/sa/TestInstanceKlassSize.java on linuxppc64 and linuxppc64le In-Reply-To: References: Message-ID: Thanks, Dan. Pushed. From: Daniel D. Daugherty Sent: Donnerstag, 5. September 2019 17:49 To: Langer, Christoph ; OpenJDK Serviceability Subject: Re: RFR(S): 8230666: Exclude serviceability/sa/TestInstanceKlassSize.java on linuxppc64 and linuxppc64le Thumbs up. This is a trivial fix so only requires a single (R)eviewer. Dan On 9/5/19 11:01 AM, Langer, Christoph wrote: Hi, please review exclusion of serviceability/sa/TestInstanceKlassSize.java for linux on the ppc platforms. The test was probably ever failing on these platforms. Martin has done some initial analysis of the problem. I've opened https://bugs.openjdk.java.net/browse/JDK-8230664 to track resolution (@Martin: can you please add some more technical detail to the bug? Thanks.) The resolution will probably take some time and the platforms are not tier1, so we shall exempt the test from being executed for the time being. This is the bug: https://bugs.openjdk.java.net/browse/JDK-8230666 This is the patch: diff -r 397b97fb989c test/hotspot/jtreg/ProblemList.txt --- a/test/hotspot/jtreg/ProblemList.txt Thu Sep 05 16:50:18 2019 +0200 +++ b/test/hotspot/jtreg/ProblemList.txt Thu Sep 05 16:54:36 2019 +0200 @@ -128,7 +128,7 @@ serviceability/sa/TestG1HeapRegion.java 8193639 solaris-all serviceability/sa/TestHeapDumpForInvokeDynamic.java 8193639 solaris-all serviceability/sa/TestHeapDumpForLargeArray.java 8193639 solaris-all -serviceability/sa/TestInstanceKlassSize.java 8193639 solaris-all +serviceability/sa/TestInstanceKlassSize.java 8193639,8230664 solaris-all,linux-ppc64le,linux-ppc64 serviceability/sa/TestInstanceKlassSizeForInterface.java 8193639 solaris-all serviceability/sa/TestIntConstant.java 8193639,8211767 solaris-all,linux-ppc64le,linux-ppc64 serviceability/sa/TestJhsdbJstackLock.java 8193639 solaris-all Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.reingruber at sap.com Fri Sep 6 14:24:00 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Fri, 6 Sep 2019 14:24:00 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Message-ID: Hi, could I please get reviews for Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to retrieve objects locked by a thread. In terms of escape analysis those references escape and optimizations like scalar replacement become invalid. The runtime currently cannot cope with objects escaping through JVMTI (try included tests). Therefore escape analysis should be disabled if an agent requests the capabilities can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to disable escape analysis, instead optimizations based on escape analysis will be reverted just before objects escape through JVMTI. I've run tier1 tests. Thanks, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8227745 From jianglizhou at google.com Fri Sep 6 15:18:37 2019 From: jianglizhou at google.com (Jiangli Zhou) Date: Fri, 6 Sep 2019 08:18:37 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects In-Reply-To: References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> Message-ID: Just to answer David's questions. Those are all very good questions! A dormant object is an unreachable object in the archived Java heap region. At the moment when the VM mapps in the 'Open' archive heap regions, all objects within the regions are considered as dormant objects. The state of an archived object changes when its reachability changes. Any archived Java object that is reachable by a live (non-dormant) object is effectively a non-dormant/live object. An archived object state is changed explicitly when it is 'installed'. HeapShared::materialize_archived_object() is called for that particular object to make GC aware about the object. For example, when a shared class is loaded, the corresponding archived mirror object is 'installed' in the shared klass. That's when the mirror object becomes alive. Another example is an archived object for static field value and all reachable objects from it. When the VM 'installs' an archived static field value back to the field, the object becomes alive explicitly. All reachable objects via the entry object also become non-dormant/alive implicitly. Please see more info in https://wiki.openjdk.java.net/display/HotSpot/Caching+Java+Heap+Objects. Will send review separately. Best, Jiangli On Thu, Sep 5, 2019 at 11:12 PM David Holmes wrote: > > On 6/09/2019 1:39 pm, Ioi Lam wrote: > > On 9/5/19 8:18 PM, David Holmes wrote: > >> Hi Ioi, > >> > >> On 6/09/2019 12:27 pm, Ioi Lam wrote: > >>> https://bugs.openjdk.java.net/browse/JDK-8230674 > >>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 > >>> > >>> > >>> Please review this small fix: > >>> > >>> When CDS is in use, archived objects are memory-mapped into the heap > >>> (currently G1GC only). These objects are partitioned into > >>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those > >>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is > >>> requested. > > >>> When a subgraph is not loaded, some of the objects in this subgraph > >>> may belong to a class that's not yet loaded. > >>> > >>> The bug happens when such an "dormant" object is dumped, but its class > >>> is not dumped because the class is not in the system dictionary. > >>> > >>> There is already code in DumperSupport::dump_instance() that tries to > >>> handle dormant objects, but it needs to be extended to cover arrays, > >>> as well as and references from non-dormant object/arrays to dormant > >>> ones. > >> > >> I have to confess I did not pay any attention to the CDS archived > >> objects work, so I don't have a firm grasp of how you have implemented > >> things. But I'm wondering how can you have a reference to a dormant > >> object from a non-dormant one? Shouldn't the act of becoming > >> non-dormant automatically cause the subgraph from that object to also > >> become non-dormant? Or do you have "read barriers" to perform the > >> changes on demand? > >> > > > > Hi David, > > > > Thanks for the review. > > > > The dormant objects are not reachable via the GC roots. They become > > non-dormant via explicit calls to JVM_InitializeFromArchive, after which > > they become reachable via the static fields of loaded classes. > > Right, so is there a distinction between non-dormant and reachable at > the time an object becomes non-dormant? I'm still unclear how a drmant > array becomes non-dormant but still contains elements that refer to > dormant objects. > > > The only issue here is heap dump is done by scanning all objects in the > > heap, including unreachable ones > > > > HeapObjectDumper obj_dumper(this, writer()); > > Universe::heap()->safe_object_iterate(&obj_dumper); > > > > that's how these dormant objects are discovered during heap dump. > > > >> That aside the code changes seem reasonable, you moved the check out > >> of DumperSupport::dump_instance and into the higher-level > >> HeapObjectDumper::do_object so that it catches instances and arrays, > >> plus you added a check for array elements. > >> > > > > I am debating whether I should put the masking code in here: > > > > void DumpWriter::write_objectID(oop o) { > > o = mask_dormant_archived_object(o); /// <---- add > > address a = (address)o; > > #ifdef _LP64 > > write_u8((u8)a); > > #else > > write_u4((u4)a); > > #endif > > } > > > > > > That way, even if a dormant object (unintentionally) becomes reachable > > via the GC roots, we won't write an invalid reference to it (the object > > "body" will not be written, so the ID will not point to anything valid). > > > > But this seems a little too aggressive to me. What do you think? > > It does seem a little aggressive as it seems to introduce the dormancy > check into a lot of places that don't need it. But as I said I don't > know this code so I'm really not the right person to ask. > > Cheers, > David > ----- > > > Thanks > > - Ioi > > From richard.reingruber at sap.com Fri Sep 6 16:28:39 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Fri, 6 Sep 2019 16:28:39 +0000 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Alex, that's a good fix for the issue. One minor thing: 89 Exception error = null; 90 for (int retry = 0; retry < 5; retry++) { 91 try { 92 log("retry: " + retry); 93 s = new Socket("localhost", port); 94 error = null; 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); 96 break; 97 } catch (ConnectException ex) { 98 log("got exception: " + ex.toString()); 99 error = ex; 100 } 101 } 102 if (error != null) { 103 throw error; 104 } Is there a reason to clear the local variable error in line 94 instead of clearing it in line 91 where each new attempt begins? Cheers, Richard. -----Original Message----- From: serviceability-dev On Behalf Of serguei.spitsyn at oracle.com Sent: Mittwoch, 4. September 2019 22:11 To: Alex Menkov ; OpenJDK Serviceability Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException Hi Alex, The fix looks good. Good simplification! Thanks, Serguei On 9/4/19 12:19, Alex Menkov wrote: > Hi all, > > Please review the fix for BadHandshakeTest test. > The problem is the test connects to the server twice and if debuggee > hasn't yet handled disconnection, the next connect gets "connection > refused" error. > Instead of adding delay before 2nd connect (we never know "good" value > for the delay and big delay can cause "accept timeout"), the test > re-tries connect in case of ConnectException. > Also improved/simplified the test slightly - debuggee is now run with > auto port assignment (used lib.jdb.Debuggee test class which > implements required functionality). > > jira: > ? https://bugs.openjdk.java.net/browse/JDK-8192057 > webrev: > http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ > > --alex From ioi.lam at oracle.com Fri Sep 6 16:43:23 2019 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 6 Sep 2019 09:43:23 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> Message-ID: <96a98c4d-6313-3176-538f-08043f807389@oracle.com> On 9/5/19 11:11 PM, David Holmes wrote: > On 6/09/2019 1:39 pm, Ioi Lam wrote: >> On 9/5/19 8:18 PM, David Holmes wrote: >>> Hi Ioi, >>> >>> On 6/09/2019 12:27 pm, Ioi Lam wrote: >>>> https://bugs.openjdk.java.net/browse/JDK-8230674 >>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >>>> >>>> >>>> Please review this small fix: >>>> >>>> When CDS is in use, archived objects are memory-mapped into the >>>> heap (currently G1GC only). These objects are partitioned into >>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >>>> requested. > >>>> When a subgraph is not loaded, some of the objects in this subgraph >>>> may belong to a class that's not yet loaded. >>>> >>>> The bug happens when such an "dormant" object is dumped, but its class >>>> is not dumped because the class is not in the system dictionary. >>>> >>>> There is already code in DumperSupport::dump_instance() that tries >>>> to handle dormant objects, but it needs to be extended to cover >>>> arrays, as well as and references from non-dormant object/arrays to >>>> dormant ones. >>> >>> I have to confess I did not pay any attention to the CDS archived >>> objects work, so I don't have a firm grasp of how you have >>> implemented things. But I'm wondering how can you have a reference >>> to a dormant object from a non-dormant one? Shouldn't the act of >>> becoming non-dormant automatically cause the subgraph from that >>> object to also become non-dormant? Or do you have "read barriers" to >>> perform the changes on demand? >>> Ah -- my bug title is not correct. I changed the bug title (and this e-mail subject) to Heap dumps should exclude dormant CDS archived objects **of unloaded classes** During the heap dump, we scan all objects in the heap, regardless of reachability. There's no way to decide reachability in HeapObjectDumper::do_object(), unless we perform an actual GC. But it's OK to include unreachable objects in the heap dump. (I guess it's useful to see how much garbage you have in the heap. There's an option to run a collection before dumping the heap.) There are 2 kinds of unreachable objects -- garbage: those that were once reachable but no longer, dormant: the archived objects that have never been reachable. Anyway, it's OK to dump dormant objects as long as their class has been loaded. The problem happens only when we dump a dormant object who class is not yet loaded (Eclipase MAT get confused when it sees an object whose class ID is invalid). So to answer your question, we can have a case with a dormant array (that contains a dormant object) like this: ??? Object[] array = {new ClassNotYetLoaded();} After my fix, the array will be dumped (we have no easy way of not doing that), but its contents becomes this in the .hprof file: ??? Object[] array = {null} Thanks - Ioi >> >> Hi David, >> >> Thanks for the review. >> >> The dormant objects are not reachable via the GC roots. They become >> non-dormant via explicit calls to JVM_InitializeFromArchive, after >> which they become reachable via the static fields of loaded classes. > > Right, so is there a distinction between non-dormant and reachable at > the time an object becomes non-dormant? I'm still unclear how a drmant > array becomes non-dormant but still contains elements that refer to > dormant objects. > >> The only issue here is heap dump is done by scanning all objects in >> the heap, including unreachable ones >> >> ?? HeapObjectDumper obj_dumper(this, writer()); >> ?? Universe::heap()->safe_object_iterate(&obj_dumper); >> >> that's how these dormant objects are discovered during heap dump. >> >>> That aside the code changes seem reasonable, you moved the check out >>> of DumperSupport::dump_instance and into the higher-level >>> HeapObjectDumper::do_object so that it catches instances and arrays, >>> plus you added a check for array elements. >>> >> >> I am debating whether I should put the masking code in here: >> >> void DumpWriter::write_objectID(oop o) { >> ?? o = mask_dormant_archived_object(o);? /// <---- add >> ?? address a = (address)o; >> #ifdef _LP64 >> ?? write_u8((u8)a); >> #else >> ?? write_u4((u4)a); >> #endif >> } >> >> >> That way, even if a dormant object (unintentionally) becomes >> reachable via the GC roots, we won't write an invalid reference to it >> (the object "body" will not be written, so the ID will not point to >> anything valid). >> >> But this seems a little too aggressive to me. What do you think? > > It does seem a little aggressive as it seems to introduce the dormancy > check into a lot of places that don't need it. But as I said I don't > know this code so I'm really not the right person to ask. > > Cheers, > David > ----- > >> Thanks >> - Ioi >> From hohensee at amazon.com Fri Sep 6 18:08:06 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 6 Sep 2019 18:08:06 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> Message-ID: <23ED3BD7-3B7E-43DE-84B8-DB52D44D362B@amazon.com> Ping. Anyone? ( Thanks, ?On 9/3/19, 12:39 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I removed ensureNonNullThreadIds() in favor of Objects.requireNonNull(ids). Thanks, Mandy, for your through reviews. May I get another reviewer to weigh in? Paul On 8/31/19, 5:06 PM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: Thanks, Mandy. I?ve finalized the CSR. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.04/. In management.cpp, I now have if (THREAD->is_Java_thread()) { return ((JavaThread*)THREAD)->cooked_allocated_bytes(); } In ThreadImpl.java, using requireNonNull would produce a different and less informative message, so I?d like to leave it as is. I changed throwIfNullThreadIds to ensureNonNullThreadIds, and throwIfThreadAllocatedMemoryNotSupported to ensureThreadAllocatedMemorySupported. I dropped the ?java.lang.? prefix from all uses of UnsupportedOperationException in both c.s.m.ThreadMXBean.java and j.l.m.ThreadMXBean.java, and did the same with SecurityException. ?@since 14? added to c.s.m.ThreadMXBean.java and the CSR. Do I need another reviewer? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 4:26 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread CSR reviewed. management.cpp 2083 java_thread = (JavaThread*)THREAD; 2084 if (java_thread->is_Java_thread()) { 2085 return java_thread->cooked_allocated_bytes(); 2086 } The cast should be done after is_Java_thread() test. ThreadImpl.java 162 private void throwIfNullThreadIds(long[] ids) { Even better: simply use Objects::requiresNonNull and this method can be removed. This suggests positive naming alternative to throwIfThreadAllocatedMemoryNotSupported - "ensureThreadAllocatedMemorySupported" (sorry I should have suggested that) ThreadMXBean.java 130 * @throws java.lang.UnsupportedOperationException if the Java virtual Nit: "java.lang." can be dropped. @since 14 is missing. Mandy On 8/30/19 3:33 PM, Hohensee, Paul wrote: Thanks for your review, Mandy. Revised webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.02/. I updated the CSR with your suggested javadoc for getCurrentThreadAllocatedBytes. It now matches that for getCurrentThreadUserTime and getCurrentThreadCputime. I also fixed the ?convenient? -> ?convenience? typos in j.l.m.ThreadMXBean.java. I meant GetOneThreads to be the possessive, but don?t feel strongly either way so I?m fine with GetOneThread. I updated ThreadImpl.java as you suggested, though in getThreadAllocatedBytes(long[] ids) I had to add a redundant-in-the-not-length-1-case check for a null ids reference. Would someone take a look at the Hotspot side and the test please? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 10:22 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread OK. That's better. Some review comments: The javadoc of getCurrentThreadAllocatedBytes() can simply say: "Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the current thread. This is a convenient method for local management use and is equivalent to calling getThreadAllocatedBytes(Thread.currentThread().getId()). src/hotspot/share/include/jmm.h GetOneThreadsAllocatedMemory: s/OneThreads/OneThread/ sun/management/ThreadImpl.java 43 private static final String THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED = 44 "Thread allocated memory measurement is not supported."; if (!isThreadAllocatedMemorySupported()) { throw new UnsupportedOperationException(THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED); } Perhaps the above can be refactored as throwIfAllocatedMemoryUnsupported() method. 391 if (ids.length == 1) { 392 sizes[0] = -1; : 398 if (ids.length == 1) { 399 long id = ids[0]; 400 sizes[0] = getThreadAllocatedMemory0( 401 Thread.currentThread().getId() == id ? 0 : id); 402 } else { It seems cleaner to handle the 1-element array case at the beginning of this method: if (ids.length == 1) { long size = getThreadAllocatedBytes(ids[0]); return new long[] { size }; } I didn't review the hotspot implementation and the test. Mandy On 8/29/19 10:01 AM, Hohensee, Paul wrote: My bad, Mandy. The webrev puts getCurrentThreadAllocatedBytes in com.sun.management.ThreadMXBean along with the current two getThreadAllocatedBytes methods for the reasons you list. I?ve updated the CSR to specify com.sun.management and added a rationale. AllocatedBytes is currently enabled by Hotspot by default because the overhead of recording TLAB occupancy is negligible. There?s no new GC code, nor will there be, so imo we don?t have to involve the GC folks. I.e., the new JMM method GetOneThreadsAllocatedBytes uses the existing cooked_allocated_bytes JavaThread method, and getCurrentThreadAllocatedBytes is the same as getThreadAllocatedBytes: it just bypasses the thread lookup code. I hadn?t tracked down what happens when getCurrentThreadUserTime and getCurrentThreadCpuTime are called before, but if I?m not mistaken, it the code in jcmd() in attachListener.cpp will call GetThreadCpuTimeWithKind in management.cpp, and it will ultimately use Thread::current() as the subject of the call, see os::current_thread_cpu_time in os_linux.cpp. That means that the CurrentThread methods should work remotely the same way they do locally. GetOneThreadsAllocatedBytes in management.cpp uses THREAD as its subject when called on behalf of getCurrentThreadAllocatedBytes, so it will also uses the current remote Java thread. Even if these methods only worked locally, there are many setups where apps are self-monitoring that could use the performance improvement. Thanks, Paul From: Mandy Chung Date: Wednesday, August 28, 2019 at 3:59 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, The CSR proposes this method in java.lang.management.ThreadMXBean as a Java SE feature. Has this been discussed with the GC team to commit measuring current thread's allocated bytes as Java SE feature? Can this be supported by all JVM implementation? What is the overhead if this is enabled by default? Does it need to be disabled? This metric is from TLAB that might be okay. This needs advice/discussion with GC experts. I see that CSR mentions it can be disabled and link to isThreadAllocatedMemoryEnabled() and setThreadAllocatedMemoryEnabled() methods but these methods are defined in com.sun.management.ThreadMXBean. As Alan points out, current thread makes sense only in local VM management. When this is monitored from a JMX client (e.g. jconsole to connect to a running JVM, "currentThreadAllowcatedBytes" attribute is the current thread in jconsole process which invoking Thread::currentThread? Mandy On 8/28/19 12:22 PM, Hohensee, Paul wrote: Please review a performance improvement for ThreadMXBean.getThreadAllocatedBytes and the addition of getCurrentThreadAllocatedBytes. JBS issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Webrev: http://cr.openjdk.java.net/~phh/8207266/webrev.00/ CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 Previous email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-July/024441.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-August/024763.html The CSR is for adding ThreadMXBean.getCurrentThreadAllocatedBytes. I?d be great for someone to review it. I took Mandy?s advice and put the fast paths in the library code. I added a new JMM method GetOneThreadsAllocatedBytes that works the same as GetThreadCpuTime: it uses a thread_id value of zero to distinguish the current thread. On my Mac laptop, the result runs 47x faster for the current thread than the old implementation. The 3 tests in test/jdk/com/sun/management/ThreadMXBean all pass. I added code to ThreadAllocatedMemory.java to test getCurrentThreadAllocatedBytes as well as variations on getThreadAllocatedBytes(id). A submit repo job is in progress. Thanks, Paul From jianglizhou at google.com Fri Sep 6 18:48:39 2019 From: jianglizhou at google.com (Jiangli Zhou) Date: Fri, 6 Sep 2019 11:48:39 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: <96a98c4d-6313-3176-538f-08043f807389@oracle.com> References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> <96a98c4d-6313-3176-538f-08043f807389@oracle.com> Message-ID: On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam wrote: > > > > On 9/5/19 11:11 PM, David Holmes wrote: > > On 6/09/2019 1:39 pm, Ioi Lam wrote: > >> On 9/5/19 8:18 PM, David Holmes wrote: > >>> Hi Ioi, > >>> > >>> On 6/09/2019 12:27 pm, Ioi Lam wrote: > >>>> https://bugs.openjdk.java.net/browse/JDK-8230674 > >>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 > >>>> > >>>> > >>>> Please review this small fix: > >>>> > >>>> When CDS is in use, archived objects are memory-mapped into the > >>>> heap (currently G1GC only). These objects are partitioned into > >>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those > >>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is > >>>> requested. > > >>>> When a subgraph is not loaded, some of the objects in this subgraph > >>>> may belong to a class that's not yet loaded. > >>>> > >>>> The bug happens when such an "dormant" object is dumped, but its class > >>>> is not dumped because the class is not in the system dictionary. > >>>> > >>>> There is already code in DumperSupport::dump_instance() that tries > >>>> to handle dormant objects, but it needs to be extended to cover > >>>> arrays, as well as and references from non-dormant object/arrays to > >>>> dormant ones. > >>> > >>> I have to confess I did not pay any attention to the CDS archived > >>> objects work, so I don't have a firm grasp of how you have > >>> implemented things. But I'm wondering how can you have a reference > >>> to a dormant object from a non-dormant one? Shouldn't the act of > >>> becoming non-dormant automatically cause the subgraph from that > >>> object to also become non-dormant? Or do you have "read barriers" to > >>> perform the changes on demand? > >>> > > Ah -- my bug title is not correct. > > I changed the bug title (and this e-mail subject) to > > Heap dumps should exclude dormant CDS archived objects **of unloaded > classes** > > During the heap dump, we scan all objects in the heap, regardless of > reachability. There's no way to decide reachability in > HeapObjectDumper::do_object(), unless we perform an actual GC. > > But it's OK to include unreachable objects in the heap dump. (I guess > it's useful to see how much garbage you have in the heap. There's an > option to run a collection before dumping the heap.) > > There are 2 kinds of unreachable objects -- garbage: those that were > once reachable but no longer, dormant: the archived objects that have > never been reachable. Currently Java object archiving framework only supports one directional state change: dormant -> live. An archived object can become a live object from dormant state, but it cannot go back to the dormant state. Need to investigate thoroughly for all cases before the 'live -> dormant' transition can be supported. All objects in the 'Open' archive heap region are associated with the builtin class loaders and their classes are not unloaded. The existing static fields for archiving within the JDK classes are selected and the associated objects do not become garbage once 'installed'. > > Anyway, it's OK to dump dormant objects as long as their class has been > loaded. The problem happens only when we dump a dormant object who class > is not yet loaded (Eclipase MAT get confused when it sees an object > whose class ID is invalid). Yes. That's a scenario needs to be handled for a tool that iterates the Java heap. A dormant object in the 'Open' archive heap region may have a 'invalid' klass since the klass may not be loaded yet at the moment. Your webrev looks reasonable to me on high level pending information for following questions. Can you please give more details on the dormant objects referenced from the arrays? What specific arrays are those? Regards, Jiangli > > So to answer your question, we can have a case with a dormant array > (that contains a dormant object) like this: > > Object[] array = {new ClassNotYetLoaded();} > > After my fix, the array will be dumped (we have no easy way of not doing > that), but its contents becomes this in the .hprof file: > > Object[] array = {null} > > Thanks > - Ioi > > > > >> > >> Hi David, > >> > >> Thanks for the review. > >> > >> The dormant objects are not reachable via the GC roots. They become > >> non-dormant via explicit calls to JVM_InitializeFromArchive, after > >> which they become reachable via the static fields of loaded classes. > > > > Right, so is there a distinction between non-dormant and reachable at > > the time an object becomes non-dormant? I'm still unclear how a drmant > > array becomes non-dormant but still contains elements that refer to > > dormant objects. > > > >> The only issue here is heap dump is done by scanning all objects in > >> the heap, including unreachable ones > >> > >> HeapObjectDumper obj_dumper(this, writer()); > >> Universe::heap()->safe_object_iterate(&obj_dumper); > >> > >> that's how these dormant objects are discovered during heap dump. > >> > >>> That aside the code changes seem reasonable, you moved the check out > >>> of DumperSupport::dump_instance and into the higher-level > >>> HeapObjectDumper::do_object so that it catches instances and arrays, > >>> plus you added a check for array elements. > >>> > >> > >> I am debating whether I should put the masking code in here: > >> > >> void DumpWriter::write_objectID(oop o) { > >> o = mask_dormant_archived_object(o); /// <---- add > >> address a = (address)o; > >> #ifdef _LP64 > >> write_u8((u8)a); > >> #else > >> write_u4((u4)a); > >> #endif > >> } > >> > >> > >> That way, even if a dormant object (unintentionally) becomes > >> reachable via the GC roots, we won't write an invalid reference to it > >> (the object "body" will not be written, so the ID will not point to > >> anything valid). > >> > >> But this seems a little too aggressive to me. What do you think? > > > > It does seem a little aggressive as it seems to introduce the dormancy > > check into a lot of places that don't need it. But as I said I don't > > know this code so I'm really not the right person to ask. > > > > Cheers, > > David > > ----- > > > >> Thanks > >> - Ioi > >> > From alexey.menkov at oracle.com Fri Sep 6 19:56:42 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Fri, 6 Sep 2019 12:56:42 -0700 Subject: RFR: JDK-8230516: invalid html in jdwp-protocol.html Message-ID: Hi all, Please review the fix for https://bugs.openjdk.java.net/browse/JDK-8230516 webrev: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/webrev/ The fix moves "anchor" spans to inside the cells. generated docs (no visual changes): old: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/0/jdwp-protocol.html new: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/1/jdwp-protocol.html --alex From daniel.daugherty at oracle.com Fri Sep 6 20:50:10 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 6 Sep 2019 16:50:10 -0400 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> Message-ID: <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Hi David, I've finally gotten back to this email thread... > FYI testing to date: > ?- tiers 1 -3 all platforms > ?- hotspot: serviceability/jvmti > ????????????????????????? /jdwp > ??????????? vmTestbase/nsk/jvmti > ????????????????????????? /jdwp > ?- JDK: com/sun/jdi You should also add: open/test/hotspot/jtreg/vmTestbase/nsk/jdb open/test/hotspot/jtreg/vmTestbase/nsk/jdi open/test/jdk/java/lang/instrument I took a quick look through the preliminary webrev and I don't see anything that worries me. Re: Thread.interrupt() and raw_wait() It would be good to see if that semantic is being tested via the JCK test suite for JVM/TI. I also very much like/appreciate the decoupling of JvmtiRawMonitors from ObjectMonitors... Thanks for tackling this crazy task. Dan On 8/15/19 2:22 AM, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 > > Preliminary webrev (still has rough edges): > http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ > > Background: > > We've had this comment for a long time: > > ?// The raw monitor subsystem is entirely distinct from normal > ?// java-synchronization or jni-synchronization.? raw monitors are not > ?// associated with objects.? They can be implemented in any manner > ?// that makes sense.? The original implementors decided to piggy-back > ?// the raw-monitor implementation on the existing Java objectMonitor > mechanism. > ?// This flaw needs to fixed.? We should reimplement raw monitors as > sui-generis. > ?// Specifically, we should not implement raw monitors via java monitors. > ?// Time permitting, we should disentangle and deconvolve the two > implementations > ?// and move the resulting raw monitor implementation over to the > JVMTI directories. > ?// Ideally, the raw monitor implementation would be built on top of > ?// park-unpark and nothing else. > > This is an attempt to do that disentangling so that we can then > consider changes to ObjectMonitor without having to worry about > JvmtiRawMonitors. But rather than building on low-level park/unpark > (which would require the same manual queue management and much of the > same complex code as exists in ObjectMonitor) I decided to try and do > this on top of PlatformMonitor. > > The reason this is just a RFC rather than RFR is that I overlooked a > non-trivial aspect of JvmtiRawMonitors: like Java monitors (as > implemented by ObjectMonitor) they interact with the Thread.interrupt > mechanism. This is not clearly stated in the JVM TI specification [1] > but only in passing by the possible errors for RawMonitorWait: > > JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again > > As I explain in the bug report there is no way to build in proper > interrupt support using PlatformMonitor as there is no way we can > "interrupt" the low-level pthread_cond_wait. But we can approximate > it. What I've done in this preliminary version is just check interrupt > state before and after the actual "wait" but we won't get woken by the > interrupt once we have actually blocked. Alternatively we could use a > periodic polling approach and wakeup every Nms to check for interruption. > > The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is not > affected by this choice as that code ignores the interrupt until the > real action it was waiting for has occurred. The interrupt is then > reposted later. > > But more generally there could be users of JvmtiRawMonitors that > expect/require that RawMonitorWait is responsive to Thread.interrupt > in a manner similar to Object.wait. And if any of them are reading > this then I'd like to know - hence this RFC :) > > FYI testing to date: > ?- tiers 1 -3 all platforms > ?- hotspot: serviceability/jvmti > ????????????????????????? /jdwp > ??????????? vmTestbase/nsk/jvmti > ????????????????????????? /jdwp > ?- JDK: com/sun/jdi > > Comments/opinions appreciated. > > Thanks, > David > > [1] > https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait From alexey.menkov at oracle.com Fri Sep 6 20:51:57 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Fri, 6 Sep 2019 13:51:57 -0700 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Richard, On 09/06/2019 09:28, Reingruber, Richard wrote: > Hi Alex, > > that's a good fix for the issue. > > One minor thing: > > 89 Exception error = null; > 90 for (int retry = 0; retry < 5; retry++) { > 91 try { > 92 log("retry: " + retry); > 93 s = new Socket("localhost", port); > 94 error = null; > 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); > 96 break; > 97 } catch (ConnectException ex) { > 98 log("got exception: " + ex.toString()); > 99 error = ex; > 100 } > 101 } > 102 if (error != null) { > 103 throw error; > 104 } > > Is there a reason to clear the local variable error in line 94 instead of clearing it > in line 91 where each new attempt begins? The logic here is: The cycle has 2 exits: - error (max retry attempts reached, error is set by last "catch") - success ("break" statement at line 96, error should be null) So error is cleared only after the socket is connected (this is the problematic operation which can cause ConnectException). Of course error can be cleared before each try - there is not functional difference. --alex > > Cheers, Richard. > > -----Original Message----- > From: serviceability-dev On Behalf Of serguei.spitsyn at oracle.com > Sent: Mittwoch, 4. September 2019 22:11 > To: Alex Menkov ; OpenJDK Serviceability > Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException > > Hi Alex, > > The fix looks good. > Good simplification! > > Thanks, > Serguei > > > On 9/4/19 12:19, Alex Menkov wrote: >> Hi all, >> >> Please review the fix for BadHandshakeTest test. >> The problem is the test connects to the server twice and if debuggee >> hasn't yet handled disconnection, the next connect gets "connection >> refused" error. >> Instead of adding delay before 2nd connect (we never know "good" value >> for the delay and big delay can cause "accept timeout"), the test >> re-tries connect in case of ConnectException. >> Also improved/simplified the test slightly - debuggee is now run with >> auto port assignment (used lib.jdb.Debuggee test class which >> implements required functionality). >> >> jira: >> ? https://bugs.openjdk.java.net/browse/JDK-8192057 >> webrev: >> http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ >> >> --alex > From daniil.x.titov at oracle.com Fri Sep 6 21:14:12 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Fri, 06 Sep 2019 14:14:12 -0700 Subject: RFR: JDK-8230516: invalid html in jdwp-protocol.html In-Reply-To: <0E3B211D-41F1-429F-8AFC-0BE3A5750B2C@oracle.com> References: <0E3B211D-41F1-429F-8AFC-0BE3A5750B2C@oracle.com> Message-ID: Hi Alex, The change looks good to me. Thanks! -Daniil ?On 9/6/19, 12:59 PM, "serviceability-dev on behalf of Alex Menkov" wrote: Hi all, Please review the fix for https://bugs.openjdk.java.net/browse/JDK-8230516 webrev: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/webrev/ The fix moves "anchor" spans to inside the cells. generated docs (no visual changes): old: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/0/jdwp-protocol.html new: http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/1/jdwp-protocol.html --alex From ioi.lam at oracle.com Fri Sep 6 22:17:13 2019 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 6 Sep 2019 15:17:13 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> <96a98c4d-6313-3176-538f-08043f807389@oracle.com> Message-ID: <3a510f34-7d34-cf02-f834-7522dadb8caa@oracle.com> On 9/6/19 11:48 AM, Jiangli Zhou wrote: > On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam wrote: >> >> >> On 9/5/19 11:11 PM, David Holmes wrote: >>> On 6/09/2019 1:39 pm, Ioi Lam wrote: >>>> On 9/5/19 8:18 PM, David Holmes wrote: >>>>> Hi Ioi, >>>>> >>>>> On 6/09/2019 12:27 pm, Ioi Lam wrote: >>>>>> https://bugs.openjdk.java.net/browse/JDK-8230674 >>>>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >>>>>> >>>>>> >>>>>> Please review this small fix: >>>>>> >>>>>> When CDS is in use, archived objects are memory-mapped into the >>>>>> heap (currently G1GC only). These objects are partitioned into >>>>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >>>>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >>>>>> requested. > >>>>>> When a subgraph is not loaded, some of the objects in this subgraph >>>>>> may belong to a class that's not yet loaded. >>>>>> >>>>>> The bug happens when such an "dormant" object is dumped, but its class >>>>>> is not dumped because the class is not in the system dictionary. >>>>>> >>>>>> There is already code in DumperSupport::dump_instance() that tries >>>>>> to handle dormant objects, but it needs to be extended to cover >>>>>> arrays, as well as and references from non-dormant object/arrays to >>>>>> dormant ones. >>>>> I have to confess I did not pay any attention to the CDS archived >>>>> objects work, so I don't have a firm grasp of how you have >>>>> implemented things. But I'm wondering how can you have a reference >>>>> to a dormant object from a non-dormant one? Shouldn't the act of >>>>> becoming non-dormant automatically cause the subgraph from that >>>>> object to also become non-dormant? Or do you have "read barriers" to >>>>> perform the changes on demand? >>>>> >> Ah -- my bug title is not correct. >> >> I changed the bug title (and this e-mail subject) to >> >> Heap dumps should exclude dormant CDS archived objects **of unloaded >> classes** >> >> During the heap dump, we scan all objects in the heap, regardless of >> reachability. There's no way to decide reachability in >> HeapObjectDumper::do_object(), unless we perform an actual GC. >> >> But it's OK to include unreachable objects in the heap dump. (I guess >> it's useful to see how much garbage you have in the heap. There's an >> option to run a collection before dumping the heap.) >> >> There are 2 kinds of unreachable objects -- garbage: those that were >> once reachable but no longer, dormant: the archived objects that have >> never been reachable. > Currently Java object archiving framework only supports one > directional state change: dormant -> live. An archived object can > become a live object from dormant state, but it cannot go back to the > dormant state. Need to investigate thoroughly for all cases before the > 'live -> dormant' transition can be supported. All objects in the > 'Open' archive heap region are associated with the builtin class > loaders and their classes are not unloaded. The existing static fields > for archiving within the JDK classes are selected and the associated > objects do not become garbage once 'installed'. > >> Anyway, it's OK to dump dormant objects as long as their class has been >> loaded. The problem happens only when we dump a dormant object who class >> is not yet loaded (Eclipase MAT get confused when it sees an object >> whose class ID is invalid). > Yes. That's a scenario needs to be handled for a tool that iterates > the Java heap. A dormant object in the 'Open' archive heap region may > have a 'invalid' klass since the klass may not be loaded yet at the > moment. > > Your webrev looks reasonable to me on high level pending information > for following questions. Can you please give more details on the > dormant objects referenced from the arrays? What specific arrays are > those? Hi Jiangli, Thanks for the review. I add the following code: ? // [id]* elements ? for (int index = 0; index < length; index++) { ??? oop o = array->obj_at(index); >> ??? if (o != NULL && mask_dormant_archived_object(o) == NULL) { ????? ResourceMark rm; ????? tty->print_cr("%s array contains %s object", array->klass()->external_name(), o->klass()->external_name()); ??? } >> ??? o = mask_dormant_archived_object(o); ??? writer->write_objectID(o); ? } and the output is: $ java -cp .? -XX:+HeapDumpAfterFullGC HelloGC Dumping heap to java_pid20956.hprof ... [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object (repeated about 20 times) It comes from java/util/jar/Attributes$Name::KNOWN_NAMES. This class is not loaded because my program doesn't use JAR files in the classpath: Thanks - Ioi > Regards, > Jiangli > > > >> So to answer your question, we can have a case with a dormant array >> (that contains a dormant object) like this: >> >> Object[] array = {new ClassNotYetLoaded();} >> >> After my fix, the array will be dumped (we have no easy way of not doing >> that), but its contents becomes this in the .hprof file: >> >> Object[] array = {null} >> >> Thanks >> - Ioi >> >> >> >>>> Hi David, >>>> >>>> Thanks for the review. >>>> >>>> The dormant objects are not reachable via the GC roots. They become >>>> non-dormant via explicit calls to JVM_InitializeFromArchive, after >>>> which they become reachable via the static fields of loaded classes. >>> Right, so is there a distinction between non-dormant and reachable at >>> the time an object becomes non-dormant? I'm still unclear how a drmant >>> array becomes non-dormant but still contains elements that refer to >>> dormant objects. >>> >>>> The only issue here is heap dump is done by scanning all objects in >>>> the heap, including unreachable ones >>>> >>>> HeapObjectDumper obj_dumper(this, writer()); >>>> Universe::heap()->safe_object_iterate(&obj_dumper); >>>> >>>> that's how these dormant objects are discovered during heap dump. >>>> >>>>> That aside the code changes seem reasonable, you moved the check out >>>>> of DumperSupport::dump_instance and into the higher-level >>>>> HeapObjectDumper::do_object so that it catches instances and arrays, >>>>> plus you added a check for array elements. >>>>> >>>> I am debating whether I should put the masking code in here: >>>> >>>> void DumpWriter::write_objectID(oop o) { >>>> o = mask_dormant_archived_object(o); /// <---- add >>>> address a = (address)o; >>>> #ifdef _LP64 >>>> write_u8((u8)a); >>>> #else >>>> write_u4((u4)a); >>>> #endif >>>> } >>>> >>>> >>>> That way, even if a dormant object (unintentionally) becomes >>>> reachable via the GC roots, we won't write an invalid reference to it >>>> (the object "body" will not be written, so the ID will not point to >>>> anything valid). >>>> >>>> But this seems a little too aggressive to me. What do you think? >>> It does seem a little aggressive as it seems to introduce the dormancy >>> check into a lot of places that don't need it. But as I said I don't >>> know this code so I'm really not the right person to ask. >>> >>> Cheers, >>> David >>> ----- >>> >>>> Thanks >>>> - Ioi >>>> From jianglizhou at google.com Fri Sep 6 23:06:15 2019 From: jianglizhou at google.com (Jiangli Zhou) Date: Fri, 6 Sep 2019 16:06:15 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: <3a510f34-7d34-cf02-f834-7522dadb8caa@oracle.com> References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> <96a98c4d-6313-3176-538f-08043f807389@oracle.com> <3a510f34-7d34-cf02-f834-7522dadb8caa@oracle.com> Message-ID: On Fri, Sep 6, 2019 at 3:17 PM Ioi Lam wrote: > > On 9/6/19 11:48 AM, Jiangli Zhou wrote: > > On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam wrote: > >> > >> > >> On 9/5/19 11:11 PM, David Holmes wrote: > >>> On 6/09/2019 1:39 pm, Ioi Lam wrote: > >>>> On 9/5/19 8:18 PM, David Holmes wrote: > >>>>> Hi Ioi, > >>>>> > >>>>> On 6/09/2019 12:27 pm, Ioi Lam wrote: > >>>>>> https://bugs.openjdk.java.net/browse/JDK-8230674 > >>>>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 > >>>>>> > >>>>>> > >>>>>> Please review this small fix: > >>>>>> > >>>>>> When CDS is in use, archived objects are memory-mapped into the > >>>>>> heap (currently G1GC only). These objects are partitioned into > >>>>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those > >>>>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is > >>>>>> requested. > > >>>>>> When a subgraph is not loaded, some of the objects in this subgraph > >>>>>> may belong to a class that's not yet loaded. > >>>>>> > >>>>>> The bug happens when such an "dormant" object is dumped, but its class > >>>>>> is not dumped because the class is not in the system dictionary. > >>>>>> > >>>>>> There is already code in DumperSupport::dump_instance() that tries > >>>>>> to handle dormant objects, but it needs to be extended to cover > >>>>>> arrays, as well as and references from non-dormant object/arrays to > >>>>>> dormant ones. > >>>>> I have to confess I did not pay any attention to the CDS archived > >>>>> objects work, so I don't have a firm grasp of how you have > >>>>> implemented things. But I'm wondering how can you have a reference > >>>>> to a dormant object from a non-dormant one? Shouldn't the act of > >>>>> becoming non-dormant automatically cause the subgraph from that > >>>>> object to also become non-dormant? Or do you have "read barriers" to > >>>>> perform the changes on demand? > >>>>> > >> Ah -- my bug title is not correct. > >> > >> I changed the bug title (and this e-mail subject) to > >> > >> Heap dumps should exclude dormant CDS archived objects **of unloaded > >> classes** > >> > >> During the heap dump, we scan all objects in the heap, regardless of > >> reachability. There's no way to decide reachability in > >> HeapObjectDumper::do_object(), unless we perform an actual GC. > >> > >> But it's OK to include unreachable objects in the heap dump. (I guess > >> it's useful to see how much garbage you have in the heap. There's an > >> option to run a collection before dumping the heap.) > >> > >> There are 2 kinds of unreachable objects -- garbage: those that were > >> once reachable but no longer, dormant: the archived objects that have > >> never been reachable. > > Currently Java object archiving framework only supports one > > directional state change: dormant -> live. An archived object can > > become a live object from dormant state, but it cannot go back to the > > dormant state. Need to investigate thoroughly for all cases before the > > 'live -> dormant' transition can be supported. All objects in the > > 'Open' archive heap region are associated with the builtin class > > loaders and their classes are not unloaded. The existing static fields > > for archiving within the JDK classes are selected and the associated > > objects do not become garbage once 'installed'. > > > >> Anyway, it's OK to dump dormant objects as long as their class has been > >> loaded. The problem happens only when we dump a dormant object who class > >> is not yet loaded (Eclipase MAT get confused when it sees an object > >> whose class ID is invalid). > > Yes. That's a scenario needs to be handled for a tool that iterates > > the Java heap. A dormant object in the 'Open' archive heap region may > > have a 'invalid' klass since the klass may not be loaded yet at the > > moment. > > > > Your webrev looks reasonable to me on high level pending information > > for following questions. Can you please give more details on the > > dormant objects referenced from the arrays? What specific arrays are > > those? > Hi Jiangli, > > Thanks for the review. I add the following code: > > // [id]* elements > for (int index = 0; index < length; index++) { > oop o = array->obj_at(index); > >> > if (o != NULL && mask_dormant_archived_object(o) == NULL) { > ResourceMark rm; > tty->print_cr("%s array contains %s object", > array->klass()->external_name(), o->klass()->external_name()); > } > >> > o = mask_dormant_archived_object(o); > writer->write_objectID(o); > } > > and the output is: > > $ java -cp . -XX:+HeapDumpAfterFullGC HelloGC > Dumping heap to java_pid20956.hprof ... > [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object > [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object > (repeated about 20 times) > > It comes from java/util/jar/Attributes$Name::KNOWN_NAMES. This class is > not loaded because my program doesn't use JAR files in the classpath: The above looks right and is expected. At this point, we would not see any archive object that first becomes live then becomes dormant again. That however will change in the future when we make the object archiving framework general enough for other JDK and application class usages. We need to work out the GC details for the live -> dormant transition when that happens. Thanks, Jiangli > > Thanks > - Ioi > > > > Regards, > > Jiangli > > > > > > > >> So to answer your question, we can have a case with a dormant array > >> (that contains a dormant object) like this: > >> > >> Object[] array = {new ClassNotYetLoaded();} > >> > >> After my fix, the array will be dumped (we have no easy way of not doing > >> that), but its contents becomes this in the .hprof file: > >> > >> Object[] array = {null} > >> > >> Thanks > >> - Ioi > >> > >> > >> > >>>> Hi David, > >>>> > >>>> Thanks for the review. > >>>> > >>>> The dormant objects are not reachable via the GC roots. They become > >>>> non-dormant via explicit calls to JVM_InitializeFromArchive, after > >>>> which they become reachable via the static fields of loaded classes. > >>> Right, so is there a distinction between non-dormant and reachable at > >>> the time an object becomes non-dormant? I'm still unclear how a drmant > >>> array becomes non-dormant but still contains elements that refer to > >>> dormant objects. > >>> > >>>> The only issue here is heap dump is done by scanning all objects in > >>>> the heap, including unreachable ones > >>>> > >>>> HeapObjectDumper obj_dumper(this, writer()); > >>>> Universe::heap()->safe_object_iterate(&obj_dumper); > >>>> > >>>> that's how these dormant objects are discovered during heap dump. > >>>> > >>>>> That aside the code changes seem reasonable, you moved the check out > >>>>> of DumperSupport::dump_instance and into the higher-level > >>>>> HeapObjectDumper::do_object so that it catches instances and arrays, > >>>>> plus you added a check for array elements. > >>>>> > >>>> I am debating whether I should put the masking code in here: > >>>> > >>>> void DumpWriter::write_objectID(oop o) { > >>>> o = mask_dormant_archived_object(o); /// <---- add > >>>> address a = (address)o; > >>>> #ifdef _LP64 > >>>> write_u8((u8)a); > >>>> #else > >>>> write_u4((u4)a); > >>>> #endif > >>>> } > >>>> > >>>> > >>>> That way, even if a dormant object (unintentionally) becomes > >>>> reachable via the GC roots, we won't write an invalid reference to it > >>>> (the object "body" will not be written, so the ID will not point to > >>>> anything valid). > >>>> > >>>> But this seems a little too aggressive to me. What do you think? > >>> It does seem a little aggressive as it seems to introduce the dormancy > >>> check into a lot of places that don't need it. But as I said I don't > >>> know this code so I'm really not the right person to ask. > >>> > >>> Cheers, > >>> David > >>> ----- > >>> > >>>> Thanks > >>>> - Ioi > >>>> > From serguei.spitsyn at oracle.com Sat Sep 7 01:08:53 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 6 Sep 2019 18:08:53 -0700 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: <4f7232e2-4a3b-f3a1-965a-830e357f7534@oracle.com> Hi Richard, This looks good to me. One question though. Suppose some methods got compiled/optimized with the escape analysis. Then nothing prevents to enable the JVMTI capabilities later and use the JVMTI functions GetOwnedMonitorInfo()and GetOwnedMonitorStackDepthInfo(). Should enabling the capabilities cause deoptimization of the already optimized compiled frames? Is this considered to be a part of the JDK-8227745? Thanks, Serguei On 9/6/19 7:24 AM, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape analysis those references escape and > optimizations like scalar replacement become invalid. > > The runtime currently cannot cope with objects escaping through JVMTI (try included > tests). Therefore escape analysis should be disabled if an agent requests the capabilities > can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 From serguei.spitsyn at oracle.com Sun Sep 8 07:12:06 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Sun, 8 Sep 2019 00:12:06 -0700 Subject: RFR: JDK-8230516: invalid html in jdwp-protocol.html In-Reply-To: References: <0E3B211D-41F1-429F-8AFC-0BE3A5750B2C@oracle.com> Message-ID: Hi Alex, +1 Thanks, Serguei On 9/6/19 14:14, Daniil Titov wrote: > Hi Alex, > > The change looks good to me. > > Thanks! > > -Daniil > > ?On 9/6/19, 12:59 PM, "serviceability-dev on behalf of Alex Menkov" wrote: > > Hi all, > > Please review the fix for > https://bugs.openjdk.java.net/browse/JDK-8230516 > webrev: > > http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/webrev/ > > The fix moves "anchor" spans to inside the cells. > > generated docs (no visual changes): > old: > > http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/0/jdwp-protocol.html > new: > > http://cr.openjdk.java.net/~amenkov/jdk14/jdwp-protocol_table_span/1/jdwp-protocol.html > > --alex > > > > From kirk.pepperdine at gmail.com Sun Sep 8 16:21:37 2019 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Sun, 8 Sep 2019 09:21:37 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> <96a98c4d-6313-3176-538f-08043f807389@oracle.com> <3a510f34-7d34-cf02-f834-7522dadb8caa@oracle.com> Message-ID: Hi, Might I add a diagnostic twist to this request. It is sometimes useful to try to determine where unreferenced objects live in heap because this can help you solve questions of nepotism. Kind regards, Kirk > On Sep 6, 2019, at 4:06 PM, Jiangli Zhou wrote: > > On Fri, Sep 6, 2019 at 3:17 PM Ioi Lam wrote: >> >> On 9/6/19 11:48 AM, Jiangli Zhou wrote: >>> On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam wrote: >>>> >>>> >>>> On 9/5/19 11:11 PM, David Holmes wrote: >>>>> On 6/09/2019 1:39 pm, Ioi Lam wrote: >>>>>> On 9/5/19 8:18 PM, David Holmes wrote: >>>>>>> Hi Ioi, >>>>>>> >>>>>>> On 6/09/2019 12:27 pm, Ioi Lam wrote: >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8230674 >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >>>>>>>> >>>>>>>> >>>>>>>> Please review this small fix: >>>>>>>> >>>>>>>> When CDS is in use, archived objects are memory-mapped into the >>>>>>>> heap (currently G1GC only). These objects are partitioned into >>>>>>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >>>>>>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >>>>>>>> requested. > >>>>>>>> When a subgraph is not loaded, some of the objects in this subgraph >>>>>>>> may belong to a class that's not yet loaded. >>>>>>>> >>>>>>>> The bug happens when such an "dormant" object is dumped, but its class >>>>>>>> is not dumped because the class is not in the system dictionary. >>>>>>>> >>>>>>>> There is already code in DumperSupport::dump_instance() that tries >>>>>>>> to handle dormant objects, but it needs to be extended to cover >>>>>>>> arrays, as well as and references from non-dormant object/arrays to >>>>>>>> dormant ones. >>>>>>> I have to confess I did not pay any attention to the CDS archived >>>>>>> objects work, so I don't have a firm grasp of how you have >>>>>>> implemented things. But I'm wondering how can you have a reference >>>>>>> to a dormant object from a non-dormant one? Shouldn't the act of >>>>>>> becoming non-dormant automatically cause the subgraph from that >>>>>>> object to also become non-dormant? Or do you have "read barriers" to >>>>>>> perform the changes on demand? >>>>>>> >>>> Ah -- my bug title is not correct. >>>> >>>> I changed the bug title (and this e-mail subject) to >>>> >>>> Heap dumps should exclude dormant CDS archived objects **of unloaded >>>> classes** >>>> >>>> During the heap dump, we scan all objects in the heap, regardless of >>>> reachability. There's no way to decide reachability in >>>> HeapObjectDumper::do_object(), unless we perform an actual GC. >>>> >>>> But it's OK to include unreachable objects in the heap dump. (I guess >>>> it's useful to see how much garbage you have in the heap. There's an >>>> option to run a collection before dumping the heap.) >>>> >>>> There are 2 kinds of unreachable objects -- garbage: those that were >>>> once reachable but no longer, dormant: the archived objects that have >>>> never been reachable. >>> Currently Java object archiving framework only supports one >>> directional state change: dormant -> live. An archived object can >>> become a live object from dormant state, but it cannot go back to the >>> dormant state. Need to investigate thoroughly for all cases before the >>> 'live -> dormant' transition can be supported. All objects in the >>> 'Open' archive heap region are associated with the builtin class >>> loaders and their classes are not unloaded. The existing static fields >>> for archiving within the JDK classes are selected and the associated >>> objects do not become garbage once 'installed'. >>> >>>> Anyway, it's OK to dump dormant objects as long as their class has been >>>> loaded. The problem happens only when we dump a dormant object who class >>>> is not yet loaded (Eclipase MAT get confused when it sees an object >>>> whose class ID is invalid). >>> Yes. That's a scenario needs to be handled for a tool that iterates >>> the Java heap. A dormant object in the 'Open' archive heap region may >>> have a 'invalid' klass since the klass may not be loaded yet at the >>> moment. >>> >>> Your webrev looks reasonable to me on high level pending information >>> for following questions. Can you please give more details on the >>> dormant objects referenced from the arrays? What specific arrays are >>> those? >> Hi Jiangli, >> >> Thanks for the review. I add the following code: >> >> // [id]* elements >> for (int index = 0; index < length; index++) { >> oop o = array->obj_at(index); >>>> >> if (o != NULL && mask_dormant_archived_object(o) == NULL) { >> ResourceMark rm; >> tty->print_cr("%s array contains %s object", >> array->klass()->external_name(), o->klass()->external_name()); >> } >>>> >> o = mask_dormant_archived_object(o); >> writer->write_objectID(o); >> } >> >> and the output is: >> >> $ java -cp . -XX:+HeapDumpAfterFullGC HelloGC >> Dumping heap to java_pid20956.hprof ... >> [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object >> [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object >> (repeated about 20 times) >> >> It comes from java/util/jar/Attributes$Name::KNOWN_NAMES. This class is >> not loaded because my program doesn't use JAR files in the classpath: > > The above looks right and is expected. At this point, we would not see > any archive object that first becomes live then becomes dormant again. > > That however will change in the future when we make the object > archiving framework general enough for other JDK and application class > usages. We need to work out the GC details for the live -> dormant > transition when that happens. > > Thanks, > Jiangli > >> >> Thanks >> - Ioi >> >> >>> Regards, >>> Jiangli >>> >>> >>> >>>> So to answer your question, we can have a case with a dormant array >>>> (that contains a dormant object) like this: >>>> >>>> Object[] array = {new ClassNotYetLoaded();} >>>> >>>> After my fix, the array will be dumped (we have no easy way of not doing >>>> that), but its contents becomes this in the .hprof file: >>>> >>>> Object[] array = {null} >>>> >>>> Thanks >>>> - Ioi >>>> >>>> >>>> >>>>>> Hi David, >>>>>> >>>>>> Thanks for the review. >>>>>> >>>>>> The dormant objects are not reachable via the GC roots. They become >>>>>> non-dormant via explicit calls to JVM_InitializeFromArchive, after >>>>>> which they become reachable via the static fields of loaded classes. >>>>> Right, so is there a distinction between non-dormant and reachable at >>>>> the time an object becomes non-dormant? I'm still unclear how a drmant >>>>> array becomes non-dormant but still contains elements that refer to >>>>> dormant objects. >>>>> >>>>>> The only issue here is heap dump is done by scanning all objects in >>>>>> the heap, including unreachable ones >>>>>> >>>>>> HeapObjectDumper obj_dumper(this, writer()); >>>>>> Universe::heap()->safe_object_iterate(&obj_dumper); >>>>>> >>>>>> that's how these dormant objects are discovered during heap dump. >>>>>> >>>>>>> That aside the code changes seem reasonable, you moved the check out >>>>>>> of DumperSupport::dump_instance and into the higher-level >>>>>>> HeapObjectDumper::do_object so that it catches instances and arrays, >>>>>>> plus you added a check for array elements. >>>>>>> >>>>>> I am debating whether I should put the masking code in here: >>>>>> >>>>>> void DumpWriter::write_objectID(oop o) { >>>>>> o = mask_dormant_archived_object(o); /// <---- add >>>>>> address a = (address)o; >>>>>> #ifdef _LP64 >>>>>> write_u8((u8)a); >>>>>> #else >>>>>> write_u4((u4)a); >>>>>> #endif >>>>>> } >>>>>> >>>>>> >>>>>> That way, even if a dormant object (unintentionally) becomes >>>>>> reachable via the GC roots, we won't write an invalid reference to it >>>>>> (the object "body" will not be written, so the ID will not point to >>>>>> anything valid). >>>>>> >>>>>> But this seems a little too aggressive to me. What do you think? >>>>> It does seem a little aggressive as it seems to introduce the dormancy >>>>> check into a lot of places that don't need it. But as I said I don't >>>>> know this code so I'm really not the right person to ask. >>>>> >>>>> Cheers, >>>>> David >>>>> ----- >>>>> >>>>>> Thanks >>>>>> - Ioi >>>>>> >> From david.holmes at oracle.com Mon Sep 9 02:15:38 2019 From: david.holmes at oracle.com (David Holmes) Date: Mon, 9 Sep 2019 12:15:38 +1000 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: Hi Dan, On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: > Hi David, > > I've finally gotten back to this email thread... Thanks. >> FYI testing to date: >> ?- tiers 1 -3 all platforms >> ?- hotspot: serviceability/jvmti >> ????????????????????????? /jdwp >> ??????????? vmTestbase/nsk/jvmti >> ????????????????????????? /jdwp >> ?- JDK: com/sun/jdi > > You should also add: > > open/test/hotspot/jtreg/vmTestbase/nsk/jdb > open/test/hotspot/jtreg/vmTestbase/nsk/jdi > open/test/jdk/java/lang/instrument Okay - in progress. Though I can't see any use of RawMonitors in any of these tests. > I took a quick look through the preliminary webrev and I don't see > anything that worries me. Thanks. I'll prepare a more polished webrev soon. > Re: Thread.interrupt() and raw_wait() > > It would be good to see if that semantic is being tested via the > JCK test suite for JVM/TI. It isn't. The only thing directly tested for RawMonitorWait is normal successful operation and reporting "not owner" when not the owner. No check for JVMTI_ERROR_INTERRUPT exists other than as input for the GetErrorName function. There's only one test in the whole test base that checks for the interrupt and that is vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if we are not interrupted before the RawMonitorWait we will wait until the full timeout elapses - which is 2 minutes by default - then return and report the interrupt. Hence the test still passes. (If it was an untimed wait that would be different of course). The more I try to convince people this change should be okay, the more uncomfortable I get with my own arguments. :) I think I'm going to implement the polling approach for checking interrupts - say 500ms. > I also very much like/appreciate the decoupling of JvmtiRawMonitors > from ObjectMonitors... Thanks for tackling this crazy task. Thanks :) David > Dan > > > > On 8/15/19 2:22 AM, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >> >> Preliminary webrev (still has rough edges): >> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >> >> Background: >> >> We've had this comment for a long time: >> >> ?// The raw monitor subsystem is entirely distinct from normal >> ?// java-synchronization or jni-synchronization.? raw monitors are not >> ?// associated with objects.? They can be implemented in any manner >> ?// that makes sense.? The original implementors decided to piggy-back >> ?// the raw-monitor implementation on the existing Java objectMonitor >> mechanism. >> ?// This flaw needs to fixed.? We should reimplement raw monitors as >> sui-generis. >> ?// Specifically, we should not implement raw monitors via java monitors. >> ?// Time permitting, we should disentangle and deconvolve the two >> implementations >> ?// and move the resulting raw monitor implementation over to the >> JVMTI directories. >> ?// Ideally, the raw monitor implementation would be built on top of >> ?// park-unpark and nothing else. >> >> This is an attempt to do that disentangling so that we can then >> consider changes to ObjectMonitor without having to worry about >> JvmtiRawMonitors. But rather than building on low-level park/unpark >> (which would require the same manual queue management and much of the >> same complex code as exists in ObjectMonitor) I decided to try and do >> this on top of PlatformMonitor. >> >> The reason this is just a RFC rather than RFR is that I overlooked a >> non-trivial aspect of JvmtiRawMonitors: like Java monitors (as >> implemented by ObjectMonitor) they interact with the Thread.interrupt >> mechanism. This is not clearly stated in the JVM TI specification [1] >> but only in passing by the possible errors for RawMonitorWait: >> >> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >> >> As I explain in the bug report there is no way to build in proper >> interrupt support using PlatformMonitor as there is no way we can >> "interrupt" the low-level pthread_cond_wait. But we can approximate >> it. What I've done in this preliminary version is just check interrupt >> state before and after the actual "wait" but we won't get woken by the >> interrupt once we have actually blocked. Alternatively we could use a >> periodic polling approach and wakeup every Nms to check for interruption. >> >> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is not >> affected by this choice as that code ignores the interrupt until the >> real action it was waiting for has occurred. The interrupt is then >> reposted later. >> >> But more generally there could be users of JvmtiRawMonitors that >> expect/require that RawMonitorWait is responsive to Thread.interrupt >> in a manner similar to Object.wait. And if any of them are reading >> this then I'd like to know - hence this RFC :) >> >> FYI testing to date: >> ?- tiers 1 -3 all platforms >> ?- hotspot: serviceability/jvmti >> ????????????????????????? /jdwp >> ??????????? vmTestbase/nsk/jvmti >> ????????????????????????? /jdwp >> ?- JDK: com/sun/jdi >> >> Comments/opinions appreciated. >> >> Thanks, >> David >> >> [1] >> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >> > From richard.reingruber at sap.com Mon Sep 9 09:07:44 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Mon, 9 Sep 2019 09:07:44 +0000 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Alex, > Of course error can be cleared before each try - there is not functional > difference. It is just a little confusing, as you can get an exception in L. 95, too. But I'm ok with it, if you prefer it like this. I would suggest, though, to sleep some ms before a retry and double the sleep time in each following retry. Best regards, Richard. -----Original Message----- From: Alex Menkov Sent: Freitag, 6. September 2019 22:52 To: Reingruber, Richard ; serguei.spitsyn at oracle.com; OpenJDK Serviceability Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException Hi Richard, On 09/06/2019 09:28, Reingruber, Richard wrote: > Hi Alex, > > that's a good fix for the issue. > > One minor thing: > > 89 Exception error = null; > 90 for (int retry = 0; retry < 5; retry++) { > 91 try { > 92 log("retry: " + retry); > 93 s = new Socket("localhost", port); > 94 error = null; > 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); > 96 break; > 97 } catch (ConnectException ex) { > 98 log("got exception: " + ex.toString()); > 99 error = ex; > 100 } > 101 } > 102 if (error != null) { > 103 throw error; > 104 } > > Is there a reason to clear the local variable error in line 94 instead of clearing it > in line 91 where each new attempt begins? The logic here is: The cycle has 2 exits: - error (max retry attempts reached, error is set by last "catch") - success ("break" statement at line 96, error should be null) So error is cleared only after the socket is connected (this is the problematic operation which can cause ConnectException). Of course error can be cleared before each try - there is not functional difference. --alex > > Cheers, Richard. > > -----Original Message----- > From: serviceability-dev On Behalf Of serguei.spitsyn at oracle.com > Sent: Mittwoch, 4. September 2019 22:11 > To: Alex Menkov ; OpenJDK Serviceability > Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException > > Hi Alex, > > The fix looks good. > Good simplification! > > Thanks, > Serguei > > > On 9/4/19 12:19, Alex Menkov wrote: >> Hi all, >> >> Please review the fix for BadHandshakeTest test. >> The problem is the test connects to the server twice and if debuggee >> hasn't yet handled disconnection, the next connect gets "connection >> refused" error. >> Instead of adding delay before 2nd connect (we never know "good" value >> for the delay and big delay can cause "accept timeout"), the test >> re-tries connect in case of ConnectException. >> Also improved/simplified the test slightly - debuggee is now run with >> auto port assignment (used lib.jdb.Debuggee test class which >> implements required functionality). >> >> jira: >> ? https://bugs.openjdk.java.net/browse/JDK-8192057 >> webrev: >> http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ >> >> --alex > From daniel.daugherty at oracle.com Mon Sep 9 15:29:59 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 9 Sep 2019 11:29:59 -0400 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: <0dd42545-847f-1bc8-3066-f5751cb0f57e@oracle.com> On 9/8/19 10:15 PM, David Holmes wrote: > Hi Dan, > > On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: >> Hi David, >> >> I've finally gotten back to this email thread... > > Thanks. > >>> FYI testing to date: >>> ?- tiers 1 -3 all platforms >>> ?- hotspot: serviceability/jvmti >>> ????????????????????????? /jdwp >>> ??????????? vmTestbase/nsk/jvmti >>> ????????????????????????? /jdwp >>> ?- JDK: com/sun/jdi >> >> You should also add: >> >> open/test/hotspot/jtreg/vmTestbase/nsk/jdb >> open/test/hotspot/jtreg/vmTestbase/nsk/jdi >> open/test/jdk/java/lang/instrument > > Okay - in progress. Though I can't see any use of RawMonitors in any > of these tests. I wouldn't expect direct use. I would expect built-on-top-of use. In particular, there are scenario tests in ndk/jdi that use JDI in different ways than straight up API tests and shake out bugs. > >> I took a quick look through the preliminary webrev and I don't see >> anything that worries me. > > Thanks. I'll prepare a more polished webrev soon. > >> Re: Thread.interrupt() and raw_wait() >> >> It would be good to see if that semantic is being tested via the >> JCK test suite for JVM/TI. > > It isn't. The only thing directly tested for RawMonitorWait is normal > successful operation and reporting "not owner" when not the owner. No > check for JVMTI_ERROR_INTERRUPT exists other than as input for the > GetErrorName function. > > There's only one test in the whole test base that checks for the > interrupt and that is > vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if we > are not interrupted before the RawMonitorWait we will wait until the > full timeout elapses - which is 2 minutes by default - then return and > report the interrupt. Hence the test still passes. (If it was an > untimed wait that would be different of course). > > The more I try to convince people this change should be okay, the more > uncomfortable I get with my own arguments. :) I think I'm going to > implement the polling approach for checking interrupts - say 500ms. I'll keep an eye open for the update... Dan > >> I also very much like/appreciate the decoupling of JvmtiRawMonitors >> from ObjectMonitors... Thanks for tackling this crazy task. > > Thanks :) > > David > >> Dan >> >> >> >> On 8/15/19 2:22 AM, David Holmes wrote: >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >>> >>> Preliminary webrev (still has rough edges): >>> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >>> >>> Background: >>> >>> We've had this comment for a long time: >>> >>> ?// The raw monitor subsystem is entirely distinct from normal >>> ?// java-synchronization or jni-synchronization.? raw monitors are not >>> ?// associated with objects.? They can be implemented in any manner >>> ?// that makes sense.? The original implementors decided to piggy-back >>> ?// the raw-monitor implementation on the existing Java >>> objectMonitor mechanism. >>> ?// This flaw needs to fixed.? We should reimplement raw monitors as >>> sui-generis. >>> ?// Specifically, we should not implement raw monitors via java >>> monitors. >>> ?// Time permitting, we should disentangle and deconvolve the two >>> implementations >>> ?// and move the resulting raw monitor implementation over to the >>> JVMTI directories. >>> ?// Ideally, the raw monitor implementation would be built on top of >>> ?// park-unpark and nothing else. >>> >>> This is an attempt to do that disentangling so that we can then >>> consider changes to ObjectMonitor without having to worry about >>> JvmtiRawMonitors. But rather than building on low-level park/unpark >>> (which would require the same manual queue management and much of >>> the same complex code as exists in ObjectMonitor) I decided to try >>> and do this on top of PlatformMonitor. >>> >>> The reason this is just a RFC rather than RFR is that I overlooked a >>> non-trivial aspect of JvmtiRawMonitors: like Java monitors (as >>> implemented by ObjectMonitor) they interact with the >>> Thread.interrupt mechanism. This is not clearly stated in the JVM TI >>> specification [1] but only in passing by the possible errors for >>> RawMonitorWait: >>> >>> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >>> >>> As I explain in the bug report there is no way to build in proper >>> interrupt support using PlatformMonitor as there is no way we can >>> "interrupt" the low-level pthread_cond_wait. But we can approximate >>> it. What I've done in this preliminary version is just check >>> interrupt state before and after the actual "wait" but we won't get >>> woken by the interrupt once we have actually blocked. Alternatively >>> we could use a periodic polling approach and wakeup every Nms to >>> check for interruption. >>> >>> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is not >>> affected by this choice as that code ignores the interrupt until the >>> real action it was waiting for has occurred. The interrupt is then >>> reposted later. >>> >>> But more generally there could be users of JvmtiRawMonitors that >>> expect/require that RawMonitorWait is responsive to Thread.interrupt >>> in a manner similar to Object.wait. And if any of them are reading >>> this then I'd like to know - hence this RFC :) >>> >>> FYI testing to date: >>> ?- tiers 1 -3 all platforms >>> ?- hotspot: serviceability/jvmti >>> ????????????????????????? /jdwp >>> ??????????? vmTestbase/nsk/jvmti >>> ????????????????????????? /jdwp >>> ?- JDK: com/sun/jdi >>> >>> Comments/opinions appreciated. >>> >>> Thanks, >>> David >>> >>> [1] >>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>> >> From adam.farley at uk.ibm.com Mon Sep 9 16:53:50 2019 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Mon, 9 Sep 2019 17:53:50 +0100 Subject: RFR: 8229378: jdwp library loader in linker_md.c quietly truncates on buffer overflow In-Reply-To: <3ea2e6ad-a32d-603c-258c-985da4e2f50a@oracle.com> References: <9bab421c-08a1-b554-6ac9-35290856ee56@oracle.com> <0c01c46b-3ff9-3016-8791-868019459d13@oracle.com> <3ea2e6ad-a32d-603c-258c-985da4e2f50a@oracle.com> Message-ID: Hi Serguei, Apologies for the delay. The errors have all been fixed, and the requested tests mostly passed, windows and linux. No test group had more failures post-fix than pre-fix, so I'm calling that a win. The new webrev can be found here: http://cr.openjdk.java.net/~afarley/8229378.2/webrev Best Regards Adam Farley IBM Runtimes "serguei.spitsyn at oracle.com" wrote on 29/08/2019 19:38:02: > From: "serguei.spitsyn at oracle.com" > To: Adam Farley8 > Cc: Chris Plummer , > daniel.daugherty at oracle.com, serviceability-dev at openjdk.java.net > Date: 29/08/2019 19:38 > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > quietly truncates on buffer overflow > > Hi Adam, > > Okay, thanks! > Serguei > > > On 8/29/19 06:26, Adam Farley8 wrote: > Hi Serguei, > > I haven't actually run a fastdebug build before. Will do that now > and address the issues. > > Once done, I'll re-run the tests I ran, and also the tests you've > listed below. > > Can you advise on how "good coverage" is determined, so I know for > future bug fixes? > > As for the up-to-date-ness, I'll update the build before doing the above. > > Expect a webrev once all of this is complete. > > Best Regards > > Adam Farley > IBM Runtimes > > > "serguei.spitsyn at oracle.com" wrote on > 29/08/2019 03:54:56: > > > From: "serguei.spitsyn at oracle.com" > > To: Adam Farley8 > > Cc: Chris Plummer , > > daniel.daugherty at oracle.com, serviceability-dev at openjdk.java.net > > Date: 29/08/2019 04:23 > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > quietly truncates on buffer overflow > > > > Hi Adam, > > > > Sorry for the latency. > > I was in process to build, test and push your fix and got the > > fastdebug build errors below. > > > > So, my question is if you've ever built the fastdebug version. > > This change is in the system-dependent code, so it has to be tested > > on both Unix and Windows. > > > > > My testing was limited to the bug specific test case I mentioned, > > and the following jdwp tests: > > > > > > test/jdk/com/sun/jdi/Jdwp* > > > test/hotspot/jtreg/serviceability/jdwp > > > > This set of tests does not provide a good coverage. > > To make sure nothing is broken you need to run the the test/jdk/com/sun/jdi > > and also the following vmTestbase tests: > > > > test/hotspot/jtreg/vmTestbase/nsk/jdi > > test/hotspot/jtreg/vmTestbase/nsk/jdb > > test/hotspot/jtreg/vmTestbase/nsk/jdwp > > > > BTW, your current webrev is not up-to-date: > > http://cr.openjdk.java.net/~afarley/8229378/webrev/ > > > > I guess, the change in the src/hotspot/share/runtime/os.cpp became > > obsolete after your previous fix that was already pushed. > > > > Thanks, > > Serguei > > > > . . . > > In file included from /scratch/sspitsyn/jdk14.1/open/src/ > > jdk.jdwp.agent/unix/native/libjdwp/linker_md.c:37:0: > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > libjdwp/linker_md.c: In function ?dll_build_name?: > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > libjdwp/util.h:46:23: error: ?Do? undeclared (first use in this function) > > #define strdup(p) Do not use this interface. > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > paths_copy = strdup(paths); > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > libjdwp/util.h:46:23: note: each undeclared identifier is reported > > only once for each function it appears in > > #define strdup(p) Do not use this interface. > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > paths_copy = strdup(paths); > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > libjdwp/util.h:46:26: error: expected ?;? before ?not? > > #define strdup(p) Do not use this interface. > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > paths_copy = strdup(paths); > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > libjdwp/util.h:38:24: error: expected ?;? before ?not? > > #define free(p) Do not use this interface. > > ^ > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > libjdwp/linker_md.c:71:5: note: in expansion of macro ?free? > > free(paths_copy); > > ^ > > gmake[3]: *** [/scratch/sspitsyn/jdk14.1/build/linux-x86_64-server- > > fastdebug/support/native/jdk.jdwp.agent/libjdwp/linker_md.o] Error 1 > > gmake[2]: *** [jdk.jdwp.agent-libs] Error 1 > > gmake[2]: *** Waiting for unfinished jobs.... > > > > ERROR: Build failed for target 'images' in configuration 'linux- > > x86_64-server-fastdebug' (exit code 2) > > > > > > > > On 8/13/19 09:28, Adam Farley8 wrote: > > Hi Serguei, Daniel, > > > > My testing was limited to the bug specific test case I mentioned, > > and the following jdwp tests: > > > > test/jdk/com/sun/jdi/Jdwp* > > test/hotspot/jtreg/serviceability/jdwp > > > > Best Regards > > > > Adam Farley > > IBM Runtimes > > > > > > "serguei.spitsyn at oracle.com" wrote on > > 13/08/2019 17:04:43: > > > > > From: "serguei.spitsyn at oracle.com" > > > To: daniel.daugherty at oracle.com, Adam Farley8 > > > , Chris Plummer > > > Cc: serviceability-dev at openjdk.java.net > > > Date: 13/08/2019 17:08 > > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > > quietly truncates on buffer overflow > > > > > > Hi Adam, > > > > > > I'm looking at your fix. > > > Also interested about your testing. > > > > > > Thanks, > > > Serguei > > > > > > On 8/13/19 08:48, Daniel D. Daugherty wrote: > > > I don't see any information about how this change was tested... > > > Is there something on another email thread? > > > > > > Dan > > > > > > > > On 8/13/19 11:41 AM, Adam Farley8 wrote: > > > Hi Chris, > > > > > > Thanks! > > > > > > I understand we need a second reviewer/sponsor to get this change > > > in. Any volunteers? > > > > > > Best Regards > > > > > > Adam Farley > > > IBM Runtimes > > > > > > > > > Chris Plummer wrote on 12/08/2019 21:35:06: > > > > > > > From: Chris Plummer > > > > To: Adam Farley8 , serviceability- > > > dev at openjdk.java.net > > > > Date: 12/08/2019 21:35 > > > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > > > quietly truncates on buffer overflow > > > > > > > > Hi Adam, > > > > > > > > It looks good to me. > > > > > > > > thanks, > > > > > > > > Chris > > > > > > > > On 8/12/19 7:34 AM, Adam Farley8 wrote: > > > > Hi All, > > > > > > > > This is a known bug, mentioned in a code comment. > > > > > > > > Here is the fix for that bug. > > > > > > > > Reviewers and sponsors requested. > > > > > > > > Short version: if you set sun.boot.library.path to > > > > something beyond a system's max path length, the > > > > current code will return an empty string (rather than > > > > printing a useful error message and shutting down). > > > > > > > > This is also a problem if you've specified multiple > > > > paths with a separator, as this code seems to wrongly > > > > assess whether the *total* length exceeds max path > > > > length. So two 200 char paths on windows will cause > > > > failure, as the total length is 400 (which is beyond > > > > max length for windows). > > > > > > > > Note that the os.cpp bit of the webrev will not be included > > > > in the final webrev, it just makes this change trivially > > > > testable. > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8229378 > > > > Webrev: http://cr.openjdk.java.net/~afarley/8229378/webrev/ > > > > > > > > > > > > Best Regards > > > > > > > > Adam Farley > > > > IBM Runtimes > > > > > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > > > number 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, > Hampshire PO6 3AU > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > > > number 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with > > number 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with > number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Mon Sep 9 18:26:37 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 9 Sep 2019 11:26:37 -0700 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Mon Sep 9 22:49:27 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 10 Sep 2019 08:49:27 +1000 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: Hi Serguei, On 10/09/2019 4:26 am, serguei.spitsyn at oracle.com wrote: > Hi David, > > On 9/8/19 19:15, David Holmes wrote: >> Hi Dan, >> >> On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: >>> Hi David, >>> >>> I've finally gotten back to this email thread... >> >> Thanks. >> >>>> FYI testing to date: >>>> ?- tiers 1 -3 all platforms >>>> ?- hotspot: serviceability/jvmti >>>> ????????????????????????? /jdwp >>>> ??????????? vmTestbase/nsk/jvmti >>>> ????????????????????????? /jdwp >>>> ?- JDK: com/sun/jdi >>> >>> You should also add: >>> >>> open/test/hotspot/jtreg/vmTestbase/nsk/jdb >>> open/test/hotspot/jtreg/vmTestbase/nsk/jdi >>> open/test/jdk/java/lang/instrument >> >> Okay - in progress. Though I can't see any use of RawMonitors in any >> of these tests. >> >>> I took a quick look through the preliminary webrev and I don't see >>> anything that worries me. >> >> Thanks. I'll prepare a more polished webrev soon. >> >>> Re: Thread.interrupt() and raw_wait() >>> >>> It would be good to see if that semantic is being tested via the >>> JCK test suite for JVM/TI. >> >> It isn't. The only thing directly tested for RawMonitorWait is normal >> successful operation and reporting "not owner" when not the owner. No >> check for JVMTI_ERROR_INTERRUPT exists other than as input for the >> GetErrorName function. > > This is most likely true. > My only concern is if RawMonitor's can be used in the JCK test libraries > (low probability). > I've asked Leonid Kuskov (JCK) to double check this (added to the > mailing list). > >> >> There's only one test in the whole test base that checks for the >> interrupt and that is >> vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if we >> are not interrupted before the RawMonitorWait we will wait until the >> full timeout elapses - which is 2 minutes by default - then return and >> report the interrupt. Hence the test still passes. (If it was an >> untimed wait that would be different of course). > > I figured the same last Friday. > One more place to care about are NSK tests libraries that are located here: > ? test/hotspot/jtreg/vmTestbase/nsk/share > > There are a couple of places where the RawMonitor is used: > ? jvmti/hotswap/HotSwap.cpp:??? if > (!NSK_JVMTI_VERIFY(jvmti->RawMonitorWait(waitLock, millis))) > ? jvmti/jvmti_tools.cpp:??? jvmtiError error = > env->RawMonitorWait(monitor, millis); > > The use in HotSwap.cpp is local. > The jvmti_tools.cpp defines this: > ? void rawMonitorWait(jvmtiEnv *env, jrawMonitorID monitor, jlong millis) { > ?? ?? jvmtiError error = env->RawMonitorWait(monitor, millis); > ?? ?? exitOnError(error); > ? } > > which is used in the jvmti/agent_tools.cpp but does not depend on > interrupting of RawMonitor's (as I can see). > One more place to mention is: > ? jvmti/DataDumpRequest/datadumpreq001/datadumpreq001.cpp > > But I see no problems there as well. > > > > The JDWP implementation is using RawMonitor's. > Please, see functions debugMonitorWait()/debugMonitorTimedWait() in > src/jdk.jdwp.agent/share/native/libjdwp/util.c. > > It expects the JVMTI_ERROR_INTERRUPT but never makes a call to the JVMTI > ThreadInterrupt(). > So, it looks like it does not depend on interrupting of RawMonitor's in > any way. > >> >> The more I try to convince people this change should be okay, the more >> uncomfortable I get with my own arguments. :) I think I'm going to >> implement the polling approach for checking interrupts - say 500ms. > > The JVMTI spec tells that the JVMTI_ERROR_INTERRUPT can be returned from > the RawMonitorWait: > https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait Yes it does and that is the only thing that implies a connection to Thread.interrupt. > which means that RawMonitorWait can be interrupted with the > Thread.Interrupt() > or JVMTI InterruptThread(): > https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#InterruptThread That's one way to interpret the fact that RawMonitorWait can return JVMTI_ERROR_INTERRUPT, but the actual interaction between Thread.interrupt and RawMonitorWait is not explicitly stated. Arguably you can just check for interruption before and after the wait, to see whether to return JVMTI_ERROR_INTERRUPT, without necessarily being able to break out of the wait itself. That's been the whole premise of this change proposal - that responsiveness to interrupts is more a quality-of-implementation issue. But in any case I've decided to try the polling approach so that we won't wait forever if interrupted but not notified. Thanks, David ----- > > > Thanks, > Serguei > >>> I also very much like/appreciate the decoupling of JvmtiRawMonitors >>> from ObjectMonitors... Thanks for tackling this crazy task. >> >> Thanks :) >> >> David >> >>> Dan >>> >>> >>> >>> On 8/15/19 2:22 AM, David Holmes wrote: >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >>>> >>>> Preliminary webrev (still has rough edges): >>>> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >>>> >>>> Background: >>>> >>>> We've had this comment for a long time: >>>> >>>> ?// The raw monitor subsystem is entirely distinct from normal >>>> ?// java-synchronization or jni-synchronization.? raw monitors are not >>>> ?// associated with objects.? They can be implemented in any manner >>>> ?// that makes sense.? The original implementors decided to piggy-back >>>> ?// the raw-monitor implementation on the existing Java >>>> objectMonitor mechanism. >>>> ?// This flaw needs to fixed.? We should reimplement raw monitors as >>>> sui-generis. >>>> ?// Specifically, we should not implement raw monitors via java >>>> monitors. >>>> ?// Time permitting, we should disentangle and deconvolve the two >>>> implementations >>>> ?// and move the resulting raw monitor implementation over to the >>>> JVMTI directories. >>>> ?// Ideally, the raw monitor implementation would be built on top of >>>> ?// park-unpark and nothing else. >>>> >>>> This is an attempt to do that disentangling so that we can then >>>> consider changes to ObjectMonitor without having to worry about >>>> JvmtiRawMonitors. But rather than building on low-level park/unpark >>>> (which would require the same manual queue management and much of >>>> the same complex code as exists in ObjectMonitor) I decided to try >>>> and do this on top of PlatformMonitor. >>>> >>>> The reason this is just a RFC rather than RFR is that I overlooked a >>>> non-trivial aspect of JvmtiRawMonitors: like Java monitors (as >>>> implemented by ObjectMonitor) they interact with the >>>> Thread.interrupt mechanism. This is not clearly stated in the JVM TI >>>> specification [1] but only in passing by the possible errors for >>>> RawMonitorWait: >>>> >>>> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >>>> >>>> As I explain in the bug report there is no way to build in proper >>>> interrupt support using PlatformMonitor as there is no way we can >>>> "interrupt" the low-level pthread_cond_wait. But we can approximate >>>> it. What I've done in this preliminary version is just check >>>> interrupt state before and after the actual "wait" but we won't get >>>> woken by the interrupt once we have actually blocked. Alternatively >>>> we could use a periodic polling approach and wakeup every Nms to >>>> check for interruption. >>>> >>>> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is not >>>> affected by this choice as that code ignores the interrupt until the >>>> real action it was waiting for has occurred. The interrupt is then >>>> reposted later. >>>> >>>> But more generally there could be users of JvmtiRawMonitors that >>>> expect/require that RawMonitorWait is responsive to Thread.interrupt >>>> in a manner similar to Object.wait. And if any of them are reading >>>> this then I'd like to know - hence this RFC :) >>>> >>>> FYI testing to date: >>>> ?- tiers 1 -3 all platforms >>>> ?- hotspot: serviceability/jvmti >>>> ????????????????????????? /jdwp >>>> ??????????? vmTestbase/nsk/jvmti >>>> ????????????????????????? /jdwp >>>> ?- JDK: com/sun/jdi >>>> >>>> Comments/opinions appreciated. >>>> >>>> Thanks, >>>> David >>>> >>>> [1] >>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>>> >>> > From serguei.spitsyn at oracle.com Mon Sep 9 23:11:49 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 9 Sep 2019 16:11:49 -0700 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: <79d37aa9-bd28-a8ab-6d4d-95facce87261@oracle.com> Hi David, On 9/9/19 15:49, David Holmes wrote: > Hi Serguei, . . . >> >> The JDWP implementation is using RawMonitor's. >> Please, see functions debugMonitorWait()/debugMonitorTimedWait() in >> src/jdk.jdwp.agent/share/native/libjdwp/util.c. >> >> It expects the JVMTI_ERROR_INTERRUPT but never makes a call to the >> JVMTI ThreadInterrupt(). >> So, it looks like it does not depend on interrupting of RawMonitor's >> in any way. >> >>> >>> The more I try to convince people this change should be okay, the >>> more uncomfortable I get with my own arguments. :) I think I'm going >>> to implement the polling approach for checking interrupts - say 500ms. >> >> The JVMTI spec tells that the JVMTI_ERROR_INTERRUPT can be returned >> from the RawMonitorWait: >> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >> > > Yes it does and that is the only thing that implies a connection to > Thread.interrupt. > >> which means that RawMonitorWait can be interrupted with the >> Thread.Interrupt() >> or JVMTI InterruptThread(): >> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#InterruptThread >> > > That's one way to interpret the fact that RawMonitorWait can return > JVMTI_ERROR_INTERRUPT, but the actual interaction between > Thread.interrupt and RawMonitorWait is not explicitly stated. Yes, it is true. The same is true about the actual interaction with the JVMTI InterruptThread(). But it seems, this function is expected to actually interrupt waiting on a RawMonitorWait() call or any other wait() calls. > Arguably you can just check for interruption before and after the > wait, to see whether to return JVMTI_ERROR_INTERRUPT, without > necessarily being able to break out of the wait itself. That's been > the whole premise of this change proposal - that responsiveness to > interrupts is more a quality-of-implementation issue. Right, it is another way to interpret it. > > But in any case I've decided to try the polling approach so that we > won't wait forever if interrupted but not notified. It sounds better. Thanks, Serguei > > Thanks, > David > ----- > >> Thanks, >> Serguei From ivan.gerasimov at oracle.com Tue Sep 10 03:41:58 2019 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 9 Sep 2019 20:41:58 -0700 Subject: RFR 8230303 : JDB hangs when running monitor command Message-ID: Hello! jdb utility has a command 'monitor ', which allows to execute the specified command every time the debuggee stops. If the modifies the list of installed monitors (the simplest example is 'monitor unmonitor 1'), then jdb hits a ConcurrentModificationException, and hangs the debuggee. While it is questionable, if modifying the monitor list has to be implemented in some specific way, it seems sensible to at least prevent a hard failure. The simplest fix appears to be to use CopyOnWriteArrayList, so that an immutable snapshot of the list will be traversed. Even if the list is modified, it wouldn't affect any iterator that might exist at that moment of time. BUGURL: https://bugs.openjdk.java.net/browse/JDK-8230303 WEBREV: http://cr.openjdk.java.net/~igerasim/8230303/00/webrev/ Mach5 control build was successfully run with hs-tier7-rt, which includes the new test and other jdb related tests. Would you please help review? Thanks in advance! -- With kind regards, Ivan Gerasimov From serguei.spitsyn at oracle.com Tue Sep 10 04:14:40 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 9 Sep 2019 21:14:40 -0700 Subject: RFR 8230303 : JDB hangs when running monitor command In-Reply-To: References: Message-ID: <18791c15-e6bb-04c0-a1bd-fe22b175560b@oracle.com> Hi Ivan, Thank you for filing the shadow bug and fixing this issue! I've targeted the bug to 14 (please, fix me if it is wrong). The fix looks good to me. Do you have any plans to backport it to the earlier releases? Thanks, Serguei On 9/9/19 20:41, Ivan Gerasimov wrote: > Hello! > > jdb utility has a command 'monitor ', which allows to execute > the specified command every time the debuggee stops. > > If the modifies the list of installed monitors (the simplest > example is 'monitor unmonitor 1'), then jdb hits a > ConcurrentModificationException, and hangs the debuggee. > > While it is questionable, if modifying the monitor list has to be > implemented in some specific way, it seems sensible to at least > prevent a hard failure. > > The simplest fix appears to be to use CopyOnWriteArrayList, so that an > immutable snapshot of the list will be traversed. > Even if the list is modified, it wouldn't affect any iterator that > might exist at that moment of time. > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8230303 > WEBREV: http://cr.openjdk.java.net/~igerasim/8230303/00/webrev/ > > Mach5 control build was successfully run with hs-tier7-rt, which > includes the new test and other jdb related tests. > > Would you please help review? > Thanks in advance! > From ivan.gerasimov at oracle.com Tue Sep 10 05:07:51 2019 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 9 Sep 2019 22:07:51 -0700 Subject: RFR 8230303 : JDB hangs when running monitor command In-Reply-To: <18791c15-e6bb-04c0-a1bd-fe22b175560b@oracle.com> References: <18791c15-e6bb-04c0-a1bd-fe22b175560b@oracle.com> Message-ID: Thank you Serguei for review! On 9/9/19 9:14 PM, serguei.spitsyn at oracle.com wrote: > Hi Ivan, > > Thank you for filing the shadow bug and fixing this issue! > I've targeted the bug to 14 (please, fix me if it is wrong). > Yes, this is correct, thanks. > The fix looks good to me. > > Do you have any plans to backport it to the earlier releases? > Yes, I'm planning to request a backport to the JDK 13 shortly. With kind regards, Ivan From ioi.lam at oracle.com Tue Sep 10 06:51:17 2019 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 9 Sep 2019 23:51:17 -0700 Subject: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes In-Reply-To: References: <1c6d0620-d0ce-cd62-1108-ce2ccdb692d8@oracle.com> <96a98c4d-6313-3176-538f-08043f807389@oracle.com> <3a510f34-7d34-cf02-f834-7522dadb8caa@oracle.com> Message-ID: <8e88644f-e9d9-d483-83a4-4736fcb4fcba@oracle.com> On 9/6/19 4:06 PM, Jiangli Zhou wrote: > On Fri, Sep 6, 2019 at 3:17 PM Ioi Lam wrote: >> On 9/6/19 11:48 AM, Jiangli Zhou wrote: >>> On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam wrote: >>>> >>>> On 9/5/19 11:11 PM, David Holmes wrote: >>>>> On 6/09/2019 1:39 pm, Ioi Lam wrote: >>>>>> On 9/5/19 8:18 PM, David Holmes wrote: >>>>>>> Hi Ioi, >>>>>>> >>>>>>> On 6/09/2019 12:27 pm, Ioi Lam wrote: >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8230674 >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01 >>>>>>>> >>>>>>>> >>>>>>>> Please review this small fix: >>>>>>>> >>>>>>>> When CDS is in use, archived objects are memory-mapped into the >>>>>>>> heap (currently G1GC only). These objects are partitioned into >>>>>>>> "subgraphs". Some of these subgraphs may not be loaded (e.g., those >>>>>>>> related to jdk.internal.math.FDBigInteger) at the time a heap dump is >>>>>>>> requested. > >>>>>>>> When a subgraph is not loaded, some of the objects in this subgraph >>>>>>>> may belong to a class that's not yet loaded. >>>>>>>> >>>>>>>> The bug happens when such an "dormant" object is dumped, but its class >>>>>>>> is not dumped because the class is not in the system dictionary. >>>>>>>> >>>>>>>> There is already code in DumperSupport::dump_instance() that tries >>>>>>>> to handle dormant objects, but it needs to be extended to cover >>>>>>>> arrays, as well as and references from non-dormant object/arrays to >>>>>>>> dormant ones. >>>>>>> I have to confess I did not pay any attention to the CDS archived >>>>>>> objects work, so I don't have a firm grasp of how you have >>>>>>> implemented things. But I'm wondering how can you have a reference >>>>>>> to a dormant object from a non-dormant one? Shouldn't the act of >>>>>>> becoming non-dormant automatically cause the subgraph from that >>>>>>> object to also become non-dormant? Or do you have "read barriers" to >>>>>>> perform the changes on demand? >>>>>>> >>>> Ah -- my bug title is not correct. >>>> >>>> I changed the bug title (and this e-mail subject) to >>>> >>>> Heap dumps should exclude dormant CDS archived objects **of unloaded >>>> classes** >>>> >>>> During the heap dump, we scan all objects in the heap, regardless of >>>> reachability. There's no way to decide reachability in >>>> HeapObjectDumper::do_object(), unless we perform an actual GC. >>>> >>>> But it's OK to include unreachable objects in the heap dump. (I guess >>>> it's useful to see how much garbage you have in the heap. There's an >>>> option to run a collection before dumping the heap.) >>>> >>>> There are 2 kinds of unreachable objects -- garbage: those that were >>>> once reachable but no longer, dormant: the archived objects that have >>>> never been reachable. >>> Currently Java object archiving framework only supports one >>> directional state change: dormant -> live. An archived object can >>> become a live object from dormant state, but it cannot go back to the >>> dormant state. Need to investigate thoroughly for all cases before the >>> 'live -> dormant' transition can be supported. All objects in the >>> 'Open' archive heap region are associated with the builtin class >>> loaders and their classes are not unloaded. The existing static fields >>> for archiving within the JDK classes are selected and the associated >>> objects do not become garbage once 'installed'. >>> >>>> Anyway, it's OK to dump dormant objects as long as their class has been >>>> loaded. The problem happens only when we dump a dormant object who class >>>> is not yet loaded (Eclipase MAT get confused when it sees an object >>>> whose class ID is invalid). >>> Yes. That's a scenario needs to be handled for a tool that iterates >>> the Java heap. A dormant object in the 'Open' archive heap region may >>> have a 'invalid' klass since the klass may not be loaded yet at the >>> moment. >>> >>> Your webrev looks reasonable to me on high level pending information >>> for following questions. Can you please give more details on the >>> dormant objects referenced from the arrays? What specific arrays are >>> those? >> Hi Jiangli, >> >> Thanks for the review. I add the following code: >> >> // [id]* elements >> for (int index = 0; index < length; index++) { >> oop o = array->obj_at(index); >> >> >> if (o != NULL && mask_dormant_archived_object(o) == NULL) { >> ResourceMark rm; >> tty->print_cr("%s array contains %s object", >> array->klass()->external_name(), o->klass()->external_name()); >> } >> >> >> o = mask_dormant_archived_object(o); >> writer->write_objectID(o); >> } >> >> and the output is: >> >> $ java -cp . -XX:+HeapDumpAfterFullGC HelloGC >> Dumping heap to java_pid20956.hprof ... >> [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object >> [Ljava.lang.Object; array contains java.util.jar.Attributes$Name object >> (repeated about 20 times) >> >> It comes from java/util/jar/Attributes$Name::KNOWN_NAMES. This class is >> not loaded because my program doesn't use JAR files in the classpath: > The above looks right and is expected. At this point, we would not see > any archive object that first becomes live then becomes dormant again. > > That however will change in the future when we make the object > archiving framework general enough for other JDK and application class > usages. We need to work out the GC details for the live -> dormant > transition when that happens. Hi Jiangli, Sure, we should look into that when we expand the scope of object archiving. I updated the webrev to log the skipped objects with -Xlog:cds+heap=debug: http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v02/ Thanks - Ioi > Thanks, > Jiangli > >> Thanks >> - Ioi >> >> >>> Regards, >>> Jiangli >>> >>> >>> >>>> So to answer your question, we can have a case with a dormant array >>>> (that contains a dormant object) like this: >>>> >>>> Object[] array = {new ClassNotYetLoaded();} >>>> >>>> After my fix, the array will be dumped (we have no easy way of not doing >>>> that), but its contents becomes this in the .hprof file: >>>> >>>> Object[] array = {null} >>>> >>>> Thanks >>>> - Ioi >>>> >>>> >>>> >>>>>> Hi David, >>>>>> >>>>>> Thanks for the review. >>>>>> >>>>>> The dormant objects are not reachable via the GC roots. They become >>>>>> non-dormant via explicit calls to JVM_InitializeFromArchive, after >>>>>> which they become reachable via the static fields of loaded classes. >>>>> Right, so is there a distinction between non-dormant and reachable at >>>>> the time an object becomes non-dormant? I'm still unclear how a drmant >>>>> array becomes non-dormant but still contains elements that refer to >>>>> dormant objects. >>>>> >>>>>> The only issue here is heap dump is done by scanning all objects in >>>>>> the heap, including unreachable ones >>>>>> >>>>>> HeapObjectDumper obj_dumper(this, writer()); >>>>>> Universe::heap()->safe_object_iterate(&obj_dumper); >>>>>> >>>>>> that's how these dormant objects are discovered during heap dump. >>>>>> >>>>>>> That aside the code changes seem reasonable, you moved the check out >>>>>>> of DumperSupport::dump_instance and into the higher-level >>>>>>> HeapObjectDumper::do_object so that it catches instances and arrays, >>>>>>> plus you added a check for array elements. >>>>>>> >>>>>> I am debating whether I should put the masking code in here: >>>>>> >>>>>> void DumpWriter::write_objectID(oop o) { >>>>>> o = mask_dormant_archived_object(o); /// <---- add >>>>>> address a = (address)o; >>>>>> #ifdef _LP64 >>>>>> write_u8((u8)a); >>>>>> #else >>>>>> write_u4((u4)a); >>>>>> #endif >>>>>> } >>>>>> >>>>>> >>>>>> That way, even if a dormant object (unintentionally) becomes >>>>>> reachable via the GC roots, we won't write an invalid reference to it >>>>>> (the object "body" will not be written, so the ID will not point to >>>>>> anything valid). >>>>>> >>>>>> But this seems a little too aggressive to me. What do you think? >>>>> It does seem a little aggressive as it seems to introduce the dormancy >>>>> check into a lot of places that don't need it. But as I said I don't >>>>> know this code so I'm really not the right person to ask. >>>>> >>>>> Cheers, >>>>> David >>>>> ----- >>>>> >>>>>> Thanks >>>>>> - Ioi >>>>>> From david.holmes at oracle.com Tue Sep 10 07:54:54 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 10 Sep 2019 17:54:54 +1000 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> Message-ID: <5fe4ee14-f406-850e-34b4-e44485b39349@oracle.com> It turns out that polling for interrupts is actually very difficult to do correctly. There is an inherent race with timing out and being signalled that can result in lost signals. Trying to account for that without introducing other problems would lead to a very complex synchronization mechanism (which would need to track waiters, notifications and a "generation" count). The end result is that we'd break the tie to ObjectMonitor code, but would replace it with new complex untried code. :( David On 10/09/2019 8:49 am, David Holmes wrote: > Hi Serguei, > > On 10/09/2019 4:26 am, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> On 9/8/19 19:15, David Holmes wrote: >>> Hi Dan, >>> >>> On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: >>>> Hi David, >>>> >>>> I've finally gotten back to this email thread... >>> >>> Thanks. >>> >>>>> FYI testing to date: >>>>> ?- tiers 1 -3 all platforms >>>>> ?- hotspot: serviceability/jvmti >>>>> ????????????????????????? /jdwp >>>>> ??????????? vmTestbase/nsk/jvmti >>>>> ????????????????????????? /jdwp >>>>> ?- JDK: com/sun/jdi >>>> >>>> You should also add: >>>> >>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdb >>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdi >>>> open/test/jdk/java/lang/instrument >>> >>> Okay - in progress. Though I can't see any use of RawMonitors in any >>> of these tests. >>> >>>> I took a quick look through the preliminary webrev and I don't see >>>> anything that worries me. >>> >>> Thanks. I'll prepare a more polished webrev soon. >>> >>>> Re: Thread.interrupt() and raw_wait() >>>> >>>> It would be good to see if that semantic is being tested via the >>>> JCK test suite for JVM/TI. >>> >>> It isn't. The only thing directly tested for RawMonitorWait is normal >>> successful operation and reporting "not owner" when not the owner. No >>> check for JVMTI_ERROR_INTERRUPT exists other than as input for the >>> GetErrorName function. >> >> This is most likely true. >> My only concern is if RawMonitor's can be used in the JCK test >> libraries (low probability). >> I've asked Leonid Kuskov (JCK) to double check this (added to the >> mailing list). >> >>> >>> There's only one test in the whole test base that checks for the >>> interrupt and that is >>> vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if we >>> are not interrupted before the RawMonitorWait we will wait until the >>> full timeout elapses - which is 2 minutes by default - then return >>> and report the interrupt. Hence the test still passes. (If it was an >>> untimed wait that would be different of course). >> >> I figured the same last Friday. >> One more place to care about are NSK tests libraries that are located >> here: >> ?? test/hotspot/jtreg/vmTestbase/nsk/share >> >> There are a couple of places where the RawMonitor is used: >> ?? jvmti/hotswap/HotSwap.cpp:??? if >> (!NSK_JVMTI_VERIFY(jvmti->RawMonitorWait(waitLock, millis))) >> ?? jvmti/jvmti_tools.cpp:??? jvmtiError error = >> env->RawMonitorWait(monitor, millis); >> >> The use in HotSwap.cpp is local. >> The jvmti_tools.cpp defines this: >> ?? void rawMonitorWait(jvmtiEnv *env, jrawMonitorID monitor, jlong >> millis) { >> ??? ?? jvmtiError error = env->RawMonitorWait(monitor, millis); >> ??? ?? exitOnError(error); >> ?? } >> >> which is used in the jvmti/agent_tools.cpp but does not depend on >> interrupting of RawMonitor's (as I can see). >> One more place to mention is: >> ?? jvmti/DataDumpRequest/datadumpreq001/datadumpreq001.cpp >> >> But I see no problems there as well. >> >> >> >> The JDWP implementation is using RawMonitor's. >> Please, see functions debugMonitorWait()/debugMonitorTimedWait() in >> src/jdk.jdwp.agent/share/native/libjdwp/util.c. >> >> It expects the JVMTI_ERROR_INTERRUPT but never makes a call to the >> JVMTI ThreadInterrupt(). >> So, it looks like it does not depend on interrupting of RawMonitor's >> in any way. >> >>> >>> The more I try to convince people this change should be okay, the >>> more uncomfortable I get with my own arguments. :) I think I'm going >>> to implement the polling approach for checking interrupts - say 500ms. >> >> The JVMTI spec tells that the JVMTI_ERROR_INTERRUPT can be returned >> from the RawMonitorWait: >> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >> > > Yes it does and that is the only thing that implies a connection to > Thread.interrupt. > >> which means that RawMonitorWait can be interrupted with the >> Thread.Interrupt() >> or JVMTI InterruptThread(): >> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#InterruptThread >> > > That's one way to interpret the fact that RawMonitorWait can return > JVMTI_ERROR_INTERRUPT, but the actual interaction between > Thread.interrupt and RawMonitorWait is not explicitly stated. Arguably > you can just check for interruption before and after the wait, to see > whether to return JVMTI_ERROR_INTERRUPT, without necessarily being able > to break out of the wait itself. That's been the whole premise of this > change proposal - that responsiveness to interrupts is more a > quality-of-implementation issue. > > But in any case I've decided to try the polling approach so that we > won't wait forever if interrupted but not notified. > > Thanks, > David > ----- > >> >> >> Thanks, >> Serguei >> >>>> I also very much like/appreciate the decoupling of JvmtiRawMonitors >>>> from ObjectMonitors... Thanks for tackling this crazy task. >>> >>> Thanks :) >>> >>> David >>> >>>> Dan >>>> >>>> >>>> >>>> On 8/15/19 2:22 AM, David Holmes wrote: >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >>>>> >>>>> Preliminary webrev (still has rough edges): >>>>> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >>>>> >>>>> Background: >>>>> >>>>> We've had this comment for a long time: >>>>> >>>>> ?// The raw monitor subsystem is entirely distinct from normal >>>>> ?// java-synchronization or jni-synchronization.? raw monitors are not >>>>> ?// associated with objects.? They can be implemented in any manner >>>>> ?// that makes sense.? The original implementors decided to piggy-back >>>>> ?// the raw-monitor implementation on the existing Java >>>>> objectMonitor mechanism. >>>>> ?// This flaw needs to fixed.? We should reimplement raw monitors >>>>> as sui-generis. >>>>> ?// Specifically, we should not implement raw monitors via java >>>>> monitors. >>>>> ?// Time permitting, we should disentangle and deconvolve the two >>>>> implementations >>>>> ?// and move the resulting raw monitor implementation over to the >>>>> JVMTI directories. >>>>> ?// Ideally, the raw monitor implementation would be built on top of >>>>> ?// park-unpark and nothing else. >>>>> >>>>> This is an attempt to do that disentangling so that we can then >>>>> consider changes to ObjectMonitor without having to worry about >>>>> JvmtiRawMonitors. But rather than building on low-level park/unpark >>>>> (which would require the same manual queue management and much of >>>>> the same complex code as exists in ObjectMonitor) I decided to try >>>>> and do this on top of PlatformMonitor. >>>>> >>>>> The reason this is just a RFC rather than RFR is that I overlooked >>>>> a non-trivial aspect of JvmtiRawMonitors: like Java monitors (as >>>>> implemented by ObjectMonitor) they interact with the >>>>> Thread.interrupt mechanism. This is not clearly stated in the JVM >>>>> TI specification [1] but only in passing by the possible errors for >>>>> RawMonitorWait: >>>>> >>>>> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >>>>> >>>>> As I explain in the bug report there is no way to build in proper >>>>> interrupt support using PlatformMonitor as there is no way we can >>>>> "interrupt" the low-level pthread_cond_wait. But we can approximate >>>>> it. What I've done in this preliminary version is just check >>>>> interrupt state before and after the actual "wait" but we won't get >>>>> woken by the interrupt once we have actually blocked. Alternatively >>>>> we could use a periodic polling approach and wakeup every Nms to >>>>> check for interruption. >>>>> >>>>> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is not >>>>> affected by this choice as that code ignores the interrupt until >>>>> the real action it was waiting for has occurred. The interrupt is >>>>> then reposted later. >>>>> >>>>> But more generally there could be users of JvmtiRawMonitors that >>>>> expect/require that RawMonitorWait is responsive to >>>>> Thread.interrupt in a manner similar to Object.wait. And if any of >>>>> them are reading this then I'd like to know - hence this RFC :) >>>>> >>>>> FYI testing to date: >>>>> ?- tiers 1 -3 all platforms >>>>> ?- hotspot: serviceability/jvmti >>>>> ????????????????????????? /jdwp >>>>> ??????????? vmTestbase/nsk/jvmti >>>>> ????????????????????????? /jdwp >>>>> ?- JDK: com/sun/jdi >>>>> >>>>> Comments/opinions appreciated. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> [1] >>>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>>>> >>>> >> From david.holmes at oracle.com Tue Sep 10 08:52:00 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 10 Sep 2019 18:52:00 +1000 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: <5fe4ee14-f406-850e-34b4-e44485b39349@oracle.com> References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> <5fe4ee14-f406-850e-34b4-e44485b39349@oracle.com> Message-ID: <774fb8b3-6c00-fcc7-0c25-858933b65f4b@oracle.com> On 10/09/2019 5:54 pm, David Holmes wrote: > It turns out that polling for interrupts is actually very difficult to > do correctly. There is an inherent race with timing out and being > signalled that can result in lost signals. Trying to account for that > without introducing other problems would lead to a very complex > synchronization mechanism (which would need to track waiters, > notifications and a "generation" count). The end result is that we'd > break the tie to ObjectMonitor code, but would replace it with new > complex untried code. :( Or ... rather than the initial "unresponsive to interrupts" approach I could take it the other way and have every raw monitor wait limited to, say 500ms, at which point it will return as a "spurious wakeup" and check the interrupt state on the way out ... David > David > > On 10/09/2019 8:49 am, David Holmes wrote: >> Hi Serguei, >> >> On 10/09/2019 4:26 am, serguei.spitsyn at oracle.com wrote: >>> Hi David, >>> >>> On 9/8/19 19:15, David Holmes wrote: >>>> Hi Dan, >>>> >>>> On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: >>>>> Hi David, >>>>> >>>>> I've finally gotten back to this email thread... >>>> >>>> Thanks. >>>> >>>>>> FYI testing to date: >>>>>> ?- tiers 1 -3 all platforms >>>>>> ?- hotspot: serviceability/jvmti >>>>>> ????????????????????????? /jdwp >>>>>> ??????????? vmTestbase/nsk/jvmti >>>>>> ????????????????????????? /jdwp >>>>>> ?- JDK: com/sun/jdi >>>>> >>>>> You should also add: >>>>> >>>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdb >>>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdi >>>>> open/test/jdk/java/lang/instrument >>>> >>>> Okay - in progress. Though I can't see any use of RawMonitors in any >>>> of these tests. >>>> >>>>> I took a quick look through the preliminary webrev and I don't see >>>>> anything that worries me. >>>> >>>> Thanks. I'll prepare a more polished webrev soon. >>>> >>>>> Re: Thread.interrupt() and raw_wait() >>>>> >>>>> It would be good to see if that semantic is being tested via the >>>>> JCK test suite for JVM/TI. >>>> >>>> It isn't. The only thing directly tested for RawMonitorWait is >>>> normal successful operation and reporting "not owner" when not the >>>> owner. No check for JVMTI_ERROR_INTERRUPT exists other than as input >>>> for the GetErrorName function. >>> >>> This is most likely true. >>> My only concern is if RawMonitor's can be used in the JCK test >>> libraries (low probability). >>> I've asked Leonid Kuskov (JCK) to double check this (added to the >>> mailing list). >>> >>>> >>>> There's only one test in the whole test base that checks for the >>>> interrupt and that is >>>> vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if >>>> we are not interrupted before the RawMonitorWait we will wait until >>>> the full timeout elapses - which is 2 minutes by default - then >>>> return and report the interrupt. Hence the test still passes. (If it >>>> was an untimed wait that would be different of course). >>> >>> I figured the same last Friday. >>> One more place to care about are NSK tests libraries that are located >>> here: >>> ?? test/hotspot/jtreg/vmTestbase/nsk/share >>> >>> There are a couple of places where the RawMonitor is used: >>> ?? jvmti/hotswap/HotSwap.cpp:??? if >>> (!NSK_JVMTI_VERIFY(jvmti->RawMonitorWait(waitLock, millis))) >>> ?? jvmti/jvmti_tools.cpp:??? jvmtiError error = >>> env->RawMonitorWait(monitor, millis); >>> >>> The use in HotSwap.cpp is local. >>> The jvmti_tools.cpp defines this: >>> ?? void rawMonitorWait(jvmtiEnv *env, jrawMonitorID monitor, jlong >>> millis) { >>> ??? ?? jvmtiError error = env->RawMonitorWait(monitor, millis); >>> ??? ?? exitOnError(error); >>> ?? } >>> >>> which is used in the jvmti/agent_tools.cpp but does not depend on >>> interrupting of RawMonitor's (as I can see). >>> One more place to mention is: >>> ?? jvmti/DataDumpRequest/datadumpreq001/datadumpreq001.cpp >>> >>> But I see no problems there as well. >>> >>> >>> >>> The JDWP implementation is using RawMonitor's. >>> Please, see functions debugMonitorWait()/debugMonitorTimedWait() in >>> src/jdk.jdwp.agent/share/native/libjdwp/util.c. >>> >>> It expects the JVMTI_ERROR_INTERRUPT but never makes a call to the >>> JVMTI ThreadInterrupt(). >>> So, it looks like it does not depend on interrupting of RawMonitor's >>> in any way. >>> >>>> >>>> The more I try to convince people this change should be okay, the >>>> more uncomfortable I get with my own arguments. :) I think I'm going >>>> to implement the polling approach for checking interrupts - say 500ms. >>> >>> The JVMTI spec tells that the JVMTI_ERROR_INTERRUPT can be returned >>> from the RawMonitorWait: >>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>> >> >> Yes it does and that is the only thing that implies a connection to >> Thread.interrupt. >> >>> which means that RawMonitorWait can be interrupted with the >>> Thread.Interrupt() >>> or JVMTI InterruptThread(): >>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#InterruptThread >>> >> >> That's one way to interpret the fact that RawMonitorWait can return >> JVMTI_ERROR_INTERRUPT, but the actual interaction between >> Thread.interrupt and RawMonitorWait is not explicitly stated. Arguably >> you can just check for interruption before and after the wait, to see >> whether to return JVMTI_ERROR_INTERRUPT, without necessarily being >> able to break out of the wait itself. That's been the whole premise of >> this change proposal - that responsiveness to interrupts is more a >> quality-of-implementation issue. >> >> But in any case I've decided to try the polling approach so that we >> won't wait forever if interrupted but not notified. >> >> Thanks, >> David >> ----- >> >>> >>> >>> Thanks, >>> Serguei >>> >>>>> I also very much like/appreciate the decoupling of JvmtiRawMonitors >>>>> from ObjectMonitors... Thanks for tackling this crazy task. >>>> >>>> Thanks :) >>>> >>>> David >>>> >>>>> Dan >>>>> >>>>> >>>>> >>>>> On 8/15/19 2:22 AM, David Holmes wrote: >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >>>>>> >>>>>> Preliminary webrev (still has rough edges): >>>>>> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >>>>>> >>>>>> Background: >>>>>> >>>>>> We've had this comment for a long time: >>>>>> >>>>>> ?// The raw monitor subsystem is entirely distinct from normal >>>>>> ?// java-synchronization or jni-synchronization.? raw monitors are >>>>>> not >>>>>> ?// associated with objects.? They can be implemented in any manner >>>>>> ?// that makes sense.? The original implementors decided to >>>>>> piggy-back >>>>>> ?// the raw-monitor implementation on the existing Java >>>>>> objectMonitor mechanism. >>>>>> ?// This flaw needs to fixed.? We should reimplement raw monitors >>>>>> as sui-generis. >>>>>> ?// Specifically, we should not implement raw monitors via java >>>>>> monitors. >>>>>> ?// Time permitting, we should disentangle and deconvolve the two >>>>>> implementations >>>>>> ?// and move the resulting raw monitor implementation over to the >>>>>> JVMTI directories. >>>>>> ?// Ideally, the raw monitor implementation would be built on top of >>>>>> ?// park-unpark and nothing else. >>>>>> >>>>>> This is an attempt to do that disentangling so that we can then >>>>>> consider changes to ObjectMonitor without having to worry about >>>>>> JvmtiRawMonitors. But rather than building on low-level >>>>>> park/unpark (which would require the same manual queue management >>>>>> and much of the same complex code as exists in ObjectMonitor) I >>>>>> decided to try and do this on top of PlatformMonitor. >>>>>> >>>>>> The reason this is just a RFC rather than RFR is that I overlooked >>>>>> a non-trivial aspect of JvmtiRawMonitors: like Java monitors (as >>>>>> implemented by ObjectMonitor) they interact with the >>>>>> Thread.interrupt mechanism. This is not clearly stated in the JVM >>>>>> TI specification [1] but only in passing by the possible errors >>>>>> for RawMonitorWait: >>>>>> >>>>>> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >>>>>> >>>>>> As I explain in the bug report there is no way to build in proper >>>>>> interrupt support using PlatformMonitor as there is no way we can >>>>>> "interrupt" the low-level pthread_cond_wait. But we can >>>>>> approximate it. What I've done in this preliminary version is just >>>>>> check interrupt state before and after the actual "wait" but we >>>>>> won't get woken by the interrupt once we have actually blocked. >>>>>> Alternatively we could use a periodic polling approach and wakeup >>>>>> every Nms to check for interruption. >>>>>> >>>>>> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is >>>>>> not affected by this choice as that code ignores the interrupt >>>>>> until the real action it was waiting for has occurred. The >>>>>> interrupt is then reposted later. >>>>>> >>>>>> But more generally there could be users of JvmtiRawMonitors that >>>>>> expect/require that RawMonitorWait is responsive to >>>>>> Thread.interrupt in a manner similar to Object.wait. And if any of >>>>>> them are reading this then I'd like to know - hence this RFC :) >>>>>> >>>>>> FYI testing to date: >>>>>> ?- tiers 1 -3 all platforms >>>>>> ?- hotspot: serviceability/jvmti >>>>>> ????????????????????????? /jdwp >>>>>> ??????????? vmTestbase/nsk/jvmti >>>>>> ????????????????????????? /jdwp >>>>>> ?- JDK: com/sun/jdi >>>>>> >>>>>> Comments/opinions appreciated. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> [1] >>>>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>>>>> >>>>> >>> From alexey.menkov at oracle.com Tue Sep 10 20:33:29 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Tue, 10 Sep 2019 13:33:29 -0700 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Richard, Updated webrev: http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev.2/ added delay between retries & moved error clearing to the beginning of the cycle. --alex On 09/09/2019 02:07, Reingruber, Richard wrote: > Hi Alex, > > > Of course error can be cleared before each try - there is not functional > > difference. > > It is just a little confusing, as you can get an exception in L. 95, too. But I'm ok with it, if you > prefer it like this. > > I would suggest, though, to sleep some ms before a retry and double the sleep time in each > following retry. > > Best regards, > Richard. > > -----Original Message----- > From: Alex Menkov > Sent: Freitag, 6. September 2019 22:52 > To: Reingruber, Richard ; serguei.spitsyn at oracle.com; OpenJDK Serviceability > Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException > > Hi Richard, > > On 09/06/2019 09:28, Reingruber, Richard wrote: >> Hi Alex, >> >> that's a good fix for the issue. >> >> One minor thing: >> >> 89 Exception error = null; >> 90 for (int retry = 0; retry < 5; retry++) { >> 91 try { >> 92 log("retry: " + retry); >> 93 s = new Socket("localhost", port); >> 94 error = null; >> 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); >> 96 break; >> 97 } catch (ConnectException ex) { >> 98 log("got exception: " + ex.toString()); >> 99 error = ex; >> 100 } >> 101 } >> 102 if (error != null) { >> 103 throw error; >> 104 } >> >> Is there a reason to clear the local variable error in line 94 instead of clearing it >> in line 91 where each new attempt begins? > > The logic here is: > The cycle has 2 exits: > - error (max retry attempts reached, error is set by last "catch") > - success ("break" statement at line 96, error should be null) > So error is cleared only after the socket is connected (this is the > problematic operation which can cause ConnectException). > > Of course error can be cleared before each try - there is not functional > difference. > > --alex > >> >> Cheers, Richard. >> >> -----Original Message----- >> From: serviceability-dev On Behalf Of serguei.spitsyn at oracle.com >> Sent: Mittwoch, 4. September 2019 22:11 >> To: Alex Menkov ; OpenJDK Serviceability >> Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException >> >> Hi Alex, >> >> The fix looks good. >> Good simplification! >> >> Thanks, >> Serguei >> >> >> On 9/4/19 12:19, Alex Menkov wrote: >>> Hi all, >>> >>> Please review the fix for BadHandshakeTest test. >>> The problem is the test connects to the server twice and if debuggee >>> hasn't yet handled disconnection, the next connect gets "connection >>> refused" error. >>> Instead of adding delay before 2nd connect (we never know "good" value >>> for the delay and big delay can cause "accept timeout"), the test >>> re-tries connect in case of ConnectException. >>> Also improved/simplified the test slightly - debuggee is now run with >>> auto port assignment (used lib.jdb.Debuggee test class which >>> implements required functionality). >>> >>> jira: >>> ? https://bugs.openjdk.java.net/browse/JDK-8192057 >>> webrev: >>> http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ >>> >>> --alex >> From serguei.spitsyn at oracle.com Tue Sep 10 20:51:27 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 10 Sep 2019 13:51:27 -0700 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Alex, It looks good. Thanks, Serguei On 9/10/19 1:33 PM, Alex Menkov wrote: > Hi Richard, > > Updated webrev: > http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev.2/ > > added delay between retries & moved error clearing to the beginning of > the cycle. > > --alex > > On 09/09/2019 02:07, Reingruber, Richard wrote: >> Hi Alex, >> >> ?? > Of course error can be cleared before each try - there is not >> functional >> ?? > difference. >> >> It is just a little confusing, as you can get an exception in L. 95, >> too. But I'm ok with it, if you >> prefer it like this. >> >> I would suggest, though, to sleep some ms before a retry and double >> the sleep time in each >> following retry. >> >> Best regards, >> Richard. >> >> -----Original Message----- >> From: Alex Menkov >> Sent: Freitag, 6. September 2019 22:52 >> To: Reingruber, Richard ; >> serguei.spitsyn at oracle.com; OpenJDK Serviceability >> >> Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java >> fails with java.net.ConnectException >> >> Hi Richard, >> >> On 09/06/2019 09:28, Reingruber, Richard wrote: >>> Hi Alex, >>> >>> that's a good fix for the issue. >>> >>> One minor thing: >>> >>> ??? 89???????????? Exception error = null; >>> ??? 90???????????? for (int retry = 0; retry < 5; retry++) { >>> ??? 91???????????????? try { >>> ??? 92???????????????????? log("retry: " + retry); >>> ??? 93???????????????????? s = new Socket("localhost", port); >>> ??? 94???????????????????? error = null; >>> ??? 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); >>> ??? 96???????????????????? break; >>> ??? 97???????????????? } catch (ConnectException ex) { >>> ??? 98???????????????????? log("got exception: " + ex.toString()); >>> ??? 99???????????????????? error = ex; >>> ?? 100???????????????? } >>> ?? 101???????????? } >>> ?? 102???????????? if (error != null) { >>> ?? 103???????????????? throw error; >>> ?? 104???????????? } >>> >>> Is there a reason to clear the local variable error in line 94 >>> instead of clearing it >>> in line 91 where each new attempt begins? >> >> The logic here is: >> The cycle has 2 exits: >> - error (max retry attempts reached, error is set by last "catch") >> - success ("break" statement at line 96, error should be null) >> So error is cleared only after the socket is connected (this is the >> problematic operation which can cause ConnectException). >> >> Of course error can be cleared before each try - there is not functional >> difference. >> >> --alex >> >>> >>> Cheers, Richard. >>> >>> -----Original Message----- >>> From: serviceability-dev >>> On Behalf Of >>> serguei.spitsyn at oracle.com >>> Sent: Mittwoch, 4. September 2019 22:11 >>> To: Alex Menkov ; OpenJDK Serviceability >>> >>> Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java >>> fails with java.net.ConnectException >>> >>> Hi Alex, >>> >>> The fix looks good. >>> Good simplification! >>> >>> Thanks, >>> Serguei >>> >>> >>> On 9/4/19 12:19, Alex Menkov wrote: >>>> Hi all, >>>> >>>> Please review the fix for BadHandshakeTest test. >>>> The problem is the test connects to the server twice and if debuggee >>>> hasn't yet handled disconnection, the next connect gets "connection >>>> refused" error. >>>> Instead of adding delay before 2nd connect (we never know "good" value >>>> for the delay and big delay can cause "accept timeout"), the test >>>> re-tries connect in case of ConnectException. >>>> Also improved/simplified the test slightly - debuggee is now run with >>>> auto port assignment (used lib.jdb.Debuggee test class which >>>> implements required functionality). >>>> >>>> jira: >>>> ? ? https://bugs.openjdk.java.net/browse/JDK-8192057 >>>> webrev: >>>> http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ >>>> >>>> --alex >>> From serguei.spitsyn at oracle.com Tue Sep 10 20:55:57 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 10 Sep 2019 13:55:57 -0700 Subject: RFR 8230303 : JDB hangs when running monitor command In-Reply-To: References: <18791c15-e6bb-04c0-a1bd-fe22b175560b@oracle.com> Message-ID: Hi Ivan, Thank you for clarifications! Serguei On 9/9/19 10:07 PM, Ivan Gerasimov wrote: > Thank you Serguei for review! > > On 9/9/19 9:14 PM, serguei.spitsyn at oracle.com wrote: >> Hi Ivan, >> >> Thank you for filing the shadow bug and fixing this issue! >> I've targeted the bug to 14 (please, fix me if it is wrong). >> > Yes, this is correct, thanks. > > >> The fix looks good to me. >> >> Do you have any plans to backport it to the earlier releases? >> > Yes, I'm planning to request a backport to the JDK 13 shortly. > > With kind regards, > > Ivan > > From leonid.mesnik at oracle.com Wed Sep 11 02:03:53 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Tue, 10 Sep 2019 19:03:53 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() Message-ID: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> Hi Could you please review following tiny fix which just add ResourceMark in JvmtiSuspendControl::print() method. The method jvmtiSuspendControl::print() might used in custom builds only for debugging purposes. So I don't know when it was used last time. I found that it crashes when I tried to use it locally. webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8230830 Leonid -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Wed Sep 11 05:03:08 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 11 Sep 2019 15:03:08 +1000 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> Message-ID: <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> Hi Leonid, On 11/09/2019 12:03 pm, Leonid Mesnik wrote: > Hi > > Could you please review following tiny fix which just add ResourceMark > in JvmtiSuspendControl::print() method. Looks fine. > The method jvmtiSuspendControl::print() might used in custom builds only > for debugging purposes. So I don't know when it was used last time. I > found that it crashes when I tried to use it locally. The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, and it already has a ResourceMark. So existing use is fine. Please ensure you test with TraceJVMTICalls enabled. Thanks, David > webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8230830 > > Leonid > > From leonid.mesnik at oracle.com Wed Sep 11 05:06:52 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Tue, 10 Sep 2019 22:06:52 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> Message-ID: <07735389-F665-463D-8C4B-8A5734234357@oracle.com> Hi Thank you for feedback. > On Sep 10, 2019, at 10:03 PM, David Holmes wrote: > > Hi Leonid, > > On 11/09/2019 12:03 pm, Leonid Mesnik wrote: >> Hi >> Could you please review following tiny fix which just add ResourceMark in JvmtiSuspendControl::print() method. > > Looks fine. > >> The method jvmtiSuspendControl::print() might used in custom builds only for debugging purposes. So I don't know when it was used last time. I found that it crashes when I tried to use it locally. > > The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, and it already has a ResourceMark. So existing use is fine. It explains why it works. I used it to track status in suspend resume investigating https://bugs.openjdk.java.net/browse/JDK-8230459 . > > Please ensure you test with TraceJVMTICalls enabled. Thanks, I have tested it with this macro enabled. Leonid > Thanks, > David > > >> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >> Leonid -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Wed Sep 11 05:12:38 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 10 Sep 2019 22:12:38 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> Message-ID: <5a41e489-cade-c556-517c-5739c40d8726@oracle.com> Looks good. Chris On 9/10/19 7:03 PM, Leonid Mesnik wrote: > Hi > > Could you please review following tiny fix which just add ResourceMark > in JvmtiSuspendControl::print() method. > > The method jvmtiSuspendControl::print() might used in custom builds > only for debugging purposes. So I don't know when it was used last > time. I found that it crashes when I tried to use it locally. > > webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8230830 > > Leonid > > From serguei.spitsyn at oracle.com Wed Sep 11 06:51:03 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 10 Sep 2019 23:51:03 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <5a41e489-cade-c556-517c-5739c40d8726@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <5a41e489-cade-c556-517c-5739c40d8726@oracle.com> Message-ID: <2fc71d64-4c88-643e-a6fc-3ace291dbb58@oracle.com> Hi Leonid, +1 Thanks, Serguei On 9/10/19 22:12, Chris Plummer wrote: > Looks good. > > Chris > > On 9/10/19 7:03 PM, Leonid Mesnik wrote: >> Hi >> >> Could you please review following tiny fix which just add >> ResourceMark in JvmtiSuspendControl::print() method. >> >> The method jvmtiSuspendControl::print() might used in custom builds >> only for debugging purposes. So I don't know when it was used last >> time. I found that it crashes when I tried to use it locally. >> >> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >> >> Leonid >> >> > From serguei.spitsyn at oracle.com Wed Sep 11 07:13:47 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 11 Sep 2019 00:13:47 -0700 Subject: RFR: 8229378: jdwp library loader in linker_md.c quietly truncates on buffer overflow In-Reply-To: References: <9bab421c-08a1-b554-6ac9-35290856ee56@oracle.com> <0c01c46b-3ff9-3016-8791-868019459d13@oracle.com> <3ea2e6ad-a32d-603c-258c-985da4e2f50a@oracle.com> Message-ID: <11c16477-c096-b807-46fd-acab1c484587@oracle.com> An HTML attachment was scrubbed... URL: From richard.reingruber at sap.com Wed Sep 11 09:19:50 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 11 Sep 2019 09:19:50 +0000 Subject: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException In-Reply-To: References: <985bd047-9ade-afbd-cde5-c29ba9bc0bd4@oracle.com> Message-ID: Hi Alex, thanks, the change looks good. Best regards, Richard. -----Original Message----- From: Alex Menkov Sent: Dienstag, 10. September 2019 22:33 To: Reingruber, Richard ; serguei.spitsyn at oracle.com; OpenJDK Serviceability Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException Hi Richard, Updated webrev: http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev.2/ added delay between retries & moved error clearing to the beginning of the cycle. --alex On 09/09/2019 02:07, Reingruber, Richard wrote: > Hi Alex, > > > Of course error can be cleared before each try - there is not functional > > difference. > > It is just a little confusing, as you can get an exception in L. 95, too. But I'm ok with it, if you > prefer it like this. > > I would suggest, though, to sleep some ms before a retry and double the sleep time in each > following retry. > > Best regards, > Richard. > > -----Original Message----- > From: Alex Menkov > Sent: Freitag, 6. September 2019 22:52 > To: Reingruber, Richard ; serguei.spitsyn at oracle.com; OpenJDK Serviceability > Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException > > Hi Richard, > > On 09/06/2019 09:28, Reingruber, Richard wrote: >> Hi Alex, >> >> that's a good fix for the issue. >> >> One minor thing: >> >> 89 Exception error = null; >> 90 for (int retry = 0; retry < 5; retry++) { >> 91 try { >> 92 log("retry: " + retry); >> 93 s = new Socket("localhost", port); >> 94 error = null; >> 95 s.getOutputStream().write("JDWP-".getBytes("UTF-8")); >> 96 break; >> 97 } catch (ConnectException ex) { >> 98 log("got exception: " + ex.toString()); >> 99 error = ex; >> 100 } >> 101 } >> 102 if (error != null) { >> 103 throw error; >> 104 } >> >> Is there a reason to clear the local variable error in line 94 instead of clearing it >> in line 91 where each new attempt begins? > > The logic here is: > The cycle has 2 exits: > - error (max retry attempts reached, error is set by last "catch") > - success ("break" statement at line 96, error should be null) > So error is cleared only after the socket is connected (this is the > problematic operation which can cause ConnectException). > > Of course error can be cleared before each try - there is not functional > difference. > > --alex > >> >> Cheers, Richard. >> >> -----Original Message----- >> From: serviceability-dev On Behalf Of serguei.spitsyn at oracle.com >> Sent: Mittwoch, 4. September 2019 22:11 >> To: Alex Menkov ; OpenJDK Serviceability >> Subject: Re: RFR: JDK-8192057: com/sun/jdi/BadHandshakeTest.java fails with java.net.ConnectException >> >> Hi Alex, >> >> The fix looks good. >> Good simplification! >> >> Thanks, >> Serguei >> >> >> On 9/4/19 12:19, Alex Menkov wrote: >>> Hi all, >>> >>> Please review the fix for BadHandshakeTest test. >>> The problem is the test connects to the server twice and if debuggee >>> hasn't yet handled disconnection, the next connect gets "connection >>> refused" error. >>> Instead of adding delay before 2nd connect (we never know "good" value >>> for the delay and big delay can cause "accept timeout"), the test >>> re-tries connect in case of ConnectException. >>> Also improved/simplified the test slightly - debuggee is now run with >>> auto port assignment (used lib.jdb.Debuggee test class which >>> implements required functionality). >>> >>> jira: >>> ? https://bugs.openjdk.java.net/browse/JDK-8192057 >>> webrev: >>> http://cr.openjdk.java.net/~amenkov/jdk14/BadHandshakeTest/webrev/ >>> >>> --alex >> From adam.farley at uk.ibm.com Wed Sep 11 11:18:30 2019 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Wed, 11 Sep 2019 12:18:30 +0100 Subject: RFR: 8229378: jdwp library loader in linker_md.c quietly truncates on buffer overflow In-Reply-To: <11c16477-c096-b807-46fd-acab1c484587@oracle.com> References: <9bab421c-08a1-b554-6ac9-35290856ee56@oracle.com> <0c01c46b-3ff9-3016-8791-868019459d13@oracle.com> <3ea2e6ad-a32d-603c-258c-985da4e2f50a@oracle.com> <11c16477-c096-b807-46fd-acab1c484587@oracle.com> Message-ID: Hi Serguei, If you're happy with the fix, then here's a webrev without the os.cpp bit. http://cr.openjdk.java.net/~afarley/8229378.3/webrev/ That was only included to make direct testing possible, and I want to make sure it doesn't get included by accident. Best Regards Adam Farley IBM Runtimes "serguei.spitsyn at oracle.com" wrote on 11/09/2019 08:13:47: > From: "serguei.spitsyn at oracle.com" > To: Adam Farley8 > Cc: Chris Plummer , > daniel.daugherty at oracle.com, serviceability-dev at openjdk.java.net > Date: 11/09/2019 08:14 > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > quietly truncates on buffer overflow > > Hi Adam, > > I'm Okay with this fix. > If nobody else have comments then I'll build it, test a little bit > and push to the jdk/jdk repo. > > Thanks, > Serguei > > > On 9/9/19 09:53, Adam Farley8 wrote: > Hi Serguei, > > Apologies for the delay. > > The errors have all been fixed, and the requested tests mostly > passed, windows and linux. > > No test group had more failures post-fix than pre-fix, so I'm > calling that a win. > > The new webrev can be found here: > > http://cr.openjdk.java.net/~afarley/8229378.2/webrev > > Best Regards > > Adam Farley > IBM Runtimes > > > "serguei.spitsyn at oracle.com" wrote on > 29/08/2019 19:38:02: > > > From: "serguei.spitsyn at oracle.com" > > To: Adam Farley8 > > Cc: Chris Plummer , > > daniel.daugherty at oracle.com, serviceability-dev at openjdk.java.net > > Date: 29/08/2019 19:38 > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > quietly truncates on buffer overflow > > > > Hi Adam, > > > > Okay, thanks! > > Serguei > > > > > > On 8/29/19 06:26, Adam Farley8 wrote: > > Hi Serguei, > > > > I haven't actually run a fastdebug build before. Will do that now > > and address the issues. > > > > Once done, I'll re-run the tests I ran, and also the tests you've > > listed below. > > > > Can you advise on how "good coverage" is determined, so I know for > > future bug fixes? > > > > As for the up-to-date-ness, I'll update the build before doing the above. > > > > Expect a webrev once all of this is complete. > > > > Best Regards > > > > Adam Farley > > IBM Runtimes > > > > > > "serguei.spitsyn at oracle.com" wrote on > > 29/08/2019 03:54:56: > > > > > From: "serguei.spitsyn at oracle.com" > > > To: Adam Farley8 > > > Cc: Chris Plummer , > > > daniel.daugherty at oracle.com, serviceability-dev at openjdk.java.net > > > Date: 29/08/2019 04:23 > > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > > quietly truncates on buffer overflow > > > > > > Hi Adam, > > > > > > Sorry for the latency. > > > I was in process to build, test and push your fix and got the > > > fastdebug build errors below. > > > > > > So, my question is if you've ever built the fastdebug version. > > > This change is in the system-dependent code, so it has to be tested > > > on both Unix and Windows. > > > > > > > My testing was limited to the bug specific test case I mentioned, > > > and the following jdwp tests: > > > > > > > > test/jdk/com/sun/jdi/Jdwp* > > > > test/hotspot/jtreg/serviceability/jdwp > > > > > > This set of tests does not provide a good coverage. > > > To make sure nothing is broken you need to run the the test/jdk/ > com/sun/jdi > > > and also the following vmTestbase tests: > > > > > > test/hotspot/jtreg/vmTestbase/nsk/jdi > > > test/hotspot/jtreg/vmTestbase/nsk/jdb > > > test/hotspot/jtreg/vmTestbase/nsk/jdwp > > > > > > BTW, your current webrev is not up-to-date: > > > http://cr.openjdk.java.net/~afarley/8229378/webrev/ > > > > > > I guess, the change in the src/hotspot/share/runtime/os.cpp became > > > obsolete after your previous fix that was already pushed. > > > > > > Thanks, > > > Serguei > > > > > > . . . > > > In file included from /scratch/sspitsyn/jdk14.1/open/src/ > > > jdk.jdwp.agent/unix/native/libjdwp/linker_md.c:37:0: > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > > libjdwp/linker_md.c: In function ?dll_build_name?: > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > > libjdwp/util.h:46:23: error: ?Do? undeclared (first use in this function) > > > #define strdup(p) Do not use this interface. > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > > paths_copy = strdup(paths); > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > > libjdwp/util.h:46:23: note: each undeclared identifier is reported > > > only once for each function it appears in > > > #define strdup(p) Do not use this interface. > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > > paths_copy = strdup(paths); > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > > libjdwp/util.h:46:26: error: expected ?;? before ?not? > > > #define strdup(p) Do not use this interface. > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > > libjdwp/linker_md.c:51:18: note: in expansion of macro ?strdup? > > > paths_copy = strdup(paths); > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/share/native/ > > > libjdwp/util.h:38:24: error: expected ?;? before ?not? > > > #define free(p) Do not use this interface. > > > ^ > > > /scratch/sspitsyn/jdk14.1/open/src/jdk.jdwp.agent/unix/native/ > > > libjdwp/linker_md.c:71:5: note: in expansion of macro ?free? > > > free(paths_copy); > > > ^ > > > gmake[3]: *** [/scratch/sspitsyn/jdk14.1/build/linux-x86_64-server- > > > fastdebug/support/native/jdk.jdwp.agent/libjdwp/linker_md.o] Error 1 > > > gmake[2]: *** [jdk.jdwp.agent-libs] Error 1 > > > gmake[2]: *** Waiting for unfinished jobs.... > > > > > > ERROR: Build failed for target 'images' in configuration 'linux- > > > x86_64-server-fastdebug' (exit code 2) > > > > > > > > > > > > On 8/13/19 09:28, Adam Farley8 wrote: > > > Hi Serguei, Daniel, > > > > > > My testing was limited to the bug specific test case I mentioned, > > > and the following jdwp tests: > > > > > > test/jdk/com/sun/jdi/Jdwp* > > > test/hotspot/jtreg/serviceability/jdwp > > > > > > Best Regards > > > > > > Adam Farley > > > IBM Runtimes > > > > > > > > > "serguei.spitsyn at oracle.com" wrote on > > > 13/08/2019 17:04:43: > > > > > > > From: "serguei.spitsyn at oracle.com" > > > > To: daniel.daugherty at oracle.com, Adam Farley8 > > > > , Chris Plummer > > > > Cc: serviceability-dev at openjdk.java.net > > > > Date: 13/08/2019 17:08 > > > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > > > quietly truncates on buffer overflow > > > > > > > > Hi Adam, > > > > > > > > I'm looking at your fix. > > > > Also interested about your testing. > > > > > > > > Thanks, > > > > Serguei > > > > > > > > On 8/13/19 08:48, Daniel D. Daugherty wrote: > > > > I don't see any information about how this change was tested... > > > > Is there something on another email thread? > > > > > > > > Dan > > > > > > > > > > > On 8/13/19 11:41 AM, Adam Farley8 wrote: > > > > Hi Chris, > > > > > > > > Thanks! > > > > > > > > I understand we need a second reviewer/sponsor to get this change > > > > in. Any volunteers? > > > > > > > > Best Regards > > > > > > > > Adam Farley > > > > IBM Runtimes > > > > > > > > > > > > Chris Plummer wrote on 12/08/2019 21:35:06: > > > > > > > > > From: Chris Plummer > > > > > To: Adam Farley8 , serviceability- > > > > dev at openjdk.java.net > > > > > Date: 12/08/2019 21:35 > > > > > Subject: Re: RFR: 8229378: jdwp library loader in linker_md.c > > > > > quietly truncates on buffer overflow > > > > > > > > > > Hi Adam, > > > > > > > > > > It looks good to me. > > > > > > > > > > thanks, > > > > > > > > > > Chris > > > > > > > > > > On 8/12/19 7:34 AM, Adam Farley8 wrote: > > > > > Hi All, > > > > > > > > > > This is a known bug, mentioned in a code comment. > > > > > > > > > > Here is the fix for that bug. > > > > > > > > > > Reviewers and sponsors requested. > > > > > > > > > > Short version: if you set sun.boot.library.path to > > > > > something beyond a system's max path length, the > > > > > current code will return an empty string (rather than > > > > > printing a useful error message and shutting down). > > > > > > > > > > This is also a problem if you've specified multiple > > > > > paths with a separator, as this code seems to wrongly > > > > > assess whether the *total* length exceeds max path > > > > > length. So two 200 char paths on windows will cause > > > > > failure, as the total length is 400 (which is beyond > > > > > max length for windows). > > > > > > > > > > Note that the os.cpp bit of the webrev will not be included > > > > > in the final webrev, it just makes this change trivially > > > > > testable. > > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8229378 > > > > > Webrev: http://cr.openjdk.java.net/~afarley/8229378/webrev/ > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > Adam Farley > > > > > IBM Runtimes > > > > > > > > > > Unless stated otherwise above: > > > > > IBM United Kingdom Limited - Registered in England and Wales with > > > > > number 741598. > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, > > Hampshire PO6 3AU > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > > > number 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, > Hampshire PO6 3AU > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > > > number 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with > > number 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with > number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Wed Sep 11 12:37:45 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 11 Sep 2019 12:37:45 +0000 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently Message-ID: Hi, please review this change for test sun/tools/jcmd/TestProcessHelper.java to make it more robust. Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ This Linux only test is starting several Java processes and then tries to figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper implementation which parses the contents of /proc//cmdline. Under some circumstances, the test already attempts to read /proc//cmdline before it actually exists or is filled with data. This can be fixed with some sleeps/retries to wait for that data to be ready. In the actual jcmd tool, such behavior of ProcessHelper. getMainClass should not be an issue because it is handled in ProcessArgumentMatcher [0]. Thanks Christoph [0] http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Wed Sep 11 12:56:53 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 11 Sep 2019 08:56:53 -0400 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <07735389-F665-463D-8C4B-8A5734234357@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> <07735389-F665-463D-8C4B-8A5734234357@oracle.com> Message-ID: <44459eb0-3b23-09d5-d8d3-0958486c223e@oracle.com> On 9/11/19 1:06 AM, Leonid Mesnik wrote: > Hi > > Thank you for feedback. > >> On Sep 10, 2019, at 10:03 PM, David Holmes > > wrote: >> >> Hi Leonid, >> >> On 11/09/2019 12:03 pm, Leonid Mesnik wrote: >>> Hi >>> Could you please review following tiny fix which just add >>> ResourceMark in JvmtiSuspendControl::print() method. >> >> Looks fine. >> >>> The method jvmtiSuspendControl::print() might used in custom builds >>> only for debugging purposes. So I don't know when it was used last >>> time. I found that it crashes when I tried to use it locally. >> >> The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, >> and it already has a ResourceMark. So existing use is fine. > > It explains why it works. So the question that comes to my mind is whether the ResourceMark that is in JvmtiEnv::NotifyFramePop() is needed for something other than the JvmtiSuspendControl::print() call? If not, then removing the one in JvmtiEnv::NotifyFramePop() in favor of the one you added in JvmtiSuspendControl::print() is a good idea. Dan > I used it to track status in suspend resume investigating > https://bugs.openjdk.java.net/browse/JDK-8230459. > >> >> Please ensure you test with TraceJVMTICalls enabled. > > Thanks, I have tested it with this macro enabled. > > Leonid >> Thanks, >> David >> >> >>> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >>> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >>> Leonid > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed Sep 11 13:59:07 2019 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 11 Sep 2019 15:59:07 +0200 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: Hi Christoph, in general I think this is fine. The increase-by-pow2 sleep time is odd but okay :) The whole things seems rather fragile and has a lot of question marks but I think your fix does not make it worse. One fun error now is that with a follow up java test reusing the PID we could get a wrong main class but I think the chances are astronomically low. Only remark, you fix this in the platform shared code, if this is a Linux only issue maybe it should be fixed in /shared/projects/openjdk/jdk-jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java instead? If not, I would remove at least the /proc//cmdline comment since this is quite platform specific. Cheers, Thomas On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph wrote: > Hi, > > > > please review this change for test sun/tools/jcmd/TestProcessHelper.java > to make it more robust. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > > > > This Linux only test is starting several Java processes and then tries to > figure out the main class by invoking jdk.jcmd's linux specific > ProcessHelper implementation which parses the contents of > /proc//cmdline. > > Under some circumstances, the test already attempts to read > /proc//cmdline before it actually exists or is filled with data. This > can be fixed with some sleeps/retries to wait for that data to be ready. > > In the actual jcmd tool, such behavior of ProcessHelper. getMainClass > should not be an issue because it is handled in ProcessArgumentMatcher [0]. > > > > Thanks > > Christoph > > > > [0] > http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Wed Sep 11 16:44:48 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 11 Sep 2019 09:44:48 -0700 Subject: RFR: 8229378: jdwp library loader in linker_md.c quietly truncates on buffer overflow In-Reply-To: References: <9bab421c-08a1-b554-6ac9-35290856ee56@oracle.com> <0c01c46b-3ff9-3016-8791-868019459d13@oracle.com> <3ea2e6ad-a32d-603c-258c-985da4e2f50a@oracle.com> <11c16477-c096-b807-46fd-acab1c484587@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Wed Sep 11 17:21:26 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 11 Sep 2019 10:21:26 -0700 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Wed Sep 11 21:37:57 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 11 Sep 2019 21:37:57 +0000 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: Hi Chris, Thomas, thanks for looking at this. I was also wondering whether a fix in ProcessHelper would be appropriate. But I think introducing retries and delays in that code can do more harm than help. For this special test case, aiming to test the ProcessHelper functionality (on Linux) only, the observed problem is that the /proc//cmdline file is not ready yet when it gets evaluated because the test can be quicker than the spawned processes. But in real life usage of jcmd this seems rather unlikely. One will probably use jcmd quite some time after a java process was started and /proc//cmdline should be ready. If then there are problems reading it, there are likely other issues which won?t go away by waiting. And for these cases the fallback is to use the attach framework, as implemented in ProcessArgumentMatcher, which provides some chance to be working still. And this fallback should also cover the exotic case when jcmd is issued too early. After all, ProcessHelper::getMainClass also documents that its result can be null. @Thomas, as for your other points: * PID reusage: Hm, maybe one can construct cases. However, I?d think the /proc/pid files should be gone after a process ends. Or at least be reconstructed if there were orphans and a new process reusing an old pid gets started. But who knows what can happen ? we?ll maybe see ?? * Comment for Linux only issue: The test is in fact a Linux only test. See line 55: * @requires os.family == "linux". So, if we?ll eventually see implementations for ProcessHelper::getMainClass on other platforms, this comment might have to be adopted. But for the time being I guess it?s fine at its current place. Would you agree? Best regards Christoph From: Chris Plummer Sent: Mittwoch, 11. September 2019 19:21 To: Thomas St?fe ; Langer, Christoph Cc: OpenJDK Serviceability Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently It does seem that the fix should be in ProcessHelper.java in getMainClass(), or maybe even getCommandLine(). Fixing it in the test implies that every user of getMainClass() should be doing something similar. But then also note what ProcessArgumentMatch.check() is doing. It also deals with getMainClass() returing null. thanks, Chris On 9/11/19 6:59 AM, Thomas St?fe wrote: Hi Christoph, in general I think this is fine. The increase-by-pow2 sleep time is odd but okay :) The whole things seems rather fragile and has a lot of question marks but I think your fix does not make it worse. One fun error now is that with a follow up java test reusing the PID we could get a wrong main class but I think the chances are astronomically low. Only remark, you fix this in the platform shared code, if this is a Linux only issue maybe it should be fixed in /shared/projects/openjdk/jdk-jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java instead? If not, I would remove at least the /proc//cmdline comment since this is quite platform specific. Cheers, Thomas On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph > wrote: Hi, please review this change for test sun/tools/jcmd/TestProcessHelper.java to make it more robust. Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ This Linux only test is starting several Java processes and then tries to figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper implementation which parses the contents of /proc//cmdline. Under some circumstances, the test already attempts to read /proc//cmdline before it actually exists or is filled with data. This can be fixed with some sleeps/retries to wait for that data to be ready. In the actual jcmd tool, such behavior of ProcessHelper. getMainClass should not be an issue because it is handled in ProcessArgumentMatcher [0]. Thanks Christoph [0] http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Wed Sep 11 22:05:40 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 11 Sep 2019 22:05:40 +0000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper Message-ID: Hi, please review an enhancement which I've identified when working with Processhelper for JDK-8230850. I noticed that ProcessHelper is an interface in common code with a static method that would lookup the actual platform implementation via reflection. This seems a little cumbersome since we can have a common dummy for ProcessHelper and override it with the platform specific implementation, leveraging the build system. The only drawback is that the test "TestProcessHelper" gets a little more sophisticated when using only package private methods in ProcessHelper, because the methods need to be made accessible for the test. But I guess that's ok, given the actual tool improvement. Bug: https://bugs.openjdk.java.net/browse/JDK-8230857 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230857.0/ Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Wed Sep 11 22:29:13 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Sep 2019 08:29:13 +1000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: Message-ID: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> Hi Christoph, On 12/09/2019 8:05 am, Langer, Christoph wrote: > Hi, > > please review an enhancement which I?ve identified when working with > Processhelper for JDK-8230850. > > I noticed that ProcessHelper is an interface in common code with a > static method that would lookup the actual platform implementation via > reflection. This seems a little cumbersome since we can have a common > dummy for ProcessHelper and override it with the platform specific > implementation, leveraging the build system. I don't see you leveraging the build system. You have two source files that compile to the same destination class file. What is ensuring the platform specific version is compiled after the generic one? Service-provider patterns use reflection to instantiate the service implementation. I don't see any problem here that needs solving. David ----- The only drawback is that > the test ?TestProcessHelper? gets a little more sophisticated when using > only package private methods in ProcessHelper, because the methods need > to be made accessible for the test. But I guess that?s ok, given the > actual tool improvement. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230857 > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230857.0/ > > Thanks > > Christoph > From leonid.mesnik at oracle.com Wed Sep 11 22:45:26 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Wed, 11 Sep 2019 15:45:26 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <44459eb0-3b23-09d5-d8d3-0958486c223e@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> <07735389-F665-463D-8C4B-8A5734234357@oracle.com> <44459eb0-3b23-09d5-d8d3-0958486c223e@oracle.com> Message-ID: Hi It is still needed for vframe *vf = vframeFor(java_thread, depth); Leonid > On Sep 11, 2019, at 5:56 AM, Daniel D. Daugherty wrote: > > On 9/11/19 1:06 AM, Leonid Mesnik wrote: >> Hi >> >> Thank you for feedback. >> >>> On Sep 10, 2019, at 10:03 PM, David Holmes > wrote: >>> >>> Hi Leonid, >>> >>> On 11/09/2019 12:03 pm, Leonid Mesnik wrote: >>>> Hi >>>> Could you please review following tiny fix which just add ResourceMark in JvmtiSuspendControl::print() method. >>> >>> Looks fine. >>> >>>> The method jvmtiSuspendControl::print() might used in custom builds only for debugging purposes. So I don't know when it was used last time. I found that it crashes when I tried to use it locally. >>> >>> The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, and it already has a ResourceMark. So existing use is fine. >> >> It explains why it works. > > So the question that comes to my mind is whether the ResourceMark > that is in JvmtiEnv::NotifyFramePop() is needed for something > other than the JvmtiSuspendControl::print() call? If not, then > removing the one in JvmtiEnv::NotifyFramePop() in favor of the > one you added in JvmtiSuspendControl::print() is a good idea. > > Dan > > >> I used it to track status in suspend resume investigating https://bugs.openjdk.java.net/browse/JDK-8230459 . >> >>> >>> Please ensure you test with TraceJVMTICalls enabled. >> >> Thanks, I have tested it with this macro enabled. >> >> Leonid >>> Thanks, >>> David >>> >>> >>>> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >>>> Leonid >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Wed Sep 11 23:38:32 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 11 Sep 2019 19:38:32 -0400 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> <07735389-F665-463D-8C4B-8A5734234357@oracle.com> <44459eb0-3b23-09d5-d8d3-0958486c223e@oracle.com> Message-ID: <76e54e33-3b5c-d6c6-b559-b98ba05149cb@oracle.com> Thanks for chasing that down. I'm good with your change. Dan On 9/11/19 6:45 PM, Leonid Mesnik wrote: > Hi > > It is still needed for > vframe *vf = vframeFor(java_thread, depth); > > Leonid > >> On Sep 11, 2019, at 5:56 AM, Daniel D. Daugherty >> > wrote: >> >> On 9/11/19 1:06 AM, Leonid Mesnik wrote: >>> Hi >>> >>> Thank you for feedback. >>> >>>> On Sep 10, 2019, at 10:03 PM, David Holmes >>> > wrote: >>>> >>>> Hi Leonid, >>>> >>>> On 11/09/2019 12:03 pm, Leonid Mesnik wrote: >>>>> Hi >>>>> Could you please review following tiny fix which just add >>>>> ResourceMark in JvmtiSuspendControl::print() method. >>>> >>>> Looks fine. >>>> >>>>> The method jvmtiSuspendControl::print() might used in custom >>>>> builds only for debugging purposes. So I don't know when it was >>>>> used last time. I found that it crashes when I tried to use it >>>>> locally. >>>> >>>> The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, >>>> and it already has a ResourceMark. So existing use is fine. >>> >>> It explains why it works. >> >> So the question that comes to my mind is whether the ResourceMark >> that is in JvmtiEnv::NotifyFramePop() is needed for something >> other than the JvmtiSuspendControl::print() call? If not, then >> removing the one in JvmtiEnv::NotifyFramePop() in favor of the >> one you added in JvmtiSuspendControl::print() is a good idea. >> >> Dan >> >> >>> I used it to track status in suspend resume investigating >>> https://bugs.openjdk.java.net/browse/JDK-8230459. >>> >>>> >>>> Please ensure you test with TraceJVMTICalls enabled. >>> >>> Thanks, I have tested it with this macro enabled. >>> >>> Leonid >>>> Thanks, >>>> David >>>> >>>> >>>>> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >>>>> Leonid >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonid.mesnik at oracle.com Wed Sep 11 23:50:45 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Wed, 11 Sep 2019 16:50:45 -0700 Subject: RFR: 8230830: No required ResourceMark in src/hotspot/share/prims/jvmtiImpl.cpp:JvmtiSuspendControl::print() In-Reply-To: <76e54e33-3b5c-d6c6-b559-b98ba05149cb@oracle.com> References: <7D331CDC-55A4-499D-B2D7-59C58CEF22BA@oracle.com> <7cb9162f-1d39-b22d-8927-4a781df0c40e@oracle.com> <07735389-F665-463D-8C4B-8A5734234357@oracle.com> <44459eb0-3b23-09d5-d8d3-0958486c223e@oracle.com> <76e54e33-3b5c-d6c6-b559-b98ba05149cb@oracle.com> Message-ID: David, Daniel, Serguei, Chris Thank you for review. Leonid > On Sep 11, 2019, at 4:38 PM, Daniel D. Daugherty wrote: > > Thanks for chasing that down. I'm good with your change. > > Dan > > > On 9/11/19 6:45 PM, Leonid Mesnik wrote: >> Hi >> >> It is still needed for >> vframe *vf = vframeFor(java_thread, depth); >> >> Leonid >> >>> On Sep 11, 2019, at 5:56 AM, Daniel D. Daugherty > wrote: >>> >>> On 9/11/19 1:06 AM, Leonid Mesnik wrote: >>>> Hi >>>> >>>> Thank you for feedback. >>>> >>>>> On Sep 10, 2019, at 10:03 PM, David Holmes > wrote: >>>>> >>>>> Hi Leonid, >>>>> >>>>> On 11/09/2019 12:03 pm, Leonid Mesnik wrote: >>>>>> Hi >>>>>> Could you please review following tiny fix which just add ResourceMark in JvmtiSuspendControl::print() method. >>>>> >>>>> Looks fine. >>>>> >>>>>> The method jvmtiSuspendControl::print() might used in custom builds only for debugging purposes. So I don't know when it was used last time. I found that it crashes when I tried to use it locally. >>>>> >>>>> The only caller is JvmtiEnv::NotifyFramePop, under TraceJVMTICalls, and it already has a ResourceMark. So existing use is fine. >>>> >>>> It explains why it works. >>> >>> So the question that comes to my mind is whether the ResourceMark >>> that is in JvmtiEnv::NotifyFramePop() is needed for something >>> other than the JvmtiSuspendControl::print() call? If not, then >>> removing the one in JvmtiEnv::NotifyFramePop() in favor of the >>> one you added in JvmtiSuspendControl::print() is a good idea. >>> >>> Dan >>> >>> >>>> I used it to track status in suspend resume investigating https://bugs.openjdk.java.net/browse/JDK-8230459 . >>>> >>>>> >>>>> Please ensure you test with TraceJVMTICalls enabled. >>>> >>>> Thanks, I have tested it with this macro enabled. >>>> >>>> Leonid >>>>> Thanks, >>>>> David >>>>> >>>>> >>>>>> webrev: http://cr.openjdk.java.net/~lmesnik/8230830/webrev.00/ >>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8230830 >>>>>> Leonid >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Sep 12 08:10:32 2019 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 12 Sep 2019 10:10:32 +0200 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: I'm fine with the patch if you would reshape the platform dependent comment. Proposal: ---- - // Depending on hw/os, process helper can return null here- // because /proc//cmdline is not ready yet. To cover that case, // give it some retries. -> + getMainClass() may return NULL, e.g. due to timing issues. Attempt some limited retries. ---- I do not need another webrev. Cheers, Thomas On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph wrote: > Hi Chris, Thomas, > > > > thanks for looking at this. I was also wondering whether a fix in > ProcessHelper would be appropriate. But I think introducing retries and > delays in that code can do more harm than help. > > > > For this special test case, aiming to test the ProcessHelper functionality > (on Linux) only, the observed problem is that the /proc//cmdline file > is not ready yet when it gets evaluated because the test can be quicker > than the spawned processes. But in real life usage of jcmd this seems > rather unlikely. One will probably use jcmd quite some time after a java > process was started and /proc//cmdline should be ready. If then there > are problems reading it, there are likely other issues which won?t go away > by waiting. And for these cases the fallback is to use the attach > framework, as implemented in ProcessArgumentMatcher, which provides some > chance to be working still. And this fallback should also cover the exotic > case when jcmd is issued too early. > > > > After all, ProcessHelper::getMainClass also documents that its result can > be null. > > > > @Thomas, as for your other points: > > - PID reusage: Hm, maybe one can construct cases. However, I?d think > the /proc/pid files should be gone after a process ends. Or at least be > reconstructed if there were orphans and a new process reusing an old pid > gets started. But who knows what can happen ? we?ll maybe see ?? > - Comment for Linux only issue: The test is in fact a Linux only test. > See line 55: * @requires os.family == "*linux*". So, if we?ll > eventually see implementations for ProcessHelper::getMainClass on other > platforms, this comment might have to be adopted. But for the time being I > guess it?s fine at its current place. > > > > Would you agree? > > > > Best regards > > Christoph > > > > *From:* Chris Plummer > *Sent:* Mittwoch, 11. September 2019 19:21 > *To:* Thomas St?fe ; Langer, Christoph < > christoph.langer at sap.com> > *Cc:* OpenJDK Serviceability > *Subject:* Re: RFR (S): 8230850: Test > sun/tools/jcmd/TestProcessHelper.java fails intermittently > > > > It does seem that the fix should be in ProcessHelper.java in > getMainClass(), or maybe even getCommandLine(). Fixing it in the test > implies that every user of getMainClass() should be doing something > similar. But then also note what ProcessArgumentMatch.check() is doing. It > also deals with getMainClass() returing null. > > thanks, > > Chris > > On 9/11/19 6:59 AM, Thomas St?fe wrote: > > Hi Christoph, > > > > in general I think this is fine. The increase-by-pow2 sleep time is odd > but okay :) > > > > The whole things seems rather fragile and has a lot of question marks but > I think your fix does not make it worse. One fun error now is that with a > follow up java test reusing the PID we could get a wrong main class but I > think the chances are astronomically low. > > > > Only remark, you fix this in the platform shared code, if this is a Linux > only issue maybe it should be fixed > in /shared/projects/openjdk/jdk-jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java > instead? If not, I would remove at least the /proc//cmdline comment > since this is quite platform specific. > > > > Cheers, Thomas > > > > > > On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph < > christoph.langer at sap.com> wrote: > > Hi, > > > > please review this change for test sun/tools/jcmd/TestProcessHelper.java > to make it more robust. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > > > > This Linux only test is starting several Java processes and then tries to > figure out the main class by invoking jdk.jcmd's linux specific > ProcessHelper implementation which parses the contents of > /proc//cmdline. > > Under some circumstances, the test already attempts to read > /proc//cmdline before it actually exists or is filled with data. This > can be fixed with some sleeps/retries to wait for that data to be ready. > > In the actual jcmd tool, such behavior of ProcessHelper. getMainClass > should not be an issue because it is handled in ProcessArgumentMatcher [0]. > > > > Thanks > > Christoph > > > > [0] > http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Sep 12 08:12:49 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 12 Sep 2019 08:12:49 +0000 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: Hi Thomas, sounds reasonable, will do. Thanks Christoph From: Thomas St?fe Sent: Donnerstag, 12. September 2019 10:11 To: Langer, Christoph Cc: Chris Plummer ; OpenJDK Serviceability Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently I'm fine with the patch if you would reshape the platform dependent comment. Proposal: ---- - // Depending on hw/os, process helper can return null here - // because /proc//cmdline is not ready yet. To cover that case, // give it some retries. -> + getMainClass() may return NULL, e.g. due to timing issues. Attempt some limited retries. ---- I do not need another webrev. Cheers, Thomas On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph > wrote: Hi Chris, Thomas, thanks for looking at this. I was also wondering whether a fix in ProcessHelper would be appropriate. But I think introducing retries and delays in that code can do more harm than help. For this special test case, aiming to test the ProcessHelper functionality (on Linux) only, the observed problem is that the /proc//cmdline file is not ready yet when it gets evaluated because the test can be quicker than the spawned processes. But in real life usage of jcmd this seems rather unlikely. One will probably use jcmd quite some time after a java process was started and /proc//cmdline should be ready. If then there are problems reading it, there are likely other issues which won?t go away by waiting. And for these cases the fallback is to use the attach framework, as implemented in ProcessArgumentMatcher, which provides some chance to be working still. And this fallback should also cover the exotic case when jcmd is issued too early. After all, ProcessHelper::getMainClass also documents that its result can be null. @Thomas, as for your other points: * PID reusage: Hm, maybe one can construct cases. However, I?d think the /proc/pid files should be gone after a process ends. Or at least be reconstructed if there were orphans and a new process reusing an old pid gets started. But who knows what can happen ? we?ll maybe see ?? * Comment for Linux only issue: The test is in fact a Linux only test. See line 55: * @requires os.family == "linux". So, if we?ll eventually see implementations for ProcessHelper::getMainClass on other platforms, this comment might have to be adopted. But for the time being I guess it?s fine at its current place. Would you agree? Best regards Christoph From: Chris Plummer > Sent: Mittwoch, 11. September 2019 19:21 To: Thomas St?fe >; Langer, Christoph > Cc: OpenJDK Serviceability > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently It does seem that the fix should be in ProcessHelper.java in getMainClass(), or maybe even getCommandLine(). Fixing it in the test implies that every user of getMainClass() should be doing something similar. But then also note what ProcessArgumentMatch.check() is doing. It also deals with getMainClass() returing null. thanks, Chris On 9/11/19 6:59 AM, Thomas St?fe wrote: Hi Christoph, in general I think this is fine. The increase-by-pow2 sleep time is odd but okay :) The whole things seems rather fragile and has a lot of question marks but I think your fix does not make it worse. One fun error now is that with a follow up java test reusing the PID we could get a wrong main class but I think the chances are astronomically low. Only remark, you fix this in the platform shared code, if this is a Linux only issue maybe it should be fixed in /shared/projects/openjdk/jdk-jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java instead? If not, I would remove at least the /proc//cmdline comment since this is quite platform specific. Cheers, Thomas On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph > wrote: Hi, please review this change for test sun/tools/jcmd/TestProcessHelper.java to make it more robust. Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ This Linux only test is starting several Java processes and then tries to figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper implementation which parses the contents of /proc//cmdline. Under some circumstances, the test already attempts to read /proc//cmdline before it actually exists or is filled with data. This can be fixed with some sleeps/retries to wait for that data to be ready. In the actual jcmd tool, such behavior of ProcessHelper. getMainClass should not be an issue because it is handled in ProcessArgumentMatcher [0]. Thanks Christoph [0] http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgehwolf at redhat.com Thu Sep 12 09:00:52 2019 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Thu, 12 Sep 2019 11:00:52 +0200 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: Message-ID: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> Hi Christoph, Have you considered to wait for TestProcess - the spawned processes - to print this on stdout: "The process started, pid: XXX" Once that's ready on stdout, checking the main class should always pass. I believe p.isAlive() check which is currrently done is insufficient. Thanks, Severin On Thu, 2019-09-12 at 08:12 +0000, Langer, Christoph wrote: > Hi Thomas, > > sounds reasonable, will do. > > Thanks > Christoph > > From: Thomas St?fe > Sent: Donnerstag, 12. September 2019 10:11 > To: Langer, Christoph > Cc: Chris Plummer ; OpenJDK Serviceability > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently > > I'm fine with the patch if you would reshape the platform dependent comment. Proposal: > > ---- > - // Depending on hw/os, process helper can return null here > - // because /proc//cmdline is not ready yet. To cover that case, > // give it some retries. > -> > + getMainClass() may return NULL, e.g. due to timing issues. Attempt some limited retries. > ---- > I do not need another webrev. > Cheers, Thomas > > > > > On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph wrote: > > Hi Chris, Thomas, > > > > thanks for looking at this. I was also wondering whether a fix in ProcessHelper would be appropriate. But I think introducing retries and delays in that code can do more harm than help. > > > > For this special test case, aiming to test the ProcessHelper functionality (on Linux) only, the observed problem is that the /proc//cmdline file is not ready yet when it gets evaluated because the test can be quicker than the spawned processes. But in real life usage of jcmd this seems rather unlikely. One will probably use jcmd quite some time after a java process was started and /proc//cmdline should be ready. If then there are problems reading it, there are likely other issues which won?t go away by waiting. And for these cases the fallback is to use the attach framework, as implemented in ProcessArgumentMatcher, which provides some chance to be working still. And this fallback should also cover the exotic case when jcmd is issued too early. > > > > After all, ProcessHelper::getMainClass also documents that its result can be null. > > > > @Thomas, as for your other points: > > PID reusage: Hm, maybe one can construct cases. However, I?d think the /proc/pid files should be gone after a process ends. Or at least be reconstructed if there were orphans and a new process reusing an old pid gets started. But who knows what can happen ? we?ll maybe see ?? > > Comment for Linux only issue: The test is in fact a Linux only test. See line 55: * @requires os.family == "linux". So, if we?ll eventually see implementations for ProcessHelper::getMainClass on other platforms, this comment might have to be adopted. But for the time being I guess it?s fine at its current place. > > > > Would you agree? > > > > Best regards > > Christoph > > > > From: Chris Plummer > > Sent: Mittwoch, 11. September 2019 19:21 > > To: Thomas St?fe ; Langer, Christoph > > Cc: OpenJDK Serviceability > > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently > > > > It does seem that the fix should be in ProcessHelper.java in getMainClass(), or maybe even getCommandLine(). Fixing it in the test implies that every user of getMainClass() should be doing something similar. But then also note what ProcessArgumentMatch.check() is doing. It also deals with getMainClass() returing null. > > > > thanks, > > > > Chris > > > > On 9/11/19 6:59 AM, Thomas St?fe wrote: > > > Hi Christoph, > > > > > > in general I think this is fine. The increase-by-pow2 sleep time is odd but okay :) > > > > > > The whole things seems rather fragile and has a lot of question marks but I think your fix does not make it worse. One fun error now is that with a follow up java test reusing the PID we could get a wrong main class but I think the chances are astronomically low. > > > > > > Only remark, you fix this in the platform shared code, if this is a Linux only issue maybe it should be fixed in /shared/projects/openjdk/jdk-jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java instead? If not, I would remove at least the /proc//cmdline comment since this is quite platform specific. > > > > > > Cheers, Thomas > > > > > > > > > On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph wrote: > > > > Hi, > > > > > > > > please review this change for test sun/tools/jcmd/TestProcessHelper.java to make it more robust. > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > > > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > > > > > > > > This Linux only test is starting several Java processes and then tries to figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper implementation which parses the contents of /proc//cmdline. > > > > Under some circumstances, the test already attempts to read /proc//cmdline before it actually exists or is filled with data. This can be fixed with some sleeps/retries to wait for that data to be ready. > > > > In the actual jcmd tool, such behavior of ProcessHelper. getMainClass should not be an issue because it is handled in ProcessArgumentMatcher [0]. > > > > > > > > Thanks > > > > Christoph > > > > > > > > [0] http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java#l86 > > > > > > > > From christoph.langer at sap.com Thu Sep 12 09:30:41 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 12 Sep 2019 09:30:41 +0000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> Message-ID: Hi David, > > please review an enhancement which I've identified when working with > > Processhelper for JDK-8230850. > > > > I noticed that ProcessHelper is an interface in common code with a > > static method that would lookup the actual platform implementation via > > reflection. This seems a little cumbersome since we can have a common > > dummy for ProcessHelper and override it with the platform specific > > implementation, leveraging the build system. > > I don't see you leveraging the build system. You have two source files > that compile to the same destination class file. What is ensuring the > platform specific version is compiled after the generic one? > > Service-provider patterns use reflection to instantiate the service > implementation. I don't see any problem here that needs solving. TL;DR: There are two source files, one in share/classes and one in linux/classes. The build system overrides the share/classes implementation with the linux/classes implementation in the linux build. This is not by coincidence and only one class is contained in the generated jdk.jcmd module. Then there won't be a need for having a service interface and a service implementation that is looked up via reflection (which is not a bad pattern by itself). I agree that it's not a big problem to be solved but still not "no problem". Here is some longer elaboration how the build system prefers specific implementations of classes and filters generic duplicates: The SetupJavaCompilation function from JavaCompilation.gmk [0] is used to compile the java sources for JDK modules. In its documentation, for argument SRC [1], it claims: "one or more directories to search for sources. The order of the source roots is significant. The first found file of a certain name has priority". In its implementation the found files are first ordered [3] and duplicates filtered out [4]. The potential source files are handed to SetupJavaCompilation in CompileJavaModules.gmk [5] and were collected by a call to FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all potential source dirs for Java classes in the module [7]. The evaluated subdirs are (in that order) $(OPENJDK_TARGET_OS)/classes, $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. Hope that explains what I'm trying to leverage here. I've uploaded an updated webrev which contains some cleanup to the Test changes: http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ Thanks Christoph [0] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l185 [1] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l157 [3] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l225 [4] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l257 [5] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l603 [6] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l555 [7] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l300 [8] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l243 From matthias.baesken at sap.com Thu Sep 12 10:11:04 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Thu, 12 Sep 2019 10:11:04 +0000 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code Message-ID: Hello, please reviews this small change . It adds ReleaseStringUTFChars calls at some places in early return cases . ( in src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp THROW_NEW_DEBUGGER_EXCEPTION contains a return , see the macro declaration 39 #define THROW_NEW_DEBUGGER_EXCEPTION(str) { throwNewDebuggerException(env, str); return;} ) Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8230901 http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Sep 12 10:21:49 2019 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 12 Sep 2019 12:21:49 +0200 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code In-Reply-To: References: Message-ID: Hi Matthias, your changes look good. an additional bug: http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 698 #ifndef _LP64 699 atoi(cmdLine_cstr); 700 if (errno) { Behaviour of atoi() in error case is undefined. errno values are not defined. See: https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html And even if atoi would set errno, this is still not enough since errno may contain a stale value. One would have to set errno=0 before the function call. If you want to fix this too 'd suggest replacing this call with strtol(). Cheers, Thomas On Thu, Sep 12, 2019 at 12:11 PM Baesken, Matthias wrote: > Hello, please reviews this small change . > > > > It adds ReleaseStringUTFChars calls at some places in early return > cases . > > ( in src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp > > > > THROW_NEW_DEBUGGER_EXCEPTION contains a return , see the macro declaration > > > > 39 #define THROW_NEW_DEBUGGER_EXCEPTION(str) { throwNewDebuggerException(env, str); return;} > > ) > > > > > > > > Bug/webrev : > > > > https://bugs.openjdk.java.net/browse/JDK-8230901 > > > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/ > > > > Thanks, Matthias > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.baesken at sap.com Thu Sep 12 11:52:18 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Thu, 12 Sep 2019 11:52:18 +0000 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code In-Reply-To: References: Message-ID: Hi Thomas, thanks for the review . You are correct about atoi . New webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/ I had 2 additional observations : 1. With OJDK on solaris 32bit gone for quite some time, we might be able to kick out the whole non _LP64 code because we are always 64 bit (maybe someone could comment if this is a safe assumption, there might be old 32bit solaris core files flying around for some reason even these days ? ) http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 696 // some older versions of libproc.so crash when trying to attach 32 bit 697 // debugger to 64 bit core file. check and throw error. 698 #ifndef _LP64 ?.. 1. The usage of atoi is commented here : https://docs.oracle.com/cd/E86824_01/html/E54766/atoi-3c.html ?However, applications should not use the atoi(), atol(), or atoll() functions unless they know the value represented by the argument will be in range for the corresponding result type? ??And here : https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html ?If the number is not known to be in range, strtol() should be used because atoi() is not required to perform any error checking? However we have a number of usages in the coding where atoi is called without knowing that the argument is in the allowed range . some examples : src/hotspot/share/runtime/arguments.cpp-382- if (match_option(option, "-Dsun.java.launcher.pid=", &tail)) { src/hotspot/share/runtime/arguments.cpp:383: _sun_java_launcher_pid = atoi(tail); src/hotspot/share/runtime/arguments.cpp-384- continue; src/java.desktop/unix/native/libawt_xawt/xawt/XToolkit.c 455 value = getenv("_AWT_MAX_POLL_TIMEOUT"); 456 if (value != NULL) { 457 AWT_MAX_POLL_TIMEOUT = atoi(value); src/java.desktop/unix/native/common/awt/X11Color.c-781- if (getenv("CMAPSIZE") != 0) { src/java.desktop/unix/native/common/awt/X11Color.c:782: cmapsize = atoi(getenv("CMAPSIZE")); Should I open a bug for these ? Best regards, Matthias From: Thomas St?fe Sent: Donnerstag, 12. September 2019 12:22 To: Baesken, Matthias Cc: serviceability-dev at openjdk.java.net Subject: Re: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code Hi Matthias, your changes look good. an additional bug: http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 698 #ifndef _LP64 699 atoi(cmdLine_cstr); 700 if (errno) { Behaviour of atoi() in error case is undefined. errno values are not defined. See: https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html And even if atoi would set errno, this is still not enough since errno may contain a stale value. One would have to set errno=0 before the function call. If you want to fix this too 'd suggest replacing this call with strtol(). Cheers, Thomas On Thu, Sep 12, 2019 at 12:11 PM Baesken, Matthias > wrote: Hello, please reviews this small change . It adds ReleaseStringUTFChars calls at some places in early return cases . ( in src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp THROW_NEW_DEBUGGER_EXCEPTION contains a return , see the macro declaration 39 #define THROW_NEW_DEBUGGER_EXCEPTION(str) { throwNewDebuggerException(env, str); return;} ) Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8230901 http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From hohensee at amazon.com Thu Sep 12 15:13:56 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 12 Sep 2019 15:13:56 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <23ED3BD7-3B7E-43DE-84B8-DB52D44D362B@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <23ED3BD7-3B7E-43DE-84B8-DB52D44D362B@amazon.com> Message-ID: Ping once more :) Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. Thanks, Paul ?On 9/6/19, 11:08 AM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: Ping. Anyone? ( Thanks, On 9/3/19, 12:39 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I removed ensureNonNullThreadIds() in favor of Objects.requireNonNull(ids). Thanks, Mandy, for your through reviews. May I get another reviewer to weigh in? Paul On 8/31/19, 5:06 PM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: Thanks, Mandy. I?ve finalized the CSR. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.04/. In management.cpp, I now have if (THREAD->is_Java_thread()) { return ((JavaThread*)THREAD)->cooked_allocated_bytes(); } In ThreadImpl.java, using requireNonNull would produce a different and less informative message, so I?d like to leave it as is. I changed throwIfNullThreadIds to ensureNonNullThreadIds, and throwIfThreadAllocatedMemoryNotSupported to ensureThreadAllocatedMemorySupported. I dropped the ?java.lang.? prefix from all uses of UnsupportedOperationException in both c.s.m.ThreadMXBean.java and j.l.m.ThreadMXBean.java, and did the same with SecurityException. ?@since 14? added to c.s.m.ThreadMXBean.java and the CSR. Do I need another reviewer? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 4:26 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread CSR reviewed. management.cpp 2083 java_thread = (JavaThread*)THREAD; 2084 if (java_thread->is_Java_thread()) { 2085 return java_thread->cooked_allocated_bytes(); 2086 } The cast should be done after is_Java_thread() test. ThreadImpl.java 162 private void throwIfNullThreadIds(long[] ids) { Even better: simply use Objects::requiresNonNull and this method can be removed. This suggests positive naming alternative to throwIfThreadAllocatedMemoryNotSupported - "ensureThreadAllocatedMemorySupported" (sorry I should have suggested that) ThreadMXBean.java 130 * @throws java.lang.UnsupportedOperationException if the Java virtual Nit: "java.lang." can be dropped. @since 14 is missing. Mandy On 8/30/19 3:33 PM, Hohensee, Paul wrote: Thanks for your review, Mandy. Revised webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.02/. I updated the CSR with your suggested javadoc for getCurrentThreadAllocatedBytes. It now matches that for getCurrentThreadUserTime and getCurrentThreadCputime. I also fixed the ?convenient? -> ?convenience? typos in j.l.m.ThreadMXBean.java. I meant GetOneThreads to be the possessive, but don?t feel strongly either way so I?m fine with GetOneThread. I updated ThreadImpl.java as you suggested, though in getThreadAllocatedBytes(long[] ids) I had to add a redundant-in-the-not-length-1-case check for a null ids reference. Would someone take a look at the Hotspot side and the test please? Paul From: Mandy Chung Date: Friday, August 30, 2019 at 10:22 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread OK. That's better. Some review comments: The javadoc of getCurrentThreadAllocatedBytes() can simply say: "Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the current thread. This is a convenient method for local management use and is equivalent to calling getThreadAllocatedBytes(Thread.currentThread().getId()). src/hotspot/share/include/jmm.h GetOneThreadsAllocatedMemory: s/OneThreads/OneThread/ sun/management/ThreadImpl.java 43 private static final String THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED = 44 "Thread allocated memory measurement is not supported."; if (!isThreadAllocatedMemorySupported()) { throw new UnsupportedOperationException(THREAD_ALLOCATED_MEMORY_NOT_SUPPORTED); } Perhaps the above can be refactored as throwIfAllocatedMemoryUnsupported() method. 391 if (ids.length == 1) { 392 sizes[0] = -1; : 398 if (ids.length == 1) { 399 long id = ids[0]; 400 sizes[0] = getThreadAllocatedMemory0( 401 Thread.currentThread().getId() == id ? 0 : id); 402 } else { It seems cleaner to handle the 1-element array case at the beginning of this method: if (ids.length == 1) { long size = getThreadAllocatedBytes(ids[0]); return new long[] { size }; } I didn't review the hotspot implementation and the test. Mandy On 8/29/19 10:01 AM, Hohensee, Paul wrote: My bad, Mandy. The webrev puts getCurrentThreadAllocatedBytes in com.sun.management.ThreadMXBean along with the current two getThreadAllocatedBytes methods for the reasons you list. I?ve updated the CSR to specify com.sun.management and added a rationale. AllocatedBytes is currently enabled by Hotspot by default because the overhead of recording TLAB occupancy is negligible. There?s no new GC code, nor will there be, so imo we don?t have to involve the GC folks. I.e., the new JMM method GetOneThreadsAllocatedBytes uses the existing cooked_allocated_bytes JavaThread method, and getCurrentThreadAllocatedBytes is the same as getThreadAllocatedBytes: it just bypasses the thread lookup code. I hadn?t tracked down what happens when getCurrentThreadUserTime and getCurrentThreadCpuTime are called before, but if I?m not mistaken, it the code in jcmd() in attachListener.cpp will call GetThreadCpuTimeWithKind in management.cpp, and it will ultimately use Thread::current() as the subject of the call, see os::current_thread_cpu_time in os_linux.cpp. That means that the CurrentThread methods should work remotely the same way they do locally. GetOneThreadsAllocatedBytes in management.cpp uses THREAD as its subject when called on behalf of getCurrentThreadAllocatedBytes, so it will also uses the current remote Java thread. Even if these methods only worked locally, there are many setups where apps are self-monitoring that could use the performance improvement. Thanks, Paul From: Mandy Chung Date: Wednesday, August 28, 2019 at 3:59 PM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, The CSR proposes this method in java.lang.management.ThreadMXBean as a Java SE feature. Has this been discussed with the GC team to commit measuring current thread's allocated bytes as Java SE feature? Can this be supported by all JVM implementation? What is the overhead if this is enabled by default? Does it need to be disabled? This metric is from TLAB that might be okay. This needs advice/discussion with GC experts. I see that CSR mentions it can be disabled and link to isThreadAllocatedMemoryEnabled() and setThreadAllocatedMemoryEnabled() methods but these methods are defined in com.sun.management.ThreadMXBean. As Alan points out, current thread makes sense only in local VM management. When this is monitored from a JMX client (e.g. jconsole to connect to a running JVM, "currentThreadAllowcatedBytes" attribute is the current thread in jconsole process which invoking Thread::currentThread? Mandy On 8/28/19 12:22 PM, Hohensee, Paul wrote: Please review a performance improvement for ThreadMXBean.getThreadAllocatedBytes and the addition of getCurrentThreadAllocatedBytes. JBS issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Webrev: http://cr.openjdk.java.net/~phh/8207266/webrev.00/ CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 Previous email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-July/024441.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2018-August/024763.html The CSR is for adding ThreadMXBean.getCurrentThreadAllocatedBytes. I?d be great for someone to review it. I took Mandy?s advice and put the fast paths in the library code. I added a new JMM method GetOneThreadsAllocatedBytes that works the same as GetThreadCpuTime: it uses a thread_id value of zero to distinguish the current thread. On my Mac laptop, the result runs 47x faster for the current thread than the old implementation. The 3 tests in test/jdk/com/sun/management/ThreadMXBean all pass. I added code to ThreadAllocatedMemory.java to test getCurrentThreadAllocatedBytes as well as variations on getThreadAllocatedBytes(id). A submit repo job is in progress. Thanks, Paul From leonid.mesnik at oracle.com Thu Sep 12 16:56:44 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Thu, 12 Sep 2019 09:56:44 -0700 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file Message-ID: Hi Could you please verify following fix which update core filename patterns. Some hosts are configured to compress core files during core dump. In such cases tests should throw skipped exception instead of failing reading compressed cores. Fixing all tests to support unpacking core.pid.gz files is going to be fixed in separate RFE. Verified that test pass if plain core is generated and skipped in other case. webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8230881 Leonid -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandy.chung at oracle.com Thu Sep 12 17:06:48 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 12 Sep 2019 10:06:48 -0700 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> Message-ID: <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> On 9/3/19 12:38 PM, Hohensee, Paul wrote: > Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I only reviewed the library side implementation that looks good.? I expect the serviceability team to review the test and hotspot change. > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. You need another reviewer to advice the following because I was not close to the ThreadsList work. 2087 ThreadsListHandle tlh; 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); 2089 2090 if (java_thread != NULL) { 2091 return java_thread->cooked_allocated_bytes(); 2092 } This looks right to me. test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java - "ThreadAllocatedMemory is expected to be disabled"); + "TEST FAILED: ThreadAllocatedMemory is expected to be disabled"); Prepending "TEST FAILED" in exception message (in several places) seems redundant since such RuntimeException is thrown and expected a test failure. + // back-to-back calls shouldn't allocate any memory + size = mbean.getThreadAllocatedBytes(id); + size1 = mbean.getThreadAllocatedBytes(id); + if (size1 != size) { Is there anything in the test can do to help guarantee this? I didn't closely review this test. The main thing I advice is to improve the reliability of this test. Put it in another way, we want to ensure that this test change will pass all the time in various test configuration. Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Sep 12 19:53:18 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 12 Sep 2019 19:53:18 +0000 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> References: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> Message-ID: Hi Severin, that seems an interesting idea for an elegant solution. However, after trying this on a decently fast linux x86 box by leveraging one of these ProcessTools::startProcess methods that would wait for a certain output to appear in the child before returning, I figured that the elapsed runtime of the test increases from about 3 seconds to 3 minutes. It evidently takes a lot longer to bootstrap a JVM and get the results of a first println than just forking a process and immediately accessing its proc filesystem. So I think we don't want to do that. I would like to go with this version (changed the comment to Thomas' suggestion): http://cr.openjdk.java.net/~clanger/webrevs/8230850.1/ Chris, are you ok with it? Thanks Christoph > -----Original Message----- > From: Severin Gehwolf > Sent: Donnerstag, 12. September 2019 11:01 > To: Langer, Christoph ; Thomas St?fe > > Cc: OpenJDK Serviceability > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java > fails intermittently > > Hi Christoph, > > Have you considered to wait for TestProcess - the spawned processes - to > print this on stdout: > > "The process started, pid: XXX" > > Once that's ready on stdout, checking the main class should always > pass. I believe p.isAlive() check which is currrently done is > insufficient. > > Thanks, > Severin > > On Thu, 2019-09-12 at 08:12 +0000, Langer, Christoph wrote: > > Hi Thomas, > > > > sounds reasonable, will do. > > > > Thanks > > Christoph > > > > From: Thomas St?fe > > Sent: Donnerstag, 12. September 2019 10:11 > > To: Langer, Christoph > > Cc: Chris Plummer ; OpenJDK Serviceability > > > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java > fails intermittently > > > > I'm fine with the patch if you would reshape the platform dependent > comment. Proposal: > > > > ---- > > - // Depending on hw/os, process helper can return null here > > - // because /proc//cmdline is not ready yet. To cover that case, > > // give it some retries. > > -> > > + getMainClass() may return NULL, e.g. due to timing issues. Attempt some > limited retries. > > ---- > > I do not need another webrev. > > Cheers, Thomas > > > > > > > > > > On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph > wrote: > > > Hi Chris, Thomas, > > > > > > thanks for looking at this. I was also wondering whether a fix in > ProcessHelper would be appropriate. But I think introducing retries and > delays in that code can do more harm than help. > > > > > > For this special test case, aiming to test the ProcessHelper functionality > (on Linux) only, the observed problem is that the /proc//cmdline file is > not ready yet when it gets evaluated because the test can be quicker than > the spawned processes. But in real life usage of jcmd this seems rather > unlikely. One will probably use jcmd quite some time after a java process was > started and /proc//cmdline should be ready. If then there are > problems reading it, there are likely other issues which won?t go away by > waiting. And for these cases the fallback is to use the attach framework, as > implemented in ProcessArgumentMatcher, which provides some chance to > be working still. And this fallback should also cover the exotic case when jcmd > is issued too early. > > > > > > After all, ProcessHelper::getMainClass also documents that its result can > be null. > > > > > > @Thomas, as for your other points: > > > PID reusage: Hm, maybe one can construct cases. However, I?d think the > /proc/pid files should be gone after a process ends. Or at least be > reconstructed if there were orphans and a new process reusing an old pid > gets started. But who knows what can happen ? we?ll maybe see ?? > > > Comment for Linux only issue: The test is in fact a Linux only test. See line > 55: * @requires os.family == "linux". So, if we?ll eventually see > implementations for ProcessHelper::getMainClass on other platforms, this > comment might have to be adopted. But for the time being I guess it?s fine at > its current place. > > > > > > Would you agree? > > > > > > Best regards > > > Christoph > > > > > > From: Chris Plummer > > > Sent: Mittwoch, 11. September 2019 19:21 > > > To: Thomas St?fe ; Langer, Christoph > > > > Cc: OpenJDK Serviceability > > > Subject: Re: RFR (S): 8230850: Test > sun/tools/jcmd/TestProcessHelper.java fails intermittently > > > > > > It does seem that the fix should be in ProcessHelper.java in > getMainClass(), or maybe even getCommandLine(). Fixing it in the test > implies that every user of getMainClass() should be doing something similar. > But then also note what ProcessArgumentMatch.check() is doing. It also > deals with getMainClass() returing null. > > > > > > thanks, > > > > > > Chris > > > > > > On 9/11/19 6:59 AM, Thomas St?fe wrote: > > > > Hi Christoph, > > > > > > > > in general I think this is fine. The increase-by-pow2 sleep time is odd > but okay :) > > > > > > > > The whole things seems rather fragile and has a lot of question marks > but I think your fix does not make it worse. One fun error now is that with a > follow up java test reusing the PID we could get a wrong main class but I think > the chances are astronomically low. > > > > > > > > Only remark, you fix this in the platform shared code, if this is a Linux > only issue maybe it should be fixed in /shared/projects/openjdk/jdk- > jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java > instead? If not, I would remove at least the /proc//cmdline comment > since this is quite platform specific. > > > > > > > > Cheers, Thomas > > > > > > > > > > > > On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph > wrote: > > > > > Hi, > > > > > > > > > > please review this change for test > sun/tools/jcmd/TestProcessHelper.java to make it more robust. > > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > > > > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > > > > > > > > > > This Linux only test is starting several Java processes and then tries to > figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper > implementation which parses the contents of /proc//cmdline. > > > > > Under some circumstances, the test already attempts to read > /proc//cmdline before it actually exists or is filled with data. This can be > fixed with some sleeps/retries to wait for that data to be ready. > > > > > In the actual jcmd tool, such behavior of ProcessHelper. getMainClass > should not be an issue because it is handled in ProcessArgumentMatcher [0]. > > > > > > > > > > Thanks > > > > > Christoph > > > > > > > > > > [0] > http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/cl > asses/sun/tools/common/ProcessArgumentMatcher.java#l86 > > > > > > > > > > > From david.holmes at oracle.com Thu Sep 12 23:46:50 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Sep 2019 09:46:50 +1000 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file In-Reply-To: References: Message-ID: Looks good! Thanks for fixing. David On 13/09/2019 2:56 am, Leonid Mesnik wrote: > Hi > > Could you please verify following fix which update core filename > patterns. Some hosts are configured to compress core files during core > dump. In such cases tests should throw skipped exception instead of > failing reading compressed cores. > Fixing all tests to support unpacking core.pid.gz files is going to be > fixed in separate RFE. > > > Verified that test pass if plain core is generated and skipped in other > case. > > > webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8230881 > > Leonid From hohensee at amazon.com Fri Sep 13 00:29:49 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 13 Sep 2019 00:29:49 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> Message-ID: Thanks for clarifying the review rules. Would someone from the serviceability team please review? New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.07/ I didn?t disturb the existing checks in the test, just added code to check the result of getThreadAllocatedBytes(long) on a non-current thread, plus the back-to-back no-allocation checks. The former wasn?t needed before because getThreadAllocatedBytes(long) was just a wrapper around getThreadAllocatedBytes(long []). This patch changes that, so I added a separate test. The latter is supposed to fail if there?s object allocation on calls to getCurrentThreadAllocatedBytes and getThreadAllocatedBytes(long). I.e., a feature, not a bug, because accumulation of transient small objects can be a performance problem. Thanks to your review, I noticed that the back-to-back check on the current thread was using getThreadAllocatedBytes(long) instead of getCurrentThreadAllocatedBytes and fixed it. I also removed all instances of ?TEST FAILED: ?. Paul From: Mandy Chung Date: Thursday, September 12, 2019 at 10:09 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread On 9/3/19 12:38 PM, Hohensee, Paul wrote: Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I only reviewed the library side implementation that looks good. I expect the serviceability team to review the test and hotspot change. Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. You need another reviewer to advice the following because I was not close to the ThreadsList work. 2087 ThreadsListHandle tlh; 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); 2089 2090 if (java_thread != NULL) { 2091 return java_thread->cooked_allocated_bytes(); 2092 } This looks right to me. test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java - "ThreadAllocatedMemory is expected to be disabled"); + "TEST FAILED: ThreadAllocatedMemory is expected to be disabled"); Prepending "TEST FAILED" in exception message (in several places) seems redundant since such RuntimeException is thrown and expected a test failure. + // back-to-back calls shouldn't allocate any memory + size = mbean.getThreadAllocatedBytes(id); + size1 = mbean.getThreadAllocatedBytes(id); + if (size1 != size) { Is there anything in the test can do to help guarantee this? I didn't closely review this test. The main thing I advice is to improve the reliability of this test. Put it in another way, we want to ensure that this test change will pass all the time in various test configuration. Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Fri Sep 13 00:53:06 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 12 Sep 2019 17:53:06 -0700 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file In-Reply-To: References: Message-ID: <6d007e8f-7696-841e-0637-033631262248@oracle.com> Looks good. Chris On 9/12/19 9:56 AM, Leonid Mesnik wrote: > Hi > > Could you please verify following fix which update core filename > patterns. Some hosts are configured to compress core files during core > dump. In such cases tests should throw skipped exception instead of > failing reading compressed cores. > Fixing all tests to support unpacking core.pid.gz files is going to be > fixed in separate RFE. > > > Verified that test pass if plain core is generated and skipped in > other case. > > > webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8230881 > > Leonid From hohensee at amazon.com Fri Sep 13 01:52:50 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 13 Sep 2019 01:52:50 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> Message-ID: And of course the back-to-back check is more or less accurate depending on the accuracy of the underlying Hotspot mechanism. So it?s possible (indeed likely with the current TLAB refill interval update) that it?ll return false negatives, but imo better to keep it anyway. From: "Hohensee, Paul" Date: Thursday, September 12, 2019 at 5:29 PM To: Mandy Chung Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Thanks for clarifying the review rules. Would someone from the serviceability team please review? New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.07/ I didn?t disturb the existing checks in the test, just added code to check the result of getThreadAllocatedBytes(long) on a non-current thread, plus the back-to-back no-allocation checks. The former wasn?t needed before because getThreadAllocatedBytes(long) was just a wrapper around getThreadAllocatedBytes(long []). This patch changes that, so I added a separate test. The latter is supposed to fail if there?s object allocation on calls to getCurrentThreadAllocatedBytes and getThreadAllocatedBytes(long). I.e., a feature, not a bug, because accumulation of transient small objects can be a performance problem. Thanks to your review, I noticed that the back-to-back check on the current thread was using getThreadAllocatedBytes(long) instead of getCurrentThreadAllocatedBytes and fixed it. I also removed all instances of ?TEST FAILED: ?. Paul From: Mandy Chung Date: Thursday, September 12, 2019 at 10:09 AM To: "Hohensee, Paul" Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread On 9/3/19 12:38 PM, Hohensee, Paul wrote: Minor update in new webrev http://cr.openjdk.java.net/~phh/8207266/webrev.05/. I only reviewed the library side implementation that looks good. I expect the serviceability team to review the test and hotspot change. Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. You need another reviewer to advice the following because I was not close to the ThreadsList work. 2087 ThreadsListHandle tlh; 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); 2089 2090 if (java_thread != NULL) { 2091 return java_thread->cooked_allocated_bytes(); 2092 } This looks right to me. test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java - "ThreadAllocatedMemory is expected to be disabled"); + "TEST FAILED: ThreadAllocatedMemory is expected to be disabled"); Prepending "TEST FAILED" in exception message (in several places) seems redundant since such RuntimeException is thrown and expected a test failure. + // back-to-back calls shouldn't allocate any memory + size = mbean.getThreadAllocatedBytes(id); + size1 = mbean.getThreadAllocatedBytes(id); + if (size1 != size) { Is there anything in the test can do to help guarantee this? I didn't closely review this test. The main thing I advice is to improve the reliability of this test. Put it in another way, we want to ensure that this test change will pass all the time in various test configuration. Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Fri Sep 13 07:50:27 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Sep 2019 17:50:27 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> Message-ID: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> Hi Paul, On 13/09/2019 10:29 am, Hohensee, Paul wrote: > Thanks for clarifying the review rules. Would someone from the > serviceability team please review? New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ One aspect of the functional change needs clarification for me - and apologies if this has been covered in the past. It seems to me that currently we only check isThreadAllocatedMemorySupported for these operations, but if I read things correctly the updated code additionally checks isThreadAllocatedMemoryEnabled, which is a behaviour change not mentioned in the CSR. > I didn?t disturb the existing checks in the test, just added code to > check the result of getThreadAllocatedBytes(long) on a non-current > thread, plus the back-to-back no-allocation checks. The former wasn?t > needed before because getThreadAllocatedBytes(long) was just a wrapper > around getThreadAllocatedBytes(long []). This patch changes that, so I > added a separate test. The latter is supposed to fail if there?s object > allocation on calls to getCurrentThreadAllocatedBytes and > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > accumulation of transient small objects can be a performance problem. > Thanks to your review, I noticed that the back-to-back check on the > current thread was using getThreadAllocatedBytes(long) instead of > getCurrentThreadAllocatedBytes and fixed it. I also removed all > instances of ?TEST FAILED: ?. The back-to-back check is not valid in general. You don't know if the first check might trigger some class loading on the return path after it has obtained the first memory value. The check might also fail if using JVMCI and some compilation related activity occurs in the current thread on the second call. Also with the introduction of handshakes its possible the current thread might hit a safepoint checks that results in it executing a handshake operation that performs allocation. Potentially there could be numerous non-deterministic actions that might occur leading to unanticipated allocation. I understand what you want to test here, I just don't think it is reliably doable. Thanks, David ----- > > Paul > > *From: *Mandy Chung > *Date: *Thursday, September 12, 2019 at 10:09 AM > *To: *"Hohensee, Paul" > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > I only reviewed the library side implementation that looks good.? I > expect the serviceability team to review the test and hotspot change. > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > You need another reviewer to advice the following because I was not > close to the ThreadsList work. > > 2087?? ThreadsListHandle tlh; > > 2088?? JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > 2089 > > 2090?? if (java_thread != NULL) { > > 2091???? return java_thread->cooked_allocated_bytes(); > > 2092?? } > > This looks right to me. > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > -??????????????? "ThreadAllocatedMemory is expected to be disabled"); > > +??????????????? "TEST FAILED: ThreadAllocatedMemory is expected to be > disabled"); > > Prepending "TEST FAILED" in exception message (in several places) > > seems redundant since such RuntimeException is thrown and expected > > a test failure. > > +??????? // back-to-back calls shouldn't allocate any memory > > +??????? size = mbean.getThreadAllocatedBytes(id); > > +??????? size1 = mbean.getThreadAllocatedBytes(id); > > +??????? if (size1 != size) { > > Is there anything in the test can do to help guarantee this? I didn't > > closely review this test.? The main thing I advice is to improve > > the reliability of this test.? Put it in another way, we want to > > ensure that this test change will pass all the time in various > > test configuration. > > Mandy > From sgehwolf at redhat.com Fri Sep 13 08:33:10 2019 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 13 Sep 2019 10:33:10 +0200 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> Message-ID: <44f49ad20c8c0b5cf2141b82f3fe6db6a9088857.camel@redhat.com> On Thu, 2019-09-12 at 19:53 +0000, Langer, Christoph wrote: > Hi Severin, > > that seems an interesting idea for an elegant solution. However, > after trying this on a decently fast linux x86 box by leveraging one > of these ProcessTools::startProcess methods that would wait for a > certain output to appear in the child before returning, I figured > that the elapsed runtime of the test increases from about 3 seconds > to 3 minutes. It evidently takes a lot longer to bootstrap a JVM and > get the results of a first println than just forking a process and > immediately accessing its proc filesystem. So I think we don't want > to do that. Wow, that's unfortunate. > I would like to go with this version (changed the comment to Thomas' > suggestion): http://cr.openjdk.java.net/~clanger/webrevs/8230850.1/ Patch seems OK in this case. Thanks, Severin > Chris, are you ok with it? > > Thanks > Christoph > > > -----Original Message----- > > From: Severin Gehwolf > > Sent: Donnerstag, 12. September 2019 11:01 > > To: Langer, Christoph ; Thomas St?fe > > > > Cc: OpenJDK Serviceability > > Subject: Re: RFR (S): 8230850: Test > > sun/tools/jcmd/TestProcessHelper.java > > fails intermittently > > > > Hi Christoph, > > > > Have you considered to wait for TestProcess - the spawned processes > > - to > > print this on stdout: > > > > "The process started, pid: XXX" > > > > Once that's ready on stdout, checking the main class should always > > pass. I believe p.isAlive() check which is currrently done is > > insufficient. > > > > Thanks, > > Severin > > > > On Thu, 2019-09-12 at 08:12 +0000, Langer, Christoph wrote: > > > Hi Thomas, > > > > > > sounds reasonable, will do. > > > > > > Thanks > > > Christoph > > > > > > From: Thomas St?fe > > > Sent: Donnerstag, 12. September 2019 10:11 > > > To: Langer, Christoph > > > Cc: Chris Plummer ; OpenJDK > > > Serviceability > > > > > Subject: Re: RFR (S): 8230850: Test > > > sun/tools/jcmd/TestProcessHelper.java > > fails intermittently > > > I'm fine with the patch if you would reshape the platform > > > dependent > > comment. Proposal: > > > ---- > > > - // Depending on hw/os, process helper can return null > > > here > > > - // because /proc//cmdline is not ready yet. To > > > cover that case, > > > // give it some retries. > > > -> > > > + getMainClass() may return NULL, e.g. due to timing issues. > > > Attempt some > > limited retries. > > > ---- > > > I do not need another webrev. > > > Cheers, Thomas > > > > > > > > > > > > > > > On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph > > wrote: > > > > Hi Chris, Thomas, > > > > > > > > thanks for looking at this. I was also wondering whether a fix > > > > in > > ProcessHelper would be appropriate. But I think introducing retries > > and > > delays in that code can do more harm than help. > > > > For this special test case, aiming to test the ProcessHelper > > > > functionality > > (on Linux) only, the observed problem is that the > > /proc//cmdline file is > > not ready yet when it gets evaluated because the test can be > > quicker than > > the spawned processes. But in real life usage of jcmd this seems > > rather > > unlikely. One will probably use jcmd quite some time after a java > > process was > > started and /proc//cmdline should be ready. If then there are > > problems reading it, there are likely other issues which won?t go > > away by > > waiting. And for these cases the fallback is to use the attach > > framework, as > > implemented in ProcessArgumentMatcher, which provides some chance > > to > > be working still. And this fallback should also cover the exotic > > case when jcmd > > is issued too early. > > > > After all, ProcessHelper::getMainClass also documents that its > > > > result can > > be null. > > > > @Thomas, as for your other points: > > > > PID reusage: Hm, maybe one can construct cases. However, I?d > > > > think the > > /proc/pid files should be gone after a process ends. Or at least be > > reconstructed if there were orphans and a new process reusing an > > old pid > > gets started. But who knows what can happen ? we?ll maybe see ?? > > > > Comment for Linux only issue: The test is in fact a Linux only > > > > test. See line > > 55: * @requires os.family == "linux". So, if we?ll eventually see > > implementations for ProcessHelper::getMainClass on other platforms, > > this > > comment might have to be adopted. But for the time being I guess > > it?s fine at > > its current place. > > > > Would you agree? > > > > > > > > Best regards > > > > Christoph > > > > > > > > From: Chris Plummer > > > > Sent: Mittwoch, 11. September 2019 19:21 > > > > To: Thomas St?fe ; Langer, Christoph > > > > > > Cc: OpenJDK Serviceability > > > > > > > > Subject: Re: RFR (S): 8230850: Test > > sun/tools/jcmd/TestProcessHelper.java fails intermittently > > > > It does seem that the fix should be in ProcessHelper.java in > > getMainClass(), or maybe even getCommandLine(). Fixing it in the > > test > > implies that every user of getMainClass() should be doing something > > similar. > > But then also note what ProcessArgumentMatch.check() is doing. It > > also > > deals with getMainClass() returing null. > > > > thanks, > > > > > > > > Chris > > > > > > > > On 9/11/19 6:59 AM, Thomas St?fe wrote: > > > > > Hi Christoph, > > > > > > > > > > in general I think this is fine. The increase-by-pow2 sleep > > > > > time is odd > > but okay :) > > > > > The whole things seems rather fragile and has a lot of > > > > > question marks > > but I think your fix does not make it worse. One fun error now is > > that with a > > follow up java test reusing the PID we could get a wrong main class > > but I think > > the chances are astronomically low. > > > > > Only remark, you fix this in the platform shared code, if > > > > > this is a Linux > > only issue maybe it should be fixed in > > /shared/projects/openjdk/jdk- > > jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java > > instead? If not, I would remove at least the /proc//cmdline > > comment > > since this is quite platform specific. > > > > > Cheers, Thomas > > > > > > > > > > > > > > > On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph > > wrote: > > > > > > Hi, > > > > > > > > > > > > please review this change for test > > sun/tools/jcmd/TestProcessHelper.java to make it more robust. > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > > > > > > Webrev: > > > > > > http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > > > > > > > > > > > > This Linux only test is starting several Java processes and > > > > > > then tries to > > figure out the main class by invoking jdk.jcmd's linux specific > > ProcessHelper > > implementation which parses the contents of /proc//cmdline. > > > > > > Under some circumstances, the test already attempts to read > > /proc//cmdline before it actually exists or is filled with > > data. This can be > > fixed with some sleeps/retries to wait for that data to be ready. > > > > > > In the actual jcmd tool, such behavior of ProcessHelper. > > > > > > getMainClass > > should not be an issue because it is handled in > > ProcessArgumentMatcher [0]. > > > > > > Thanks > > > > > > Christoph > > > > > > > > > > > > [0] > > http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/cl > > asses/sun/tools/common/ProcessArgumentMatcher.java#l86 From matthias.baesken at sap.com Fri Sep 13 10:01:39 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Fri, 13 Sep 2019 10:01:39 +0000 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code References: Message-ID: Hello , my colleague Ralf pointed out that the NULL-check of the result of GetStringUTFChars should be done right after the GetStringUTFChars so I moved the NULL-check up : http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.2/ Best regards, Matthias Hi Thomas, thanks for the review . You are correct about atoi . New webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/ I had 2 additional observations : 1. With OJDK on solaris 32bit gone for quite some time, we might be able to kick out the whole non _LP64 code because we are always 64 bit (maybe someone could comment if this is a safe assumption, there might be old 32bit solaris core files flying around for some reason even these days ? ) http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 696 // some older versions of libproc.so crash when trying to attach 32 bit 697 // debugger to 64 bit core file. check and throw error. 698 #ifndef _LP64 ?.. 1. The usage of atoi is commented here : https://docs.oracle.com/cd/E86824_01/html/E54766/atoi-3c.html ?However, applications should not use the atoi(), atol(), or atoll() functions unless they know the value represented by the argument will be in range for the corresponding result type? ??And here : https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html ?If the number is not known to be in range, strtol() should be used because atoi() is not required to perform any error checking? However we have a number of usages in the coding where atoi is called without knowing that the argument is in the allowed range . some examples : src/hotspot/share/runtime/arguments.cpp-382- if (match_option(option, "-Dsun.java.launcher.pid=", &tail)) { src/hotspot/share/runtime/arguments.cpp:383: _sun_java_launcher_pid = atoi(tail); src/hotspot/share/runtime/arguments.cpp-384- continue; src/java.desktop/unix/native/libawt_xawt/xawt/XToolkit.c 455 value = getenv("_AWT_MAX_POLL_TIMEOUT"); 456 if (value != NULL) { 457 AWT_MAX_POLL_TIMEOUT = atoi(value); src/java.desktop/unix/native/common/awt/X11Color.c-781- if (getenv("CMAPSIZE") != 0) { src/java.desktop/unix/native/common/awt/X11Color.c:782: cmapsize = atoi(getenv("CMAPSIZE")); Should I open a bug for these ? Best regards, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.reingruber at sap.com Fri Sep 13 14:12:24 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Fri, 13 Sep 2019 14:12:24 +0000 Subject: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken Message-ID: Hi, could I please get reviews for Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8230956 JVMTI provides functions to follow references beginning at the roots of the object graph and it provides functions to iterate all objects on the heap[1][2]. These functions are means to access objects which are otherwise local to a Java thread. In terms of escape analysis these local objects escape through these JVMTI functions invalidating optimizations based on escape analysis. Example: - Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is scalar replaced. - JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all instances of C. - A uses GetObjectsWithTags() to retrieve the tagged instances of C. - Error: I is missing because its allocation was eliminated / scalar replaced. Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap functions that let objects escape. Currently it is not possible to revert EA based optimizations just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the JVMTI capability can_tag_objects is taken. But this is not sufficient, because there may be compiled frames on stack with EA based optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then it does not help to disable escape analysis or invalidate compiled methods with ea based optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete solution to the issue. An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after calling JVMTI AddCapabilities(), even with an empty set of capabilities, JvmtiExport::can_walk_any_space() will return true. I've run tier1 tests. Thanks, Richard. [1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap [2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0 From chris.plummer at oracle.com Fri Sep 13 17:49:00 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 13 Sep 2019 10:49:00 -0700 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: References: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> Message-ID: <73cc892e-abf2-d9cc-22c3-058ca7b01ebd@oracle.com> 3 minutes??? Sounds like something is wrong. What is the JVM doing during this time? Chris On 9/12/19 12:53 PM, Langer, Christoph wrote: > Hi Severin, > > that seems an interesting idea for an elegant solution. However, after trying this on a decently fast linux x86 box by leveraging one of these ProcessTools::startProcess methods that would wait for a certain output to appear in the child before returning, I figured that the elapsed runtime of the test increases from about 3 seconds to 3 minutes. It evidently takes a lot longer to bootstrap a JVM and get the results of a first println than just forking a process and immediately accessing its proc filesystem. So I think we don't want to do that. > > I would like to go with this version (changed the comment to Thomas' suggestion): http://cr.openjdk.java.net/~clanger/webrevs/8230850.1/ > > Chris, are you ok with it? > > Thanks > Christoph > >> -----Original Message----- >> From: Severin Gehwolf >> Sent: Donnerstag, 12. September 2019 11:01 >> To: Langer, Christoph ; Thomas St?fe >> >> Cc: OpenJDK Serviceability >> Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java >> fails intermittently >> >> Hi Christoph, >> >> Have you considered to wait for TestProcess - the spawned processes - to >> print this on stdout: >> >> "The process started, pid: XXX" >> >> Once that's ready on stdout, checking the main class should always >> pass. I believe p.isAlive() check which is currrently done is >> insufficient. >> >> Thanks, >> Severin >> >> On Thu, 2019-09-12 at 08:12 +0000, Langer, Christoph wrote: >>> Hi Thomas, >>> >>> sounds reasonable, will do. >>> >>> Thanks >>> Christoph >>> >>> From: Thomas St?fe >>> Sent: Donnerstag, 12. September 2019 10:11 >>> To: Langer, Christoph >>> Cc: Chris Plummer ; OpenJDK Serviceability >> >>> Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java >> fails intermittently >>> I'm fine with the patch if you would reshape the platform dependent >> comment. Proposal: >>> ---- >>> - // Depending on hw/os, process helper can return null here >>> - // because /proc//cmdline is not ready yet. To cover that case, >>> // give it some retries. >>> -> >>> + getMainClass() may return NULL, e.g. due to timing issues. Attempt some >> limited retries. >>> ---- >>> I do not need another webrev. >>> Cheers, Thomas >>> >>> >>> >>> >>> On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph >> wrote: >>>> Hi Chris, Thomas, >>>> >>>> thanks for looking at this. I was also wondering whether a fix in >> ProcessHelper would be appropriate. But I think introducing retries and >> delays in that code can do more harm than help. >>>> For this special test case, aiming to test the ProcessHelper functionality >> (on Linux) only, the observed problem is that the /proc//cmdline file is >> not ready yet when it gets evaluated because the test can be quicker than >> the spawned processes. But in real life usage of jcmd this seems rather >> unlikely. One will probably use jcmd quite some time after a java process was >> started and /proc//cmdline should be ready. If then there are >> problems reading it, there are likely other issues which won?t go away by >> waiting. And for these cases the fallback is to use the attach framework, as >> implemented in ProcessArgumentMatcher, which provides some chance to >> be working still. And this fallback should also cover the exotic case when jcmd >> is issued too early. >>>> After all, ProcessHelper::getMainClass also documents that its result can >> be null. >>>> @Thomas, as for your other points: >>>> PID reusage: Hm, maybe one can construct cases. However, I?d think the >> /proc/pid files should be gone after a process ends. Or at least be >> reconstructed if there were orphans and a new process reusing an old pid >> gets started. But who knows what can happen ? we?ll maybe see ?? >>>> Comment for Linux only issue: The test is in fact a Linux only test. See line >> 55: * @requires os.family == "linux". So, if we?ll eventually see >> implementations for ProcessHelper::getMainClass on other platforms, this >> comment might have to be adopted. But for the time being I guess it?s fine at >> its current place. >>>> Would you agree? >>>> >>>> Best regards >>>> Christoph >>>> >>>> From: Chris Plummer >>>> Sent: Mittwoch, 11. September 2019 19:21 >>>> To: Thomas St?fe ; Langer, Christoph >> >>>> Cc: OpenJDK Serviceability >>>> Subject: Re: RFR (S): 8230850: Test >> sun/tools/jcmd/TestProcessHelper.java fails intermittently >>>> It does seem that the fix should be in ProcessHelper.java in >> getMainClass(), or maybe even getCommandLine(). Fixing it in the test >> implies that every user of getMainClass() should be doing something similar. >> But then also note what ProcessArgumentMatch.check() is doing. It also >> deals with getMainClass() returing null. >>>> thanks, >>>> >>>> Chris >>>> >>>> On 9/11/19 6:59 AM, Thomas St?fe wrote: >>>>> Hi Christoph, >>>>> >>>>> in general I think this is fine. The increase-by-pow2 sleep time is odd >> but okay :) >>>>> The whole things seems rather fragile and has a lot of question marks >> but I think your fix does not make it worse. One fun error now is that with a >> follow up java test reusing the PID we could get a wrong main class but I think >> the chances are astronomically low. >>>>> Only remark, you fix this in the platform shared code, if this is a Linux >> only issue maybe it should be fixed in /shared/projects/openjdk/jdk- >> jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java >> instead? If not, I would remove at least the /proc//cmdline comment >> since this is quite platform specific. >>>>> Cheers, Thomas >>>>> >>>>> >>>>> On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph >> wrote: >>>>>> Hi, >>>>>> >>>>>> please review this change for test >> sun/tools/jcmd/TestProcessHelper.java to make it more robust. >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 >>>>>> Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ >>>>>> >>>>>> This Linux only test is starting several Java processes and then tries to >> figure out the main class by invoking jdk.jcmd's linux specific ProcessHelper >> implementation which parses the contents of /proc//cmdline. >>>>>> Under some circumstances, the test already attempts to read >> /proc//cmdline before it actually exists or is filled with data. This can be >> fixed with some sleeps/retries to wait for that data to be ready. >>>>>> In the actual jcmd tool, such behavior of ProcessHelper. getMainClass >> should not be an issue because it is handled in ProcessArgumentMatcher [0]. >>>>>> Thanks >>>>>> Christoph >>>>>> >>>>>> [0] >> http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/cl >> asses/sun/tools/common/ProcessArgumentMatcher.java#l86 >>>> From hohensee at amazon.com Fri Sep 13 19:11:30 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 13 Sep 2019 19:11:30 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> Message-ID: <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> Hi David, thanks for your comments. New webrev in http://cr.openjdk.java.net/~phh/8207266/webrev.08/ Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. Paul ?On 9/13/19, 12:50 AM, "David Holmes" wrote: Hi Paul, On 13/09/2019 10:29 am, Hohensee, Paul wrote: > Thanks for clarifying the review rules. Would someone from the > serviceability team please review? New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ One aspect of the functional change needs clarification for me - and apologies if this has been covered in the past. It seems to me that currently we only check isThreadAllocatedMemorySupported for these operations, but if I read things correctly the updated code additionally checks isThreadAllocatedMemoryEnabled, which is a behaviour change not mentioned in the CSR. > I didn?t disturb the existing checks in the test, just added code to > check the result of getThreadAllocatedBytes(long) on a non-current > thread, plus the back-to-back no-allocation checks. The former wasn?t > needed before because getThreadAllocatedBytes(long) was just a wrapper > around getThreadAllocatedBytes(long []). This patch changes that, so I > added a separate test. The latter is supposed to fail if there?s object > allocation on calls to getCurrentThreadAllocatedBytes and > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > accumulation of transient small objects can be a performance problem. > Thanks to your review, I noticed that the back-to-back check on the > current thread was using getThreadAllocatedBytes(long) instead of > getCurrentThreadAllocatedBytes and fixed it. I also removed all > instances of ?TEST FAILED: ?. The back-to-back check is not valid in general. You don't know if the first check might trigger some class loading on the return path after it has obtained the first memory value. The check might also fail if using JVMCI and some compilation related activity occurs in the current thread on the second call. Also with the introduction of handshakes its possible the current thread might hit a safepoint checks that results in it executing a handshake operation that performs allocation. Potentially there could be numerous non-deterministic actions that might occur leading to unanticipated allocation. I understand what you want to test here, I just don't think it is reliably doable. Thanks, David ----- > > Paul > > *From: *Mandy Chung > *Date: *Thursday, September 12, 2019 at 10:09 AM > *To: *"Hohensee, Paul" > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > I only reviewed the library side implementation that looks good. I > expect the serviceability team to review the test and hotspot change. > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > You need another reviewer to advice the following because I was not > close to the ThreadsList work. > > 2087 ThreadsListHandle tlh; > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > 2089 > > 2090 if (java_thread != NULL) { > > 2091 return java_thread->cooked_allocated_bytes(); > > 2092 } > > This looks right to me. > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > - "ThreadAllocatedMemory is expected to be disabled"); > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > disabled"); > > Prepending "TEST FAILED" in exception message (in several places) > > seems redundant since such RuntimeException is thrown and expected > > a test failure. > > + // back-to-back calls shouldn't allocate any memory > > + size = mbean.getThreadAllocatedBytes(id); > > + size1 = mbean.getThreadAllocatedBytes(id); > > + if (size1 != size) { > > Is there anything in the test can do to help guarantee this? I didn't > > closely review this test. The main thing I advice is to improve > > the reliability of this test. Put it in another way, we want to > > ensure that this test change will pass all the time in various > > test configuration. > > Mandy > From serguei.spitsyn at oracle.com Fri Sep 13 21:58:06 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 13 Sep 2019 14:58:06 -0700 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code In-Reply-To: References: Message-ID: Hi Matthias, The fix looks good to me. Thank you for catching and fixing this! Thanks, Serguei On 9/13/19 3:01 AM, Baesken, Matthias wrote: > > Hello , my colleague? Ralf pointed out that? the? NULL-check? of the > result of GetStringUTFChars > > should be done right ?after the ?GetStringUTFChars?? so I moved the > NULL-check up : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.2/ > > > Best regards, Matthias > > Hi Thomas,?? thanks for the review . > > You are correct about atoi . > > New webrev ?: > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/ > > > I had 2 additional? observations? : > > 1. With ?OJDK on solaris 32bit gone for quite some time, we might be? > able? to kick out the whole? non _LP64? code? because we are > always 64 bit > > (maybe? someone could comment if this is a safe assumption,? there > might be old 32bit solaris core files flying around for some reason > even these days ? ) > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html > > > 696?? // some older versions of libproc.so crash when trying to attach > 32 bit > > 697?? // debugger to 64 bit core file. check and throw error. > > 698 #ifndef _LP64 > > ?.. > > 2. The usage of atoi is commented? here : > > https://docs.oracle.com/cd/E86824_01/html/E54766/atoi-3c.html > > ?However, applications should not use the atoi(), atol(), or > atoll()?functions unless they know the value represented by the > argument will be in range for the corresponding result type? > > ????? ??And here : > > https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html > > ?If the number is not known to be in range, /strtol/() > ?should > be used because /atoi/() is not required to perform any error checking? > > However? we ?have? a number of? usages in the coding?? where? atoi is > called without knowing that? the argument is in the allowed range . > > some examples : > > src/hotspot/share/runtime/arguments.cpp-382- if (match_option(option, > "-Dsun.java.launcher.pid=", &tail)) { > > src/hotspot/share/runtime/arguments.cpp:383: _sun_java_launcher_pid = > atoi(tail); > > src/hotspot/share/runtime/arguments.cpp-384- continue; > > src/java.desktop/unix/native/libawt_xawt/xawt/XToolkit.c > > 455??? value = getenv("_AWT_MAX_POLL_TIMEOUT"); > > 456??? if (value != NULL) { > > 457??????? AWT_MAX_POLL_TIMEOUT = atoi(value); > > src/java.desktop/unix/native/common/awt/X11Color.c-781- if > (getenv("CMAPSIZE") != 0) { > > src/java.desktop/unix/native/common/awt/X11Color.c:782: cmapsize = > atoi(getenv("CMAPSIZE")); > > Should I open a bug for these ? > > Best regards, Matthias > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri Sep 13 22:18:20 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 13 Sep 2019 15:18:20 -0700 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code In-Reply-To: References: Message-ID: <9dca6ade-aba5-daf1-fa26-07b3d0bad35e@oracle.com> Hi Matthias, On 9/12/19 4:52 AM, Baesken, Matthias wrote: > > Hi Thomas,?? thanks for the review . > > You are correct about atoi . > > New webrev ?: > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/ > > > I had 2 additional? observations? : > > 1. With ?OJDK on solaris 32bit gone for quite some time, we might be > able? to kick out the whole? non _LP64? code? because we are > always 64 bit > > (maybe? someone could comment if this is a safe assumption,? there > might be old 32bit solaris core files flying around for some reason > even these days ? ) > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html > > > 696?? // some older versions of libproc.so crash when trying to attach > 32 bit > > 697?? // debugger to 64 bit core file. check and throw error. > > 698 #ifndef _LP64 > > ?.. > > 2. The usage of atoi is commented? here : > > https://docs.oracle.com/cd/E86824_01/html/E54766/atoi-3c.html > > ?However, applications should not use the atoi(), atol(), or > atoll()?functions unless they know the value represented by the > argument will be in range for the corresponding result type? > > ????? ??And here : > > https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html > > ?If the number is not known to be in range, /strtol/() > ?should > be used because /atoi/() is not required to perform any error checking? > > However? we ?have? a number of? usages in the coding?? where? atoi is > called without knowing that? the argument is in the allowed range . > > some examples : > > src/hotspot/share/runtime/arguments.cpp-382- if (match_option(option, > "-Dsun.java.launcher.pid=", &tail)) { > > src/hotspot/share/runtime/arguments.cpp:383: _sun_java_launcher_pid = > atoi(tail); > > src/hotspot/share/runtime/arguments.cpp-384- continue; > > src/java.desktop/unix/native/libawt_xawt/xawt/XToolkit.c > > 455??? value = getenv("_AWT_MAX_POLL_TIMEOUT"); > > 456??? if (value != NULL) { > > 457??????? AWT_MAX_POLL_TIMEOUT = atoi(value); > > src/java.desktop/unix/native/common/awt/X11Color.c-781- if > (getenv("CMAPSIZE") != 0) { > > src/java.desktop/unix/native/common/awt/X11Color.c:782: cmapsize = > atoi(getenv("CMAPSIZE")); > > Should I open a bug for these ? > Probably, two different bug are needed: hotspot/runtime and AWT. Thanks, Serguei > Best regards, Matthias > > *From:*Thomas St?fe > *Sent:* Donnerstag, 12. September 2019 12:22 > *To:* Baesken, Matthias > *Cc:* serviceability-dev at openjdk.java.net > *Subject:* Re: RFR [XS]: 8230901: missing ReleaseStringUTFChars in > servicability native code > > Hi Matthias, > > your changes look good. > > > an additional bug: > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html > > > > ?698 #ifndef _LP64 > ?699 ? atoi(cmdLine_cstr); > ?700 ? if (errno) { > > Behaviour of atoi() in error case is undefined. errno values are not > defined. > > See: https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html > > And even if atoi would set errno, this is still not enough since errno > may contain a stale value. One would have to set errno=0 before the > function call. > > > If you want to fix this too 'd suggest replacing this call with strtol(). > > Cheers, Thomas > > On Thu, Sep 12, 2019 at 12:11 PM Baesken, Matthias > > wrote: > > Hello, please reviews this small change . > > It adds? ReleaseStringUTFChars calls? at some places in early > return cases . > > ( in src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp > > THROW_NEW_DEBUGGER_EXCEPTION contains a return , see the macro > declaration > > 39 #define THROW_NEW_DEBUGGER_EXCEPTION(str) { > throwNewDebuggerException(env, str); return;} > > ) > > Bug/webrev : > > https://bugs.openjdk.java.net/browse/JDK-8230901 > > http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/ > > > Thanks, Matthias > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Fri Sep 13 22:34:38 2019 From: david.holmes at oracle.com (David Holmes) Date: Sat, 14 Sep 2019 08:34:38 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> Message-ID: <28a1cc57-dee3-7beb-19ef-d434ddf038ca@oracle.com> Hi Paul, On 14/09/2019 5:11 am, Hohensee, Paul wrote: > Hi David, thanks for your comments. New webrev in > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. Thanks for clarifying. > You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. Updated test looks fine. Nothing further from me. Thanks, David ----- > Paul > > ?On 9/13/19, 12:50 AM, "David Holmes" wrote: > > Hi Paul, > > On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > Thanks for clarifying the review rules. Would someone from the > > serviceability team please review? New webrev at > > > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > One aspect of the functional change needs clarification for me - and > apologies if this has been covered in the past. It seems to me that > currently we only check isThreadAllocatedMemorySupported for these > operations, but if I read things correctly the updated code additionally > checks isThreadAllocatedMemoryEnabled, which is a behaviour change not > mentioned in the CSR. > > > I didn?t disturb the existing checks in the test, just added code to > > check the result of getThreadAllocatedBytes(long) on a non-current > > thread, plus the back-to-back no-allocation checks. The former wasn?t > > needed before because getThreadAllocatedBytes(long) was just a wrapper > > around getThreadAllocatedBytes(long []). This patch changes that, so I > > added a separate test. The latter is supposed to fail if there?s object > > allocation on calls to getCurrentThreadAllocatedBytes and > > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > > accumulation of transient small objects can be a performance problem. > > Thanks to your review, I noticed that the back-to-back check on the > > current thread was using getThreadAllocatedBytes(long) instead of > > getCurrentThreadAllocatedBytes and fixed it. I also removed all > > instances of ?TEST FAILED: ?. > > The back-to-back check is not valid in general. You don't know if the > first check might trigger some class loading on the return path after it > has obtained the first memory value. The check might also fail if using > JVMCI and some compilation related activity occurs in the current thread > on the second call. Also with the introduction of handshakes its > possible the current thread might hit a safepoint checks that results in > it executing a handshake operation that performs allocation. Potentially > there could be numerous non-deterministic actions that might occur > leading to unanticipated allocation. > > I understand what you want to test here, I just don't think it is > reliably doable. > > Thanks, > David > ----- > > > > > Paul > > > > *From: *Mandy Chung > > *Date: *Thursday, September 12, 2019 at 10:09 AM > > *To: *"Hohensee, Paul" > > *Cc: *OpenJDK Serviceability , > > "hotspot-gc-dev at openjdk.java.net" > > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > > can be quicker for self thread > > > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > > > > I only reviewed the library side implementation that looks good. I > > expect the serviceability team to review the test and hotspot change. > > > > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > > > > You need another reviewer to advice the following because I was not > > close to the ThreadsList work. > > > > 2087 ThreadsListHandle tlh; > > > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > > 2089 > > > > 2090 if (java_thread != NULL) { > > > > 2091 return java_thread->cooked_allocated_bytes(); > > > > 2092 } > > > > This looks right to me. > > > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > > - "ThreadAllocatedMemory is expected to be disabled"); > > > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > > disabled"); > > > > Prepending "TEST FAILED" in exception message (in several places) > > > > seems redundant since such RuntimeException is thrown and expected > > > > a test failure. > > > > + // back-to-back calls shouldn't allocate any memory > > > > + size = mbean.getThreadAllocatedBytes(id); > > > > + size1 = mbean.getThreadAllocatedBytes(id); > > > > + if (size1 != size) { > > > > Is there anything in the test can do to help guarantee this? I didn't > > > > closely review this test. The main thing I advice is to improve > > > > the reliability of this test. Put it in another way, we want to > > > > ensure that this test change will pass all the time in various > > > > test configuration. > > > > Mandy > > > > From serguei.spitsyn at oracle.com Fri Sep 13 23:33:29 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 13 Sep 2019 16:33:29 -0700 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file In-Reply-To: References: Message-ID: <9de2c900-16d5-d38a-e707-266313b39a5d@oracle.com> Hi Leonid, +1 Thanks, Serguei On 9/12/19 4:46 PM, David Holmes wrote: > Looks good! > > Thanks for fixing. > > David > > On 13/09/2019 2:56 am, Leonid Mesnik wrote: >> Hi >> >> Could you please verify following fix which update core filename >> patterns. Some hosts are configured to compress core files during >> core dump. In such cases tests should throw skipped exception instead >> of failing reading compressed cores. >> Fixing all tests to support unpacking core.pid.gz files is going to >> be fixed in separate RFE. >> >> >> Verified that test pass if plain core is generated and skipped in >> other case. >> >> >> webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8230881 >> >> Leonid From leonid.mesnik at oracle.com Fri Sep 13 23:42:17 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Fri, 13 Sep 2019 16:42:17 -0700 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file In-Reply-To: <9de2c900-16d5-d38a-e707-266313b39a5d@oracle.com> References: <9de2c900-16d5-d38a-e707-266313b39a5d@oracle.com> Message-ID: <9487A838-FB4C-4E40-83F1-E094B27063ED@oracle.com> Thank you for your review. I filed another bug (rfe?) to support testing on hosts with compressing cores https://bugs.openjdk.java.net/browse/JDK-8230942 Will fix it later. Leonid > On Sep 13, 2019, at 4:33 PM, serguei.spitsyn at oracle.com wrote: > > Hi Leonid, > > +1 > > Thanks, > Serguei > > On 9/12/19 4:46 PM, David Holmes wrote: >> Looks good! >> >> Thanks for fixing. >> >> David >> >> On 13/09/2019 2:56 am, Leonid Mesnik wrote: >>> Hi >>> >>> Could you please verify following fix which update core filename patterns. Some hosts are configured to compress core files during core dump. In such cases tests should throw skipped exception instead of failing reading compressed cores. >>> Fixing all tests to support unpacking core.pid.gz files is going to be fixed in separate RFE. >>> >>> >>> Verified that test pass if plain core is generated and skipped in other case. >>> >>> >>> webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ >>> bug: https://bugs.openjdk.java.net/browse/JDK-8230881 >>> >>> Leonid > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri Sep 13 23:58:30 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 13 Sep 2019 16:58:30 -0700 Subject: RFR: 8230881: serviceability/sa/TestJmapCore tests fail with java.lang.RuntimeException: Could not find dump file In-Reply-To: <9487A838-FB4C-4E40-83F1-E094B27063ED@oracle.com> References: <9de2c900-16d5-d38a-e707-266313b39a5d@oracle.com> <9487A838-FB4C-4E40-83F1-E094B27063ED@oracle.com> Message-ID: <41f2a5ea-b783-98b5-3bbe-285583312f6f@oracle.com> Hi Leonid, Thank you for filing the bug! I've changed it to Enhancement. Thanks, Serguei On 9/13/19 4:42 PM, Leonid Mesnik wrote: > Thank you for your review. > > I filed another bug (rfe?) to support testing on hosts with > compressing cores > > https://bugs.openjdk.java.net/browse/JDK-8230942 > > Will fix it later. > > Leonid > >> On Sep 13, 2019, at 4:33 PM, serguei.spitsyn at oracle.com >> wrote: >> >> Hi Leonid, >> >> +1 >> >> Thanks, >> Serguei >> >> On 9/12/19 4:46 PM, David Holmes wrote: >>> Looks good! >>> >>> Thanks for fixing. >>> >>> David >>> >>> On 13/09/2019 2:56 am, Leonid Mesnik wrote: >>>> Hi >>>> >>>> Could you please verify following fix which update core filename >>>> patterns. Some hosts are configured to compress core files during >>>> core dump. In such cases tests should throw skipped exception >>>> instead of failing reading compressed cores. >>>> Fixing all tests to support unpacking core.pid.gz files is going to >>>> be fixed in separate RFE. >>>> >>>> >>>> Verified that test pass if plain core is generated and skipped in >>>> other case. >>>> >>>> >>>> webrev: http://cr.openjdk.java.net/~lmesnik/8230881/webrev.00/ >>>> >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8230881 >>>> >>>> Leonid >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Sat Sep 14 00:48:51 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 13 Sep 2019 17:48:51 -0700 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> Message-ID: <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> Hi Paul, It looks pretty good in general. http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html It would be nice to refactor the java main() method as it becomes too big. Two ways ofgetCurrentThreadAllocatedBytes() testing are good candidates to become separate methods. 98 long size1 = mbean.getThreadAllocatedBytes(id); Just wanted to double check if you wanted to invoke the getCurrentThreadAllocatedBytes() instead as it is a part of: 85 // First way, getCurrentThreadAllocatedBytes Thanks, Serguei On 9/13/19 12:11 PM, Hohensee, Paul wrote: > Hi David, thanks for your comments. New webrev in > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. > > You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. > > Paul > > ?On 9/13/19, 12:50 AM, "David Holmes" wrote: > > Hi Paul, > > On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > Thanks for clarifying the review rules. Would someone from the > > serviceability team please review? New webrev at > > > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > One aspect of the functional change needs clarification for me - and > apologies if this has been covered in the past. It seems to me that > currently we only check isThreadAllocatedMemorySupported for these > operations, but if I read things correctly the updated code additionally > checks isThreadAllocatedMemoryEnabled, which is a behaviour change not > mentioned in the CSR. > > > I didn?t disturb the existing checks in the test, just added code to > > check the result of getThreadAllocatedBytes(long) on a non-current > > thread, plus the back-to-back no-allocation checks. The former wasn?t > > needed before because getThreadAllocatedBytes(long) was just a wrapper > > around getThreadAllocatedBytes(long []). This patch changes that, so I > > added a separate test. The latter is supposed to fail if there?s object > > allocation on calls to getCurrentThreadAllocatedBytes and > > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > > accumulation of transient small objects can be a performance problem. > > Thanks to your review, I noticed that the back-to-back check on the > > current thread was using getThreadAllocatedBytes(long) instead of > > getCurrentThreadAllocatedBytes and fixed it. I also removed all > > instances of ?TEST FAILED: ?. > > The back-to-back check is not valid in general. You don't know if the > first check might trigger some class loading on the return path after it > has obtained the first memory value. The check might also fail if using > JVMCI and some compilation related activity occurs in the current thread > on the second call. Also with the introduction of handshakes its > possible the current thread might hit a safepoint checks that results in > it executing a handshake operation that performs allocation. Potentially > there could be numerous non-deterministic actions that might occur > leading to unanticipated allocation. > > I understand what you want to test here, I just don't think it is > reliably doable. > > Thanks, > David > ----- > > > > > Paul > > > > *From: *Mandy Chung > > *Date: *Thursday, September 12, 2019 at 10:09 AM > > *To: *"Hohensee, Paul" > > *Cc: *OpenJDK Serviceability , > > "hotspot-gc-dev at openjdk.java.net" > > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > > can be quicker for self thread > > > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > > > > I only reviewed the library side implementation that looks good. I > > expect the serviceability team to review the test and hotspot change. > > > > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > > > > You need another reviewer to advice the following because I was not > > close to the ThreadsList work. > > > > 2087 ThreadsListHandle tlh; > > > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > > 2089 > > > > 2090 if (java_thread != NULL) { > > > > 2091 return java_thread->cooked_allocated_bytes(); > > > > 2092 } > > > > This looks right to me. > > > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > > - "ThreadAllocatedMemory is expected to be disabled"); > > > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > > disabled"); > > > > Prepending "TEST FAILED" in exception message (in several places) > > > > seems redundant since such RuntimeException is thrown and expected > > > > a test failure. > > > > + // back-to-back calls shouldn't allocate any memory > > > > + size = mbean.getThreadAllocatedBytes(id); > > > > + size1 = mbean.getThreadAllocatedBytes(id); > > > > + if (size1 != size) { > > > > Is there anything in the test can do to help guarantee this? I didn't > > > > closely review this test. The main thing I advice is to improve > > > > the reliability of this test. Put it in another way, we want to > > > > ensure that this test change will pass all the time in various > > > > test configuration. > > > > Mandy > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Sun Sep 15 05:57:15 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Sun, 15 Sep 2019 05:57:15 +0000 Subject: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java fails intermittently In-Reply-To: <73cc892e-abf2-d9cc-22c3-058ca7b01ebd@oracle.com> References: <538fbd9d96e88ef3179f6ad7c410e4311e87c420.camel@redhat.com> <73cc892e-abf2-d9cc-22c3-058ca7b01ebd@oracle.com> Message-ID: Hi Chris, I guess it's the multitude of JVMs that get started one after each other and not in parallel. So I'll push my fix then. Best regards Christoph > -----Original Message----- > From: Chris Plummer > Sent: Freitag, 13. September 2019 19:49 > To: Langer, Christoph ; Severin Gehwolf > ; Thomas St?fe > Cc: OpenJDK Serviceability > Subject: Re: RFR (S): 8230850: Test sun/tools/jcmd/TestProcessHelper.java > fails intermittently > > 3 minutes??? Sounds like something is wrong. What is the JVM doing > during this time? > > Chris > > On 9/12/19 12:53 PM, Langer, Christoph wrote: > > Hi Severin, > > > > that seems an interesting idea for an elegant solution. However, after > trying this on a decently fast linux x86 box by leveraging one of these > ProcessTools::startProcess methods that would wait for a certain output to > appear in the child before returning, I figured that the elapsed runtime of the > test increases from about 3 seconds to 3 minutes. It evidently takes a lot > longer to bootstrap a JVM and get the results of a first println than just > forking a process and immediately accessing its proc filesystem. So I think we > don't want to do that. > > > > I would like to go with this version (changed the comment to Thomas' > suggestion): http://cr.openjdk.java.net/~clanger/webrevs/8230850.1/ > > > > Chris, are you ok with it? > > > > Thanks > > Christoph > > > >> -----Original Message----- > >> From: Severin Gehwolf > >> Sent: Donnerstag, 12. September 2019 11:01 > >> To: Langer, Christoph ; Thomas St?fe > >> > >> Cc: OpenJDK Serviceability > >> Subject: Re: RFR (S): 8230850: Test > sun/tools/jcmd/TestProcessHelper.java > >> fails intermittently > >> > >> Hi Christoph, > >> > >> Have you considered to wait for TestProcess - the spawned processes - to > >> print this on stdout: > >> > >> "The process started, pid: XXX" > >> > >> Once that's ready on stdout, checking the main class should always > >> pass. I believe p.isAlive() check which is currrently done is > >> insufficient. > >> > >> Thanks, > >> Severin > >> > >> On Thu, 2019-09-12 at 08:12 +0000, Langer, Christoph wrote: > >>> Hi Thomas, > >>> > >>> sounds reasonable, will do. > >>> > >>> Thanks > >>> Christoph > >>> > >>> From: Thomas St?fe > >>> Sent: Donnerstag, 12. September 2019 10:11 > >>> To: Langer, Christoph > >>> Cc: Chris Plummer ; OpenJDK Serviceability > >> > >>> Subject: Re: RFR (S): 8230850: Test > sun/tools/jcmd/TestProcessHelper.java > >> fails intermittently > >>> I'm fine with the patch if you would reshape the platform dependent > >> comment. Proposal: > >>> ---- > >>> - // Depending on hw/os, process helper can return null here > >>> - // because /proc//cmdline is not ready yet. To cover that > case, > >>> // give it some retries. > >>> -> > >>> + getMainClass() may return NULL, e.g. due to timing issues. Attempt > some > >> limited retries. > >>> ---- > >>> I do not need another webrev. > >>> Cheers, Thomas > >>> > >>> > >>> > >>> > >>> On Wed, Sep 11, 2019 at 11:37 PM Langer, Christoph > >> wrote: > >>>> Hi Chris, Thomas, > >>>> > >>>> thanks for looking at this. I was also wondering whether a fix in > >> ProcessHelper would be appropriate. But I think introducing retries and > >> delays in that code can do more harm than help. > >>>> For this special test case, aiming to test the ProcessHelper functionality > >> (on Linux) only, the observed problem is that the /proc//cmdline file > is > >> not ready yet when it gets evaluated because the test can be quicker than > >> the spawned processes. But in real life usage of jcmd this seems rather > >> unlikely. One will probably use jcmd quite some time after a java process > was > >> started and /proc//cmdline should be ready. If then there are > >> problems reading it, there are likely other issues which won?t go away by > >> waiting. And for these cases the fallback is to use the attach framework, > as > >> implemented in ProcessArgumentMatcher, which provides some chance > to > >> be working still. And this fallback should also cover the exotic case when > jcmd > >> is issued too early. > >>>> After all, ProcessHelper::getMainClass also documents that its result > can > >> be null. > >>>> @Thomas, as for your other points: > >>>> PID reusage: Hm, maybe one can construct cases. However, I?d think > the > >> /proc/pid files should be gone after a process ends. Or at least be > >> reconstructed if there were orphans and a new process reusing an old pid > >> gets started. But who knows what can happen ? we?ll maybe see ?? > >>>> Comment for Linux only issue: The test is in fact a Linux only test. See > line > >> 55: * @requires os.family == "linux". So, if we?ll eventually see > >> implementations for ProcessHelper::getMainClass on other platforms, > this > >> comment might have to be adopted. But for the time being I guess it?s > fine at > >> its current place. > >>>> Would you agree? > >>>> > >>>> Best regards > >>>> Christoph > >>>> > >>>> From: Chris Plummer > >>>> Sent: Mittwoch, 11. September 2019 19:21 > >>>> To: Thomas St?fe ; Langer, Christoph > >> > >>>> Cc: OpenJDK Serviceability > >>>> Subject: Re: RFR (S): 8230850: Test > >> sun/tools/jcmd/TestProcessHelper.java fails intermittently > >>>> It does seem that the fix should be in ProcessHelper.java in > >> getMainClass(), or maybe even getCommandLine(). Fixing it in the test > >> implies that every user of getMainClass() should be doing something > similar. > >> But then also note what ProcessArgumentMatch.check() is doing. It also > >> deals with getMainClass() returing null. > >>>> thanks, > >>>> > >>>> Chris > >>>> > >>>> On 9/11/19 6:59 AM, Thomas St?fe wrote: > >>>>> Hi Christoph, > >>>>> > >>>>> in general I think this is fine. The increase-by-pow2 sleep time is odd > >> but okay :) > >>>>> The whole things seems rather fragile and has a lot of question marks > >> but I think your fix does not make it worse. One fun error now is that with > a > >> follow up java test reusing the PID we could get a wrong main class but I > think > >> the chances are astronomically low. > >>>>> Only remark, you fix this in the platform shared code, if this is a Linux > >> only issue maybe it should be fixed in /shared/projects/openjdk/jdk- > >> jdk/source/src/jdk.jcmd/linux/classes/sun/tools/ProcessHelper.java > >> instead? If not, I would remove at least the /proc//cmdline > comment > >> since this is quite platform specific. > >>>>> Cheers, Thomas > >>>>> > >>>>> > >>>>> On Wed, Sep 11, 2019 at 2:39 PM Langer, Christoph > >> wrote: > >>>>>> Hi, > >>>>>> > >>>>>> please review this change for test > >> sun/tools/jcmd/TestProcessHelper.java to make it more robust. > >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230850 > >>>>>> Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8230850.0/ > >>>>>> > >>>>>> This Linux only test is starting several Java processes and then tries > to > >> figure out the main class by invoking jdk.jcmd's linux specific > ProcessHelper > >> implementation which parses the contents of /proc//cmdline. > >>>>>> Under some circumstances, the test already attempts to read > >> /proc//cmdline before it actually exists or is filled with data. This can > be > >> fixed with some sleeps/retries to wait for that data to be ready. > >>>>>> In the actual jcmd tool, such behavior of ProcessHelper. > getMainClass > >> should not be an issue because it is handled in ProcessArgumentMatcher > [0]. > >>>>>> Thanks > >>>>>> Christoph > >>>>>> > >>>>>> [0] > >> > http://hg.openjdk.java.net/jdk/jdk/file/8b08eaf9a0eb/src/jdk.jcmd/share/cl > >> asses/sun/tools/common/ProcessArgumentMatcher.java#l86 > >>>> > From hohensee at amazon.com Sun Sep 15 09:52:30 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Sun, 15 Sep 2019 09:52:30 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> Message-ID: Hi, Serguei, thanks for the review. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.09/ I refactored the test?s main() method, and you?re correct, getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in that context: fixed. Paul From: "serguei.spitsyn at oracle.com" Organization: Oracle Corporation Date: Friday, September 13, 2019 at 5:50 PM To: "Hohensee, Paul" , David Holmes , Mandy Chung Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, It looks pretty good in general. http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html It would be nice to refactor the java main() method as it becomes too big. Two ways of getCurrentThreadAllocatedBytes() testing are good candidates to become separate methods. 98 long size1 = mbean.getThreadAllocatedBytes(id); Just wanted to double check if you wanted to invoke the getCurrentThreadAllocatedBytes() instead as it is a part of: 85 // First way, getCurrentThreadAllocatedBytes Thanks, Serguei On 9/13/19 12:11 PM, Hohensee, Paul wrote: Hi David, thanks for your comments. New webrev in http://cr.openjdk.java.net/~phh/8207266/webrev.08/ Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. Paul On 9/13/19, 12:50 AM, "David Holmes" wrote: Hi Paul, On 13/09/2019 10:29 am, Hohensee, Paul wrote: > Thanks for clarifying the review rules. Would someone from the > serviceability team please review? New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ One aspect of the functional change needs clarification for me - and apologies if this has been covered in the past. It seems to me that currently we only check isThreadAllocatedMemorySupported for these operations, but if I read things correctly the updated code additionally checks isThreadAllocatedMemoryEnabled, which is a behaviour change not mentioned in the CSR. > I didn?t disturb the existing checks in the test, just added code to > check the result of getThreadAllocatedBytes(long) on a non-current > thread, plus the back-to-back no-allocation checks. The former wasn?t > needed before because getThreadAllocatedBytes(long) was just a wrapper > around getThreadAllocatedBytes(long []). This patch changes that, so I > added a separate test. The latter is supposed to fail if there?s object > allocation on calls to getCurrentThreadAllocatedBytes and > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > accumulation of transient small objects can be a performance problem. > Thanks to your review, I noticed that the back-to-back check on the > current thread was using getThreadAllocatedBytes(long) instead of > getCurrentThreadAllocatedBytes and fixed it. I also removed all > instances of ?TEST FAILED: ?. The back-to-back check is not valid in general. You don't know if the first check might trigger some class loading on the return path after it has obtained the first memory value. The check might also fail if using JVMCI and some compilation related activity occurs in the current thread on the second call. Also with the introduction of handshakes its possible the current thread might hit a safepoint checks that results in it executing a handshake operation that performs allocation. Potentially there could be numerous non-deterministic actions that might occur leading to unanticipated allocation. I understand what you want to test here, I just don't think it is reliably doable. Thanks, David ----- > > Paul > > *From: *Mandy Chung > *Date: *Thursday, September 12, 2019 at 10:09 AM > *To: *"Hohensee, Paul" > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > I only reviewed the library side implementation that looks good. I > expect the serviceability team to review the test and hotspot change. > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > You need another reviewer to advice the following because I was not > close to the ThreadsList work. > > 2087 ThreadsListHandle tlh; > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > 2089 > > 2090 if (java_thread != NULL) { > > 2091 return java_thread->cooked_allocated_bytes(); > > 2092 } > > This looks right to me. > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > - "ThreadAllocatedMemory is expected to be disabled"); > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > disabled"); > > Prepending "TEST FAILED" in exception message (in several places) > > seems redundant since such RuntimeException is thrown and expected > > a test failure. > > + // back-to-back calls shouldn't allocate any memory > > + size = mbean.getThreadAllocatedBytes(id); > > + size1 = mbean.getThreadAllocatedBytes(id); > > + if (size1 != size) { > > Is there anything in the test can do to help guarantee this? I didn't > > closely review this test. The main thing I advice is to improve > > the reliability of this test. Put it in another way, we want to > > ensure that this test change will pass all the time in various > > test configuration. > > Mandy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniil.x.titov at oracle.com Mon Sep 16 18:18:49 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Mon, 16 Sep 2019 11:18:49 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> Message-ID: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Hello, After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. src/hotspot/share/runtime/threadSMR.cpp 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 [4] https://bugs.openjdk.java.net/browse/JDK-8229391 ?Thank you, Daniil > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > Hi Daniil, > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > Hi David, > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > the changes you suggested: > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > - ThreadTableCreate_lock is made _safepoint_check_always; > > Okay. > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > the thread table is changed to grow on demand by the thread that is doing the addition; > > Okay - I'm happy to get the serviceThread out of the picture here. > > > - fixed nits and formatting issues. > > Okay. > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > >>> as Daniel suggested. > >> Not sure it's best to combine these, but if they are limited to the > >> changes in management.cpp only then that may be okay. > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > limited to management.cpp (plus a new test) so I left them in the webrev but > > I also could move it in the separate issue if required. > > I'd prefer this part of be separated out, but won't insist. Let's see if > Dan or Serguei have a strong opinion. > > > > src/hotspot/share/runtime/threadSMR.cpp > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > I think it cleaner/better to just use > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > I had to leave this code unchanged since it turned out the threadObj is null > > when VM is destroyed: > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > C [libjli.so+0x4333] JavaMain+0x2c3 > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > This is actually nothing to do with the VM being destroyed, but is an > issue with JNI_AttachCurrentThread and its interaction with the > ThreadSMR iterators. The attach process is: > - create JavaThread > - mark as "is attaching via jni" > - add to ThreadsList > - create java.lang.Thread object (you can only execute Java code after > you are attached) > - mark as "attach completed" > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > iterator but will have a NULL java.lang.Thread object. > > We special-case attaching threads in a number of places in the VM and I > think we should be explicitly doing something here to filter out > attaching threads, rather than just being tolerant of a NULL j.l.Thread > object. Specifically in ThreadsSMRSupport::add_thread: > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > ThreadTable::add_thread(tid, thread); > } > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > which covers the case the JNI attach encountered an error trying to > create the j.l.Thread object. > > >> src/hotspot/share/services/threadTable.cpp > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > >> The is_dead parameter still bothers me here. I can't make enough sense > >> out of the template code in ConcurrentHashtable to see why we have to > >> have it, but I'm concerned that its very existence means we perhaps > >> should not be trying to extend CHT in this context. ?? > > > > My understanding is that is_dead parameter provides a mechanism for > > ConcurrentHashtable to remove stale entries that were not explicitly > > removed by calling ConcurrentHashTable::remove() method. > > I think that just because in our case we don't use this mechanism doesn't > > mean we should not use ConcurrentHashTable. > > Can you confirm that this usage is okay with Robbin Ehn please. He's > back from vacation this week. > > >> I would still want to see what impact this has on thread > >> startup cost, both with and without the table being initialized. > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > 100,000 threads are created and started for about 15200 ms. If the thread table > > is off the test takes about 14800 ms. Based on this information the enabled > > thread table makes the thread startup about 2.7% slower. > > That doesn't sound very good. I think we may need to Claes involved to > help investigate overall performance impact here. > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > No further code comments. > > I didn't look at the test in detail. > > Thanks, > David > > > Thanks! > > --Daniil > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > Hi Daniil, > > > > Overall I think this is a reasonable approach but I would still like to > > see some performance and footprint numbers, both to verify it fixes the > > problem reported, and that we are not getting penalized elsewhere. > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > Hi David, Daniel, and Serguei, > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > The initialization allows the created but unpopulated, or partially > > populated, table to be seen by other threads - is that your intention? > > It seems it should be okay as the other threads will then race with the > > initializing thread to add specific entries, and this is a concurrent > > map so that should be functionally correct. But if so then I think you > > can also reduce the scope of the ThreadTableCreate_lock so that it > > covers creation of the table only, not the initial population of the table. > > > > I like the approach of only initializing the table when needed and using > > that to control when the add/remove-thread code needs to update the > > table. But I would still want to see what impact this has on thread > > startup cost, both with and without the table being initialized. > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > as Daniel suggested. > > > > Not sure it's best to combine these, but if they are limited to the > > changes in management.cpp only then that may be okay. It helps to be > > able to focus on the table related changes without being distracted by > > other optimizations. > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > to strip it of the all functionality that is not required in the thread table case. > > > > The revised version seems better in that regard. But I still have a > > concern, see below. > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > growing the thread table when required. > > > > Yes but why? Why can't this table be grown on demand by the thread that > > is doing the addition? For other tables we may have to delegate to the > > service thread because the current thread cannot perform the action, or > > it doesn't want to perform it at the time the need for the resize is > > detected (e.g. its detected at a safepoint and you want the resize to > > happen later outside the safepoint). It's not apparent to me that such > > restrictions apply here. > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > Ok. > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > Some specific code comments: > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > false, Monitor::_safepoint_check_never); > > > > I think this needs to be a _safepoint_check_always lock. The table will > > be created by regular JavaThreads and they should (nearly) always be > > checking for safepoints if they are going to block acquiring the lock. > > And it isn't at all obvious that the thread doing the creation can't go > > to a safepoint whilst this lock is held. > > > > --- > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > Nit: > > > > 618 JavaThread* thread = thread_at(i); > > > > you could reuse the new java_thread local you introduced at line 613 and > > just rename that "new" variable to "thread" so you don't have to change > > all other uses. > > > > 628 } else if (java_thread != NULL && ... > > > > You don't need to check != NULL here as you only get here when > > java_thread is not NULL. > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > non-null threadObj. > > > > --- > > > > src/hotspot/share/services/management.cpp > > > > 1323 if (THREAD->is_Java_thread()) { > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > These calls can only be made on a JavaThread so this be simplified to > > remove the is_Java_thread() call. Similarly in other places. > > > > --- > > > > src/hotspot/share/services/threadTable.cpp > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > I believe hotspot style is to not indent the access modifiers in C++ > > class declarations, so the above would just be: > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > etc. > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > 61 _tid(tid),_java_thread(java_thread) {} > > > > line 61 should be indented as it continues line 60. > > > > 67 class ThreadTableConfig : public AllStatic { > > ... > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > The is_dead parameter still bothers me here. I can't make enough sense > > out of the template code in ConcurrentHashtable to see why we have to > > have it, but I'm concerned that its very existence means we perhaps > > should not be trying to extend CHT in this context. ?? > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > line 116 should be indented, though in this case I think a better layout > > would be: > > > > 115 size_t start_size_log = > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > DefaultThreadTableSizeLog; > > > > 131 double ThreadTable::get_load_factor() { > > 132 return (double)_items_count/_current_size; > > 133 } > > > > Not sure that is doing what you want/expect. It will perform integer > > division and then cast that whole integer to a double. If you want > > double arithmetic you need: > > > > return ((double)_items_count)/_current_size; > > > > 180 jlong _tid; > > 181 uintx _hash; > > > > Nit: no need for all those spaces before the variable name. > > > > 183 ThreadTableLookup(jlong tid) > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > line 184 should be indented. > > > > 201 ThreadGet():_return(NULL) {} > > > > Nit: need space after : > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > 212 _has_work = false; > > > > line 211 is indented one space too far. > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > Nit: need space after , > > > > 252 return _local_table->remove(thread,lookup); > > > > Nit: need space after , > > > > Thanks, > > David > > ------ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > Hi Serguei and David, > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > in src/hotspot/share/services/management.cpp so I think that table > > > needs to enabled and populated only if it is going to be used. > > > > > > I've taken a look at the webrev below and I see that David has > > > followed up with additional comments. Before I do a crawl through > > > code review for this, I would like to see the ThreadTable stuff > > > made optional and David's other comments addressed. > > > > > > Another possible optimization is for callers of > > > find_JavaThread_from_java_tid() to save the calling thread's > > > tid value before they loop and if the current tid == saved_tid > > > then use the current JavaThread* instead of calling > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > Dan > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > From: > > > > Organization: Oracle Corporation > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > Hi Daniil, > > > > > > > > I have several quick comments. > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > 619 // to the thread table. > > > > 620 for (uint i = 0; i < length(); i++) { > > > > 621 JavaThread* thread = thread_at(i); > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > 624 return thread; > > > > 625 } > > > > 626 } > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > 628 return java_thread; > > > > 629 } > > > > 630 return NULL; > > > > 631 } > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > 633 oop tobj = java_thread->threadObj(); > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > 635 // or is starting to exit. > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > 638 } > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > A space is missed after the comma: > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > An empty line is needed before L632. > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > It'd better to list parameters in the opposite order. > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > Thanks, > > > > Serguei > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > implementation!) will need careful examination. We have to be concerned > > > > about the cost of maintaining it when it may never even be queried. You > > > > would need to look at footprint cost and performance impact. > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > next few days. I will try to look at this asap next week, but we will > > > > need a lot more data on it. > > > > > > > > Thanks, > > > > David > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > in the thread table. > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > > > > > Best regards, > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From alexey.menkov at oracle.com Mon Sep 16 19:24:23 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Mon, 16 Sep 2019 12:24:23 -0700 Subject: Ping: Re: RFR: JDK-8186825: some memory leak issues in the transport_startTransport In-Reply-To: <9531fb93-e9e0-5897-a466-8422244477ca@oracle.com> References: <9531fb93-e9e0-5897-a466-8422244477ca@oracle.com> Message-ID: <2744cb9d-5d6f-0321-97a9-f2dae8c6f5d9@oracle.com> Need 2nd review for the fix --alex On 09/05/2019 12:08, serguei.spitsyn at oracle.com wrote: > Hi Alex, > > Looks good to me. > > Thanks, > Serguei > > > On 9/5/19 11:20, Alex Menkov wrote: >> Hi all, >> >> Please review the fix for >> ? https://bugs.openjdk.java.net/browse/JDK-8186825 >> webrev: >> http://cr.openjdk.java.net/~amenkov/jdk14/jdwp_memory_leak/webrev/ >> >> TransportInfo structure is used to pass data from main thread >> (transport_startTransport function) to acceptThread/attachThread and >> should be released by acceptThread/attachThread. >> >> --alex > From hohensee at amazon.com Mon Sep 16 21:18:22 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Mon, 16 Sep 2019 21:18:22 +0000 Subject: Ping: Re: RFR: JDK-8186825: some memory leak issues in the transport_startTransport In-Reply-To: <2744cb9d-5d6f-0321-97a9-f2dae8c6f5d9@oracle.com> References: <9531fb93-e9e0-5897-a466-8422244477ca@oracle.com> <2744cb9d-5d6f-0321-97a9-f2dae8c6f5d9@oracle.com> Message-ID: <3AFDE38B-FC0A-406A-BC58-FE1D92250A0C@amazon.com> Looks good Paul ?On 9/16/19, 12:25 PM, "serviceability-dev on behalf of Alex Menkov" wrote: Need 2nd review for the fix --alex On 09/05/2019 12:08, serguei.spitsyn at oracle.com wrote: > Hi Alex, > > Looks good to me. > > Thanks, > Serguei > > > On 9/5/19 11:20, Alex Menkov wrote: >> Hi all, >> >> Please review the fix for >> https://bugs.openjdk.java.net/browse/JDK-8186825 >> webrev: >> http://cr.openjdk.java.net/~amenkov/jdk14/jdwp_memory_leak/webrev/ >> >> TransportInfo structure is used to pass data from main thread >> (transport_startTransport function) to acceptThread/attachThread and >> should be released by acceptThread/attachThread. >> >> --alex > From david.holmes at oracle.com Mon Sep 16 23:01:15 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 09:01:15 +1000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> Message-ID: <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> Hi Christoph, Sorry for the delay getting back you. cc'd build-dev to get some clarification on the below ... On 12/09/2019 7:30 pm, Langer, Christoph wrote: > Hi David, > >>> please review an enhancement which I've identified when working with >>> Processhelper for JDK-8230850. >>> >>> I noticed that ProcessHelper is an interface in common code with a >>> static method that would lookup the actual platform implementation via >>> reflection. This seems a little cumbersome since we can have a common >>> dummy for ProcessHelper and override it with the platform specific >>> implementation, leveraging the build system. >> >> I don't see you leveraging the build system. You have two source files >> that compile to the same destination class file. What is ensuring the >> platform specific version is compiled after the generic one? >> >> Service-provider patterns use reflection to instantiate the service >> implementation. I don't see any problem here that needs solving. > > TL;DR: > There are two source files, one in share/classes and one in linux/classes. The build system overrides the share/classes implementation with the linux/classes implementation in the linux build. This is not by coincidence and only one class is contained in the generated jdk.jcmd module. Then there won't be a need for having a service interface and a service implementation that is looked up via reflection (which is not a bad pattern by itself). I agree that it's not a big problem to be solved but still not "no problem". > Here is some longer elaboration how the build system prefers specific implementations of classes and filters generic duplicates: > The SetupJavaCompilation function from JavaCompilation.gmk [0] is used to compile the java sources for JDK modules. In its documentation, for argument SRC [1], it claims: "one or more directories to search for sources. The order of the source roots is significant. The first found file of a certain name has priority". In its implementation the found files are first ordered [3] and duplicates filtered out [4]. > The potential source files are handed to SetupJavaCompilation in CompileJavaModules.gmk [5] and were collected by a call to FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all potential source dirs for Java classes in the module [7]. The evaluated subdirs are (in that order) $(OPENJDK_TARGET_OS)/classes, $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. > Hope that explains what I'm trying to leverage here. I'm not 100% certain that what you describe actually ensures what you want it to ensure. I can't reconcile "the first found file ... has priority" with the fact found files are sorted and duplicates eliminated. It is the sorting that concerns me as it suggests linux/Foo.java might replace shared/Foo.java, but if we're on Windows then we have a problem! That said there is also this comment: # Order src files according to the order of the src dirs. Correct odering is # needed for correct overriding between different source roots. I'd need the build team to clarify what "correct overriding" is actually defined as. Thanks, David ----- > I've uploaded an updated webrev which contains some cleanup to the Test changes: http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ > > Thanks > Christoph > > [0] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l185 > [1] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l157 > [3] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l225 > [4] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l257 > [5] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l603 > [6] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l555 > [7] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l300 > [8] http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l243 > > From david.holmes at oracle.com Tue Sep 17 02:26:48 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 12:26:48 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: Hi Daniil, Thanks again for your perseverance on this one. I think there is a problem with initialization of the thread table. Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid and has commenced execution of ThreadTable::lazy_initialize, but not yet marked _is_initialized as true. Now two new threads (T2 and T3) are created and start running - they aren't added to the ThreadTable yet because it isn't initialized. Now T0 also calls ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If _is_initialized is still false T0 will attempt initialization but once it gets the lock it will see the table has now been initialized by T1. It will then proceed to update the table with its own ThreadList content - adding T2 and T3. That is all fine. But now suppose T0 initially sees _is_initialized as true, it will do nothing in lazy_initialize and simply return to find_JavaThread_from_java_tid. But now T2 and T3 are missing from the ThreadTable and nothing will cause them to be added. More generally any ThreadsList that is created after the ThreadsList that will be used for initialization, may contain threads that will not be added to the table. Thanks, David On 17/09/2019 4:18 am, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Tue Sep 17 04:36:01 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Mon, 16 Sep 2019 21:36:01 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Hi David, The case you have described is exact the reason why we still have a code inside ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds the requested thread to the thread table if it is not there ( lines 614-613 below). The assumption is that it's quite uncommon and even if this is the case the linear scan happens only once per such thread. 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { 612 ThreadTable::lazy_initialize(this); 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); 614 if (thread == NULL) { 615 // If the thread is not found in the table find it 616 // with a linear search and add to the table. 617 for (uint i = 0; i < length(); i++) { 618 thread = thread_at(i); 619 oop tobj = thread->threadObj(); 620 // Ignore the thread if it hasn't run yet, has exited 621 // or is starting to exit. 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } 631 } 632 } else if (!thread->is_exiting()) { 633 return thread; 634 } 635 return NULL; 636 } Thanks, Daniil ?On 9/16/19, 7:27 PM, "David Holmes" wrote: Hi Daniil, Thanks again for your perseverance on this one. I think there is a problem with initialization of the thread table. Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid and has commenced execution of ThreadTable::lazy_initialize, but not yet marked _is_initialized as true. Now two new threads (T2 and T3) are created and start running - they aren't added to the ThreadTable yet because it isn't initialized. Now T0 also calls ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If _is_initialized is still false T0 will attempt initialization but once it gets the lock it will see the table has now been initialized by T1. It will then proceed to update the table with its own ThreadList content - adding T2 and T3. That is all fine. But now suppose T0 initially sees _is_initialized as true, it will do nothing in lazy_initialize and simply return to find_JavaThread_from_java_tid. But now T2 and T3 are missing from the ThreadTable and nothing will cause them to be added. More generally any ThreadsList that is created after the ThreadsList that will be used for initialization, may contain threads that will not be added to the table. Thanks, David On 17/09/2019 4:18 am, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Tue Sep 17 05:17:03 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 15:17:03 +1000 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Message-ID: Hi Daniil, On 17/09/2019 2:36 pm, Daniil Titov wrote: > Hi David, > > The case you have described is exact the reason why we still have a code inside > ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > the requested thread to the thread table if it is not there ( lines 614-613 below). The > assumption is that it's quite uncommon and even if this is the case the linear scan happens > only once per such thread. But that is a linear scan of the current threadslist which doesn't contain the new threads ... hmmm need to think more on this. If the tid I'm looking for is valid it should belong to a thread that existed before the current ThreadsList was created to process that tid. So it would only be a problem if the tid was speculatively produced and matched one of those new threads. But that is a race anyway. Okay. Thanks for clarifying. David ----- > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > > Thanks, > Daniil > > ?On 9/16/19, 7:27 PM, "David Holmes" wrote: > > Hi Daniil, > > Thanks again for your perseverance on this one. > > I think there is a problem with initialization of the thread table. > Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid > and has commenced execution of ThreadTable::lazy_initialize, but not yet > marked _is_initialized as true. Now two new threads (T2 and T3) are > created and start running - they aren't added to the ThreadTable yet > because it isn't initialized. Now T0 also calls > ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList > that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If > _is_initialized is still false T0 will attempt initialization but once > it gets the lock it will see the table has now been initialized by T1. > It will then proceed to update the table with its own ThreadList content > - adding T2 and T3. That is all fine. But now suppose T0 initially sees > _is_initialized as true, it will do nothing in lazy_initialize and > simply return to find_JavaThread_from_java_tid. But now T2 and T3 are > missing from the ThreadTable and nothing will cause them to be added. > > More generally any ThreadsList that is created after the ThreadsList > that will be used for initialization, may contain threads that will not > be added to the table. > > Thanks, > David > > On 17/09/2019 4:18 am, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Tue Sep 17 08:53:54 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 01:53:54 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Tue Sep 17 09:04:42 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 02:04:42 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Tue Sep 17 09:10:10 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 02:10:10 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Message-ID: <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Hi Daniil, On 9/16/19 21:36, Daniil Titov wrote: > Hi David, > > The case you have described is exact the reason why we still have a code inside > ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > the requested thread to the thread table if it is not there ( lines 614-613 below). I disagree because it is easy to avoid concurrent ThreadTable initialization (please, see my separate email). The reason for this code is to cover a case of late/lazy ThreadTable initialization. Thanks, Serguei > The > assumption is that it's quite uncommon and even if this is the case the linear scan happens > only once per such thread. > > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > > Thanks, > Daniil > > ?On 9/16/19, 7:27 PM, "David Holmes" wrote: > > Hi Daniil, > > Thanks again for your perseverance on this one. > > I think there is a problem with initialization of the thread table. > Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid > and has commenced execution of ThreadTable::lazy_initialize, but not yet > marked _is_initialized as true. Now two new threads (T2 and T3) are > created and start running - they aren't added to the ThreadTable yet > because it isn't initialized. Now T0 also calls > ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList > that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If > _is_initialized is still false T0 will attempt initialization but once > it gets the lock it will see the table has now been initialized by T1. > It will then proceed to update the table with its own ThreadList content > - adding T2 and T3. That is all fine. But now suppose T0 initially sees > _is_initialized as true, it will do nothing in lazy_initialize and > simply return to find_JavaThread_from_java_tid. But now T2 and T3 are > missing from the ThreadTable and nothing will cause them to be added. > > More generally any ThreadsList that is created after the ThreadsList > that will be used for initialization, may contain threads that will not > be added to the table. > > Thanks, > David > > On 17/09/2019 4:18 am, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Tue Sep 17 09:26:29 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 02:26:29 -0700 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Tue Sep 17 10:46:07 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 20:46:07 +1000 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Message-ID: Hi Serguei, On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > > On 9/16/19 21:36, Daniil Titov wrote: >> Hi David, >> >> The case you have described is exact the reason why we still have a >> code inside >> ThreadsList::find_JavaThread_from_java_tid() method that does a linear >> scan and adds >> ? the requested thread to the thread table if it is not there ( lines >> 614-613 below). > > I disagree because it is easy to avoid concurrent ThreadTable > initialization (please, see my separate email). > The reason for this code is to cover a case of late/lazy ThreadTable > initialization. I'm not sure I follow. With the current code if two threads are racing to initialize the ThreadTable with ThreadsLists that contain a different set of threads then there are two possibilities with regards to the interleaving. Assume T1 initializes the table with its set of threads and so finds the tid it is looking for in the table. Meanwhile T2 is racing with the initialization logic: - If T2 sees _is_initialized then lazy_initialization does nothing for T2, and the additional threads in its ThreadsList (say T3 and T4) are not added to the table. But the specific thread associated with the tid (say T3) will be found by linear search of the ThreadsList and then added. If any other threads come searching for T4 they too will not find it in the ThreadTable but instead perform the linear search of their ThreadsList (and add it). - if T2 doesn't see _is_initialized at first it will try to acquire the lock, and eventually see _is_initialized is true, at which point it will try to add all of its thread's to the table (so T3 and T4 will be added). When lazy_initialize returns, T3 will be found in the table and returned. If any other threads come searching for T4 they will also find it in the table. With your suggested code change this second case is not possible so for any racing initialization the lookup of any threads not in the original ThreadsList will always result in using the linear search before adding to the table. Both seem correct to me. Which one is more efficient will totally depend on the number of differences between the ThreadsLists and whether the code ever tries to look up those additional threads. If we assume racing initialization is likely to be rare anyway (because generally one thread is in charge of doing the monitoring) then the choice seems somewhat arbitrary. Cheers, David ----- > Thanks, > Serguei > >> ??? The >> assumption is that it's quite uncommon and even if this is the case >> the linear scan happens >> only once per such thread. >> >> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >> java_tid) const { >> ? 612?? ThreadTable::lazy_initialize(this); >> ? 613?? JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); >> ? 614?? if (thread == NULL) { >> ? 615???? // If the thread is not found in the table find it >> ? 616???? // with a linear search and add to the table. >> ? 617???? for (uint i = 0; i < length(); i++) { >> ? 618?????? thread = thread_at(i); >> ? 619?????? oop tobj = thread->threadObj(); >> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ? 621?????? // or is starting to exit. >> ? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ? 623???????? MutexLocker ml(Threads_lock); >> ? 624???????? // Must be inside the lock to ensure that we don't add >> the thread to the table >> ? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ? 626???????? if (!thread->is_exiting()) { >> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ? 628?????????? return thread; >> ? 629???????? } >> ? 630?????? } >> ? 631???? } >> ? 632?? } else if (!thread->is_exiting()) { >> ? 633?????? return thread; >> ? 634?? } >> ? 635?? return NULL; >> ? 636 } >> >> Thanks, >> Daniil >> >> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >> >> ???? Hi Daniil, >> ???? Thanks again for your perseverance on this one. >> ???? I think there is a problem with initialization of the thread table. >> ???? Suppose thread T1 has called >> ThreadsList::find_JavaThread_from_java_tid >> ???? and has commenced execution of ThreadTable::lazy_initialize, but >> not yet >> ???? marked _is_initialized as true. Now two new threads (T2 and T3) are >> ???? created and start running - they aren't added to the ThreadTable yet >> ???? because it isn't initialized. Now T0 also calls >> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >> ThreadsList >> ???? that contains T2 and T3. It also calls >> ThreadTable::lazy_initialize. If >> ???? _is_initialized is still false T0 will attempt initialization but >> once >> ???? it gets the lock it will see the table has now been initialized >> by T1. >> ???? It will then proceed to update the table with its own ThreadList >> content >> ???? - adding T2 and T3. That is all fine. But now suppose T0 >> initially sees >> ???? _is_initialized as true, it will do nothing in lazy_initialize and >> ???? simply return to find_JavaThread_from_java_tid. But now T2 and T3 >> are >> ???? missing from the ThreadTable and nothing will cause them to be >> added. >> ???? More generally any ThreadsList that is created after the ThreadsList >> ???? that will be used for initialization, may contain threads that >> will not >> ???? be added to the table. >> ???? Thanks, >> ???? David >> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on the >> performance (thanks a lot Claes for helping with it!) the conclusion >> was that the impact on the thread startup time is not a blocker for >> this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just to >> remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] >> minus changes in src/hotspot/share/services/management.cpp (that were >> factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code that >> adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev:? https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that may >> be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue if >> required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c]? jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333]? JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159]? ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me here. >> I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In case >> when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time the >> thread table >> ???? >????????? >????? >????? >?? is initialized with the threads from >> the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only initializing >> the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, but >> if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then that >> may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in that >> regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori or >> deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably originally >> suggested in the patch attached to the JBS issue, should be used.? It >> will make >> ???? >????????? >????? >????? > the backporting more complicated, >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? +?? def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324?????????? JavaThread* >> current_thread = (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not indent >> the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60???? ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = size_log >> > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? >????? DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong????????? _tid; >> ???? >????????? >????? >????? 181???? uintx???????? _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces before >> the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183???? ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201???? ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211??? assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil Titov >> wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with an >> unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about the >> maintaining the thread table when it may never even be queried, one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it on >> and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as David >> H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and if >> the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot c/c++ >> files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616???? if (java_thread == >> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? >?? 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? >?? 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? >?? 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? >?? 620???????? for (uint i = 0; >> i < length(); i++) { >> ???? >????????? >????? >????? >????? >?? 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? >?? 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? >?? 623 >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? >?? 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? >?? 625???????????? } >> ???? >????????? >????? >????? >????? >?? 626???????? } >> ???? >????????? >????? >????? >????? >?? 627???? } else if >> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? >?? 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629???? } >> ???? >????????? >????? >????? >????? >?? 630???? return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633???? oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634???? // Ignore the thread >> if it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635???? // or is starting to >> exit. >> ???? >????????? >????? >????? >????? >?? 636???? return (tobj != NULL >> && !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? >?? 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name 'is_valid_java_thread' >> looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? >????? implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Unfortunately I'm just >> about to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the mapping >> between the thread ids and JavaThread objects and replaces the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> > From matthias.baesken at sap.com Tue Sep 17 11:07:43 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Tue, 17 Sep 2019 11:07:43 +0000 Subject: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code In-Reply-To: <9dca6ade-aba5-daf1-fa26-07b3d0bad35e@oracle.com> References: <9dca6ade-aba5-daf1-fa26-07b3d0bad35e@oracle.com> Message-ID: Hi Serguei and Thomas , thanks for the reviews. >Should I open a bug for these ? > Probably, two different bug are needed: hotspot/runtime and AWT. Regarding the atoi on input provided by getenv - I?ll open 2 bugs for this. Best regards, Matthias From: serguei.spitsyn at oracle.com Sent: Samstag, 14. September 2019 00:18 To: Baesken, Matthias ; Thomas St?fe Cc: serviceability-dev at openjdk.java.net Subject: Re: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code Hi Matthias, On 9/12/19 4:52 AM, Baesken, Matthias wrote: Hi Thomas, thanks for the review . You are correct about atoi . New webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/ I had 2 additional observations : 1. With OJDK on solaris 32bit gone for quite some time, we might be able to kick out the whole non _LP64 code because we are always 64 bit (maybe someone could comment if this is a safe assumption, there might be old 32bit solaris core files flying around for some reason even these days ? ) http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.1/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 696 // some older versions of libproc.so crash when trying to attach 32 bit 697 // debugger to 64 bit core file. check and throw error. 698 #ifndef _LP64 ?.. 1. The usage of atoi is commented here : https://docs.oracle.com/cd/E86824_01/html/E54766/atoi-3c.html ?However, applications should not use the atoi(), atol(), or atoll() functions unless they know the value represented by the argument will be in range for the corresponding result type? ??And here : https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html ?If the number is not known to be in range, strtol() should be used because atoi() is not required to perform any error checking? However we have a number of usages in the coding where atoi is called without knowing that the argument is in the allowed range . some examples : src/hotspot/share/runtime/arguments.cpp-382- if (match_option(option, "-Dsun.java.launcher.pid=", &tail)) { src/hotspot/share/runtime/arguments.cpp:383: _sun_java_launcher_pid = atoi(tail); src/hotspot/share/runtime/arguments.cpp-384- continue; src/java.desktop/unix/native/libawt_xawt/xawt/XToolkit.c 455 value = getenv("_AWT_MAX_POLL_TIMEOUT"); 456 if (value != NULL) { 457 AWT_MAX_POLL_TIMEOUT = atoi(value); src/java.desktop/unix/native/common/awt/X11Color.c-781- if (getenv("CMAPSIZE") != 0) { src/java.desktop/unix/native/common/awt/X11Color.c:782: cmapsize = atoi(getenv("CMAPSIZE")); Should I open a bug for these ? Probably, two different bug are needed: hotspot/runtime and AWT. Thanks, Serguei Best regards, Matthias From: Thomas St?fe Sent: Donnerstag, 12. September 2019 12:22 To: Baesken, Matthias Cc: serviceability-dev at openjdk.java.net Subject: Re: RFR [XS]: 8230901: missing ReleaseStringUTFChars in servicability native code Hi Matthias, your changes look good. an additional bug: http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp.frames.html 698 #ifndef _LP64 699 atoi(cmdLine_cstr); 700 if (errno) { Behaviour of atoi() in error case is undefined. errno values are not defined. See: https://pubs.opengroup.org/onlinepubs/009695399/functions/atoi.html And even if atoi would set errno, this is still not enough since errno may contain a stale value. One would have to set errno=0 before the function call. If you want to fix this too 'd suggest replacing this call with strtol(). Cheers, Thomas On Thu, Sep 12, 2019 at 12:11 PM Baesken, Matthias > wrote: Hello, please reviews this small change . It adds ReleaseStringUTFChars calls at some places in early return cases . ( in src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp THROW_NEW_DEBUGGER_EXCEPTION contains a return , see the macro declaration 39 #define THROW_NEW_DEBUGGER_EXCEPTION(str) { throwNewDebuggerException(env, str); return;} ) Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8230901 http://cr.openjdk.java.net/~mbaesken/webrevs/8230901.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus.ihse.bursie at oracle.com Tue Sep 17 11:26:19 2019 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 17 Sep 2019 13:26:19 +0200 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> Message-ID: <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> On 2019-09-17 01:01, David Holmes wrote: > Hi Christoph, > > Sorry for the delay getting back you. > > cc'd build-dev to get some clarification on the below ... > > On 12/09/2019 7:30 pm, Langer, Christoph wrote: >> Hi David, >> >>>> please review an enhancement which I've identified when working with >>>> Processhelper for JDK-8230850. >>>> >>>> I noticed that ProcessHelper is an interface in common code with a >>>> static method that would lookup the actual platform implementation via >>>> reflection. This seems a little cumbersome since we can have a common >>>> dummy for ProcessHelper and override it with the platform specific >>>> implementation, leveraging the build system. >>> >>> I don't see you leveraging the build system. You have two source files >>> that compile to the same destination class file. What is ensuring the >>> platform specific version is compiled after the generic one? >>> >>> Service-provider patterns use reflection to instantiate the service >>> implementation. I don't see any problem here that needs solving. >> >> TL;DR: >> There are two source files, one in share/classes and one in >> linux/classes. The build system overrides the share/classes >> implementation with the linux/classes implementation in the linux >> build. This is not by coincidence and only one class is contained in >> the generated jdk.jcmd module. Then there won't be a need for having >> a service interface and a service implementation that is looked up >> via reflection (which is not a bad pattern by itself). I agree that >> it's not a big problem to be solved but still not "no problem". >> Here is some longer elaboration how the build system prefers specific >> implementations of classes and filters generic duplicates: >> The SetupJavaCompilation function from JavaCompilation.gmk [0] is >> used to compile the java sources for JDK modules. In its >> documentation, for argument SRC [1], it claims: "one or more >> directories to search for sources. The order of the source roots is >> significant. The first found file of a certain name has priority". In >> its implementation the found files are first ordered [3] and >> duplicates filtered out [4]. >> The potential source files are handed to SetupJavaCompilation in >> CompileJavaModules.gmk [5] and were collected by a call to >> FindModuleSrcDirs [6].? FindModuleSrcDirs iterates over all potential >> source dirs for Java classes in the module [7]. The evaluated subdirs >> are (in that order) $(OPENJDK_TARGET_OS)/classes, >> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. >> Hope that explains what I'm trying to leverage here. > > I'm not 100% certain that what you describe actually ensures what you > want it to ensure. I can't reconcile "the first found file ... has > priority" with the fact found files are sorted and duplicates > eliminated. It is the sorting that concerns me as it suggests > linux/Foo.java might replace shared/Foo.java, but if we're on Windows > then we have a problem! That said there is also this comment: > > # Order src files according to the order of the src dirs. Correct > odering is > # needed for correct overriding between different source roots. > > I'd need the build team to clarify what "correct overriding" is > actually defined as. David, Christoph is correct. linux/Foo.java will override share/Foo.java. I don't remember how the magic in JavaCompilation.gmk works anymore :-), but we have relied on this behavior in other places for a long time, so I'm pretty certain it is still working correctly. Presumably, the $(sort ...) is there to remove (identical) duplicates, which is a side-effect of sort. /Magnus > > Thanks, > David > ----- > >> I've uploaded an updated webrev which contains some cleanup to the >> Test changes: http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ >> >> Thanks >> Christoph >> >> [0] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l185 >> [1] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l157 >> [3] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l225 >> [4] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l257 >> [5] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l603 >> [6] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l555 >> [7] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l300 >> [8] >> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l243 >> >> From david.holmes at oracle.com Tue Sep 17 12:59:40 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 22:59:40 +1000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> Message-ID: <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> Hi Magnus, On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: > On 2019-09-17 01:01, David Holmes wrote: >> Hi Christoph, >> >> Sorry for the delay getting back you. >> >> cc'd build-dev to get some clarification on the below ... >> >> On 12/09/2019 7:30 pm, Langer, Christoph wrote: >>> Hi David, >>> >>>>> please review an enhancement which I've identified when working with >>>>> Processhelper for JDK-8230850. >>>>> >>>>> I noticed that ProcessHelper is an interface in common code with a >>>>> static method that would lookup the actual platform implementation via >>>>> reflection. This seems a little cumbersome since we can have a common >>>>> dummy for ProcessHelper and override it with the platform specific >>>>> implementation, leveraging the build system. >>>> >>>> I don't see you leveraging the build system. You have two source files >>>> that compile to the same destination class file. What is ensuring the >>>> platform specific version is compiled after the generic one? >>>> >>>> Service-provider patterns use reflection to instantiate the service >>>> implementation. I don't see any problem here that needs solving. >>> >>> TL;DR: >>> There are two source files, one in share/classes and one in >>> linux/classes. The build system overrides the share/classes >>> implementation with the linux/classes implementation in the linux >>> build. This is not by coincidence and only one class is contained in >>> the generated jdk.jcmd module. Then there won't be a need for having >>> a service interface and a service implementation that is looked up >>> via reflection (which is not a bad pattern by itself). I agree that >>> it's not a big problem to be solved but still not "no problem". >>> Here is some longer elaboration how the build system prefers specific >>> implementations of classes and filters generic duplicates: >>> The SetupJavaCompilation function from JavaCompilation.gmk [0] is >>> used to compile the java sources for JDK modules. In its >>> documentation, for argument SRC [1], it claims: "one or more >>> directories to search for sources. The order of the source roots is >>> significant. The first found file of a certain name has priority". In >>> its implementation the found files are first ordered [3] and >>> duplicates filtered out [4]. >>> The potential source files are handed to SetupJavaCompilation in >>> CompileJavaModules.gmk [5] and were collected by a call to >>> FindModuleSrcDirs [6].? FindModuleSrcDirs iterates over all potential >>> source dirs for Java classes in the module [7]. The evaluated subdirs >>> are (in that order) $(OPENJDK_TARGET_OS)/classes, >>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. >>> Hope that explains what I'm trying to leverage here. >> >> I'm not 100% certain that what you describe actually ensures what you >> want it to ensure. I can't reconcile "the first found file ... has >> priority" with the fact found files are sorted and duplicates >> eliminated. It is the sorting that concerns me as it suggests >> linux/Foo.java might replace shared/Foo.java, but if we're on Windows >> then we have a problem! That said there is also this comment: >> >> # Order src files according to the order of the src dirs. Correct >> odering is >> # needed for correct overriding between different source roots. >> >> I'd need the build team to clarify what "correct overriding" is >> actually defined as. > David, > > Christoph is correct. linux/Foo.java will override share/Foo.java. I > don't remember how the magic in JavaCompilation.gmk works anymore :-), > but we have relied on this behavior in other places for a long time, so > I'm pretty certain it is still working correctly. Presumably, the $(sort > ...) is there to remove (identical) duplicates, which is a side-effect > of sort. Thanks for confirming. I'd still like to understand exactly what these overriding rules are though. It's not a mechanism I was aware of. Thanks, David > /Magnus > >> >> Thanks, >> David >> ----- >> >>> I've uploaded an updated webrev which contains some cleanup to the >>> Test changes: http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ >>> >>> Thanks >>> Christoph >>> >>> [0] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l185 >>> >>> [1] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l157 >>> >>> [3] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l225 >>> >>> [4] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/JavaCompilation.gmk#l257 >>> >>> [5] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l603 >>> >>> [6] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/CompileJavaModules.gmk#l555 >>> >>> [7] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l300 >>> >>> [8] >>> http://hg.openjdk.java.net/jdk/jdk/file/ea93d6a9f720/make/common/Modules.gmk#l243 >>> >>> >>> > From erik.joelsson at oracle.com Tue Sep 17 13:39:25 2019 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Tue, 17 Sep 2019 06:39:25 -0700 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> Message-ID: <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Hello, On 2019-09-17 05:59, David Holmes wrote: > Hi Magnus, > > On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: >> On 2019-09-17 01:01, David Holmes wrote: >>> Hi Christoph, >>> >>> Sorry for the delay getting back you. >>> >>> cc'd build-dev to get some clarification on the below ... >>> >>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: >>>> Hi David, >>>> >>>>>> please review an enhancement which I've identified when working with >>>>>> Processhelper for JDK-8230850. >>>>>> >>>>>> I noticed that ProcessHelper is an interface in common code with a >>>>>> static method that would lookup the actual platform >>>>>> implementation via >>>>>> reflection. This seems a little cumbersome since we can have a >>>>>> common >>>>>> dummy for ProcessHelper and override it with the platform specific >>>>>> implementation, leveraging the build system. >>>>> >>>>> I don't see you leveraging the build system. You have two source >>>>> files >>>>> that compile to the same destination class file. What is ensuring the >>>>> platform specific version is compiled after the generic one? >>>>> >>>>> Service-provider patterns use reflection to instantiate the service >>>>> implementation. I don't see any problem here that needs solving. >>>> >>>> TL;DR: >>>> There are two source files, one in share/classes and one in >>>> linux/classes. The build system overrides the share/classes >>>> implementation with the linux/classes implementation in the linux >>>> build. This is not by coincidence and only one class is contained >>>> in the generated jdk.jcmd module. Then there won't be a need for >>>> having a service interface and a service implementation that is >>>> looked up via reflection (which is not a bad pattern by itself). I >>>> agree that it's not a big problem to be solved but still not "no >>>> problem". >>>> Here is some longer elaboration how the build system prefers >>>> specific implementations of classes and filters generic duplicates: >>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] is >>>> used to compile the java sources for JDK modules. In its >>>> documentation, for argument SRC [1], it claims: "one or more >>>> directories to search for sources. The order of the source roots is >>>> significant. The first found file of a certain name has priority". >>>> In its implementation the found files are first ordered [3] and >>>> duplicates filtered out [4]. >>>> The potential source files are handed to SetupJavaCompilation in >>>> CompileJavaModules.gmk [5] and were collected by a call to >>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all >>>> potential source dirs for Java classes in the module [7]. The >>>> evaluated subdirs are (in that order) $(OPENJDK_TARGET_OS)/classes, >>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. >>>> Hope that explains what I'm trying to leverage here. >>> >>> I'm not 100% certain that what you describe actually ensures what >>> you want it to ensure. I can't reconcile "the first found file ... >>> has priority" with the fact found files are sorted and duplicates >>> eliminated. It is the sorting that concerns me as it suggests >>> linux/Foo.java might replace shared/Foo.java, but if we're on >>> Windows then we have a problem! That said there is also this comment: >>> >>> # Order src files according to the order of the src dirs. Correct >>> odering is >>> # needed for correct overriding between different source roots. >>> >>> I'd need the build team to clarify what "correct overriding" is >>> actually defined as. >> David, >> >> Christoph is correct. linux/Foo.java will override share/Foo.java. I >> don't remember how the magic in JavaCompilation.gmk works anymore >> :-), but we have relied on this behavior in other places for a long >> time, so I'm pretty certain it is still working correctly. >> Presumably, the $(sort ...) is there to remove (identical) >> duplicates, which is a side-effect of sort. > > Thanks for confirming. I'd still like to understand exactly what these > overriding rules are though. It's not a mechanism I was aware of. > SetupJavaCompilation is indeed behaving as Christoph describes and it is by design. I implemented support for this behavior in: https://bugs.openjdk.java.net/browse/JDK-8079344 The relevant parts of SetupJavaCompilation look like this: ? # Order src files according to the order of the src dirs. Correct odering is ? # needed for correct overriding between different source roots. ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) The second line orders the src files by the src roots. (We used to just call find for one src root at a time, but the above actually performs better due only running 1 external process) Further down we have this: ? ifneq ($$($1_KEEP_DUPS), true) ??? # Remove duplicate source files by keeping the first found of each duplicate. ??? # This allows for automatic overrides with custom or platform specific versions ??? # source files. ??? # ??? # For the smart javac wrapper case, add each removed file to an extra exclude ??? # file list to prevent sjavac from finding duplicate sources. ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), $$(s))) \ ??????? $$(if $$($1_$$(relative_src)), \ ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) ? endif This loop is a bit hairy to wrap your head around. It's iterating over all the src files, in the order of importance. The variable relative_src is the path from the src root, the part that is common to all duplicate src files. The variables on the form $1_$$(relative_src) basically act as a hash map (string->boolean). So for each src file, if the relative path for it has already been seen, add it to an exclude list, else mark it as seen and add it to the return list. /Erik From hohensee at amazon.com Tue Sep 17 14:10:26 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 17 Sep 2019 14:10:26 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> Message-ID: <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> Thanks, Serguei. :) David, are you ok with the patch? Paul From: "serguei.spitsyn at oracle.com" Date: Tuesday, September 17, 2019 at 2:26 AM To: "Hohensee, Paul" , David Holmes , Mandy Chung Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, Thank you for refactoring and fixing the test. It looks great now! Thanks, Serguei On 9/15/19 02:52, Hohensee, Paul wrote: Hi, Serguei, thanks for the review. New webrev at http://cr.openjdk.java.net/~phh/8207266/webrev.09/ I refactored the test?s main() method, and you?re correct, getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in that context: fixed. Paul From: "serguei.spitsyn at oracle.com" Organization: Oracle Corporation Date: Friday, September 13, 2019 at 5:50 PM To: "Hohensee, Paul" , David Holmes , Mandy Chung Cc: OpenJDK Serviceability , "hotspot-gc-dev at openjdk.java.net" Subject: Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Hi Paul, It looks pretty good in general. http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html It would be nice to refactor the java main() method as it becomes too big. Two ways of getCurrentThreadAllocatedBytes() testing are good candidates to become separate methods. 98 long size1 = mbean.getThreadAllocatedBytes(id); Just wanted to double check if you wanted to invoke the getCurrentThreadAllocatedBytes() instead as it is a part of: 85 // First way, getCurrentThreadAllocatedBytes Thanks, Serguei On 9/13/19 12:11 PM, Hohensee, Paul wrote: Hi David, thanks for your comments. New webrev in http://cr.openjdk.java.net/~phh/8207266/webrev.08/ Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. Paul On 9/13/19, 12:50 AM, "David Holmes" wrote: Hi Paul, On 13/09/2019 10:29 am, Hohensee, Paul wrote: > Thanks for clarifying the review rules. Would someone from the > serviceability team please review? New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.07/ One aspect of the functional change needs clarification for me - and apologies if this has been covered in the past. It seems to me that currently we only check isThreadAllocatedMemorySupported for these operations, but if I read things correctly the updated code additionally checks isThreadAllocatedMemoryEnabled, which is a behaviour change not mentioned in the CSR. > I didn?t disturb the existing checks in the test, just added code to > check the result of getThreadAllocatedBytes(long) on a non-current > thread, plus the back-to-back no-allocation checks. The former wasn?t > needed before because getThreadAllocatedBytes(long) was just a wrapper > around getThreadAllocatedBytes(long []). This patch changes that, so I > added a separate test. The latter is supposed to fail if there?s object > allocation on calls to getCurrentThreadAllocatedBytes and > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > accumulation of transient small objects can be a performance problem. > Thanks to your review, I noticed that the back-to-back check on the > current thread was using getThreadAllocatedBytes(long) instead of > getCurrentThreadAllocatedBytes and fixed it. I also removed all > instances of ?TEST FAILED: ?. The back-to-back check is not valid in general. You don't know if the first check might trigger some class loading on the return path after it has obtained the first memory value. The check might also fail if using JVMCI and some compilation related activity occurs in the current thread on the second call. Also with the introduction of handshakes its possible the current thread might hit a safepoint checks that results in it executing a handshake operation that performs allocation. Potentially there could be numerous non-deterministic actions that might occur leading to unanticipated allocation. I understand what you want to test here, I just don't think it is reliably doable. Thanks, David ----- > > Paul > > *From: *Mandy Chung > *Date: *Thursday, September 12, 2019 at 10:09 AM > *To: *"Hohensee, Paul" > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > I only reviewed the library side implementation that looks good. I > expect the serviceability team to review the test and hotspot change. > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > You need another reviewer to advice the following because I was not > close to the ThreadsList work. > > 2087 ThreadsListHandle tlh; > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > 2089 > > 2090 if (java_thread != NULL) { > > 2091 return java_thread->cooked_allocated_bytes(); > > 2092 } > > This looks right to me. > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > - "ThreadAllocatedMemory is expected to be disabled"); > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > disabled"); > > Prepending "TEST FAILED" in exception message (in several places) > > seems redundant since such RuntimeException is thrown and expected > > a test failure. > > + // back-to-back calls shouldn't allocate any memory > > + size = mbean.getThreadAllocatedBytes(id); > > + size1 = mbean.getThreadAllocatedBytes(id); > > + if (size1 != size) { > > Is there anything in the test can do to help guarantee this? I didn't > > closely review this test. The main thing I advice is to improve > > the reliability of this test. Put it in another way, we want to > > ensure that this test change will pass all the time in various > > test configuration. > > Mandy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Tue Sep 17 22:50:24 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Sep 2019 08:50:24 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> Message-ID: <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> On 18/09/2019 12:10 am, Hohensee, Paul wrote: > Thanks, Serguei. :) > > David, are you ok with the patch? Yep, nothing further from me. David > Paul > > *From: *"serguei.spitsyn at oracle.com" > *Date: *Tuesday, September 17, 2019 at 2:26 AM > *To: *"Hohensee, Paul" , David Holmes > , Mandy Chung > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > Hi Paul, > > Thank you for refactoring and fixing the test. > It looks great now! > > Thanks, > Serguei > > > On 9/15/19 02:52, Hohensee, Paul wrote: > > Hi, Serguei, thanks for the review. New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > I refactored the test?s main() method, and you?re correct, > getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > that context: fixed. > > Paul > > *From: *"serguei.spitsyn at oracle.com" > > > *Organization: *Oracle Corporation > *Date: *Friday, September 13, 2019 at 5:50 PM > *To: *"Hohensee, Paul" > , David Holmes > , Mandy Chung > > *Cc: *OpenJDK Serviceability > , > "hotspot-gc-dev at openjdk.java.net" > > > > *Subject: *Re: RFR (M): 8207266: > ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > Hi Paul, > > It looks pretty good in general. > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > It would be nice to refactor the java main() method as it becomes > too big. > Two ways ofgetCurrentThreadAllocatedBytes() testing are good candidates > to become separate methods. > > ? 98?? ??????long size1 = mbean.getThreadAllocatedBytes(id); > > Just wanted to double check if you wanted to invoke > the getCurrentThreadAllocatedBytes() instead as it is > a part of: > > ? 85???????? // First way, getCurrentThreadAllocatedBytes > > > Thanks, > Serguei > > On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > Hi David, thanks for your comments. New webrev in > > > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > > Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. > > > > You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. > > > > Paul > > > > On 9/13/19, 12:50 AM, "David Holmes" wrote: > > > > ??? Hi Paul, > > > > ????On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > ??? > Thanks for clarifying the review rules. Would someone from the > > ????> serviceability team please review? New webrev at > > ??? > > > ????>http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > > ????One aspect of the functional change needs clarification for me - and > > ????apologies if this has been covered in the past. It seems to me that > > ????currently we only check isThreadAllocatedMemorySupported for these > > ????operations, but if I read things correctly the updated code additionally > > ????checks isThreadAllocatedMemoryEnabled, which is a behaviour change not > > ????mentioned in the CSR. > > > > ????> I didn?t disturb the existing checks in the test, just added code to > > ????> check the result of getThreadAllocatedBytes(long) on a non-current > > ????> thread, plus the back-to-back no-allocation checks. The former wasn?t > > ????> needed before because getThreadAllocatedBytes(long) was just a wrapper > > ????> around getThreadAllocatedBytes(long []). This patch changes that, so I > > ????> added a separate test. The latter is supposed to fail if there?s object > > ????> allocation on calls to getCurrentThreadAllocatedBytes and > > ????> getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > > ????> accumulation of transient small objects can be a performance problem. > > ????> Thanks to your review, I noticed that the back-to-back check on the > > ????> current thread was using getThreadAllocatedBytes(long) instead of > > ????> getCurrentThreadAllocatedBytes and fixed it. I also removed all > > ????> instances of ?TEST FAILED: ?. > > > > ????The back-to-back check is not valid in general. You don't know if the > > ????first check might trigger some class loading on the return path after it > > ????has obtained the first memory value. The check might also fail if using > > ????JVMCI and some compilation related activity occurs in the current thread > > ????on the second call. Also with the introduction of handshakes its > > ????possible the current thread might hit a safepoint checks that results in > > ????it executing a handshake operation that performs allocation. Potentially > > ????there could be numerous non-deterministic actions that might occur > > ????leading to unanticipated allocation. > > > > ????I understand what you want to test here, I just don't think it is > > ????reliably doable. > > > > ????Thanks, > > ??? David > > ??? ----- > > > > ????> > > ????> Paul > > ??? > > > ????> *From: *Mandy Chung > > ??? > *Date: *Thursday, September 12, 2019 at 10:09 AM > > ??? > *To: *"Hohensee, Paul" > > ??? > *Cc: *OpenJDK Serviceability , > > ????>"hotspot-gc-dev at openjdk.java.net" > > ??? > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > > ????> can be quicker for self thread > > ??? > > > ????> On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > ??? > > > ????>???? Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > ??? > > > ????> > > ????> I only reviewed the library side implementation that looks good.? I > > ????> expect the serviceability team to review the test and hotspot change. > > ??? > > > ????> > > ????>???? Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > ??? > > > ????> > > ????> You need another reviewer to advice the following because I was not > > ????> close to the ThreadsList work. > > ??? > > > ????> 2087?? ThreadsListHandle tlh; > > ??? > > > ????> 2088?? JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > ??? > > > ????> 2089 > > ??? > > > ????> 2090?? if (java_thread != NULL) { > > ??? > > > ????> 2091???? return java_thread->cooked_allocated_bytes(); > > ??? > > > ????> 2092?? } > > ??? > > > ????> This looks right to me. > > ??? > > > ????> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > ??? > > > ????> -??????????????? "ThreadAllocatedMemory is expected to be disabled"); > > ??? > > > ????> +??????????????? "TEST FAILED: ThreadAllocatedMemory is expected to be > > ????> disabled"); > > ??? > > > ????> Prepending "TEST FAILED" in exception message (in several places) > > ??? > > > ????> seems redundant since such RuntimeException is thrown and expected > > ??? > > > ????> a test failure. > > ??? > > > ????> +??????? // back-to-back calls shouldn't allocate any memory > > ??? > > > ????> +??????? size = mbean.getThreadAllocatedBytes(id); > > ??? > > > ????> +??????? size1 = mbean.getThreadAllocatedBytes(id); > > ??? > > > ????> +??????? if (size1 != size) { > > ??? > > > ????> Is there anything in the test can do to help guarantee this? I didn't > > ??? > > > ????> closely review this test.? The main thing I advice is to improve > > ??? > > > ????> the reliability of this test.? Put it in another way, we want to > > ??? > > > ????> ensure that this test change will pass all the time in various > > ??? > > > ????> test configuration. > > ??? > > > ????> Mandy > > ??? > > > > > > > > From david.holmes at oracle.com Tue Sep 17 23:12:49 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Sep 2019 09:12:49 +1000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Message-ID: Hi Erik, Thanks for the additional details (I can't say I fully understand them :) ). David On 17/09/2019 11:39 pm, Erik Joelsson wrote: > Hello, > > On 2019-09-17 05:59, David Holmes wrote: >> Hi Magnus, >> >> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: >>> On 2019-09-17 01:01, David Holmes wrote: >>>> Hi Christoph, >>>> >>>> Sorry for the delay getting back you. >>>> >>>> cc'd build-dev to get some clarification on the below ... >>>> >>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: >>>>> Hi David, >>>>> >>>>>>> please review an enhancement which I've identified when working with >>>>>>> Processhelper for JDK-8230850. >>>>>>> >>>>>>> I noticed that ProcessHelper is an interface in common code with a >>>>>>> static method that would lookup the actual platform >>>>>>> implementation via >>>>>>> reflection. This seems a little cumbersome since we can have a >>>>>>> common >>>>>>> dummy for ProcessHelper and override it with the platform specific >>>>>>> implementation, leveraging the build system. >>>>>> >>>>>> I don't see you leveraging the build system. You have two source >>>>>> files >>>>>> that compile to the same destination class file. What is ensuring the >>>>>> platform specific version is compiled after the generic one? >>>>>> >>>>>> Service-provider patterns use reflection to instantiate the service >>>>>> implementation. I don't see any problem here that needs solving. >>>>> >>>>> TL;DR: >>>>> There are two source files, one in share/classes and one in >>>>> linux/classes. The build system overrides the share/classes >>>>> implementation with the linux/classes implementation in the linux >>>>> build. This is not by coincidence and only one class is contained >>>>> in the generated jdk.jcmd module. Then there won't be a need for >>>>> having a service interface and a service implementation that is >>>>> looked up via reflection (which is not a bad pattern by itself). I >>>>> agree that it's not a big problem to be solved but still not "no >>>>> problem". >>>>> Here is some longer elaboration how the build system prefers >>>>> specific implementations of classes and filters generic duplicates: >>>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] is >>>>> used to compile the java sources for JDK modules. In its >>>>> documentation, for argument SRC [1], it claims: "one or more >>>>> directories to search for sources. The order of the source roots is >>>>> significant. The first found file of a certain name has priority". >>>>> In its implementation the found files are first ordered [3] and >>>>> duplicates filtered out [4]. >>>>> The potential source files are handed to SetupJavaCompilation in >>>>> CompileJavaModules.gmk [5] and were collected by a call to >>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all >>>>> potential source dirs for Java classes in the module [7]. The >>>>> evaluated subdirs are (in that order) $(OPENJDK_TARGET_OS)/classes, >>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. >>>>> Hope that explains what I'm trying to leverage here. >>>> >>>> I'm not 100% certain that what you describe actually ensures what >>>> you want it to ensure. I can't reconcile "the first found file ... >>>> has priority" with the fact found files are sorted and duplicates >>>> eliminated. It is the sorting that concerns me as it suggests >>>> linux/Foo.java might replace shared/Foo.java, but if we're on >>>> Windows then we have a problem! That said there is also this comment: >>>> >>>> # Order src files according to the order of the src dirs. Correct >>>> odering is >>>> # needed for correct overriding between different source roots. >>>> >>>> I'd need the build team to clarify what "correct overriding" is >>>> actually defined as. >>> David, >>> >>> Christoph is correct. linux/Foo.java will override share/Foo.java. I >>> don't remember how the magic in JavaCompilation.gmk works anymore >>> :-), but we have relied on this behavior in other places for a long >>> time, so I'm pretty certain it is still working correctly. >>> Presumably, the $(sort ...) is there to remove (identical) >>> duplicates, which is a side-effect of sort. >> >> Thanks for confirming. I'd still like to understand exactly what these >> overriding rules are though. It's not a mechanism I was aware of. >> > SetupJavaCompilation is indeed behaving as Christoph describes and it is > by design. I implemented support for this behavior in: > > https://bugs.openjdk.java.net/browse/JDK-8079344 > > The relevant parts of SetupJavaCompilation look like this: > > ? # Order src files according to the order of the src dirs. Correct > odering is > ? # needed for correct overriding between different source roots. > ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) > ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ > ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) > > The second line orders the src files by the src roots. (We used to just > call find for one src root at a time, but the above actually performs > better due only running 1 external process) > > Further down we have this: > > ? ifneq ($$($1_KEEP_DUPS), true) > ??? # Remove duplicate source files by keeping the first found of each > duplicate. > ??? # This allows for automatic overrides with custom or platform > specific versions > ??? # source files. > ??? # > ??? # For the smart javac wrapper case, add each removed file to an > extra exclude > ??? # file list to prevent sjavac from finding duplicate sources. > ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ > ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), > $$(s))) \ > ??????? $$(if $$($1_$$(relative_src)), \ > ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ > ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) > ? endif > > This loop is a bit hairy to wrap your head around. It's iterating over > all the src files, in the order of importance. The variable relative_src > is the path from the src root, the part that is common to all duplicate > src files. The variables on the form $1_$$(relative_src) basically act > as a hash map (string->boolean). So for each src file, if the relative > path for it has already been seen, add it to an exclude list, else mark > it as seen and add it to the return list. > > /Erik > From daniil.x.titov at oracle.com Wed Sep 18 00:13:57 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 17 Sep 2019 17:13:57 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> Message-ID: <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Hi Serguei, Please find below my answers to the concerns you mentioned in the previous email. 1. > I have a concern about the checks for thread->is_exiting(). > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > Does it make sense to develop a test that would cover these cases? I agree, it doesn't really provide any protection so it makes sense just remove it. The current implementation find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. 2. > - the lines 105-108 can result in adding exiting threads into the ThreadTable I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() 3. > I would suggest to rewrite this fragment in a safe way: > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > as: > { > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > return; > } > create_table(threads->length()); > _is_initialized = true; > } It was an intension to not block while populating the table with the threads from the current thread list. There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for it to complete since the requested thread could be not present in the thread list that triggers the thread table initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table and thus avoid the linear scan when these thread are looked up for the first time. 4. >> The case you have described is exact the reason why we still have a code inside >> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >> the requested thread to the thread table if it is not there ( lines 614-613 below). > I disagree because it is easy to avoid concurrent ThreadTable > initialization (please, see my separate email). > The reason for this code is to cover a case of late/lazy ThreadTable > initialization. David Holmes replied to this in a separate email providing a very detailed explanation of the possible cases and how the proposed implementation satisfies them. Best regards, Daniil From: "serguei.spitsyn at oracle.com" Date: Tuesday, September 17, 2019 at 1:53 AM To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) Hi Daniil, Thank you for you patience in working on this issue! Also, I like that the current thread related optimizations in management.cpp were factored out. It was a good idea to separate them. I have a concern about the checks for thread->is_exiting(). The threads are added to and removed from the ThreadTable under protection of Threads_lock. However, the thread->is_exiting() checks are not protected, and so, they are racy. There is a couple of such checks to mention: 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { 612 ThreadTable::lazy_initialize(this); 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); 614 if (thread == NULL) { 615 // If the thread is not found in the table find it 616 // with a linear search and add to the table. 617 for (uint i = 0; i < length(); i++) { 618 thread = thread_at(i); 619 oop tobj = thread->threadObj(); 620 // Ignore the thread if it hasn't run yet, has exited 621 // or is starting to exit. 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } 631 } 632 } else if (!thread->is_exiting()) { 633 return thread; 634 } 635 return NULL; 636 } ? ... 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { 94 if (!_is_initialized) { 95 { 96 MutexLocker ml(ThreadTableCreate_lock); 97 if (!_is_initialized) { 98 create_table(threads->length()); 99 _is_initialized = true; 100 } 101 } 102 for (uint i = 0; i < threads->length(); i++) { 103 JavaThread* thread = threads->thread_at(i); 104 oop tobj = thread->threadObj(); 105 if (tobj != NULL && !thread->is_exiting()) { 106 jlong java_tid = java_lang_Thread::thread_id(tobj); 107 add_thread(java_tid, thread); 108 } 109 } 110 } 111 } A thread may start exiting right after the checks at the lines 626 and 105. So that: ?- the lines 632-633 are useless as they do not really protect from returning an exiting thread ?- the lines 105-108 can result in adding exiting threads into the ThreadTable Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they are protected with the Threads_lock. But the returned thread still can exit after that. It is interesting what might happen if an exiting thread is returned by the ThreadsList::find_JavaThread_from_java_tid (). Does it make sense to develop a test that would cover these cases? Thanks, Serguei On 9/16/19 11:18, Daniil Titov wrote: Hello, After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. src/hotspot/share/runtime/threadSMR.cpp 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 [4] https://bugs.openjdk.java.net/browse/JDK-8229391 ?Thank you, Daniil > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > Hi Daniil, > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > Hi David, > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > the changes you suggested: > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > - ThreadTableCreate_lock is made _safepoint_check_always; > > Okay. > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > the thread table is changed to grow on demand by the thread that is doing the addition; > > Okay - I'm happy to get the serviceThread out of the picture here. > > > - fixed nits and formatting issues. > > Okay. > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > >>> as Daniel suggested. > >> Not sure it's best to combine these, but if they are limited to the > >> changes in management.cpp only then that may be okay. > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > limited to management.cpp (plus a new test) so I left them in the webrev but > > I also could move it in the separate issue if required. > > I'd prefer this part of be separated out, but won't insist. Let's see if > Dan or Serguei have a strong opinion. > > > > src/hotspot/share/runtime/threadSMR.cpp > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > I think it cleaner/better to just use > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > I had to leave this code unchanged since it turned out the threadObj is null > > when VM is destroyed: > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > C [libjli.so+0x4333] JavaMain+0x2c3 > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > This is actually nothing to do with the VM being destroyed, but is an > issue with JNI_AttachCurrentThread and its interaction with the > ThreadSMR iterators. The attach process is: > - create JavaThread > - mark as "is attaching via jni" > - add to ThreadsList > - create java.lang.Thread object (you can only execute Java code after > you are attached) > - mark as "attach completed" > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > iterator but will have a NULL java.lang.Thread object. > > We special-case attaching threads in a number of places in the VM and I > think we should be explicitly doing something here to filter out > attaching threads, rather than just being tolerant of a NULL j.l.Thread > object. Specifically in ThreadsSMRSupport::add_thread: > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > ThreadTable::add_thread(tid, thread); > } > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > which covers the case the JNI attach encountered an error trying to > create the j.l.Thread object. > > >> src/hotspot/share/services/threadTable.cpp > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > >> The is_dead parameter still bothers me here. I can't make enough sense > >> out of the template code in ConcurrentHashtable to see why we have to > >> have it, but I'm concerned that its very existence means we perhaps > >> should not be trying to extend CHT in this context. ?? > > > > My understanding is that is_dead parameter provides a mechanism for > > ConcurrentHashtable to remove stale entries that were not explicitly > > removed by calling ConcurrentHashTable::remove() method. > > I think that just because in our case we don't use this mechanism doesn't > > mean we should not use ConcurrentHashTable. > > Can you confirm that this usage is okay with Robbin Ehn please. He's > back from vacation this week. > > >> I would still want to see what impact this has on thread > >> startup cost, both with and without the table being initialized. > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > 100,000 threads are created and started for about 15200 ms. If the thread table > > is off the test takes about 14800 ms. Based on this information the enabled > > thread table makes the thread startup about 2.7% slower. > > That doesn't sound very good. I think we may need to Claes involved to > help investigate overall performance impact here. > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > No further code comments. > > I didn't look at the test in detail. > > Thanks, > David > > > Thanks! > > --Daniil > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > Overall I think this is a reasonable approach but I would still like to > > see some performance and footprint numbers, both to verify it fixes the > > problem reported, and that we are not getting penalized elsewhere. > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > Hi David, Daniel, and Serguei, > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > The initialization allows the created but unpopulated, or partially > > populated, table to be seen by other threads - is that your intention? > > It seems it should be okay as the other threads will then race with the > > initializing thread to add specific entries, and this is a concurrent > > map so that should be functionally correct. But if so then I think you > > can also reduce the scope of the ThreadTableCreate_lock so that it > > covers creation of the table only, not the initial population of the table. > > > > I like the approach of only initializing the table when needed and using > > that to control when the add/remove-thread code needs to update the > > table. But I would still want to see what impact this has on thread > > startup cost, both with and without the table being initialized. > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > as Daniel suggested. > > > > Not sure it's best to combine these, but if they are limited to the > > changes in management.cpp only then that may be okay. It helps to be > > able to focus on the table related changes without being distracted by > > other optimizations. > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > to strip it of the all functionality that is not required in the thread table case. > > > > The revised version seems better in that regard. But I still have a > > concern, see below. > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > growing the thread table when required. > > > > Yes but why? Why can't this table be grown on demand by the thread that > > is doing the addition? For other tables we may have to delegate to the > > service thread because the current thread cannot perform the action, or > > it doesn't want to perform it at the time the need for the resize is > > detected (e.g. its detected at a safepoint and you want the resize to > > happen later outside the safepoint). It's not apparent to me that such > > restrictions apply here. > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > Ok. > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > Some specific code comments: > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > false, Monitor::_safepoint_check_never); > > > > I think this needs to be a _safepoint_check_always lock. The table will > > be created by regular JavaThreads and they should (nearly) always be > > checking for safepoints if they are going to block acquiring the lock. > > And it isn't at all obvious that the thread doing the creation can't go > > to a safepoint whilst this lock is held. > > > > --- > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > Nit: > > > > 618 JavaThread* thread = thread_at(i); > > > > you could reuse the new java_thread local you introduced at line 613 and > > just rename that "new" variable to "thread" so you don't have to change > > all other uses. > > > > 628 } else if (java_thread != NULL && ... > > > > You don't need to check != NULL here as you only get here when > > java_thread is not NULL. > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > non-null threadObj. > > > > --- > > > > src/hotspot/share/services/management.cpp > > > > 1323 if (THREAD->is_Java_thread()) { > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > These calls can only be made on a JavaThread so this be simplified to > > remove the is_Java_thread() call. Similarly in other places. > > > > --- > > > > src/hotspot/share/services/threadTable.cpp > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > I believe hotspot style is to not indent the access modifiers in C++ > > class declarations, so the above would just be: > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > etc. > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > 61 _tid(tid),_java_thread(java_thread) {} > > > > line 61 should be indented as it continues line 60. > > > > 67 class ThreadTableConfig : public AllStatic { > > ... > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > The is_dead parameter still bothers me here. I can't make enough sense > > out of the template code in ConcurrentHashtable to see why we have to > > have it, but I'm concerned that its very existence means we perhaps > > should not be trying to extend CHT in this context. ?? > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > line 116 should be indented, though in this case I think a better layout > > would be: > > > > 115 size_t start_size_log = > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > DefaultThreadTableSizeLog; > > > > 131 double ThreadTable::get_load_factor() { > > 132 return (double)_items_count/_current_size; > > 133 } > > > > Not sure that is doing what you want/expect. It will perform integer > > division and then cast that whole integer to a double. If you want > > double arithmetic you need: > > > > return ((double)_items_count)/_current_size; > > > > 180 jlong _tid; > > 181 uintx _hash; > > > > Nit: no need for all those spaces before the variable name. > > > > 183 ThreadTableLookup(jlong tid) > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > line 184 should be indented. > > > > 201 ThreadGet():_return(NULL) {} > > > > Nit: need space after : > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > 212 _has_work = false; > > > > line 211 is indented one space too far. > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > Nit: need space after , > > > > 252 return _local_table->remove(thread,lookup); > > > > Nit: need space after , > > > > Thanks, > > David > > ------ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > Hi Serguei and David, > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > in src/hotspot/share/services/management.cpp so I think that table > > > needs to enabled and populated only if it is going to be used. > > > > > > I've taken a look at the webrev below and I see that David has > > > followed up with additional comments. Before I do a crawl through > > > code review for this, I would like to see the ThreadTable stuff > > > made optional and David's other comments addressed. > > > > > > Another possible optimization is for callers of > > > find_JavaThread_from_java_tid() to save the calling thread's > > > tid value before they loop and if the current tid == saved_tid > > > then use the current JavaThread* instead of calling > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > Dan > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > Organization: Oracle Corporation > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > Hi Daniil, > > > > > > > > I have several quick comments. > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > 619 // to the thread table. > > > > 620 for (uint i = 0; i < length(); i++) { > > > > 621 JavaThread* thread = thread_at(i); > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > 624 return thread; > > > > 625 } > > > > 626 } > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > 628 return java_thread; > > > > 629 } > > > > 630 return NULL; > > > > 631 } > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > 633 oop tobj = java_thread->threadObj(); > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > 635 // or is starting to exit. > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > 638 } > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > A space is missed after the comma: > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > An empty line is needed before L632. > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > It'd better to list parameters in the opposite order. > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > Thanks, > > > > Serguei > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > implementation!) will need careful examination. We have to be concerned > > > > about the cost of maintaining it when it may never even be queried. You > > > > would need to look at footprint cost and performance impact. > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > next few days. I will try to look at this asap next week, but we will > > > > need a lot more data on it. > > > > > > > > Thanks, > > > > David > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > in the thread table. > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > > > > > Best regards, > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Wed Sep 18 06:26:43 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Sep 2019 16:26:43 +1000 Subject: RFR (XXXXS): 8231162: JVMTI RawMonitorWait triggers assertion failure: Only JavaThreads can be interruptible Message-ID: Bug: https://bugs.openjdk.java.net/browse/JDK-8231162 webrev: http://cr.openjdk.java.net/~dholmes/8231162/webrev/ - r = rmonitor->raw_wait(millis, true, thread); + r = rmonitor->raw_wait(millis, false, thread); Non-JavaThreads are not interruptible and so "true" should not have been being passed. This tripped over the assertions added as part of the movement of the interrupt code to JavaThread under JDK-8230424. Dan: FYI I overlooked this because I already rewrote all this RawMonitor logic under "8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor" to do the right thing, but of course that hasn't been pushed yet. And this isn't detected until tier 4 testing. Thanks, David From serguei.spitsyn at oracle.com Wed Sep 18 07:13:15 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 00:13:15 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Message-ID: <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Wed Sep 18 07:25:29 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Sep 2019 17:25:29 +1000 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> Message-ID: <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> Hi Serguei, In the interests of full disclosure I was the one who told Daniil to not hold the lock while populating the table. I'll leave you two to work out which way you want to go there. Thanks, David On 18/09/2019 5:13 pm, serguei.spitsyn at oracle.com wrote: > Hi David, > > > On 9/17/19 03:46, David Holmes wrote: >> Hi Serguei, >> >> On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: >>> Hi Daniil, >>> >>> >>> On 9/16/19 21:36, Daniil Titov wrote: >>>> Hi David, >>>> >>>> The case you have described is exact the reason why we still have a >>>> code inside >>>> ThreadsList::find_JavaThread_from_java_tid() method that does a >>>> linear scan and adds >>>> ? the requested thread to the thread table if it is not there ( >>>> lines 614-613 below). >>> >>> I disagree because it is easy to avoid concurrent ThreadTable >>> initialization (please, see my separate email). >>> The reason for this code is to cover a case of late/lazy ThreadTable >>> initialization. >> >> I'm not sure I follow. With the current code if two threads are racing >> to initialize the ThreadTable with ThreadsLists that contain a >> different set of threads then there are two possibilities with regards >> to the interleaving. Assume T1 initializes the table with its set of >> threads and so finds the tid it is looking for in the table. Meanwhile >> T2 is racing with the initialization logic: >> >> - If T2 sees _is_initialized then lazy_initialization does nothing for >> T2, and the additional threads in its ThreadsList (say T3 and T4) are >> not added to the table. But the specific thread associated with the >> tid (say T3) will be found by linear search of the ThreadsList and >> then added. If any other threads come searching for T4 they too will >> not find it in the ThreadTable but instead perform the linear search >> of their ThreadsList (and add it). >> >> - if T2 doesn't see _is_initialized at first it will try to acquire >> the lock, and eventually see _is_initialized is true, at which point >> it will try to add all of its thread's to the table (so T3 and T4 will >> be added). When lazy_initialize returns, T3 will be found in the table >> and returned. If any other threads come searching for T4 they will >> also find it in the table. > > My main concerns are simplicity and reliability. > I do no care much about extra overhead at the ThreadTable initialization. > A probability of the ThreadTable::lazy_initialize() being called > concurrently is low. > Also, it might happen only once for the whole VM process execution. > > I was wrong by thinking that adding new threads to the ThreadTable after > its initialization > will result in thread linear search as well. So my conclusion was that > we should not care > if it happens once more at lazy initialization point. > But now I see that after ThreadTable::is_initialized() returns true the > ThreadsSMRSupport::add_thread() > makes a call to the ThreadTable::add_thread(). So, no more linear search > will happen. > > > However, it seems to me, a possible concurrent lazy initialization in > the webrev.06 introduces > its own extra overhead - competing threads (suppose, we have just two of > them) will do > the same amount of work concurrently: > ?? - all indirect memory readings, lookup/comparisons and other checks > will be performed twice > ?? - new ThreadTableEntry can be allocated twice for some threads in > the list > ???? (depending on how ConcurrentHashTable is implemented, there can be > a potential a memory leak) > > So, I doubt we win much in performance here but can loose in reliability. > > > I'd suggest to simplify the lazy initialization code and make it more > reliable this way: > > if (!_is_initialized) { > MutexLocker ml(ThreadTableCreate_lock); > if (!_is_initialized) { > create_table(threads->length()); > _is_initialized = true; > } > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL && !thread->is_exiting()) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > add_thread(java_tid, thread); > } > } > } > >> With your suggested code change this second case is not possible so >> for any racing initialization the lookup of any threads not in the >> original ThreadsList will always result in using the linear search >> before adding to the table. > > Yes, but I did not care about this. > The overhead is expected to be lower than the lazy initialization cost, > especially because the probability of concurrent initialization is low. > >> Both seem correct to me. Which one is more efficient will totally >> depend on the number of differences between the ThreadsLists and >> whether the code ever tries to look up those additional threads. If we >> assume racing initialization is likely to be rare anyway (because >> generally one thread is in charge of doing the monitoring) then the >> choice seems somewhat arbitrary. > > I agree, but have a little preference in favor of simplicity. > It was a good discussion though. :) > > Thanks, > Serguei > >> Cheers, >> David >> ----- >> >>> Thanks, >>> Serguei >>> >>>> ??? The >>>> assumption is that it's quite uncommon and even if this is the case >>>> the linear scan happens >>>> only once per such thread. >>>> >>>> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >>>> java_tid) const { >>>> ? 612?? ThreadTable::lazy_initialize(this); >>>> ? 613?? JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); >>>> ? 614?? if (thread == NULL) { >>>> ? 615???? // If the thread is not found in the table find it >>>> ? 616???? // with a linear search and add to the table. >>>> ? 617???? for (uint i = 0; i < length(); i++) { >>>> ? 618?????? thread = thread_at(i); >>>> ? 619?????? oop tobj = thread->threadObj(); >>>> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >>>> ? 621?????? // or is starting to exit. >>>> ? 622?????? if (tobj != NULL && java_tid == >>>> java_lang_Thread::thread_id(tobj)) { >>>> ? 623???????? MutexLocker ml(Threads_lock); >>>> ? 624???????? // Must be inside the lock to ensure that we don't add >>>> the thread to the table >>>> ? 625???????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ? 626???????? if (!thread->is_exiting()) { >>>> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >>>> ? 628?????????? return thread; >>>> ? 629???????? } >>>> ? 630?????? } >>>> ? 631???? } >>>> ? 632?? } else if (!thread->is_exiting()) { >>>> ? 633?????? return thread; >>>> ? 634?? } >>>> ? 635?? return NULL; >>>> ? 636 } >>>> >>>> Thanks, >>>> Daniil >>>> >>>> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >>>> >>>> ???? Hi Daniil, >>>> ???? Thanks again for your perseverance on this one. >>>> ???? I think there is a problem with initialization of the thread >>>> table. >>>> ???? Suppose thread T1 has called >>>> ThreadsList::find_JavaThread_from_java_tid >>>> ???? and has commenced execution of ThreadTable::lazy_initialize, >>>> but not yet >>>> ???? marked _is_initialized as true. Now two new threads (T2 and T3) >>>> are >>>> ???? created and start running - they aren't added to the >>>> ThreadTable yet >>>> ???? because it isn't initialized. Now T0 also calls >>>> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >>>> ThreadsList >>>> ???? that contains T2 and T3. It also calls >>>> ThreadTable::lazy_initialize. If >>>> ???? _is_initialized is still false T0 will attempt initialization >>>> but once >>>> ???? it gets the lock it will see the table has now been initialized >>>> by T1. >>>> ???? It will then proceed to update the table with its own >>>> ThreadList content >>>> ???? - adding T2 and T3. That is all fine. But now suppose T0 >>>> initially sees >>>> ???? _is_initialized as true, it will do nothing in lazy_initialize and >>>> ???? simply return to find_JavaThread_from_java_tid. But now T2 and >>>> T3 are >>>> ???? missing from the ThreadTable and nothing will cause them to be >>>> added. >>>> ???? More generally any ThreadsList that is created after the >>>> ThreadsList >>>> ???? that will be used for initialization, may contain threads that >>>> will not >>>> ???? be added to the table. >>>> ???? Thanks, >>>> ???? David >>>> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >>>> ???? > Hello, >>>> ???? > >>>> ???? > After investigating with Claes the impact of this change on >>>> the performance (thanks a lot Claes for helping with it!) the >>>> conclusion was that the impact on the thread startup time is not a >>>> blocker for this change. >>>> ???? > >>>> ???? > I also measured the memory footprint using Native Memory >>>> Tracking and results showed around 40 bytes per live thread. >>>> ???? > >>>> ???? > Please review a new version of the fix, webrev.06 [1].? Just >>>> to remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 >>>> [3] minus changes in src/hotspot/share/services/management.cpp (that >>>> were factored out to a separate issue [4]) and plus a change in >>>> ThreadsList::find_JavaThread_from_java_tid() method (please, see >>>> below)? that addresses the problem Robbin found and puts the code >>>> that adds a new thread to the thread table inside Threads_lock. >>>> ???? > >>>> ???? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? > >>>> ???? > 622?????? if (tobj != NULL && java_tid == >>>> java_lang_Thread::thread_id(tobj)) { >>>> ???? > 623???????? MutexLocker ml(Threads_lock); >>>> ???? > 624???????? // Must be inside the lock to ensure that we >>>> don't add the thread to the table >>>> ???? > 625???????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ???? > 626???????? if (!thread->is_exiting()) { >>>> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >>>> ???? > 628?????????? return thread; >>>> ???? > 629???????? } >>>> ???? > 630?????? } >>>> ???? > >>>> ???? > [1] Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >>>> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >>>> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >>>> ???? > >>>> ???? > ?Thank you, >>>> ???? > Daniil >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? >????????? > >>>> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >>>> wrote: >>>> ???? >????????? > >>>> ???? >????????? >????? Hi Daniil, >>>> ???? >????????? > >>>> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >>>> ???? >????????? >????? > Hi David, >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > Thank you for your detailed review. Please >>>> review a new version of the fix that includes >>>> ???? >????????? >????? > the changes you suggested: >>>> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced >>>> to cover the creation of the table only; >>>> ???? >????????? >????? > - ThreadTableCreate_lock is made >>>> _safepoint_check_always; >>>> ???? >????????? > >>>> ???? >????????? >????? Okay. >>>> ???? >????????? > >>>> ???? >????????? >????? > - ServiceThread is no longer responsible >>>> for the resizing of the thread table, instead, >>>> ???? >????????? >????? >??? the thread table is changed to grow on >>>> demand by the thread that is doing the addition; >>>> ???? >????????? > >>>> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >>>> of the picture here. >>>> ???? >????????? > >>>> ???? >????????? >????? > - fixed nits and formatting issues. >>>> ???? >????????? > >>>> ???? >????????? >????? Okay. >>>> ???? >????????? > >>>> ???? >????????? >????? >>> The change also includes additional >>>> optimization for some callers of find_JavaThread_from_java_tid() >>>> ???? >????????? >????? >>>?? as Daniel suggested. >>>> ???? >????????? >????? >> Not sure it's best to combine these, but >>>> if they are limited to the >>>> ???? >????????? >????? >> changes in management.cpp only then that >>>> may be okay. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > The additional optimization for some >>>> callers of find_JavaThread_from_java_tid() is >>>> ???? >????????? >????? > limited to management.cpp (plus a new test) >>>> so I left them in the webrev? but >>>> ???? >????????? >????? > I also could move it in the separate issue >>>> if required. >>>> ???? >????????? > >>>> ???? >????????? >????? I'd prefer this part of be separated out, but >>>> won't insist. Let's see if >>>> ???? >????????? >????? Dan or Serguei have a strong opinion. >>>> ???? >????????? > >>>> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? >????????? >????? >??? >755???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >??? > 926???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >?? >? I think it cleaner/better to just use >>>> ???? >????????? >????? >?? > jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >>>> JavaThread and it has to have a >>>> ???? >????????? >????? >?? > non-null threadObj. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > I had to leave this code unchanged since it >>>> turned out the threadObj is null >>>> ???? >????????? >????? > when VM is destroyed: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >>>> oopDesc::long_field(int) const+0x67 >>>> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >>>> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >>>> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >>>> Threads::add(JavaThread*, bool)+0x82 >>>> ???? >????????? >????? > V? [libjvm.so+0xef8369] >>>> attach_current_thread.part.197+0xc9 >>>> ???? >????????? >????? > V? [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c >>>> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >>>> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >>>> ???? >????????? > >>>> ???? >????????? >????? This is actually nothing to do with the VM >>>> being destroyed, but is an >>>> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >>>> interaction with the >>>> ???? >????????? >????? ThreadSMR iterators. The attach process is: >>>> ???? >????????? >????? - create JavaThread >>>> ???? >????????? >????? - mark as "is attaching via jni" >>>> ???? >????????? >????? - add to ThreadsList >>>> ???? >????????? >????? - create java.lang.Thread object (you can >>>> only execute Java code after >>>> ???? >????????? >????? you are attached) >>>> ???? >????????? >????? - mark as "attach completed" >>>> ???? >????????? > >>>> ???? >????????? >????? So while a thread "is attaching" it will be >>>> seen by the ThreadSMR thread >>>> ???? >????????? >????? iterator but will have a NULL >>>> java.lang.Thread object. >>>> ???? >????????? > >>>> ???? >????????? >????? We special-case attaching threads in a number >>>> of places in the VM and I >>>> ???? >????????? >????? think we should be explicitly doing something >>>> here to filter out >>>> ???? >????????? >????? attaching threads, rather than just being >>>> tolerant of a NULL j.l.Thread >>>> ???? >????????? >????? object. Specifically in >>>> ThreadsSMRSupport::add_thread: >>>> ???? >????????? > >>>> ???? >????????? >????? if (ThreadTable::is_initialized() && >>>> !thread->is_attaching_via_jni()) { >>>> ???? >????????? >???????? jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >>>> ???? >????????? >????? } >>>> ???? >????????? > >>>> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >>>> we can use the same guard, >>>> ???? >????????? >????? which covers the case the JNI attach >>>> encountered an error trying to >>>> ???? >????????? >????? create the j.l.Thread object. >>>> ???? >????????? > >>>> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >>>> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >>>> value, bool* is_dead) { >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >> The is_dead parameter still bothers me >>>> here. I can't make enough sense >>>> ???? >????????? >????? >> out of the template code in >>>> ConcurrentHashtable to see why we have to >>>> ???? >????????? >????? >> have it, but I'm concerned that its very >>>> existence means we perhaps >>>> ???? >????????? >????? >> should not be trying to extend CHT in this >>>> context. ?? >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > My understanding is that is_dead parameter >>>> provides a mechanism for >>>> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >>>> that were not explicitly >>>> ???? >????????? >????? > removed by calling >>>> ConcurrentHashTable::remove() method. >>>> ???? >????????? >????? > I think that just because in our case we >>>> don't use this mechanism doesn't >>>> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >>>> ???? >????????? > >>>> ???? >????????? >????? Can you confirm that this usage is okay with >>>> Robbin Ehn please. He's >>>> ???? >????????? >????? back from vacation this week. >>>> ???? >????????? > >>>> ???? >????????? >????? >> I would still want to see what impact this >>>> has on thread >>>> ???? >????????? >????? >> startup cost, both with and without the >>>> table being initialized. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > I run a test that initializes the table by >>>> calling ThreadMXBean.get getThreadInfo(), >>>> ???? >????????? >????? > starts some threads as a worm-up, and then >>>> creates and starts 100,000 threads >>>> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >>>> case when the thread table is enabled >>>> ???? >????????? >????? > 100,000 threads are created and started >>>> for about 15200 ms. If the thread table >>>> ???? >????????? >????? > is off the test takes about 14800 ms. Based >>>> on this information the enabled >>>> ???? >????????? >????? > thread table makes the thread startup about >>>> 2.7% slower. >>>> ???? >????????? > >>>> ???? >????????? >????? That doesn't sound very good. I think we may >>>> need to Claes involved to >>>> ???? >????????? >????? help investigate overall performance impact >>>> here. >>>> ???? >????????? > >>>> ???? >????????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >>>> ???? >????????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? > >>>> ???? >????????? >????? No further code comments. >>>> ???? >????????? > >>>> ???? >????????? >????? I didn't look at the test in detail. >>>> ???? >????????? > >>>> ???? >????????? >????? Thanks, >>>> ???? >????????? >????? David >>>> ???? >????????? > >>>> ???? >????????? >????? > Thanks! >>>> ???? >????????? >????? > --Daniil >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >>>> wrote: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Hi Daniil, >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Overall I think this is a reasonable >>>> approach but I would still like to >>>> ???? >????????? >????? >????? see some performance and footprint >>>> numbers, both to verify it fixes the >>>> ???? >????????? >????? >????? problem reported, and that we are not >>>> getting penalized elsewhere. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov >>>> wrote: >>>> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > Please review the new version of the >>>> fix, that makes the thread table initialization on demand and >>>> ???? >????????? >????? >????? > moves it inside >>>> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >>>> the thread table >>>> ???? >????????? >????? >????? >?? is initialized with the threads >>>> from the current thread list. We don't want to hold Threads_lock >>>> ???? >????????? >????? >????? > inside >>>> find_JavaThread_from_java_tid(),? thus new threads still could be >>>> created? while the thread >>>> ???? >????????? >????? >????? > table is being initialized . Such >>>> threads will be found by the linear search and added to the thread >>>> table >>>> ???? >????????? >????? >????? > later, in >>>> ThreadsList::find_JavaThread_from_java_tid(). >>>> ????? >????????? >????? > >>>> ???? >????????? >????? >????? The initialization allows the created >>>> but unpopulated, or partially >>>> ???? >????????? >????? >????? populated, table to be seen by other >>>> threads - is that your intention? >>>> ???? >????????? >????? >????? It seems it should be okay as the >>>> other threads will then race with the >>>> ???? >????????? >????? >????? initializing thread to add specific >>>> entries, and this is a concurrent >>>> ???? >????????? >????? >????? map so that should be functionally >>>> correct. But if so then I think you >>>> ???? >????????? >????? >????? can also reduce the scope of the >>>> ThreadTableCreate_lock so that it >>>> ???? >????????? >????? >????? covers creation of the table only, not >>>> the initial population of the table. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I like the approach of only >>>> initializing the table when needed and using >>>> ???? >????????? >????? >????? that to control when the >>>> add/remove-thread code needs to update the >>>> ???? >????????? >????? >????? table. But I would still want to see >>>> what impact this has on thread >>>> ???? >????????? >????? >????? startup cost, both with and without >>>> the table being initialized. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > The change also includes additional >>>> optimization for some callers of find_JavaThread_from_java_tid() >>>> ???? >????????? >????? >????? > as Daniel suggested. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Not sure it's best to combine these, >>>> but if they are limited to the >>>> ???? >????????? >????? >????? changes in management.cpp only then >>>> that may be okay. It helps to be >>>> ???? >????????? >????? >????? able to focus on the table related >>>> changes without being distracted by >>>> ???? >????????? >????? >????? other optimizations. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > That is correct that >>>> ResolvedMethodTable was used as a blueprint for the thread table, >>>> however, I tried >>>> ???? >????????? >????? >????? > to strip it of the all functionality >>>> that is not required in the thread table case. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? The revised version seems better in >>>> that regard. But I still have a >>>> ???? >????????? >????? >????? concern, see below. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > We need to have the thread table >>>> resizable and allow it to grow as the number of threads increases to >>>> avoid >>>> ???? >????????? >????? >????? > reserving excessive memory a-priori >>>> or deteriorating lookup times. The ServiceThread is responsible for >>>> ???? >????????? >????? >????? > growing the thread table when required. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Yes but why? Why can't this table be >>>> grown on demand by the thread that >>>> ???? >????????? >????? >????? is doing the addition? For other >>>> tables we may have to delegate to the >>>> ???? >????????? >????? >????? service thread because the current >>>> thread cannot perform the action, or >>>> ???? >????????? >????? >????? it doesn't want to perform it at the >>>> time the need for the resize is >>>> ???? >????????? >????? >????? detected (e.g. its detected at a >>>> safepoint and you want the resize to >>>> ???? >????????? >????? >????? happen later outside the safepoint). >>>> It's not apparent to me that such >>>> ???? >????????? >????? >????? restrictions apply here. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > There is no ConcurrentHashTable >>>> available in Java 8 and for backporting this fix to Java 8 another >>>> implementation >>>> ???? >????????? >????? >????? > of the hash table, probably >>>> originally suggested in the patch attached to the JBS issue, should >>>> be used.? It will make >>>> ???? >????????? >????? >????? > the backporting more complicated, >>>> however, adding a new Implementation of the hash table in Java 14 >>>> while it >>>> ???? >????????? >????? >????? > already has ConcurrentHashTable >>>> doesn't seem? reasonable for me. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Ok. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > Webrev: >>>> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Some specific code comments: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >>>> PaddedMutex? , special, >>>> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I think this needs to be a >>>> _safepoint_check_always lock. The table will >>>> ???? >????????? >????? >????? be created by regular JavaThreads and >>>> they should (nearly) always be >>>> ???? >????????? >????? >????? checking for safepoints if they are >>>> going to block acquiring the lock. >>>> ???? >????????? >????? >????? And it isn't at all obvious that the >>>> thread doing the creation can't go >>>> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >>>> thread_at(i); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? you could reuse the new java_thread >>>> local you introduced at line 613 and >>>> ???? >????????? >????? >????? just rename that "new" variable to >>>> "thread" so you don't have to change >>>> ???? >????????? >????? >????? all other uses. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL >>>> && ... >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? You don't need to check != NULL here >>>> as you only get here when >>>> ???? >????????? >????? >????? java_thread is not NULL. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 755???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >??????? 926???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I think it cleaner/better to just use >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? as we know thread is not NULL, it is a >>>> JavaThread and it has to have a >>>> ???? >????????? >????? >????? non-null threadObj. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/services/management.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 1323???????? if >>>> (THREAD->is_Java_thread()) { >>>> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >>>> (JavaThread*)THREAD; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? These calls can only be made on a >>>> JavaThread so this be simplified to >>>> ???? >????????? >????? >????? remove the is_Java_thread() call. >>>> Similarly in other places. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >>>> CHeapObj { >>>> ???? >????????? >????? >???????? 56?? private: >>>> ???? >????????? >????? >???????? 57???? jlong _tid; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I believe hotspot style is to not >>>> indent the access modifiers in C++ >>>> ???? >????????? >????? >????? class declarations, so the above would >>>> just be: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >>>> CHeapObj { >>>> ???? >????????? >????? >???????? 56 private: >>>> ???? >????????? >????? >???????? 57?? jlong _tid; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? etc. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >>>> JavaThread* java_thread) : >>>> ???? >????????? >????? >??????? 61 >>>> _tid(tid),_java_thread(java_thread) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 61 should be indented as it >>>> continues line 60. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >>>> AllStatic { >>>> ???? >????????? >????? >???????? ... >>>> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >>>> const& value, bool* is_dead) { >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? The is_dead parameter still bothers me >>>> here. I can't make enough sense >>>> ???? >????????? >????? >????? out of the template code in >>>> ConcurrentHashtable to see why we have to >>>> ???? >????????? >????? >????? have it, but I'm concerned that its >>>> very existence means we perhaps >>>> ???? >????????? >????? >????? should not be trying to extend CHT in >>>> this context. ?? >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>> size_log > DefaultThreadTableSizeLog >>>> ???? >????????? >????? >??????? 116?? ? size_log : >>>> DefaultThreadTableSizeLog; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 116 should be indented, though in >>>> this case I think a better layout >>>> ???? >????????? >????? >????? would be: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>> ???? >????????? >????? >??????? 116?????? size_log > >>>> DefaultThreadTableSizeLog ? size_log : >>>> ???? >????????? >????? > DefaultThreadTableSizeLog; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 131 double >>>> ThreadTable::get_load_factor() { >>>> ???? >????????? >????? >??????? 132?? return >>>> (double)_items_count/_current_size; >>>> ???? >????????? >????? >??????? 133 } >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Not sure that is doing what you >>>> want/expect. It will perform integer >>>> ???? >????????? >????? >????? division and then cast that whole >>>> integer to a double. If you want >>>> ???? >????????? >????? >????? double arithmetic you need: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? return >>>> ((double)_items_count)/_current_size; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 180???? jlong _tid; >>>> ???? >????????? >????? >????? 181???? uintx _hash; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: no need for all those spaces >>>> before the variable name. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >>>> ???? >????????? >????? >??????? 184???? : _tid(tid), >>>> _hash(primitive_hash(tid)) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 184 should be indented. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after : >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 211 assert(_is_initialized, "Thread >>>> table is not initialized"); >>>> ???? >????????? >????? >??????? 212?? _has_work = false; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 211 is indented one space too far. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 229 ThreadTableEntry* entry = new >>>> ThreadTableEntry(tid,java_thread); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after , >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 252?? return >>>> _local_table->remove(thread,lookup); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after , >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Thanks, >>>> ???? >????????? >????? >????? David >>>> ???? >????????? >????? >????? ------ >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? > --Daniil >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >>>> Daugherty" wrote: >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >>>> Titov wrote: >>>> ???? >????????? >????? >????? >????? > Hi Serguei and David, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Serguei is right, >>>> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with >>>> an unmatched java_tid. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Please find a new version of >>>> the fix that includes the changes Serguei suggested. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Regarding the concern about >>>> the maintaining the thread table when it may never even be queried, >>>> one of >>>> ???? >????????? >????? >????? >????? > the options could be to add >>>> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >>>> the calls to the thread table >>>> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >>>> add_thread() and remove_thread() methods to check this flag. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > When >>>> ThreadsList::find_JavaThread_from_java_tid() is called for the first >>>> time it could check if ThreadTable ::isEnabled >>>> ???? >????????? >????? >????? >????? > Is on and if not then set it >>>> on and populate the thread table with all existing threads from the >>>> thread list. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? I have the same concerns as >>>> David H. about this new ThreadTable. >>>> ???? >????????? >????? >????? > >>>> ThreadsList::find_JavaThread_from_java_tid() is only called from code >>>> ???? >????????? >????? >????? >????? in >>>> src/hotspot/share/services/management.cpp so I think that table >>>> ???? >????????? >????? >????? >????? needs to enabled and populated >>>> only if it is going to be used. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? I've taken a look at the webrev >>>> below and I see that David has >>>> ???? >????????? >????? >????? >????? followed up with additional >>>> comments. Before I do a crawl through >>>> ???? >????????? >????? >????? >????? code review for this, I would >>>> like to see the ThreadTable stuff >>>> ???? >????????? >????? >????? >????? made optional and David's other >>>> comments addressed. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? Another possible optimization >>>> is for callers of >>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>> save the calling thread's >>>> ???? >????????? >????? >????? >????? tid value before they loop and >>>> if the current tid == saved_tid >>>> ???? >????????? >????? >????? >????? then use the current >>>> JavaThread* instead of calling >>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>> get the JavaThread*. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? Dan >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >>>> ???? >????????? >????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? >????? > --Daniil >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > From: >>>> >>>> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >>>> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 >>>> at 7:56 PM >>>> ???? >????????? >????? >????? >????? > To: Daniil Titov >>>> , OpenJDK Serviceability >>>> , >>>> "hotspot-runtime-dev at openjdk.java.net" >>>> , "jmx-dev at openjdk.java.net" >>>> >>>> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >>>> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >>>> maxDepth) >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > I have several quick comments. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The indent in the hotspot >>>> c/c++ files has to be 2, not 4. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >>>> >>>> ???? >????????? >????? >????? >????? > 614 JavaThread* >>>> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>> java_thread = ThreadTable::find_thread(java_tid); >>>> ???? >????????? >????? >????? >????? > 616???? if (java_thread == >>>> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >>>> ???? >????????? >????? >????? >????? > 617???????? // >>>> ThreadsSMRSupport::add_thread() is not called for the primordial >>>> ???? >????????? >????? >????? >????? > 618???????? // thread. Thus, >>>> we find this thread with a linear search and add it >>>> ???? >????????? >????? >????? >????? > 619???????? // to the thread >>>> table. >>>> ???? >????????? >????? >????? >????? > 620???????? for (uint i = 0; >>>> i < length(); i++) { >>>> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >>>> thread = thread_at(i); >>>> ???? >????????? >????? >????? >????? > 622???????????? if >>>> (is_valid_java_thread(java_tid,thread)) { >>>> ???? >????????? >????? >????? >????? > 623 >>>> ThreadTable::add_thread(java_tid, thread); >>>> ???? >????????? >????? >????? >????? > 624???????????????? return >>>> thread; >>>> ???? >????????? >????? >????? >????? > 625???????????? } >>>> ???? >????????? >????? >????? >????? > 626???????? } >>>> ???? >????????? >????? >????? >????? > 627???? } else if >>>> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>> ???? >????????? >????? >????? >????? > 628???????? return java_thread; >>>> ???? >????????? >????? >????? >????? > 629???? } >>>> ???? >????????? >????? >????? >????? > 630???? return NULL; >>>> ???? >????????? >????? >????? >????? >?? 631 } >>>> ???? >????????? >????? >????? >????? >?? 632 bool >>>> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >>>> java_thread) { >>>> ???? >????????? >????? >????? >????? > 633???? oop tobj = >>>> java_thread->threadObj(); >>>> ???? >????????? >????? >????? >????? > 634???? // Ignore the thread >>>> if it hasn't run yet, has exited >>>> ???? >????????? >????? >????? >????? > 635???? // or is starting to >>>> exit. >>>> ???? >????????? >????? >????? >????? > 636???? return (tobj != NULL >>>> && !java_thread->is_exiting() && >>>> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >>>> java_lang_Thread::thread_id(tobj)); >>>> ???? >????????? >????? >????? >????? >?? 638 } >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>> java_thread = ThreadTable::find_thread(java_tid); >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >>>> find_thread() to find_thread_by_tid(). >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > A space is missed after the >>>> comma: >>>> ???? >????????? >????? >????? >????? >??? 622 if >>>> (is_valid_java_thread(java_tid,thread)) { >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > An empty line is needed >>>> before L632. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The name >>>> 'is_valid_java_thread' looks wrong (or confusing) to me. >>>> ???? >????????? >????? >????? >????? > Something like >>>> 'is_alive_java_thread_with_tid()' would be better. >>>> ???? >????????? >????? >????? >????? > It'd better to list >>>> parameters in the opposite order. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The call to >>>> is_valid_java_thread() is confusing: >>>> ???? >????????? >????? >????? >????? >???? 627 } else if >>>> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Why would the call >>>> ThreadTable::find_thread(java_tid) return a JavaThread with an >>>> unmatched java_tid? >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks, >>>> ???? >????????? >????? >????? >????? > Serguei >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >>>> Holmes" wrote: >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >????? Hi Daniil, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >????? The definition and use >>>> of this hashtable (yet another hashtable >>>> ???? >????????? >????? >????? >????? > implementation!) will need >>>> careful examination. We have to be concerned >>>> ???? >????????? >????? >????? >????? > about the cost of maintaining >>>> it when it may never even be queried. You >>>> ???? >????????? >????? >????? >????? > would need to look at >>>> footprint cost and performance impact. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Unfortunately I'm just about >>>> to board a plane and will be out for the >>>> ???? >????????? >????? >????? >????? > next few days. I will try to >>>> look at this asap next week, but we will >>>> ???? >????????? >????? >????? >????? > need a lot more data on it. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks, >>>> ???? >????????? >????? >????? >????? > David >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >>>> Titov wrote: >>>> ???? >????????? >????? >????? >????? > Please review the change that >>>> improves performance of ThreadMXBean MXBean methods returning the >>>> ???? >????????? >????? >????? >????? > information for specific >>>> threads. The change introduces the thread table that uses >>>> ConcurrentHashTable >>>> ???? >????????? >????? >????? >????? > to store one-to-one the >>>> mapping between the thread ids and JavaThread objects and replaces >>>> the linear >>>> ???? >????????? >????? >????? >????? > search over the thread list >>>> in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with >>>> the lookup >>>> ???? >????????? >????? >????? >????? > in the thread table. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 >>>> and tier3 tests successfully passed. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >>>> ???? >????????? >????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Best regards, >>>> ???? >????????? >????? >????? >????? > Daniil >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? > >>>> ???? >????????? > >>>> ???? >????????? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> >>>> >>> > From serguei.spitsyn at oracle.com Wed Sep 18 07:34:55 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 00:34:55 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> Message-ID: Hi David, On 9/18/19 00:25, David Holmes wrote: > Hi Serguei, > > In the interests of full disclosure I was the one who told Daniil to > not hold the lock while populating the table. I well understand your point to not hold the lock while populating the table. > > I'll leave you two to work out which way you want to go there. In fact, I have no strong opinion here, just my gut feelings. :) And I'm open for any solution. Just wanted to share my thoughts with you, guys, to have all reasoning on the table. Let me briefly talk to Daniil tomorrow. Thanks, Serguei > Thanks, > David > > On 18/09/2019 5:13 pm, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> >> On 9/17/19 03:46, David Holmes wrote: >>> Hi Serguei, >>> >>> On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: >>>> Hi Daniil, >>>> >>>> >>>> On 9/16/19 21:36, Daniil Titov wrote: >>>>> Hi David, >>>>> >>>>> The case you have described is exact the reason why we still have >>>>> a code inside >>>>> ThreadsList::find_JavaThread_from_java_tid() method that does a >>>>> linear scan and adds >>>>> ? the requested thread to the thread table if it is not there ( >>>>> lines 614-613 below). >>>> >>>> I disagree because it is easy to avoid concurrent ThreadTable >>>> initialization (please, see my separate email). >>>> The reason for this code is to cover a case of late/lazy >>>> ThreadTable initialization. >>> >>> I'm not sure I follow. With the current code if two threads are >>> racing to initialize the ThreadTable with ThreadsLists that contain >>> a different set of threads then there are two possibilities with >>> regards to the interleaving. Assume T1 initializes the table with >>> its set of threads and so finds the tid it is looking for in the >>> table. Meanwhile T2 is racing with the initialization logic: >>> >>> - If T2 sees _is_initialized then lazy_initialization does nothing >>> for T2, and the additional threads in its ThreadsList (say T3 and >>> T4) are not added to the table. But the specific thread associated >>> with the tid (say T3) will be found by linear search of the >>> ThreadsList and then added. If any other threads come searching for >>> T4 they too will not find it in the ThreadTable but instead perform >>> the linear search of their ThreadsList (and add it). >>> >>> - if T2 doesn't see _is_initialized at first it will try to acquire >>> the lock, and eventually see _is_initialized is true, at which point >>> it will try to add all of its thread's to the table (so T3 and T4 >>> will be added). When lazy_initialize returns, T3 will be found in >>> the table and returned. If any other threads come searching for T4 >>> they will also find it in the table. >> >> My main concerns are simplicity and reliability. >> I do no care much about extra overhead at the ThreadTable >> initialization. >> A probability of the ThreadTable::lazy_initialize() being called >> concurrently is low. >> Also, it might happen only once for the whole VM process execution. >> >> I was wrong by thinking that adding new threads to the ThreadTable >> after its initialization >> will result in thread linear search as well. So my conclusion was >> that we should not care >> if it happens once more at lazy initialization point. >> But now I see that after ThreadTable::is_initialized() returns true >> the ThreadsSMRSupport::add_thread() >> makes a call to the ThreadTable::add_thread(). So, no more linear >> search will happen. >> >> >> However, it seems to me, a possible concurrent lazy initialization in >> the webrev.06 introduces >> its own extra overhead - competing threads (suppose, we have just two >> of them) will do >> the same amount of work concurrently: >> ??? - all indirect memory readings, lookup/comparisons and other >> checks will be performed twice >> ??? - new ThreadTableEntry can be allocated twice for some threads in >> the list >> ????? (depending on how ConcurrentHashTable is implemented, there can >> be a potential a memory leak) >> >> So, I doubt we win much in performance here but can loose in >> reliability. >> >> >> I'd suggest to simplify the lazy initialization code and make it more >> reliable this way: >> >> ????? if (!_is_initialized) { >> ??????? MutexLocker ml(ThreadTableCreate_lock); >> ??????? if (!_is_initialized) { >> ????????? create_table(threads->length()); >> ????????? _is_initialized = true; >> ??????? } >> ??????? for (uint i = 0; i < threads->length(); i++) { >> ????????? JavaThread* thread = threads->thread_at(i); >> ????????? oop tobj = thread->threadObj(); >> ????????? if (tobj != NULL && !thread->is_exiting()) { >> ??????????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ??????????? add_thread(java_tid, thread); >> ????????? } >> ??????? } >> ????? } >> >>> With your suggested code change this second case is not possible so >>> for any racing initialization the lookup of any threads not in the >>> original ThreadsList will always result in using the linear search >>> before adding to the table. >> >> Yes, but I did not care about this. >> The overhead is expected to be lower than the lazy initialization cost, >> especially because the probability of concurrent initialization is low. >> >>> Both seem correct to me. Which one is more efficient will totally >>> depend on the number of differences between the ThreadsLists and >>> whether the code ever tries to look up those additional threads. If >>> we assume racing initialization is likely to be rare anyway (because >>> generally one thread is in charge of doing the monitoring) then the >>> choice seems somewhat arbitrary. >> >> I agree, but have a little preference in favor of simplicity. >> It was a good discussion though. :) >> >> Thanks, >> Serguei >> >>> Cheers, >>> David >>> ----- >>> >>>> Thanks, >>>> Serguei >>>> >>>>> ??? The >>>>> assumption is that it's quite uncommon and even if this is the >>>>> case the linear scan happens >>>>> only once per such thread. >>>>> >>>>> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >>>>> java_tid) const { >>>>> ? 612?? ThreadTable::lazy_initialize(this); >>>>> ? 613?? JavaThread* thread = >>>>> ThreadTable::find_thread_by_tid(java_tid); >>>>> ? 614?? if (thread == NULL) { >>>>> ? 615???? // If the thread is not found in the table find it >>>>> ? 616???? // with a linear search and add to the table. >>>>> ? 617???? for (uint i = 0; i < length(); i++) { >>>>> ? 618?????? thread = thread_at(i); >>>>> ? 619?????? oop tobj = thread->threadObj(); >>>>> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >>>>> ? 621?????? // or is starting to exit. >>>>> ? 622?????? if (tobj != NULL && java_tid == >>>>> java_lang_Thread::thread_id(tobj)) { >>>>> ? 623???????? MutexLocker ml(Threads_lock); >>>>> ? 624???????? // Must be inside the lock to ensure that we don't >>>>> add the thread to the table >>>>> ? 625???????? // that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> ? 626???????? if (!thread->is_exiting()) { >>>>> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >>>>> ? 628?????????? return thread; >>>>> ? 629???????? } >>>>> ? 630?????? } >>>>> ? 631???? } >>>>> ? 632?? } else if (!thread->is_exiting()) { >>>>> ? 633?????? return thread; >>>>> ? 634?? } >>>>> ? 635?? return NULL; >>>>> ? 636 } >>>>> >>>>> Thanks, >>>>> Daniil >>>>> >>>>> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >>>>> >>>>> ???? Hi Daniil, >>>>> ???? Thanks again for your perseverance on this one. >>>>> ???? I think there is a problem with initialization of the thread >>>>> table. >>>>> ???? Suppose thread T1 has called >>>>> ThreadsList::find_JavaThread_from_java_tid >>>>> ???? and has commenced execution of ThreadTable::lazy_initialize, >>>>> but not yet >>>>> ???? marked _is_initialized as true. Now two new threads (T2 and >>>>> T3) are >>>>> ???? created and start running - they aren't added to the >>>>> ThreadTable yet >>>>> ???? because it isn't initialized. Now T0 also calls >>>>> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >>>>> ThreadsList >>>>> ???? that contains T2 and T3. It also calls >>>>> ThreadTable::lazy_initialize. If >>>>> ???? _is_initialized is still false T0 will attempt initialization >>>>> but once >>>>> ???? it gets the lock it will see the table has now been >>>>> initialized by T1. >>>>> ???? It will then proceed to update the table with its own >>>>> ThreadList content >>>>> ???? - adding T2 and T3. That is all fine. But now suppose T0 >>>>> initially sees >>>>> ???? _is_initialized as true, it will do nothing in >>>>> lazy_initialize and >>>>> ???? simply return to find_JavaThread_from_java_tid. But now T2 >>>>> and T3 are >>>>> ???? missing from the ThreadTable and nothing will cause them to >>>>> be added. >>>>> ???? More generally any ThreadsList that is created after the >>>>> ThreadsList >>>>> ???? that will be used for initialization, may contain threads >>>>> that will not >>>>> ???? be added to the table. >>>>> ???? Thanks, >>>>> ???? David >>>>> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >>>>> ???? > Hello, >>>>> ???? > >>>>> ???? > After investigating with Claes the impact of this change on >>>>> the performance (thanks a lot Claes for helping with it!) the >>>>> conclusion was that the impact on the thread startup time is not a >>>>> blocker for this change. >>>>> ???? > >>>>> ???? > I also measured the memory footprint using Native Memory >>>>> Tracking and results showed around 40 bytes per live thread. >>>>> ???? > >>>>> ???? > Please review a new version of the fix, webrev.06 [1].? >>>>> Just to remind,? webrev.05 was abandoned and webrev.06 [1] is >>>>> webrev.04 [3] minus changes in >>>>> src/hotspot/share/services/management.cpp (that were factored out >>>>> to a separate issue [4]) and plus a change in >>>>> ThreadsList::find_JavaThread_from_java_tid() method (please, see >>>>> below)? that addresses the problem Robbin found and puts the code >>>>> that adds a new thread to the thread table inside Threads_lock. >>>>> ???? > >>>>> ???? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? > >>>>> ???? > 622?????? if (tobj != NULL && java_tid == >>>>> java_lang_Thread::thread_id(tobj)) { >>>>> ???? > 623???????? MutexLocker ml(Threads_lock); >>>>> ???? > 624???????? // Must be inside the lock to ensure that we >>>>> don't add the thread to the table >>>>> ???? > 625???????? // that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> ???? > 626???????? if (!thread->is_exiting()) { >>>>> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >>>>> ???? > 628?????????? return thread; >>>>> ???? > 629???????? } >>>>> ???? > 630?????? } >>>>> ???? > >>>>> ???? > [1] Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >>>>> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >>>>> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >>>>> ???? > >>>>> ???? > ?Thank you, >>>>> ???? > Daniil >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? >????????? > >>>>> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >>>>> wrote: >>>>> ???? >????????? > >>>>> ???? >????????? >????? Hi Daniil, >>>>> ???? >????????? > >>>>> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >>>>> ???? >????????? >????? > Hi David, >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > Thank you for your detailed review. >>>>> Please review a new version of the fix that includes >>>>> ???? >????????? >????? > the changes you suggested: >>>>> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced >>>>> to cover the creation of the table only; >>>>> ???? >????????? >????? > - ThreadTableCreate_lock is made >>>>> _safepoint_check_always; >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > - ServiceThread is no longer responsible >>>>> for the resizing of the thread table, instead, >>>>> ???? >????????? >????? >??? the thread table is changed to grow on >>>>> demand by the thread that is doing the addition; >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay - I'm happy to get the serviceThread >>>>> out of the picture here. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > - fixed nits and formatting issues. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >>> The change also includes additional >>>>> optimization for some callers of find_JavaThread_from_java_tid() >>>>> ???? >????????? >????? >>>?? as Daniel suggested. >>>>> ???? >????????? >????? >> Not sure it's best to combine these, but >>>>> if they are limited to the >>>>> ???? >????????? >????? >> changes in management.cpp only then that >>>>> may be okay. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > The additional optimization for some >>>>> callers of find_JavaThread_from_java_tid() is >>>>> ???? >????????? >????? > limited to management.cpp (plus a new >>>>> test) so I left them in the webrev? but >>>>> ???? >????????? >????? > I also could move it in the separate >>>>> issue if required. >>>>> ???? >????????? > >>>>> ???? >????????? >????? I'd prefer this part of be separated out, >>>>> but won't insist. Let's see if >>>>> ???? >????????? >????? Dan or Serguei have a strong opinion. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? >????????? >????? >??? >755???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >??? > 926???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >?? >? I think it cleaner/better to just use >>>>> ???? >????????? >????? >?? > jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? >????? >?? > as we know thread is not NULL, it is >>>>> a JavaThread and it has to have a >>>>> ???? >????????? >????? >?? > non-null threadObj. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > I had to leave this code unchanged since >>>>> it turned out the threadObj is null >>>>> ???? >????????? >????? > when VM is destroyed: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >>>>> oopDesc::long_field(int) const+0x67 >>>>> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >>>>> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >>>>> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >>>>> Threads::add(JavaThread*, bool)+0x82 >>>>> ???? >????????? >????? > V? [libjvm.so+0xef8369] >>>>> attach_current_thread.part.197+0xc9 >>>>> ???? >????????? >????? > V? [libjvm.so+0xec136c] >>>>> jni_DestroyJavaVM+0x6c >>>>> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >>>>> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >>>>> ???? >????????? > >>>>> ???? >????????? >????? This is actually nothing to do with the VM >>>>> being destroyed, but is an >>>>> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >>>>> interaction with the >>>>> ???? >????????? >????? ThreadSMR iterators. The attach process is: >>>>> ???? >????????? >????? - create JavaThread >>>>> ???? >????????? >????? - mark as "is attaching via jni" >>>>> ???? >????????? >????? - add to ThreadsList >>>>> ???? >????????? >????? - create java.lang.Thread object (you can >>>>> only execute Java code after >>>>> ???? >????????? >????? you are attached) >>>>> ???? >????????? >????? - mark as "attach completed" >>>>> ???? >????????? > >>>>> ???? >????????? >????? So while a thread "is attaching" it will be >>>>> seen by the ThreadSMR thread >>>>> ???? >????????? >????? iterator but will have a NULL >>>>> java.lang.Thread object. >>>>> ???? >????????? > >>>>> ???? >????????? >????? We special-case attaching threads in a >>>>> number of places in the VM and I >>>>> ???? >????????? >????? think we should be explicitly doing >>>>> something here to filter out >>>>> ???? >????????? >????? attaching threads, rather than just being >>>>> tolerant of a NULL j.l.Thread >>>>> ???? >????????? >????? object. Specifically in >>>>> ThreadsSMRSupport::add_thread: >>>>> ???? >????????? > >>>>> ???? >????????? >????? if (ThreadTable::is_initialized() && >>>>> !thread->is_attaching_via_jni()) { >>>>> ???? >????????? >???????? jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? > ThreadTable::add_thread(tid, thread); >>>>> ???? >????????? >????? } >>>>> ???? >????????? > >>>>> ???? >????????? >????? Note that in >>>>> ThreadsSMRSupport::remove_thread we can use the same guard, >>>>> ???? >????????? >????? which covers the case the JNI attach >>>>> encountered an error trying to >>>>> ???? >????????? >????? create the j.l.Thread object. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >>>>> ???? >????????? >????? >> 71???? static uintx get_hash(Value >>>>> const& value, bool* is_dead) { >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >> The is_dead parameter still bothers me >>>>> here. I can't make enough sense >>>>> ???? >????????? >????? >> out of the template code in >>>>> ConcurrentHashtable to see why we have to >>>>> ???? >????????? >????? >> have it, but I'm concerned that its very >>>>> existence means we perhaps >>>>> ???? >????????? >????? >> should not be trying to extend CHT in >>>>> this context. ?? >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > My understanding is that is_dead >>>>> parameter provides a mechanism for >>>>> ???? >????????? >????? > ConcurrentHashtable to remove stale >>>>> entries that were not explicitly >>>>> ???? >????????? >????? > removed by calling >>>>> ConcurrentHashTable::remove() method. >>>>> ???? >????????? >????? > I think that just because in our case we >>>>> don't use this mechanism doesn't >>>>> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Can you confirm that this usage is okay >>>>> with Robbin Ehn please. He's >>>>> ???? >????????? >????? back from vacation this week. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >> I would still want to see what impact >>>>> this has on thread >>>>> ???? >????????? >????? >> startup cost, both with and without the >>>>> table being initialized. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > I run a test that initializes the table >>>>> by calling ThreadMXBean.get getThreadInfo(), >>>>> ???? >????????? >????? > starts some threads as a worm-up, and >>>>> then creates and starts 100,000 threads >>>>> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >>>>> case when the thread table is enabled >>>>> ???? >????????? >????? > 100,000 threads are created and started? >>>>> for about 15200 ms. If the thread table >>>>> ???? >????????? >????? > is off the test takes about 14800 ms. >>>>> Based on this information the enabled >>>>> ???? >????????? >????? > thread table makes the thread startup >>>>> about 2.7% slower. >>>>> ???? >????????? > >>>>> ???? >????????? >????? That doesn't sound very good. I think we >>>>> may need to Claes involved to >>>>> ???? >????????? >????? help investigate overall performance impact >>>>> here. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >>>>> ???? >????????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? > >>>>> ???? >????????? >????? No further code comments. >>>>> ???? >????????? > >>>>> ???? >????????? >????? I didn't look at the test in detail. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Thanks, >>>>> ???? >????????? >????? David >>>>> ???? >????????? > >>>>> ???? >????????? >????? > Thanks! >>>>> ???? >????????? >????? > --Daniil >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >>>>> wrote: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Hi Daniil, >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Overall I think this is a reasonable >>>>> approach but I would still like to >>>>> ???? >????????? >????? >????? see some performance and footprint >>>>> numbers, both to verify it fixes the >>>>> ???? >????????? >????? >????? problem reported, and that we are >>>>> not getting penalized elsewhere. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov >>>>> wrote: >>>>> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > Please review the new version of >>>>> the fix, that makes the thread table initialization on demand and >>>>> ???? >????????? >????? >????? > moves it inside >>>>> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >>>>> the thread table >>>>> ???? >????????? >????? >????? >?? is initialized with the threads >>>>> from the current thread list. We don't want to hold Threads_lock >>>>> ???? >????????? >????? >????? > inside >>>>> find_JavaThread_from_java_tid(),? thus new threads still could be >>>>> created? while the thread >>>>> ???? >????????? >????? >????? > table is being initialized . Such >>>>> threads will be found by the linear search and added to the thread >>>>> table >>>>> ???? >????????? >????? >????? > later, in >>>>> ThreadsList::find_JavaThread_from_java_tid(). >>>>> ????? >????????? >????? > >>>>> ???? >????????? >????? >????? The initialization allows the >>>>> created but unpopulated, or partially >>>>> ???? >????????? >????? >????? populated, table to be seen by other >>>>> threads - is that your intention? >>>>> ???? >????????? >????? >????? It seems it should be okay as the >>>>> other threads will then race with the >>>>> ???? >????????? >????? >????? initializing thread to add specific >>>>> entries, and this is a concurrent >>>>> ???? >????????? >????? >????? map so that should be functionally >>>>> correct. But if so then I think you >>>>> ???? >????????? >????? >????? can also reduce the scope of the >>>>> ThreadTableCreate_lock so that it >>>>> ???? >????????? >????? >????? covers creation of the table only, >>>>> not the initial population of the table. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I like the approach of only >>>>> initializing the table when needed and using >>>>> ???? >????????? >????? >????? that to control when the >>>>> add/remove-thread code needs to update the >>>>> ???? >????????? >????? >????? table. But I would still want to see >>>>> what impact this has on thread >>>>> ???? >????????? >????? >????? startup cost, both with and without >>>>> the table being initialized. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > The change also includes >>>>> additional optimization for some callers of >>>>> find_JavaThread_from_java_tid() >>>>> ???? >????????? >????? >????? > as Daniel suggested. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Not sure it's best to combine these, >>>>> but if they are limited to the >>>>> ???? >????????? >????? >????? changes in management.cpp only then >>>>> that may be okay. It helps to be >>>>> ???? >????????? >????? >????? able to focus on the table related >>>>> changes without being distracted by >>>>> ???? >????????? >????? >????? other optimizations. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > That is correct that >>>>> ResolvedMethodTable was used as a blueprint for the thread table, >>>>> however, I tried >>>>> ???? >????????? >????? >????? > to strip it of the all >>>>> functionality that is not required in the thread table case. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? The revised version seems better in >>>>> that regard. But I still have a >>>>> ???? >????????? >????? >????? concern, see below. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > We need to have the thread table >>>>> resizable and allow it to grow as the number of threads increases >>>>> to avoid >>>>> ???? >????????? >????? >????? > reserving excessive memory >>>>> a-priori or deteriorating lookup times. The ServiceThread is >>>>> responsible for >>>>> ???? >????????? >????? >????? > growing the thread table when >>>>> required. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Yes but why? Why can't this table be >>>>> grown on demand by the thread that >>>>> ???? >????????? >????? >????? is doing the addition? For other >>>>> tables we may have to delegate to the >>>>> ???? >????????? >????? >????? service thread because the current >>>>> thread cannot perform the action, or >>>>> ???? >????????? >????? >????? it doesn't want to perform it at the >>>>> time the need for the resize is >>>>> ???? >????????? >????? >????? detected (e.g. its detected at a >>>>> safepoint and you want the resize to >>>>> ???? >????????? >????? >????? happen later outside the safepoint). >>>>> It's not apparent to me that such >>>>> ???? >????????? >????? >????? restrictions apply here. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > There is no ConcurrentHashTable >>>>> available in Java 8 and for backporting this fix to Java 8 another >>>>> implementation >>>>> ???? >????????? >????? >????? > of the hash table, probably >>>>> originally suggested in the patch attached to the JBS issue, >>>>> should be used.? It will make >>>>> ???? >????????? >????? >????? > the backporting more complicated,? >>>>> however, adding a new Implementation of the hash table in Java 14 >>>>> while it >>>>> ???? >????????? >????? >????? > already has ConcurrentHashTable >>>>> doesn't seem? reasonable for me. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Ok. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > Webrev: >>>>> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Some specific code comments: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >>>>> PaddedMutex? , special, >>>>> ???? >????????? >????? >????? false, >>>>> Monitor::_safepoint_check_never); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I think this needs to be a >>>>> _safepoint_check_always lock. The table will >>>>> ???? >????????? >????? >????? be created by regular JavaThreads >>>>> and they should (nearly) always be >>>>> ???? >????????? >????? >????? checking for safepoints if they are >>>>> going to block acquiring the lock. >>>>> ???? >????????? >????? >????? And it isn't at all obvious that the >>>>> thread doing the creation can't go >>>>> ???? >????????? >????? >????? to a safepoint whilst this lock is >>>>> held. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 618 JavaThread* thread = >>>>> thread_at(i); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? you could reuse the new java_thread >>>>> local you introduced at line 613 and >>>>> ???? >????????? >????? >????? just rename that "new" variable to >>>>> "thread" so you don't have to change >>>>> ???? >????????? >????? >????? all other uses. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL >>>>> && ... >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? You don't need to check != NULL here >>>>> as you only get here when >>>>> ???? >????????? >????? >????? java_thread is not NULL. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 755???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >??????? 926???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I think it cleaner/better to just use >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? as we know thread is not NULL, it is >>>>> a JavaThread and it has to have a >>>>> ???? >????????? >????? >????? non-null threadObj. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/services/management.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 1323???????? if >>>>> (THREAD->is_Java_thread()) { >>>>> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >>>>> (JavaThread*)THREAD; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? These calls can only be made on a >>>>> JavaThread so this be simplified to >>>>> ???? >????????? >????? >????? remove the is_Java_thread() call. >>>>> Similarly in other places. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : >>>>> public CHeapObj { >>>>> ???? >????????? >????? >???????? 56?? private: >>>>> ???? >????????? >????? >???????? 57???? jlong _tid; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I believe hotspot style is to not >>>>> indent the access modifiers in C++ >>>>> ???? >????????? >????? >????? class declarations, so the above >>>>> would just be: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : >>>>> public CHeapObj { >>>>> ???? >????????? >????? >???????? 56 private: >>>>> ???? >????????? >????? >???????? 57?? jlong _tid; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? etc. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >>>>> JavaThread* java_thread) : >>>>> ???? >????????? >????? >??????? 61 >>>>> _tid(tid),_java_thread(java_thread) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 61 should be indented as it >>>>> continues line 60. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 67 class ThreadTableConfig : >>>>> public AllStatic { >>>>> ???? >????????? >????? >???????? ... >>>>> ???? >????????? >????? >???????? 71???? static uintx >>>>> get_hash(Value const& value, bool* is_dead) { >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? The is_dead parameter still bothers >>>>> me here. I can't make enough sense >>>>> ???? >????????? >????? >????? out of the template code in >>>>> ConcurrentHashtable to see why we have to >>>>> ???? >????????? >????? >????? have it, but I'm concerned that its >>>>> very existence means we perhaps >>>>> ???? >????????? >????? >????? should not be trying to extend CHT >>>>> in this context. ?? >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>>> size_log > DefaultThreadTableSizeLog >>>>> ???? >????????? >????? >??????? 116?? ? size_log : >>>>> DefaultThreadTableSizeLog; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 116 should be indented, though >>>>> in this case I think a better layout >>>>> ???? >????????? >????? >????? would be: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>>> ???? >????????? >????? >??????? 116 size_log > >>>>> DefaultThreadTableSizeLog ? size_log : >>>>> ???? >????????? >????? > DefaultThreadTableSizeLog; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 131 double >>>>> ThreadTable::get_load_factor() { >>>>> ???? >????????? >????? >??????? 132?? return >>>>> (double)_items_count/_current_size; >>>>> ???? >????????? >????? >??????? 133 } >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Not sure that is doing what you >>>>> want/expect. It will perform integer >>>>> ???? >????????? >????? >????? division and then cast that whole >>>>> integer to a double. If you want >>>>> ???? >????????? >????? >????? double arithmetic you need: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? return >>>>> ((double)_items_count)/_current_size; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 180???? jlong _tid; >>>>> ???? >????????? >????? >????? 181???? uintx _hash; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: no need for all those spaces >>>>> before the variable name. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >>>>> ???? >????????? >????? >??????? 184???? : _tid(tid), >>>>> _hash(primitive_hash(tid)) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 184 should be indented. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after : >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 211 assert(_is_initialized, >>>>> "Thread table is not initialized"); >>>>> ???? >????????? >????? >??????? 212?? _has_work = false; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 211 is indented one space too far. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 229 ThreadTableEntry* entry = new >>>>> ThreadTableEntry(tid,java_thread); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after , >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 252?? return >>>>> _local_table->remove(thread,lookup); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after , >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Thanks, >>>>> ???? >????????? >????? >????? David >>>>> ???? >????????? >????? >????? ------ >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? > --Daniil >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >>>>> Daugherty" wrote: >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >>>>> Titov wrote: >>>>> ???? >????????? >????? >????? >????? > Hi Serguei and David, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Serguei is right, >>>>> ThreadTable::find_thread(java_tid) cannot? return a JavaThread >>>>> with an unmatched java_tid. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Please find a new version >>>>> of the fix that includes the changes Serguei suggested. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Regarding the concern about >>>>> the maintaining the thread table when it may never even be >>>>> queried, one of >>>>> ???? >????????? >????? >????? >????? > the options could be to add >>>>> ThreadTable ::isEnabled flag, set it to "false" by default, and >>>>> wrap the calls to the thread table >>>>> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >>>>> add_thread() and remove_thread() methods to check this flag. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > When >>>>> ThreadsList::find_JavaThread_from_java_tid() is called for the >>>>> first time it could check if ThreadTable ::isEnabled >>>>> ???? >????????? >????? >????? >????? > Is on and if not then set >>>>> it on and populate the thread table with all existing threads from >>>>> the thread list. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? I have the same concerns as >>>>> David H. about this new ThreadTable. >>>>> ???? >????????? >????? >????? > >>>>> ThreadsList::find_JavaThread_from_java_tid() is only called from code >>>>> ???? >????????? >????? >????? >????? in >>>>> src/hotspot/share/services/management.cpp so I think that table >>>>> ???? >????????? >????? >????? >????? needs to enabled and >>>>> populated only if it is going to be used. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? I've taken a look at the >>>>> webrev below and I see that David has >>>>> ???? >????????? >????? >????? >????? followed up with additional >>>>> comments. Before I do a crawl through >>>>> ???? >????????? >????? >????? >????? code review for this, I would >>>>> like to see the ThreadTable stuff >>>>> ???? >????????? >????? >????? >????? made optional and David's >>>>> other comments addressed. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? Another possible optimization >>>>> is for callers of >>>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>>> save the calling thread's >>>>> ???? >????????? >????? >????? >????? tid value before they loop >>>>> and if the current tid == saved_tid >>>>> ???? >????????? >????? >????? >????? then use the current >>>>> JavaThread* instead of calling >>>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>>> get the JavaThread*. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? Dan >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >>>>> ???? >????????? >????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? >????? > --Daniil >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > From: >>>>> >>>>> ???? >????????? >????? >????? >????? > Organization: Oracle >>>>> Corporation >>>>> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 >>>>> at 7:56 PM >>>>> ???? >????????? >????? >????? >????? > To: Daniil Titov >>>>> , OpenJDK Serviceability >>>>> , >>>>> "hotspot-runtime-dev at openjdk.java.net" >>>>> , "jmx-dev at openjdk.java.net" >>>>> >>>>> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >>>>> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >>>>> maxDepth) >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > I have several quick comments. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The indent in the hotspot >>>>> c/c++ files has to be 2, not 4. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >>>>> >>>>> ???? >????????? >????? >????? >????? > 614 JavaThread* >>>>> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >>>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>>> java_thread = ThreadTable::find_thread(java_tid); >>>>> ???? >????????? >????? >????? >????? > 616???? if (java_thread == >>>>> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >>>>> ???? >????????? >????? >????? >????? > 617???????? // >>>>> ThreadsSMRSupport::add_thread() is not called for the primordial >>>>> ???? >????????? >????? >????? >????? > 618???????? // thread. >>>>> Thus, we find this thread with a linear search and add it >>>>> ???? >????????? >????? >????? >????? > 619???????? // to the >>>>> thread table. >>>>> ???? >????????? >????? >????? >????? > 620???????? for (uint i = >>>>> 0; i < length(); i++) { >>>>> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >>>>> thread = thread_at(i); >>>>> ???? >????????? >????? >????? >????? > 622???????????? if >>>>> (is_valid_java_thread(java_tid,thread)) { >>>>> ???? >????????? >????? >????? >????? > 623???????????????? >>>>> ThreadTable::add_thread(java_tid, thread); >>>>> ???? >????????? >????? >????? >????? > 624???????????????? return >>>>> thread; >>>>> ???? >????????? >????? >????? >????? > 625???????????? } >>>>> ???? >????????? >????? >????? >????? > 626???????? } >>>>> ???? >????????? >????? >????? >????? > 627???? } else if >>>>> (java_thread != NULL && is_valid_java_thread(java_tid, >>>>> java_thread)) { >>>>> ???? >????????? >????? >????? >????? > 628???????? return >>>>> java_thread; >>>>> ???? >????????? >????? >????? >????? > 629???? } >>>>> ???? >????????? >????? >????? >????? > 630???? return NULL; >>>>> ???? >????????? >????? >????? >????? > 631 } >>>>> ???? >????????? >????? >????? >????? > 632 bool >>>>> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >>>>> java_thread) { >>>>> ???? >????????? >????? >????? >????? > 633???? oop tobj = >>>>> java_thread->threadObj(); >>>>> ???? >????????? >????? >????? >????? > 634???? // Ignore the >>>>> thread if it hasn't run yet, has exited >>>>> ???? >????????? >????? >????? >????? > 635???? // or is starting >>>>> to exit. >>>>> ???? >????????? >????? >????? >????? > 636???? return (tobj != >>>>> NULL && !java_thread->is_exiting() && >>>>> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >>>>> java_lang_Thread::thread_id(tobj)); >>>>> ???? >????????? >????? >????? >????? > 638 } >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>>> java_thread = ThreadTable::find_thread(java_tid); >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > I'd suggest to rename >>>>> find_thread() to find_thread_by_tid(). >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > A space is missed after the >>>>> comma: >>>>> ???? >????????? >????? >????? >????? > 622 if >>>>> (is_valid_java_thread(java_tid,thread)) { >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > An empty line is needed >>>>> before L632. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The name >>>>> 'is_valid_java_thread' looks wrong (or confusing) to me. >>>>> ???? >????????? >????? >????? >????? > Something like >>>>> 'is_alive_java_thread_with_tid()' would be better. >>>>> ???? >????????? >????? >????? >????? > It'd better to list >>>>> parameters in the opposite order. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The call to >>>>> is_valid_java_thread() is confusing: >>>>> ???? >????????? >????? >????? >????? > 627 } else if (java_thread >>>>> != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Why would the call >>>>> ThreadTable::find_thread(java_tid) return a JavaThread with an >>>>> unmatched java_tid? >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks, >>>>> ???? >????????? >????? >????? >????? > Serguei >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >>>>> Holmes" wrote: >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The definition and use of >>>>> this hashtable (yet another hashtable >>>>> ???? >????????? >????? >????? >????? > implementation!) will need >>>>> careful examination. We have to be concerned >>>>> ???? >????????? >????? >????? >????? > about the cost of >>>>> maintaining it when it may never even be queried. You >>>>> ???? >????????? >????? >????? >????? > would need to look at >>>>> footprint cost and performance impact. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Unfortunately I'm just >>>>> about to board a plane and will be out for the >>>>> ???? >????????? >????? >????? >????? > next few days. I will try >>>>> to look at this asap next week, but we will >>>>> ???? >????????? >????? >????? >????? > need a lot more data on it. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks, >>>>> ???? >????????? >????? >????? >????? > David >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >>>>> Titov wrote: >>>>> ???? >????????? >????? >????? >????? > Please review the change >>>>> that improves performance of ThreadMXBean MXBean methods returning >>>>> the >>>>> ???? >????????? >????? >????? >????? > information for specific >>>>> threads. The change introduces the thread table that uses >>>>> ConcurrentHashTable >>>>> ???? >????????? >????? >????? >????? > to store one-to-one the >>>>> mapping between the thread ids and JavaThread objects and replaces >>>>> the linear >>>>> ???? >????????? >????? >????? >????? > search over the thread list >>>>> in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method >>>>> with the lookup >>>>> ???? >????????? >????? >????? >????? > in the thread table. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 >>>>> and tier3 tests successfully passed. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >>>>> ???? >????????? >????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Best regards, >>>>> ???? >????????? >????? >????? >????? > Daniil >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? > >>>>> ???? >????????? > >>>>> ???? >????????? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> >>>>> >>>> >> From serguei.spitsyn at oracle.com Wed Sep 18 08:01:25 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 01:01:25 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: "serguei.spitsyn at oracle.com" > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ? ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > ?- the lines 632-633 are useless as they do not really protect from returning an exiting thread > ?- the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniel.daugherty at oracle.com Wed Sep 18 13:32:09 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 09:32:09 -0400 Subject: RFR (XXXXS): 8231162: JVMTI RawMonitorWait triggers assertion failure: Only JavaThreads can be interruptible In-Reply-To: References: Message-ID: <3ffac0c6-9dd6-41db-4993-1e9c3435768c@oracle.com> Thumbs up! This is a trivial change and only needs a single (R)eviewer. Did you rerun the failing tests to make sure this is the only issue? Dan P.S. I've done the "test a stack of patches" together and have something break when you push just one of the patches... just recently in fact. :-( At least yours didn't happen until Tier4... :-) On 9/18/19 2:26 AM, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8231162 > webrev: http://cr.openjdk.java.net/~dholmes/8231162/webrev/ > > -????? r = rmonitor->raw_wait(millis, true, thread); > +????? r = rmonitor->raw_wait(millis, false, thread); > > Non-JavaThreads are not interruptible and so "true" should not have > been being passed. This tripped over the assertions added as part of > the movement of the interrupt code to JavaThread under JDK-8230424. > > Dan: FYI I overlooked this because I already rewrote all this > RawMonitor logic under "8229160: Reimplement JvmtiRawMonitor to use > PlatformMonitor" to do the right thing, but of course that hasn't been > pushed yet. And this isn't detected until tier 4 testing. > > Thanks, > David From daniel.daugherty at oracle.com Wed Sep 18 13:36:00 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 09:36:00 -0400 Subject: RFR (XXXXS): 8231162: JVMTI RawMonitorWait triggers assertion failure: Only JavaThreads can be interruptible In-Reply-To: <3ffac0c6-9dd6-41db-4993-1e9c3435768c@oracle.com> References: <3ffac0c6-9dd6-41db-4993-1e9c3435768c@oracle.com> Message-ID: <7a6609f7-9349-31d7-5419-a34577adf560@oracle.com> Forgot to say: I'm really happy you put in that new assert: ? if (interruptible) { ??? assert(THREAD->is_Java_thread(), "Only JavaThreads can be interruptible"); Dan On 9/18/19 9:32 AM, Daniel D. Daugherty wrote: > Thumbs up! This is a trivial change and only needs a single (R)eviewer. > > Did you rerun the failing tests to make sure this is the only issue? > > Dan > > P.S. > I've done the "test a stack of patches" together and have something break > when you push just one of the patches... just recently in fact. :-( > At least yours didn't happen until Tier4... :-) > > On 9/18/19 2:26 AM, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231162 >> webrev: http://cr.openjdk.java.net/~dholmes/8231162/webrev/ >> >> -????? r = rmonitor->raw_wait(millis, true, thread); >> +????? r = rmonitor->raw_wait(millis, false, thread); >> >> Non-JavaThreads are not interruptible and so "true" should not have >> been being passed. This tripped over the assertions added as part of >> the movement of the interrupt code to JavaThread under JDK-8230424. >> >> Dan: FYI I overlooked this because I already rewrote all this >> RawMonitor logic under "8229160: Reimplement JvmtiRawMonitor to use >> PlatformMonitor" to do the right thing, but of course that hasn't >> been pushed yet. And this isn't detected until tier 4 testing. >> >> Thanks, >> David > From hohensee at amazon.com Wed Sep 18 15:03:07 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Wed, 18 Sep 2019 15:03:07 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> Message-ID: <7D09AF54-54DC-4064-83F9-29EA671C3484@amazon.com> Thanks David (and Mandy and Serguei). Pushed. ?On 9/17/19, 3:51 PM, "David Holmes" wrote: On 18/09/2019 12:10 am, Hohensee, Paul wrote: > Thanks, Serguei. :) > > David, are you ok with the patch? Yep, nothing further from me. David > Paul > > *From: *"serguei.spitsyn at oracle.com" > *Date: *Tuesday, September 17, 2019 at 2:26 AM > *To: *"Hohensee, Paul" , David Holmes > , Mandy Chung > *Cc: *OpenJDK Serviceability , > "hotspot-gc-dev at openjdk.java.net" > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > > Hi Paul, > > Thank you for refactoring and fixing the test. > It looks great now! > > Thanks, > Serguei > > > On 9/15/19 02:52, Hohensee, Paul wrote: > > Hi, Serguei, thanks for the review. New webrev at > > http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > I refactored the test?s main() method, and you?re correct, > getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > that context: fixed. > > Paul > > *From: *"serguei.spitsyn at oracle.com" > > > *Organization: *Oracle Corporation > *Date: *Friday, September 13, 2019 at 5:50 PM > *To: *"Hohensee, Paul" > , David Holmes > , Mandy Chung > > *Cc: *OpenJDK Serviceability > , > "hotspot-gc-dev at openjdk.java.net" > > > > *Subject: *Re: RFR (M): 8207266: > ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > Hi Paul, > > It looks pretty good in general. > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > It would be nice to refactor the java main() method as it becomes > too big. > Two ways ofgetCurrentThreadAllocatedBytes() testing are good candidates > to become separate methods. > > 98 long size1 = mbean.getThreadAllocatedBytes(id); > > Just wanted to double check if you wanted to invoke > the getCurrentThreadAllocatedBytes() instead as it is > a part of: > > 85 // First way, getCurrentThreadAllocatedBytes > > > Thanks, > Serguei > > On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > Hi David, thanks for your comments. New webrev in > > > > http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > > Both the old and new versions of the code check that thread allocated memory is both supported and enabled. The existing version of getThreadAllocatedBytes(long []) calls verifyThreadAllocatedMemory(long []), which checks inline to make sure thread allocated memory is supported, then calls isThreadAllocatedMemoryEnabled() to verify that it's enabled. isThreadAllocatedMemoryEnabled() duplicates (!) the support check and returns the enabled flag. I removed the redundant check in the new version. > > > > You're of course correct about the back-to-back check. Application code can't know when the runtime will hijack a thread for its own purposes. I've removed the check. > > > > Paul > > > > On 9/13/19, 12:50 AM, "David Holmes" wrote: > > > > Hi Paul, > > > > On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > > Thanks for clarifying the review rules. Would someone from the > > > serviceability team please review? New webrev at > > > > > >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > > One aspect of the functional change needs clarification for me - and > > apologies if this has been covered in the past. It seems to me that > > currently we only check isThreadAllocatedMemorySupported for these > > operations, but if I read things correctly the updated code additionally > > checks isThreadAllocatedMemoryEnabled, which is a behaviour change not > > mentioned in the CSR. > > > > > I didn?t disturb the existing checks in the test, just added code to > > > check the result of getThreadAllocatedBytes(long) on a non-current > > > thread, plus the back-to-back no-allocation checks. The former wasn?t > > > needed before because getThreadAllocatedBytes(long) was just a wrapper > > > around getThreadAllocatedBytes(long []). This patch changes that, so I > > > added a separate test. The latter is supposed to fail if there?s object > > > allocation on calls to getCurrentThreadAllocatedBytes and > > > getThreadAllocatedBytes(long). I.e., a feature, not a bug, because > > > accumulation of transient small objects can be a performance problem. > > > Thanks to your review, I noticed that the back-to-back check on the > > > current thread was using getThreadAllocatedBytes(long) instead of > > > getCurrentThreadAllocatedBytes and fixed it. I also removed all > > > instances of ?TEST FAILED: ?. > > > > The back-to-back check is not valid in general. You don't know if the > > first check might trigger some class loading on the return path after it > > has obtained the first memory value. The check might also fail if using > > JVMCI and some compilation related activity occurs in the current thread > > on the second call. Also with the introduction of handshakes its > > possible the current thread might hit a safepoint checks that results in > > it executing a handshake operation that performs allocation. Potentially > > there could be numerous non-deterministic actions that might occur > > leading to unanticipated allocation. > > > > I understand what you want to test here, I just don't think it is > > reliably doable. > > > > Thanks, > > David > > ----- > > > > > > > > Paul > > > > > > *From: *Mandy Chung > > > *Date: *Thursday, September 12, 2019 at 10:09 AM > > > *To: *"Hohensee, Paul" > > > *Cc: *OpenJDK Serviceability , > > >"hotspot-gc-dev at openjdk.java.net" > > > *Subject: *Re: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() > > > can be quicker for self thread > > > > > > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > > > > Minor update in new webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > > > > > > > I only reviewed the library side implementation that looks good. I > > > expect the serviceability team to review the test and hotspot change. > > > > > > > > > Need a confirmatory review to push this. If I understand the rules correctly, it doesn't need a Reviewer review since Mandy's already reviewed it, it just needs a Committer review. > > > > > > > > > You need another reviewer to advice the following because I was not > > > close to the ThreadsList work. > > > > > > 2087 ThreadsListHandle tlh; > > > > > > 2088 JavaThread* java_thread = tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > > > > 2089 > > > > > > 2090 if (java_thread != NULL) { > > > > > > 2091 return java_thread->cooked_allocated_bytes(); > > > > > > 2092 } > > > > > > This looks right to me. > > > > > > test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > > > > - "ThreadAllocatedMemory is expected to be disabled"); > > > > > > + "TEST FAILED: ThreadAllocatedMemory is expected to be > > > disabled"); > > > > > > Prepending "TEST FAILED" in exception message (in several places) > > > > > > seems redundant since such RuntimeException is thrown and expected > > > > > > a test failure. > > > > > > + // back-to-back calls shouldn't allocate any memory > > > > > > + size = mbean.getThreadAllocatedBytes(id); > > > > > > + size1 = mbean.getThreadAllocatedBytes(id); > > > > > > + if (size1 != size) { > > > > > > Is there anything in the test can do to help guarantee this? I didn't > > > > > > closely review this test. The main thing I advice is to improve > > > > > > the reliability of this test. Put it in another way, we want to > > > > > > ensure that this test change will pass all the time in various > > > > > > test configuration. > > > > > > Mandy > > > > > > > > > > From hohensee at amazon.com Wed Sep 18 17:56:28 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Wed, 18 Sep 2019 17:56:28 +0000 Subject: RFA Backport CSR to 11u: 8231194: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Message-ID: <4A7D9E4F-0D48-4C31-B092-55E03A45E0D1@amazon.com> Please review/approve an 11u CSR backport request. https://bugs.openjdk.java.net/browse/JDK-8231194 Original issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Original CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 11u backport issue: https://bugs.openjdk.java.net/browse/JDK-8231193 Email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-August/029011.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029033.html Thanks, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Wed Sep 18 20:26:15 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 16:26:15 -0400 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: Message-ID: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Forwarding this to the serviceability-dev at ... alias and Bcc'ing jdk-dev at ... which is a really broad alias... Dan On 9/18/19 2:47 PM, Sam Thomas wrote: > Hi, > > I'm trying to understand if a class will load as soon as all the > transformers return. The aim is to get a class reference of a class I have > seen in my transformer. > I'm currently using Class.forName(). > > Also if there is way to get the same without triggering class loading - if > the class is not loaded return a null reference. I ask this because there > is a scenario where there are two agents on JBoss where one asks for the > class reference using Class.forName() and the other went and performed a > redefineClass() on it and I end up getting a Failed to define class. > > Thanks > ./Sam From chris.plummer at oracle.com Wed Sep 18 21:11:17 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 18 Sep 2019 14:11:17 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr Message-ID: Hello, Please review the following changes: http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ https://bugs.openjdk.java.net/browse/JDK-8228625 There are actually numerous ways that JShellHeapDumpTest.java fails. One is a test bug, being addressed here, and the rest all seem to be SA bugs. Those are now being covered by JDK-8230872. All the issues seem to stem from the fact that the test spawns a jshell process, and then immediately does a "jhsdb jmap" on the process before jshell has fully started up. The test bug happens when the jmap succeeds, but jshell has not yet entered the main java thread. Thus the search for "JShellToolProvider" in the output fails. It expects "JShellToolProvider" to be in the output because it is part of a method name in the main thread, and the test dump all the thread stacks contained in the jmap generated hprof file. When the test fails in this way, you can see the stack dump in the output, but the main thread is missing. There's a couple of ways to fix this. One is to just add a delay (10s seems to be more than enough), and the other is to retry the "jhsdb jmap" command until the stack contains the JShellToolProvider symbol. I chose the later because doing a 10s delay masks the SA issues that are now covered by JDK-8230872. In a way the 10s delay is a better fix, because it makes this test pass every time, but I did not like that it also hid real SA problems in JDK-8230872. My plan for now is to do this retry fix, and then if there are too many failures due to JDK-8230872, then also add a 10s delay, with the intention of removing it once JDK-8230872 if fixed. From what I can see, JDK-8230872 failures happen on about 1% of the runs. I made a few of other changes. One was to no longer redirect stderr from the jmap process as was done from the following: processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); This causes the output not to appear in the OutputAnalyzer output, resulting in the following not working: ??????????? output.shouldNotContain("null"); Also I added code to dump the output of the jshell process so you can see if the jshell prompt was ever generated. thanks, Chris From david.holmes at oracle.com Wed Sep 18 21:31:06 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 07:31:06 +1000 Subject: RFR (XXXXS): 8231162: JVMTI RawMonitorWait triggers assertion failure: Only JavaThreads can be interruptible In-Reply-To: <3ffac0c6-9dd6-41db-4993-1e9c3435768c@oracle.com> References: <3ffac0c6-9dd6-41db-4993-1e9c3435768c@oracle.com> Message-ID: <5094af97-6f27-ae7e-9953-fe0abb02fd56@oracle.com> Hi Dan, On 18/09/2019 11:32 pm, Daniel D. Daugherty wrote: > Thumbs up! This is a trivial change and only needs a single (R)eviewer. Thanks for the review. I missed you by 10 minutes last night :( > Did you rerun the failing tests to make sure this is the only issue? Yes all the failing tests pass now. Thanks, David > Dan > > P.S. > I've done the "test a stack of patches" together and have something break > when you push just one of the patches... just recently in fact. :-( > At least yours didn't happen until Tier4... :-) > > On 9/18/19 2:26 AM, David Holmes wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231162 >> webrev: http://cr.openjdk.java.net/~dholmes/8231162/webrev/ >> >> -????? r = rmonitor->raw_wait(millis, true, thread); >> +????? r = rmonitor->raw_wait(millis, false, thread); >> >> Non-JavaThreads are not interruptible and so "true" should not have >> been being passed. This tripped over the assertions added as part of >> the movement of the interrupt code to JavaThread under JDK-8230424. >> >> Dan: FYI I overlooked this because I already rewrote all this >> RawMonitor logic under "8229160: Reimplement JvmtiRawMonitor to use >> PlatformMonitor" to do the right thing, but of course that hasn't been >> pushed yet. And this isn't detected until tier 4 testing. >> >> Thanks, >> David > From alexey.menkov at oracle.com Wed Sep 18 22:01:25 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Wed, 18 Sep 2019 15:01:25 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: Message-ID: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> Hi Chris, Did you think about waiting for jshell prompt ("jshell>") before run "jhsdb jmap" command instead of delay or re-tries? --alex On 09/18/2019 14:11, Chris Plummer wrote: > Hello, > > Please review the following changes: > > http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ > https://bugs.openjdk.java.net/browse/JDK-8228625 > > There are actually numerous ways that JShellHeapDumpTest.java fails. One > is a test bug, being addressed here, and the rest all seem to be SA > bugs. Those are now being covered by JDK-8230872. All the issues seem to > stem from the fact that the test spawns a jshell process, and then > immediately does a "jhsdb jmap" on the process before jshell has fully > started up. > > The test bug happens when the jmap succeeds, but jshell has not yet > entered the main java thread. Thus the search for "JShellToolProvider" > in the output fails. It expects "JShellToolProvider" to be in the output > because it is part of a method name in the main thread, and the test > dump all the thread stacks contained in the jmap generated hprof file. > When the test fails in this way, you can see the stack dump in the > output, but the main thread is missing. > > There's a couple of ways to fix this. One is to just add a delay (10s > seems to be more than enough), and the other is to retry the "jhsdb > jmap" command until the stack contains the JShellToolProvider symbol. I > chose the later because doing a 10s delay masks the SA issues that are > now covered by JDK-8230872. In a way the 10s delay is a better fix, > because it makes this test pass every time, but I did not like that it > also hid real SA problems in JDK-8230872. My plan for now is to do this > retry fix, and then if there are too many failures due to JDK-8230872, > then also add a 10s delay, with the intention of removing it once > JDK-8230872 if fixed. From what I can see, JDK-8230872 failures happen > on about 1% of the runs. > > I made a few of other changes. One was to no longer redirect stderr from > the jmap process as was done from the following: > > processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); > > This causes the output not to appear in the OutputAnalyzer output, > resulting in the following not working: > > ??????????? output.shouldNotContain("null"); > > Also I added code to dump the output of the jshell process so you can > see if the jshell prompt was ever generated. > > thanks, > > Chris > From chris.plummer at oracle.com Wed Sep 18 22:44:05 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 18 Sep 2019 15:44:05 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> Message-ID: <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Is there an easy way of doing this? Currently the jshell process is just spawned using Runtime.exec(). Chris On 9/18/19 3:01 PM, Alex Menkov wrote: > Hi Chris, > > Did you think about waiting for jshell prompt ("jshell>") before run > "jhsdb jmap" command instead of delay or re-tries? > > --alex > > On 09/18/2019 14:11, Chris Plummer wrote: >> Hello, >> >> Please review the following changes: >> >> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >> https://bugs.openjdk.java.net/browse/JDK-8228625 >> >> There are actually numerous ways that JShellHeapDumpTest.java fails. >> One is a test bug, being addressed here, and the rest all seem to be >> SA bugs. Those are now being covered by JDK-8230872. All the issues >> seem to stem from the fact that the test spawns a jshell process, and >> then immediately does a "jhsdb jmap" on the process before jshell has >> fully started up. >> >> The test bug happens when the jmap succeeds, but jshell has not yet >> entered the main java thread. Thus the search for >> "JShellToolProvider" in the output fails. It expects >> "JShellToolProvider" to be in the output because it is part of a >> method name in the main thread, and the test dump all the thread >> stacks contained in the jmap generated hprof file. When the test >> fails in this way, you can see the stack dump in the output, but the >> main thread is missing. >> >> There's a couple of ways to fix this. One is to just add a delay (10s >> seems to be more than enough), and the other is to retry the "jhsdb >> jmap" command until the stack contains the JShellToolProvider symbol. >> I chose the later because doing a 10s delay masks the SA issues that >> are now covered by JDK-8230872. In a way the 10s delay is a better >> fix, because it makes this test pass every time, but I did not like >> that it also hid real SA problems in JDK-8230872. My plan for now is >> to do this retry fix, and then if there are too many failures due to >> JDK-8230872, then also add a 10s delay, with the intention of >> removing it once JDK-8230872 if fixed. From what I can see, >> JDK-8230872 failures happen on about 1% of the runs. >> >> I made a few of other changes. One was to no longer redirect stderr >> from the jmap process as was done from the following: >> >> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >> >> This causes the output not to appear in the OutputAnalyzer output, >> resulting in the following not working: >> >> ???????????? output.shouldNotContain("null"); >> >> Also I added code to dump the output of the jshell process so you can >> see if the jshell prompt was ever generated. >> >> thanks, >> >> Chris >> From alexey.menkov at oracle.com Wed Sep 18 23:29:36 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Wed, 18 Sep 2019 16:29:36 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: You can use jdk.test.lib classes to simplify the things. Something like ProcessBuilder pb = new ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); Process p = ProcessTools.startProcess("JShell", pb, s -> { // warm-up predicate return s.contains(">jshell"); }); --alex On 09/18/2019 15:44, Chris Plummer wrote: > Is there an easy way of doing this? Currently the jshell process is just > spawned using Runtime.exec(). > > Chris > > On 9/18/19 3:01 PM, Alex Menkov wrote: >> Hi Chris, >> >> Did you think about waiting for jshell prompt ("jshell>") before run >> "jhsdb jmap" command instead of delay or re-tries? >> >> --alex >> >> On 09/18/2019 14:11, Chris Plummer wrote: >>> Hello, >>> >>> Please review the following changes: >>> >>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>> >>> There are actually numerous ways that JShellHeapDumpTest.java fails. >>> One is a test bug, being addressed here, and the rest all seem to be >>> SA bugs. Those are now being covered by JDK-8230872. All the issues >>> seem to stem from the fact that the test spawns a jshell process, and >>> then immediately does a "jhsdb jmap" on the process before jshell has >>> fully started up. >>> >>> The test bug happens when the jmap succeeds, but jshell has not yet >>> entered the main java thread. Thus the search for >>> "JShellToolProvider" in the output fails. It expects >>> "JShellToolProvider" to be in the output because it is part of a >>> method name in the main thread, and the test dump all the thread >>> stacks contained in the jmap generated hprof file. When the test >>> fails in this way, you can see the stack dump in the output, but the >>> main thread is missing. >>> >>> There's a couple of ways to fix this. One is to just add a delay (10s >>> seems to be more than enough), and the other is to retry the "jhsdb >>> jmap" command until the stack contains the JShellToolProvider symbol. >>> I chose the later because doing a 10s delay masks the SA issues that >>> are now covered by JDK-8230872. In a way the 10s delay is a better >>> fix, because it makes this test pass every time, but I did not like >>> that it also hid real SA problems in JDK-8230872. My plan for now is >>> to do this retry fix, and then if there are too many failures due to >>> JDK-8230872, then also add a 10s delay, with the intention of >>> removing it once JDK-8230872 if fixed. From what I can see, >>> JDK-8230872 failures happen on about 1% of the runs. >>> >>> I made a few of other changes. One was to no longer redirect stderr >>> from the jmap process as was done from the following: >>> >>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>> >>> This causes the output not to appear in the OutputAnalyzer output, >>> resulting in the following not working: >>> >>> ???????????? output.shouldNotContain("null"); >>> >>> Also I added code to dump the output of the jshell process so you can >>> see if the jshell prompt was ever generated. >>> >>> thanks, >>> >>> Chris >>> > > From alexey.menkov at oracle.com Wed Sep 18 23:39:16 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Wed, 18 Sep 2019 16:39:16 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: <690e1b0e-8f9c-ecda-d9bf-1232a52f03c8@oracle.com> Oh, I mean s.contains("jshell>") --alex On 09/18/2019 16:29, Alex Menkov wrote: > You can use jdk.test.lib classes to simplify the things. > Something like > > ProcessBuilder pb = new > ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); > Process p = ProcessTools.startProcess("JShell", pb, > ??? s -> {? // warm-up predicate > ??????? return s.contains(">jshell"); > ??? }); > > --alex > > On 09/18/2019 15:44, Chris Plummer wrote: >> Is there an easy way of doing this? Currently the jshell process is >> just spawned using Runtime.exec(). >> >> Chris >> >> On 9/18/19 3:01 PM, Alex Menkov wrote: >>> Hi Chris, >>> >>> Did you think about waiting for jshell prompt ("jshell>") before run >>> "jhsdb jmap" command instead of delay or re-tries? >>> >>> --alex >>> >>> On 09/18/2019 14:11, Chris Plummer wrote: >>>> Hello, >>>> >>>> Please review the following changes: >>>> >>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>> >>>> There are actually numerous ways that JShellHeapDumpTest.java fails. >>>> One is a test bug, being addressed here, and the rest all seem to be >>>> SA bugs. Those are now being covered by JDK-8230872. All the issues >>>> seem to stem from the fact that the test spawns a jshell process, >>>> and then immediately does a "jhsdb jmap" on the process before >>>> jshell has fully started up. >>>> >>>> The test bug happens when the jmap succeeds, but jshell has not yet >>>> entered the main java thread. Thus the search for >>>> "JShellToolProvider" in the output fails. It expects >>>> "JShellToolProvider" to be in the output because it is part of a >>>> method name in the main thread, and the test dump all the thread >>>> stacks contained in the jmap generated hprof file. When the test >>>> fails in this way, you can see the stack dump in the output, but the >>>> main thread is missing. >>>> >>>> There's a couple of ways to fix this. One is to just add a delay >>>> (10s seems to be more than enough), and the other is to retry the >>>> "jhsdb jmap" command until the stack contains the JShellToolProvider >>>> symbol. I chose the later because doing a 10s delay masks the SA >>>> issues that are now covered by JDK-8230872. In a way the 10s delay >>>> is a better fix, because it makes this test pass every time, but I >>>> did not like that it also hid real SA problems in JDK-8230872. My >>>> plan for now is to do this retry fix, and then if there are too many >>>> failures due to JDK-8230872, then also add a 10s delay, with the >>>> intention of removing it once JDK-8230872 if fixed. From what I can >>>> see, JDK-8230872 failures happen on about 1% of the runs. >>>> >>>> I made a few of other changes. One was to no longer redirect stderr >>>> from the jmap process as was done from the following: >>>> >>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>> >>>> This causes the output not to appear in the OutputAnalyzer output, >>>> resulting in the following not working: >>>> >>>> ???????????? output.shouldNotContain("null"); >>>> >>>> Also I added code to dump the output of the jshell process so you >>>> can see if the jshell prompt was ever generated. >>>> >>>> thanks, >>>> >>>> Chris >>>> >> >> From david.holmes at oracle.com Wed Sep 18 23:40:32 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 09:40:32 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> Message-ID: <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> Paul, Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: [2019-09-18T22:59:32,349Z] /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: error: ServerThreadMXBeanNew is not abstract and does not override abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean and possibly other issues as we are seeing hundreds of failures. David On 18/09/2019 8:50 am, David Holmes wrote: > On 18/09/2019 12:10 am, Hohensee, Paul wrote: >> Thanks, Serguei. :) >> >> David, are you ok with the patch? > > Yep, nothing further from me. > > David > >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> *Date: *Tuesday, September 17, 2019 at 2:26 AM >> *To: *"Hohensee, Paul" , David Holmes >> , Mandy Chung >> *Cc: *OpenJDK Serviceability , >> "hotspot-gc-dev at openjdk.java.net" >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> >> Hi Paul, >> >> Thank you for refactoring and fixing the test. >> It looks great now! >> >> Thanks, >> Serguei >> >> >> On 9/15/19 02:52, Hohensee, Paul wrote: >> >> ??? Hi, Serguei, thanks for the review. New webrev at >> >> ??? http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> >> ??? I refactored the test?s main() method, and you?re correct, >> ??? getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in >> ??? that context: fixed. >> >> ??? Paul >> >> ??? *From: *"serguei.spitsyn at oracle.com" >> ??? >> ??? >> ??? *Organization: *Oracle Corporation >> ??? *Date: *Friday, September 13, 2019 at 5:50 PM >> ??? *To: *"Hohensee, Paul" >> ??? , David Holmes >> ??? , Mandy Chung >> ??? >> ??? *Cc: *OpenJDK Serviceability >> ??? , >> ??? "hotspot-gc-dev at openjdk.java.net" >> ??? >> ??? >> ??? >> ??? *Subject: *Re: RFR (M): 8207266: >> ??? ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> >> ??? Hi Paul, >> >> ??? It looks pretty good in general. >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> >> >> ??? It would be nice to refactor the java main() method as it becomes >> ??? too big. >> ??? Two ways ofgetCurrentThreadAllocatedBytes() testing are good >> candidates >> ??? to become separate methods. >> >> ???? ? 98?? ??????long size1 = mbean.getThreadAllocatedBytes(id); >> >> ??? Just wanted to double check if you wanted to invoke >> ??? the getCurrentThreadAllocatedBytes() instead as it is >> ??? a part of: >> >> ???? ? 85???????? // First way, getCurrentThreadAllocatedBytes >> >> >> ??? Thanks, >> ??? Serguei >> >> ??? On 9/13/19 12:11 PM, Hohensee, Paul wrote: >> >> ??????? Hi David, thanks for your comments. New webrev in >> >> >> ??????? http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> >> >> ??????? Both the old and new versions of the code check that thread >> allocated memory is both supported and enabled. The existing version >> of getThreadAllocatedBytes(long []) calls >> verifyThreadAllocatedMemory(long []), which checks inline to make sure >> thread allocated memory is supported, then calls >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and >> returns the enabled flag. I removed the redundant check in the new >> version. >> >> >> ??????? You're of course correct about the back-to-back check. >> Application code can't know when the runtime will hijack a thread for >> its own purposes. I've removed the check. >> >> >> ??????? Paul >> >> >> ??????? On 9/13/19, 12:50 AM, "David Holmes" >> ? wrote: >> >> >> ???????? ??? Hi Paul, >> >> >> ???????? ????On 13/09/2019 10:29 am, Hohensee, Paul wrote: >> >> ???????? ??? > Thanks for clarifying the review rules. Would someone >> from the >> >> ???????? ????> serviceability team please review? New webrev at >> >> ???????? ??? > >> >> ???????? ????>http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> >> >> ???????? ????One aspect of the functional change needs clarification >> for me - and >> >> ???????? ????apologies if this has been covered in the past. It seems >> to me that >> >> ???????? ????currently we only check isThreadAllocatedMemorySupported >> for these >> >> ???????? ????operations, but if I read things correctly the updated >> code additionally >> >> ???????? ????checks isThreadAllocatedMemoryEnabled, which is a >> behaviour change not >> >> ???????? ????mentioned in the CSR. >> >> >> ???????? ????> I didn?t disturb the existing checks in the test, just >> added code to >> >> ???????? ????> check the result of getThreadAllocatedBytes(long) on a >> non-current >> >> ???????? ????> thread, plus the back-to-back no-allocation checks. The >> former wasn?t >> >> ???????? ????> needed before because getThreadAllocatedBytes(long) was >> just a wrapper >> >> ???????? ????> around getThreadAllocatedBytes(long []). This patch >> changes that, so I >> >> ???????? ????> added a separate test. The latter is supposed to fail >> if there?s object >> >> ???????? ????> allocation on calls to getCurrentThreadAllocatedBytes and >> >> ???????? ????> getThreadAllocatedBytes(long). I.e., a feature, not a >> bug, because >> >> ???????? ????> accumulation of transient small objects can be a >> performance problem. >> >> ???????? ????> Thanks to your review, I noticed that the back-to-back >> check on the >> >> ???????? ????> current thread was using getThreadAllocatedBytes(long) >> instead of >> >> ???????? ????> getCurrentThreadAllocatedBytes and fixed it. I also >> removed all >> >> ???????? ????> instances of ?TEST FAILED: ?. >> >> >> ???????? ????The back-to-back check is not valid in general. You don't >> know if the >> >> ???????? ????first check might trigger some class loading on the >> return path after it >> >> ???????? ????has obtained the first memory value. The check might also >> fail if using >> >> ???????? ????JVMCI and some compilation related activity occurs in the >> current thread >> >> ???????? ????on the second call. Also with the introduction of >> handshakes its >> >> ???????? ????possible the current thread might hit a safepoint checks >> that results in >> >> ???????? ????it executing a handshake operation that performs >> allocation. Potentially >> >> ???????? ????there could be numerous non-deterministic actions that >> might occur >> >> ???????? ????leading to unanticipated allocation. >> >> >> ???????? ????I understand what you want to test here, I just don't >> think it is >> >> ???????? ????reliably doable. >> >> >> ???????? ????Thanks, >> >> ???????? ??? David >> >> ???????? ??? ----- >> >> >> ???????? ????> >> >> ???????? ????> Paul >> >> ???????? ??? > >> >> ???????? ????> *From: *Mandy Chung >> >> >> ???????? ??? > *Date: *Thursday, September 12, 2019 at 10:09 AM >> >> ???????? ??? > *To: *"Hohensee, Paul" >> >> >> ???????? ??? > *Cc: *OpenJDK >> Serviceability >> , >> >> ???????? ????>"hotspot-gc-dev at openjdk.java.net" >> >> >> >> >> ???????? ??? > *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() >> >> ???????? ????> can be quicker for self thread >> >> ???????? ??? > >> >> ???????? ????> On 9/3/19 12:38 PM, Hohensee, Paul wrote: >> >> ???????? ??? > >> >> ???????? ????>???? Minor update in new >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> >> ???????? ??? > >> >> ???????? ????> >> >> ???????? ????> I only reviewed the library side implementation that >> looks good.? I >> >> ???????? ????> expect the serviceability team to review the test and >> hotspot change. >> >> ???????? ??? > >> >> ???????? ????> >> >> ???????? ????>???? Need a confirmatory review to push this. If I >> understand the rules correctly, it doesn't need a Reviewer review >> since Mandy's already reviewed it, it just needs a Committer review. >> >> ???????? ??? > >> >> ???????? ????> >> >> ???????? ????> You need another reviewer to advice the following >> because I was not >> >> ???????? ????> close to the ThreadsList work. >> >> ???????? ??? > >> >> ???????? ????> 2087?? ThreadsListHandle tlh; >> >> ???????? ??? > >> >> ???????? ????> 2088?? JavaThread* java_thread = >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> >> ???????? ??? > >> >> ???????? ????> 2089 >> >> ???????? ??? > >> >> ???????? ????> 2090?? if (java_thread != NULL) { >> >> ???????? ??? > >> >> ???????? ????> 2091???? return java_thread->cooked_allocated_bytes(); >> >> ???????? ??? > >> >> ???????? ????> 2092?? } >> >> ???????? ??? > >> >> ???????? ????> This looks right to me. >> >> ???????? ??? > >> >> ???????? ????> >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> >> ???????? ??? > >> >> ???????? ????> -??????????????? "ThreadAllocatedMemory is expected to >> be disabled"); >> >> ???????? ??? > >> >> ???????? ????> +??????????????? "TEST FAILED: ThreadAllocatedMemory is >> expected to be >> >> ???????? ????> disabled"); >> >> ???????? ??? > >> >> ???????? ????> Prepending "TEST FAILED" in exception message (in >> several places) >> >> ???????? ??? > >> >> ???????? ????> seems redundant since such RuntimeException is thrown >> and expected >> >> ???????? ??? > >> >> ???????? ????> a test failure. >> >> ???????? ??? > >> >> ???????? ????> +??????? // back-to-back calls shouldn't allocate any >> memory >> >> ???????? ??? > >> >> ???????? ????> +??????? size = mbean.getThreadAllocatedBytes(id); >> >> ???????? ??? > >> >> ???????? ????> +??????? size1 = mbean.getThreadAllocatedBytes(id); >> >> ???????? ??? > >> >> ???????? ????> +??????? if (size1 != size) { >> >> ???????? ??? > >> >> ???????? ????> Is there anything in the test can do to help guarantee >> this? I didn't >> >> ???????? ??? > >> >> ???????? ????> closely review this test.? The main thing I advice is >> to improve >> >> ???????? ??? > >> >> ???????? ????> the reliability of this test.? Put it in another way, >> we want to >> >> ???????? ??? > >> >> ???????? ????> ensure that this test change will pass all the time in >> various >> >> ???????? ??? > >> >> ???????? ????> test configuration. >> >> ???????? ??? > >> >> ???????? ????> Mandy >> >> ???????? ??? > >> >> >> >> >> From hohensee at amazon.com Wed Sep 18 23:47:02 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Wed, 18 Sep 2019 23:47:02 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> Message-ID: I'll take a look. ?On 9/18/19, 4:40 PM, "David Holmes" wrote: Paul, Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: [2019-09-18T22:59:32,349Z] /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: error: ServerThreadMXBeanNew is not abstract and does not override abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean and possibly other issues as we are seeing hundreds of failures. David On 18/09/2019 8:50 am, David Holmes wrote: > On 18/09/2019 12:10 am, Hohensee, Paul wrote: >> Thanks, Serguei. :) >> >> David, are you ok with the patch? > > Yep, nothing further from me. > > David > >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> *Date: *Tuesday, September 17, 2019 at 2:26 AM >> *To: *"Hohensee, Paul" , David Holmes >> , Mandy Chung >> *Cc: *OpenJDK Serviceability , >> "hotspot-gc-dev at openjdk.java.net" >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> >> Hi Paul, >> >> Thank you for refactoring and fixing the test. >> It looks great now! >> >> Thanks, >> Serguei >> >> >> On 9/15/19 02:52, Hohensee, Paul wrote: >> >> Hi, Serguei, thanks for the review. New webrev at >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> >> I refactored the test?s main() method, and you?re correct, >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in >> that context: fixed. >> >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> >> >> *Organization: *Oracle Corporation >> *Date: *Friday, September 13, 2019 at 5:50 PM >> *To: *"Hohensee, Paul" >> , David Holmes >> , Mandy Chung >> >> *Cc: *OpenJDK Serviceability >> , >> "hotspot-gc-dev at openjdk.java.net" >> >> >> >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> >> Hi Paul, >> >> It looks pretty good in general. >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> >> >> It would be nice to refactor the java main() method as it becomes >> too big. >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good >> candidates >> to become separate methods. >> >> 98 long size1 = mbean.getThreadAllocatedBytes(id); >> >> Just wanted to double check if you wanted to invoke >> the getCurrentThreadAllocatedBytes() instead as it is >> a part of: >> >> 85 // First way, getCurrentThreadAllocatedBytes >> >> >> Thanks, >> Serguei >> >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: >> >> Hi David, thanks for your comments. New webrev in >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> >> >> Both the old and new versions of the code check that thread >> allocated memory is both supported and enabled. The existing version >> of getThreadAllocatedBytes(long []) calls >> verifyThreadAllocatedMemory(long []), which checks inline to make sure >> thread allocated memory is supported, then calls >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and >> returns the enabled flag. I removed the redundant check in the new >> version. >> >> >> You're of course correct about the back-to-back check. >> Application code can't know when the runtime will hijack a thread for >> its own purposes. I've removed the check. >> >> >> Paul >> >> >> On 9/13/19, 12:50 AM, "David Holmes" >> wrote: >> >> >> Hi Paul, >> >> >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: >> >> > Thanks for clarifying the review rules. Would someone >> from the >> >> > serviceability team please review? New webrev at >> >> > >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> >> >> One aspect of the functional change needs clarification >> for me - and >> >> apologies if this has been covered in the past. It seems >> to me that >> >> currently we only check isThreadAllocatedMemorySupported >> for these >> >> operations, but if I read things correctly the updated >> code additionally >> >> checks isThreadAllocatedMemoryEnabled, which is a >> behaviour change not >> >> mentioned in the CSR. >> >> >> > I didn?t disturb the existing checks in the test, just >> added code to >> >> > check the result of getThreadAllocatedBytes(long) on a >> non-current >> >> > thread, plus the back-to-back no-allocation checks. The >> former wasn?t >> >> > needed before because getThreadAllocatedBytes(long) was >> just a wrapper >> >> > around getThreadAllocatedBytes(long []). This patch >> changes that, so I >> >> > added a separate test. The latter is supposed to fail >> if there?s object >> >> > allocation on calls to getCurrentThreadAllocatedBytes and >> >> > getThreadAllocatedBytes(long). I.e., a feature, not a >> bug, because >> >> > accumulation of transient small objects can be a >> performance problem. >> >> > Thanks to your review, I noticed that the back-to-back >> check on the >> >> > current thread was using getThreadAllocatedBytes(long) >> instead of >> >> > getCurrentThreadAllocatedBytes and fixed it. I also >> removed all >> >> > instances of ?TEST FAILED: ?. >> >> >> The back-to-back check is not valid in general. You don't >> know if the >> >> first check might trigger some class loading on the >> return path after it >> >> has obtained the first memory value. The check might also >> fail if using >> >> JVMCI and some compilation related activity occurs in the >> current thread >> >> on the second call. Also with the introduction of >> handshakes its >> >> possible the current thread might hit a safepoint checks >> that results in >> >> it executing a handshake operation that performs >> allocation. Potentially >> >> there could be numerous non-deterministic actions that >> might occur >> >> leading to unanticipated allocation. >> >> >> I understand what you want to test here, I just don't >> think it is >> >> reliably doable. >> >> >> Thanks, >> >> David >> >> ----- >> >> >> > >> >> > Paul >> >> > >> >> > *From: *Mandy Chung >> >> >> > *Date: *Thursday, September 12, 2019 at 10:09 AM >> >> > *To: *"Hohensee, Paul" >> >> >> > *Cc: *OpenJDK >> Serviceability >> , >> >> >"hotspot-gc-dev at openjdk.java.net" >> >> >> >> >> > *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() >> >> > can be quicker for self thread >> >> > >> >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: >> >> > >> >> > Minor update in new >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> >> > >> >> > >> >> > I only reviewed the library side implementation that >> looks good. I >> >> > expect the serviceability team to review the test and >> hotspot change. >> >> > >> >> > >> >> > Need a confirmatory review to push this. If I >> understand the rules correctly, it doesn't need a Reviewer review >> since Mandy's already reviewed it, it just needs a Committer review. >> >> > >> >> > >> >> > You need another reviewer to advice the following >> because I was not >> >> > close to the ThreadsList work. >> >> > >> >> > 2087 ThreadsListHandle tlh; >> >> > >> >> > 2088 JavaThread* java_thread = >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> >> > >> >> > 2089 >> >> > >> >> > 2090 if (java_thread != NULL) { >> >> > >> >> > 2091 return java_thread->cooked_allocated_bytes(); >> >> > >> >> > 2092 } >> >> > >> >> > This looks right to me. >> >> > >> >> > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> >> > >> >> > - "ThreadAllocatedMemory is expected to >> be disabled"); >> >> > >> >> > + "TEST FAILED: ThreadAllocatedMemory is >> expected to be >> >> > disabled"); >> >> > >> >> > Prepending "TEST FAILED" in exception message (in >> several places) >> >> > >> >> > seems redundant since such RuntimeException is thrown >> and expected >> >> > >> >> > a test failure. >> >> > >> >> > + // back-to-back calls shouldn't allocate any >> memory >> >> > >> >> > + size = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + size1 = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + if (size1 != size) { >> >> > >> >> > Is there anything in the test can do to help guarantee >> this? I didn't >> >> > >> >> > closely review this test. The main thing I advice is >> to improve >> >> > >> >> > the reliability of this test. Put it in another way, >> we want to >> >> > >> >> > ensure that this test change will pass all the time in >> various >> >> > >> >> > test configuration. >> >> > >> >> > Mandy >> >> > >> >> >> >> >> From hohensee at amazon.com Thu Sep 19 00:00:58 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:00:58 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> Message-ID: <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis public default long getCurrentThreadAllocatedBytes() { return -1; } Shall I go with that, or reverse the original patch? ?On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: I'll take a look. On 9/18/19, 4:40 PM, "David Holmes" wrote: Paul, Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: [2019-09-18T22:59:32,349Z] /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: error: ServerThreadMXBeanNew is not abstract and does not override abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean and possibly other issues as we are seeing hundreds of failures. David On 18/09/2019 8:50 am, David Holmes wrote: > On 18/09/2019 12:10 am, Hohensee, Paul wrote: >> Thanks, Serguei. :) >> >> David, are you ok with the patch? > > Yep, nothing further from me. > > David > >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> *Date: *Tuesday, September 17, 2019 at 2:26 AM >> *To: *"Hohensee, Paul" , David Holmes >> , Mandy Chung >> *Cc: *OpenJDK Serviceability , >> "hotspot-gc-dev at openjdk.java.net" >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> >> Hi Paul, >> >> Thank you for refactoring and fixing the test. >> It looks great now! >> >> Thanks, >> Serguei >> >> >> On 9/15/19 02:52, Hohensee, Paul wrote: >> >> Hi, Serguei, thanks for the review. New webrev at >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> >> I refactored the test?s main() method, and you?re correct, >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in >> that context: fixed. >> >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> >> >> *Organization: *Oracle Corporation >> *Date: *Friday, September 13, 2019 at 5:50 PM >> *To: *"Hohensee, Paul" >> , David Holmes >> , Mandy Chung >> >> *Cc: *OpenJDK Serviceability >> , >> "hotspot-gc-dev at openjdk.java.net" >> >> >> >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> >> Hi Paul, >> >> It looks pretty good in general. >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> >> >> It would be nice to refactor the java main() method as it becomes >> too big. >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good >> candidates >> to become separate methods. >> >> 98 long size1 = mbean.getThreadAllocatedBytes(id); >> >> Just wanted to double check if you wanted to invoke >> the getCurrentThreadAllocatedBytes() instead as it is >> a part of: >> >> 85 // First way, getCurrentThreadAllocatedBytes >> >> >> Thanks, >> Serguei >> >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: >> >> Hi David, thanks for your comments. New webrev in >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> >> >> Both the old and new versions of the code check that thread >> allocated memory is both supported and enabled. The existing version >> of getThreadAllocatedBytes(long []) calls >> verifyThreadAllocatedMemory(long []), which checks inline to make sure >> thread allocated memory is supported, then calls >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and >> returns the enabled flag. I removed the redundant check in the new >> version. >> >> >> You're of course correct about the back-to-back check. >> Application code can't know when the runtime will hijack a thread for >> its own purposes. I've removed the check. >> >> >> Paul >> >> >> On 9/13/19, 12:50 AM, "David Holmes" >> wrote: >> >> >> Hi Paul, >> >> >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: >> >> > Thanks for clarifying the review rules. Would someone >> from the >> >> > serviceability team please review? New webrev at >> >> > >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> >> >> One aspect of the functional change needs clarification >> for me - and >> >> apologies if this has been covered in the past. It seems >> to me that >> >> currently we only check isThreadAllocatedMemorySupported >> for these >> >> operations, but if I read things correctly the updated >> code additionally >> >> checks isThreadAllocatedMemoryEnabled, which is a >> behaviour change not >> >> mentioned in the CSR. >> >> >> > I didn?t disturb the existing checks in the test, just >> added code to >> >> > check the result of getThreadAllocatedBytes(long) on a >> non-current >> >> > thread, plus the back-to-back no-allocation checks. The >> former wasn?t >> >> > needed before because getThreadAllocatedBytes(long) was >> just a wrapper >> >> > around getThreadAllocatedBytes(long []). This patch >> changes that, so I >> >> > added a separate test. The latter is supposed to fail >> if there?s object >> >> > allocation on calls to getCurrentThreadAllocatedBytes and >> >> > getThreadAllocatedBytes(long). I.e., a feature, not a >> bug, because >> >> > accumulation of transient small objects can be a >> performance problem. >> >> > Thanks to your review, I noticed that the back-to-back >> check on the >> >> > current thread was using getThreadAllocatedBytes(long) >> instead of >> >> > getCurrentThreadAllocatedBytes and fixed it. I also >> removed all >> >> > instances of ?TEST FAILED: ?. >> >> >> The back-to-back check is not valid in general. You don't >> know if the >> >> first check might trigger some class loading on the >> return path after it >> >> has obtained the first memory value. The check might also >> fail if using >> >> JVMCI and some compilation related activity occurs in the >> current thread >> >> on the second call. Also with the introduction of >> handshakes its >> >> possible the current thread might hit a safepoint checks >> that results in >> >> it executing a handshake operation that performs >> allocation. Potentially >> >> there could be numerous non-deterministic actions that >> might occur >> >> leading to unanticipated allocation. >> >> >> I understand what you want to test here, I just don't >> think it is >> >> reliably doable. >> >> >> Thanks, >> >> David >> >> ----- >> >> >> > >> >> > Paul >> >> > >> >> > *From: *Mandy Chung >> >> >> > *Date: *Thursday, September 12, 2019 at 10:09 AM >> >> > *To: *"Hohensee, Paul" >> >> >> > *Cc: *OpenJDK >> Serviceability >> , >> >> >"hotspot-gc-dev at openjdk.java.net" >> >> >> >> >> > *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() >> >> > can be quicker for self thread >> >> > >> >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: >> >> > >> >> > Minor update in new >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> >> > >> >> > >> >> > I only reviewed the library side implementation that >> looks good. I >> >> > expect the serviceability team to review the test and >> hotspot change. >> >> > >> >> > >> >> > Need a confirmatory review to push this. If I >> understand the rules correctly, it doesn't need a Reviewer review >> since Mandy's already reviewed it, it just needs a Committer review. >> >> > >> >> > >> >> > You need another reviewer to advice the following >> because I was not >> >> > close to the ThreadsList work. >> >> > >> >> > 2087 ThreadsListHandle tlh; >> >> > >> >> > 2088 JavaThread* java_thread = >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> >> > >> >> > 2089 >> >> > >> >> > 2090 if (java_thread != NULL) { >> >> > >> >> > 2091 return java_thread->cooked_allocated_bytes(); >> >> > >> >> > 2092 } >> >> > >> >> > This looks right to me. >> >> > >> >> > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> >> > >> >> > - "ThreadAllocatedMemory is expected to >> be disabled"); >> >> > >> >> > + "TEST FAILED: ThreadAllocatedMemory is >> expected to be >> >> > disabled"); >> >> > >> >> > Prepending "TEST FAILED" in exception message (in >> several places) >> >> > >> >> > seems redundant since such RuntimeException is thrown >> and expected >> >> > >> >> > a test failure. >> >> > >> >> > + // back-to-back calls shouldn't allocate any >> memory >> >> > >> >> > + size = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + size1 = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + if (size1 != size) { >> >> > >> >> > Is there anything in the test can do to help guarantee >> this? I didn't >> >> > >> >> > closely review this test. The main thing I advice is >> to improve >> >> > >> >> > the reliability of this test. Put it in another way, >> we want to >> >> > >> >> > ensure that this test change will pass all the time in >> various >> >> > >> >> > test configuration. >> >> > >> >> > Mandy >> >> > >> >> >> >> >> From hohensee at amazon.com Thu Sep 19 00:10:02 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:10:02 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: I've filed https://bugs.openjdk.java.net/browse/JDK-8231209 for this quick fix. A better fix is to support getCurrentThreadAllocatedBytes in these tests. ?On 9/18/19, 5:02 PM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis public default long getCurrentThreadAllocatedBytes() { return -1; } Shall I go with that, or reverse the original patch? On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: I'll take a look. On 9/18/19, 4:40 PM, "David Holmes" wrote: Paul, Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: [2019-09-18T22:59:32,349Z] /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: error: ServerThreadMXBeanNew is not abstract and does not override abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean and possibly other issues as we are seeing hundreds of failures. David On 18/09/2019 8:50 am, David Holmes wrote: > On 18/09/2019 12:10 am, Hohensee, Paul wrote: >> Thanks, Serguei. :) >> >> David, are you ok with the patch? > > Yep, nothing further from me. > > David > >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> *Date: *Tuesday, September 17, 2019 at 2:26 AM >> *To: *"Hohensee, Paul" , David Holmes >> , Mandy Chung >> *Cc: *OpenJDK Serviceability , >> "hotspot-gc-dev at openjdk.java.net" >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> >> Hi Paul, >> >> Thank you for refactoring and fixing the test. >> It looks great now! >> >> Thanks, >> Serguei >> >> >> On 9/15/19 02:52, Hohensee, Paul wrote: >> >> Hi, Serguei, thanks for the review. New webrev at >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> >> I refactored the test?s main() method, and you?re correct, >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in >> that context: fixed. >> >> Paul >> >> *From: *"serguei.spitsyn at oracle.com" >> >> >> *Organization: *Oracle Corporation >> *Date: *Friday, September 13, 2019 at 5:50 PM >> *To: *"Hohensee, Paul" >> , David Holmes >> , Mandy Chung >> >> *Cc: *OpenJDK Serviceability >> , >> "hotspot-gc-dev at openjdk.java.net" >> >> >> >> *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> >> Hi Paul, >> >> It looks pretty good in general. >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> >> >> It would be nice to refactor the java main() method as it becomes >> too big. >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good >> candidates >> to become separate methods. >> >> 98 long size1 = mbean.getThreadAllocatedBytes(id); >> >> Just wanted to double check if you wanted to invoke >> the getCurrentThreadAllocatedBytes() instead as it is >> a part of: >> >> 85 // First way, getCurrentThreadAllocatedBytes >> >> >> Thanks, >> Serguei >> >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: >> >> Hi David, thanks for your comments. New webrev in >> >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> >> >> Both the old and new versions of the code check that thread >> allocated memory is both supported and enabled. The existing version >> of getThreadAllocatedBytes(long []) calls >> verifyThreadAllocatedMemory(long []), which checks inline to make sure >> thread allocated memory is supported, then calls >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and >> returns the enabled flag. I removed the redundant check in the new >> version. >> >> >> You're of course correct about the back-to-back check. >> Application code can't know when the runtime will hijack a thread for >> its own purposes. I've removed the check. >> >> >> Paul >> >> >> On 9/13/19, 12:50 AM, "David Holmes" >> wrote: >> >> >> Hi Paul, >> >> >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: >> >> > Thanks for clarifying the review rules. Would someone >> from the >> >> > serviceability team please review? New webrev at >> >> > >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> >> >> One aspect of the functional change needs clarification >> for me - and >> >> apologies if this has been covered in the past. It seems >> to me that >> >> currently we only check isThreadAllocatedMemorySupported >> for these >> >> operations, but if I read things correctly the updated >> code additionally >> >> checks isThreadAllocatedMemoryEnabled, which is a >> behaviour change not >> >> mentioned in the CSR. >> >> >> > I didn?t disturb the existing checks in the test, just >> added code to >> >> > check the result of getThreadAllocatedBytes(long) on a >> non-current >> >> > thread, plus the back-to-back no-allocation checks. The >> former wasn?t >> >> > needed before because getThreadAllocatedBytes(long) was >> just a wrapper >> >> > around getThreadAllocatedBytes(long []). This patch >> changes that, so I >> >> > added a separate test. The latter is supposed to fail >> if there?s object >> >> > allocation on calls to getCurrentThreadAllocatedBytes and >> >> > getThreadAllocatedBytes(long). I.e., a feature, not a >> bug, because >> >> > accumulation of transient small objects can be a >> performance problem. >> >> > Thanks to your review, I noticed that the back-to-back >> check on the >> >> > current thread was using getThreadAllocatedBytes(long) >> instead of >> >> > getCurrentThreadAllocatedBytes and fixed it. I also >> removed all >> >> > instances of ?TEST FAILED: ?. >> >> >> The back-to-back check is not valid in general. You don't >> know if the >> >> first check might trigger some class loading on the >> return path after it >> >> has obtained the first memory value. The check might also >> fail if using >> >> JVMCI and some compilation related activity occurs in the >> current thread >> >> on the second call. Also with the introduction of >> handshakes its >> >> possible the current thread might hit a safepoint checks >> that results in >> >> it executing a handshake operation that performs >> allocation. Potentially >> >> there could be numerous non-deterministic actions that >> might occur >> >> leading to unanticipated allocation. >> >> >> I understand what you want to test here, I just don't >> think it is >> >> reliably doable. >> >> >> Thanks, >> >> David >> >> ----- >> >> >> > >> >> > Paul >> >> > >> >> > *From: *Mandy Chung >> >> >> > *Date: *Thursday, September 12, 2019 at 10:09 AM >> >> > *To: *"Hohensee, Paul" >> >> >> > *Cc: *OpenJDK >> Serviceability >> , >> >> >"hotspot-gc-dev at openjdk.java.net" >> >> >> >> >> > *Subject: *Re: RFR (M): 8207266: >> ThreadMXBean::getThreadAllocatedBytes() >> >> > can be quicker for self thread >> >> > >> >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: >> >> > >> >> > Minor update in new >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> >> > >> >> > >> >> > I only reviewed the library side implementation that >> looks good. I >> >> > expect the serviceability team to review the test and >> hotspot change. >> >> > >> >> > >> >> > Need a confirmatory review to push this. If I >> understand the rules correctly, it doesn't need a Reviewer review >> since Mandy's already reviewed it, it just needs a Committer review. >> >> > >> >> > >> >> > You need another reviewer to advice the following >> because I was not >> >> > close to the ThreadsList work. >> >> > >> >> > 2087 ThreadsListHandle tlh; >> >> > >> >> > 2088 JavaThread* java_thread = >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> >> > >> >> > 2089 >> >> > >> >> > 2090 if (java_thread != NULL) { >> >> > >> >> > 2091 return java_thread->cooked_allocated_bytes(); >> >> > >> >> > 2092 } >> >> > >> >> > This looks right to me. >> >> > >> >> > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> >> > >> >> > - "ThreadAllocatedMemory is expected to >> be disabled"); >> >> > >> >> > + "TEST FAILED: ThreadAllocatedMemory is >> expected to be >> >> > disabled"); >> >> > >> >> > Prepending "TEST FAILED" in exception message (in >> several places) >> >> > >> >> > seems redundant since such RuntimeException is thrown >> and expected >> >> > >> >> > a test failure. >> >> > >> >> > + // back-to-back calls shouldn't allocate any >> memory >> >> > >> >> > + size = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + size1 = mbean.getThreadAllocatedBytes(id); >> >> > >> >> > + if (size1 != size) { >> >> > >> >> > Is there anything in the test can do to help guarantee >> this? I didn't >> >> > >> >> > closely review this test. The main thing I advice is >> to improve >> >> > >> >> > the reliability of this test. Put it in another way, >> we want to >> >> > >> >> > ensure that this test change will pass all the time in >> various >> >> > >> >> > test configuration. >> >> > >> >> > Mandy >> >> > >> >> >> >> >> From daniel.daugherty at oracle.com Thu Sep 19 00:14:13 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:14:13 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> > Shall I go with that, or reverse the original patch? I'm a bit worried about what else might show up since the NSK monitoring tests were not run prior to this push. I vote for backing out the fix until proper testing has been done (and at least the one problem fixed...) Dan On 9/18/19 8:00 PM, Hohensee, Paul wrote: > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > public default long getCurrentThreadAllocatedBytes() { > return -1; > } > > Shall I go with that, or reverse the original patch? > > ?On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > I'll take a look. > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > Paul, > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > [2019-09-18T22:59:32,349Z] > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > error: ServerThreadMXBeanNew is not abstract and does not override > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > and possibly other issues as we are seeing hundreds of failures. > > David > > On 18/09/2019 8:50 am, David Holmes wrote: > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > >> Thanks, Serguei. :) > >> > >> David, are you ok with the patch? > > > > Yep, nothing further from me. > > > > David > > > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > >> *To: *"Hohensee, Paul" , David Holmes > >> , Mandy Chung > >> *Cc: *OpenJDK Serviceability , > >> "hotspot-gc-dev at openjdk.java.net" > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > >> > >> Hi Paul, > >> > >> Thank you for refactoring and fixing the test. > >> It looks great now! > >> > >> Thanks, > >> Serguei > >> > >> > >> On 9/15/19 02:52, Hohensee, Paul wrote: > >> > >> Hi, Serguei, thanks for the review. New webrev at > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > >> > >> I refactored the test?s main() method, and you?re correct, > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > >> that context: fixed. > >> > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> > >> > >> *Organization: *Oracle Corporation > >> *Date: *Friday, September 13, 2019 at 5:50 PM > >> *To: *"Hohensee, Paul" > >> , David Holmes > >> , Mandy Chung > >> > >> *Cc: *OpenJDK Serviceability > >> , > >> "hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >> thread > >> > >> Hi Paul, > >> > >> It looks pretty good in general. > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > >> > >> > >> It would be nice to refactor the java main() method as it becomes > >> too big. > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > >> candidates > >> to become separate methods. > >> > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > >> > >> Just wanted to double check if you wanted to invoke > >> the getCurrentThreadAllocatedBytes() instead as it is > >> a part of: > >> > >> 85 // First way, getCurrentThreadAllocatedBytes > >> > >> > >> Thanks, > >> Serguei > >> > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > >> > >> Hi David, thanks for your comments. New webrev in > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > >> > >> > >> Both the old and new versions of the code check that thread > >> allocated memory is both supported and enabled. The existing version > >> of getThreadAllocatedBytes(long []) calls > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > >> thread allocated memory is supported, then calls > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > >> returns the enabled flag. I removed the redundant check in the new > >> version. > >> > >> > >> You're of course correct about the back-to-back check. > >> Application code can't know when the runtime will hijack a thread for > >> its own purposes. I've removed the check. > >> > >> > >> Paul > >> > >> > >> On 9/13/19, 12:50 AM, "David Holmes" > >> wrote: > >> > >> > >> Hi Paul, > >> > >> > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > >> > >> > Thanks for clarifying the review rules. Would someone > >> from the > >> > >> > serviceability team please review? New webrev at > >> > >> > > >> > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > >> > >> > >> One aspect of the functional change needs clarification > >> for me - and > >> > >> apologies if this has been covered in the past. It seems > >> to me that > >> > >> currently we only check isThreadAllocatedMemorySupported > >> for these > >> > >> operations, but if I read things correctly the updated > >> code additionally > >> > >> checks isThreadAllocatedMemoryEnabled, which is a > >> behaviour change not > >> > >> mentioned in the CSR. > >> > >> > >> > I didn?t disturb the existing checks in the test, just > >> added code to > >> > >> > check the result of getThreadAllocatedBytes(long) on a > >> non-current > >> > >> > thread, plus the back-to-back no-allocation checks. The > >> former wasn?t > >> > >> > needed before because getThreadAllocatedBytes(long) was > >> just a wrapper > >> > >> > around getThreadAllocatedBytes(long []). This patch > >> changes that, so I > >> > >> > added a separate test. The latter is supposed to fail > >> if there?s object > >> > >> > allocation on calls to getCurrentThreadAllocatedBytes and > >> > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > >> bug, because > >> > >> > accumulation of transient small objects can be a > >> performance problem. > >> > >> > Thanks to your review, I noticed that the back-to-back > >> check on the > >> > >> > current thread was using getThreadAllocatedBytes(long) > >> instead of > >> > >> > getCurrentThreadAllocatedBytes and fixed it. I also > >> removed all > >> > >> > instances of ?TEST FAILED: ?. > >> > >> > >> The back-to-back check is not valid in general. You don't > >> know if the > >> > >> first check might trigger some class loading on the > >> return path after it > >> > >> has obtained the first memory value. The check might also > >> fail if using > >> > >> JVMCI and some compilation related activity occurs in the > >> current thread > >> > >> on the second call. Also with the introduction of > >> handshakes its > >> > >> possible the current thread might hit a safepoint checks > >> that results in > >> > >> it executing a handshake operation that performs > >> allocation. Potentially > >> > >> there could be numerous non-deterministic actions that > >> might occur > >> > >> leading to unanticipated allocation. > >> > >> > >> I understand what you want to test here, I just don't > >> think it is > >> > >> reliably doable. > >> > >> > >> Thanks, > >> > >> David > >> > >> ----- > >> > >> > >> > > >> > >> > Paul > >> > >> > > >> > >> > *From: *Mandy Chung > >> > >> > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > >> > >> > *To: *"Hohensee, Paul" > >> > >> > >> > *Cc: *OpenJDK > >> Serviceability > >> , > >> > >> >"hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> > >> > *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() > >> > >> > can be quicker for self thread > >> > >> > > >> > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > >> > >> > > >> > >> > Minor update in new > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > >> > >> > > >> > >> > > >> > >> > I only reviewed the library side implementation that > >> looks good. I > >> > >> > expect the serviceability team to review the test and > >> hotspot change. > >> > >> > > >> > >> > > >> > >> > Need a confirmatory review to push this. If I > >> understand the rules correctly, it doesn't need a Reviewer review > >> since Mandy's already reviewed it, it just needs a Committer review. > >> > >> > > >> > >> > > >> > >> > You need another reviewer to advice the following > >> because I was not > >> > >> > close to the ThreadsList work. > >> > >> > > >> > >> > 2087 ThreadsListHandle tlh; > >> > >> > > >> > >> > 2088 JavaThread* java_thread = > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > >> > >> > > >> > >> > 2089 > >> > >> > > >> > >> > 2090 if (java_thread != NULL) { > >> > >> > > >> > >> > 2091 return java_thread->cooked_allocated_bytes(); > >> > >> > > >> > >> > 2092 } > >> > >> > > >> > >> > This looks right to me. > >> > >> > > >> > >> > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > >> > >> > > >> > >> > - "ThreadAllocatedMemory is expected to > >> be disabled"); > >> > >> > > >> > >> > + "TEST FAILED: ThreadAllocatedMemory is > >> expected to be > >> > >> > disabled"); > >> > >> > > >> > >> > Prepending "TEST FAILED" in exception message (in > >> several places) > >> > >> > > >> > >> > seems redundant since such RuntimeException is thrown > >> and expected > >> > >> > > >> > >> > a test failure. > >> > >> > > >> > >> > + // back-to-back calls shouldn't allocate any > >> memory > >> > >> > > >> > >> > + size = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + size1 = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + if (size1 != size) { > >> > >> > > >> > >> > Is there anything in the test can do to help guarantee > >> this? I didn't > >> > >> > > >> > >> > closely review this test. The main thing I advice is > >> to improve > >> > >> > > >> > >> > the reliability of this test. Put it in another way, > >> we want to > >> > >> > > >> > >> > ensure that this test change will pass all the time in > >> various > >> > >> > > >> > >> > test configuration. > >> > >> > > >> > >> > Mandy > >> > >> > > >> > >> > >> > >> > >> > > > > From hohensee at amazon.com Thu Sep 19 00:17:04 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:17:04 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> Message-ID: <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> Is there a tool that will generate a reversal patch? ?On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > Shall I go with that, or reverse the original patch? I'm a bit worried about what else might show up since the NSK monitoring tests were not run prior to this push. I vote for backing out the fix until proper testing has been done (and at least the one problem fixed...) Dan On 9/18/19 8:00 PM, Hohensee, Paul wrote: > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > public default long getCurrentThreadAllocatedBytes() { > return -1; > } > > Shall I go with that, or reverse the original patch? > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > I'll take a look. > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > Paul, > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > [2019-09-18T22:59:32,349Z] > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > error: ServerThreadMXBeanNew is not abstract and does not override > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > and possibly other issues as we are seeing hundreds of failures. > > David > > On 18/09/2019 8:50 am, David Holmes wrote: > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > >> Thanks, Serguei. :) > >> > >> David, are you ok with the patch? > > > > Yep, nothing further from me. > > > > David > > > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > >> *To: *"Hohensee, Paul" , David Holmes > >> , Mandy Chung > >> *Cc: *OpenJDK Serviceability , > >> "hotspot-gc-dev at openjdk.java.net" > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > >> > >> Hi Paul, > >> > >> Thank you for refactoring and fixing the test. > >> It looks great now! > >> > >> Thanks, > >> Serguei > >> > >> > >> On 9/15/19 02:52, Hohensee, Paul wrote: > >> > >> Hi, Serguei, thanks for the review. New webrev at > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > >> > >> I refactored the test?s main() method, and you?re correct, > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > >> that context: fixed. > >> > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> > >> > >> *Organization: *Oracle Corporation > >> *Date: *Friday, September 13, 2019 at 5:50 PM > >> *To: *"Hohensee, Paul" > >> , David Holmes > >> , Mandy Chung > >> > >> *Cc: *OpenJDK Serviceability > >> , > >> "hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >> thread > >> > >> Hi Paul, > >> > >> It looks pretty good in general. > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > >> > >> > >> It would be nice to refactor the java main() method as it becomes > >> too big. > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > >> candidates > >> to become separate methods. > >> > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > >> > >> Just wanted to double check if you wanted to invoke > >> the getCurrentThreadAllocatedBytes() instead as it is > >> a part of: > >> > >> 85 // First way, getCurrentThreadAllocatedBytes > >> > >> > >> Thanks, > >> Serguei > >> > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > >> > >> Hi David, thanks for your comments. New webrev in > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > >> > >> > >> Both the old and new versions of the code check that thread > >> allocated memory is both supported and enabled. The existing version > >> of getThreadAllocatedBytes(long []) calls > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > >> thread allocated memory is supported, then calls > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > >> returns the enabled flag. I removed the redundant check in the new > >> version. > >> > >> > >> You're of course correct about the back-to-back check. > >> Application code can't know when the runtime will hijack a thread for > >> its own purposes. I've removed the check. > >> > >> > >> Paul > >> > >> > >> On 9/13/19, 12:50 AM, "David Holmes" > >> wrote: > >> > >> > >> Hi Paul, > >> > >> > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > >> > >> > Thanks for clarifying the review rules. Would someone > >> from the > >> > >> > serviceability team please review? New webrev at > >> > >> > > >> > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > >> > >> > >> One aspect of the functional change needs clarification > >> for me - and > >> > >> apologies if this has been covered in the past. It seems > >> to me that > >> > >> currently we only check isThreadAllocatedMemorySupported > >> for these > >> > >> operations, but if I read things correctly the updated > >> code additionally > >> > >> checks isThreadAllocatedMemoryEnabled, which is a > >> behaviour change not > >> > >> mentioned in the CSR. > >> > >> > >> > I didn?t disturb the existing checks in the test, just > >> added code to > >> > >> > check the result of getThreadAllocatedBytes(long) on a > >> non-current > >> > >> > thread, plus the back-to-back no-allocation checks. The > >> former wasn?t > >> > >> > needed before because getThreadAllocatedBytes(long) was > >> just a wrapper > >> > >> > around getThreadAllocatedBytes(long []). This patch > >> changes that, so I > >> > >> > added a separate test. The latter is supposed to fail > >> if there?s object > >> > >> > allocation on calls to getCurrentThreadAllocatedBytes and > >> > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > >> bug, because > >> > >> > accumulation of transient small objects can be a > >> performance problem. > >> > >> > Thanks to your review, I noticed that the back-to-back > >> check on the > >> > >> > current thread was using getThreadAllocatedBytes(long) > >> instead of > >> > >> > getCurrentThreadAllocatedBytes and fixed it. I also > >> removed all > >> > >> > instances of ?TEST FAILED: ?. > >> > >> > >> The back-to-back check is not valid in general. You don't > >> know if the > >> > >> first check might trigger some class loading on the > >> return path after it > >> > >> has obtained the first memory value. The check might also > >> fail if using > >> > >> JVMCI and some compilation related activity occurs in the > >> current thread > >> > >> on the second call. Also with the introduction of > >> handshakes its > >> > >> possible the current thread might hit a safepoint checks > >> that results in > >> > >> it executing a handshake operation that performs > >> allocation. Potentially > >> > >> there could be numerous non-deterministic actions that > >> might occur > >> > >> leading to unanticipated allocation. > >> > >> > >> I understand what you want to test here, I just don't > >> think it is > >> > >> reliably doable. > >> > >> > >> Thanks, > >> > >> David > >> > >> ----- > >> > >> > >> > > >> > >> > Paul > >> > >> > > >> > >> > *From: *Mandy Chung > >> > >> > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > >> > >> > *To: *"Hohensee, Paul" > >> > >> > >> > *Cc: *OpenJDK > >> Serviceability > >> , > >> > >> >"hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> > >> > *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() > >> > >> > can be quicker for self thread > >> > >> > > >> > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > >> > >> > > >> > >> > Minor update in new > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > >> > >> > > >> > >> > > >> > >> > I only reviewed the library side implementation that > >> looks good. I > >> > >> > expect the serviceability team to review the test and > >> hotspot change. > >> > >> > > >> > >> > > >> > >> > Need a confirmatory review to push this. If I > >> understand the rules correctly, it doesn't need a Reviewer review > >> since Mandy's already reviewed it, it just needs a Committer review. > >> > >> > > >> > >> > > >> > >> > You need another reviewer to advice the following > >> because I was not > >> > >> > close to the ThreadsList work. > >> > >> > > >> > >> > 2087 ThreadsListHandle tlh; > >> > >> > > >> > >> > 2088 JavaThread* java_thread = > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > >> > >> > > >> > >> > 2089 > >> > >> > > >> > >> > 2090 if (java_thread != NULL) { > >> > >> > > >> > >> > 2091 return java_thread->cooked_allocated_bytes(); > >> > >> > > >> > >> > 2092 } > >> > >> > > >> > >> > This looks right to me. > >> > >> > > >> > >> > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > >> > >> > > >> > >> > - "ThreadAllocatedMemory is expected to > >> be disabled"); > >> > >> > > >> > >> > + "TEST FAILED: ThreadAllocatedMemory is > >> expected to be > >> > >> > disabled"); > >> > >> > > >> > >> > Prepending "TEST FAILED" in exception message (in > >> several places) > >> > >> > > >> > >> > seems redundant since such RuntimeException is thrown > >> and expected > >> > >> > > >> > >> > a test failure. > >> > >> > > >> > >> > + // back-to-back calls shouldn't allocate any > >> memory > >> > >> > > >> > >> > + size = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + size1 = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + if (size1 != size) { > >> > >> > > >> > >> > Is there anything in the test can do to help guarantee > >> this? I didn't > >> > >> > > >> > >> > closely review this test. The main thing I advice is > >> to improve > >> > >> > > >> > >> > the reliability of this test. Put it in another way, > >> we want to > >> > >> > > >> > >> > ensure that this test change will pass all the time in > >> various > >> > >> > > >> > >> > test configuration. > >> > >> > > >> > >> > Mandy > >> > >> > > >> > >> > >> > >> > >> > > > > From daniel.daugherty at oracle.com Thu Sep 19 00:18:43 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:18:43 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> References: <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> Message-ID: % hg backout is the usual way to do this... Dan On 9/18/19 8:17 PM, Hohensee, Paul wrote: > Is there a tool that will generate a reversal patch? > > ?On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > Shall I go with that, or reverse the original patch? > > I'm a bit worried about what else might show up since the > NSK monitoring tests were not run prior to this push. > > I vote for backing out the fix until proper testing has > been done (and at least the one problem fixed...) > > Dan > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > public default long getCurrentThreadAllocatedBytes() { > > return -1; > > } > > > > Shall I go with that, or reverse the original patch? > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > I'll take a look. > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > Paul, > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > [2019-09-18T22:59:32,349Z] > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > error: ServerThreadMXBeanNew is not abstract and does not override > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > and possibly other issues as we are seeing hundreds of failures. > > > > David > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > >> Thanks, Serguei. :) > > >> > > >> David, are you ok with the patch? > > > > > > Yep, nothing further from me. > > > > > > David > > > > > >> Paul > > >> > > >> *From: *"serguei.spitsyn at oracle.com" > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > >> *To: *"Hohensee, Paul" , David Holmes > > >> , Mandy Chung > > >> *Cc: *OpenJDK Serviceability , > > >> "hotspot-gc-dev at openjdk.java.net" > > >> *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > >> > > >> Hi Paul, > > >> > > >> Thank you for refactoring and fixing the test. > > >> It looks great now! > > >> > > >> Thanks, > > >> Serguei > > >> > > >> > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > >> > > >> Hi, Serguei, thanks for the review. New webrev at > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > >> > > >> I refactored the test?s main() method, and you?re correct, > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > >> that context: fixed. > > >> > > >> Paul > > >> > > >> *From: *"serguei.spitsyn at oracle.com" > > >> > > >> > > >> *Organization: *Oracle Corporation > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > >> *To: *"Hohensee, Paul" > > >> , David Holmes > > >> , Mandy Chung > > >> > > >> *Cc: *OpenJDK Serviceability > > >> , > > >> "hotspot-gc-dev at openjdk.java.net" > > >> > > >> > > >> > > >> *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > >> thread > > >> > > >> Hi Paul, > > >> > > >> It looks pretty good in general. > > >> > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > >> > > >> > > >> It would be nice to refactor the java main() method as it becomes > > >> too big. > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > >> candidates > > >> to become separate methods. > > >> > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > >> > > >> Just wanted to double check if you wanted to invoke > > >> the getCurrentThreadAllocatedBytes() instead as it is > > >> a part of: > > >> > > >> 85 // First way, getCurrentThreadAllocatedBytes > > >> > > >> > > >> Thanks, > > >> Serguei > > >> > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > >> > > >> Hi David, thanks for your comments. New webrev in > > >> > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > >> > > >> > > >> Both the old and new versions of the code check that thread > > >> allocated memory is both supported and enabled. The existing version > > >> of getThreadAllocatedBytes(long []) calls > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > >> thread allocated memory is supported, then calls > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > >> returns the enabled flag. I removed the redundant check in the new > > >> version. > > >> > > >> > > >> You're of course correct about the back-to-back check. > > >> Application code can't know when the runtime will hijack a thread for > > >> its own purposes. I've removed the check. > > >> > > >> > > >> Paul > > >> > > >> > > >> On 9/13/19, 12:50 AM, "David Holmes" > > >> wrote: > > >> > > >> > > >> Hi Paul, > > >> > > >> > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > >> > > >> > Thanks for clarifying the review rules. Would someone > > >> from the > > >> > > >> > serviceability team please review? New webrev at > > >> > > >> > > > >> > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > >> > > >> > > >> One aspect of the functional change needs clarification > > >> for me - and > > >> > > >> apologies if this has been covered in the past. It seems > > >> to me that > > >> > > >> currently we only check isThreadAllocatedMemorySupported > > >> for these > > >> > > >> operations, but if I read things correctly the updated > > >> code additionally > > >> > > >> checks isThreadAllocatedMemoryEnabled, which is a > > >> behaviour change not > > >> > > >> mentioned in the CSR. > > >> > > >> > > >> > I didn?t disturb the existing checks in the test, just > > >> added code to > > >> > > >> > check the result of getThreadAllocatedBytes(long) on a > > >> non-current > > >> > > >> > thread, plus the back-to-back no-allocation checks. The > > >> former wasn?t > > >> > > >> > needed before because getThreadAllocatedBytes(long) was > > >> just a wrapper > > >> > > >> > around getThreadAllocatedBytes(long []). This patch > > >> changes that, so I > > >> > > >> > added a separate test. The latter is supposed to fail > > >> if there?s object > > >> > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > >> > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > >> bug, because > > >> > > >> > accumulation of transient small objects can be a > > >> performance problem. > > >> > > >> > Thanks to your review, I noticed that the back-to-back > > >> check on the > > >> > > >> > current thread was using getThreadAllocatedBytes(long) > > >> instead of > > >> > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > >> removed all > > >> > > >> > instances of ?TEST FAILED: ?. > > >> > > >> > > >> The back-to-back check is not valid in general. You don't > > >> know if the > > >> > > >> first check might trigger some class loading on the > > >> return path after it > > >> > > >> has obtained the first memory value. The check might also > > >> fail if using > > >> > > >> JVMCI and some compilation related activity occurs in the > > >> current thread > > >> > > >> on the second call. Also with the introduction of > > >> handshakes its > > >> > > >> possible the current thread might hit a safepoint checks > > >> that results in > > >> > > >> it executing a handshake operation that performs > > >> allocation. Potentially > > >> > > >> there could be numerous non-deterministic actions that > > >> might occur > > >> > > >> leading to unanticipated allocation. > > >> > > >> > > >> I understand what you want to test here, I just don't > > >> think it is > > >> > > >> reliably doable. > > >> > > >> > > >> Thanks, > > >> > > >> David > > >> > > >> ----- > > >> > > >> > > >> > > > >> > > >> > Paul > > >> > > >> > > > >> > > >> > *From: *Mandy Chung > > >> > > >> > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > >> > > >> > *To: *"Hohensee, Paul" > > >> > > >> > > >> > *Cc: *OpenJDK > > >> Serviceability > > >> , > > >> > > >> >"hotspot-gc-dev at openjdk.java.net" > > >> > > >> > > >> > > >> > > >> > *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() > > >> > > >> > can be quicker for self thread > > >> > > >> > > > >> > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > >> > > >> > > > >> > > >> > Minor update in new > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > >> > > >> > > > >> > > >> > > > >> > > >> > I only reviewed the library side implementation that > > >> looks good. I > > >> > > >> > expect the serviceability team to review the test and > > >> hotspot change. > > >> > > >> > > > >> > > >> > > > >> > > >> > Need a confirmatory review to push this. If I > > >> understand the rules correctly, it doesn't need a Reviewer review > > >> since Mandy's already reviewed it, it just needs a Committer review. > > >> > > >> > > > >> > > >> > > > >> > > >> > You need another reviewer to advice the following > > >> because I was not > > >> > > >> > close to the ThreadsList work. > > >> > > >> > > > >> > > >> > 2087 ThreadsListHandle tlh; > > >> > > >> > > > >> > > >> > 2088 JavaThread* java_thread = > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > >> > > >> > > > >> > > >> > 2089 > > >> > > >> > > > >> > > >> > 2090 if (java_thread != NULL) { > > >> > > >> > > > >> > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > >> > > >> > > > >> > > >> > 2092 } > > >> > > >> > > > >> > > >> > This looks right to me. > > >> > > >> > > > >> > > >> > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > >> > > >> > > > >> > > >> > - "ThreadAllocatedMemory is expected to > > >> be disabled"); > > >> > > >> > > > >> > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > >> expected to be > > >> > > >> > disabled"); > > >> > > >> > > > >> > > >> > Prepending "TEST FAILED" in exception message (in > > >> several places) > > >> > > >> > > > >> > > >> > seems redundant since such RuntimeException is thrown > > >> and expected > > >> > > >> > > > >> > > >> > a test failure. > > >> > > >> > > > >> > > >> > + // back-to-back calls shouldn't allocate any > > >> memory > > >> > > >> > > > >> > > >> > + size = mbean.getThreadAllocatedBytes(id); > > >> > > >> > > > >> > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > >> > > >> > > > >> > > >> > + if (size1 != size) { > > >> > > >> > > > >> > > >> > Is there anything in the test can do to help guarantee > > >> this? I didn't > > >> > > >> > > > >> > > >> > closely review this test. The main thing I advice is > > >> to improve > > >> > > >> > > > >> > > >> > the reliability of this test. Put it in another way, > > >> we want to > > >> > > >> > > > >> > > >> > ensure that this test change will pass all the time in > > >> various > > >> > > >> > > > >> > > >> > test configuration. > > >> > > >> > > > >> > > >> > Mandy > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > > From hohensee at amazon.com Thu Sep 19 00:21:30 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:21:30 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> Message-ID: <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> Never having done this before, is it hg backout -r ? Do I file a JBS issue for the reversion? Seems necessary. ?On 9/18/19, 5:18 PM, "Daniel D. Daugherty" wrote: % hg backout is the usual way to do this... Dan On 9/18/19 8:17 PM, Hohensee, Paul wrote: > Is there a tool that will generate a reversal patch? > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > Shall I go with that, or reverse the original patch? > > I'm a bit worried about what else might show up since the > NSK monitoring tests were not run prior to this push. > > I vote for backing out the fix until proper testing has > been done (and at least the one problem fixed...) > > Dan > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > public default long getCurrentThreadAllocatedBytes() { > > return -1; > > } > > > > Shall I go with that, or reverse the original patch? > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > I'll take a look. > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > Paul, > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > [2019-09-18T22:59:32,349Z] > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > error: ServerThreadMXBeanNew is not abstract and does not override > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > and possibly other issues as we are seeing hundreds of failures. > > > > David > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > >> Thanks, Serguei. :) > > >> > > >> David, are you ok with the patch? > > > > > > Yep, nothing further from me. > > > > > > David > > > > > >> Paul > > >> > > >> *From: *"serguei.spitsyn at oracle.com" > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > >> *To: *"Hohensee, Paul" , David Holmes > > >> , Mandy Chung > > >> *Cc: *OpenJDK Serviceability , > > >> "hotspot-gc-dev at openjdk.java.net" > > >> *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > >> > > >> Hi Paul, > > >> > > >> Thank you for refactoring and fixing the test. > > >> It looks great now! > > >> > > >> Thanks, > > >> Serguei > > >> > > >> > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > >> > > >> Hi, Serguei, thanks for the review. New webrev at > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > >> > > >> I refactored the test?s main() method, and you?re correct, > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > >> that context: fixed. > > >> > > >> Paul > > >> > > >> *From: *"serguei.spitsyn at oracle.com" > > >> > > >> > > >> *Organization: *Oracle Corporation > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > >> *To: *"Hohensee, Paul" > > >> , David Holmes > > >> , Mandy Chung > > >> > > >> *Cc: *OpenJDK Serviceability > > >> , > > >> "hotspot-gc-dev at openjdk.java.net" > > >> > > >> > > >> > > >> *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > >> thread > > >> > > >> Hi Paul, > > >> > > >> It looks pretty good in general. > > >> > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > >> > > >> > > >> It would be nice to refactor the java main() method as it becomes > > >> too big. > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > >> candidates > > >> to become separate methods. > > >> > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > >> > > >> Just wanted to double check if you wanted to invoke > > >> the getCurrentThreadAllocatedBytes() instead as it is > > >> a part of: > > >> > > >> 85 // First way, getCurrentThreadAllocatedBytes > > >> > > >> > > >> Thanks, > > >> Serguei > > >> > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > >> > > >> Hi David, thanks for your comments. New webrev in > > >> > > >> > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > >> > > >> > > >> Both the old and new versions of the code check that thread > > >> allocated memory is both supported and enabled. The existing version > > >> of getThreadAllocatedBytes(long []) calls > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > >> thread allocated memory is supported, then calls > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > >> returns the enabled flag. I removed the redundant check in the new > > >> version. > > >> > > >> > > >> You're of course correct about the back-to-back check. > > >> Application code can't know when the runtime will hijack a thread for > > >> its own purposes. I've removed the check. > > >> > > >> > > >> Paul > > >> > > >> > > >> On 9/13/19, 12:50 AM, "David Holmes" > > >> wrote: > > >> > > >> > > >> Hi Paul, > > >> > > >> > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > >> > > >> > Thanks for clarifying the review rules. Would someone > > >> from the > > >> > > >> > serviceability team please review? New webrev at > > >> > > >> > > > >> > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > >> > > >> > > >> One aspect of the functional change needs clarification > > >> for me - and > > >> > > >> apologies if this has been covered in the past. It seems > > >> to me that > > >> > > >> currently we only check isThreadAllocatedMemorySupported > > >> for these > > >> > > >> operations, but if I read things correctly the updated > > >> code additionally > > >> > > >> checks isThreadAllocatedMemoryEnabled, which is a > > >> behaviour change not > > >> > > >> mentioned in the CSR. > > >> > > >> > > >> > I didn?t disturb the existing checks in the test, just > > >> added code to > > >> > > >> > check the result of getThreadAllocatedBytes(long) on a > > >> non-current > > >> > > >> > thread, plus the back-to-back no-allocation checks. The > > >> former wasn?t > > >> > > >> > needed before because getThreadAllocatedBytes(long) was > > >> just a wrapper > > >> > > >> > around getThreadAllocatedBytes(long []). This patch > > >> changes that, so I > > >> > > >> > added a separate test. The latter is supposed to fail > > >> if there?s object > > >> > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > >> > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > >> bug, because > > >> > > >> > accumulation of transient small objects can be a > > >> performance problem. > > >> > > >> > Thanks to your review, I noticed that the back-to-back > > >> check on the > > >> > > >> > current thread was using getThreadAllocatedBytes(long) > > >> instead of > > >> > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > >> removed all > > >> > > >> > instances of ?TEST FAILED: ?. > > >> > > >> > > >> The back-to-back check is not valid in general. You don't > > >> know if the > > >> > > >> first check might trigger some class loading on the > > >> return path after it > > >> > > >> has obtained the first memory value. The check might also > > >> fail if using > > >> > > >> JVMCI and some compilation related activity occurs in the > > >> current thread > > >> > > >> on the second call. Also with the introduction of > > >> handshakes its > > >> > > >> possible the current thread might hit a safepoint checks > > >> that results in > > >> > > >> it executing a handshake operation that performs > > >> allocation. Potentially > > >> > > >> there could be numerous non-deterministic actions that > > >> might occur > > >> > > >> leading to unanticipated allocation. > > >> > > >> > > >> I understand what you want to test here, I just don't > > >> think it is > > >> > > >> reliably doable. > > >> > > >> > > >> Thanks, > > >> > > >> David > > >> > > >> ----- > > >> > > >> > > >> > > > >> > > >> > Paul > > >> > > >> > > > >> > > >> > *From: *Mandy Chung > > >> > > >> > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > >> > > >> > *To: *"Hohensee, Paul" > > >> > > >> > > >> > *Cc: *OpenJDK > > >> Serviceability > > >> , > > >> > > >> >"hotspot-gc-dev at openjdk.java.net" > > >> > > >> > > >> > > >> > > >> > *Subject: *Re: RFR (M): 8207266: > > >> ThreadMXBean::getThreadAllocatedBytes() > > >> > > >> > can be quicker for self thread > > >> > > >> > > > >> > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > >> > > >> > > > >> > > >> > Minor update in new > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > >> > > >> > > > >> > > >> > > > >> > > >> > I only reviewed the library side implementation that > > >> looks good. I > > >> > > >> > expect the serviceability team to review the test and > > >> hotspot change. > > >> > > >> > > > >> > > >> > > > >> > > >> > Need a confirmatory review to push this. If I > > >> understand the rules correctly, it doesn't need a Reviewer review > > >> since Mandy's already reviewed it, it just needs a Committer review. > > >> > > >> > > > >> > > >> > > > >> > > >> > You need another reviewer to advice the following > > >> because I was not > > >> > > >> > close to the ThreadsList work. > > >> > > >> > > > >> > > >> > 2087 ThreadsListHandle tlh; > > >> > > >> > > > >> > > >> > 2088 JavaThread* java_thread = > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > >> > > >> > > > >> > > >> > 2089 > > >> > > >> > > > >> > > >> > 2090 if (java_thread != NULL) { > > >> > > >> > > > >> > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > >> > > >> > > > >> > > >> > 2092 } > > >> > > >> > > > >> > > >> > This looks right to me. > > >> > > >> > > > >> > > >> > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > >> > > >> > > > >> > > >> > - "ThreadAllocatedMemory is expected to > > >> be disabled"); > > >> > > >> > > > >> > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > >> expected to be > > >> > > >> > disabled"); > > >> > > >> > > > >> > > >> > Prepending "TEST FAILED" in exception message (in > > >> several places) > > >> > > >> > > > >> > > >> > seems redundant since such RuntimeException is thrown > > >> and expected > > >> > > >> > > > >> > > >> > a test failure. > > >> > > >> > > > >> > > >> > + // back-to-back calls shouldn't allocate any > > >> memory > > >> > > >> > > > >> > > >> > + size = mbean.getThreadAllocatedBytes(id); > > >> > > >> > > > >> > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > >> > > >> > > > >> > > >> > + if (size1 != size) { > > >> > > >> > > > >> > > >> > Is there anything in the test can do to help guarantee > > >> this? I didn't > > >> > > >> > > > >> > > >> > closely review this test. The main thing I advice is > > >> to improve > > >> > > >> > > > >> > > >> > the reliability of this test. Put it in another way, > > >> we want to > > >> > > >> > > > >> > > >> > ensure that this test change will pass all the time in > > >> various > > >> > > >> > > > >> > > >> > test configuration. > > >> > > >> > > > >> > > >> > Mandy > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > > From david.holmes at oracle.com Thu Sep 19 00:21:40 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 10:21:40 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: <67eb44c3-a9e9-7377-a4e5-01f07ccc5462@oracle.com> Hi Paul, On 19/09/2019 10:00 am, Hohensee, Paul wrote: > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > public default long getCurrentThreadAllocatedBytes() { > return -1; > } > > Shall I go with that, or reverse the original patch? Any default implementation must follow the specification for the new method: * equivalent to calling: * 
  *   {@link #getThreadAllocatedBytes 
getThreadAllocatedBytes}(Thread.currentThread().getId());
  *
The CSR request for this seems to be quite misleading in the compatibility section: "Adds a method with well-understood semantics and leaves existing method semantics unchanged. Existing programs will continue to compile and execute as before. " It may have been assumed, based on the wording, that there would be a default implementation, otherwise the second sentence is obviously not correct. But it's clear from the specification section that this was not going to be a default method. It may be better to back this out and redo with a revised CSR request. Thanks, David > ?On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > I'll take a look. > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > Paul, > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > [2019-09-18T22:59:32,349Z] > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > error: ServerThreadMXBeanNew is not abstract and does not override > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > and possibly other issues as we are seeing hundreds of failures. > > David > > On 18/09/2019 8:50 am, David Holmes wrote: > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > >> Thanks, Serguei. :) > >> > >> David, are you ok with the patch? > > > > Yep, nothing further from me. > > > > David > > > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > >> *To: *"Hohensee, Paul" , David Holmes > >> , Mandy Chung > >> *Cc: *OpenJDK Serviceability , > >> "hotspot-gc-dev at openjdk.java.net" > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > >> > >> Hi Paul, > >> > >> Thank you for refactoring and fixing the test. > >> It looks great now! > >> > >> Thanks, > >> Serguei > >> > >> > >> On 9/15/19 02:52, Hohensee, Paul wrote: > >> > >> Hi, Serguei, thanks for the review. New webrev at > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > >> > >> I refactored the test?s main() method, and you?re correct, > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > >> that context: fixed. > >> > >> Paul > >> > >> *From: *"serguei.spitsyn at oracle.com" > >> > >> > >> *Organization: *Oracle Corporation > >> *Date: *Friday, September 13, 2019 at 5:50 PM > >> *To: *"Hohensee, Paul" > >> , David Holmes > >> , Mandy Chung > >> > >> *Cc: *OpenJDK Serviceability > >> , > >> "hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >> thread > >> > >> Hi Paul, > >> > >> It looks pretty good in general. > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > >> > >> > >> It would be nice to refactor the java main() method as it becomes > >> too big. > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > >> candidates > >> to become separate methods. > >> > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > >> > >> Just wanted to double check if you wanted to invoke > >> the getCurrentThreadAllocatedBytes() instead as it is > >> a part of: > >> > >> 85 // First way, getCurrentThreadAllocatedBytes > >> > >> > >> Thanks, > >> Serguei > >> > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > >> > >> Hi David, thanks for your comments. New webrev in > >> > >> > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > >> > >> > >> Both the old and new versions of the code check that thread > >> allocated memory is both supported and enabled. The existing version > >> of getThreadAllocatedBytes(long []) calls > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > >> thread allocated memory is supported, then calls > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > >> returns the enabled flag. I removed the redundant check in the new > >> version. > >> > >> > >> You're of course correct about the back-to-back check. > >> Application code can't know when the runtime will hijack a thread for > >> its own purposes. I've removed the check. > >> > >> > >> Paul > >> > >> > >> On 9/13/19, 12:50 AM, "David Holmes" > >> wrote: > >> > >> > >> Hi Paul, > >> > >> > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > >> > >> > Thanks for clarifying the review rules. Would someone > >> from the > >> > >> > serviceability team please review? New webrev at > >> > >> > > >> > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > >> > >> > >> One aspect of the functional change needs clarification > >> for me - and > >> > >> apologies if this has been covered in the past. It seems > >> to me that > >> > >> currently we only check isThreadAllocatedMemorySupported > >> for these > >> > >> operations, but if I read things correctly the updated > >> code additionally > >> > >> checks isThreadAllocatedMemoryEnabled, which is a > >> behaviour change not > >> > >> mentioned in the CSR. > >> > >> > >> > I didn?t disturb the existing checks in the test, just > >> added code to > >> > >> > check the result of getThreadAllocatedBytes(long) on a > >> non-current > >> > >> > thread, plus the back-to-back no-allocation checks. The > >> former wasn?t > >> > >> > needed before because getThreadAllocatedBytes(long) was > >> just a wrapper > >> > >> > around getThreadAllocatedBytes(long []). This patch > >> changes that, so I > >> > >> > added a separate test. The latter is supposed to fail > >> if there?s object > >> > >> > allocation on calls to getCurrentThreadAllocatedBytes and > >> > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > >> bug, because > >> > >> > accumulation of transient small objects can be a > >> performance problem. > >> > >> > Thanks to your review, I noticed that the back-to-back > >> check on the > >> > >> > current thread was using getThreadAllocatedBytes(long) > >> instead of > >> > >> > getCurrentThreadAllocatedBytes and fixed it. I also > >> removed all > >> > >> > instances of ?TEST FAILED: ?. > >> > >> > >> The back-to-back check is not valid in general. You don't > >> know if the > >> > >> first check might trigger some class loading on the > >> return path after it > >> > >> has obtained the first memory value. The check might also > >> fail if using > >> > >> JVMCI and some compilation related activity occurs in the > >> current thread > >> > >> on the second call. Also with the introduction of > >> handshakes its > >> > >> possible the current thread might hit a safepoint checks > >> that results in > >> > >> it executing a handshake operation that performs > >> allocation. Potentially > >> > >> there could be numerous non-deterministic actions that > >> might occur > >> > >> leading to unanticipated allocation. > >> > >> > >> I understand what you want to test here, I just don't > >> think it is > >> > >> reliably doable. > >> > >> > >> Thanks, > >> > >> David > >> > >> ----- > >> > >> > >> > > >> > >> > Paul > >> > >> > > >> > >> > *From: *Mandy Chung > >> > >> > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > >> > >> > *To: *"Hohensee, Paul" > >> > >> > >> > *Cc: *OpenJDK > >> Serviceability > >> , > >> > >> >"hotspot-gc-dev at openjdk.java.net" > >> > >> > >> > >> > >> > *Subject: *Re: RFR (M): 8207266: > >> ThreadMXBean::getThreadAllocatedBytes() > >> > >> > can be quicker for self thread > >> > >> > > >> > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > >> > >> > > >> > >> > Minor update in new > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > >> > >> > > >> > >> > > >> > >> > I only reviewed the library side implementation that > >> looks good. I > >> > >> > expect the serviceability team to review the test and > >> hotspot change. > >> > >> > > >> > >> > > >> > >> > Need a confirmatory review to push this. If I > >> understand the rules correctly, it doesn't need a Reviewer review > >> since Mandy's already reviewed it, it just needs a Committer review. > >> > >> > > >> > >> > > >> > >> > You need another reviewer to advice the following > >> because I was not > >> > >> > close to the ThreadsList work. > >> > >> > > >> > >> > 2087 ThreadsListHandle tlh; > >> > >> > > >> > >> > 2088 JavaThread* java_thread = > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > >> > >> > > >> > >> > 2089 > >> > >> > > >> > >> > 2090 if (java_thread != NULL) { > >> > >> > > >> > >> > 2091 return java_thread->cooked_allocated_bytes(); > >> > >> > > >> > >> > 2092 } > >> > >> > > >> > >> > This looks right to me. > >> > >> > > >> > >> > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > >> > >> > > >> > >> > - "ThreadAllocatedMemory is expected to > >> be disabled"); > >> > >> > > >> > >> > + "TEST FAILED: ThreadAllocatedMemory is > >> expected to be > >> > >> > disabled"); > >> > >> > > >> > >> > Prepending "TEST FAILED" in exception message (in > >> several places) > >> > >> > > >> > >> > seems redundant since such RuntimeException is thrown > >> and expected > >> > >> > > >> > >> > a test failure. > >> > >> > > >> > >> > + // back-to-back calls shouldn't allocate any > >> memory > >> > >> > > >> > >> > + size = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + size1 = mbean.getThreadAllocatedBytes(id); > >> > >> > > >> > >> > + if (size1 != size) { > >> > >> > > >> > >> > Is there anything in the test can do to help guarantee > >> this? I didn't > >> > >> > > >> > >> > closely review this test. The main thing I advice is > >> to improve > >> > >> > > >> > >> > the reliability of this test. Put it in another way, > >> we want to > >> > >> > > >> > >> > ensure that this test change will pass all the time in > >> various > >> > >> > > >> > >> > test configuration. > >> > >> > > >> > >> > Mandy > >> > >> > > >> > >> > >> > >> > >> > > > > From daniel.daugherty at oracle.com Thu Sep 19 00:25:19 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:25:19 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> References: <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> Message-ID: <6139732f-a914-c59c-2119-517e480a279c@oracle.com> I created this sub-task for you: JDK-8231210 [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread https://bugs.openjdk.java.net/browse/JDK-8231210 If you would prefer, I can handle this backout for you. Dan On 9/18/19 8:21 PM, Hohensee, Paul wrote: > Never having done this before, is it > > hg backout -r > > ? Do I file a JBS issue for the reversion? Seems necessary. > > ?On 9/18/19, 5:18 PM, "Daniel D. Daugherty" wrote: > > % hg backout > > is the usual way to do this... > > Dan > > > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > > Is there a tool that will generate a reversal patch? > > > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > > > Shall I go with that, or reverse the original patch? > > > > I'm a bit worried about what else might show up since the > > NSK monitoring tests were not run prior to this push. > > > > I vote for backing out the fix until proper testing has > > been done (and at least the one problem fixed...) > > > > Dan > > > > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > > > public default long getCurrentThreadAllocatedBytes() { > > > return -1; > > > } > > > > > > Shall I go with that, or reverse the original patch? > > > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > I'll take a look. > > > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > > > Paul, > > > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > > > [2019-09-18T22:59:32,349Z] > > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > > error: ServerThreadMXBeanNew is not abstract and does not override > > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > > > and possibly other issues as we are seeing hundreds of failures. > > > > > > David > > > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > > >> Thanks, Serguei. :) > > > >> > > > >> David, are you ok with the patch? > > > > > > > > Yep, nothing further from me. > > > > > > > > David > > > > > > > >> Paul > > > >> > > > >> *From: *"serguei.spitsyn at oracle.com" > > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > > >> *To: *"Hohensee, Paul" , David Holmes > > > >> , Mandy Chung > > > >> *Cc: *OpenJDK Serviceability , > > > >> "hotspot-gc-dev at openjdk.java.net" > > > >> *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > > >> > > > >> Hi Paul, > > > >> > > > >> Thank you for refactoring and fixing the test. > > > >> It looks great now! > > > >> > > > >> Thanks, > > > >> Serguei > > > >> > > > >> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > > >> > > > >> Hi, Serguei, thanks for the review. New webrev at > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > > >> > > > >> I refactored the test?s main() method, and you?re correct, > > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > > >> that context: fixed. > > > >> > > > >> Paul > > > >> > > > >> *From: *"serguei.spitsyn at oracle.com" > > > >> > > > >> > > > >> *Organization: *Oracle Corporation > > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > > >> *To: *"Hohensee, Paul" > > > >> , David Holmes > > > >> , Mandy Chung > > > >> > > > >> *Cc: *OpenJDK Serviceability > > > >> , > > > >> "hotspot-gc-dev at openjdk.java.net" > > > >> > > > >> > > > >> > > > >> *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > > >> thread > > > >> > > > >> Hi Paul, > > > >> > > > >> It looks pretty good in general. > > > >> > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > > >> > > > >> > > > >> It would be nice to refactor the java main() method as it becomes > > > >> too big. > > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > > >> candidates > > > >> to become separate methods. > > > >> > > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> Just wanted to double check if you wanted to invoke > > > >> the getCurrentThreadAllocatedBytes() instead as it is > > > >> a part of: > > > >> > > > >> 85 // First way, getCurrentThreadAllocatedBytes > > > >> > > > >> > > > >> Thanks, > > > >> Serguei > > > >> > > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > > >> > > > >> Hi David, thanks for your comments. New webrev in > > > >> > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > >> > > > >> > > > >> Both the old and new versions of the code check that thread > > > >> allocated memory is both supported and enabled. The existing version > > > >> of getThreadAllocatedBytes(long []) calls > > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > > >> thread allocated memory is supported, then calls > > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > > >> returns the enabled flag. I removed the redundant check in the new > > > >> version. > > > >> > > > >> > > > >> You're of course correct about the back-to-back check. > > > >> Application code can't know when the runtime will hijack a thread for > > > >> its own purposes. I've removed the check. > > > >> > > > >> > > > >> Paul > > > >> > > > >> > > > >> On 9/13/19, 12:50 AM, "David Holmes" > > > >> wrote: > > > >> > > > >> > > > >> Hi Paul, > > > >> > > > >> > > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > > >> > > > >> > Thanks for clarifying the review rules. Would someone > > > >> from the > > > >> > > > >> > serviceability team please review? New webrev at > > > >> > > > >> > > > > >> > > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > >> > > > >> > > > >> One aspect of the functional change needs clarification > > > >> for me - and > > > >> > > > >> apologies if this has been covered in the past. It seems > > > >> to me that > > > >> > > > >> currently we only check isThreadAllocatedMemorySupported > > > >> for these > > > >> > > > >> operations, but if I read things correctly the updated > > > >> code additionally > > > >> > > > >> checks isThreadAllocatedMemoryEnabled, which is a > > > >> behaviour change not > > > >> > > > >> mentioned in the CSR. > > > >> > > > >> > > > >> > I didn?t disturb the existing checks in the test, just > > > >> added code to > > > >> > > > >> > check the result of getThreadAllocatedBytes(long) on a > > > >> non-current > > > >> > > > >> > thread, plus the back-to-back no-allocation checks. The > > > >> former wasn?t > > > >> > > > >> > needed before because getThreadAllocatedBytes(long) was > > > >> just a wrapper > > > >> > > > >> > around getThreadAllocatedBytes(long []). This patch > > > >> changes that, so I > > > >> > > > >> > added a separate test. The latter is supposed to fail > > > >> if there?s object > > > >> > > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > > >> > > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > > >> bug, because > > > >> > > > >> > accumulation of transient small objects can be a > > > >> performance problem. > > > >> > > > >> > Thanks to your review, I noticed that the back-to-back > > > >> check on the > > > >> > > > >> > current thread was using getThreadAllocatedBytes(long) > > > >> instead of > > > >> > > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > > >> removed all > > > >> > > > >> > instances of ?TEST FAILED: ?. > > > >> > > > >> > > > >> The back-to-back check is not valid in general. You don't > > > >> know if the > > > >> > > > >> first check might trigger some class loading on the > > > >> return path after it > > > >> > > > >> has obtained the first memory value. The check might also > > > >> fail if using > > > >> > > > >> JVMCI and some compilation related activity occurs in the > > > >> current thread > > > >> > > > >> on the second call. Also with the introduction of > > > >> handshakes its > > > >> > > > >> possible the current thread might hit a safepoint checks > > > >> that results in > > > >> > > > >> it executing a handshake operation that performs > > > >> allocation. Potentially > > > >> > > > >> there could be numerous non-deterministic actions that > > > >> might occur > > > >> > > > >> leading to unanticipated allocation. > > > >> > > > >> > > > >> I understand what you want to test here, I just don't > > > >> think it is > > > >> > > > >> reliably doable. > > > >> > > > >> > > > >> Thanks, > > > >> > > > >> David > > > >> > > > >> ----- > > > >> > > > >> > > > >> > > > > >> > > > >> > Paul > > > >> > > > >> > > > > >> > > > >> > *From: *Mandy Chung > > > >> > > > >> > > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > > >> > > > >> > *To: *"Hohensee, Paul" > > > >> > > > >> > > > >> > *Cc: *OpenJDK > > > >> Serviceability > > > >> , > > > >> > > > >> >"hotspot-gc-dev at openjdk.java.net" > > > >> > > > >> > > > >> > > > >> > > > >> > *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() > > > >> > > > >> > can be quicker for self thread > > > >> > > > >> > > > > >> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > >> > > > >> > > > > >> > > > >> > Minor update in new > > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > I only reviewed the library side implementation that > > > >> looks good. I > > > >> > > > >> > expect the serviceability team to review the test and > > > >> hotspot change. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > Need a confirmatory review to push this. If I > > > >> understand the rules correctly, it doesn't need a Reviewer review > > > >> since Mandy's already reviewed it, it just needs a Committer review. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > You need another reviewer to advice the following > > > >> because I was not > > > >> > > > >> > close to the ThreadsList work. > > > >> > > > >> > > > > >> > > > >> > 2087 ThreadsListHandle tlh; > > > >> > > > >> > > > > >> > > > >> > 2088 JavaThread* java_thread = > > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > >> > > > >> > > > > >> > > > >> > 2089 > > > >> > > > >> > > > > >> > > > >> > 2090 if (java_thread != NULL) { > > > >> > > > >> > > > > >> > > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > > >> > > > >> > > > > >> > > > >> > 2092 } > > > >> > > > >> > > > > >> > > > >> > This looks right to me. > > > >> > > > >> > > > > >> > > > >> > > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > >> > > > >> > > > > >> > > > >> > - "ThreadAllocatedMemory is expected to > > > >> be disabled"); > > > >> > > > >> > > > > >> > > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > > >> expected to be > > > >> > > > >> > disabled"); > > > >> > > > >> > > > > >> > > > >> > Prepending "TEST FAILED" in exception message (in > > > >> several places) > > > >> > > > >> > > > > >> > > > >> > seems redundant since such RuntimeException is thrown > > > >> and expected > > > >> > > > >> > > > > >> > > > >> > a test failure. > > > >> > > > >> > > > > >> > > > >> > + // back-to-back calls shouldn't allocate any > > > >> memory > > > >> > > > >> > > > > >> > > > >> > + size = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> > > > > >> > > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> > > > > >> > > > >> > + if (size1 != size) { > > > >> > > > >> > > > > >> > > > >> > Is there anything in the test can do to help guarantee > > > >> this? I didn't > > > >> > > > >> > > > > >> > > > >> > closely review this test. The main thing I advice is > > > >> to improve > > > >> > > > >> > > > > >> > > > >> > the reliability of this test. Put it in another way, > > > >> we want to > > > >> > > > >> > > > > >> > > > >> > ensure that this test change will pass all the time in > > > >> various > > > >> > > > >> > > > > >> > > > >> > test configuration. > > > >> > > > >> > > > > >> > > > >> > Mandy > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > From daniel.daugherty at oracle.com Thu Sep 19 00:34:41 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:34:41 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <6139732f-a914-c59c-2119-517e480a279c@oracle.com> References: <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> Message-ID: I have the backout ready to go. About to send out a webrev... Dan On 9/18/19 8:25 PM, Daniel D. Daugherty wrote: > I created this sub-task for you: > > JDK-8231210 [BACKOUT] JDK-8207266 > ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > https://bugs.openjdk.java.net/browse/JDK-8231210 > > If you would prefer, I can handle this backout for you. > > Dan > > > On 9/18/19 8:21 PM, Hohensee, Paul wrote: >> Never having done this before, is it >> >> hg backout -r >> >> ? Do I file a JBS issue for the reversion? Seems necessary. >> >> ?On 9/18/19, 5:18 PM, "Daniel D. Daugherty" >> wrote: >> >> ???? % hg backout >> ???? ???? is the usual way to do this... >> ???? ???? Dan >> ???? ???? ???? On 9/18/19 8:17 PM, Hohensee, Paul wrote: >> ???? > Is there a tool that will generate a reversal patch? >> ???? > >> ???? > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" >> wrote: >> ???? > >> ???? >?????? > Shall I go with that, or reverse the original patch? >> ???? > >> ???? >????? I'm a bit worried about what else might show up since the >> ???? >????? NSK monitoring tests were not run prior to this push. >> ???? > >> ???? >????? I vote for backing out the fix until proper testing has >> ???? >????? been done (and at least the one problem fixed...) >> ???? > >> ???? >????? Dan >> ???? > >> ???? > >> ???? >????? On 9/18/19 8:00 PM, Hohensee, Paul wrote: >> ???? >????? > They all implement com.sun.management.ThreadMXBean, so >> adding a getCurrentThreadAllocatedBytes broke them. Potential fix is >> to give it a default implementation, vis >> ???? >????? > >> ???? >????? >????? public default long >> getCurrentThreadAllocatedBytes() { >> ???? >????? >????????? return -1; >> ???? >????? >????? } >> ???? >????? > >> ???? >????? > Shall I go with that, or reverse the original patch? >> ???? >????? > >> ???? >????? > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of >> Hohensee, Paul" > behalf of hohensee at amazon.com> wrote: >> ???? >????? > >> ???? >????? >????? I'll take a look. >> ???? >????? > >> ???? >????? >????? On 9/18/19, 4:40 PM, "David Holmes" >> wrote: >> ???? >????? > >> ???? >????? >????????? Paul, >> ???? >????? > >> ???? >????? >????????? Unfortunately this patch has broken the >> vmTestbase/nsk/monitoring tests: >> ???? >????? > >> ???? >????? >????????? [2019-09-18T22:59:32,349Z] >> ???? >????? > >> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: >> ???? >????? >????????? error: ServerThreadMXBeanNew is not abstract >> and does not override >> ???? >????? >????????? abstract method >> getCurrentThreadAllocatedBytes() in ThreadMXBean >> ???? >????? > >> ???? >????? >????????? and possibly other issues as we are seeing >> hundreds of failures. >> ???? >????? > >> ???? >????? >????????? David >> ???? >????? > >> ???? >????? >????????? On 18/09/2019 8:50 am, David Holmes wrote: >> ???? >????? >????????? > On 18/09/2019 12:10 am, Hohensee, Paul wrote: >> ???? >????? >????????? >> Thanks, Serguei. :) >> ???? >????? >????????? >> >> ???? >????? >????????? >> David, are you ok with the patch? >> ???? >????? >????????? > >> ???? >????? >????????? > Yep, nothing further from me. >> ???? >????? >????????? > >> ???? >????? >????????? > David >> ???? >????? >????????? > >> ???? >????? >????????? >> Paul >> ???? >????? >????????? >> >> ???? >????? >????????? >> *From: *"serguei.spitsyn at oracle.com" >> >> ???? >????? >????????? >> *Date: *Tuesday, September 17, 2019 at 2:26 AM >> ???? >????? >????????? >> *To: *"Hohensee, Paul" >> , David Holmes >> ???? >????? >????????? >> , Mandy Chung >> >> ???? >????? >????????? >> *Cc: *OpenJDK Serviceability >> , >> ???? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >> >> ???? >????? >????????? >> *Subject: *Re: RFR (M): 8207266: >> ???? >????? >????????? >> ThreadMXBean::getThreadAllocatedBytes() can >> be quicker for self thread >> ???? >????? >????????? >> >> ???? >????? >????????? >> Hi Paul, >> ???? >????? >????????? >> >> ???? >????? >????????? >> Thank you for refactoring and fixing the test. >> ???? >????? >????????? >> It looks great now! >> ???? >????? >????????? >> >> ???? >????? >????????? >> Thanks, >> ???? >????? >????????? >> Serguei >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> On 9/15/19 02:52, Hohensee, Paul wrote: >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? Hi, Serguei, thanks for the review. New >> webrev at >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? I refactored the test?s main() method, >> and you?re correct, >> ???? >????? >????????? >> getThreadAllocatedBytes should be >> getCurrentThreadAllocatedBytes in >> ???? >????? >????????? >>???? that context: fixed. >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? Paul >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? *From: *"serguei.spitsyn at oracle.com" >> ???? >????? >????????? >> >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? *Organization: *Oracle Corporation >> ???? >????? >????????? >>???? *Date: *Friday, September 13, 2019 at >> 5:50 PM >> ???? >????? >????????? >>???? *To: *"Hohensee, Paul" >> >> ???? >????? >????????? >> , David Holmes >> >> ???? >????? >????????? >> , Mandy Chung >> ???? >????? >????????? >> >> >> ???? >????? >????????? >>???? *Cc: *OpenJDK Serviceability >> >> ???? >????? >????????? >> , >> ???? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? *Subject: *Re: RFR (M): 8207266: >> ???? >????? >????????? >> ThreadMXBean::getThreadAllocatedBytes() can >> be quicker for self >> ???? >????? >????????? >> thread >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? Hi Paul, >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? It looks pretty good in general. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? It would be nice to refactor the java >> main() method as it becomes >> ???? >????? >????????? >>???? too big. >> ???? >????? >????????? >>???? Two ways >> ofgetCurrentThreadAllocatedBytes() testing are good >> ???? >????? >????????? >> candidates >> ???? >????? >????????? >>???? to become separate methods. >> ???? >????? >????????? >> >> ???? >????? >????????? >>??????? 98???????? long size1 = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? Just wanted to double check if you >> wanted to invoke >> ???? >????? >????????? >>???? the getCurrentThreadAllocatedBytes() >> instead as it is >> ???? >????? >????????? >>???? a part of: >> ???? >????? >????????? >> >> ???? >????? >????????? >>??????? 85???????? // First way, >> getCurrentThreadAllocatedBytes >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? Thanks, >> ???? >????? >????????? >>???? Serguei >> ???? >????? >????????? >> >> ???? >????? >????????? >>???? On 9/13/19 12:11 PM, Hohensee, Paul wrote: >> ???? >????? >????????? >> >> ???? >????? >????????? >>???????? Hi David, thanks for your comments. >> New webrev in >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???????? Both the old and new versions of >> the code check that thread >> ???? >????? >????????? >> allocated memory is both supported and >> enabled. The existing version >> ???? >????? >????????? >> of getThreadAllocatedBytes(long []) calls >> ???? >????? >????????? >> verifyThreadAllocatedMemory(long []), which >> checks inline to make sure >> ???? >????? >????????? >> thread allocated memory is supported, then >> calls >> ???? >????? >????????? >> isThreadAllocatedMemoryEnabled() to verify >> that it's enabled. >> ???? >????? >????????? >> isThreadAllocatedMemoryEnabled() duplicates >> (!) the support check and >> ???? >????? >????????? >> returns the enabled flag. I removed the >> redundant check in the new >> ???? >????? >????????? >> version. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???????? You're of course correct about the >> back-to-back check. >> ???? >????? >????????? >> Application code can't know when the >> runtime will hijack a thread for >> ???? >????? >????????? >> its own purposes. I've removed the check. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???????? Paul >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>???????? On 9/13/19, 12:50 AM, "David >> Holmes" >> ???? >????? >????????? >> ? wrote: >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? Hi Paul, >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? On 13/09/2019 10:29 am, >> Hohensee, Paul wrote: >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Thanks for clarifying the >> review rules. Would someone >> ???? >????? >????????? >> from the >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > serviceability team please >> review? New webrev at >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? One aspect of the functional >> change needs clarification >> ???? >????? >????????? >> for me - and >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? apologies if this has been >> covered in the past. It seems >> ???? >????? >????????? >> to me that >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? currently we only check >> isThreadAllocatedMemorySupported >> ???? >????? >????????? >> for these >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? operations, but if I read >> things correctly the updated >> ???? >????? >????????? >> code additionally >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? checks >> isThreadAllocatedMemoryEnabled, which is a >> ???? >????? >????????? >> behaviour change not >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? mentioned in the CSR. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > I didn?t disturb the >> existing checks in the test, just >> ???? >????? >????????? >> added code to >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > check the result of >> getThreadAllocatedBytes(long) on a >> ???? >????? >????????? >> non-current >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > thread, plus the >> back-to-back no-allocation checks. The >> ???? >????? >????????? >> former wasn?t >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > needed before because >> getThreadAllocatedBytes(long) was >> ???? >????? >????????? >> just a wrapper >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > around >> getThreadAllocatedBytes(long []). This patch >> ???? >????? >????????? >> changes that, so I >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > added a separate test. The >> latter is supposed to fail >> ???? >????? >????????? >> if there?s object >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > allocation on calls to >> getCurrentThreadAllocatedBytes and >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> getThreadAllocatedBytes(long). I.e., a feature, not a >> ???? >????? >????????? >> bug, because >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > accumulation of transient >> small objects can be a >> ???? >????? >????????? >> performance problem. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Thanks to your review, I >> noticed that the back-to-back >> ???? >????? >????????? >> check on the >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > current thread was using >> getThreadAllocatedBytes(long) >> ???? >????? >????????? >> instead of >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> getCurrentThreadAllocatedBytes and fixed it. I also >> ???? >????? >????????? >> removed all >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > instances of ?TEST FAILED: ?. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? The back-to-back check is not >> valid in general. You don't >> ???? >????? >????????? >> know if the >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? first check might trigger some >> class loading on the >> ???? >????? >????????? >> return path after it >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? has obtained the first memory >> value. The check might also >> ???? >????? >????????? >> fail if using >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? JVMCI and some compilation >> related activity occurs in the >> ???? >????? >????????? >> current thread >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? on the second call. Also with >> the introduction of >> ???? >????? >????????? >> handshakes its >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? possible the current thread >> might hit a safepoint checks >> ???? >????? >????????? >> that results in >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? it executing a handshake >> operation that performs >> ???? >????? >????????? >> allocation. Potentially >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? there could be numerous >> non-deterministic actions that >> ???? >????? >????????? >> might occur >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? leading to unanticipated >> allocation. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? I understand what you want to >> test here, I just don't >> ???? >????? >????????? >> think it is >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? reliably doable. >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? Thanks, >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? David >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? ----- >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Paul >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > *From: *Mandy >> Chung >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > *Date: *Thursday, September >> 12, 2019 at 10:09 AM >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > *To: *"Hohensee, >> Paul" >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > *Cc: *OpenJDK >> ???? >????? >????????? >> >> Serviceability >> ???? >????? >????????? >> , >> ???? >????? >????????? >> >> ???? >????? >????????? >> >"hotspot-gc-dev at openjdk.java.net" >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > *Subject: *Re: RFR (M): >> 8207266: >> ???? >????? >????????? >> ThreadMXBean::getThreadAllocatedBytes() >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > can be quicker for self thread >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > On 9/3/19 12:38 PM, >> Hohensee, Paul wrote: >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Minor update in new >> ???? >????? >????????? >> >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > I only reviewed the library >> side implementation that >> ???? >????? >????????? >> looks good.? I >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > expect the serviceability >> team to review the test and >> ???? >????? >????????? >> hotspot change. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? >???? Need a confirmatory >> review to push this. If I >> ???? >????? >????????? >> understand the rules correctly, it doesn't >> need a Reviewer review >> ???? >????? >????????? >> since Mandy's already reviewed it, it just >> needs a Committer review. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > You need another reviewer to >> advice the following >> ???? >????? >????????? >> because I was not >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > close to the ThreadsList work. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2087 ThreadsListHandle tlh; >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2088 JavaThread* java_thread = >> ???? >????? >????????? >> >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2089 >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2090 if (java_thread != NULL) { >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2091 return >> java_thread->cooked_allocated_bytes(); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > 2092?? } >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > This looks right to me. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > -??????????????? >> "ThreadAllocatedMemory is expected to >> ???? >????? >????????? >> be disabled"); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > +??????????????? "TEST >> FAILED: ThreadAllocatedMemory is >> ???? >????? >????????? >> expected to be >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > disabled"); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Prepending "TEST FAILED" in >> exception message (in >> ???? >????? >????????? >> several places) >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > seems redundant since such >> RuntimeException is thrown >> ???? >????? >????????? >> and expected >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > a test failure. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > + // back-to-back calls >> shouldn't allocate any >> ???? >????? >????????? >> memory >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > + size = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > + size1 = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > + if (size1 != size) { >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Is there anything in the >> test can do to help guarantee >> ???? >????? >????????? >> this? I didn't >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > closely review this test.? >> The main thing I advice is >> ???? >????? >????????? >> to improve >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > the reliability of this >> test.? Put it in another way, >> ???? >????? >????????? >> we want to >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > ensure that this test change >> will pass all the time in >> ???? >????? >????????? >> various >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > test configuration. >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > Mandy >> ???? >????? >????????? >> >> ???? >????? >????????? >>????????????? > >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? >????????? >> >> ???? >????? > >> ???? >????? > >> ???? >????? > >> ???? >????? > >> ???? > >> ???? > >> ???? > >> > From hohensee at amazon.com Thu Sep 19 00:36:26 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:36:26 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <6139732f-a914-c59c-2119-517e480a279c@oracle.com> References: <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> Message-ID: <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> And I filed 8231211 for the same thing. :) Yes, please handle it, because it will go faster since I don't have access to a fast machine (just my laptop). Webrev here: http://cr.openjdk.java.net/~phh/8231211/webrev.00/ Thanks, ?On 9/18/19, 5:25 PM, "Daniel D. Daugherty" wrote: I created this sub-task for you: JDK-8231210 [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread https://bugs.openjdk.java.net/browse/JDK-8231210 If you would prefer, I can handle this backout for you. Dan On 9/18/19 8:21 PM, Hohensee, Paul wrote: > Never having done this before, is it > > hg backout -r > > ? Do I file a JBS issue for the reversion? Seems necessary. > > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" wrote: > > % hg backout > > is the usual way to do this... > > Dan > > > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > > Is there a tool that will generate a reversal patch? > > > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > > > Shall I go with that, or reverse the original patch? > > > > I'm a bit worried about what else might show up since the > > NSK monitoring tests were not run prior to this push. > > > > I vote for backing out the fix until proper testing has > > been done (and at least the one problem fixed...) > > > > Dan > > > > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > > > public default long getCurrentThreadAllocatedBytes() { > > > return -1; > > > } > > > > > > Shall I go with that, or reverse the original patch? > > > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > I'll take a look. > > > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > > > Paul, > > > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > > > [2019-09-18T22:59:32,349Z] > > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > > error: ServerThreadMXBeanNew is not abstract and does not override > > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > > > and possibly other issues as we are seeing hundreds of failures. > > > > > > David > > > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > > >> Thanks, Serguei. :) > > > >> > > > >> David, are you ok with the patch? > > > > > > > > Yep, nothing further from me. > > > > > > > > David > > > > > > > >> Paul > > > >> > > > >> *From: *"serguei.spitsyn at oracle.com" > > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > > >> *To: *"Hohensee, Paul" , David Holmes > > > >> , Mandy Chung > > > >> *Cc: *OpenJDK Serviceability , > > > >> "hotspot-gc-dev at openjdk.java.net" > > > >> *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > > >> > > > >> Hi Paul, > > > >> > > > >> Thank you for refactoring and fixing the test. > > > >> It looks great now! > > > >> > > > >> Thanks, > > > >> Serguei > > > >> > > > >> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > > >> > > > >> Hi, Serguei, thanks for the review. New webrev at > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > > >> > > > >> I refactored the test?s main() method, and you?re correct, > > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > > >> that context: fixed. > > > >> > > > >> Paul > > > >> > > > >> *From: *"serguei.spitsyn at oracle.com" > > > >> > > > >> > > > >> *Organization: *Oracle Corporation > > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > > >> *To: *"Hohensee, Paul" > > > >> , David Holmes > > > >> , Mandy Chung > > > >> > > > >> *Cc: *OpenJDK Serviceability > > > >> , > > > >> "hotspot-gc-dev at openjdk.java.net" > > > >> > > > >> > > > >> > > > >> *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > > >> thread > > > >> > > > >> Hi Paul, > > > >> > > > >> It looks pretty good in general. > > > >> > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > > >> > > > >> > > > >> It would be nice to refactor the java main() method as it becomes > > > >> too big. > > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > > >> candidates > > > >> to become separate methods. > > > >> > > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> Just wanted to double check if you wanted to invoke > > > >> the getCurrentThreadAllocatedBytes() instead as it is > > > >> a part of: > > > >> > > > >> 85 // First way, getCurrentThreadAllocatedBytes > > > >> > > > >> > > > >> Thanks, > > > >> Serguei > > > >> > > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > > >> > > > >> Hi David, thanks for your comments. New webrev in > > > >> > > > >> > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > >> > > > >> > > > >> Both the old and new versions of the code check that thread > > > >> allocated memory is both supported and enabled. The existing version > > > >> of getThreadAllocatedBytes(long []) calls > > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > > >> thread allocated memory is supported, then calls > > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > > >> returns the enabled flag. I removed the redundant check in the new > > > >> version. > > > >> > > > >> > > > >> You're of course correct about the back-to-back check. > > > >> Application code can't know when the runtime will hijack a thread for > > > >> its own purposes. I've removed the check. > > > >> > > > >> > > > >> Paul > > > >> > > > >> > > > >> On 9/13/19, 12:50 AM, "David Holmes" > > > >> wrote: > > > >> > > > >> > > > >> Hi Paul, > > > >> > > > >> > > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > > >> > > > >> > Thanks for clarifying the review rules. Would someone > > > >> from the > > > >> > > > >> > serviceability team please review? New webrev at > > > >> > > > >> > > > > >> > > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > >> > > > >> > > > >> One aspect of the functional change needs clarification > > > >> for me - and > > > >> > > > >> apologies if this has been covered in the past. It seems > > > >> to me that > > > >> > > > >> currently we only check isThreadAllocatedMemorySupported > > > >> for these > > > >> > > > >> operations, but if I read things correctly the updated > > > >> code additionally > > > >> > > > >> checks isThreadAllocatedMemoryEnabled, which is a > > > >> behaviour change not > > > >> > > > >> mentioned in the CSR. > > > >> > > > >> > > > >> > I didn?t disturb the existing checks in the test, just > > > >> added code to > > > >> > > > >> > check the result of getThreadAllocatedBytes(long) on a > > > >> non-current > > > >> > > > >> > thread, plus the back-to-back no-allocation checks. The > > > >> former wasn?t > > > >> > > > >> > needed before because getThreadAllocatedBytes(long) was > > > >> just a wrapper > > > >> > > > >> > around getThreadAllocatedBytes(long []). This patch > > > >> changes that, so I > > > >> > > > >> > added a separate test. The latter is supposed to fail > > > >> if there?s object > > > >> > > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > > >> > > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > > >> bug, because > > > >> > > > >> > accumulation of transient small objects can be a > > > >> performance problem. > > > >> > > > >> > Thanks to your review, I noticed that the back-to-back > > > >> check on the > > > >> > > > >> > current thread was using getThreadAllocatedBytes(long) > > > >> instead of > > > >> > > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > > >> removed all > > > >> > > > >> > instances of ?TEST FAILED: ?. > > > >> > > > >> > > > >> The back-to-back check is not valid in general. You don't > > > >> know if the > > > >> > > > >> first check might trigger some class loading on the > > > >> return path after it > > > >> > > > >> has obtained the first memory value. The check might also > > > >> fail if using > > > >> > > > >> JVMCI and some compilation related activity occurs in the > > > >> current thread > > > >> > > > >> on the second call. Also with the introduction of > > > >> handshakes its > > > >> > > > >> possible the current thread might hit a safepoint checks > > > >> that results in > > > >> > > > >> it executing a handshake operation that performs > > > >> allocation. Potentially > > > >> > > > >> there could be numerous non-deterministic actions that > > > >> might occur > > > >> > > > >> leading to unanticipated allocation. > > > >> > > > >> > > > >> I understand what you want to test here, I just don't > > > >> think it is > > > >> > > > >> reliably doable. > > > >> > > > >> > > > >> Thanks, > > > >> > > > >> David > > > >> > > > >> ----- > > > >> > > > >> > > > >> > > > > >> > > > >> > Paul > > > >> > > > >> > > > > >> > > > >> > *From: *Mandy Chung > > > >> > > > >> > > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > > >> > > > >> > *To: *"Hohensee, Paul" > > > >> > > > >> > > > >> > *Cc: *OpenJDK > > > >> Serviceability > > > >> , > > > >> > > > >> >"hotspot-gc-dev at openjdk.java.net" > > > >> > > > >> > > > >> > > > >> > > > >> > *Subject: *Re: RFR (M): 8207266: > > > >> ThreadMXBean::getThreadAllocatedBytes() > > > >> > > > >> > can be quicker for self thread > > > >> > > > >> > > > > >> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > >> > > > >> > > > > >> > > > >> > Minor update in new > > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > I only reviewed the library side implementation that > > > >> looks good. I > > > >> > > > >> > expect the serviceability team to review the test and > > > >> hotspot change. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > Need a confirmatory review to push this. If I > > > >> understand the rules correctly, it doesn't need a Reviewer review > > > >> since Mandy's already reviewed it, it just needs a Committer review. > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > You need another reviewer to advice the following > > > >> because I was not > > > >> > > > >> > close to the ThreadsList work. > > > >> > > > >> > > > > >> > > > >> > 2087 ThreadsListHandle tlh; > > > >> > > > >> > > > > >> > > > >> > 2088 JavaThread* java_thread = > > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > >> > > > >> > > > > >> > > > >> > 2089 > > > >> > > > >> > > > > >> > > > >> > 2090 if (java_thread != NULL) { > > > >> > > > >> > > > > >> > > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > > >> > > > >> > > > > >> > > > >> > 2092 } > > > >> > > > >> > > > > >> > > > >> > This looks right to me. > > > >> > > > >> > > > > >> > > > >> > > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > >> > > > >> > > > > >> > > > >> > - "ThreadAllocatedMemory is expected to > > > >> be disabled"); > > > >> > > > >> > > > > >> > > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > > >> expected to be > > > >> > > > >> > disabled"); > > > >> > > > >> > > > > >> > > > >> > Prepending "TEST FAILED" in exception message (in > > > >> several places) > > > >> > > > >> > > > > >> > > > >> > seems redundant since such RuntimeException is thrown > > > >> and expected > > > >> > > > >> > > > > >> > > > >> > a test failure. > > > >> > > > >> > > > > >> > > > >> > + // back-to-back calls shouldn't allocate any > > > >> memory > > > >> > > > >> > > > > >> > > > >> > + size = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> > > > > >> > > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > > >> > > > >> > > > > >> > > > >> > + if (size1 != size) { > > > >> > > > >> > > > > >> > > > >> > Is there anything in the test can do to help guarantee > > > >> this? I didn't > > > >> > > > >> > > > > >> > > > >> > closely review this test. The main thing I advice is > > > >> to improve > > > >> > > > >> > > > > >> > > > >> > the reliability of this test. Put it in another way, > > > >> we want to > > > >> > > > >> > > > > >> > > > >> > ensure that this test change will pass all the time in > > > >> various > > > >> > > > >> > > > > >> > > > >> > test configuration. > > > >> > > > >> > > > > >> > > > >> > Mandy > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > From hohensee at amazon.com Thu Sep 19 00:38:26 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:38:26 +0000 Subject: RFA Backport CSR to 11u: 8231194: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <4A7D9E4F-0D48-4C31-B092-55E03A45E0D1@amazon.com> References: <4A7D9E4F-0D48-4C31-B092-55E03A45E0D1@amazon.com> Message-ID: <64BC7D9E-E79D-43B1-98F0-BE75B84EF7A9@amazon.com> Please belay this, there?s a problem with the original patch. Thanks, From: "Hohensee, Paul" Date: Wednesday, September 18, 2019 at 10:56 AM To: OpenJDK Serviceability Cc: "jdk-updates-dev at openjdk.java.net" Subject: RFA Backport CSR to 11u: 8231194: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Please review/approve an 11u CSR backport request. https://bugs.openjdk.java.net/browse/JDK-8231194 Original issue: https://bugs.openjdk.java.net/browse/JDK-8207266 Original CSR: https://bugs.openjdk.java.net/browse/JDK-8230311 11u backport issue: https://bugs.openjdk.java.net/browse/JDK-8231193 Email threads: https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-August/029011.html https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029033.html Thanks, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Thu Sep 19 00:43:25 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 18 Sep 2019 17:43:25 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: Ok. It was a bit unclear to me why the author went with Runtime.exec() in the first place. I'll try ProcessTools. That will probably however, hide 8230872. I might need to write another test for it. Chris On 9/18/19 4:29 PM, Alex Menkov wrote: > You can use jdk.test.lib classes to simplify the things. > Something like > > ProcessBuilder pb = new > ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); > Process p = ProcessTools.startProcess("JShell", pb, > ??? s -> {? // warm-up predicate > ??????? return s.contains(">jshell"); > ??? }); > > --alex > > On 09/18/2019 15:44, Chris Plummer wrote: >> Is there an easy way of doing this? Currently the jshell process is >> just spawned using Runtime.exec(). >> >> Chris >> >> On 9/18/19 3:01 PM, Alex Menkov wrote: >>> Hi Chris, >>> >>> Did you think about waiting for jshell prompt ("jshell>") before run >>> "jhsdb jmap" command instead of delay or re-tries? >>> >>> --alex >>> >>> On 09/18/2019 14:11, Chris Plummer wrote: >>>> Hello, >>>> >>>> Please review the following changes: >>>> >>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>> >>>> There are actually numerous ways that JShellHeapDumpTest.java >>>> fails. One is a test bug, being addressed here, and the rest all >>>> seem to be SA bugs. Those are now being covered by JDK-8230872. All >>>> the issues seem to stem from the fact that the test spawns a jshell >>>> process, and then immediately does a "jhsdb jmap" on the process >>>> before jshell has fully started up. >>>> >>>> The test bug happens when the jmap succeeds, but jshell has not yet >>>> entered the main java thread. Thus the search for >>>> "JShellToolProvider" in the output fails. It expects >>>> "JShellToolProvider" to be in the output because it is part of a >>>> method name in the main thread, and the test dump all the thread >>>> stacks contained in the jmap generated hprof file. When the test >>>> fails in this way, you can see the stack dump in the output, but >>>> the main thread is missing. >>>> >>>> There's a couple of ways to fix this. One is to just add a delay >>>> (10s seems to be more than enough), and the other is to retry the >>>> "jhsdb jmap" command until the stack contains the >>>> JShellToolProvider symbol. I chose the later because doing a 10s >>>> delay masks the SA issues that are now covered by JDK-8230872. In a >>>> way the 10s delay is a better fix, because it makes this test pass >>>> every time, but I did not like that it also hid real SA problems in >>>> JDK-8230872. My plan for now is to do this retry fix, and then if >>>> there are too many failures due to JDK-8230872, then also add a 10s >>>> delay, with the intention of removing it once JDK-8230872 if fixed. >>>> From what I can see, JDK-8230872 failures happen on about 1% of the >>>> runs. >>>> >>>> I made a few of other changes. One was to no longer redirect stderr >>>> from the jmap process as was done from the following: >>>> >>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>> >>>> This causes the output not to appear in the OutputAnalyzer output, >>>> resulting in the following not working: >>>> >>>> ???????????? output.shouldNotContain("null"); >>>> >>>> Also I added code to dump the output of the jshell process so you >>>> can see if the jshell prompt was ever generated. >>>> >>>> thanks, >>>> >>>> Chris >>>> >> >> From david.holmes at oracle.com Thu Sep 19 00:42:53 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 10:42:53 +1000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi Richard, On 7/09/2019 12:24 am, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape analysis those references escape and > optimizations like scalar replacement become invalid. What bothers me about this is that it seems like escape analysis is doing something wrong in this case. If the object is thread-local but is being synchronized upon then either: a) the synchronization is elided and so the object will not appear in the set of owned monitors; or b) the fact synchronization occurs renders the object ineligible to be considered thread-local, and so there is no problem with it appearing in the set of owned monitors I think there is a bigger design philosophy issue here about the interaction of escape analysis and debugging/management frameworks in general. I'd like to see a very clear description on exactly how they should interact. Cheers, David > The runtime currently cannot cope with objects escaping through JVMTI (try included > tests). Therefore escape analysis should be disabled if an agent requests the capabilities > can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > From daniel.daugherty at oracle.com Thu Sep 19 00:44:00 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:44:00 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> References: <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> Message-ID: <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> For some reason, the backout that I did does not match the backout that you did so I'm trying to figure that out. Dan On 9/18/19 8:36 PM, Hohensee, Paul wrote: > And I filed 8231211 for the same thing. :) > > Yes, please handle it, because it will go faster since I don't have access to a fast machine (just my laptop). > > Webrev here: > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > Thanks, > > ?On 9/18/19, 5:25 PM, "Daniel D. Daugherty" wrote: > > I created this sub-task for you: > > JDK-8231210 [BACKOUT] JDK-8207266 > ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > https://bugs.openjdk.java.net/browse/JDK-8231210 > > If you would prefer, I can handle this backout for you. > > Dan > > > On 9/18/19 8:21 PM, Hohensee, Paul wrote: > > Never having done this before, is it > > > > hg backout -r > > > > ? Do I file a JBS issue for the reversion? Seems necessary. > > > > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" wrote: > > > > % hg backout > > > > is the usual way to do this... > > > > Dan > > > > > > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > > > Is there a tool that will generate a reversal patch? > > > > > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > > > > > Shall I go with that, or reverse the original patch? > > > > > > I'm a bit worried about what else might show up since the > > > NSK monitoring tests were not run prior to this push. > > > > > > I vote for backing out the fix until proper testing has > > > been done (and at least the one problem fixed...) > > > > > > Dan > > > > > > > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > > > > > public default long getCurrentThreadAllocatedBytes() { > > > > return -1; > > > > } > > > > > > > > Shall I go with that, or reverse the original patch? > > > > > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > I'll take a look. > > > > > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > > > > > Paul, > > > > > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > > > > > [2019-09-18T22:59:32,349Z] > > > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > > > error: ServerThreadMXBeanNew is not abstract and does not override > > > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > > > > > and possibly other issues as we are seeing hundreds of failures. > > > > > > > > David > > > > > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > > > >> Thanks, Serguei. :) > > > > >> > > > > >> David, are you ok with the patch? > > > > > > > > > > Yep, nothing further from me. > > > > > > > > > > David > > > > > > > > > >> Paul > > > > >> > > > > >> *From: *"serguei.spitsyn at oracle.com" > > > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > > > >> *To: *"Hohensee, Paul" , David Holmes > > > > >> , Mandy Chung > > > > >> *Cc: *OpenJDK Serviceability , > > > > >> "hotspot-gc-dev at openjdk.java.net" > > > > >> *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> Thank you for refactoring and fixing the test. > > > > >> It looks great now! > > > > >> > > > > >> Thanks, > > > > >> Serguei > > > > >> > > > > >> > > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > > > >> > > > > >> Hi, Serguei, thanks for the review. New webrev at > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > > > >> > > > > >> I refactored the test?s main() method, and you?re correct, > > > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > > > >> that context: fixed. > > > > >> > > > > >> Paul > > > > >> > > > > >> *From: *"serguei.spitsyn at oracle.com" > > > > >> > > > > >> > > > > >> *Organization: *Oracle Corporation > > > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > > > >> *To: *"Hohensee, Paul" > > > > >> , David Holmes > > > > >> , Mandy Chung > > > > >> > > > > >> *Cc: *OpenJDK Serviceability > > > > >> , > > > > >> "hotspot-gc-dev at openjdk.java.net" > > > > >> > > > > >> > > > > >> > > > > >> *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > > > >> thread > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> It looks pretty good in general. > > > > >> > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > > > >> > > > > >> > > > > >> It would be nice to refactor the java main() method as it becomes > > > > >> too big. > > > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > > > >> candidates > > > > >> to become separate methods. > > > > >> > > > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> Just wanted to double check if you wanted to invoke > > > > >> the getCurrentThreadAllocatedBytes() instead as it is > > > > >> a part of: > > > > >> > > > > >> 85 // First way, getCurrentThreadAllocatedBytes > > > > >> > > > > >> > > > > >> Thanks, > > > > >> Serguei > > > > >> > > > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > > > >> > > > > >> Hi David, thanks for your comments. New webrev in > > > > >> > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > > >> > > > > >> > > > > >> Both the old and new versions of the code check that thread > > > > >> allocated memory is both supported and enabled. The existing version > > > > >> of getThreadAllocatedBytes(long []) calls > > > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > > > >> thread allocated memory is supported, then calls > > > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > > > >> returns the enabled flag. I removed the redundant check in the new > > > > >> version. > > > > >> > > > > >> > > > > >> You're of course correct about the back-to-back check. > > > > >> Application code can't know when the runtime will hijack a thread for > > > > >> its own purposes. I've removed the check. > > > > >> > > > > >> > > > > >> Paul > > > > >> > > > > >> > > > > >> On 9/13/19, 12:50 AM, "David Holmes" > > > > >> wrote: > > > > >> > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> > > > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > > > >> > > > > >> > Thanks for clarifying the review rules. Would someone > > > > >> from the > > > > >> > > > > >> > serviceability team please review? New webrev at > > > > >> > > > > >> > > > > > >> > > > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > > >> > > > > >> > > > > >> One aspect of the functional change needs clarification > > > > >> for me - and > > > > >> > > > > >> apologies if this has been covered in the past. It seems > > > > >> to me that > > > > >> > > > > >> currently we only check isThreadAllocatedMemorySupported > > > > >> for these > > > > >> > > > > >> operations, but if I read things correctly the updated > > > > >> code additionally > > > > >> > > > > >> checks isThreadAllocatedMemoryEnabled, which is a > > > > >> behaviour change not > > > > >> > > > > >> mentioned in the CSR. > > > > >> > > > > >> > > > > >> > I didn?t disturb the existing checks in the test, just > > > > >> added code to > > > > >> > > > > >> > check the result of getThreadAllocatedBytes(long) on a > > > > >> non-current > > > > >> > > > > >> > thread, plus the back-to-back no-allocation checks. The > > > > >> former wasn?t > > > > >> > > > > >> > needed before because getThreadAllocatedBytes(long) was > > > > >> just a wrapper > > > > >> > > > > >> > around getThreadAllocatedBytes(long []). This patch > > > > >> changes that, so I > > > > >> > > > > >> > added a separate test. The latter is supposed to fail > > > > >> if there?s object > > > > >> > > > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > > > >> > > > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > > > >> bug, because > > > > >> > > > > >> > accumulation of transient small objects can be a > > > > >> performance problem. > > > > >> > > > > >> > Thanks to your review, I noticed that the back-to-back > > > > >> check on the > > > > >> > > > > >> > current thread was using getThreadAllocatedBytes(long) > > > > >> instead of > > > > >> > > > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > > > >> removed all > > > > >> > > > > >> > instances of ?TEST FAILED: ?. > > > > >> > > > > >> > > > > >> The back-to-back check is not valid in general. You don't > > > > >> know if the > > > > >> > > > > >> first check might trigger some class loading on the > > > > >> return path after it > > > > >> > > > > >> has obtained the first memory value. The check might also > > > > >> fail if using > > > > >> > > > > >> JVMCI and some compilation related activity occurs in the > > > > >> current thread > > > > >> > > > > >> on the second call. Also with the introduction of > > > > >> handshakes its > > > > >> > > > > >> possible the current thread might hit a safepoint checks > > > > >> that results in > > > > >> > > > > >> it executing a handshake operation that performs > > > > >> allocation. Potentially > > > > >> > > > > >> there could be numerous non-deterministic actions that > > > > >> might occur > > > > >> > > > > >> leading to unanticipated allocation. > > > > >> > > > > >> > > > > >> I understand what you want to test here, I just don't > > > > >> think it is > > > > >> > > > > >> reliably doable. > > > > >> > > > > >> > > > > >> Thanks, > > > > >> > > > > >> David > > > > >> > > > > >> ----- > > > > >> > > > > >> > > > > >> > > > > > >> > > > > >> > Paul > > > > >> > > > > >> > > > > > >> > > > > >> > *From: *Mandy Chung > > > > >> > > > > >> > > > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > > > >> > > > > >> > *To: *"Hohensee, Paul" > > > > >> > > > > >> > > > > >> > *Cc: *OpenJDK > > > > >> Serviceability > > > > >> , > > > > >> > > > > >> >"hotspot-gc-dev at openjdk.java.net" > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() > > > > >> > > > > >> > can be quicker for self thread > > > > >> > > > > >> > > > > > >> > > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > > >> > > > > >> > > > > > >> > > > > >> > Minor update in new > > > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > I only reviewed the library side implementation that > > > > >> looks good. I > > > > >> > > > > >> > expect the serviceability team to review the test and > > > > >> hotspot change. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > Need a confirmatory review to push this. If I > > > > >> understand the rules correctly, it doesn't need a Reviewer review > > > > >> since Mandy's already reviewed it, it just needs a Committer review. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > You need another reviewer to advice the following > > > > >> because I was not > > > > >> > > > > >> > close to the ThreadsList work. > > > > >> > > > > >> > > > > > >> > > > > >> > 2087 ThreadsListHandle tlh; > > > > >> > > > > >> > > > > > >> > > > > >> > 2088 JavaThread* java_thread = > > > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > > >> > > > > >> > > > > > >> > > > > >> > 2089 > > > > >> > > > > >> > > > > > >> > > > > >> > 2090 if (java_thread != NULL) { > > > > >> > > > > >> > > > > > >> > > > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > > > >> > > > > >> > > > > > >> > > > > >> > 2092 } > > > > >> > > > > >> > > > > > >> > > > > >> > This looks right to me. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > > >> > > > > >> > > > > > >> > > > > >> > - "ThreadAllocatedMemory is expected to > > > > >> be disabled"); > > > > >> > > > > >> > > > > > >> > > > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > > > >> expected to be > > > > >> > > > > >> > disabled"); > > > > >> > > > > >> > > > > > >> > > > > >> > Prepending "TEST FAILED" in exception message (in > > > > >> several places) > > > > >> > > > > >> > > > > > >> > > > > >> > seems redundant since such RuntimeException is thrown > > > > >> and expected > > > > >> > > > > >> > > > > > >> > > > > >> > a test failure. > > > > >> > > > > >> > > > > > >> > > > > >> > + // back-to-back calls shouldn't allocate any > > > > >> memory > > > > >> > > > > >> > > > > > >> > > > > >> > + size = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> > > > > > >> > > > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> > > > > > >> > > > > >> > + if (size1 != size) { > > > > >> > > > > >> > > > > > >> > > > > >> > Is there anything in the test can do to help guarantee > > > > >> this? I didn't > > > > >> > > > > >> > > > > > >> > > > > >> > closely review this test. The main thing I advice is > > > > >> to improve > > > > >> > > > > >> > > > > > >> > > > > >> > the reliability of this test. Put it in another way, > > > > >> we want to > > > > >> > > > > >> > > > > > >> > > > > >> > ensure that this test change will pass all the time in > > > > >> various > > > > >> > > > > >> > > > > > >> > > > > >> > test configuration. > > > > >> > > > > >> > > > > > >> > > > > >> > Mandy > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniel.daugherty at oracle.com Thu Sep 19 00:53:42 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 20:53:42 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> References: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> Message-ID: Looks like the issue is different versions of 'hg' in use. When I import Paul's patch from his webrev using my 'hg' and then export it again, it matches my version of the backout. I have done a mechanical verification that the backout is an exact reversal for Paul's original changeset. I'm planning to push the changeset with the following info: 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Reviewed-by: phh, dholmes Everyone good with this? Dan On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: > For some reason, the backout that I did does not match the backout > that you did so I'm trying to figure that out. > > Dan > > > > On 9/18/19 8:36 PM, Hohensee, Paul wrote: >> And I filed 8231211 for the same thing. :) >> >> Yes, please handle it, because it will go faster since I don't have >> access to a fast machine (just my laptop). >> >> Webrev here: >> >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> >> Thanks, >> >> ?On 9/18/19, 5:25 PM, "Daniel D. Daugherty" >> wrote: >> >> ???? I created this sub-task for you: >> ???? ???? JDK-8231210 [BACKOUT] JDK-8207266 >> ???? ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> ???? https://bugs.openjdk.java.net/browse/JDK-8231210 >> ???? ???? If you would prefer, I can handle this backout for you. >> ???? ???? Dan >> ???? ???? ???? On 9/18/19 8:21 PM, Hohensee, Paul wrote: >> ???? > Never having done this before, is it >> ???? > >> ???? > hg backout -r >> ???? > >> ???? > ? Do I file a JBS issue for the reversion? Seems necessary. >> ???? > >> ???? > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" >> wrote: >> ???? > >> ???? >????? % hg backout >> ???? > >> ???? >????? is the usual way to do this... >> ???? > >> ???? >????? Dan >> ???? > >> ???? > >> ???? >????? On 9/18/19 8:17 PM, Hohensee, Paul wrote: >> ???? >????? > Is there a tool that will generate a reversal patch? >> ???? >????? > >> ???? >????? > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" >> wrote: >> ???? >????? > >> ???? >????? >?????? > Shall I go with that, or reverse the original >> patch? >> ???? >????? > >> ???? >????? >????? I'm a bit worried about what else might show up >> since the >> ???? >????? >????? NSK monitoring tests were not run prior to this push. >> ???? >????? > >> ???? >????? >????? I vote for backing out the fix until proper >> testing has >> ???? >????? >????? been done (and at least the one problem fixed...) >> ???? >????? > >> ???? >????? >????? Dan >> ???? >????? > >> ???? >????? > >> ???? >????? >????? On 9/18/19 8:00 PM, Hohensee, Paul wrote: >> ???? >????? >????? > They all implement >> com.sun.management.ThreadMXBean, so adding a >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give >> it a default implementation, vis >> ???? >????? >????? > >> ???? >????? >????? >????? public default long >> getCurrentThreadAllocatedBytes() { >> ???? >????? >????? >????????? return -1; >> ???? >????? >????? >????? } >> ???? >????? >????? > >> ???? >????? >????? > Shall I go with that, or reverse the original >> patch? >> ???? >????? >????? > >> ???? >????? >????? > On 9/18/19, 4:48 PM, "serviceability-dev on >> behalf of Hohensee, Paul" >> > hohensee at amazon.com> wrote: >> ???? >????? >????? > >> ???? >????? >????? >????? I'll take a look. >> ???? >????? >????? > >> ???? >????? >????? >????? On 9/18/19, 4:40 PM, "David Holmes" >> wrote: >> ???? >????? >????? > >> ???? >????? >????? >????????? Paul, >> ???? >????? >????? > >> ???? >????? >????? >????????? Unfortunately this patch has broken the >> vmTestbase/nsk/monitoring tests: >> ???? >????? >????? > >> ???? >????? >????? > [2019-09-18T22:59:32,349Z] >> ???? >????? >????? > >> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: >> ???? >????? >????? >????????? error: ServerThreadMXBeanNew is not >> abstract and does not override >> ???? >????? >????? >????????? abstract method >> getCurrentThreadAllocatedBytes() in ThreadMXBean >> ???? >????? >????? > >> ???? >????? >????? >????????? and possibly other issues as we are >> seeing hundreds of failures. >> ???? >????? >????? > >> ???? >????? >????? >????????? David >> ???? >????? >????? > >> ???? >????? >????? >????????? On 18/09/2019 8:50 am, David Holmes wrote: >> ???? >????? >????? >????????? > On 18/09/2019 12:10 am, Hohensee, >> Paul wrote: >> ???? >????? >????? >????????? >> Thanks, Serguei. :) >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> David, are you ok with the patch? >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > Yep, nothing further from me. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > David >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? >> Paul >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> *From: *"serguei.spitsyn at oracle.com" >> >> ???? >????? >????? >????????? >> *Date: *Tuesday, September 17, 2019 >> at 2:26 AM >> ???? >????? >????? >????????? >> *To: *"Hohensee, Paul" >> , David Holmes >> ???? >????? >????? >????????? >> , Mandy >> Chung >> ???? >????? >????? >????????? >> *Cc: *OpenJDK Serviceability >> , >> ???? >????? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >> >> ???? >????? >????? >????????? >> *Subject: *Re: RFR (M): 8207266: >> ???? >????? >????? >????????? >> >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> Hi Paul, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> Thank you for refactoring and fixing >> the test. >> ???? >????? >????? >????????? >> It looks great now! >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> Thanks, >> ???? >????? >????? >????????? >> Serguei >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> On 9/15/19 02:52, Hohensee, Paul wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? Hi, Serguei, thanks for the >> review. New webrev at >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? I refactored the test?s main() >> method, and you?re correct, >> ???? >????? >????? >????????? >> getThreadAllocatedBytes should be >> getCurrentThreadAllocatedBytes in >> ???? >????? >????? >????????? >>???? that context: fixed. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? Paul >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? *From: >> *"serguei.spitsyn at oracle.com" >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> *Organization: *Oracle Corporation >> ???? >????? >????? >????????? >>???? *Date: *Friday, September 13, >> 2019 at 5:50 PM >> ???? >????? >????? >????????? >>???? *To: *"Hohensee, Paul" >> >> ???? >????? >????? >????????? >> , David >> Holmes >> ???? >????? >????? >????????? >> , >> Mandy Chung >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >>???? *Cc: *OpenJDK Serviceability >> >> ???? >????? >????? >????????? >> >> , >> ???? >????? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >>???? *Subject: *Re: RFR (M): 8207266: >> ???? >????? >????? >????????? >> >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> ???? >????? >????? >????????? >> thread >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? Hi Paul, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? It looks pretty good in general. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? It would be nice to refactor the >> java main() method as it becomes >> ???? >????? >????? >????????? >>???? too big. >> ???? >????? >????? >????????? >>???? Two ways >> ofgetCurrentThreadAllocatedBytes() testing are good >> ???? >????? >????? >????????? >> candidates >> ???? >????? >????? >????????? >>???? to become separate methods. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> 98???????? long size1 = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? Just wanted to double check if >> you wanted to invoke >> ???? >????? >????? >????????? >>???? the >> getCurrentThreadAllocatedBytes() instead as it is >> ???? >????? >????? >????????? >>???? a part of: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> 85???????? // First way, >> getCurrentThreadAllocatedBytes >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? Thanks, >> ???? >????? >????? >????????? >>???? Serguei >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???? On 9/13/19 12:11 PM, Hohensee, >> Paul wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???????? Hi David, thanks for your >> comments. New webrev in >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???????? Both the old and new >> versions of the code check that thread >> ???? >????? >????? >????????? >> allocated memory is both supported >> and enabled. The existing version >> ???? >????? >????? >????????? >> of getThreadAllocatedBytes(long []) >> calls >> ???? >????? >????? >????????? >> verifyThreadAllocatedMemory(long >> []), which checks inline to make sure >> ???? >????? >????? >????????? >> thread allocated memory is >> supported, then calls >> ???? >????? >????? >????????? >> isThreadAllocatedMemoryEnabled() to >> verify that it's enabled. >> ???? >????? >????? >????????? >> isThreadAllocatedMemoryEnabled() >> duplicates (!) the support check and >> ???? >????? >????? >????????? >> returns the enabled flag. I removed >> the redundant check in the new >> ???? >????? >????? >????????? >> version. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???????? You're of course correct >> about the back-to-back check. >> ???? >????? >????? >????????? >> Application code can't know when the >> runtime will hijack a thread for >> ???? >????? >????? >????????? >> its own purposes. I've removed the >> check. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???????? Paul >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>???????? On 9/13/19, 12:50 AM, "David >> Holmes" >> ???? >????? >????? >????????? >> ? >> wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? Hi Paul, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? On 13/09/2019 10:29 am, >> Hohensee, Paul wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Thanks for clarifying the review >> rules. Would someone >> ???? >????? >????? >????????? >> from the >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > serviceability team please review? >> New webrev at >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? One aspect of the >> functional change needs clarification >> ???? >????? >????? >????????? >> for me - and >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> apologies if this has been covered >> in the past. It seems >> ???? >????? >????? >????????? >> to me that >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> currently we only check >> isThreadAllocatedMemorySupported >> ???? >????? >????? >????????? >> for these >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> operations, but if I read things >> correctly the updated >> ???? >????? >????? >????????? >> code additionally >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> checks >> isThreadAllocatedMemoryEnabled, which is a >> ???? >????? >????? >????????? >> behaviour change not >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> mentioned in the CSR. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > I didn?t disturb the existing >> checks in the test, just >> ???? >????? >????? >????????? >> added code to >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > check the result of >> getThreadAllocatedBytes(long) on a >> ???? >????? >????? >????????? >> non-current >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > thread, plus the back-to-back >> no-allocation checks. The >> ???? >????? >????? >????????? >> former wasn?t >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > needed before because >> getThreadAllocatedBytes(long) was >> ???? >????? >????? >????????? >> just a wrapper >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > around >> getThreadAllocatedBytes(long []). This patch >> ???? >????? >????? >????????? >> changes that, so I >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > added a separate test. The latter >> is supposed to fail >> ???? >????? >????? >????????? >> if there?s object >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > allocation on calls to >> getCurrentThreadAllocatedBytes and >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > getThreadAllocatedBytes(long). >> I.e., a feature, not a >> ???? >????? >????? >????????? >> bug, because >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > accumulation of transient small >> objects can be a >> ???? >????? >????? >????????? >> performance problem. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Thanks to your review, I noticed >> that the back-to-back >> ???? >????? >????? >????????? >> check on the >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > current thread was using >> getThreadAllocatedBytes(long) >> ???? >????? >????? >????????? >> instead of >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > getCurrentThreadAllocatedBytes and >> fixed it. I also >> ???? >????? >????? >????????? >> removed all >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > instances of ?TEST FAILED: ?. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? The back-to-back check >> is not valid in general. You don't >> ???? >????? >????? >????????? >> know if the >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> first check might trigger some class >> loading on the >> ???? >????? >????? >????????? >> return path after it >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? has obtained the first >> memory value. The check might also >> ???? >????? >????? >????????? >> fail if using >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> JVMCI and some compilation related >> activity occurs in the >> ???? >????? >????? >????????? >> current thread >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? on the second call. >> Also with the introduction of >> ???? >????? >????? >????????? >> handshakes its >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> possible the current thread might >> hit a safepoint checks >> ???? >????? >????? >????????? >> that results in >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? it executing a >> handshake operation that performs >> ???? >????? >????? >????????? >> allocation. Potentially >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> there could be numerous >> non-deterministic actions that >> ???? >????? >????? >????????? >> might occur >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> leading to unanticipated allocation. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????????????? I understand what you >> want to test here, I just don't >> ???? >????? >????? >????????? >> think it is >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> reliably doable. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> Thanks, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> David >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> ----- >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Paul >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > *From: *Mandy >> Chung >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > *Date: *Thursday, September 12, >> 2019 at 10:09 AM >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > *To: *"Hohensee, >> Paul" >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > *Cc: *OpenJDK >> ???? >????? >????? >????????? >> >> Serviceability >> ???? >????? >????? >????????? >> >> , >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >"hotspot-gc-dev at openjdk.java.net" >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > *Subject: *Re: RFR (M): 8207266: >> ???? >????? >????? >????????? >> ThreadMXBean::getThreadAllocatedBytes() >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > can be quicker for self thread >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > On 9/3/19 12:38 PM, Hohensee, Paul >> wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >???? Minor update in new >> ???? >????? >????? >????????? >> >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > I only reviewed the library side >> implementation that >> ???? >????? >????? >????????? >> looks good.? I >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > expect the serviceability team to >> review the test and >> ???? >????? >????? >????????? >> hotspot change. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >???? Need a confirmatory review to >> push this. If I >> ???? >????? >????? >????????? >> understand the rules correctly, it >> doesn't need a Reviewer review >> ???? >????? >????? >????????? >> since Mandy's already reviewed it, >> it just needs a Committer review. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > You need another reviewer to >> advice the following >> ???? >????? >????? >????????? >> because I was not >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > close to the ThreadsList work. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2087?? ThreadsListHandle tlh; >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2088?? JavaThread* java_thread = >> ???? >????? >????? >????????? >> >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2089 >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2090?? if (java_thread != NULL) { >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2091???? return >> java_thread->cooked_allocated_bytes(); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > 2092?? } >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > This looks right to me. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > -??????????????? >> "ThreadAllocatedMemory is expected to >> ???? >????? >????? >????????? >> be disabled"); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > +??????????????? "TEST FAILED: >> ThreadAllocatedMemory is >> ???? >????? >????? >????????? >> expected to be >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > disabled"); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Prepending "TEST FAILED" in >> exception message (in >> ???? >????? >????? >????????? >> several places) >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > seems redundant since such >> RuntimeException is thrown >> ???? >????? >????? >????????? >> and expected >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > a test failure. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > +??????? // back-to-back calls >> shouldn't allocate any >> ???? >????? >????? >????????? >> memory >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > +??????? size = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > +??????? size1 = >> mbean.getThreadAllocatedBytes(id); >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > +??????? if (size1 != size) { >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Is there anything in the test can >> do to help guarantee >> ???? >????? >????? >????????? >> this? I didn't >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > closely review this test.? The >> main thing I advice is >> ???? >????? >????? >????????? >> to improve >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > the reliability of this test.? Put >> it in another way, >> ???? >????? >????? >????????? >> we want to >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > ensure that this test change will >> pass all the time in >> ???? >????? >????? >????????? >> various >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > test configuration. >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > Mandy >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> > >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? > >> ???? >????? > >> ???? >????? > >> ???? > >> ???? > >> ???? > >> > From hohensee at amazon.com Thu Sep 19 00:54:54 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:54:54 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> References: <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> Message-ID: <5B1CED54-3764-4152-8FDB-C51978A6225C@amazon.com> I used "hg backout -r 56298". ?On 9/18/19, 5:46 PM, "Daniel D. Daugherty" wrote: For some reason, the backout that I did does not match the backout that you did so I'm trying to figure that out. Dan On 9/18/19 8:36 PM, Hohensee, Paul wrote: > And I filed 8231211 for the same thing. :) > > Yes, please handle it, because it will go faster since I don't have access to a fast machine (just my laptop). > > Webrev here: > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > Thanks, > > On 9/18/19, 5:25 PM, "Daniel D. Daugherty" wrote: > > I created this sub-task for you: > > JDK-8231210 [BACKOUT] JDK-8207266 > ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > https://bugs.openjdk.java.net/browse/JDK-8231210 > > If you would prefer, I can handle this backout for you. > > Dan > > > On 9/18/19 8:21 PM, Hohensee, Paul wrote: > > Never having done this before, is it > > > > hg backout -r > > > > ? Do I file a JBS issue for the reversion? Seems necessary. > > > > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" wrote: > > > > % hg backout > > > > is the usual way to do this... > > > > Dan > > > > > > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > > > Is there a tool that will generate a reversal patch? > > > > > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" wrote: > > > > > > > Shall I go with that, or reverse the original patch? > > > > > > I'm a bit worried about what else might show up since the > > > NSK monitoring tests were not run prior to this push. > > > > > > I vote for backing out the fix until proper testing has > > > been done (and at least the one problem fixed...) > > > > > > Dan > > > > > > > > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > > > > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > > > > > > > public default long getCurrentThreadAllocatedBytes() { > > > > return -1; > > > > } > > > > > > > > Shall I go with that, or reverse the original patch? > > > > > > > > On 9/18/19, 4:48 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > I'll take a look. > > > > > > > > On 9/18/19, 4:40 PM, "David Holmes" wrote: > > > > > > > > Paul, > > > > > > > > Unfortunately this patch has broken the vmTestbase/nsk/monitoring tests: > > > > > > > > [2019-09-18T22:59:32,349Z] > > > > /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > > > > error: ServerThreadMXBeanNew is not abstract and does not override > > > > abstract method getCurrentThreadAllocatedBytes() in ThreadMXBean > > > > > > > > and possibly other issues as we are seeing hundreds of failures. > > > > > > > > David > > > > > > > > On 18/09/2019 8:50 am, David Holmes wrote: > > > > > On 18/09/2019 12:10 am, Hohensee, Paul wrote: > > > > >> Thanks, Serguei. :) > > > > >> > > > > >> David, are you ok with the patch? > > > > > > > > > > Yep, nothing further from me. > > > > > > > > > > David > > > > > > > > > >> Paul > > > > >> > > > > >> *From: *"serguei.spitsyn at oracle.com" > > > > >> *Date: *Tuesday, September 17, 2019 at 2:26 AM > > > > >> *To: *"Hohensee, Paul" , David Holmes > > > > >> , Mandy Chung > > > > >> *Cc: *OpenJDK Serviceability , > > > > >> "hotspot-gc-dev at openjdk.java.net" > > > > >> *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> Thank you for refactoring and fixing the test. > > > > >> It looks great now! > > > > >> > > > > >> Thanks, > > > > >> Serguei > > > > >> > > > > >> > > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > > > > >> > > > > >> Hi, Serguei, thanks for the review. New webrev at > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > > > > >> > > > > >> I refactored the test?s main() method, and you?re correct, > > > > >> getThreadAllocatedBytes should be getCurrentThreadAllocatedBytes in > > > > >> that context: fixed. > > > > >> > > > > >> Paul > > > > >> > > > > >> *From: *"serguei.spitsyn at oracle.com" > > > > >> > > > > >> > > > > >> *Organization: *Oracle Corporation > > > > >> *Date: *Friday, September 13, 2019 at 5:50 PM > > > > >> *To: *"Hohensee, Paul" > > > > >> , David Holmes > > > > >> , Mandy Chung > > > > >> > > > > >> *Cc: *OpenJDK Serviceability > > > > >> , > > > > >> "hotspot-gc-dev at openjdk.java.net" > > > > >> > > > > >> > > > > >> > > > > >> *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > > > > >> thread > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> It looks pretty good in general. > > > > >> > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > > > > >> > > > > >> > > > > >> It would be nice to refactor the java main() method as it becomes > > > > >> too big. > > > > >> Two ways ofgetCurrentThreadAllocatedBytes() testing are good > > > > >> candidates > > > > >> to become separate methods. > > > > >> > > > > >> 98 long size1 = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> Just wanted to double check if you wanted to invoke > > > > >> the getCurrentThreadAllocatedBytes() instead as it is > > > > >> a part of: > > > > >> > > > > >> 85 // First way, getCurrentThreadAllocatedBytes > > > > >> > > > > >> > > > > >> Thanks, > > > > >> Serguei > > > > >> > > > > >> On 9/13/19 12:11 PM, Hohensee, Paul wrote: > > > > >> > > > > >> Hi David, thanks for your comments. New webrev in > > > > >> > > > > >> > > > > >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > > > > >> > > > > >> > > > > >> Both the old and new versions of the code check that thread > > > > >> allocated memory is both supported and enabled. The existing version > > > > >> of getThreadAllocatedBytes(long []) calls > > > > >> verifyThreadAllocatedMemory(long []), which checks inline to make sure > > > > >> thread allocated memory is supported, then calls > > > > >> isThreadAllocatedMemoryEnabled() to verify that it's enabled. > > > > >> isThreadAllocatedMemoryEnabled() duplicates (!) the support check and > > > > >> returns the enabled flag. I removed the redundant check in the new > > > > >> version. > > > > >> > > > > >> > > > > >> You're of course correct about the back-to-back check. > > > > >> Application code can't know when the runtime will hijack a thread for > > > > >> its own purposes. I've removed the check. > > > > >> > > > > >> > > > > >> Paul > > > > >> > > > > >> > > > > >> On 9/13/19, 12:50 AM, "David Holmes" > > > > >> wrote: > > > > >> > > > > >> > > > > >> Hi Paul, > > > > >> > > > > >> > > > > >> On 13/09/2019 10:29 am, Hohensee, Paul wrote: > > > > >> > > > > >> > Thanks for clarifying the review rules. Would someone > > > > >> from the > > > > >> > > > > >> > serviceability team please review? New webrev at > > > > >> > > > > >> > > > > > >> > > > > >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > > > > >> > > > > >> > > > > >> One aspect of the functional change needs clarification > > > > >> for me - and > > > > >> > > > > >> apologies if this has been covered in the past. It seems > > > > >> to me that > > > > >> > > > > >> currently we only check isThreadAllocatedMemorySupported > > > > >> for these > > > > >> > > > > >> operations, but if I read things correctly the updated > > > > >> code additionally > > > > >> > > > > >> checks isThreadAllocatedMemoryEnabled, which is a > > > > >> behaviour change not > > > > >> > > > > >> mentioned in the CSR. > > > > >> > > > > >> > > > > >> > I didn?t disturb the existing checks in the test, just > > > > >> added code to > > > > >> > > > > >> > check the result of getThreadAllocatedBytes(long) on a > > > > >> non-current > > > > >> > > > > >> > thread, plus the back-to-back no-allocation checks. The > > > > >> former wasn?t > > > > >> > > > > >> > needed before because getThreadAllocatedBytes(long) was > > > > >> just a wrapper > > > > >> > > > > >> > around getThreadAllocatedBytes(long []). This patch > > > > >> changes that, so I > > > > >> > > > > >> > added a separate test. The latter is supposed to fail > > > > >> if there?s object > > > > >> > > > > >> > allocation on calls to getCurrentThreadAllocatedBytes and > > > > >> > > > > >> > getThreadAllocatedBytes(long). I.e., a feature, not a > > > > >> bug, because > > > > >> > > > > >> > accumulation of transient small objects can be a > > > > >> performance problem. > > > > >> > > > > >> > Thanks to your review, I noticed that the back-to-back > > > > >> check on the > > > > >> > > > > >> > current thread was using getThreadAllocatedBytes(long) > > > > >> instead of > > > > >> > > > > >> > getCurrentThreadAllocatedBytes and fixed it. I also > > > > >> removed all > > > > >> > > > > >> > instances of ?TEST FAILED: ?. > > > > >> > > > > >> > > > > >> The back-to-back check is not valid in general. You don't > > > > >> know if the > > > > >> > > > > >> first check might trigger some class loading on the > > > > >> return path after it > > > > >> > > > > >> has obtained the first memory value. The check might also > > > > >> fail if using > > > > >> > > > > >> JVMCI and some compilation related activity occurs in the > > > > >> current thread > > > > >> > > > > >> on the second call. Also with the introduction of > > > > >> handshakes its > > > > >> > > > > >> possible the current thread might hit a safepoint checks > > > > >> that results in > > > > >> > > > > >> it executing a handshake operation that performs > > > > >> allocation. Potentially > > > > >> > > > > >> there could be numerous non-deterministic actions that > > > > >> might occur > > > > >> > > > > >> leading to unanticipated allocation. > > > > >> > > > > >> > > > > >> I understand what you want to test here, I just don't > > > > >> think it is > > > > >> > > > > >> reliably doable. > > > > >> > > > > >> > > > > >> Thanks, > > > > >> > > > > >> David > > > > >> > > > > >> ----- > > > > >> > > > > >> > > > > >> > > > > > >> > > > > >> > Paul > > > > >> > > > > >> > > > > > >> > > > > >> > *From: *Mandy Chung > > > > >> > > > > >> > > > > >> > *Date: *Thursday, September 12, 2019 at 10:09 AM > > > > >> > > > > >> > *To: *"Hohensee, Paul" > > > > >> > > > > >> > > > > >> > *Cc: *OpenJDK > > > > >> Serviceability > > > > >> , > > > > >> > > > > >> >"hotspot-gc-dev at openjdk.java.net" > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > *Subject: *Re: RFR (M): 8207266: > > > > >> ThreadMXBean::getThreadAllocatedBytes() > > > > >> > > > > >> > can be quicker for self thread > > > > >> > > > > >> > > > > > >> > > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul wrote: > > > > >> > > > > >> > > > > > >> > > > > >> > Minor update in new > > > > >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > I only reviewed the library side implementation that > > > > >> looks good. I > > > > >> > > > > >> > expect the serviceability team to review the test and > > > > >> hotspot change. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > Need a confirmatory review to push this. If I > > > > >> understand the rules correctly, it doesn't need a Reviewer review > > > > >> since Mandy's already reviewed it, it just needs a Committer review. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > You need another reviewer to advice the following > > > > >> because I was not > > > > >> > > > > >> > close to the ThreadsList work. > > > > >> > > > > >> > > > > > >> > > > > >> > 2087 ThreadsListHandle tlh; > > > > >> > > > > >> > > > > > >> > > > > >> > 2088 JavaThread* java_thread = > > > > >> tlh.list()->find_JavaThread_from_java_tid(thread_id); > > > > >> > > > > >> > > > > > >> > > > > >> > 2089 > > > > >> > > > > >> > > > > > >> > > > > >> > 2090 if (java_thread != NULL) { > > > > >> > > > > >> > > > > > >> > > > > >> > 2091 return java_thread->cooked_allocated_bytes(); > > > > >> > > > > >> > > > > > >> > > > > >> > 2092 } > > > > >> > > > > >> > > > > > >> > > > > >> > This looks right to me. > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > > > > >> > > > > >> > > > > > >> > > > > >> > - "ThreadAllocatedMemory is expected to > > > > >> be disabled"); > > > > >> > > > > >> > > > > > >> > > > > >> > + "TEST FAILED: ThreadAllocatedMemory is > > > > >> expected to be > > > > >> > > > > >> > disabled"); > > > > >> > > > > >> > > > > > >> > > > > >> > Prepending "TEST FAILED" in exception message (in > > > > >> several places) > > > > >> > > > > >> > > > > > >> > > > > >> > seems redundant since such RuntimeException is thrown > > > > >> and expected > > > > >> > > > > >> > > > > > >> > > > > >> > a test failure. > > > > >> > > > > >> > > > > > >> > > > > >> > + // back-to-back calls shouldn't allocate any > > > > >> memory > > > > >> > > > > >> > > > > > >> > > > > >> > + size = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> > > > > > >> > > > > >> > + size1 = mbean.getThreadAllocatedBytes(id); > > > > >> > > > > >> > > > > > >> > > > > >> > + if (size1 != size) { > > > > >> > > > > >> > > > > > >> > > > > >> > Is there anything in the test can do to help guarantee > > > > >> this? I didn't > > > > >> > > > > >> > > > > > >> > > > > >> > closely review this test. The main thing I advice is > > > > >> to improve > > > > >> > > > > >> > > > > > >> > > > > >> > the reliability of this test. Put it in another way, > > > > >> we want to > > > > >> > > > > >> > > > > > >> > > > > >> > ensure that this test change will pass all the time in > > > > >> various > > > > >> > > > > >> > > > > > >> > > > > >> > test configuration. > > > > >> > > > > >> > > > > > >> > > > > >> > Mandy > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hohensee at amazon.com Thu Sep 19 00:55:26 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 00:55:26 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> Message-ID: I am. ?On 9/18/19, 5:54 PM, "Daniel D. Daugherty" wrote: Looks like the issue is different versions of 'hg' in use. When I import Paul's patch from his webrev using my 'hg' and then export it again, it matches my version of the backout. I have done a mechanical verification that the backout is an exact reversal for Paul's original changeset. I'm planning to push the changeset with the following info: 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Reviewed-by: phh, dholmes Everyone good with this? Dan On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: > For some reason, the backout that I did does not match the backout > that you did so I'm trying to figure that out. > > Dan > > > > On 9/18/19 8:36 PM, Hohensee, Paul wrote: >> And I filed 8231211 for the same thing. :) >> >> Yes, please handle it, because it will go faster since I don't have >> access to a fast machine (just my laptop). >> >> Webrev here: >> >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> >> Thanks, >> >> On 9/18/19, 5:25 PM, "Daniel D. Daugherty" >> wrote: >> >> I created this sub-task for you: >> JDK-8231210 [BACKOUT] JDK-8207266 >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> thread >> https://bugs.openjdk.java.net/browse/JDK-8231210 >> If you would prefer, I can handle this backout for you. >> Dan >> On 9/18/19 8:21 PM, Hohensee, Paul wrote: >> > Never having done this before, is it >> > >> > hg backout -r >> > >> > ? Do I file a JBS issue for the reversion? Seems necessary. >> > >> > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" >> wrote: >> > >> > % hg backout >> > >> > is the usual way to do this... >> > >> > Dan >> > >> > >> > On 9/18/19 8:17 PM, Hohensee, Paul wrote: >> > > Is there a tool that will generate a reversal patch? >> > > >> > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" >> wrote: >> > > >> > > > Shall I go with that, or reverse the original >> patch? >> > > >> > > I'm a bit worried about what else might show up >> since the >> > > NSK monitoring tests were not run prior to this push. >> > > >> > > I vote for backing out the fix until proper >> testing has >> > > been done (and at least the one problem fixed...) >> > > >> > > Dan >> > > >> > > >> > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: >> > > > They all implement >> com.sun.management.ThreadMXBean, so adding a >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give >> it a default implementation, vis >> > > > >> > > > public default long >> getCurrentThreadAllocatedBytes() { >> > > > return -1; >> > > > } >> > > > >> > > > Shall I go with that, or reverse the original >> patch? >> > > > >> > > > On 9/18/19, 4:48 PM, "serviceability-dev on >> behalf of Hohensee, Paul" >> > hohensee at amazon.com> wrote: >> > > > >> > > > I'll take a look. >> > > > >> > > > On 9/18/19, 4:40 PM, "David Holmes" >> wrote: >> > > > >> > > > Paul, >> > > > >> > > > Unfortunately this patch has broken the >> vmTestbase/nsk/monitoring tests: >> > > > >> > > > [2019-09-18T22:59:32,349Z] >> > > > >> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: >> > > > error: ServerThreadMXBeanNew is not >> abstract and does not override >> > > > abstract method >> getCurrentThreadAllocatedBytes() in ThreadMXBean >> > > > >> > > > and possibly other issues as we are >> seeing hundreds of failures. >> > > > >> > > > David >> > > > >> > > > On 18/09/2019 8:50 am, David Holmes wrote: >> > > > > On 18/09/2019 12:10 am, Hohensee, >> Paul wrote: >> > > > >> Thanks, Serguei. :) >> > > > >> >> > > > >> David, are you ok with the patch? >> > > > > >> > > > > Yep, nothing further from me. >> > > > > >> > > > > David >> > > > > >> > > > >> Paul >> > > > >> >> > > > >> *From: *"serguei.spitsyn at oracle.com" >> >> > > > >> *Date: *Tuesday, September 17, 2019 >> at 2:26 AM >> > > > >> *To: *"Hohensee, Paul" >> , David Holmes >> > > > >> , Mandy >> Chung >> > > > >> *Cc: *OpenJDK Serviceability >> , >> > > > >> "hotspot-gc-dev at openjdk.java.net" >> >> > > > >> *Subject: *Re: RFR (M): 8207266: >> > > > >> >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >> > > > >> >> > > > >> Hi Paul, >> > > > >> >> > > > >> Thank you for refactoring and fixing >> the test. >> > > > >> It looks great now! >> > > > >> >> > > > >> Thanks, >> > > > >> Serguei >> > > > >> >> > > > >> >> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: >> > > > >> >> > > > >> Hi, Serguei, thanks for the >> review. New webrev at >> > > > >> >> > > > >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >> > > > >> >> > > > >> I refactored the test?s main() >> method, and you?re correct, >> > > > >> getThreadAllocatedBytes should be >> getCurrentThreadAllocatedBytes in >> > > > >> that context: fixed. >> > > > >> >> > > > >> Paul >> > > > >> >> > > > >> *From: >> *"serguei.spitsyn at oracle.com" >> > > > >> >> >> > > > >> >> > > > >> *Organization: *Oracle Corporation >> > > > >> *Date: *Friday, September 13, >> 2019 at 5:50 PM >> > > > >> *To: *"Hohensee, Paul" >> >> > > > >> , David >> Holmes >> > > > >> , >> Mandy Chung >> > > > >> >> >> > > > >> *Cc: *OpenJDK Serviceability >> >> > > > >> >> , >> > > > >> "hotspot-gc-dev at openjdk.java.net" >> > > > >> >> >> > > > >> >> > > > >> >> >> > > > >> *Subject: *Re: RFR (M): 8207266: >> > > > >> >> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >> > > > >> thread >> > > > >> >> > > > >> Hi Paul, >> > > > >> >> > > > >> It looks pretty good in general. >> > > > >> >> > > > >> >> > > > >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >> > > > >> >> > > > >> >> > > > >> It would be nice to refactor the >> java main() method as it becomes >> > > > >> too big. >> > > > >> Two ways >> ofgetCurrentThreadAllocatedBytes() testing are good >> > > > >> candidates >> > > > >> to become separate methods. >> > > > >> >> > > > >> 98 long size1 = >> mbean.getThreadAllocatedBytes(id); >> > > > >> >> > > > >> Just wanted to double check if >> you wanted to invoke >> > > > >> the >> getCurrentThreadAllocatedBytes() instead as it is >> > > > >> a part of: >> > > > >> >> > > > >> 85 // First way, >> getCurrentThreadAllocatedBytes >> > > > >> >> > > > >> >> > > > >> Thanks, >> > > > >> Serguei >> > > > >> >> > > > >> On 9/13/19 12:11 PM, Hohensee, >> Paul wrote: >> > > > >> >> > > > >> Hi David, thanks for your >> comments. New webrev in >> > > > >> >> > > > >> >> > > > >> >> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >> > > > >> >> > > > >> >> > > > >> Both the old and new >> versions of the code check that thread >> > > > >> allocated memory is both supported >> and enabled. The existing version >> > > > >> of getThreadAllocatedBytes(long []) >> calls >> > > > >> verifyThreadAllocatedMemory(long >> []), which checks inline to make sure >> > > > >> thread allocated memory is >> supported, then calls >> > > > >> isThreadAllocatedMemoryEnabled() to >> verify that it's enabled. >> > > > >> isThreadAllocatedMemoryEnabled() >> duplicates (!) the support check and >> > > > >> returns the enabled flag. I removed >> the redundant check in the new >> > > > >> version. >> > > > >> >> > > > >> >> > > > >> You're of course correct >> about the back-to-back check. >> > > > >> Application code can't know when the >> runtime will hijack a thread for >> > > > >> its own purposes. I've removed the >> check. >> > > > >> >> > > > >> >> > > > >> Paul >> > > > >> >> > > > >> >> > > > >> On 9/13/19, 12:50 AM, "David >> Holmes" >> > > > >> >> wrote: >> > > > >> >> > > > >> >> > > > >> Hi Paul, >> > > > >> >> > > > >> >> > > > >> On 13/09/2019 10:29 am, >> Hohensee, Paul wrote: >> > > > >> >> > > > >> > Thanks for clarifying the review >> rules. Would someone >> > > > >> from the >> > > > >> >> > > > >> > serviceability team please review? >> New webrev at >> > > > >> >> > > > >> > >> > > > >> >> > > > >> >> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >> > > > >> >> > > > >> >> > > > >> One aspect of the >> functional change needs clarification >> > > > >> for me - and >> > > > >> >> > > > >> apologies if this has been covered >> in the past. It seems >> > > > >> to me that >> > > > >> >> > > > >> currently we only check >> isThreadAllocatedMemorySupported >> > > > >> for these >> > > > >> >> > > > >> operations, but if I read things >> correctly the updated >> > > > >> code additionally >> > > > >> >> > > > >> checks >> isThreadAllocatedMemoryEnabled, which is a >> > > > >> behaviour change not >> > > > >> >> > > > >> mentioned in the CSR. >> > > > >> >> > > > >> >> > > > >> > I didn?t disturb the existing >> checks in the test, just >> > > > >> added code to >> > > > >> >> > > > >> > check the result of >> getThreadAllocatedBytes(long) on a >> > > > >> non-current >> > > > >> >> > > > >> > thread, plus the back-to-back >> no-allocation checks. The >> > > > >> former wasn?t >> > > > >> >> > > > >> > needed before because >> getThreadAllocatedBytes(long) was >> > > > >> just a wrapper >> > > > >> >> > > > >> > around >> getThreadAllocatedBytes(long []). This patch >> > > > >> changes that, so I >> > > > >> >> > > > >> > added a separate test. The latter >> is supposed to fail >> > > > >> if there?s object >> > > > >> >> > > > >> > allocation on calls to >> getCurrentThreadAllocatedBytes and >> > > > >> >> > > > >> > getThreadAllocatedBytes(long). >> I.e., a feature, not a >> > > > >> bug, because >> > > > >> >> > > > >> > accumulation of transient small >> objects can be a >> > > > >> performance problem. >> > > > >> >> > > > >> > Thanks to your review, I noticed >> that the back-to-back >> > > > >> check on the >> > > > >> >> > > > >> > current thread was using >> getThreadAllocatedBytes(long) >> > > > >> instead of >> > > > >> >> > > > >> > getCurrentThreadAllocatedBytes and >> fixed it. I also >> > > > >> removed all >> > > > >> >> > > > >> > instances of ?TEST FAILED: ?. >> > > > >> >> > > > >> >> > > > >> The back-to-back check >> is not valid in general. You don't >> > > > >> know if the >> > > > >> >> > > > >> first check might trigger some class >> loading on the >> > > > >> return path after it >> > > > >> >> > > > >> has obtained the first >> memory value. The check might also >> > > > >> fail if using >> > > > >> >> > > > >> JVMCI and some compilation related >> activity occurs in the >> > > > >> current thread >> > > > >> >> > > > >> on the second call. >> Also with the introduction of >> > > > >> handshakes its >> > > > >> >> > > > >> possible the current thread might >> hit a safepoint checks >> > > > >> that results in >> > > > >> >> > > > >> it executing a >> handshake operation that performs >> > > > >> allocation. Potentially >> > > > >> >> > > > >> there could be numerous >> non-deterministic actions that >> > > > >> might occur >> > > > >> >> > > > >> leading to unanticipated allocation. >> > > > >> >> > > > >> >> > > > >> I understand what you >> want to test here, I just don't >> > > > >> think it is >> > > > >> >> > > > >> reliably doable. >> > > > >> >> > > > >> >> > > > >> Thanks, >> > > > >> >> > > > >> David >> > > > >> >> > > > >> ----- >> > > > >> >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Paul >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > *From: *Mandy >> Chung >> > > > >> >> > > > >> >> > > > >> > *Date: *Thursday, September 12, >> 2019 at 10:09 AM >> > > > >> >> > > > >> > *To: *"Hohensee, >> Paul" >> > > > >> >> > > > >> >> > > > >> > *Cc: *OpenJDK >> > > > >> >> Serviceability >> > > > >> >> , >> > > > >> >> > > > >> >"hotspot-gc-dev at openjdk.java.net" >> > > > >> >> >> > > > >> >> > > > >> >> >> > > > >> >> > > > >> > *Subject: *Re: RFR (M): 8207266: >> > > > >> ThreadMXBean::getThreadAllocatedBytes() >> > > > >> >> > > > >> > can be quicker for self thread >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul >> wrote: >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Minor update in new >> > > > >> >> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > I only reviewed the library side >> implementation that >> > > > >> looks good. I >> > > > >> >> > > > >> > expect the serviceability team to >> review the test and >> > > > >> hotspot change. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Need a confirmatory review to >> push this. If I >> > > > >> understand the rules correctly, it >> doesn't need a Reviewer review >> > > > >> since Mandy's already reviewed it, >> it just needs a Committer review. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > You need another reviewer to >> advice the following >> > > > >> because I was not >> > > > >> >> > > > >> > close to the ThreadsList work. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2087 ThreadsListHandle tlh; >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2088 JavaThread* java_thread = >> > > > >> >> tlh.list()->find_JavaThread_from_java_tid(thread_id); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2089 >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2090 if (java_thread != NULL) { >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2091 return >> java_thread->cooked_allocated_bytes(); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > 2092 } >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > This looks right to me. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > >> > > > >> >> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > - >> "ThreadAllocatedMemory is expected to >> > > > >> be disabled"); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > + "TEST FAILED: >> ThreadAllocatedMemory is >> > > > >> expected to be >> > > > >> >> > > > >> > disabled"); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Prepending "TEST FAILED" in >> exception message (in >> > > > >> several places) >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > seems redundant since such >> RuntimeException is thrown >> > > > >> and expected >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > a test failure. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > + // back-to-back calls >> shouldn't allocate any >> > > > >> memory >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > + size = >> mbean.getThreadAllocatedBytes(id); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > + size1 = >> mbean.getThreadAllocatedBytes(id); >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > + if (size1 != size) { >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Is there anything in the test can >> do to help guarantee >> > > > >> this? I didn't >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > closely review this test. The >> main thing I advice is >> > > > >> to improve >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > the reliability of this test. Put >> it in another way, >> > > > >> we want to >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > ensure that this test change will >> pass all the time in >> > > > >> various >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > test configuration. >> > > > >> >> > > > >> > >> > > > >> >> > > > >> > Mandy >> > > > >> >> > > > >> > >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > >> > >> > From david.holmes at oracle.com Thu Sep 19 01:00:07 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 11:00:07 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> Message-ID: <114ed65b-4b13-22aa-1849-f2737f7a9779@oracle.com> Ship it! Thanks Dan! David On 19/09/2019 10:53 am, Daniel D. Daugherty wrote: > Looks like the issue is different versions of 'hg' in use. > > When I import Paul's patch from his webrev using my 'hg' and > then export it again, it matches my version of the backout. > > I have done a mechanical verification that the backout is an > exact reversal for Paul's original changeset. > > I'm planning to push the changeset with the following info: > > > 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > Reviewed-by: phh, dholmes > > Everyone good with this? > > Dan > > On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: >> For some reason, the backout that I did does not match the backout >> that you did so I'm trying to figure that out. >> >> Dan >> >> >> >> On 9/18/19 8:36 PM, Hohensee, Paul wrote: >>> And I filed 8231211 for the same thing. :) >>> >>> Yes, please handle it, because it will go faster since I don't have >>> access to a fast machine (just my laptop). >>> >>> Webrev here: >>> >>> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >>> >>> Thanks, >>> >>> ?On 9/18/19, 5:25 PM, "Daniel D. Daugherty" >>> wrote: >>> >>> ???? I created this sub-task for you: >>> ???? ???? JDK-8231210 [BACKOUT] JDK-8207266 >>> ???? ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >>> thread >>> ???? https://bugs.openjdk.java.net/browse/JDK-8231210 >>> ???? ???? If you would prefer, I can handle this backout for you. >>> ???? ???? Dan >>> ???? ???? ???? On 9/18/19 8:21 PM, Hohensee, Paul wrote: >>> ???? > Never having done this before, is it >>> ???? > >>> ???? > hg backout -r >>> ???? > >>> ???? > ? Do I file a JBS issue for the reversion? Seems necessary. >>> ???? > >>> ???? > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" >>> wrote: >>> ???? > >>> ???? >????? % hg backout >>> ???? > >>> ???? >????? is the usual way to do this... >>> ???? > >>> ???? >????? Dan >>> ???? > >>> ???? > >>> ???? >????? On 9/18/19 8:17 PM, Hohensee, Paul wrote: >>> ???? >????? > Is there a tool that will generate a reversal patch? >>> ???? >????? > >>> ???? >????? > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" >>> wrote: >>> ???? >????? > >>> ???? >????? >?????? > Shall I go with that, or reverse the original >>> patch? >>> ???? >????? > >>> ???? >????? >????? I'm a bit worried about what else might show up >>> since the >>> ???? >????? >????? NSK monitoring tests were not run prior to this push. >>> ???? >????? > >>> ???? >????? >????? I vote for backing out the fix until proper >>> testing has >>> ???? >????? >????? been done (and at least the one problem fixed...) >>> ???? >????? > >>> ???? >????? >????? Dan >>> ???? >????? > >>> ???? >????? > >>> ???? >????? >????? On 9/18/19 8:00 PM, Hohensee, Paul wrote: >>> ???? >????? >????? > They all implement >>> com.sun.management.ThreadMXBean, so adding a >>> getCurrentThreadAllocatedBytes broke them. Potential fix is to give >>> it a default implementation, vis >>> ???? >????? >????? > >>> ???? >????? >????? >????? public default long >>> getCurrentThreadAllocatedBytes() { >>> ???? >????? >????? >????????? return -1; >>> ???? >????? >????? >????? } >>> ???? >????? >????? > >>> ???? >????? >????? > Shall I go with that, or reverse the original >>> patch? >>> ???? >????? >????? > >>> ???? >????? >????? > On 9/18/19, 4:48 PM, "serviceability-dev on >>> behalf of Hohensee, Paul" >>> >> hohensee at amazon.com> wrote: >>> ???? >????? >????? > >>> ???? >????? >????? >????? I'll take a look. >>> ???? >????? >????? > >>> ???? >????? >????? >????? On 9/18/19, 4:40 PM, "David Holmes" >>> wrote: >>> ???? >????? >????? > >>> ???? >????? >????? >????????? Paul, >>> ???? >????? >????? > >>> ???? >????? >????? >????????? Unfortunately this patch has broken the >>> vmTestbase/nsk/monitoring tests: >>> ???? >????? >????? > >>> ???? >????? >????? > [2019-09-18T22:59:32,349Z] >>> ???? >????? >????? > >>> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: >>> >>> ???? >????? >????? >????????? error: ServerThreadMXBeanNew is not >>> abstract and does not override >>> ???? >????? >????? >????????? abstract method >>> getCurrentThreadAllocatedBytes() in ThreadMXBean >>> ???? >????? >????? > >>> ???? >????? >????? >????????? and possibly other issues as we are >>> seeing hundreds of failures. >>> ???? >????? >????? > >>> ???? >????? >????? >????????? David >>> ???? >????? >????? > >>> ???? >????? >????? >????????? On 18/09/2019 8:50 am, David Holmes wrote: >>> ???? >????? >????? >????????? > On 18/09/2019 12:10 am, Hohensee, >>> Paul wrote: >>> ???? >????? >????? >????????? >> Thanks, Serguei. :) >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> David, are you ok with the patch? >>> ???? >????? >????? >????????? > >>> ???? >????? >????? >????????? > Yep, nothing further from me. >>> ???? >????? >????? >????????? > >>> ???? >????? >????? >????????? > David >>> ???? >????? >????? >????????? > >>> ???? >????? >????? >????????? >> Paul >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> *From: *"serguei.spitsyn at oracle.com" >>> >>> ???? >????? >????? >????????? >> *Date: *Tuesday, September 17, 2019 >>> at 2:26 AM >>> ???? >????? >????? >????????? >> *To: *"Hohensee, Paul" >>> , David Holmes >>> ???? >????? >????? >????????? >> , Mandy >>> Chung >>> ???? >????? >????? >????????? >> *Cc: *OpenJDK Serviceability >>> , >>> ???? >????? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >>> >>> ???? >????? >????? >????????? >> *Subject: *Re: RFR (M): 8207266: >>> ???? >????? >????? >????????? >> >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> Hi Paul, >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> Thank you for refactoring and fixing >>> the test. >>> ???? >????? >????? >????????? >> It looks great now! >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> Thanks, >>> ???? >????? >????? >????????? >> Serguei >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> On 9/15/19 02:52, Hohensee, Paul wrote: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? Hi, Serguei, thanks for the >>> review. New webrev at >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? I refactored the test?s main() >>> method, and you?re correct, >>> ???? >????? >????? >????????? >> getThreadAllocatedBytes should be >>> getCurrentThreadAllocatedBytes in >>> ???? >????? >????? >????????? >>???? that context: fixed. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? Paul >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? *From: >>> *"serguei.spitsyn at oracle.com" >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> *Organization: *Oracle Corporation >>> ???? >????? >????? >????????? >>???? *Date: *Friday, September 13, >>> 2019 at 5:50 PM >>> ???? >????? >????? >????????? >>???? *To: *"Hohensee, Paul" >>> >>> ???? >????? >????? >????????? >> , David >>> Holmes >>> ???? >????? >????? >????????? >> , >>> Mandy Chung >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >>???? *Cc: *OpenJDK Serviceability >>> >>> ???? >????? >????? >????????? >> >>> , >>> ???? >????? >????? >????????? >> "hotspot-gc-dev at openjdk.java.net" >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >>???? *Subject: *Re: RFR (M): 8207266: >>> ???? >????? >????? >????????? >> >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >>> ???? >????? >????? >????????? >> thread >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? Hi Paul, >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? It looks pretty good in general. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >>> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? It would be nice to refactor the >>> java main() method as it becomes >>> ???? >????? >????? >????????? >>???? too big. >>> ???? >????? >????? >????????? >>???? Two ways >>> ofgetCurrentThreadAllocatedBytes() testing are good >>> ???? >????? >????? >????????? >> candidates >>> ???? >????? >????? >????????? >>???? to become separate methods. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> 98???????? long size1 = >>> mbean.getThreadAllocatedBytes(id); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? Just wanted to double check if >>> you wanted to invoke >>> ???? >????? >????? >????????? >>???? the >>> getCurrentThreadAllocatedBytes() instead as it is >>> ???? >????? >????? >????????? >>???? a part of: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> 85???????? // First way, >>> getCurrentThreadAllocatedBytes >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? Thanks, >>> ???? >????? >????? >????????? >>???? Serguei >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???? On 9/13/19 12:11 PM, Hohensee, >>> Paul wrote: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???????? Hi David, thanks for your >>> comments. New webrev in >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???????? Both the old and new >>> versions of the code check that thread >>> ???? >????? >????? >????????? >> allocated memory is both supported >>> and enabled. The existing version >>> ???? >????? >????? >????????? >> of getThreadAllocatedBytes(long []) >>> calls >>> ???? >????? >????? >????????? >> verifyThreadAllocatedMemory(long >>> []), which checks inline to make sure >>> ???? >????? >????? >????????? >> thread allocated memory is >>> supported, then calls >>> ???? >????? >????? >????????? >> isThreadAllocatedMemoryEnabled() to >>> verify that it's enabled. >>> ???? >????? >????? >????????? >> isThreadAllocatedMemoryEnabled() >>> duplicates (!) the support check and >>> ???? >????? >????? >????????? >> returns the enabled flag. I removed >>> the redundant check in the new >>> ???? >????? >????? >????????? >> version. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???????? You're of course correct >>> about the back-to-back check. >>> ???? >????? >????? >????????? >> Application code can't know when the >>> runtime will hijack a thread for >>> ???? >????? >????? >????????? >> its own purposes. I've removed the >>> check. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???????? Paul >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>???????? On 9/13/19, 12:50 AM, "David >>> Holmes" >>> ???? >????? >????? >????????? >> wrote: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? Hi Paul, >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? On 13/09/2019 10:29 am, >>> Hohensee, Paul wrote: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Thanks for clarifying the review >>> rules. Would someone >>> ???? >????? >????? >????????? >> from the >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > serviceability team please review? >>> New webrev at >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? One aspect of the >>> functional change needs clarification >>> ???? >????? >????? >????????? >> for me - and >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> apologies if this has been covered >>> in the past. It seems >>> ???? >????? >????? >????????? >> to me that >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> currently we only check >>> isThreadAllocatedMemorySupported >>> ???? >????? >????? >????????? >> for these >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> operations, but if I read things >>> correctly the updated >>> ???? >????? >????? >????????? >> code additionally >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> checks >>> isThreadAllocatedMemoryEnabled, which is a >>> ???? >????? >????? >????????? >> behaviour change not >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> mentioned in the CSR. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > I didn?t disturb the existing >>> checks in the test, just >>> ???? >????? >????? >????????? >> added code to >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > check the result of >>> getThreadAllocatedBytes(long) on a >>> ???? >????? >????? >????????? >> non-current >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > thread, plus the back-to-back >>> no-allocation checks. The >>> ???? >????? >????? >????????? >> former wasn?t >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > needed before because >>> getThreadAllocatedBytes(long) was >>> ???? >????? >????? >????????? >> just a wrapper >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > around >>> getThreadAllocatedBytes(long []). This patch >>> ???? >????? >????? >????????? >> changes that, so I >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > added a separate test. The latter >>> is supposed to fail >>> ???? >????? >????? >????????? >> if there?s object >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > allocation on calls to >>> getCurrentThreadAllocatedBytes and >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > getThreadAllocatedBytes(long). >>> I.e., a feature, not a >>> ???? >????? >????? >????????? >> bug, because >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > accumulation of transient small >>> objects can be a >>> ???? >????? >????? >????????? >> performance problem. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Thanks to your review, I noticed >>> that the back-to-back >>> ???? >????? >????? >????????? >> check on the >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > current thread was using >>> getThreadAllocatedBytes(long) >>> ???? >????? >????? >????????? >> instead of >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > getCurrentThreadAllocatedBytes and >>> fixed it. I also >>> ???? >????? >????? >????????? >> removed all >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > instances of ?TEST FAILED: ?. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? The back-to-back check >>> is not valid in general. You don't >>> ???? >????? >????? >????????? >> know if the >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> first check might trigger some class >>> loading on the >>> ???? >????? >????? >????????? >> return path after it >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? has obtained the first >>> memory value. The check might also >>> ???? >????? >????? >????????? >> fail if using >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> JVMCI and some compilation related >>> activity occurs in the >>> ???? >????? >????? >????????? >> current thread >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? on the second call. >>> Also with the introduction of >>> ???? >????? >????? >????????? >> handshakes its >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> possible the current thread might >>> hit a safepoint checks >>> ???? >????? >????? >????????? >> that results in >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? it executing a >>> handshake operation that performs >>> ???? >????? >????? >????????? >> allocation. Potentially >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> there could be numerous >>> non-deterministic actions that >>> ???? >????? >????? >????????? >> might occur >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> leading to unanticipated allocation. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >>????????????? I understand what you >>> want to test here, I just don't >>> ???? >????? >????? >????????? >> think it is >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> reliably doable. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> Thanks, >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> David >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> ----- >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Paul >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > *From: *Mandy >>> Chung >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > *Date: *Thursday, September 12, >>> 2019 at 10:09 AM >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > *To: *"Hohensee, >>> Paul" >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > *Cc: *OpenJDK >>> ???? >????? >????? >????????? >> >>> Serviceability >>> ???? >????? >????? >????????? >> >>> , >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >"hotspot-gc-dev at openjdk.java.net" >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > *Subject: *Re: RFR (M): 8207266: >>> ???? >????? >????? >????????? >> ThreadMXBean::getThreadAllocatedBytes() >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > can be quicker for self thread >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > On 9/3/19 12:38 PM, Hohensee, Paul >>> wrote: >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >???? Minor update in new >>> ???? >????? >????? >????????? >> >>> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > I only reviewed the library side >>> implementation that >>> ???? >????? >????? >????????? >> looks good.? I >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > expect the serviceability team to >>> review the test and >>> ???? >????? >????? >????????? >> hotspot change. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >???? Need a confirmatory review to >>> push this. If I >>> ???? >????? >????? >????????? >> understand the rules correctly, it >>> doesn't need a Reviewer review >>> ???? >????? >????? >????????? >> since Mandy's already reviewed it, >>> it just needs a Committer review. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > You need another reviewer to >>> advice the following >>> ???? >????? >????? >????????? >> because I was not >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > close to the ThreadsList work. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2087?? ThreadsListHandle tlh; >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2088?? JavaThread* java_thread = >>> ???? >????? >????? >????????? >> >>> tlh.list()->find_JavaThread_from_java_tid(thread_id); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2089 >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2090?? if (java_thread != NULL) { >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2091???? return >>> java_thread->cooked_allocated_bytes(); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > 2092?? } >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > This looks right to me. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > - "ThreadAllocatedMemory is >>> expected to >>> ???? >????? >????? >????????? >> be disabled"); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > +??????????????? "TEST FAILED: >>> ThreadAllocatedMemory is >>> ???? >????? >????? >????????? >> expected to be >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > disabled"); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Prepending "TEST FAILED" in >>> exception message (in >>> ???? >????? >????? >????????? >> several places) >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > seems redundant since such >>> RuntimeException is thrown >>> ???? >????? >????? >????????? >> and expected >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > a test failure. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > +??????? // back-to-back calls >>> shouldn't allocate any >>> ???? >????? >????? >????????? >> memory >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > +??????? size = >>> mbean.getThreadAllocatedBytes(id); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > +??????? size1 = >>> mbean.getThreadAllocatedBytes(id); >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > +??????? if (size1 != size) { >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Is there anything in the test can >>> do to help guarantee >>> ???? >????? >????? >????????? >> this? I didn't >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > closely review this test.? The >>> main thing I advice is >>> ???? >????? >????? >????????? >> to improve >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > the reliability of this test.? Put >>> it in another way, >>> ???? >????? >????? >????????? >> we want to >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > ensure that this test change will >>> pass all the time in >>> ???? >????? >????? >????????? >> various >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > test configuration. >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > Mandy >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> > >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? >????????? >> >>> ???? >????? >????? > >>> ???? >????? >????? > >>> ???? >????? >????? > >>> ???? >????? >????? > >>> ???? >????? > >>> ???? >????? > >>> ???? >????? > >>> ???? > >>> ???? > >>> ???? > >>> >> > From hohensee at amazon.com Thu Sep 19 01:05:40 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 01:05:40 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <114ed65b-4b13-22aa-1849-f2737f7a9779@oracle.com> References: <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> <114ed65b-4b13-22aa-1849-f2737f7a9779@oracle.com> Message-ID: <5DE94C8C-B398-48A4-AB0C-2CEE51A99DA0@amazon.com> +1, thanks! My apologies for the bad patch. I'll file another issue and run every test that mentions ThreadMXBean. At least, I know how to revert a patch now. Paul ?On 9/18/19, 6:00 PM, "David Holmes" wrote: Ship it! Thanks Dan! David On 19/09/2019 10:53 am, Daniel D. Daugherty wrote: > Looks like the issue is different versions of 'hg' in use. > > When I import Paul's patch from his webrev using my 'hg' and > then export it again, it matches my version of the backout. > > I have done a mechanical verification that the backout is an > exact reversal for Paul's original changeset. > > I'm planning to push the changeset with the following info: > > > 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() > can be quicker for self thread > Reviewed-by: phh, dholmes > > Everyone good with this? > > Dan > > On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: >> For some reason, the backout that I did does not match the backout >> that you did so I'm trying to figure that out. >> >> Dan >> >> >> >> On 9/18/19 8:36 PM, Hohensee, Paul wrote: >>> And I filed 8231211 for the same thing. :) >>> >>> Yes, please handle it, because it will go faster since I don't have >>> access to a fast machine (just my laptop). >>> >>> Webrev here: >>> >>> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >>> >>> Thanks, >>> >>> On 9/18/19, 5:25 PM, "Daniel D. Daugherty" >>> wrote: >>> >>> I created this sub-task for you: >>> JDK-8231210 [BACKOUT] JDK-8207266 >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >>> thread >>> https://bugs.openjdk.java.net/browse/JDK-8231210 >>> If you would prefer, I can handle this backout for you. >>> Dan >>> On 9/18/19 8:21 PM, Hohensee, Paul wrote: >>> > Never having done this before, is it >>> > >>> > hg backout -r >>> > >>> > ? Do I file a JBS issue for the reversion? Seems necessary. >>> > >>> > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" >>> wrote: >>> > >>> > % hg backout >>> > >>> > is the usual way to do this... >>> > >>> > Dan >>> > >>> > >>> > On 9/18/19 8:17 PM, Hohensee, Paul wrote: >>> > > Is there a tool that will generate a reversal patch? >>> > > >>> > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" >>> wrote: >>> > > >>> > > > Shall I go with that, or reverse the original >>> patch? >>> > > >>> > > I'm a bit worried about what else might show up >>> since the >>> > > NSK monitoring tests were not run prior to this push. >>> > > >>> > > I vote for backing out the fix until proper >>> testing has >>> > > been done (and at least the one problem fixed...) >>> > > >>> > > Dan >>> > > >>> > > >>> > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: >>> > > > They all implement >>> com.sun.management.ThreadMXBean, so adding a >>> getCurrentThreadAllocatedBytes broke them. Potential fix is to give >>> it a default implementation, vis >>> > > > >>> > > > public default long >>> getCurrentThreadAllocatedBytes() { >>> > > > return -1; >>> > > > } >>> > > > >>> > > > Shall I go with that, or reverse the original >>> patch? >>> > > > >>> > > > On 9/18/19, 4:48 PM, "serviceability-dev on >>> behalf of Hohensee, Paul" >>> >> hohensee at amazon.com> wrote: >>> > > > >>> > > > I'll take a look. >>> > > > >>> > > > On 9/18/19, 4:40 PM, "David Holmes" >>> wrote: >>> > > > >>> > > > Paul, >>> > > > >>> > > > Unfortunately this patch has broken the >>> vmTestbase/nsk/monitoring tests: >>> > > > >>> > > > [2019-09-18T22:59:32,349Z] >>> > > > >>> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: >>> >>> > > > error: ServerThreadMXBeanNew is not >>> abstract and does not override >>> > > > abstract method >>> getCurrentThreadAllocatedBytes() in ThreadMXBean >>> > > > >>> > > > and possibly other issues as we are >>> seeing hundreds of failures. >>> > > > >>> > > > David >>> > > > >>> > > > On 18/09/2019 8:50 am, David Holmes wrote: >>> > > > > On 18/09/2019 12:10 am, Hohensee, >>> Paul wrote: >>> > > > >> Thanks, Serguei. :) >>> > > > >> >>> > > > >> David, are you ok with the patch? >>> > > > > >>> > > > > Yep, nothing further from me. >>> > > > > >>> > > > > David >>> > > > > >>> > > > >> Paul >>> > > > >> >>> > > > >> *From: *"serguei.spitsyn at oracle.com" >>> >>> > > > >> *Date: *Tuesday, September 17, 2019 >>> at 2:26 AM >>> > > > >> *To: *"Hohensee, Paul" >>> , David Holmes >>> > > > >> , Mandy >>> Chung >>> > > > >> *Cc: *OpenJDK Serviceability >>> , >>> > > > >> "hotspot-gc-dev at openjdk.java.net" >>> >>> > > > >> *Subject: *Re: RFR (M): 8207266: >>> > > > >> >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread >>> > > > >> >>> > > > >> Hi Paul, >>> > > > >> >>> > > > >> Thank you for refactoring and fixing >>> the test. >>> > > > >> It looks great now! >>> > > > >> >>> > > > >> Thanks, >>> > > > >> Serguei >>> > > > >> >>> > > > >> >>> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: >>> > > > >> >>> > > > >> Hi, Serguei, thanks for the >>> review. New webrev at >>> > > > >> >>> > > > >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ >>> > > > >> >>> > > > >> I refactored the test?s main() >>> method, and you?re correct, >>> > > > >> getThreadAllocatedBytes should be >>> getCurrentThreadAllocatedBytes in >>> > > > >> that context: fixed. >>> > > > >> >>> > > > >> Paul >>> > > > >> >>> > > > >> *From: >>> *"serguei.spitsyn at oracle.com" >>> > > > >> >>> >>> > > > >> >>> > > > >> *Organization: *Oracle Corporation >>> > > > >> *Date: *Friday, September 13, >>> 2019 at 5:50 PM >>> > > > >> *To: *"Hohensee, Paul" >>> >>> > > > >> , David >>> Holmes >>> > > > >> , >>> Mandy Chung >>> > > > >> >>> >>> > > > >> *Cc: *OpenJDK Serviceability >>> >>> > > > >> >>> , >>> > > > >> "hotspot-gc-dev at openjdk.java.net" >>> > > > >> >>> >>> > > > >> >>> > > > >> >>> >>> > > > >> *Subject: *Re: RFR (M): 8207266: >>> > > > >> >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self >>> > > > >> thread >>> > > > >> >>> > > > >> Hi Paul, >>> > > > >> >>> > > > >> It looks pretty good in general. >>> > > > >> >>> > > > >> >>> > > > >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html >>> >>> > > > >> >>> > > > >> >>> > > > >> It would be nice to refactor the >>> java main() method as it becomes >>> > > > >> too big. >>> > > > >> Two ways >>> ofgetCurrentThreadAllocatedBytes() testing are good >>> > > > >> candidates >>> > > > >> to become separate methods. >>> > > > >> >>> > > > >> 98 long size1 = >>> mbean.getThreadAllocatedBytes(id); >>> > > > >> >>> > > > >> Just wanted to double check if >>> you wanted to invoke >>> > > > >> the >>> getCurrentThreadAllocatedBytes() instead as it is >>> > > > >> a part of: >>> > > > >> >>> > > > >> 85 // First way, >>> getCurrentThreadAllocatedBytes >>> > > > >> >>> > > > >> >>> > > > >> Thanks, >>> > > > >> Serguei >>> > > > >> >>> > > > >> On 9/13/19 12:11 PM, Hohensee, >>> Paul wrote: >>> > > > >> >>> > > > >> Hi David, thanks for your >>> comments. New webrev in >>> > > > >> >>> > > > >> >>> > > > >> >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ >>> > > > >> >>> > > > >> >>> > > > >> Both the old and new >>> versions of the code check that thread >>> > > > >> allocated memory is both supported >>> and enabled. The existing version >>> > > > >> of getThreadAllocatedBytes(long []) >>> calls >>> > > > >> verifyThreadAllocatedMemory(long >>> []), which checks inline to make sure >>> > > > >> thread allocated memory is >>> supported, then calls >>> > > > >> isThreadAllocatedMemoryEnabled() to >>> verify that it's enabled. >>> > > > >> isThreadAllocatedMemoryEnabled() >>> duplicates (!) the support check and >>> > > > >> returns the enabled flag. I removed >>> the redundant check in the new >>> > > > >> version. >>> > > > >> >>> > > > >> >>> > > > >> You're of course correct >>> about the back-to-back check. >>> > > > >> Application code can't know when the >>> runtime will hijack a thread for >>> > > > >> its own purposes. I've removed the >>> check. >>> > > > >> >>> > > > >> >>> > > > >> Paul >>> > > > >> >>> > > > >> >>> > > > >> On 9/13/19, 12:50 AM, "David >>> Holmes" >>> > > > >> wrote: >>> > > > >> >>> > > > >> >>> > > > >> Hi Paul, >>> > > > >> >>> > > > >> >>> > > > >> On 13/09/2019 10:29 am, >>> Hohensee, Paul wrote: >>> > > > >> >>> > > > >> > Thanks for clarifying the review >>> rules. Would someone >>> > > > >> from the >>> > > > >> >>> > > > >> > serviceability team please review? >>> New webrev at >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> >>> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ >>> > > > >> >>> > > > >> >>> > > > >> One aspect of the >>> functional change needs clarification >>> > > > >> for me - and >>> > > > >> >>> > > > >> apologies if this has been covered >>> in the past. It seems >>> > > > >> to me that >>> > > > >> >>> > > > >> currently we only check >>> isThreadAllocatedMemorySupported >>> > > > >> for these >>> > > > >> >>> > > > >> operations, but if I read things >>> correctly the updated >>> > > > >> code additionally >>> > > > >> >>> > > > >> checks >>> isThreadAllocatedMemoryEnabled, which is a >>> > > > >> behaviour change not >>> > > > >> >>> > > > >> mentioned in the CSR. >>> > > > >> >>> > > > >> >>> > > > >> > I didn?t disturb the existing >>> checks in the test, just >>> > > > >> added code to >>> > > > >> >>> > > > >> > check the result of >>> getThreadAllocatedBytes(long) on a >>> > > > >> non-current >>> > > > >> >>> > > > >> > thread, plus the back-to-back >>> no-allocation checks. The >>> > > > >> former wasn?t >>> > > > >> >>> > > > >> > needed before because >>> getThreadAllocatedBytes(long) was >>> > > > >> just a wrapper >>> > > > >> >>> > > > >> > around >>> getThreadAllocatedBytes(long []). This patch >>> > > > >> changes that, so I >>> > > > >> >>> > > > >> > added a separate test. The latter >>> is supposed to fail >>> > > > >> if there?s object >>> > > > >> >>> > > > >> > allocation on calls to >>> getCurrentThreadAllocatedBytes and >>> > > > >> >>> > > > >> > getThreadAllocatedBytes(long). >>> I.e., a feature, not a >>> > > > >> bug, because >>> > > > >> >>> > > > >> > accumulation of transient small >>> objects can be a >>> > > > >> performance problem. >>> > > > >> >>> > > > >> > Thanks to your review, I noticed >>> that the back-to-back >>> > > > >> check on the >>> > > > >> >>> > > > >> > current thread was using >>> getThreadAllocatedBytes(long) >>> > > > >> instead of >>> > > > >> >>> > > > >> > getCurrentThreadAllocatedBytes and >>> fixed it. I also >>> > > > >> removed all >>> > > > >> >>> > > > >> > instances of ?TEST FAILED: ?. >>> > > > >> >>> > > > >> >>> > > > >> The back-to-back check >>> is not valid in general. You don't >>> > > > >> know if the >>> > > > >> >>> > > > >> first check might trigger some class >>> loading on the >>> > > > >> return path after it >>> > > > >> >>> > > > >> has obtained the first >>> memory value. The check might also >>> > > > >> fail if using >>> > > > >> >>> > > > >> JVMCI and some compilation related >>> activity occurs in the >>> > > > >> current thread >>> > > > >> >>> > > > >> on the second call. >>> Also with the introduction of >>> > > > >> handshakes its >>> > > > >> >>> > > > >> possible the current thread might >>> hit a safepoint checks >>> > > > >> that results in >>> > > > >> >>> > > > >> it executing a >>> handshake operation that performs >>> > > > >> allocation. Potentially >>> > > > >> >>> > > > >> there could be numerous >>> non-deterministic actions that >>> > > > >> might occur >>> > > > >> >>> > > > >> leading to unanticipated allocation. >>> > > > >> >>> > > > >> >>> > > > >> I understand what you >>> want to test here, I just don't >>> > > > >> think it is >>> > > > >> >>> > > > >> reliably doable. >>> > > > >> >>> > > > >> >>> > > > >> Thanks, >>> > > > >> >>> > > > >> David >>> > > > >> >>> > > > >> ----- >>> > > > >> >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Paul >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > *From: *Mandy >>> Chung >>> > > > >> >>> > > > >> >>> > > > >> > *Date: *Thursday, September 12, >>> 2019 at 10:09 AM >>> > > > >> >>> > > > >> > *To: *"Hohensee, >>> Paul" >>> > > > >> >>> > > > >> >>> > > > >> > *Cc: *OpenJDK >>> > > > >> >>> Serviceability >>> > > > >> >>> , >>> > > > >> >>> > > > >> >"hotspot-gc-dev at openjdk.java.net" >>> > > > >> >>> >>> > > > >> >>> > > > >> >>> >>> > > > >> >>> > > > >> > *Subject: *Re: RFR (M): 8207266: >>> > > > >> ThreadMXBean::getThreadAllocatedBytes() >>> > > > >> >>> > > > >> > can be quicker for self thread >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul >>> wrote: >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Minor update in new >>> > > > >> >>> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > I only reviewed the library side >>> implementation that >>> > > > >> looks good. I >>> > > > >> >>> > > > >> > expect the serviceability team to >>> review the test and >>> > > > >> hotspot change. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Need a confirmatory review to >>> push this. If I >>> > > > >> understand the rules correctly, it >>> doesn't need a Reviewer review >>> > > > >> since Mandy's already reviewed it, >>> it just needs a Committer review. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > You need another reviewer to >>> advice the following >>> > > > >> because I was not >>> > > > >> >>> > > > >> > close to the ThreadsList work. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2087 ThreadsListHandle tlh; >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2088 JavaThread* java_thread = >>> > > > >> >>> tlh.list()->find_JavaThread_from_java_tid(thread_id); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2089 >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2090 if (java_thread != NULL) { >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2091 return >>> java_thread->cooked_allocated_bytes(); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > 2092 } >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > This looks right to me. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > >>> > > > >> >>> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > - "ThreadAllocatedMemory is >>> expected to >>> > > > >> be disabled"); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > + "TEST FAILED: >>> ThreadAllocatedMemory is >>> > > > >> expected to be >>> > > > >> >>> > > > >> > disabled"); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Prepending "TEST FAILED" in >>> exception message (in >>> > > > >> several places) >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > seems redundant since such >>> RuntimeException is thrown >>> > > > >> and expected >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > a test failure. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > + // back-to-back calls >>> shouldn't allocate any >>> > > > >> memory >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > + size = >>> mbean.getThreadAllocatedBytes(id); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > + size1 = >>> mbean.getThreadAllocatedBytes(id); >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > + if (size1 != size) { >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Is there anything in the test can >>> do to help guarantee >>> > > > >> this? I didn't >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > closely review this test. The >>> main thing I advice is >>> > > > >> to improve >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > the reliability of this test. Put >>> it in another way, >>> > > > >> we want to >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > ensure that this test change will >>> pass all the time in >>> > > > >> various >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > test configuration. >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> > Mandy >>> > > > >> >>> > > > >> > >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> > >>> > >>> > >>> >> > From daniel.daugherty at oracle.com Thu Sep 19 01:08:11 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 18 Sep 2019 21:08:11 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <5DE94C8C-B398-48A4-AB0C-2CEE51A99DA0@amazon.com> References: <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> <114ed65b-4b13-22aa-1849-f2737f7a9779@oracle.com> <5DE94C8C-B398-48A4-AB0C-2CEE51A99DA0@amazon.com> Message-ID: <35e2e812-5341-e193-7783-5d45ec77ca89@oracle.com> We should probably repurpose ??? JDK-8231209 Many com.sun.management.ThreadMXBean test failures after 8207266 as your REDO bug. Dan On 9/18/19 9:05 PM, Hohensee, Paul wrote: > +1, thanks! > > My apologies for the bad patch. I'll file another issue and run every test that mentions ThreadMXBean. At least, I know how to revert a patch now. > > Paul > > ?On 9/18/19, 6:00 PM, "David Holmes" wrote: > > Ship it! > > Thanks Dan! > > David > > On 19/09/2019 10:53 am, Daniel D. Daugherty wrote: > > Looks like the issue is different versions of 'hg' in use. > > > > When I import Paul's patch from his webrev using my 'hg' and > > then export it again, it matches my version of the backout. > > > > I have done a mechanical verification that the backout is an > > exact reversal for Paul's original changeset. > > > > I'm planning to push the changeset with the following info: > > > > > > 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() > > can be quicker for self thread > > Reviewed-by: phh, dholmes > > > > Everyone good with this? > > > > Dan > > > > On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: > >> For some reason, the backout that I did does not match the backout > >> that you did so I'm trying to figure that out. > >> > >> Dan > >> > >> > >> > >> On 9/18/19 8:36 PM, Hohensee, Paul wrote: > >>> And I filed 8231211 for the same thing. :) > >>> > >>> Yes, please handle it, because it will go faster since I don't have > >>> access to a fast machine (just my laptop). > >>> > >>> Webrev here: > >>> > >>> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >>> > >>> Thanks, > >>> > >>> On 9/18/19, 5:25 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > >>> I created this sub-task for you: > >>> JDK-8231210 [BACKOUT] JDK-8207266 > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >>> thread > >>> https://bugs.openjdk.java.net/browse/JDK-8231210 > >>> If you would prefer, I can handle this backout for you. > >>> Dan > >>> On 9/18/19 8:21 PM, Hohensee, Paul wrote: > >>> > Never having done this before, is it > >>> > > >>> > hg backout -r > >>> > > >>> > ? Do I file a JBS issue for the reversion? Seems necessary. > >>> > > >>> > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > > >>> > % hg backout > >>> > > >>> > is the usual way to do this... > >>> > > >>> > Dan > >>> > > >>> > > >>> > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > >>> > > Is there a tool that will generate a reversal patch? > >>> > > > >>> > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > > > >>> > > > Shall I go with that, or reverse the original > >>> patch? > >>> > > > >>> > > I'm a bit worried about what else might show up > >>> since the > >>> > > NSK monitoring tests were not run prior to this push. > >>> > > > >>> > > I vote for backing out the fix until proper > >>> testing has > >>> > > been done (and at least the one problem fixed...) > >>> > > > >>> > > Dan > >>> > > > >>> > > > >>> > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > >>> > > > They all implement > >>> com.sun.management.ThreadMXBean, so adding a > >>> getCurrentThreadAllocatedBytes broke them. Potential fix is to give > >>> it a default implementation, vis > >>> > > > > >>> > > > public default long > >>> getCurrentThreadAllocatedBytes() { > >>> > > > return -1; > >>> > > > } > >>> > > > > >>> > > > Shall I go with that, or reverse the original > >>> patch? > >>> > > > > >>> > > > On 9/18/19, 4:48 PM, "serviceability-dev on > >>> behalf of Hohensee, Paul" > >>> >>> hohensee at amazon.com> wrote: > >>> > > > > >>> > > > I'll take a look. > >>> > > > > >>> > > > On 9/18/19, 4:40 PM, "David Holmes" > >>> wrote: > >>> > > > > >>> > > > Paul, > >>> > > > > >>> > > > Unfortunately this patch has broken the > >>> vmTestbase/nsk/monitoring tests: > >>> > > > > >>> > > > [2019-09-18T22:59:32,349Z] > >>> > > > > >>> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > >>> > >>> > > > error: ServerThreadMXBeanNew is not > >>> abstract and does not override > >>> > > > abstract method > >>> getCurrentThreadAllocatedBytes() in ThreadMXBean > >>> > > > > >>> > > > and possibly other issues as we are > >>> seeing hundreds of failures. > >>> > > > > >>> > > > David > >>> > > > > >>> > > > On 18/09/2019 8:50 am, David Holmes wrote: > >>> > > > > On 18/09/2019 12:10 am, Hohensee, > >>> Paul wrote: > >>> > > > >> Thanks, Serguei. :) > >>> > > > >> > >>> > > > >> David, are you ok with the patch? > >>> > > > > > >>> > > > > Yep, nothing further from me. > >>> > > > > > >>> > > > > David > >>> > > > > > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> *From: *"serguei.spitsyn at oracle.com" > >>> > >>> > > > >> *Date: *Tuesday, September 17, 2019 > >>> at 2:26 AM > >>> > > > >> *To: *"Hohensee, Paul" > >>> , David Holmes > >>> > > > >> , Mandy > >>> Chung > >>> > > > >> *Cc: *OpenJDK Serviceability > >>> , > >>> > > > >> "hotspot-gc-dev at openjdk.java.net" > >>> > >>> > > > >> *Subject: *Re: RFR (M): 8207266: > >>> > > > >> > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> Thank you for refactoring and fixing > >>> the test. > >>> > > > >> It looks great now! > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> Serguei > >>> > > > >> > >>> > > > >> > >>> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > >>> > > > >> > >>> > > > >> Hi, Serguei, thanks for the > >>> review. New webrev at > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > >>> > > > >> > >>> > > > >> I refactored the test?s main() > >>> method, and you?re correct, > >>> > > > >> getThreadAllocatedBytes should be > >>> getCurrentThreadAllocatedBytes in > >>> > > > >> that context: fixed. > >>> > > > >> > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> *From: > >>> *"serguei.spitsyn at oracle.com" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> *Organization: *Oracle Corporation > >>> > > > >> *Date: *Friday, September 13, > >>> 2019 at 5:50 PM > >>> > > > >> *To: *"Hohensee, Paul" > >>> > >>> > > > >> , David > >>> Holmes > >>> > > > >> , > >>> Mandy Chung > >>> > > > >> > >>> > >>> > > > >> *Cc: *OpenJDK Serviceability > >>> > >>> > > > >> > >>> , > >>> > > > >> "hotspot-gc-dev at openjdk.java.net" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > >>> > >>> > > > >> *Subject: *Re: RFR (M): 8207266: > >>> > > > >> > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >>> > > > >> thread > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> It looks pretty good in general. > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > >>> > >>> > > > >> > >>> > > > >> > >>> > > > >> It would be nice to refactor the > >>> java main() method as it becomes > >>> > > > >> too big. > >>> > > > >> Two ways > >>> ofgetCurrentThreadAllocatedBytes() testing are good > >>> > > > >> candidates > >>> > > > >> to become separate methods. > >>> > > > >> > >>> > > > >> 98 long size1 = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> Just wanted to double check if > >>> you wanted to invoke > >>> > > > >> the > >>> getCurrentThreadAllocatedBytes() instead as it is > >>> > > > >> a part of: > >>> > > > >> > >>> > > > >> 85 // First way, > >>> getCurrentThreadAllocatedBytes > >>> > > > >> > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> Serguei > >>> > > > >> > >>> > > > >> On 9/13/19 12:11 PM, Hohensee, > >>> Paul wrote: > >>> > > > >> > >>> > > > >> Hi David, thanks for your > >>> comments. New webrev in > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > >>> > > > >> > >>> > > > >> > >>> > > > >> Both the old and new > >>> versions of the code check that thread > >>> > > > >> allocated memory is both supported > >>> and enabled. The existing version > >>> > > > >> of getThreadAllocatedBytes(long []) > >>> calls > >>> > > > >> verifyThreadAllocatedMemory(long > >>> []), which checks inline to make sure > >>> > > > >> thread allocated memory is > >>> supported, then calls > >>> > > > >> isThreadAllocatedMemoryEnabled() to > >>> verify that it's enabled. > >>> > > > >> isThreadAllocatedMemoryEnabled() > >>> duplicates (!) the support check and > >>> > > > >> returns the enabled flag. I removed > >>> the redundant check in the new > >>> > > > >> version. > >>> > > > >> > >>> > > > >> > >>> > > > >> You're of course correct > >>> about the back-to-back check. > >>> > > > >> Application code can't know when the > >>> runtime will hijack a thread for > >>> > > > >> its own purposes. I've removed the > >>> check. > >>> > > > >> > >>> > > > >> > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> > >>> > > > >> On 9/13/19, 12:50 AM, "David > >>> Holmes" > >>> > > > >> wrote: > >>> > > > >> > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> > >>> > > > >> On 13/09/2019 10:29 am, > >>> Hohensee, Paul wrote: > >>> > > > >> > >>> > > > >> > Thanks for clarifying the review > >>> rules. Would someone > >>> > > > >> from the > >>> > > > >> > >>> > > > >> > serviceability team please review? > >>> New webrev at > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > >>> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > >>> > > > >> > >>> > > > >> > >>> > > > >> One aspect of the > >>> functional change needs clarification > >>> > > > >> for me - and > >>> > > > >> > >>> > > > >> apologies if this has been covered > >>> in the past. It seems > >>> > > > >> to me that > >>> > > > >> > >>> > > > >> currently we only check > >>> isThreadAllocatedMemorySupported > >>> > > > >> for these > >>> > > > >> > >>> > > > >> operations, but if I read things > >>> correctly the updated > >>> > > > >> code additionally > >>> > > > >> > >>> > > > >> checks > >>> isThreadAllocatedMemoryEnabled, which is a > >>> > > > >> behaviour change not > >>> > > > >> > >>> > > > >> mentioned in the CSR. > >>> > > > >> > >>> > > > >> > >>> > > > >> > I didn?t disturb the existing > >>> checks in the test, just > >>> > > > >> added code to > >>> > > > >> > >>> > > > >> > check the result of > >>> getThreadAllocatedBytes(long) on a > >>> > > > >> non-current > >>> > > > >> > >>> > > > >> > thread, plus the back-to-back > >>> no-allocation checks. The > >>> > > > >> former wasn?t > >>> > > > >> > >>> > > > >> > needed before because > >>> getThreadAllocatedBytes(long) was > >>> > > > >> just a wrapper > >>> > > > >> > >>> > > > >> > around > >>> getThreadAllocatedBytes(long []). This patch > >>> > > > >> changes that, so I > >>> > > > >> > >>> > > > >> > added a separate test. The latter > >>> is supposed to fail > >>> > > > >> if there?s object > >>> > > > >> > >>> > > > >> > allocation on calls to > >>> getCurrentThreadAllocatedBytes and > >>> > > > >> > >>> > > > >> > getThreadAllocatedBytes(long). > >>> I.e., a feature, not a > >>> > > > >> bug, because > >>> > > > >> > >>> > > > >> > accumulation of transient small > >>> objects can be a > >>> > > > >> performance problem. > >>> > > > >> > >>> > > > >> > Thanks to your review, I noticed > >>> that the back-to-back > >>> > > > >> check on the > >>> > > > >> > >>> > > > >> > current thread was using > >>> getThreadAllocatedBytes(long) > >>> > > > >> instead of > >>> > > > >> > >>> > > > >> > getCurrentThreadAllocatedBytes and > >>> fixed it. I also > >>> > > > >> removed all > >>> > > > >> > >>> > > > >> > instances of ?TEST FAILED: ?. > >>> > > > >> > >>> > > > >> > >>> > > > >> The back-to-back check > >>> is not valid in general. You don't > >>> > > > >> know if the > >>> > > > >> > >>> > > > >> first check might trigger some class > >>> loading on the > >>> > > > >> return path after it > >>> > > > >> > >>> > > > >> has obtained the first > >>> memory value. The check might also > >>> > > > >> fail if using > >>> > > > >> > >>> > > > >> JVMCI and some compilation related > >>> activity occurs in the > >>> > > > >> current thread > >>> > > > >> > >>> > > > >> on the second call. > >>> Also with the introduction of > >>> > > > >> handshakes its > >>> > > > >> > >>> > > > >> possible the current thread might > >>> hit a safepoint checks > >>> > > > >> that results in > >>> > > > >> > >>> > > > >> it executing a > >>> handshake operation that performs > >>> > > > >> allocation. Potentially > >>> > > > >> > >>> > > > >> there could be numerous > >>> non-deterministic actions that > >>> > > > >> might occur > >>> > > > >> > >>> > > > >> leading to unanticipated allocation. > >>> > > > >> > >>> > > > >> > >>> > > > >> I understand what you > >>> want to test here, I just don't > >>> > > > >> think it is > >>> > > > >> > >>> > > > >> reliably doable. > >>> > > > >> > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> > >>> > > > >> David > >>> > > > >> > >>> > > > >> ----- > >>> > > > >> > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Paul > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > *From: *Mandy > >>> Chung > >>> > > > >> > >>> > > > >> > >>> > > > >> > *Date: *Thursday, September 12, > >>> 2019 at 10:09 AM > >>> > > > >> > >>> > > > >> > *To: *"Hohensee, > >>> Paul" > >>> > > > >> > >>> > > > >> > >>> > > > >> > *Cc: *OpenJDK > >>> > > > >> > >>> Serviceability > >>> > > > >> > >>> , > >>> > > > >> > >>> > > > >> >"hotspot-gc-dev at openjdk.java.net" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > *Subject: *Re: RFR (M): 8207266: > >>> > > > >> ThreadMXBean::getThreadAllocatedBytes() > >>> > > > >> > >>> > > > >> > can be quicker for self thread > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul > >>> wrote: > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Minor update in new > >>> > > > >> > >>> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > I only reviewed the library side > >>> implementation that > >>> > > > >> looks good. I > >>> > > > >> > >>> > > > >> > expect the serviceability team to > >>> review the test and > >>> > > > >> hotspot change. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Need a confirmatory review to > >>> push this. If I > >>> > > > >> understand the rules correctly, it > >>> doesn't need a Reviewer review > >>> > > > >> since Mandy's already reviewed it, > >>> it just needs a Committer review. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > You need another reviewer to > >>> advice the following > >>> > > > >> because I was not > >>> > > > >> > >>> > > > >> > close to the ThreadsList work. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2087 ThreadsListHandle tlh; > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2088 JavaThread* java_thread = > >>> > > > >> > >>> tlh.list()->find_JavaThread_from_java_tid(thread_id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2089 > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2090 if (java_thread != NULL) { > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2091 return > >>> java_thread->cooked_allocated_bytes(); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2092 } > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > This looks right to me. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > - "ThreadAllocatedMemory is > >>> expected to > >>> > > > >> be disabled"); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + "TEST FAILED: > >>> ThreadAllocatedMemory is > >>> > > > >> expected to be > >>> > > > >> > >>> > > > >> > disabled"); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Prepending "TEST FAILED" in > >>> exception message (in > >>> > > > >> several places) > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > seems redundant since such > >>> RuntimeException is thrown > >>> > > > >> and expected > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > a test failure. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + // back-to-back calls > >>> shouldn't allocate any > >>> > > > >> memory > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + size = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + size1 = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + if (size1 != size) { > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Is there anything in the test can > >>> do to help guarantee > >>> > > > >> this? I didn't > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > closely review this test. The > >>> main thing I advice is > >>> > > > >> to improve > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > the reliability of this test. Put > >>> it in another way, > >>> > > > >> we want to > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > ensure that this test change will > >>> pass all the time in > >>> > > > >> various > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > test configuration. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Mandy > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> > >> > > > > From hohensee at amazon.com Thu Sep 19 01:09:14 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 01:09:14 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <35e2e812-5341-e193-7783-5d45ec77ca89@oracle.com> References: <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <64069096-23e3-efd5-44cd-2fff0618d28f@oracle.com> <9A151A35-DA66-4B0F-B67F-E7F5BECF205A@amazon.com> <586EB17C-1974-4FEF-A751-34B5B359BA61@amazon.com> <6139732f-a914-c59c-2119-517e480a279c@oracle.com> <8E8AE26F-5A7B-499C-BB6A-5EC954E42BD7@amazon.com> <0d316e76-44cc-4f65-a96d-e1207b1c94e6@oracle.com> <114ed65b-4b13-22aa-1849-f2737f7a9779@oracle.com> <5DE94C8C-B398-48A4-AB0C-2CEE51A99DA0@amazon.com> <35e2e812-5341-e193-7783-5d45ec77ca89@oracle.com> Message-ID: <3234DE48-C3C7-4E92-BE23-B3B638EB8BC4@amazon.com> Good idea, will do. ?On 9/18/19, 6:08 PM, "Daniel D. Daugherty" wrote: We should probably repurpose JDK-8231209 Many com.sun.management.ThreadMXBean test failures after 8207266 as your REDO bug. Dan On 9/18/19 9:05 PM, Hohensee, Paul wrote: > +1, thanks! > > My apologies for the bad patch. I'll file another issue and run every test that mentions ThreadMXBean. At least, I know how to revert a patch now. > > Paul > > On 9/18/19, 6:00 PM, "David Holmes" wrote: > > Ship it! > > Thanks Dan! > > David > > On 19/09/2019 10:53 am, Daniel D. Daugherty wrote: > > Looks like the issue is different versions of 'hg' in use. > > > > When I import Paul's patch from his webrev using my 'hg' and > > then export it again, it matches my version of the backout. > > > > I have done a mechanical verification that the backout is an > > exact reversal for Paul's original changeset. > > > > I'm planning to push the changeset with the following info: > > > > > > 8231210: [BACKOUT] JDK-8207266 ThreadMXBean::getThreadAllocatedBytes() > > can be quicker for self thread > > Reviewed-by: phh, dholmes > > > > Everyone good with this? > > > > Dan > > > > On 9/18/19 8:44 PM, Daniel D. Daugherty wrote: > >> For some reason, the backout that I did does not match the backout > >> that you did so I'm trying to figure that out. > >> > >> Dan > >> > >> > >> > >> On 9/18/19 8:36 PM, Hohensee, Paul wrote: > >>> And I filed 8231211 for the same thing. :) > >>> > >>> Yes, please handle it, because it will go faster since I don't have > >>> access to a fast machine (just my laptop). > >>> > >>> Webrev here: > >>> > >>> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >>> > >>> Thanks, > >>> > >>> On 9/18/19, 5:25 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > >>> I created this sub-task for you: > >>> JDK-8231210 [BACKOUT] JDK-8207266 > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >>> thread > >>> https://bugs.openjdk.java.net/browse/JDK-8231210 > >>> If you would prefer, I can handle this backout for you. > >>> Dan > >>> On 9/18/19 8:21 PM, Hohensee, Paul wrote: > >>> > Never having done this before, is it > >>> > > >>> > hg backout -r > >>> > > >>> > ? Do I file a JBS issue for the reversion? Seems necessary. > >>> > > >>> > On 9/18/19, 5:18 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > > >>> > % hg backout > >>> > > >>> > is the usual way to do this... > >>> > > >>> > Dan > >>> > > >>> > > >>> > On 9/18/19 8:17 PM, Hohensee, Paul wrote: > >>> > > Is there a tool that will generate a reversal patch? > >>> > > > >>> > > On 9/18/19, 5:14 PM, "Daniel D. Daugherty" > >>> wrote: > >>> > > > >>> > > > Shall I go with that, or reverse the original > >>> patch? > >>> > > > >>> > > I'm a bit worried about what else might show up > >>> since the > >>> > > NSK monitoring tests were not run prior to this push. > >>> > > > >>> > > I vote for backing out the fix until proper > >>> testing has > >>> > > been done (and at least the one problem fixed...) > >>> > > > >>> > > Dan > >>> > > > >>> > > > >>> > > On 9/18/19 8:00 PM, Hohensee, Paul wrote: > >>> > > > They all implement > >>> com.sun.management.ThreadMXBean, so adding a > >>> getCurrentThreadAllocatedBytes broke them. Potential fix is to give > >>> it a default implementation, vis > >>> > > > > >>> > > > public default long > >>> getCurrentThreadAllocatedBytes() { > >>> > > > return -1; > >>> > > > } > >>> > > > > >>> > > > Shall I go with that, or reverse the original > >>> patch? > >>> > > > > >>> > > > On 9/18/19, 4:48 PM, "serviceability-dev on > >>> behalf of Hohensee, Paul" > >>> >>> hohensee at amazon.com> wrote: > >>> > > > > >>> > > > I'll take a look. > >>> > > > > >>> > > > On 9/18/19, 4:40 PM, "David Holmes" > >>> wrote: > >>> > > > > >>> > > > Paul, > >>> > > > > >>> > > > Unfortunately this patch has broken the > >>> vmTestbase/nsk/monitoring tests: > >>> > > > > >>> > > > [2019-09-18T22:59:32,349Z] > >>> > > > > >>> /scratch/mesos/jib-master/install/jdk-14+15-615/src.full/open/test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java:32: > >>> > >>> > > > error: ServerThreadMXBeanNew is not > >>> abstract and does not override > >>> > > > abstract method > >>> getCurrentThreadAllocatedBytes() in ThreadMXBean > >>> > > > > >>> > > > and possibly other issues as we are > >>> seeing hundreds of failures. > >>> > > > > >>> > > > David > >>> > > > > >>> > > > On 18/09/2019 8:50 am, David Holmes wrote: > >>> > > > > On 18/09/2019 12:10 am, Hohensee, > >>> Paul wrote: > >>> > > > >> Thanks, Serguei. :) > >>> > > > >> > >>> > > > >> David, are you ok with the patch? > >>> > > > > > >>> > > > > Yep, nothing further from me. > >>> > > > > > >>> > > > > David > >>> > > > > > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> *From: *"serguei.spitsyn at oracle.com" > >>> > >>> > > > >> *Date: *Tuesday, September 17, 2019 > >>> at 2:26 AM > >>> > > > >> *To: *"Hohensee, Paul" > >>> , David Holmes > >>> > > > >> , Mandy > >>> Chung > >>> > > > >> *Cc: *OpenJDK Serviceability > >>> , > >>> > > > >> "hotspot-gc-dev at openjdk.java.net" > >>> > >>> > > > >> *Subject: *Re: RFR (M): 8207266: > >>> > > > >> > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> Thank you for refactoring and fixing > >>> the test. > >>> > > > >> It looks great now! > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> Serguei > >>> > > > >> > >>> > > > >> > >>> > > > >> On 9/15/19 02:52, Hohensee, Paul wrote: > >>> > > > >> > >>> > > > >> Hi, Serguei, thanks for the > >>> review. New webrev at > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.09/ > >>> > > > >> > >>> > > > >> I refactored the test?s main() > >>> method, and you?re correct, > >>> > > > >> getThreadAllocatedBytes should be > >>> getCurrentThreadAllocatedBytes in > >>> > > > >> that context: fixed. > >>> > > > >> > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> *From: > >>> *"serguei.spitsyn at oracle.com" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> *Organization: *Oracle Corporation > >>> > > > >> *Date: *Friday, September 13, > >>> 2019 at 5:50 PM > >>> > > > >> *To: *"Hohensee, Paul" > >>> > >>> > > > >> , David > >>> Holmes > >>> > > > >> , > >>> Mandy Chung > >>> > > > >> > >>> > >>> > > > >> *Cc: *OpenJDK Serviceability > >>> > >>> > > > >> > >>> , > >>> > > > >> "hotspot-gc-dev at openjdk.java.net" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > >>> > >>> > > > >> *Subject: *Re: RFR (M): 8207266: > >>> > > > >> > >>> ThreadMXBean::getThreadAllocatedBytes() can be quicker for self > >>> > > > >> thread > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> It looks pretty good in general. > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java.frames.html > >>> > >>> > > > >> > >>> > > > >> > >>> > > > >> It would be nice to refactor the > >>> java main() method as it becomes > >>> > > > >> too big. > >>> > > > >> Two ways > >>> ofgetCurrentThreadAllocatedBytes() testing are good > >>> > > > >> candidates > >>> > > > >> to become separate methods. > >>> > > > >> > >>> > > > >> 98 long size1 = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> Just wanted to double check if > >>> you wanted to invoke > >>> > > > >> the > >>> getCurrentThreadAllocatedBytes() instead as it is > >>> > > > >> a part of: > >>> > > > >> > >>> > > > >> 85 // First way, > >>> getCurrentThreadAllocatedBytes > >>> > > > >> > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> Serguei > >>> > > > >> > >>> > > > >> On 9/13/19 12:11 PM, Hohensee, > >>> Paul wrote: > >>> > > > >> > >>> > > > >> Hi David, thanks for your > >>> comments. New webrev in > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> http://cr.openjdk.java.net/~phh/8207266/webrev.08/ > >>> > > > >> > >>> > > > >> > >>> > > > >> Both the old and new > >>> versions of the code check that thread > >>> > > > >> allocated memory is both supported > >>> and enabled. The existing version > >>> > > > >> of getThreadAllocatedBytes(long []) > >>> calls > >>> > > > >> verifyThreadAllocatedMemory(long > >>> []), which checks inline to make sure > >>> > > > >> thread allocated memory is > >>> supported, then calls > >>> > > > >> isThreadAllocatedMemoryEnabled() to > >>> verify that it's enabled. > >>> > > > >> isThreadAllocatedMemoryEnabled() > >>> duplicates (!) the support check and > >>> > > > >> returns the enabled flag. I removed > >>> the redundant check in the new > >>> > > > >> version. > >>> > > > >> > >>> > > > >> > >>> > > > >> You're of course correct > >>> about the back-to-back check. > >>> > > > >> Application code can't know when the > >>> runtime will hijack a thread for > >>> > > > >> its own purposes. I've removed the > >>> check. > >>> > > > >> > >>> > > > >> > >>> > > > >> Paul > >>> > > > >> > >>> > > > >> > >>> > > > >> On 9/13/19, 12:50 AM, "David > >>> Holmes" > >>> > > > >> wrote: > >>> > > > >> > >>> > > > >> > >>> > > > >> Hi Paul, > >>> > > > >> > >>> > > > >> > >>> > > > >> On 13/09/2019 10:29 am, > >>> Hohensee, Paul wrote: > >>> > > > >> > >>> > > > >> > Thanks for clarifying the review > >>> rules. Would someone > >>> > > > >> from the > >>> > > > >> > >>> > > > >> > serviceability team please review? > >>> New webrev at > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > >>> >http://cr.openjdk.java.net/~phh/8207266/webrev.07/ > >>> > > > >> > >>> > > > >> > >>> > > > >> One aspect of the > >>> functional change needs clarification > >>> > > > >> for me - and > >>> > > > >> > >>> > > > >> apologies if this has been covered > >>> in the past. It seems > >>> > > > >> to me that > >>> > > > >> > >>> > > > >> currently we only check > >>> isThreadAllocatedMemorySupported > >>> > > > >> for these > >>> > > > >> > >>> > > > >> operations, but if I read things > >>> correctly the updated > >>> > > > >> code additionally > >>> > > > >> > >>> > > > >> checks > >>> isThreadAllocatedMemoryEnabled, which is a > >>> > > > >> behaviour change not > >>> > > > >> > >>> > > > >> mentioned in the CSR. > >>> > > > >> > >>> > > > >> > >>> > > > >> > I didn?t disturb the existing > >>> checks in the test, just > >>> > > > >> added code to > >>> > > > >> > >>> > > > >> > check the result of > >>> getThreadAllocatedBytes(long) on a > >>> > > > >> non-current > >>> > > > >> > >>> > > > >> > thread, plus the back-to-back > >>> no-allocation checks. The > >>> > > > >> former wasn?t > >>> > > > >> > >>> > > > >> > needed before because > >>> getThreadAllocatedBytes(long) was > >>> > > > >> just a wrapper > >>> > > > >> > >>> > > > >> > around > >>> getThreadAllocatedBytes(long []). This patch > >>> > > > >> changes that, so I > >>> > > > >> > >>> > > > >> > added a separate test. The latter > >>> is supposed to fail > >>> > > > >> if there?s object > >>> > > > >> > >>> > > > >> > allocation on calls to > >>> getCurrentThreadAllocatedBytes and > >>> > > > >> > >>> > > > >> > getThreadAllocatedBytes(long). > >>> I.e., a feature, not a > >>> > > > >> bug, because > >>> > > > >> > >>> > > > >> > accumulation of transient small > >>> objects can be a > >>> > > > >> performance problem. > >>> > > > >> > >>> > > > >> > Thanks to your review, I noticed > >>> that the back-to-back > >>> > > > >> check on the > >>> > > > >> > >>> > > > >> > current thread was using > >>> getThreadAllocatedBytes(long) > >>> > > > >> instead of > >>> > > > >> > >>> > > > >> > getCurrentThreadAllocatedBytes and > >>> fixed it. I also > >>> > > > >> removed all > >>> > > > >> > >>> > > > >> > instances of ?TEST FAILED: ?. > >>> > > > >> > >>> > > > >> > >>> > > > >> The back-to-back check > >>> is not valid in general. You don't > >>> > > > >> know if the > >>> > > > >> > >>> > > > >> first check might trigger some class > >>> loading on the > >>> > > > >> return path after it > >>> > > > >> > >>> > > > >> has obtained the first > >>> memory value. The check might also > >>> > > > >> fail if using > >>> > > > >> > >>> > > > >> JVMCI and some compilation related > >>> activity occurs in the > >>> > > > >> current thread > >>> > > > >> > >>> > > > >> on the second call. > >>> Also with the introduction of > >>> > > > >> handshakes its > >>> > > > >> > >>> > > > >> possible the current thread might > >>> hit a safepoint checks > >>> > > > >> that results in > >>> > > > >> > >>> > > > >> it executing a > >>> handshake operation that performs > >>> > > > >> allocation. Potentially > >>> > > > >> > >>> > > > >> there could be numerous > >>> non-deterministic actions that > >>> > > > >> might occur > >>> > > > >> > >>> > > > >> leading to unanticipated allocation. > >>> > > > >> > >>> > > > >> > >>> > > > >> I understand what you > >>> want to test here, I just don't > >>> > > > >> think it is > >>> > > > >> > >>> > > > >> reliably doable. > >>> > > > >> > >>> > > > >> > >>> > > > >> Thanks, > >>> > > > >> > >>> > > > >> David > >>> > > > >> > >>> > > > >> ----- > >>> > > > >> > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Paul > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > *From: *Mandy > >>> Chung > >>> > > > >> > >>> > > > >> > >>> > > > >> > *Date: *Thursday, September 12, > >>> 2019 at 10:09 AM > >>> > > > >> > >>> > > > >> > *To: *"Hohensee, > >>> Paul" > >>> > > > >> > >>> > > > >> > >>> > > > >> > *Cc: *OpenJDK > >>> > > > >> > >>> Serviceability > >>> > > > >> > >>> , > >>> > > > >> > >>> > > > >> >"hotspot-gc-dev at openjdk.java.net" > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > >>> > >>> > > > >> > >>> > > > >> > *Subject: *Re: RFR (M): 8207266: > >>> > > > >> ThreadMXBean::getThreadAllocatedBytes() > >>> > > > >> > >>> > > > >> > can be quicker for self thread > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > On 9/3/19 12:38 PM, Hohensee, Paul > >>> wrote: > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Minor update in new > >>> > > > >> > >>> webrevhttp://cr.openjdk.java.net/~phh/8207266/webrev.05/. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > I only reviewed the library side > >>> implementation that > >>> > > > >> looks good. I > >>> > > > >> > >>> > > > >> > expect the serviceability team to > >>> review the test and > >>> > > > >> hotspot change. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Need a confirmatory review to > >>> push this. If I > >>> > > > >> understand the rules correctly, it > >>> doesn't need a Reviewer review > >>> > > > >> since Mandy's already reviewed it, > >>> it just needs a Committer review. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > You need another reviewer to > >>> advice the following > >>> > > > >> because I was not > >>> > > > >> > >>> > > > >> > close to the ThreadsList work. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2087 ThreadsListHandle tlh; > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2088 JavaThread* java_thread = > >>> > > > >> > >>> tlh.list()->find_JavaThread_from_java_tid(thread_id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2089 > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2090 if (java_thread != NULL) { > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2091 return > >>> java_thread->cooked_allocated_bytes(); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > 2092 } > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > This looks right to me. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> test/jdk/com/sun/management/ThreadMXBean/ThreadAllocatedMemory.java > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > - "ThreadAllocatedMemory is > >>> expected to > >>> > > > >> be disabled"); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + "TEST FAILED: > >>> ThreadAllocatedMemory is > >>> > > > >> expected to be > >>> > > > >> > >>> > > > >> > disabled"); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Prepending "TEST FAILED" in > >>> exception message (in > >>> > > > >> several places) > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > seems redundant since such > >>> RuntimeException is thrown > >>> > > > >> and expected > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > a test failure. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + // back-to-back calls > >>> shouldn't allocate any > >>> > > > >> memory > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + size = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + size1 = > >>> mbean.getThreadAllocatedBytes(id); > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > + if (size1 != size) { > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Is there anything in the test can > >>> do to help guarantee > >>> > > > >> this? I didn't > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > closely review this test. The > >>> main thing I advice is > >>> > > > >> to improve > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > the reliability of this test. Put > >>> it in another way, > >>> > > > >> we want to > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > ensure that this test change will > >>> pass all the time in > >>> > > > >> various > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > test configuration. > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > Mandy > >>> > > > >> > >>> > > > >> > > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> > >> > > > > From mandy.chung at oracle.com Thu Sep 19 02:57:18 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 18 Sep 2019 19:57:18 -0700 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: On 9/18/19 5:00 PM, Hohensee, Paul wrote: > They all implement com.sun.management.ThreadMXBean, so adding a getCurrentThreadAllocatedBytes broke them. Potential fix is to give it a default implementation, vis > > public default long getCurrentThreadAllocatedBytes() { > return -1; > } > com.sun.management.ThreadMXBean (and other platform MXBeans) is a "sealed" interface which should only be implemented by JDK. Unfortunately we don't have the sealed type feature yet.? Yes it needs to be a default method.? I think it should throw UOE. ???? * @implSpec ???? * The default implementation throws {@code UnsupportedOperationException}. The @throw UOE can make it clear that it does not support current thread memory allocation measurement. Mandy From david.holmes at oracle.com Thu Sep 19 03:15:39 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 13:15:39 +1000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: On 19/09/2019 12:57 pm, Mandy Chung wrote: > On 9/18/19 5:00 PM, Hohensee, Paul wrote: >> They all implement com.sun.management.ThreadMXBean, so adding a >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it >> a default implementation, vis >> >> ???? public default long getCurrentThreadAllocatedBytes() { >> ???????? return -1; >> ???? } >> > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > "sealed" interface which should only be implemented by JDK. Didn't realize that. I don't recall knowing about PlatformManagedObject. Sealed types will at least allow this to be enforced, though I have to wonder what the tests are doing here. > Unfortunately we don't have the sealed type feature yet.? Yes it needs > to be a default method.? I think it should throw UOE. > > ???? * @implSpec > ???? * The default implementation throws {@code > UnsupportedOperationException}. > > The @throw UOE can make it clear that it does not support current thread > memory allocation measurement. Yes that seems a reasonable default if we don't want this to be implemented outside the platform. Thanks, David > Mandy From chris.plummer at oracle.com Thu Sep 19 04:44:12 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 18 Sep 2019 21:44:12 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: I got this to work, although it increased the test time from about 30s to over 3m. I looked into it a bit and it appears to be due to the size of the generate hprof file. It used to be about 300k, but now is nearly 7mb. I guess that's because jshell has been run for longer and probably allocated more data. I need to do some more testing and a bit of cleanup. I'll get another review out tomorrow. Chris Suddenly the test time is taking On 9/18/19 5:43 PM, Chris Plummer wrote: > Ok. It was a bit unclear to me why the author went with Runtime.exec() > in the first place. I'll try ProcessTools. That will probably however, > hide 8230872. I might need to write another test for it. > > Chris > > On 9/18/19 4:29 PM, Alex Menkov wrote: >> You can use jdk.test.lib classes to simplify the things. >> Something like >> >> ProcessBuilder pb = new >> ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); >> Process p = ProcessTools.startProcess("JShell", pb, >> ??? s -> {? // warm-up predicate >> ??????? return s.contains(">jshell"); >> ??? }); >> >> --alex >> >> On 09/18/2019 15:44, Chris Plummer wrote: >>> Is there an easy way of doing this? Currently the jshell process is >>> just spawned using Runtime.exec(). >>> >>> Chris >>> >>> On 9/18/19 3:01 PM, Alex Menkov wrote: >>>> Hi Chris, >>>> >>>> Did you think about waiting for jshell prompt ("jshell>") before >>>> run "jhsdb jmap" command instead of delay or re-tries? >>>> >>>> --alex >>>> >>>> On 09/18/2019 14:11, Chris Plummer wrote: >>>>> Hello, >>>>> >>>>> Please review the following changes: >>>>> >>>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>>> >>>>> There are actually numerous ways that JShellHeapDumpTest.java >>>>> fails. One is a test bug, being addressed here, and the rest all >>>>> seem to be SA bugs. Those are now being covered by JDK-8230872. >>>>> All the issues seem to stem from the fact that the test spawns a >>>>> jshell process, and then immediately does a "jhsdb jmap" on the >>>>> process before jshell has fully started up. >>>>> >>>>> The test bug happens when the jmap succeeds, but jshell has not >>>>> yet entered the main java thread. Thus the search for >>>>> "JShellToolProvider" in the output fails. It expects >>>>> "JShellToolProvider" to be in the output because it is part of a >>>>> method name in the main thread, and the test dump all the thread >>>>> stacks contained in the jmap generated hprof file. When the test >>>>> fails in this way, you can see the stack dump in the output, but >>>>> the main thread is missing. >>>>> >>>>> There's a couple of ways to fix this. One is to just add a delay >>>>> (10s seems to be more than enough), and the other is to retry the >>>>> "jhsdb jmap" command until the stack contains the >>>>> JShellToolProvider symbol. I chose the later because doing a 10s >>>>> delay masks the SA issues that are now covered by JDK-8230872. In >>>>> a way the 10s delay is a better fix, because it makes this test >>>>> pass every time, but I did not like that it also hid real SA >>>>> problems in JDK-8230872. My plan for now is to do this retry fix, >>>>> and then if there are too many failures due to JDK-8230872, then >>>>> also add a 10s delay, with the intention of removing it once >>>>> JDK-8230872 if fixed. From what I can see, JDK-8230872 failures >>>>> happen on about 1% of the runs. >>>>> >>>>> I made a few of other changes. One was to no longer redirect >>>>> stderr from the jmap process as was done from the following: >>>>> >>>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>>> >>>>> This causes the output not to appear in the OutputAnalyzer output, >>>>> resulting in the following not working: >>>>> >>>>> ???????????? output.shouldNotContain("null"); >>>>> >>>>> Also I added code to dump the output of the jshell process so you >>>>> can see if the jshell prompt was ever generated. >>>>> >>>>> thanks, >>>>> >>>>> Chris >>>>> >>> >>> > > From christoph.langer at sap.com Thu Sep 19 09:47:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 19 Sep 2019 09:47:16 +0000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Message-ID: Hi, @Erik, Magnus: Thanks for stepping in to explain things. Now back to the actual change: Is this ok then (@David)? Any other reviews from somebody else? http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ Thank you! Best regards Christoph > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 18. September 2019 01:13 > To: Erik Joelsson ; Magnus Ihse Bursie > ; Langer, Christoph > ; OpenJDK Serviceability dev at openjdk.java.net>; build-dev > Subject: Re: RFR: 8230857: Avoid reflection in > sun.tools.common.ProcessHelper > > Hi Erik, > > Thanks for the additional details (I can't say I fully understand them :) ). > > David > > On 17/09/2019 11:39 pm, Erik Joelsson wrote: > > Hello, > > > > On 2019-09-17 05:59, David Holmes wrote: > >> Hi Magnus, > >> > >> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: > >>> On 2019-09-17 01:01, David Holmes wrote: > >>>> Hi Christoph, > >>>> > >>>> Sorry for the delay getting back you. > >>>> > >>>> cc'd build-dev to get some clarification on the below ... > >>>> > >>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: > >>>>> Hi David, > >>>>> > >>>>>>> please review an enhancement which I've identified when working > with > >>>>>>> Processhelper for JDK-8230850. > >>>>>>> > >>>>>>> I noticed that ProcessHelper is an interface in common code with a > >>>>>>> static method that would lookup the actual platform > >>>>>>> implementation via > >>>>>>> reflection. This seems a little cumbersome since we can have a > >>>>>>> common > >>>>>>> dummy for ProcessHelper and override it with the platform specific > >>>>>>> implementation, leveraging the build system. > >>>>>> > >>>>>> I don't see you leveraging the build system. You have two source > >>>>>> files > >>>>>> that compile to the same destination class file. What is ensuring the > >>>>>> platform specific version is compiled after the generic one? > >>>>>> > >>>>>> Service-provider patterns use reflection to instantiate the service > >>>>>> implementation. I don't see any problem here that needs solving. > >>>>> > >>>>> TL;DR: > >>>>> There are two source files, one in share/classes and one in > >>>>> linux/classes. The build system overrides the share/classes > >>>>> implementation with the linux/classes implementation in the linux > >>>>> build. This is not by coincidence and only one class is contained > >>>>> in the generated jdk.jcmd module. Then there won't be a need for > >>>>> having a service interface and a service implementation that is > >>>>> looked up via reflection (which is not a bad pattern by itself). I > >>>>> agree that it's not a big problem to be solved but still not "no > >>>>> problem". > >>>>> Here is some longer elaboration how the build system prefers > >>>>> specific implementations of classes and filters generic duplicates: > >>>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] is > >>>>> used to compile the java sources for JDK modules. In its > >>>>> documentation, for argument SRC [1], it claims: "one or more > >>>>> directories to search for sources. The order of the source roots is > >>>>> significant. The first found file of a certain name has priority". > >>>>> In its implementation the found files are first ordered [3] and > >>>>> duplicates filtered out [4]. > >>>>> The potential source files are handed to SetupJavaCompilation in > >>>>> CompileJavaModules.gmk [5] and were collected by a call to > >>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all > >>>>> potential source dirs for Java classes in the module [7]. The > >>>>> evaluated subdirs are (in that order) > $(OPENJDK_TARGET_OS)/classes, > >>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. > >>>>> Hope that explains what I'm trying to leverage here. > >>>> > >>>> I'm not 100% certain that what you describe actually ensures what > >>>> you want it to ensure. I can't reconcile "the first found file ... > >>>> has priority" with the fact found files are sorted and duplicates > >>>> eliminated. It is the sorting that concerns me as it suggests > >>>> linux/Foo.java might replace shared/Foo.java, but if we're on > >>>> Windows then we have a problem! That said there is also this > comment: > >>>> > >>>> # Order src files according to the order of the src dirs. Correct > >>>> odering is > >>>> # needed for correct overriding between different source roots. > >>>> > >>>> I'd need the build team to clarify what "correct overriding" is > >>>> actually defined as. > >>> David, > >>> > >>> Christoph is correct. linux/Foo.java will override share/Foo.java. I > >>> don't remember how the magic in JavaCompilation.gmk works anymore > >>> :-), but we have relied on this behavior in other places for a long > >>> time, so I'm pretty certain it is still working correctly. > >>> Presumably, the $(sort ...) is there to remove (identical) > >>> duplicates, which is a side-effect of sort. > >> > >> Thanks for confirming. I'd still like to understand exactly what these > >> overriding rules are though. It's not a mechanism I was aware of. > >> > > SetupJavaCompilation is indeed behaving as Christoph describes and it is > > by design. I implemented support for this behavior in: > > > > https://bugs.openjdk.java.net/browse/JDK-8079344 > > > > The relevant parts of SetupJavaCompilation look like this: > > > > ? # Order src files according to the order of the src dirs. Correct > > odering is > > ? # needed for correct overriding between different source roots. > > ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) > > ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ > > ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) > > > > The second line orders the src files by the src roots. (We used to just > > call find for one src root at a time, but the above actually performs > > better due only running 1 external process) > > > > Further down we have this: > > > > ? ifneq ($$($1_KEEP_DUPS), true) > > ??? # Remove duplicate source files by keeping the first found of each > > duplicate. > > ??? # This allows for automatic overrides with custom or platform > > specific versions > > ??? # source files. > > ??? # > > ??? # For the smart javac wrapper case, add each removed file to an > > extra exclude > > ??? # file list to prevent sjavac from finding duplicate sources. > > ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ > > ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), > > $$(s))) \ > > ??????? $$(if $$($1_$$(relative_src)), \ > > ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ > > ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) > > ? endif > > > > This loop is a bit hairy to wrap your head around. It's iterating over > > all the src files, in the order of importance. The variable relative_src > > is the path from the src root, the part that is common to all duplicate > > src files. The variables on the form $1_$$(relative_src) basically act > > as a hash map (string->boolean). So for each src file, if the relative > > path for it has already been seen, add it to an exclude list, else mark > > it as seen and add it to the return list. > > > > /Erik > > From david.holmes at oracle.com Thu Sep 19 11:56:13 2019 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Sep 2019 21:56:13 +1000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Message-ID: Hi Christoph, On 19/09/2019 7:47 pm, Langer, Christoph wrote: > Hi, > > @Erik, Magnus: Thanks for stepping in to explain things. > > Now back to the actual change: Is this ok then (@David)? Any other reviews from somebody else? > > http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ It seems okay. For the test I'm unclear on exactly how to ensure things are accessible, but presumably the +open is sufficient and works under all circumstances. Thanks, David > Thank you! > > Best regards > Christoph > >> -----Original Message----- >> From: David Holmes >> Sent: Mittwoch, 18. September 2019 01:13 >> To: Erik Joelsson ; Magnus Ihse Bursie >> ; Langer, Christoph >> ; OpenJDK Serviceability > dev at openjdk.java.net>; build-dev >> Subject: Re: RFR: 8230857: Avoid reflection in >> sun.tools.common.ProcessHelper >> >> Hi Erik, >> >> Thanks for the additional details (I can't say I fully understand them :) ). >> >> David >> >> On 17/09/2019 11:39 pm, Erik Joelsson wrote: >>> Hello, >>> >>> On 2019-09-17 05:59, David Holmes wrote: >>>> Hi Magnus, >>>> >>>> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: >>>>> On 2019-09-17 01:01, David Holmes wrote: >>>>>> Hi Christoph, >>>>>> >>>>>> Sorry for the delay getting back you. >>>>>> >>>>>> cc'd build-dev to get some clarification on the below ... >>>>>> >>>>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: >>>>>>> Hi David, >>>>>>> >>>>>>>>> please review an enhancement which I've identified when working >> with >>>>>>>>> Processhelper for JDK-8230850. >>>>>>>>> >>>>>>>>> I noticed that ProcessHelper is an interface in common code with a >>>>>>>>> static method that would lookup the actual platform >>>>>>>>> implementation via >>>>>>>>> reflection. This seems a little cumbersome since we can have a >>>>>>>>> common >>>>>>>>> dummy for ProcessHelper and override it with the platform specific >>>>>>>>> implementation, leveraging the build system. >>>>>>>> >>>>>>>> I don't see you leveraging the build system. You have two source >>>>>>>> files >>>>>>>> that compile to the same destination class file. What is ensuring the >>>>>>>> platform specific version is compiled after the generic one? >>>>>>>> >>>>>>>> Service-provider patterns use reflection to instantiate the service >>>>>>>> implementation. I don't see any problem here that needs solving. >>>>>>> >>>>>>> TL;DR: >>>>>>> There are two source files, one in share/classes and one in >>>>>>> linux/classes. The build system overrides the share/classes >>>>>>> implementation with the linux/classes implementation in the linux >>>>>>> build. This is not by coincidence and only one class is contained >>>>>>> in the generated jdk.jcmd module. Then there won't be a need for >>>>>>> having a service interface and a service implementation that is >>>>>>> looked up via reflection (which is not a bad pattern by itself). I >>>>>>> agree that it's not a big problem to be solved but still not "no >>>>>>> problem". >>>>>>> Here is some longer elaboration how the build system prefers >>>>>>> specific implementations of classes and filters generic duplicates: >>>>>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] is >>>>>>> used to compile the java sources for JDK modules. In its >>>>>>> documentation, for argument SRC [1], it claims: "one or more >>>>>>> directories to search for sources. The order of the source roots is >>>>>>> significant. The first found file of a certain name has priority". >>>>>>> In its implementation the found files are first ordered [3] and >>>>>>> duplicates filtered out [4]. >>>>>>> The potential source files are handed to SetupJavaCompilation in >>>>>>> CompileJavaModules.gmk [5] and were collected by a call to >>>>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all >>>>>>> potential source dirs for Java classes in the module [7]. The >>>>>>> evaluated subdirs are (in that order) >> $(OPENJDK_TARGET_OS)/classes, >>>>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per [8]. >>>>>>> Hope that explains what I'm trying to leverage here. >>>>>> >>>>>> I'm not 100% certain that what you describe actually ensures what >>>>>> you want it to ensure. I can't reconcile "the first found file ... >>>>>> has priority" with the fact found files are sorted and duplicates >>>>>> eliminated. It is the sorting that concerns me as it suggests >>>>>> linux/Foo.java might replace shared/Foo.java, but if we're on >>>>>> Windows then we have a problem! That said there is also this >> comment: >>>>>> >>>>>> # Order src files according to the order of the src dirs. Correct >>>>>> odering is >>>>>> # needed for correct overriding between different source roots. >>>>>> >>>>>> I'd need the build team to clarify what "correct overriding" is >>>>>> actually defined as. >>>>> David, >>>>> >>>>> Christoph is correct. linux/Foo.java will override share/Foo.java. I >>>>> don't remember how the magic in JavaCompilation.gmk works anymore >>>>> :-), but we have relied on this behavior in other places for a long >>>>> time, so I'm pretty certain it is still working correctly. >>>>> Presumably, the $(sort ...) is there to remove (identical) >>>>> duplicates, which is a side-effect of sort. >>>> >>>> Thanks for confirming. I'd still like to understand exactly what these >>>> overriding rules are though. It's not a mechanism I was aware of. >>>> >>> SetupJavaCompilation is indeed behaving as Christoph describes and it is >>> by design. I implemented support for this behavior in: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8079344 >>> >>> The relevant parts of SetupJavaCompilation look like this: >>> >>> ? # Order src files according to the order of the src dirs. Correct >>> odering is >>> ? # needed for correct overriding between different source roots. >>> ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) >>> ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ >>> ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) >>> >>> The second line orders the src files by the src roots. (We used to just >>> call find for one src root at a time, but the above actually performs >>> better due only running 1 external process) >>> >>> Further down we have this: >>> >>> ? ifneq ($$($1_KEEP_DUPS), true) >>> ??? # Remove duplicate source files by keeping the first found of each >>> duplicate. >>> ??? # This allows for automatic overrides with custom or platform >>> specific versions >>> ??? # source files. >>> ??? # >>> ??? # For the smart javac wrapper case, add each removed file to an >>> extra exclude >>> ??? # file list to prevent sjavac from finding duplicate sources. >>> ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ >>> ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), >>> $$(s))) \ >>> ??????? $$(if $$($1_$$(relative_src)), \ >>> ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ >>> ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) >>> ? endif >>> >>> This loop is a bit hairy to wrap your head around. It's iterating over >>> all the src files, in the order of importance. The variable relative_src >>> is the path from the src root, the part that is common to all duplicate >>> src files. The variables on the form $1_$$(relative_src) basically act >>> as a hash map (string->boolean). So for each src file, if the relative >>> path for it has already been seen, add it to an exclude list, else mark >>> it as seen and add it to the return list. >>> >>> /Erik >>> From hohensee at amazon.com Thu Sep 19 13:17:09 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 13:17:09 +0000 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <588a91ec-8d4a-1157-5d72-88bb1eef1e6e@oracle.com> <30EA5D0C-1AEC-4242-B17B-CA4D39ECAF71@amazon.com> <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> Message-ID: <98E2CF97-D8C0-4956-9EA4-76FC4F0CBFCA@amazon.com> I'll have the default method throw UOE. That's the same as the other default methods do. The necessary test fix is in test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, which needs a new getCurrentThreadAllocatedBytes method, defined as public long getCurrentThreadAllocatedBytes() { return (Long) invokeMethod("getCurrentThreadAllocatedBytes", new Object[] { }, new String[] { }); } With this fix, the 134 tests in test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. Preliminary webrev at http://cr.openjdk.java.net/~phh/8231211/webrev.00/ Is it worth adding getCurrentThreadAllocatedBytes tests to the test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes set? Paul ?On 9/18/19, 8:16 PM, "David Holmes" wrote: On 19/09/2019 12:57 pm, Mandy Chung wrote: > On 9/18/19 5:00 PM, Hohensee, Paul wrote: >> They all implement com.sun.management.ThreadMXBean, so adding a >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it >> a default implementation, vis >> >> public default long getCurrentThreadAllocatedBytes() { >> return -1; >> } >> > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > "sealed" interface which should only be implemented by JDK. Didn't realize that. I don't recall knowing about PlatformManagedObject. Sealed types will at least allow this to be enforced, though I have to wonder what the tests are doing here. > Unfortunately we don't have the sealed type feature yet. Yes it needs > to be a default method. I think it should throw UOE. > > * @implSpec > * The default implementation throws {@code > UnsupportedOperationException}. > > The @throw UOE can make it clear that it does not support current thread > memory allocation measurement. Yes that seems a reasonable default if we don't want this to be implemented outside the platform. Thanks, David > Mandy From daniel.daugherty at oracle.com Thu Sep 19 13:31:05 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 19 Sep 2019 09:31:05 -0400 Subject: RFR (M): 8207266: ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <98E2CF97-D8C0-4956-9EA4-76FC4F0CBFCA@amazon.com> References: <0d42d653-d158-a6e4-45b6-84f087c7e592@oracle.com> <03A2509C-5587-448A-82F8-9240EA040326@amazon.com> <6f674d71-58f6-bc79-7d08-7bcc24e3b0fa@oracle.com> <5252a51d-4217-000b-1444-a088bb8a6a58@oracle.com> <873119A8-C595-4B73-AD0B-1625D6CAC47D@amazon.com> <56ea7c5f-8c91-9a05-6d95-255bfd0c154d@oracle.com> <5417BEA4-AECD-4130-B269-19847C0092B3@amazon.com> <1561d09b-68ff-55fa-128a-045798a3d6a9@oracle.com> <6732ee43-532f-377d-f37c-fe4e0f9becfe@oracle.com> <93FFE1B3-C1BA-4568-9402-48EB74BB089B@amazon.com> <98E2CF97-D8C0-4956-9EA4-76FC4F0CBFCA@amazon.com> Message-ID: <6924431f-87df-732b-43ac-2df931d384d2@oracle.com> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ The redo bug is 8231209. 8231211 is closed as a dup of 8231210. Dan On 9/19/19 9:17 AM, Hohensee, Paul wrote: > I'll have the default method throw UOE. That's the same as the other default methods do. > > The necessary test fix is in test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, which needs a new getCurrentThreadAllocatedBytes method, defined as > > public long getCurrentThreadAllocatedBytes() { > return (Long) invokeMethod("getCurrentThreadAllocatedBytes", > new Object[] { }, > new String[] { }); > } > > With this fix, the 134 tests in test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. Preliminary webrev at > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > Is it worth adding getCurrentThreadAllocatedBytes tests to the test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes set? > > Paul > > ?On 9/18/19, 8:16 PM, "David Holmes" wrote: > > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> They all implement com.sun.management.ThreadMXBean, so adding a > >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it > >> a default implementation, vis > >> > >> public default long getCurrentThreadAllocatedBytes() { > >> return -1; > >> } > >> > > > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > > "sealed" interface which should only be implemented by JDK. > > Didn't realize that. I don't recall knowing about PlatformManagedObject. > Sealed types will at least allow this to be enforced, though I have to > wonder what the tests are doing here. > > > Unfortunately we don't have the sealed type feature yet. Yes it needs > > to be a default method. I think it should throw UOE. > > > > * @implSpec > > * The default implementation throws {@code > > UnsupportedOperationException}. > > > > The @throw UOE can make it clear that it does not support current thread > > memory allocation measurement. > > Yes that seems a reasonable default if we don't want this to be > implemented outside the platform. > > Thanks, > David > > > Mandy > > From richard.reingruber at sap.com Thu Sep 19 16:42:42 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Thu, 19 Sep 2019 16:42:42 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi David, thanks for looking at this issue. And my appologies for the lengthy mail... > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > optimizations like scalar replacement become invalid. > > What bothers me about this is that it seems like escape analysis is > doing something wrong in this case. Yes it is. > If the object is thread-local but is > being synchronized upon then either: The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis does not recognize this. That's what it is doing wrong. Consequently the state of the virtual machine, as observed through JVMTI, is wrong. See below... > a) the synchronization is elided and so the object will not appear in > the set of owned monitors; or > b) the fact synchronization occurs renders the object ineligible to be > considered thread-local, and so there is no problem with it appearing in > the set of owned monitors > > I think there is a bigger design philosophy issue here about the > interaction of escape analysis and debugging/management frameworks in > general. I'd like to see a very clear description on exactly how they > should interact. > I don't see too many design alternatives here. The JVMTI implementation has to present the correct state of the virtual machine according to the spec. I think it fails to do so in this case. Please look again at the test: 172 public long dontinline_endlessLoop() { 173 long cs = checkSum; 174 while (doLoop && loopCount-- > 0) { 175 targetIsInLoop = true; 176 checkSum += checkSum % ++cs; 177 } 178 loopCount = 3; 179 targetIsInLoop = false; 180 return checkSum; 181 } 249 public void dontinline_testMethod() { 250 LockCls l1 = new LockCls(); // to be scalar replaced 251 synchronized (l1) { 252 inlinedTestMethodWithNestedLocking(l1); 253 } 254 } 255 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { 257 synchronized (l1) { // nested 258 dontinline_endlessLoop(); 259 } 260 } This is the stack when the agent calls GetOwnedMonitorInfo() dontinline_endlessLoop() at line 176 inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame dontinline_testMethod() at line 252 // compiled frame The state of the _virtual_ machine at that point is obvious: - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. - That instance was locked by a monitorenter bytecode at line 251 This could be proven by interpreting the execution trace bytecode by bytecode using paper and pencil (hope you won't make me do this, though ;)) JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does not. From a more philosophical point of view compiled code is free to change the state of the physical machine in a way such that it cannot be mapped to a valid state of the virtual machine after each and every machine instruction. But it must reach points in its execution trace, where it is actually possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These points are called safepoints. The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm state is presented. EA does not take into account that the lock state can be observed using GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability can_get_owned_monitor_info is taken. With the fix the test passes. Note that for the very same reason EA is disabled if can_access_local_variables is taken, because the JVMTI implementation cannot handout references to objects stored in local variables if they were scalar replaced. With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA based optimizations just-in-time before local objects escape. Note that EA opts are already reverted today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() and relock_objects() in class Deoptimization) Thanks and cheers, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8227745 -----Original Message----- From: David Holmes Sent: Donnerstag, 19. September 2019 02:43 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Richard, On 7/09/2019 12:24 am, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape analysis those references escape and > optimizations like scalar replacement become invalid. What bothers me about this is that it seems like escape analysis is doing something wrong in this case. If the object is thread-local but is being synchronized upon then either: a) the synchronization is elided and so the object will not appear in the set of owned monitors; or b) the fact synchronization occurs renders the object ineligible to be considered thread-local, and so there is no problem with it appearing in the set of owned monitors I think there is a bigger design philosophy issue here about the interaction of escape analysis and debugging/management frameworks in general. I'd like to see a very clear description on exactly how they should interact. Cheers, David > The runtime currently cannot cope with objects escaping through JVMTI (try included > tests). Therefore escape analysis should be disabled if an agent requests the capabilities > can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > From hohensee at amazon.com Thu Sep 19 16:39:31 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 16:39:31 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread Message-ID: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Off by 2 error. Changed the subject to reflect 8231209. http://cr.openjdk.java.net/~phh/8231209/webrev.00/ Paul ?On 9/19/19, 6:31 AM, "Daniel D. Daugherty" wrote: > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ The redo bug is 8231209. 8231211 is closed as a dup of 8231210. Dan On 9/19/19 9:17 AM, Hohensee, Paul wrote: > I'll have the default method throw UOE. That's the same as the other default methods do. > > The necessary test fix is in test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, which needs a new getCurrentThreadAllocatedBytes method, defined as > > public long getCurrentThreadAllocatedBytes() { > return (Long) invokeMethod("getCurrentThreadAllocatedBytes", > new Object[] { }, > new String[] { }); > } > > With this fix, the 134 tests in test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. Preliminary webrev at > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > Is it worth adding getCurrentThreadAllocatedBytes tests to the test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes set? > > Paul > > On 9/18/19, 8:16 PM, "David Holmes" wrote: > > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> They all implement com.sun.management.ThreadMXBean, so adding a > >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it > >> a default implementation, vis > >> > >> public default long getCurrentThreadAllocatedBytes() { > >> return -1; > >> } > >> > > > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > > "sealed" interface which should only be implemented by JDK. > > Didn't realize that. I don't recall knowing about PlatformManagedObject. > Sealed types will at least allow this to be enforced, though I have to > wonder what the tests are doing here. > > > Unfortunately we don't have the sealed type feature yet. Yes it needs > > to be a default method. I think it should throw UOE. > > > > * @implSpec > > * The default implementation throws {@code > > UnsupportedOperationException}. > > > > The @throw UOE can make it clear that it does not support current thread > > memory allocation measurement. > > Yes that seems a reasonable default if we don't want this to be > implemented outside the platform. > > Thanks, > David > > > Mandy > > From hohensee at amazon.com Thu Sep 19 16:52:21 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 19 Sep 2019 16:52:21 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: More formally, Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ Thanks, ?On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, Paul" wrote: Off by 2 error. Changed the subject to reflect 8231209. http://cr.openjdk.java.net/~phh/8231209/webrev.00/ Paul On 9/19/19, 6:31 AM, "Daniel D. Daugherty" wrote: > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ The redo bug is 8231209. 8231211 is closed as a dup of 8231210. Dan On 9/19/19 9:17 AM, Hohensee, Paul wrote: > I'll have the default method throw UOE. That's the same as the other default methods do. > > The necessary test fix is in test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, which needs a new getCurrentThreadAllocatedBytes method, defined as > > public long getCurrentThreadAllocatedBytes() { > return (Long) invokeMethod("getCurrentThreadAllocatedBytes", > new Object[] { }, > new String[] { }); > } > > With this fix, the 134 tests in test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. Preliminary webrev at > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > Is it worth adding getCurrentThreadAllocatedBytes tests to the test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes set? > > Paul > > On 9/18/19, 8:16 PM, "David Holmes" wrote: > > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> They all implement com.sun.management.ThreadMXBean, so adding a > >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it > >> a default implementation, vis > >> > >> public default long getCurrentThreadAllocatedBytes() { > >> return -1; > >> } > >> > > > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > > "sealed" interface which should only be implemented by JDK. > > Didn't realize that. I don't recall knowing about PlatformManagedObject. > Sealed types will at least allow this to be enforced, though I have to > wonder what the tests are doing here. > > > Unfortunately we don't have the sealed type feature yet. Yes it needs > > to be a default method. I think it should throw UOE. > > > > * @implSpec > > * The default implementation throws {@code > > UnsupportedOperationException}. > > > > The @throw UOE can make it clear that it does not support current thread > > memory allocation measurement. > > Yes that seems a reasonable default if we don't want this to be > implemented outside the platform. > > Thanks, > David > > > Mandy > > From chris.plummer at oracle.com Thu Sep 19 21:40:55 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 19 Sep 2019 14:40:55 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: One thing I didn't mention before is that using "jshell> " as the line predicate does not work because jshell does not produce a \n after generating this prompt, thus it's not actually a line and no attempt will be made to match it, so eventually it times out. I fixed this by using a snippet from one of the previous 2 lines" ? Welcome to JShell -- Version 14-internal ? For an introduction type: /help intro I now search for "Welcome to JShell", which is working. I think this is related to additional issues I am having with this fix. I used to see JDK-8230872 happen about 1% of the time after the original fix I had for this CR (8228625). However, now it fails about 9% of the time. It seems that this new fix is making it more likely that jshell will be in a state that will uncover jmap bugs. If I add another 10 second delay to the test, all the jmap problems go away. This started to get me thinking that maybe "jmap --pid" is just not reliable. We do have a few other tests for this, but I believe they all wait until the target process has all threads blocked before issuing the jmap command. This might be why they don't see problems. In the case of this test, since I don't fully wait for the jshell> prompt, there is still probably some jshell activity when jmap attaches. Waiting an extra 10 seconds gets us past this activity, and likely jshell is blocked waiting for input. In fact even just waiting 1 additional second seems to be long enough. Chris On 9/18/19 9:44 PM, Chris Plummer wrote: > I got this to work, although it increased the test time from about 30s > to over 3m. I looked into it a bit and it appears to be due to the > size of the generate hprof file. It used to be about 300k, but now is > nearly 7mb. I guess that's because jshell has been run for longer and > probably allocated more data. > > I need to do some more testing and a bit of cleanup. I'll get another > review out tomorrow. > > Chris > > Suddenly the test time is taking > > On 9/18/19 5:43 PM, Chris Plummer wrote: >> Ok. It was a bit unclear to me why the author went with >> Runtime.exec() in the first place. I'll try ProcessTools. That will >> probably however, hide 8230872. I might need to write another test >> for it. >> >> Chris >> >> On 9/18/19 4:29 PM, Alex Menkov wrote: >>> You can use jdk.test.lib classes to simplify the things. >>> Something like >>> >>> ProcessBuilder pb = new >>> ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); >>> Process p = ProcessTools.startProcess("JShell", pb, >>> ??? s -> {? // warm-up predicate >>> ??????? return s.contains(">jshell"); >>> ??? }); >>> >>> --alex >>> >>> On 09/18/2019 15:44, Chris Plummer wrote: >>>> Is there an easy way of doing this? Currently the jshell process is >>>> just spawned using Runtime.exec(). >>>> >>>> Chris >>>> >>>> On 9/18/19 3:01 PM, Alex Menkov wrote: >>>>> Hi Chris, >>>>> >>>>> Did you think about waiting for jshell prompt ("jshell>") before >>>>> run "jhsdb jmap" command instead of delay or re-tries? >>>>> >>>>> --alex >>>>> >>>>> On 09/18/2019 14:11, Chris Plummer wrote: >>>>>> Hello, >>>>>> >>>>>> Please review the following changes: >>>>>> >>>>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>>>> >>>>>> There are actually numerous ways that JShellHeapDumpTest.java >>>>>> fails. One is a test bug, being addressed here, and the rest all >>>>>> seem to be SA bugs. Those are now being covered by JDK-8230872. >>>>>> All the issues seem to stem from the fact that the test spawns a >>>>>> jshell process, and then immediately does a "jhsdb jmap" on the >>>>>> process before jshell has fully started up. >>>>>> >>>>>> The test bug happens when the jmap succeeds, but jshell has not >>>>>> yet entered the main java thread. Thus the search for >>>>>> "JShellToolProvider" in the output fails. It expects >>>>>> "JShellToolProvider" to be in the output because it is part of a >>>>>> method name in the main thread, and the test dump all the thread >>>>>> stacks contained in the jmap generated hprof file. When the test >>>>>> fails in this way, you can see the stack dump in the output, but >>>>>> the main thread is missing. >>>>>> >>>>>> There's a couple of ways to fix this. One is to just add a delay >>>>>> (10s seems to be more than enough), and the other is to retry the >>>>>> "jhsdb jmap" command until the stack contains the >>>>>> JShellToolProvider symbol. I chose the later because doing a 10s >>>>>> delay masks the SA issues that are now covered by JDK-8230872. In >>>>>> a way the 10s delay is a better fix, because it makes this test >>>>>> pass every time, but I did not like that it also hid real SA >>>>>> problems in JDK-8230872. My plan for now is to do this retry fix, >>>>>> and then if there are too many failures due to JDK-8230872, then >>>>>> also add a 10s delay, with the intention of removing it once >>>>>> JDK-8230872 if fixed. From what I can see, JDK-8230872 failures >>>>>> happen on about 1% of the runs. >>>>>> >>>>>> I made a few of other changes. One was to no longer redirect >>>>>> stderr from the jmap process as was done from the following: >>>>>> >>>>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>>>> >>>>>> This causes the output not to appear in the OutputAnalyzer >>>>>> output, resulting in the following not working: >>>>>> >>>>>> ???????????? output.shouldNotContain("null"); >>>>>> >>>>>> Also I added code to dump the output of the jshell process so you >>>>>> can see if the jshell prompt was ever generated. >>>>>> >>>>>> thanks, >>>>>> >>>>>> Chris >>>>>> >>>> >>>> >> >> > > From mandy.chung at oracle.com Thu Sep 19 22:00:35 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 19 Sep 2019 15:00:35 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: <1afdd496-a369-a4bd-0542-4d22eed65f96@oracle.com> On 9/19/19 9:52 AM, Hohensee, Paul wrote: > More formally, > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > The new metric you added is an attribute (not an operation).? So the change in ServerThreadMXBeanNew should call getLongAttribute inherited from ServerMXBean superclass. The change to com.sun.management.ThreadMXBean is fine.? The CSR needs an amendment.?? I am asking Joe [2] for his recommendation whether you should reuse JDK-8230311 or file a new CSR. Mandy [1] test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerMXBean.java [2] https://bugs.openjdk.java.net/browse/JDK-8230311?focusedCommentId=14290493&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14290493 From david.holmes at oracle.com Thu Sep 19 22:58:44 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 08:58:44 +1000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: On 20/09/2019 2:42 am, Reingruber, Richard wrote: > Hi David, > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > optimizations like scalar replacement become invalid. > > > > What bothers me about this is that it seems like escape analysis is > > doing something wrong in this case. > > Yes it is. > > > If the object is thread-local but is > > being synchronized upon then either: > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > machine, as observed through JVMTI, is wrong. See below... You seem to have completely missed my point. If the object is local and is synchronized upon then the synchronization can be elided (and should be) in which case it won't appear in GetOwnedMonitorInfo and so does not escape. If the synchronization cannot be elided then the object cannot be considered local. That is how Escape Analysis should be operating here IMHO. Cheers, David ----- > > a) the synchronization is elided and so the object will not appear in > > the set of owned monitors; or > > b) the fact synchronization occurs renders the object ineligible to be > > considered thread-local, and so there is no problem with it appearing in > > the set of owned monitors > > > > I think there is a bigger design philosophy issue here about the > > interaction of escape analysis and debugging/management frameworks in > > general. I'd like to see a very clear description on exactly how they > > should interact. > > > > I don't see too many design alternatives here. The JVMTI implementation has to present the correct > state of the virtual machine according to the spec. I think it fails to do so in this case. > > Please look again at the test: > > 172 public long dontinline_endlessLoop() { > 173 long cs = checkSum; > 174 while (doLoop && loopCount-- > 0) { > 175 targetIsInLoop = true; > 176 checkSum += checkSum % ++cs; > 177 } > 178 loopCount = 3; > 179 targetIsInLoop = false; > 180 return checkSum; > 181 } > > 249 public void dontinline_testMethod() { > 250 LockCls l1 = new LockCls(); // to be scalar replaced > 251 synchronized (l1) { > 252 inlinedTestMethodWithNestedLocking(l1); > 253 } > 254 } > 255 > 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { > 257 synchronized (l1) { // nested > 258 dontinline_endlessLoop(); > 259 } > 260 } > > This is the stack when the agent calls GetOwnedMonitorInfo() > > dontinline_endlessLoop() at line 176 > inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame > dontinline_testMethod() at line 252 // compiled frame > > The state of the _virtual_ machine at that point is obvious: > > - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. > - That instance was locked by a monitorenter bytecode at line 251 > > This could be proven by interpreting the execution trace bytecode by bytecode using paper and > pencil (hope you won't make me do this, though ;)) > > JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call > GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does > not. > > From a more philosophical point of view compiled code is free to change the state of the physical > machine in a way such that it cannot be mapped to a valid state of the virtual machine after each > and every machine instruction. But it must reach points in its execution trace, where it is actually > possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These > points are called safepoints. > > The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm > state is presented. EA does not take into account that the lock state can be observed using > GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability > can_get_owned_monitor_info is taken. With the fix the test passes. > > Note that for the very same reason EA is disabled if can_access_local_variables is taken, because > the JVMTI implementation cannot handout references to objects stored in local variables if they were > scalar replaced. > > With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA > based optimizations just-in-time before local objects escape. Note that EA opts are already reverted > today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() > and relock_objects() in class Deoptimization) > > Thanks and cheers, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > > -----Original Message----- > From: David Holmes > Sent: Donnerstag, 19. September 2019 02:43 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 7/09/2019 12:24 am, Reingruber, Richard wrote: >> Hi, >> >> could I please get reviews for >> >> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >> >> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> retrieve objects locked by a thread. In terms of escape analysis those references escape and >> optimizations like scalar replacement become invalid. > > What bothers me about this is that it seems like escape analysis is > doing something wrong in this case. If the object is thread-local but is > being synchronized upon then either: > a) the synchronization is elided and so the object will not appear in > the set of owned monitors; or > b) the fact synchronization occurs renders the object ineligible to be > considered thread-local, and so there is no problem with it appearing in > the set of owned monitors > > I think there is a bigger design philosophy issue here about the > interaction of escape analysis and debugging/management frameworks in > general. I'd like to see a very clear description on exactly how they > should interact. > > Cheers, > David > >> The runtime currently cannot cope with objects escaping through JVMTI (try included >> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >> >> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >> objects escape through JVMTI. >> >> I've run tier1 tests. >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> From david.holmes at oracle.com Thu Sep 19 23:06:56 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 09:06:56 +1000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: Hi Paul, On 20/09/2019 2:52 am, Hohensee, Paul wrote: > More formally, > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ I'm assuming there are only two changes here: 1. The new method is now a default method that throws UOE. That seems fine. 2. You implemented the new method in the test class. I don't understand why you did that. The test can't be calling the new method. Now that it is a default method we will get past the compilation failure that caused the problem. So no change to the test should be needed AFAICS. A new CSR request is needed. Just copy everything across from the old, with the updated spec. But please also mention this is a PlatformManagedObject in the compatibility discussion. Thanks, David > Thanks, > > ?On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > Off by 2 error. Changed the subject to reflect 8231209. > > http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > Paul > > On 9/19/19, 6:31 AM, "Daniel D. Daugherty" wrote: > > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > The redo bug is 8231209. 8231211 is closed as a dup of 8231210. > > Dan > > > On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > I'll have the default method throw UOE. That's the same as the other default methods do. > > > > The necessary test fix is in test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > public long getCurrentThreadAllocatedBytes() { > > return (Long) invokeMethod("getCurrentThreadAllocatedBytes", > > new Object[] { }, > > new String[] { }); > > } > > > > With this fix, the 134 tests in test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. Preliminary webrev at > > > > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > Is it worth adding getCurrentThreadAllocatedBytes tests to the test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes set? > > > > Paul > > > > On 9/18/19, 8:16 PM, "David Holmes" wrote: > > > > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > >> They all implement com.sun.management.ThreadMXBean, so adding a > > >> getCurrentThreadAllocatedBytes broke them. Potential fix is to give it > > >> a default implementation, vis > > >> > > >> public default long getCurrentThreadAllocatedBytes() { > > >> return -1; > > >> } > > >> > > > > > > com.sun.management.ThreadMXBean (and other platform MXBeans) is a > > > "sealed" interface which should only be implemented by JDK. > > > > Didn't realize that. I don't recall knowing about PlatformManagedObject. > > Sealed types will at least allow this to be enforced, though I have to > > wonder what the tests are doing here. > > > > > Unfortunately we don't have the sealed type feature yet. Yes it needs > > > to be a default method. I think it should throw UOE. > > > > > > * @implSpec > > > * The default implementation throws {@code > > > UnsupportedOperationException}. > > > > > > The @throw UOE can make it clear that it does not support current thread > > > memory allocation measurement. > > > > Yes that seems a reasonable default if we don't want this to be > > implemented outside the platform. > > > > Thanks, > > David > > > > > Mandy > > > > > > > > > From mandy.chung at oracle.com Thu Sep 19 23:33:15 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 19 Sep 2019 16:33:15 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: On 9/19/19 4:06 PM, David Holmes wrote: > > 2. You implemented the new method in the test class. > > I don't understand why you did that. The test can't be calling the new > method. Now that it is a default method we will get past the > compilation failure that caused the problem. So no change to the test > should be needed AFAICS. > Maybe.? It can verify easily. ? I don't know what the test does and whether it expects a good return value rather than UOE. Implementing the new method has no harm but as I commented early, it is a long attribute and not an operation. I'm okay with either way. Mandy From david.holmes at oracle.com Thu Sep 19 23:56:42 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 09:56:42 +1000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: <0f42fdd1-290d-7226-3dae-e281280c3b67@oracle.com> On 20/09/2019 9:33 am, Mandy Chung wrote: > On 9/19/19 4:06 PM, David Holmes wrote: >> >> 2. You implemented the new method in the test class. >> >> I don't understand why you did that. The test can't be calling the new >> method. Now that it is a default method we will get past the >> compilation failure that caused the problem. So no change to the test >> should be needed AFAICS. >> > > Maybe.? It can verify easily. ? I don't know what the test does and > whether it expects a good return value rather than UOE. The test doesn't call it. It is a new method that the test knows nothing about. David ----- > Implementing the new method has no harm but as I commented early, it is > a long attribute and not an operation. > > I'm okay with either way. > > Mandy From daniil.x.titov at oracle.com Fri Sep 20 00:30:02 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Thu, 19 Sep 2019 17:30:02 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: Hi David and Serguei, Please review new version of the fix that includes the changes Serguei suggested: 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread that has just passed the removal point in ThreadsSMRSupport::remove_thread() The changes are in ThreadTable::lazy_initialize() method only. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! --Daniil ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: "serguei.spitsyn at oracle.com" > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Fri Sep 20 01:15:25 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 11:15:25 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> Hi Daniil, On 20/09/2019 10:30 am, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list Ok. > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() That seems good too. I agree with previous comments about the general race with is_exiting in terms of how this API behaves. But there's no change in that behaviour with your changes AFAICS. The main concern, now addressed, is that you mustn't be able to add a thread that never gets removed. Thanks, David ----- > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: "serguei.spitsyn at oracle.com" > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Fri Sep 20 02:54:31 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Thu, 19 Sep 2019 19:54:31 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> Message-ID: <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> Hi David, Thank you for reviewing this version of the fix. > I agree with previous comments about the general race with is_exiting in > terms of how this API behaves. But there's no change in that behaviour > with your changes AFAICS. Could you please say am I right that you are referring here to the discussion about the fact that find_JavaThread_from_java_tid() still could return the thread that is exiting? If so then, yes, this behavior was not changed. We discussed it with Serguei and the conclusion was that since the current implementation behaves exact the same way then even if decide somehow address this then it makes sense to do it in a separate issue. It also is not clear at this moment what the possible solution could be. The obvious one, just to hold Thread_locks in the callers of find_JavaThread_from_java_tid() (management.cpp) is too expensive and doesn't look as acceptable. Thanks! Best regards, Daniil ?On 9/19/19, 6:15 PM, "David Holmes" wrote: Hi Daniil, On 20/09/2019 10:30 am, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list Ok. > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() That seems good too. I agree with previous comments about the general race with is_exiting in terms of how this API behaves. But there's no change in that behaviour with your changes AFAICS. The main concern, now addressed, is that you mustn't be able to add a thread that never gets removed. Thanks, David ----- > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: "serguei.spitsyn at oracle.com" > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Fri Sep 20 03:24:03 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 13:24:03 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> Message-ID: <0810ce57-ed4f-0156-c5ee-be4de87c11ab@oracle.com> On 20/09/2019 12:54 pm, Daniil Titov wrote: > Hi David, > > Thank you for reviewing this version of the fix. > >> I agree with previous comments about the general race with is_exiting in >> terms of how this API behaves. But there's no change in that behaviour >> with your changes AFAICS. > > Could you please say am I right that you are referring here to the discussion about > the fact that find_JavaThread_from_java_tid() still could return the thread that > is exiting? Yes you are right. > If so then, yes, this behavior was not changed. We discussed it with Serguei > and the conclusion was that since the current implementation behaves exact the same way > then even if decide somehow address this then it makes sense to do it in a separate issue. > It also is not clear at this moment what the possible solution could be. The obvious one, > just to hold Thread_locks in the callers of find_JavaThread_from_java_tid() (management.cpp) > is too expensive and doesn't look as acceptable. As far as I am concerned it is just an optimization to try to avoid returning a JavaThread that is in the process of exiting. It is an unavoidable race. Even holding the Threads_lock doesn't really help as the thread can still be "exited" for all intents and purposes even if the JavaThread can't truly terminate. The ThreadSMR now ensures that it is always safe to inspect a JavaThread. If we catch such a thread late in its termination sequence then its threadObj will be NULL and we'll just skip it; or we'll access it in a VM op and see it is_exiting, or some combination of such checks. So I don't see any bug here that needs to be fixed. Cheers, David > Thanks! > > Best regards, > Daniil > > > ?On 9/19/19, 6:15 PM, "David Holmes" wrote: > > Hi Daniil, > > On 20/09/2019 10:30 am, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > Ok. > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > That seems good too. > > I agree with previous comments about the general race with is_exiting in > terms of how this API behaves. But there's no change in that behaviour > with your changes AFAICS. The main concern, now addressed, is that you > mustn't be able to add a thread that never gets removed. > > Thanks, > David > ----- > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > > > Hi Daniil, > > > > On 9/17/19 17:13, Daniil Titov wrote: > > > Hi Serguei, > > > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > > > 1. > > > > I have a concern about the checks for thread->is_exiting(). > > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > >> It is interesting what might happen if an exiting thread is returned by the > > >> ThreadsList::find_JavaThread_from_java_tid (). > > >> Does it make sense to develop a test that would cover these cases? > > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > > > Now, I'm not that confident about it. :) > > > > > The current implementation > > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > > > If I understand it correctly, the jt->threadObj() can remain non-NULL > > for some time while jt->is_exiting() == true. > > It is not clear how reliable is to use it. > > But this is a pre-existing issue. It is not you who introduced it. :) > > > > So, we can skip it for now. > > But for the record, we may have a source of intermittent issues. > > > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > > > Understand. > > > > > 2. > > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > > > > Okay, thanks! > > > > > 3. > > >> I would suggest to rewrite this fragment in a safe way: > > >> 95 { > > >> 96 MutexLocker ml(ThreadTableCreate_lock); > > >> 97 if (!_is_initialized) { > > >> 98 create_table(threads->length()); > > >> 99 _is_initialized = true; > > >> 100 } > > >> 101 } > > >> as: > > >> { > > >> MutexLocker ml(ThreadTableCreate_lock); > > >> if (_is_initialized) { > > >> return; > > > > } > > > > create_table(threads->length()); > > > > _is_initialized = true; > > > > } > > > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > > and thus avoid the linear scan when these thread are looked up for the first time. > > > > > > I've replied to David in another email. > > Let's talk once more about it tomorrow. > > > > > > > 4. > > >>> The case you have described is exact the reason why we still have a code inside > > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > > >> I disagree because it is easy to avoid concurrent ThreadTable > > >> initialization (please, see my separate email). > > >> The reason for this code is to cover a case of late/lazy ThreadTable > > >> initialization. > > > David Holmes replied to this in a separate email providing a very detailed > > > explanation of the possible cases and how the proposed implementation satisfies them. > > > > Yes. Please, see above. > > > > Thanks, > > Serguei > > > > > Best regards, > > > Daniil > > > > > > From: "serguei.spitsyn at oracle.com" > > > Date: Tuesday, September 17, 2019 at 1:53 AM > > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > Hi Daniil, > > > > > > Thank you for you patience in working on this issue! > > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > > It was a good idea to separate them. > > > > > > I have a concern about the checks for thread->is_exiting(). > > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > > > There is a couple of such checks to mention: > > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > 612 ThreadTable::lazy_initialize(this); > > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > > 614 if (thread == NULL) { > > > 615 // If the thread is not found in the table find it > > > 616 // with a linear search and add to the table. > > > 617 for (uint i = 0; i < length(); i++) { > > > 618 thread = thread_at(i); > > > 619 oop tobj = thread->threadObj(); > > > 620 // Ignore the thread if it hasn't run yet, has exited > > > 621 // or is starting to exit. > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > > 623 MutexLocker ml(Threads_lock); > > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > 626 if (!thread->is_exiting()) { > > > 627 ThreadTable::add_thread(java_tid, thread); > > > 628 return thread; > > > 629 } > > > 630 } > > > 631 } > > > 632 } else if (!thread->is_exiting()) { > > > 633 return thread; > > > 634 } > > > 635 return NULL; > > > 636 } > > > ... > > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > > 94 if (!_is_initialized) { > > > 95 { > > > 96 MutexLocker ml(ThreadTableCreate_lock); > > > 97 if (!_is_initialized) { > > > 98 create_table(threads->length()); > > > 99 _is_initialized = true; > > > 100 } > > > 101 } > > > 102 for (uint i = 0; i < threads->length(); i++) { > > > 103 JavaThread* thread = threads->thread_at(i); > > > 104 oop tobj = thread->threadObj(); > > > 105 if (tobj != NULL && !thread->is_exiting()) { > > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > > 107 add_thread(java_tid, thread); > > > 108 } > > > 109 } > > > 110 } > > > 111 } > > > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > > So that: > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > > are protected with the Threads_lock. But the returned thread still can exit after that. > > > It is interesting what might happen if an exiting thread is returned by the > > > ThreadsList::find_JavaThread_from_java_tid (). > > > > > > Does it make sense to develop a test that would cover these cases? > > > > > > Thanks, > > > Serguei > > > > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > > Hello, > > > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > > 623 MutexLocker ml(Threads_lock); > > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > 626 if (!thread->is_exiting()) { > > > 627 ThreadTable::add_thread(java_tid, thread); > > > 628 return thread; > > > 629 } > > > 630 } > > > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > > > ?Thank you, > > > Daniil > > > > > > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > > Hi David, > > > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > > the changes you suggested: > > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > > > Okay. > > > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > > > - fixed nits and formatting issues. > > > > > > > > Okay. > > > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > >>> as Daniel suggested. > > > > >> Not sure it's best to combine these, but if they are limited to the > > > > >> changes in management.cpp only then that may be okay. > > > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > > I also could move it in the separate issue if required. > > > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > > Dan or Serguei have a strong opinion. > > > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > > non-null threadObj. > > > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > > when VM is destroyed: > > > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > > issue with JNI_AttachCurrentThread and its interaction with the > > > > ThreadSMR iterators. The attach process is: > > > > - create JavaThread > > > > - mark as "is attaching via jni" > > > > - add to ThreadsList > > > > - create java.lang.Thread object (you can only execute Java code after > > > > you are attached) > > > > - mark as "attach completed" > > > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > > iterator but will have a NULL java.lang.Thread object. > > > > > > > > We special-case attaching threads in a number of places in the VM and I > > > > think we should be explicitly doing something here to filter out > > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > ThreadTable::add_thread(tid, thread); > > > > } > > > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > > which covers the case the JNI attach encountered an error trying to > > > > create the j.l.Thread object. > > > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > > >> have it, but I'm concerned that its very existence means we perhaps > > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > > removed by calling ConcurrentHashTable::remove() method. > > > > > I think that just because in our case we don't use this mechanism doesn't > > > > > mean we should not use ConcurrentHashTable. > > > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > > back from vacation this week. > > > > > > > > >> I would still want to see what impact this has on thread > > > > >> startup cost, both with and without the table being initialized. > > > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > > thread table makes the thread startup about 2.7% slower. > > > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > > help investigate overall performance impact here. > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > No further code comments. > > > > > > > > I didn't look at the test in detail. > > > > > > > > Thanks, > > > > David > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > > see some performance and footprint numbers, both to verify it fixes the > > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > > populated, table to be seen by other threads - is that your intention? > > > > > It seems it should be okay as the other threads will then race with the > > > > > initializing thread to add specific entries, and this is a concurrent > > > > > map so that should be functionally correct. But if so then I think you > > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > > > I like the approach of only initializing the table when needed and using > > > > > that to control when the add/remove-thread code needs to update the > > > > > table. But I would still want to see what impact this has on thread > > > > > startup cost, both with and without the table being initialized. > > > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > > as Daniel suggested. > > > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > > changes in management.cpp only then that may be okay. It helps to be > > > > > able to focus on the table related changes without being distracted by > > > > > other optimizations. > > > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > > > The revised version seems better in that regard. But I still have a > > > > > concern, see below. > > > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > > growing the thread table when required. > > > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > > is doing the addition? For other tables we may have to delegate to the > > > > > service thread because the current thread cannot perform the action, or > > > > > it doesn't want to perform it at the time the need for the resize is > > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > > happen later outside the safepoint). It's not apparent to me that such > > > > > restrictions apply here. > > > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > > > Ok. > > > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > > > Some specific code comments: > > > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > > false, Monitor::_safepoint_check_never); > > > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > > be created by regular JavaThreads and they should (nearly) always be > > > > > checking for safepoints if they are going to block acquiring the lock. > > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > > to a safepoint whilst this lock is held. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > > > Nit: > > > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > > just rename that "new" variable to "thread" so you don't have to change > > > > > all other uses. > > > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > > > You don't need to check != NULL here as you only get here when > > > > > java_thread is not NULL. > > > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > > > I think it cleaner/better to just use > > > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > > 56 private: > > > > > 57 jlong _tid; > > > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > > class declarations, so the above would just be: > > > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > > 56 private: > > > > > 57 jlong _tid; > > > > > > > > > > etc. > > > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > > ... > > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > > have it, but I'm concerned that its very existence means we perhaps > > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > > would be: > > > > > > > > > > 115 size_t start_size_log = > > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > > DefaultThreadTableSizeLog; > > > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > > 132 return (double)_items_count/_current_size; > > > > > 133 } > > > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > > division and then cast that whole integer to a double. If you want > > > > > double arithmetic you need: > > > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > > > 180 jlong _tid; > > > > > 181 uintx _hash; > > > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > > > line 184 should be indented. > > > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > > > Nit: need space after : > > > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > > 212 _has_work = false; > > > > > > > > > > line 211 is indented one space too far. > > > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > > > Nit: need space after , > > > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > > > Nit: need space after , > > > > > > > > > > Thanks, > > > > > David > > > > > ------ > > > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > > Hi Serguei and David, > > > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > > followed up with additional comments. Before I do a crawl through > > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > > made optional and David's other comments addressed. > > > > > > > > > > > > Another possible optimization is for callers of > > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > > tid value before they loop and if the current tid == saved_tid > > > > > > then use the current JavaThread* instead of calling > > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > > > Thanks! > > > > > > > --Daniil > > > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > > Organization: Oracle Corporation > > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > > 619 // to the thread table. > > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > > 624 return thread; > > > > > > > 625 } > > > > > > > 626 } > > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > 628 return java_thread; > > > > > > > 629 } > > > > > > > 630 return NULL; > > > > > > > 631 } > > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > > 635 // or is starting to exit. > > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > > 638 } > > > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > > > A space is missed after the comma: > > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Serguei > > > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > > need a lot more data on it. > > > > > > > > > > > > > > Thanks, > > > > > > > David > > > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > > in the thread table. > > > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Best regards, > > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Fri Sep 20 05:30:39 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 19 Sep 2019 22:30:39 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Fri Sep 20 05:31:18 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 19 Sep 2019 22:31:18 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: Here's a new webrev: http://cr.openjdk.java.net/~cjplummer/8228625/webrev.01/ I decided to add a 2 second delay to make sure the test always passes and avoids the issues in JDK-8230872. After some internal discussion there seems to be some consensus that SA is probably not very stable when attaching to an active process, and I'd rather not have those issues disrupting this test. I filed a CR to write a new test without the 2 second delay so we can still reproduce these issues. See: https://bugs.openjdk.java.net/browse/JDK-8231288 Besides now using ProcessTools.startProcess() with a line predicate instead of Runtime.exec(), I also did a bit of cleaning up of variable names and added a few more diagnostic printlns. thanks, Chris On 9/19/19 2:40 PM, Chris Plummer wrote: > One thing I didn't mention before is that using "jshell> " as the line > predicate does not work because jshell does not produce a \n after > generating this prompt, thus it's not actually a line and no attempt > will be made to match it, so eventually it times out. I fixed this by > using a snippet from one of the previous 2 lines" > > ? Welcome to JShell -- Version 14-internal > ? For an introduction type: /help intro > > I now search for "Welcome to JShell", which is working. > > I think this is related to additional issues I am having with this > fix. I used to see JDK-8230872 happen about 1% of the time after the > original fix I had for this CR (8228625). However, now it fails about > 9% of the time. It seems that this new fix is making it more likely > that jshell will be in a state that will uncover jmap bugs. If I add > another 10 second delay to the test, all the jmap problems go away. > > This started to get me thinking that maybe "jmap --pid" is just not > reliable. We do have a few other tests for this, but I believe they > all wait until the target process has all threads blocked before > issuing the jmap command. This might be why they don't see problems. > In the case of this test, since I don't fully wait for the jshell> > prompt, there is still probably some jshell activity when jmap > attaches. Waiting an extra 10 seconds gets us past this activity, and > likely jshell is blocked waiting for input. In fact even just waiting > 1 additional second seems to be long enough. > > Chris > > On 9/18/19 9:44 PM, Chris Plummer wrote: >> I got this to work, although it increased the test time from about >> 30s to over 3m. I looked into it a bit and it appears to be due to >> the size of the generate hprof file. It used to be about 300k, but >> now is nearly 7mb. I guess that's because jshell has been run for >> longer and probably allocated more data. >> >> I need to do some more testing and a bit of cleanup. I'll get another >> review out tomorrow. >> >> Chris >> >> Suddenly the test time is taking >> >> On 9/18/19 5:43 PM, Chris Plummer wrote: >>> Ok. It was a bit unclear to me why the author went with >>> Runtime.exec() in the first place. I'll try ProcessTools. That will >>> probably however, hide 8230872. I might need to write another test >>> for it. >>> >>> Chris >>> >>> On 9/18/19 4:29 PM, Alex Menkov wrote: >>>> You can use jdk.test.lib classes to simplify the things. >>>> Something like >>>> >>>> ProcessBuilder pb = new >>>> ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); >>>> Process p = ProcessTools.startProcess("JShell", pb, >>>> ??? s -> {? // warm-up predicate >>>> ??????? return s.contains(">jshell"); >>>> ??? }); >>>> >>>> --alex >>>> >>>> On 09/18/2019 15:44, Chris Plummer wrote: >>>>> Is there an easy way of doing this? Currently the jshell process >>>>> is just spawned using Runtime.exec(). >>>>> >>>>> Chris >>>>> >>>>> On 9/18/19 3:01 PM, Alex Menkov wrote: >>>>>> Hi Chris, >>>>>> >>>>>> Did you think about waiting for jshell prompt ("jshell>") before >>>>>> run "jhsdb jmap" command instead of delay or re-tries? >>>>>> >>>>>> --alex >>>>>> >>>>>> On 09/18/2019 14:11, Chris Plummer wrote: >>>>>>> Hello, >>>>>>> >>>>>>> Please review the following changes: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>>>>> >>>>>>> There are actually numerous ways that JShellHeapDumpTest.java >>>>>>> fails. One is a test bug, being addressed here, and the rest all >>>>>>> seem to be SA bugs. Those are now being covered by JDK-8230872. >>>>>>> All the issues seem to stem from the fact that the test spawns a >>>>>>> jshell process, and then immediately does a "jhsdb jmap" on the >>>>>>> process before jshell has fully started up. >>>>>>> >>>>>>> The test bug happens when the jmap succeeds, but jshell has not >>>>>>> yet entered the main java thread. Thus the search for >>>>>>> "JShellToolProvider" in the output fails. It expects >>>>>>> "JShellToolProvider" to be in the output because it is part of a >>>>>>> method name in the main thread, and the test dump all the thread >>>>>>> stacks contained in the jmap generated hprof file. When the test >>>>>>> fails in this way, you can see the stack dump in the output, but >>>>>>> the main thread is missing. >>>>>>> >>>>>>> There's a couple of ways to fix this. One is to just add a delay >>>>>>> (10s seems to be more than enough), and the other is to retry >>>>>>> the "jhsdb jmap" command until the stack contains the >>>>>>> JShellToolProvider symbol. I chose the later because doing a 10s >>>>>>> delay masks the SA issues that are now covered by JDK-8230872. >>>>>>> In a way the 10s delay is a better fix, because it makes this >>>>>>> test pass every time, but I did not like that it also hid real >>>>>>> SA problems in JDK-8230872. My plan for now is to do this retry >>>>>>> fix, and then if there are too many failures due to JDK-8230872, >>>>>>> then also add a 10s delay, with the intention of removing it >>>>>>> once JDK-8230872 if fixed. From what I can see, JDK-8230872 >>>>>>> failures happen on about 1% of the runs. >>>>>>> >>>>>>> I made a few of other changes. One was to no longer redirect >>>>>>> stderr from the jmap process as was done from the following: >>>>>>> >>>>>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>>>>> >>>>>>> This causes the output not to appear in the OutputAnalyzer >>>>>>> output, resulting in the following not working: >>>>>>> >>>>>>> ???????????? output.shouldNotContain("null"); >>>>>>> >>>>>>> Also I added code to dump the output of the jshell process so >>>>>>> you can see if the jshell prompt was ever generated. >>>>>>> >>>>>>> thanks, >>>>>>> >>>>>>> Chris >>>>>>> >>>>> >>>>> >>> >>> >> >> > > From david.holmes at oracle.com Fri Sep 20 05:35:10 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 15:35:10 +1000 Subject: RFC: 8229160: Reimplement JvmtiRawMonitor to use PlatformMonitor In-Reply-To: <774fb8b3-6c00-fcc7-0c25-858933b65f4b@oracle.com> References: <842a1d43-bdcc-3345-2731-d92b477e3ad3@oracle.com> <75acc5c5-b45f-a764-55d6-692dd7ca0a1a@oracle.com> <5fe4ee14-f406-850e-34b4-e44485b39349@oracle.com> <774fb8b3-6c00-fcc7-0c25-858933b65f4b@oracle.com> Message-ID: I've abandoned this effort and am instead taking a simpler approach under a new bug id: https://bugs.openjdk.java.net/browse/JDK-8231289 RFR coming in a few days. David ----- On 10/09/2019 6:52 pm, David Holmes wrote: > On 10/09/2019 5:54 pm, David Holmes wrote: >> It turns out that polling for interrupts is actually very difficult to >> do correctly. There is an inherent race with timing out and being >> signalled that can result in lost signals. Trying to account for that >> without introducing other problems would lead to a very complex >> synchronization mechanism (which would need to track waiters, >> notifications and a "generation" count). The end result is that we'd >> break the tie to ObjectMonitor code, but would replace it with new >> complex untried code. :( > > Or ... rather than the initial "unresponsive to interrupts" approach I > could take it the other way and have every raw monitor wait limited to, > say 500ms, at which point it will return as a "spurious wakeup" and > check the interrupt state on the way out ... > > David > >> David >> >> On 10/09/2019 8:49 am, David Holmes wrote: >>> Hi Serguei, >>> >>> On 10/09/2019 4:26 am, serguei.spitsyn at oracle.com wrote: >>>> Hi David, >>>> >>>> On 9/8/19 19:15, David Holmes wrote: >>>>> Hi Dan, >>>>> >>>>> On 7/09/2019 6:50 am, Daniel D. Daugherty wrote: >>>>>> Hi David, >>>>>> >>>>>> I've finally gotten back to this email thread... >>>>> >>>>> Thanks. >>>>> >>>>>>> FYI testing to date: >>>>>>> ?- tiers 1 -3 all platforms >>>>>>> ?- hotspot: serviceability/jvmti >>>>>>> ????????????????????????? /jdwp >>>>>>> ??????????? vmTestbase/nsk/jvmti >>>>>>> ????????????????????????? /jdwp >>>>>>> ?- JDK: com/sun/jdi >>>>>> >>>>>> You should also add: >>>>>> >>>>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdb >>>>>> open/test/hotspot/jtreg/vmTestbase/nsk/jdi >>>>>> open/test/jdk/java/lang/instrument >>>>> >>>>> Okay - in progress. Though I can't see any use of RawMonitors in >>>>> any of these tests. >>>>> >>>>>> I took a quick look through the preliminary webrev and I don't see >>>>>> anything that worries me. >>>>> >>>>> Thanks. I'll prepare a more polished webrev soon. >>>>> >>>>>> Re: Thread.interrupt() and raw_wait() >>>>>> >>>>>> It would be good to see if that semantic is being tested via the >>>>>> JCK test suite for JVM/TI. >>>>> >>>>> It isn't. The only thing directly tested for RawMonitorWait is >>>>> normal successful operation and reporting "not owner" when not the >>>>> owner. No check for JVMTI_ERROR_INTERRUPT exists other than as >>>>> input for the GetErrorName function. >>>> >>>> This is most likely true. >>>> My only concern is if RawMonitor's can be used in the JCK test >>>> libraries (low probability). >>>> I've asked Leonid Kuskov (JCK) to double check this (added to the >>>> mailing list). >>>> >>>>> >>>>> There's only one test in the whole test base that checks for the >>>>> interrupt and that is >>>>> vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/. In that test if >>>>> we are not interrupted before the RawMonitorWait we will wait until >>>>> the full timeout elapses - which is 2 minutes by default - then >>>>> return and report the interrupt. Hence the test still passes. (If >>>>> it was an untimed wait that would be different of course). >>>> >>>> I figured the same last Friday. >>>> One more place to care about are NSK tests libraries that are >>>> located here: >>>> ?? test/hotspot/jtreg/vmTestbase/nsk/share >>>> >>>> There are a couple of places where the RawMonitor is used: >>>> ?? jvmti/hotswap/HotSwap.cpp:??? if >>>> (!NSK_JVMTI_VERIFY(jvmti->RawMonitorWait(waitLock, millis))) >>>> ?? jvmti/jvmti_tools.cpp:??? jvmtiError error = >>>> env->RawMonitorWait(monitor, millis); >>>> >>>> The use in HotSwap.cpp is local. >>>> The jvmti_tools.cpp defines this: >>>> ?? void rawMonitorWait(jvmtiEnv *env, jrawMonitorID monitor, jlong >>>> millis) { >>>> ??? ?? jvmtiError error = env->RawMonitorWait(monitor, millis); >>>> ??? ?? exitOnError(error); >>>> ?? } >>>> >>>> which is used in the jvmti/agent_tools.cpp but does not depend on >>>> interrupting of RawMonitor's (as I can see). >>>> One more place to mention is: >>>> ?? jvmti/DataDumpRequest/datadumpreq001/datadumpreq001.cpp >>>> >>>> But I see no problems there as well. >>>> >>>> >>>> >>>> The JDWP implementation is using RawMonitor's. >>>> Please, see functions debugMonitorWait()/debugMonitorTimedWait() in >>>> src/jdk.jdwp.agent/share/native/libjdwp/util.c. >>>> >>>> It expects the JVMTI_ERROR_INTERRUPT but never makes a call to the >>>> JVMTI ThreadInterrupt(). >>>> So, it looks like it does not depend on interrupting of RawMonitor's >>>> in any way. >>>> >>>>> >>>>> The more I try to convince people this change should be okay, the >>>>> more uncomfortable I get with my own arguments. :) I think I'm >>>>> going to implement the polling approach for checking interrupts - >>>>> say 500ms. >>>> >>>> The JVMTI spec tells that the JVMTI_ERROR_INTERRUPT can be returned >>>> from the RawMonitorWait: >>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>>> >>> >>> Yes it does and that is the only thing that implies a connection to >>> Thread.interrupt. >>> >>>> which means that RawMonitorWait can be interrupted with the >>>> Thread.Interrupt() >>>> or JVMTI InterruptThread(): >>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#InterruptThread >>>> >>> >>> That's one way to interpret the fact that RawMonitorWait can return >>> JVMTI_ERROR_INTERRUPT, but the actual interaction between >>> Thread.interrupt and RawMonitorWait is not explicitly stated. >>> Arguably you can just check for interruption before and after the >>> wait, to see whether to return JVMTI_ERROR_INTERRUPT, without >>> necessarily being able to break out of the wait itself. That's been >>> the whole premise of this change proposal - that responsiveness to >>> interrupts is more a quality-of-implementation issue. >>> >>> But in any case I've decided to try the polling approach so that we >>> won't wait forever if interrupted but not notified. >>> >>> Thanks, >>> David >>> ----- >>> >>>> >>>> >>>> Thanks, >>>> Serguei >>>> >>>>>> I also very much like/appreciate the decoupling of JvmtiRawMonitors >>>>>> from ObjectMonitors... Thanks for tackling this crazy task. >>>>> >>>>> Thanks :) >>>>> >>>>> David >>>>> >>>>>> Dan >>>>>> >>>>>> >>>>>> >>>>>> On 8/15/19 2:22 AM, David Holmes wrote: >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8229160 >>>>>>> >>>>>>> Preliminary webrev (still has rough edges): >>>>>>> http://cr.openjdk.java.net/~dholmes/8229160/webrev.prelim/ >>>>>>> >>>>>>> Background: >>>>>>> >>>>>>> We've had this comment for a long time: >>>>>>> >>>>>>> ?// The raw monitor subsystem is entirely distinct from normal >>>>>>> ?// java-synchronization or jni-synchronization.? raw monitors >>>>>>> are not >>>>>>> ?// associated with objects.? They can be implemented in any manner >>>>>>> ?// that makes sense.? The original implementors decided to >>>>>>> piggy-back >>>>>>> ?// the raw-monitor implementation on the existing Java >>>>>>> objectMonitor mechanism. >>>>>>> ?// This flaw needs to fixed.? We should reimplement raw monitors >>>>>>> as sui-generis. >>>>>>> ?// Specifically, we should not implement raw monitors via java >>>>>>> monitors. >>>>>>> ?// Time permitting, we should disentangle and deconvolve the two >>>>>>> implementations >>>>>>> ?// and move the resulting raw monitor implementation over to the >>>>>>> JVMTI directories. >>>>>>> ?// Ideally, the raw monitor implementation would be built on top of >>>>>>> ?// park-unpark and nothing else. >>>>>>> >>>>>>> This is an attempt to do that disentangling so that we can then >>>>>>> consider changes to ObjectMonitor without having to worry about >>>>>>> JvmtiRawMonitors. But rather than building on low-level >>>>>>> park/unpark (which would require the same manual queue management >>>>>>> and much of the same complex code as exists in ObjectMonitor) I >>>>>>> decided to try and do this on top of PlatformMonitor. >>>>>>> >>>>>>> The reason this is just a RFC rather than RFR is that I >>>>>>> overlooked a non-trivial aspect of JvmtiRawMonitors: like Java >>>>>>> monitors (as implemented by ObjectMonitor) they interact with the >>>>>>> Thread.interrupt mechanism. This is not clearly stated in the JVM >>>>>>> TI specification [1] but only in passing by the possible errors >>>>>>> for RawMonitorWait: >>>>>>> >>>>>>> JVMTI_ERROR_INTERRUPT??? Wait was interrupted, try again >>>>>>> >>>>>>> As I explain in the bug report there is no way to build in proper >>>>>>> interrupt support using PlatformMonitor as there is no way we can >>>>>>> "interrupt" the low-level pthread_cond_wait. But we can >>>>>>> approximate it. What I've done in this preliminary version is >>>>>>> just check interrupt state before and after the actual "wait" but >>>>>>> we won't get woken by the interrupt once we have actually >>>>>>> blocked. Alternatively we could use a periodic polling approach >>>>>>> and wakeup every Nms to check for interruption. >>>>>>> >>>>>>> The only use of JvmtiRawMonitors in the JDK libraries (JDWP) is >>>>>>> not affected by this choice as that code ignores the interrupt >>>>>>> until the real action it was waiting for has occurred. The >>>>>>> interrupt is then reposted later. >>>>>>> >>>>>>> But more generally there could be users of JvmtiRawMonitors that >>>>>>> expect/require that RawMonitorWait is responsive to >>>>>>> Thread.interrupt in a manner similar to Object.wait. And if any >>>>>>> of them are reading this then I'd like to know - hence this RFC :) >>>>>>> >>>>>>> FYI testing to date: >>>>>>> ?- tiers 1 -3 all platforms >>>>>>> ?- hotspot: serviceability/jvmti >>>>>>> ????????????????????????? /jdwp >>>>>>> ??????????? vmTestbase/nsk/jvmti >>>>>>> ????????????????????????? /jdwp >>>>>>> ?- JDK: com/sun/jdi >>>>>>> >>>>>>> Comments/opinions appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> [1] >>>>>>> https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#RawMonitorWait >>>>>>> >>>>>> >>>> From serguei.spitsyn at oracle.com Fri Sep 20 06:02:26 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 19 Sep 2019 23:02:26 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> Hi Paul, I have almost the same comments as David: ?- the same two spots of changes identified ?- the addition of the default method was expected ?- the change in test is a surprise (I also doubt, it is really needed) ?- new CSR is needed Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( Thanks, Serguei On 9/19/19 16:06, David Holmes wrote: > Hi Paul, > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: >> More formally, >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > I'm assuming there are only two changes here: > > 1. The new method is now a default method that throws UOE. > > That seems fine. > > 2. You implemented the new method in the test class. > > I don't understand why you did that. The test can't be calling the new > method. Now that it is a default method we will get past the > compilation failure that caused the problem. So no change to the test > should be needed AFAICS. > > A new CSR request is needed. Just copy everything across from the old, > with the updated spec. But please also mention this is a > PlatformManagedObject in the compatibility discussion. > > Thanks, > David > >> Thanks, >> >> ?On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, >> Paul" > hohensee at amazon.com> wrote: >> >> ???? Off by 2 error. Changed the subject to reflect 8231209. >> ???? ???? http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> ???? ???? Paul >> ???? ???? On 9/19/19, 6:31 AM, "Daniel D. Daugherty" >> wrote: >> ???? ????????? > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> ???????? ???????? The redo bug is 8231209. 8231211 is closed as a dup >> of 8231210. >> ???????? ???????? Dan >> ???????? ???????? ???????? On 9/19/19 9:17 AM, Hohensee, Paul wrote: >> ???????? > I'll have the default method throw UOE. That's the same as >> the other default methods do. >> ???????? > >> ???????? > The necessary test fix is in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, >> which needs a new getCurrentThreadAllocatedBytes method, defined as >> ???????? > >> ???????? >????? public long getCurrentThreadAllocatedBytes() { >> ???????? >????????? return (Long) >> invokeMethod("getCurrentThreadAllocatedBytes", >> ???????? >????????????? new Object[] { }, >> ???????? >????????????? new String[] { }); >> ???????? >????? } >> ???????? > >> ???????? > With this fix, the 134 tests in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. >> Preliminary webrev at >> ???????? > >> ???????? > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> ???????? > >> ???????? > Is it worth adding getCurrentThreadAllocatedBytes tests to >> the >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes >> set? >> ???????? > >> ???????? > Paul >> ???????? > >> ???????? > On 9/18/19, 8:16 PM, "David Holmes" >> wrote: >> ???????? > >> ???????? >????? On 19/09/2019 12:57 pm, Mandy Chung wrote: >> ???????? >????? > On 9/18/19 5:00 PM, Hohensee, Paul wrote: >> ???????? >????? >> They all implement >> com.sun.management.ThreadMXBean, so adding a >> ???????? >????? >> getCurrentThreadAllocatedBytes broke them. >> Potential fix is to give it >> ???????? >????? >> a default implementation, vis >> ???????? >????? >> >> ???????? >????? >>????? public default long >> getCurrentThreadAllocatedBytes() { >> ???????? >????? >>????????? return -1; >> ???????? >????? >>????? } >> ???????? >????? >> >> ???????? >????? > >> ???????? >????? > com.sun.management.ThreadMXBean (and other platform >> MXBeans) is a >> ???????? >????? > "sealed" interface which should only be implemented >> by JDK. >> ???????? > >> ???????? >????? Didn't realize that. I don't recall knowing about >> PlatformManagedObject. >> ???????? >????? Sealed types will at least allow this to be enforced, >> though I have to >> ???????? >????? wonder what the tests are doing here. >> ???????? > >> ???????? >????? > Unfortunately we don't have the sealed type feature >> yet.? Yes it needs >> ???????? >????? > to be a default method.? I think it should throw UOE. >> ???????? >????? > >> ???????? >????? >?????? * @implSpec >> ???????? >????? >?????? * The default implementation throws {@code >> ???????? >????? > UnsupportedOperationException}. >> ???????? >????? > >> ???????? >????? > The @throw UOE can make it clear that it does not >> support current thread >> ???????? >????? > memory allocation measurement. >> ???????? > >> ???????? >????? Yes that seems a reasonable default if we don't want >> this to be >> ???????? >????? implemented outside the platform. >> ???????? > >> ???????? >????? Thanks, >> ???????? >????? David >> ???????? > >> ???????? >????? > Mandy >> ???????? > >> ???????? > >> From richard.reingruber at sap.com Fri Sep 20 08:31:05 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Fri, 20 Sep 2019 08:31:05 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi David, > On 20/09/2019 2:42 am, Reingruber, Richard wrote: > > Hi David, > > > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > > optimizations like scalar replacement become invalid. > > > > > > What bothers me about this is that it seems like escape analysis is > > > doing something wrong in this case. > > > > Yes it is. > > > > > If the object is thread-local but is > > > being synchronized upon then either: > > > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > > machine, as observed through JVMTI, is wrong. See below... > > You seem to have completely missed my point. If the object is local and > is synchronized upon then the synchronization can be elided (and should > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > escape. If the synchronization cannot be elided then the object cannot > be considered local. That is how Escape Analysis should be operating > here IMHO. I presume we agree that it is the state of the abstract virtual machine that must be observed through JVMTI, right? The locking state of an object O after a monitorenter on O is locked on the abstract vm. The JIT can still elide synchronization based on a prove that it is actually redundant for the computation. But at a safepoint JVMTI must report O as locked, because that's its state on the abstract virtual machine. Cheers, Richard. -----Original Message----- From: David Holmes Sent: Freitag, 20. September 2019 00:59 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken On 20/09/2019 2:42 am, Reingruber, Richard wrote: > Hi David, > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > optimizations like scalar replacement become invalid. > > > > What bothers me about this is that it seems like escape analysis is > > doing something wrong in this case. > > Yes it is. > > > If the object is thread-local but is > > being synchronized upon then either: > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > machine, as observed through JVMTI, is wrong. See below... You seem to have completely missed my point. If the object is local and is synchronized upon then the synchronization can be elided (and should be) in which case it won't appear in GetOwnedMonitorInfo and so does not escape. If the synchronization cannot be elided then the object cannot be considered local. That is how Escape Analysis should be operating here IMHO. Cheers, David ----- > > a) the synchronization is elided and so the object will not appear in > > the set of owned monitors; or > > b) the fact synchronization occurs renders the object ineligible to be > > considered thread-local, and so there is no problem with it appearing in > > the set of owned monitors > > > > I think there is a bigger design philosophy issue here about the > > interaction of escape analysis and debugging/management frameworks in > > general. I'd like to see a very clear description on exactly how they > > should interact. > > > > I don't see too many design alternatives here. The JVMTI implementation has to present the correct > state of the virtual machine according to the spec. I think it fails to do so in this case. > > Please look again at the test: > > 172 public long dontinline_endlessLoop() { > 173 long cs = checkSum; > 174 while (doLoop && loopCount-- > 0) { > 175 targetIsInLoop = true; > 176 checkSum += checkSum % ++cs; > 177 } > 178 loopCount = 3; > 179 targetIsInLoop = false; > 180 return checkSum; > 181 } > > 249 public void dontinline_testMethod() { > 250 LockCls l1 = new LockCls(); // to be scalar replaced > 251 synchronized (l1) { > 252 inlinedTestMethodWithNestedLocking(l1); > 253 } > 254 } > 255 > 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { > 257 synchronized (l1) { // nested > 258 dontinline_endlessLoop(); > 259 } > 260 } > > This is the stack when the agent calls GetOwnedMonitorInfo() > > dontinline_endlessLoop() at line 176 > inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame > dontinline_testMethod() at line 252 // compiled frame > > The state of the _virtual_ machine at that point is obvious: > > - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. > - That instance was locked by a monitorenter bytecode at line 251 > > This could be proven by interpreting the execution trace bytecode by bytecode using paper and > pencil (hope you won't make me do this, though ;)) > > JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call > GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does > not. > > From a more philosophical point of view compiled code is free to change the state of the physical > machine in a way such that it cannot be mapped to a valid state of the virtual machine after each > and every machine instruction. But it must reach points in its execution trace, where it is actually > possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These > points are called safepoints. > > The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm > state is presented. EA does not take into account that the lock state can be observed using > GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability > can_get_owned_monitor_info is taken. With the fix the test passes. > > Note that for the very same reason EA is disabled if can_access_local_variables is taken, because > the JVMTI implementation cannot handout references to objects stored in local variables if they were > scalar replaced. > > With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA > based optimizations just-in-time before local objects escape. Note that EA opts are already reverted > today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() > and relock_objects() in class Deoptimization) > > Thanks and cheers, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > > -----Original Message----- > From: David Holmes > Sent: Donnerstag, 19. September 2019 02:43 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 7/09/2019 12:24 am, Reingruber, Richard wrote: >> Hi, >> >> could I please get reviews for >> >> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >> >> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> retrieve objects locked by a thread. In terms of escape analysis those references escape and >> optimizations like scalar replacement become invalid. > > What bothers me about this is that it seems like escape analysis is > doing something wrong in this case. If the object is thread-local but is > being synchronized upon then either: > a) the synchronization is elided and so the object will not appear in > the set of owned monitors; or > b) the fact synchronization occurs renders the object ineligible to be > considered thread-local, and so there is no problem with it appearing in > the set of owned monitors > > I think there is a bigger design philosophy issue here about the > interaction of escape analysis and debugging/management frameworks in > general. I'd like to see a very clear description on exactly how they > should interact. > > Cheers, > David > >> The runtime currently cannot cope with objects escaping through JVMTI (try included >> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >> >> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >> objects escape through JVMTI. >> >> I've run tier1 tests. >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> From serguei.spitsyn at oracle.com Fri Sep 20 08:31:22 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 01:31:22 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: <1c4b0962-b418-216b-4657-78957cf4b9cf@oracle.com> An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Fri Sep 20 09:07:12 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 19:07:12 +1000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi Richard, On 20/09/2019 6:31 pm, Reingruber, Richard wrote: > Hi David, > > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: > > > Hi David, > > > > > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > > > optimizations like scalar replacement become invalid. > > > > > > > > What bothers me about this is that it seems like escape analysis is > > > > doing something wrong in this case. > > > > > > Yes it is. > > > > > > > If the object is thread-local but is > > > > being synchronized upon then either: > > > > > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > > > machine, as observed through JVMTI, is wrong. See below... > > > > You seem to have completely missed my point. If the object is local and > > is synchronized upon then the synchronization can be elided (and should > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > escape. If the synchronization cannot be elided then the object cannot > > be considered local. That is how Escape Analysis should be operating > > here IMHO. > > I presume we agree that it is the state of the abstract virtual machine that must be observed > through JVMTI, right? > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > abstract virtual machine. I don't agree. If the locking can be elided then it is completely elided. If the state of the "abstract VM" had to be perfectly preserved then we wouldn't be able to do the optimisations referred to in JLS 12.6: "Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable." I place lock elision in the same category of "optimising transformations" that changes what would "naively" be expected. Now this should be explicitly covered in the JLS/JVMS somewhere but I'm having trouble finding exactly where. This article discusses lock elision: https://www.ibm.com/developerworks/library/j-jtp10185/index.html and states: "It stands to reason, then, that if a thread enters a synchronized block protected by a lock that no other thread will ever synchronize on, then that synchronization has no effect and can therefore be removed by the optimizer. (The Java Language Specification explicitly allows this optimization.)" I'll have to ping Brian to see if he recalls exactly where this is covered. :) David ----- > > Cheers, Richard. > > -----Original Message----- > From: David Holmes > Sent: Freitag, 20. September 2019 00:59 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >> Hi David, >> >> thanks for looking at this issue. And my appologies for the lengthy mail... >> >> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >> > > optimizations like scalar replacement become invalid. >> > >> > What bothers me about this is that it seems like escape analysis is >> > doing something wrong in this case. >> >> Yes it is. >> >> > If the object is thread-local but is >> > being synchronized upon then either: >> >> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >> machine, as observed through JVMTI, is wrong. See below... > > You seem to have completely missed my point. If the object is local and > is synchronized upon then the synchronization can be elided (and should > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > escape. If the synchronization cannot be elided then the object cannot > be considered local. That is how Escape Analysis should be operating > here IMHO. > > Cheers, > David > ----- > >> > a) the synchronization is elided and so the object will not appear in >> > the set of owned monitors; or >> > b) the fact synchronization occurs renders the object ineligible to be >> > considered thread-local, and so there is no problem with it appearing in >> > the set of owned monitors >> > >> > I think there is a bigger design philosophy issue here about the >> > interaction of escape analysis and debugging/management frameworks in >> > general. I'd like to see a very clear description on exactly how they >> > should interact. >> > >> >> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >> state of the virtual machine according to the spec. I think it fails to do so in this case. >> >> Please look again at the test: >> >> 172 public long dontinline_endlessLoop() { >> 173 long cs = checkSum; >> 174 while (doLoop && loopCount-- > 0) { >> 175 targetIsInLoop = true; >> 176 checkSum += checkSum % ++cs; >> 177 } >> 178 loopCount = 3; >> 179 targetIsInLoop = false; >> 180 return checkSum; >> 181 } >> >> 249 public void dontinline_testMethod() { >> 250 LockCls l1 = new LockCls(); // to be scalar replaced >> 251 synchronized (l1) { >> 252 inlinedTestMethodWithNestedLocking(l1); >> 253 } >> 254 } >> 255 >> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >> 257 synchronized (l1) { // nested >> 258 dontinline_endlessLoop(); >> 259 } >> 260 } >> >> This is the stack when the agent calls GetOwnedMonitorInfo() >> >> dontinline_endlessLoop() at line 176 >> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >> dontinline_testMethod() at line 252 // compiled frame >> >> The state of the _virtual_ machine at that point is obvious: >> >> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >> - That instance was locked by a monitorenter bytecode at line 251 >> >> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >> pencil (hope you won't make me do this, though ;)) >> >> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >> not. >> >> From a more philosophical point of view compiled code is free to change the state of the physical >> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >> and every machine instruction. But it must reach points in its execution trace, where it is actually >> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >> points are called safepoints. >> >> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >> state is presented. EA does not take into account that the lock state can be observed using >> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >> can_get_owned_monitor_info is taken. With the fix the test passes. >> >> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >> the JVMTI implementation cannot handout references to objects stored in local variables if they were >> scalar replaced. >> >> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >> and relock_objects() in class Deoptimization) >> >> Thanks and cheers, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> >> -----Original Message----- >> From: David Holmes >> Sent: Donnerstag, 19. September 2019 02:43 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> Hi Richard, >> >> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>> Hi, >>> >>> could I please get reviews for >>> >>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>> >>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> optimizations like scalar replacement become invalid. >> >> What bothers me about this is that it seems like escape analysis is >> doing something wrong in this case. If the object is thread-local but is >> being synchronized upon then either: >> a) the synchronization is elided and so the object will not appear in >> the set of owned monitors; or >> b) the fact synchronization occurs renders the object ineligible to be >> considered thread-local, and so there is no problem with it appearing in >> the set of owned monitors >> >> I think there is a bigger design philosophy issue here about the >> interaction of escape analysis and debugging/management frameworks in >> general. I'd like to see a very clear description on exactly how they >> should interact. >> >> Cheers, >> David >> >>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>> >>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>> objects escape through JVMTI. >>> >>> I've run tier1 tests. >>> >>> Thanks, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>> From daniel.fuchs at oracle.com Fri Sep 20 11:16:43 2019 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 20 Sep 2019 12:16:43 +0100 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> Message-ID: <7de738c3-a9b8-5d49-870a-ae7392ead85a@oracle.com> Hi Paul, It might be worth double checking what happens if you create a MXBean proxy to access the com.sun.management.ThreadMXBean in a remote server: https://download.java.net/java/early_access/jdk14/docs/api/java.management/java/lang/management/ManagementFactory.html#getPlatformMXBean(javax.management.MBeanServerConnection,java.lang.Class) and https://download.java.net/java/early_access/jdk14/docs/api/java.management/javax/management/MBeanServerInvocationHandler.html#newProxyInstance(javax.management.MBeanServerConnection,javax.management.ObjectName,java.lang.Class,boolean) are two different ways of doing this. Will getCurrentThreadAllocatedBytes() return something (however uninteresting that might be) or will the default implementation be triggered on the client side and UnsuportedOperationsException be thrown directly without invoking the server? Whatever the result is - it might be worth mentioning in the CSR. Best regards, -- daniel On 19/09/2019 17:39, Hohensee, Paul wrote: > Off by 2 error. Changed the subject to reflect 8231209. > > http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > Paul From Michael.Rasmussen at roguewave.com Fri Sep 20 12:05:12 2019 From: Michael.Rasmussen at roguewave.com (Michael Rasmussen) Date: Fri, 20 Sep 2019 12:05:12 +0000 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> References: , <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Message-ID: On 9/18/19 2:47 PM, Sam Thomas wrote: > Hi, > > I'm trying to understand if a class will load as soon as all the transformers return. The aim is to get a class reference of a class I have seen in my transformer. Short answer: no. Trying to define a class can cause other classes to try to be loaded before (for instance the super types), meaning when you're done transforming class A, the same thread might try to load multiple other classes before it finishes defining A. And if any of these fail, A will also fail to be defined. > Also if there is way to get the same without triggering class loading - if the class is not loaded return a null reference. ClassLoader has a protected method?findLoadedClass that checks if a class is loaded by that classloader and returns the Class instance if so, or null otherwise. But as mentioned, the method is protected so normally not accessible outside the classloader itself. /Michael From daniil.x.titov at oracle.com Fri Sep 20 15:42:15 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Fri, 20 Sep 2019 08:42:15 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Hi Serguei, > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); If I understood you correctly in the code snippet you sent you meant to use Threads_lock, not ThreadTableCreate_lock, right? The original idea was to do a minimal amount of work while holding the lock and hold the lock for as short period of time as possible to not block other threads when it is not necessary. With the suggested approach no new threads could be started until the thread table is created and populated with all threads running inside a Java application and in case of large app there could be thousands of them. And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock is higher than the rank of ThreadTableCreate_lock. So choosing between blocking new threads from starting and potentially allowing some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. Thanks! Best regards, Daniil From: "serguei.spitsyn at oracle.com" Date: Thursday, September 19, 2019 at 10:30 PM To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) Hi Daniil, I think, it is better to grab the thread_lock just once at lazy initialization. It would look simpler, something, like this would work: void ThreadTable::lazy_initialize(const ThreadsList *threads) { if (_is_initialized) { return; } MutexLocker ml(ThreadTableCreate_lock); if (_is_initialized) { // There is no obvious benefits in allowing the thread table // being concurrently populated during the initalization. return; } create_table(threads->length()); _is_initialized = true; for (uint i = 0; i < threads->length(); i++) { JavaThread* thread = threads->thread_at(i); oop tobj = thread->threadObj(); if (tobj != NULL) { jlong java_tid = java_lang_Thread::thread_id(tobj); if (!thread->is_exiting()) { // Must be inside the lock to ensure that we don't add the thread to the table // that has just passed the removal point in ThreadsSMRSupport::remove_thread() add_thread(java_tid, thread); } } } } Otherwise, concurrent executions of the find_JavaThread_from_java_tid() will sometimes do a linear search of threads that are not included yet to the ThreadTable from the ThreadsList (which is used for lazy initialization). Instead, it is better to wait for the lazy_initialization() to complete. Thanks, Serguei On 9/19/19 17:30, Daniil Titov wrote: Hi David and Serguei, Please review new version of the fix that includes the changes Serguei suggested: 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread that has just passed the removal point in ThreadsSMRSupport::remove_thread() The changes are in ThreadTable::lazy_initialize() method only. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! --Daniil ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From richard.reingruber at sap.com Fri Sep 20 16:18:04 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Fri, 20 Sep 2019 16:18:04 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi David, > > > You seem to have completely missed my point. If the object is local and > > > is synchronized upon then the synchronization can be elided (and should > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > > escape. If the synchronization cannot be elided then the object cannot > > > be considered local. That is how Escape Analysis should be operating > > > here IMHO. > > > > I presume we agree that it is the state of the abstract virtual machine that must be observed > > through JVMTI, right? > > > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > > abstract virtual machine. > > I don't agree. If the locking can be elided then it is completely > elided. If the state of the "abstract VM" had to be perfectly preserved > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > "Optimizing transformations of a program can be designed that reduce the > number of objects that are reachable to be less than those which would > naively be considered reachable." Interestingly this optimization is disabled, if a JVMTI agent can_access_local_variables. (see ciMethod::liveness_at_bci()) > I place lock elision in the same category of "optimising > transformations" that changes what would "naively" be expected. Now this > should be explicitly covered in the JLS/JVMS somewhere but I'm having > trouble finding exactly where. This article discusses lock elision: > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > and states: > > "It stands to reason, then, that if a thread enters a synchronized block > protected by a lock that no other thread will ever synchronize on, then > that synchronization has no effect and can therefore be removed by the > optimizer. (The Java Language Specification explicitly allows this > optimization.)" > > I'll have to ping Brian to see if he recalls exactly where this is > covered. :) In the formal specification of the memory model, I'd guess. Can be confusing, though, if you see in your debugger, that your object gets locked (interpreted), then its not locked (compiled) and then in the same activation suddenly locked, because an uncommon trap was hit. Thanks and a good weekend to you! - Actually to every reader :) Richard. -----Original Message----- From: David Holmes Sent: Freitag, 20. September 2019 11:07 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Richard, On 20/09/2019 6:31 pm, Reingruber, Richard wrote: > Hi David, > > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: > > > Hi David, > > > > > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > > > optimizations like scalar replacement become invalid. > > > > > > > > What bothers me about this is that it seems like escape analysis is > > > > doing something wrong in this case. > > > > > > Yes it is. > > > > > > > If the object is thread-local but is > > > > being synchronized upon then either: > > > > > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > > > machine, as observed through JVMTI, is wrong. See below... > > > > You seem to have completely missed my point. If the object is local and > > is synchronized upon then the synchronization can be elided (and should > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > escape. If the synchronization cannot be elided then the object cannot > > be considered local. That is how Escape Analysis should be operating > > here IMHO. > > I presume we agree that it is the state of the abstract virtual machine that must be observed > through JVMTI, right? > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > abstract virtual machine. I don't agree. If the locking can be elided then it is completely elided. If the state of the "abstract VM" had to be perfectly preserved then we wouldn't be able to do the optimisations referred to in JLS 12.6: "Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable." I place lock elision in the same category of "optimising transformations" that changes what would "naively" be expected. Now this should be explicitly covered in the JLS/JVMS somewhere but I'm having trouble finding exactly where. This article discusses lock elision: https://www.ibm.com/developerworks/library/j-jtp10185/index.html and states: "It stands to reason, then, that if a thread enters a synchronized block protected by a lock that no other thread will ever synchronize on, then that synchronization has no effect and can therefore be removed by the optimizer. (The Java Language Specification explicitly allows this optimization.)" I'll have to ping Brian to see if he recalls exactly where this is covered. :) David ----- > > Cheers, Richard. > > -----Original Message----- > From: David Holmes > Sent: Freitag, 20. September 2019 00:59 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >> Hi David, >> >> thanks for looking at this issue. And my appologies for the lengthy mail... >> >> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >> > > optimizations like scalar replacement become invalid. >> > >> > What bothers me about this is that it seems like escape analysis is >> > doing something wrong in this case. >> >> Yes it is. >> >> > If the object is thread-local but is >> > being synchronized upon then either: >> >> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >> machine, as observed through JVMTI, is wrong. See below... > > You seem to have completely missed my point. If the object is local and > is synchronized upon then the synchronization can be elided (and should > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > escape. If the synchronization cannot be elided then the object cannot > be considered local. That is how Escape Analysis should be operating > here IMHO. > > Cheers, > David > ----- > >> > a) the synchronization is elided and so the object will not appear in >> > the set of owned monitors; or >> > b) the fact synchronization occurs renders the object ineligible to be >> > considered thread-local, and so there is no problem with it appearing in >> > the set of owned monitors >> > >> > I think there is a bigger design philosophy issue here about the >> > interaction of escape analysis and debugging/management frameworks in >> > general. I'd like to see a very clear description on exactly how they >> > should interact. >> > >> >> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >> state of the virtual machine according to the spec. I think it fails to do so in this case. >> >> Please look again at the test: >> >> 172 public long dontinline_endlessLoop() { >> 173 long cs = checkSum; >> 174 while (doLoop && loopCount-- > 0) { >> 175 targetIsInLoop = true; >> 176 checkSum += checkSum % ++cs; >> 177 } >> 178 loopCount = 3; >> 179 targetIsInLoop = false; >> 180 return checkSum; >> 181 } >> >> 249 public void dontinline_testMethod() { >> 250 LockCls l1 = new LockCls(); // to be scalar replaced >> 251 synchronized (l1) { >> 252 inlinedTestMethodWithNestedLocking(l1); >> 253 } >> 254 } >> 255 >> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >> 257 synchronized (l1) { // nested >> 258 dontinline_endlessLoop(); >> 259 } >> 260 } >> >> This is the stack when the agent calls GetOwnedMonitorInfo() >> >> dontinline_endlessLoop() at line 176 >> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >> dontinline_testMethod() at line 252 // compiled frame >> >> The state of the _virtual_ machine at that point is obvious: >> >> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >> - That instance was locked by a monitorenter bytecode at line 251 >> >> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >> pencil (hope you won't make me do this, though ;)) >> >> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >> not. >> >> From a more philosophical point of view compiled code is free to change the state of the physical >> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >> and every machine instruction. But it must reach points in its execution trace, where it is actually >> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >> points are called safepoints. >> >> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >> state is presented. EA does not take into account that the lock state can be observed using >> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >> can_get_owned_monitor_info is taken. With the fix the test passes. >> >> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >> the JVMTI implementation cannot handout references to objects stored in local variables if they were >> scalar replaced. >> >> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >> and relock_objects() in class Deoptimization) >> >> Thanks and cheers, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> >> -----Original Message----- >> From: David Holmes >> Sent: Donnerstag, 19. September 2019 02:43 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> Hi Richard, >> >> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>> Hi, >>> >>> could I please get reviews for >>> >>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>> >>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> optimizations like scalar replacement become invalid. >> >> What bothers me about this is that it seems like escape analysis is >> doing something wrong in this case. If the object is thread-local but is >> being synchronized upon then either: >> a) the synchronization is elided and so the object will not appear in >> the set of owned monitors; or >> b) the fact synchronization occurs renders the object ineligible to be >> considered thread-local, and so there is no problem with it appearing in >> the set of owned monitors >> >> I think there is a bigger design philosophy issue here about the >> interaction of escape analysis and debugging/management frameworks in >> general. I'd like to see a very clear description on exactly how they >> should interact. >> >> Cheers, >> David >> >>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>> >>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>> objects escape through JVMTI. >>> >>> I've run tier1 tests. >>> >>> Thanks, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>> From chris.plummer at oracle.com Fri Sep 20 16:36:09 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 20 Sep 2019 09:36:09 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: <1c4b0962-b418-216b-4657-78957cf4b9cf@oracle.com> References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> <1c4b0962-b418-216b-4657-78957cf4b9cf@oracle.com> Message-ID: <832a3a20-34b9-6944-7d6c-961f85debdd9@oracle.com> An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri Sep 20 18:06:45 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 11:06:45 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Message-ID: <53347b95-0a5d-7143-1a74-4951b4097489@oracle.com> Hi Daniil, You are right that we also have to grab the Threads_lock around the add_thread call to avoid adding exiting threads. Let me think a little bit more here. Thanks, Serguei On 9/20/19 08:42, Daniil Titov wrote: > Hi Serguei, > >> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> if (_is_initialized) { >> return; > > } >> MutexLocker ml(ThreadTableCreate_lock); > > If I understood you correctly in the code snippet you sent you meant to use > Threads_lock, not ThreadTableCreate_lock, right? > > The original idea was to do a minimal amount of work while holding the lock and > hold the lock for as short period of time as possible to not block other threads when it is not necessary. > > With the suggested approach no new threads could be started until the thread table is created and > populated with all threads running inside a Java application and in case of large app there could be > thousands of them. > > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around > thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock > is higher than the rank of ThreadTableCreate_lock. > > So choosing between blocking new threads from starting and potentially allowing > some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. > > Thanks! > > Best regards, > Daniil > > > > From: "serguei.spitsyn at oracle.com" > Date: Thursday, September 19, 2019 at 10:30 PM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > I think, it is better to grab the thread_lock just once at lazy initialization. > It would look simpler, something, like this would work: > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > // There is no obvious benefits in allowing the thread table > // being concurrently populated during the initalization. > return; > } > create_table(threads->length()); > _is_initialized = true; > > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > if (!thread->is_exiting()) { > // Must be inside the lock to ensure that we don't add the thread to the table > // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > add_thread(java_tid, thread); > } > } > } > } > > Otherwise, concurrent executions of the find_JavaThread_from_java_tid() > will sometimes do a linear search of threads that are not included yet to > the ThreadTable from the ThreadsList (which is used for lazy initialization). > Instead, it is better to wait for the lazy_initialization() to complete. > Thanks, > Serguei > > > On 9/19/19 17:30, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From christoph.langer at sap.com Fri Sep 20 21:13:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 20 Sep 2019 21:13:16 +0000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Message-ID: Thanks David! May I get another review? Best regards Christoph > -----Original Message----- > From: David Holmes > Sent: Donnerstag, 19. September 2019 13:56 > To: Langer, Christoph ; Erik Joelsson > ; Magnus Ihse Bursie > ; OpenJDK Serviceability dev at openjdk.java.net>; build-dev > Subject: Re: RFR: 8230857: Avoid reflection in > sun.tools.common.ProcessHelper > > Hi Christoph, > > On 19/09/2019 7:47 pm, Langer, Christoph wrote: > > Hi, > > > > @Erik, Magnus: Thanks for stepping in to explain things. > > > > Now back to the actual change: Is this ok then (@David)? Any other > reviews from somebody else? > > > > http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ > > It seems okay. > > For the test I'm unclear on exactly how to ensure things are accessible, > but presumably the +open is sufficient and works under all circumstances. > > Thanks, > David > > > Thank you! > > > > Best regards > > Christoph > > > >> -----Original Message----- > >> From: David Holmes > >> Sent: Mittwoch, 18. September 2019 01:13 > >> To: Erik Joelsson ; Magnus Ihse Bursie > >> ; Langer, Christoph > >> ; OpenJDK Serviceability >> dev at openjdk.java.net>; build-dev > >> Subject: Re: RFR: 8230857: Avoid reflection in > >> sun.tools.common.ProcessHelper > >> > >> Hi Erik, > >> > >> Thanks for the additional details (I can't say I fully understand them :) ). > >> > >> David > >> > >> On 17/09/2019 11:39 pm, Erik Joelsson wrote: > >>> Hello, > >>> > >>> On 2019-09-17 05:59, David Holmes wrote: > >>>> Hi Magnus, > >>>> > >>>> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: > >>>>> On 2019-09-17 01:01, David Holmes wrote: > >>>>>> Hi Christoph, > >>>>>> > >>>>>> Sorry for the delay getting back you. > >>>>>> > >>>>>> cc'd build-dev to get some clarification on the below ... > >>>>>> > >>>>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: > >>>>>>> Hi David, > >>>>>>> > >>>>>>>>> please review an enhancement which I've identified when > working > >> with > >>>>>>>>> Processhelper for JDK-8230850. > >>>>>>>>> > >>>>>>>>> I noticed that ProcessHelper is an interface in common code with > a > >>>>>>>>> static method that would lookup the actual platform > >>>>>>>>> implementation via > >>>>>>>>> reflection. This seems a little cumbersome since we can have a > >>>>>>>>> common > >>>>>>>>> dummy for ProcessHelper and override it with the platform > specific > >>>>>>>>> implementation, leveraging the build system. > >>>>>>>> > >>>>>>>> I don't see you leveraging the build system. You have two source > >>>>>>>> files > >>>>>>>> that compile to the same destination class file. What is ensuring > the > >>>>>>>> platform specific version is compiled after the generic one? > >>>>>>>> > >>>>>>>> Service-provider patterns use reflection to instantiate the service > >>>>>>>> implementation. I don't see any problem here that needs solving. > >>>>>>> > >>>>>>> TL;DR: > >>>>>>> There are two source files, one in share/classes and one in > >>>>>>> linux/classes. The build system overrides the share/classes > >>>>>>> implementation with the linux/classes implementation in the linux > >>>>>>> build. This is not by coincidence and only one class is contained > >>>>>>> in the generated jdk.jcmd module. Then there won't be a need for > >>>>>>> having a service interface and a service implementation that is > >>>>>>> looked up via reflection (which is not a bad pattern by itself). I > >>>>>>> agree that it's not a big problem to be solved but still not "no > >>>>>>> problem". > >>>>>>> Here is some longer elaboration how the build system prefers > >>>>>>> specific implementations of classes and filters generic duplicates: > >>>>>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] > is > >>>>>>> used to compile the java sources for JDK modules. In its > >>>>>>> documentation, for argument SRC [1], it claims: "one or more > >>>>>>> directories to search for sources. The order of the source roots is > >>>>>>> significant. The first found file of a certain name has priority". > >>>>>>> In its implementation the found files are first ordered [3] and > >>>>>>> duplicates filtered out [4]. > >>>>>>> The potential source files are handed to SetupJavaCompilation in > >>>>>>> CompileJavaModules.gmk [5] and were collected by a call to > >>>>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all > >>>>>>> potential source dirs for Java classes in the module [7]. The > >>>>>>> evaluated subdirs are (in that order) > >> $(OPENJDK_TARGET_OS)/classes, > >>>>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per > [8]. > >>>>>>> Hope that explains what I'm trying to leverage here. > >>>>>> > >>>>>> I'm not 100% certain that what you describe actually ensures what > >>>>>> you want it to ensure. I can't reconcile "the first found file ... > >>>>>> has priority" with the fact found files are sorted and duplicates > >>>>>> eliminated. It is the sorting that concerns me as it suggests > >>>>>> linux/Foo.java might replace shared/Foo.java, but if we're on > >>>>>> Windows then we have a problem! That said there is also this > >> comment: > >>>>>> > >>>>>> # Order src files according to the order of the src dirs. Correct > >>>>>> odering is > >>>>>> # needed for correct overriding between different source roots. > >>>>>> > >>>>>> I'd need the build team to clarify what "correct overriding" is > >>>>>> actually defined as. > >>>>> David, > >>>>> > >>>>> Christoph is correct. linux/Foo.java will override share/Foo.java. I > >>>>> don't remember how the magic in JavaCompilation.gmk works > anymore > >>>>> :-), but we have relied on this behavior in other places for a long > >>>>> time, so I'm pretty certain it is still working correctly. > >>>>> Presumably, the $(sort ...) is there to remove (identical) > >>>>> duplicates, which is a side-effect of sort. > >>>> > >>>> Thanks for confirming. I'd still like to understand exactly what these > >>>> overriding rules are though. It's not a mechanism I was aware of. > >>>> > >>> SetupJavaCompilation is indeed behaving as Christoph describes and it is > >>> by design. I implemented support for this behavior in: > >>> > >>> https://bugs.openjdk.java.net/browse/JDK-8079344 > >>> > >>> The relevant parts of SetupJavaCompilation look like this: > >>> > >>> ? # Order src files according to the order of the src dirs. Correct > >>> odering is > >>> ? # needed for correct overriding between different source roots. > >>> ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) > >>> ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ > >>> ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) > >>> > >>> The second line orders the src files by the src roots. (We used to just > >>> call find for one src root at a time, but the above actually performs > >>> better due only running 1 external process) > >>> > >>> Further down we have this: > >>> > >>> ? ifneq ($$($1_KEEP_DUPS), true) > >>> ??? # Remove duplicate source files by keeping the first found of each > >>> duplicate. > >>> ??? # This allows for automatic overrides with custom or platform > >>> specific versions > >>> ??? # source files. > >>> ??? # > >>> ??? # For the smart javac wrapper case, add each removed file to an > >>> extra exclude > >>> ??? # file list to prevent sjavac from finding duplicate sources. > >>> ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ > >>> ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), > >>> $$(s))) \ > >>> ??????? $$(if $$($1_$$(relative_src)), \ > >>> ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ > >>> ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) > >>> ? endif > >>> > >>> This loop is a bit hairy to wrap your head around. It's iterating over > >>> all the src files, in the order of importance. The variable relative_src > >>> is the path from the src root, the part that is common to all duplicate > >>> src files. The variables on the form $1_$$(relative_src) basically act > >>> as a hash map (string->boolean). So for each src file, if the relative > >>> path for it has already been seen, add it to an exclude list, else mark > >>> it as seen and add it to the return list. > >>> > >>> /Erik > >>> From daniel.daugherty at oracle.com Fri Sep 20 21:59:09 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 17:59:09 -0400 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Daniil, Thanks for sticking with this project through the many versions. Sorry this review is late... On 9/19/19 8:30 PM, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ src/hotspot/share/runtime/mutexLocker.hpp ??? No comments. src/hotspot/share/runtime/mutexLocker.cpp ??? No comments. src/hotspot/share/runtime/threadSMR.cpp ??? L623: ??????? MutexLocker ml(Threads_lock); ??? L626: ??????? if (!thread->is_exiting()) { ??????? Re: discussion about is_exiting() ??????? The header comment is pretty clear: ????????? src/hotspot/share/runtime/thread.hpp: ??????????? // thread has called JavaThread::exit() or is terminated ??????????? bool is_exiting() const; ??????? is_exiting() might become true right after you have called it, ??????? but its purpose is to ask the question and not prevent the ??????? condition from becoming true. As David said, you should consider ??????? it an optimization. If you happen to see the condition is true, ??????? then you know that the JavaThread isn't going to be around much ??????? longer and should act accordingly. ??????? The is_exiting() implementation is: ????????? inline bool JavaThread::is_exiting() const { ? ? ? ? ? ? // Use load-acquire so that setting of _terminated by ? ? ? ? ? ? // JavaThread::exit() is seen more quickly. ? ? ? ? ? ? TerminatedTypes l_terminated = (TerminatedTypes) ? ? ? ? ?? ???? OrderAccess::load_acquire((volatile jint *) &_terminated); ? ? ? ? ??? return l_terminated == _thread_exiting || check_is_terminated(l_terminated); ????????? } ??????? and it depends on the JavaThread's _terminated field value. ? ? ? ? ? // JavaThread termination support ? ? ? ? ? enum TerminatedTypes { ? ? ? ? ?? _not_terminated = 0xDEAD - 2, ? ? ? ? ?? _thread_exiting,???????????????????????????? // JavaThread::exit() has been called for this thread ? ? ? ? ?? _thread_terminated,????????????????????????? // JavaThread is removed from thread list ? ? ? ? ?? _vm_exited?????????????????????????????????? // JavaThread is still executing native code, but VM is terminated ????????????????????????????????? ? ? ? ? ????????????? // only VM_Exit can set _vm_exited ? ? ? ??? }; ??????? so the JavaThread's _terminated field can get set to ??????? _thread_exiting independent of the Threads_lock, but ??????? it can't get set to _thread_terminated without the ??????? Threads_lock. ??????? So by grabbing the Threads_lock on L623, you make sure ??????? that ThreadTable::add_thread(java_tid, thread) does not ??????? add a JavaThread that's not on the ThreadsList. It might ??????? still become is_exiting() == true right after your ????????? L626???????? if (!thread->is_exiting()) { ??????? but it will still be on the main ThreadsList. And that ??????? means that when the JavaThread is removed from the main ??????? ThreadsList, you'll still call: ????????? L931: ??? ThreadTable::remove_thread(tid); ??? L624: ??????? // Must be inside the lock to ensure that we don't add the thread to the table ??????? typo: s/the thread/a thread/ ??? L633: ????? return thread; ??????? nit - L633 - indented too far (should be 2 spaces) src/hotspot/share/services/threadTable.hpp ??? L42: ? static void lazy_initialize(const ThreadsList *threads); ??????? nit - put space between '*' the variable: ????????? static void lazy_initialize(const ThreadsList* threads); ??????? like you do in your other decls. ??? L45: ? // Lookup and inserts ??????? Perhaps:? // Lookup and list management ??? L60-61 - nit - please delete these blank lines. src/hotspot/share/services/threadTable.cpp ??? L28: #include "runtime/timerTrace.hpp" ??????? nit - This should be after threadSMR.hpp... (alpha sorted order) ??? L39: static const size_t DefaultThreadTableSizeLog = 8; ??????? nit - your other 'static const' are not CamelCase. Why is this one? ??? L45: static ThreadTableHash* volatile _local_table = NULL; ??? L50: static volatile size_t _current_size = 0; ??? L51: static volatile size_t _items_count = 0; ??????? nit - can you group the file statics together? (up with L41). ??? L60:???? _tid(tid),_java_thread(java_thread) {} ??????? nit - space after ',' ??? L62?? jlong tid() const { return _tid;} ??? L63?? JavaThread* thread() const {return _java_thread;} ??????? nit - space before '}' ??????? nit - space after '{' on L63. ??? L70: ??? static uintx get_hash(Value const& value, bool* is_dead) { ??????? Parameter 'is_dead' is not used. ??? L74: ??? static void* allocate_node(size_t size, Value const& value) { ??????? Parameter 'value' is not used. ??? L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { ??????? Re: discussion about lazy_initialize() racing with ??????????? ThreadsList::find_JavaThread_from_java_tid() ??????? There's a couple of aspects to these two pieces of code racing ??????? with each other and racing with new thread creation. Racing with ??????? new thread creation is the easy one: ????????? If a new thread isn't added to the ThreadTable by ????????? ThreadsSMRSupport::add_thread() calling ThreadTable::add_thread(), ????????? then the point in the future where someone calls ????????? find_JavaThread_from_java_tid() will add it to the table due to ????????? the linear search when ThreadTable::find_thread_by_tid() ????????? returns NULL. ?????? As for multi-threads calling ThreadsList::find_JavaThread_from_java_tid() ?????? at the same time which results in multi-threads in lazy_initialize() ?????? at the same time... ?????? - ThreadTable creation will be linear due to ThreadTableCreate_lock. ???????? After _is_initialized is set to true, then no more callers to ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. ?????? - Once the ThreadTable is created, then multi-threads can be ???????? executing the for-loop to add their ThreadsList entries to ???????? the ThreadTable. There will be a bit of Threads_lock contention ???????? as each of the multi-threads tries to add their entries and ???????? there will be some wasted work since the multi-threads will ???????? likely have similar ThreadLists. ?????? Of course, once _is_initialized is set to true, then any caller ?????? to lazy_initialize() will return quickly and ?????? ThreadsList::find_JavaThread_from_java_tid() will call ?????? ThreadTable::find_thread_by_tid(). If the target java_tid isn't ?????? found, then we do the linear search thing here and add the ?????? the entry if we find a match in our current ThreadsList. Since ?????? we're only adding the one here, we only contend for the Threads_lock ?????? here if we find it. ?????? If ThreadsList::find_JavaThread_from_java_tid() is called with a ?????? target java_tid for a JavaThread that was created after the ?????? ThreadsList object that the caller has in hand for the ?????? find_JavaThread_from_java_tid() call, then, of course, that ?????? target 'java_tid' won't be found because the JavaThread was ?????? added the main ThreadsList _after_ the ThreadsList object was ?????? created by the caller. Of course, you have to ask where the ?????? target java_tid value came from since the JavaThread wasn't ?????? around when the ThreadsList::find_JavaThread_from_java_tid() ?????? call was made with that target java_tid value... ??? L99: ??????? // being concurently populated during the initalization. ??????? Typos? Perhaps: ???????????????? // to be concurrently populated during initialization. ??????? But I think those two comment lines are more appropriate above ??????? this line: ??????? L96: ????? MutexLocker ml(ThreadTableCreate_lock); ??? L112: ????????? // Must be inside the lock to ensure that we don't add the thread to the table ??????? typo: s/the thread/a thread/ ??? L141: ? return ((double)_items_count)/_current_size; ??????? nit - need spaces around '/'. ??? L177: ? bool equals(ThreadTableEntry **value, bool* is_dead) { ??????? nit - put space between '**' the variable: ??????????? bool equals(ThreadTableEntry** value, ??????? Parameter 'is_dead' is not used. ??? L214: ? while(true) { ??????? nit - space before '('. Short version: Thumbs up. Longer version: I don't think I've spotted anything other than nits here. Mostly I've just looked for multi-threaded races, proper usage of the Thread-SMR stuff, and minimal impact in the case where the new ThreadsTable is never needed. Dan P.S. ThreadTable is a bit of misnomer. What you really have here is a ThreadIdTable, but I'm really late to the code review flow with that comment... > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil From serguei.spitsyn at oracle.com Fri Sep 20 23:15:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 16:15:35 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Message-ID: <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> Hi Dan, Please, find a minor correction below. On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes >> Serguei suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ . . . > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??????? Re: discussion about lazy_initialize() racing with > ??????????? ThreadsList::find_JavaThread_from_java_tid() > > ??????? There's a couple of aspects to these two pieces of code racing > ??????? with each other and racing with new thread creation. Racing with > ??????? new thread creation is the easy one: > > ????????? If a new thread isn't added to the ThreadTable by > ????????? ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > ????????? then the point in the future where someone calls > ????????? find_JavaThread_from_java_tid() will add it to the table due to > ????????? the linear search when ThreadTable::find_thread_by_tid() > ????????? returns NULL. > > ?????? As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > ?????? at the same time which results in multi-threads in > lazy_initialize() > ?????? at the same time... > > ?????? - ThreadTable creation will be linear due to > ThreadTableCreate_lock. > ???????? After _is_initialized is set to true, then no more callers to > ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. > ?????? - Once the ThreadTable is created, then multi-threads can be > ???????? executing the for-loop to add their ThreadsList entries to > ???????? the ThreadTable. I guess, there is a confusion here. The lines 97-101 were recently added into the latest webrev: 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { 94 if (!_is_initialized) { 95 { 96 MutexLocker ml(ThreadTableCreate_lock); 97 if (_is_initialized) { 98 // There is no obvious benefits in allowing the thread table 99 // being concurently populated during the initalization. 100 return; 101 } It prevents multi-threads executing the for-loop to add their ThreadsList entries to the ThreadTable. Instead, these threads may not find the requested threads in the ThreadTable and so, will start linear search in their ThreadsList to add them into the ThreadTable. . . . > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... Agreed, ThreadIdTable is better name for this table. Thanks, Serguei > > >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil > From hohensee at amazon.com Fri Sep 20 23:28:46 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 20 Sep 2019 23:28:46 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <7de738c3-a9b8-5d49-870a-ae7392ead85a@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <7de738c3-a9b8-5d49-870a-ae7392ead85a@oracle.com> Message-ID: java.lang.management.ThreadMXBean has two default methods that throw UnsupportedOperationException: ThreadInfo[] getThreadInfo(long[], boolean, boolean, int) ThreadInfo[] dumpAllThreads(boolean, boolean, int) Without actually testing it, is it safe to assume that since com.sun.management.ThreadMXBean extends j.l.m.ThreadMXBean, the new c.s.m.ThreadMXBean method public default long getCurrentThreadAllocatedBytes() { throw new UnsupportedOperationException(); } will behave the same way? Thanks, Paul ?On 9/20/19, 4:15 AM, "Daniel Fuchs" wrote: Hi Paul, It might be worth double checking what happens if you create a MXBean proxy to access the com.sun.management.ThreadMXBean in a remote server: https://download.java.net/java/early_access/jdk14/docs/api/java.management/java/lang/management/ManagementFactory.html#getPlatformMXBean(javax.management.MBeanServerConnection,java.lang.Class) and https://download.java.net/java/early_access/jdk14/docs/api/java.management/javax/management/MBeanServerInvocationHandler.html#newProxyInstance(javax.management.MBeanServerConnection,javax.management.ObjectName,java.lang.Class,boolean) are two different ways of doing this. Will getCurrentThreadAllocatedBytes() return something (however uninteresting that might be) or will the default implementation be triggered on the client side and UnsuportedOperationsException be thrown directly without invoking the server? Whatever the result is - it might be worth mentioning in the CSR. Best regards, -- daniel On 19/09/2019 17:39, Hohensee, Paul wrote: > Off by 2 error. Changed the subject to reflect 8231209. > > http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > Paul From serguei.spitsyn at oracle.com Fri Sep 20 23:50:48 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 16:50:48 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Message-ID: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Hi Daniil, Yes, the Threads_lock is still needed around thread->is_exiting() check and add_thread(). > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet > and then the nested Threads_lock around thread->is_exiting() and > add_thread(java_tid, thread) lines then it will not work since the rank > of Threads_lock is higher than the rank of ThreadTableCreate_lock. The ThreadTableCreate_lock is only used in the lazy_initialize() function. It looks safe to adjust the ThreadTableCreate_lock to fix the above problem. Then the approach below will work as needed. void ThreadTable::lazy_initialize(const ThreadsList *threads) { if (_is_initialized) { return; } MutexLocker ml(ThreadTableCreate_lock); if (_is_initialized) { // There is no obvious benefits in allowing the thread table // being concurrently populated during the initalization. return; } create_table(threads->length()); _is_initialized = true; for (uint i = 0; i < threads->length(); i++) { JavaThread* thread = threads->thread_at(i); oop tobj = thread->threadObj(); if (tobj != NULL) { jlong java_tid = java_lang_Thread::thread_id(tobj); MutexLocker ml(Threads_lock); if (!thread->is_exiting()) { // Must be inside the lock to ensure that we don't add the thread to the table // that has just passed the removal point in ThreadsSMRSupport::remove_thread() add_thread(java_tid, thread); } } } } If you rename ThreadTable to ThreadIdTable then the ThreadTableCreate_lock has to be renamed to ThreadIdTableCreate_lock. Thanks, Serguei On 9/20/19 8:42 AM, Daniil Titov wrote: > Hi Serguei, > >> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> if (_is_initialized) { >> return; > > } >> MutexLocker ml(ThreadTableCreate_lock); > > If I understood you correctly in the code snippet you sent you meant to use > Threads_lock, not ThreadTableCreate_lock, right? > > The original idea was to do a minimal amount of work while holding the lock and > hold the lock for as short period of time as possible to not block other threads when it is not necessary. > > With the suggested approach no new threads could be started until the thread table is created and > populated with all threads running inside a Java application and in case of large app there could be > thousands of them. > > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around > thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock > is higher than the rank of ThreadTableCreate_lock. > > So choosing between blocking new threads from starting and potentially allowing > some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. > > Thanks! > > Best regards, > Daniil > > > > From: "serguei.spitsyn at oracle.com" > Date: Thursday, September 19, 2019 at 10:30 PM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > I think, it is better to grab the thread_lock just once at lazy initialization. > It would look simpler, something, like this would work: > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > // There is no obvious benefits in allowing the thread table > // being concurrently populated during the initalization. > return; > } > create_table(threads->length()); > _is_initialized = true; > > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > if (!thread->is_exiting()) { > // Must be inside the lock to ensure that we don't add the thread to the table > // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > add_thread(java_tid, thread); > } > } > } > } > > Otherwise, concurrent executions of the find_JavaThread_from_java_tid() > will sometimes do a linear search of threads that are not included yet to > the ThreadTable from the ThreadsList (which is used for lazy initialization). > Instead, it is better to wait for the lazy_initialization() to complete. > Thanks, > Serguei > > > On 9/19/19 17:30, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Sat Sep 21 00:07:01 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 17:07:01 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Message-ID: <43f900b8-538f-6270-033b-cf71748e7f23@oracle.com> On 9/20/19 4:50 PM, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Yes, the Threads_lock is still needed around thread->is_exiting() > check and add_thread(). > >> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >> and then the nested Threads_lock around thread->is_exiting() and >> add_thread(java_tid, thread) lines then it will not work since the rank >> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. > > The ThreadTableCreate_lock is only used in the lazy_initialize() > function. > It looks safe to adjust the ThreadTableCreate_lock to fix the above > problem. Sorry, I wanted to say: "to adjust the ThreadTableCreate_lock rank". Thanks, Serguei > Then the approach below will work as needed. > > ?void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??? if (_is_initialized) { > ????? return; > ??? } > ??? MutexLocker ml(ThreadTableCreate_lock); > ??? if (_is_initialized) { > ????? // There is no obvious benefits in allowing the thread table > ????? // being concurrently populated during the initalization. > ????? return; > ??? } > ??? create_table(threads->length()); > ??? _is_initialized = true; > > ??? for (uint i = 0; i < threads->length(); i++) { > ????? JavaThread* thread = threads->thread_at(i); > ????? oop tobj = thread->threadObj(); > ????? if (tobj != NULL) { > ??????? jlong java_tid = java_lang_Thread::thread_id(tobj); > ??????? MutexLocker ml(Threads_lock); > ??????? if (!thread->is_exiting()) { > ????????? // Must be inside the lock to ensure that we don't add the > thread to the table > ????????? // that has just passed the removal point in > ThreadsSMRSupport::remove_thread() > ????????? add_thread(java_tid, thread); > ??????? } > ????? } > ??? } > ? } > > > If you rename ThreadTable to ThreadIdTable then the > ThreadTableCreate_lock > has to be renamed to ThreadIdTableCreate_lock. > > Thanks, > Serguei > > On 9/20/19 8:42 AM, Daniil Titov wrote: >> Hi Serguei, >> >>> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ????? if (_is_initialized) { >>> ????? return; >> ? >?? } >>> ??? MutexLocker ml(ThreadTableCreate_lock); >> >> If I understood you correctly in the code snippet you sent you meant >> to? use >> Threads_lock, not ThreadTableCreate_lock, right? >> >> The original idea was to do a minimal amount of work while holding >> the lock and >> ? hold the lock for as short period of time as possible to not block >> other threads when it is not necessary. >> >> With the suggested approach no new threads could be started until the >> thread table is created and >> populated with all threads running inside a Java application and in >> case of large app there could be >> thousands of them. >> >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your >> snippet and then the nested Threads_lock around >> thread->is_exiting() and add_thread(java_tid, thread) lines then it >> will not work since the rank of Threads_lock >> is higher than the rank of ThreadTableCreate_lock. >> >> So choosing between blocking new threads from starting and >> potentially allowing >> some other monitoring thread? to do a one-time linear scan I think it >> makes sense to choose the latter. >> >> Thanks! >> >> Best regards, >> Daniil >> >> >> >> From: "serguei.spitsyn at oracle.com" >> Date: Thursday, September 19, 2019 at 10:30 PM >> To: Daniil Titov , Robbin Ehn >> , David Holmes , >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> , Claes Redestad >> Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> >> Hi Daniil, >> >> I think, it is better to grab the thread_lock just once at lazy >> initialization. >> It would look simpler, something, like this would work: >> ?? void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } >> >> Otherwise, concurrent executions of the find_JavaThread_from_java_tid() >> will sometimes do a linear search of threads that are not included >> yet to >> the ThreadTable from the ThreadsList (which is used for lazy >> initialization). >> Instead, it is better to wait for the lazy_initialization() to complete. >> Thanks, >> Serguei >> >> >> On 9/19/19 17:30, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes >> Serguei suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil >> >> ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com wrote: >> >> ???? Hi Daniil, >> ???? ???? On 9/17/19 17:13, Daniil Titov wrote: >> ???? > Hi Serguei, >> ???? > >> ???? > Please find below my answers to the concerns you mentioned in >> the previous email. >> ???? > >> ???? > 1. >> ???? >?? > I have a concern about the checks for thread->is_exiting(). >> ???? >?? > - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >> It is interesting what might happen if an exiting thread is >> returned by the >> ???? >> ThreadsList::find_JavaThread_from_java_tid (). >> ???? >> Does it make sense to develop a test that would cover these >> cases? >> ???? > I agree, it doesn't really provide any protection so it makes >> sense just remove it. >> ???? ???? Now, I'm not that confident about it. :) >> ???? ???? >?? The current implementation >> ???? > find_JavaThread_from_java_tid()? doesn't provide such >> protection as well, since the thread could start exiting >> ???? > immediately after method find_JavaThread_from_java_tid() >> returns, so the assumption is that the callers of >> ???? > find_JavaThread_from_java_tid()? are expecting to deal with >> such threads and? looking on some of them shows that >> ???? > they usually try to retrieve threadObj or a thread statistic >> object and if it is NULL that just do nothing. >> ???? ???? If I understand it correctly, the jt->threadObj() can >> remain non-NULL >> ???? for some time while jt->is_exiting() == true. >> ???? It is not clear how reliable is to use it. >> ???? But this is a pre-existing issue. It is not you who introduced >> it. :) >> ???? ???? So, we can skip it for now. >> ???? But for the record, we may have a source of intermittent issues. >> ???? ???? > I'm not sure we could cover this specific case with the >> test. The window between find_JavaThread_from_java_tid() returns and >> the caller >> ???? > continues the execution is too small. The window between the >> thread started exiting and removed itself from the thread table is >> very small as well. >> ???? ???? Understand. >> ???? ???? > 2. >> ???? >>?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? >?? I agree, it was missed, we need to wrap this code inside >> Thread_lock in the similar way as it is done >> find_JavaThread_from_java_tid() >> ???? ???? ???? Okay, thanks! >> ???? ???? > 3. >> ???? >> I would suggest to rewrite this fragment in a safe way: >> ???? >>?? 95???? { >> ???? >>?? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>?? 97?????? if (!_is_initialized) { >> ???? >>?? 98???????? create_table(threads->length()); >> ???? >>?? 99???????? _is_initialized = true; >> ???? >> 100?????? } >> ???? >> 101???? } >> ???? >> as: >> ???? >>??????? { >> ???? >>????????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>????????? if (_is_initialized) { >> ???? >>??????????? return; >> ???? >?? >??????? } >> ???? >?? >??????? create_table(threads->length()); >> ???? >?? >??????? _is_initialized = true; >> ???? >?? >????? } >> ???? > >> ???? > It was an intension to not block? while populating the table >> with the threads from the current thread list. >> ???? > There is no needs to have other threads that call >> find_JavaThread_from_java_tid()? be blocked and waiting for >> ???? >?? it to complete since the requested thread could be not >> present in the thread list that triggers the thread table >> ???? >?? initialization. Plus in case of racing initialization it >> allows threads from not original? thread lists be added to the table >> ???? > and thus avoid the linear scan when these thread are looked up >> for the first time. >> ???? ???? ???? I've replied to David in another email. >> ???? Let's talk once more about it tomorrow. >> ???? ???? ???? > 4. >> ???? >>> The case you have described is exact the reason why we still >> have a code inside >> ???? >>> ThreadsList::find_JavaThread_from_java_tid() method that >> does a linear scan and adds >> ???? >>>??? the requested thread to the thread table if it is not >> there ( lines 614-613 below). >> ???? >> I disagree because it is easy to avoid concurrent ThreadTable >> ???? >> initialization (please, see my separate email). >> ???? >> The reason for this code is to cover a case of late/lazy >> ThreadTable >> ???? >> initialization. >> ???? > David Holmes replied to this in a separate email providing a >> very detailed >> ???? > explanation of the possible cases and how the proposed >> implementation satisfies them. >> ???? ???? Yes. Please, see above. >> ???? ???? Thanks, >> ???? Serguei >> ???? ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > From: mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com >> ???? > Date: Tuesday, September 17, 2019 at 1:53 AM >> ???? > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn >> mailto:robbin.ehn at oracle.com, David Holmes >> mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, >> OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, >> Claes Redestad mailto:claes.redestad at oracle.com >> ???? > Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> ???? > >> ???? > Hi Daniil, >> ???? > >> ???? > Thank you for you patience in working on this issue! >> ???? > Also, I like that the current thread related optimizations in >> management.cpp were factored out. >> ???? > It was a good idea to separate them. >> ???? > >> ???? > I have a concern about the checks for thread->is_exiting(). >> ???? > The threads are added to and removed from the ThreadTable >> under protection of Threads_lock. >> ???? > However, the thread->is_exiting() checks are not protected, >> and so, they are racy. >> ???? > >> ???? > There is a couple of such checks to mention: >> ???? >?? 611 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >?? 612?? ThreadTable::lazy_initialize(this); >> ???? >?? 613?? JavaThread* thread = >> ThreadTable::find_thread_by_tid(java_tid); >> ???? >?? 614?? if (thread == NULL) { >> ???? >?? 615???? // If the thread is not found in the table find it >> ???? >?? 616???? // with a linear search and add to the table. >> ???? >?? 617???? for (uint i = 0; i < length(); i++) { >> ???? >?? 618?????? thread = thread_at(i); >> ???? >?? 619?????? oop tobj = thread->threadObj(); >> ???? >?? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ???? >?? 621?????? // or is starting to exit. >> ???? >?? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? >?? 623???????? MutexLocker ml(Threads_lock); >> ???? >?? 624???????? // Must be inside the lock to ensure that we >> don't add the thread to the table >> ???? >?? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >?? 626???????? if (!thread->is_exiting()) { >> ???? >?? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? >?? 628?????????? return thread; >> ???? >?? 629???????? } >> ???? >?? 630?????? } >> ???? >?? 631???? } >> ???? >?? 632?? } else if (!thread->is_exiting()) { >> ???? >?? 633?????? return thread; >> ???? >?? 634?? } >> ???? >?? 635?? return NULL; >> ???? >?? 636 } >> ???? >??? ... >> ???? >??? 93 void ThreadTable::lazy_initialize(const ThreadsList >> *threads) { >> ???? >??? 94?? if (!_is_initialized) { >> ???? >??? 95???? { >> ???? >??? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >??? 97?????? if (!_is_initialized) { >> ???? >??? 98???????? create_table(threads->length()); >> ???? >??? 99???????? _is_initialized = true; >> ???? >?? 100?????? } >> ???? >?? 101???? } >> ???? >?? 102???? for (uint i = 0; i < threads->length(); i++) { >> ???? >?? 103?????? JavaThread* thread = threads->thread_at(i); >> ???? >?? 104?????? oop tobj = thread->threadObj(); >> ???? >?? 105?????? if (tobj != NULL && !thread->is_exiting()) { >> ???? >?? 106???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???? >?? 107???????? add_thread(java_tid, thread); >> ???? >?? 108?????? } >> ???? >?? 109???? } >> ???? >?? 110?? } >> ???? >?? 111 } >> ???? > >> ???? > A thread may start exiting right after the checks at the lines >> 626 and 105. >> ???? > So that: >> ???? >?? - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? > >> ???? > Please, note, the lines 626-629 are safe in terms of addition >> to the ThreadTable as they >> ???? > are protected with the Threads_lock. But the returned thread >> still can exit after that. >> ???? > It is interesting what might happen if an exiting thread is >> returned by the >> ???? > ThreadsList::find_JavaThread_from_java_tid (). >> ???? > >> ???? > Does it make sense to develop a test that would cover these >> cases? >> ???? > >> ???? > Thanks, >> ???? > Serguei >> ???? > >> ???? > >> ???? > On 9/16/19 11:18, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on >> the performance (thanks a lot Claes for helping with it!) the >> conclusion was that the impact on the thread startup time is not a >> blocker for this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just >> to remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 >> [3] minus changes in src/hotspot/share/services/management.cpp (that >> were factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code >> that adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that >> may be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue >> if required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >> case when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >> the thread table >> ???? >????????? >????? >????? >?? is initialized with the threads >> from the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only >> initializing the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, >> but if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then >> that may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in >> that regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori >> or deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably >> originally suggested in the patch attached to the JBS issue, should >> be used.? It will make >> ???? >????????? >????? >????? > the backporting more complicated,? >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && >> ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >> (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not >> indent the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its >> very existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> size_log > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? > DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong _tid; >> ???? >????????? >????? >????? 181???? uintx _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces >> before the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211 assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" mailto:daniel.daugherty at oracle.com wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with >> an unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about >> the maintaining the thread table when it may never even be queried, >> one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it >> on and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as >> David H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >> save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and >> if the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >> get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> mailto:serguei.spitsyn at oracle.com >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability >> mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot >> c/c++ files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615 JavaThread* java_thread >> = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616 if (java_thread == NULL >> && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? > 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? > 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? > 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? > 620???????? for (uint i = 0; i >> < length(); i++) { >> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? > 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > 623???????????????? >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? > 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? > 625???????????? } >> ???? >????????? >????? >????? >????? > 626???????? } >> ???? >????????? >????? >????? >????? >?? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629 } >> ???? >????????? >????? >????? >????? >?? 630 return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633 oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634 // Ignore the thread if >> it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635 // or is starting to exit. >> ???? >????????? >????? >????? >????? >?? 636 return (tobj != NULL && >> !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615 JavaThread* java_thread >> = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the >> comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name >> 'is_valid_java_thread' looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? > implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Unfortunately I'm just about >> to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the >> mapping between the thread ids and JavaThread objects and replaces >> the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> >> >> >> >> > From daniel.daugherty at oracle.com Sat Sep 21 01:07:29 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 21:07:29 -0400 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> Message-ID: <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: > Hi Dan, > > Please, find a minor correction below. > > > On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >> Daniil, >> >> Thanks for sticking with this project through the many versions. >> Sorry this review is late... >> >> >> On 9/19/19 8:30 PM, Daniil Titov wrote: >>> Hi David and Serguei, >>> >>> Please review new version of the fix that includes the changes >>> Serguei suggested: >>> ? 1. If racing threads initialize the thread table only one of these >>> threads will populate the table with the threads from the thread list >>> ? 2. The code that adds the thread to the tread table is put inside >>> Threads_lock to ensure that we cannot accidentally add the thread >>> ????? that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> >>> The changes are in ThreadTable::lazy_initialize() method only. >>> >>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>> successfully passed. >>> >>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > . . . >> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ??????? Re: discussion about lazy_initialize() racing with >> ??????????? ThreadsList::find_JavaThread_from_java_tid() >> >> ??????? There's a couple of aspects to these two pieces of code racing >> ??????? with each other and racing with new thread creation. Racing with >> ??????? new thread creation is the easy one: >> >> ????????? If a new thread isn't added to the ThreadTable by >> ????????? ThreadsSMRSupport::add_thread() calling >> ThreadTable::add_thread(), >> ????????? then the point in the future where someone calls >> ????????? find_JavaThread_from_java_tid() will add it to the table >> due to >> ????????? the linear search when ThreadTable::find_thread_by_tid() >> ????????? returns NULL. >> >> ?????? As for multi-threads calling >> ThreadsList::find_JavaThread_from_java_tid() >> ?????? at the same time which results in multi-threads in >> lazy_initialize() >> ?????? at the same time... >> >> ?????? - ThreadTable creation will be linear due to >> ThreadTableCreate_lock. >> ???????? After _is_initialized is set to true, then no more callers to >> ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. >> ?????? - Once the ThreadTable is created, then multi-threads can be >> ???????? executing the for-loop to add their ThreadsList entries to >> ???????? the ThreadTable. > > I guess, there is a confusion here. > The lines 97-101 were recently added into the latest webrev: > > ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ? 94?? if (!_is_initialized) { > ? 95???? { > ? 96?????? MutexLocker ml(ThreadTableCreate_lock); > ? 97?????? if (_is_initialized) { > ? 98???????? // There is no obvious benefits in allowing the thread table > ? 99???????? // being concurently populated during the initalization. > ?100???????? return; > ?101?????? } > > It prevents multi-threads executing the for-loop to add their > ThreadsList entries to the ThreadTable. > Instead, these threads may not find the requested threads in the > ThreadTable and so, will start linear search in their ThreadsList > to add them into the ThreadTable. The ThreadTableCreate_lock is only held from here: ? L95: ??? { ? L96: ????? MutexLocker ml(ThreadTableCreate_lock); to here: ? L103: ????? _is_initialized = true; ? L104: ??? } and the for-loop starts here: ? L105: ??? for (uint i = 0; i < threads->length(); i++) { so the ThreadTable_Create_lock does not linearize the for-loop. I'm quoting webrev.07... which I believe is the latest. Dan > > . . . >> Short version: Thumbs up. >> >> Longer version: I don't think I've spotted anything other than nits >> here. >> Mostly I've just looked for multi-threaded races, proper usage of the >> Thread-SMR stuff, and minimal impact in the case where the new >> ThreadsTable is never needed. >> >> Dan >> >> P.S. >> ThreadTable is a bit of misnomer. What you really have here is >> a ThreadIdTable, but I'm really late to the code review flow >> with that comment... > > Agreed, ThreadIdTable is better name for this table. > > Thanks, > Serguei > > >> >> >>> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> >>> Thank you! >>> --Daniil >> > From daniel.daugherty at oracle.com Sat Sep 21 01:11:27 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 21:11:27 -0400 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: > On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >> Hi Dan, >> >> Please, find a minor correction below. >> >> >> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>> Daniil, >>> >>> Thanks for sticking with this project through the many versions. >>> Sorry this review is late... >>> >>> >>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>> Hi David and Serguei, >>>> >>>> Please review new version of the fix that includes the changes >>>> Serguei suggested: >>>> ? 1. If racing threads initialize the thread table only one of >>>> these threads will populate the table with the threads from the >>>> thread list >>>> ? 2. The code that adds the thread to the tread table is put inside >>>> Threads_lock to ensure that we cannot accidentally add the thread >>>> ????? that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> >>>> The changes are in ThreadTable::lazy_initialize() method only. >>>> >>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>> successfully passed. >>>> >>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> . . . >>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ??????? Re: discussion about lazy_initialize() racing with >>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>> >>> ??????? There's a couple of aspects to these two pieces of code racing >>> ??????? with each other and racing with new thread creation. Racing >>> with >>> ??????? new thread creation is the easy one: >>> >>> ????????? If a new thread isn't added to the ThreadTable by >>> ????????? ThreadsSMRSupport::add_thread() calling >>> ThreadTable::add_thread(), >>> ????????? then the point in the future where someone calls >>> ????????? find_JavaThread_from_java_tid() will add it to the table >>> due to >>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>> ????????? returns NULL. >>> >>> ?????? As for multi-threads calling >>> ThreadsList::find_JavaThread_from_java_tid() >>> ?????? at the same time which results in multi-threads in >>> lazy_initialize() >>> ?????? at the same time... >>> >>> ?????? - ThreadTable creation will be linear due to >>> ThreadTableCreate_lock. >>> ???????? After _is_initialized is set to true, then no more callers to >>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>> block. >>> ?????? - Once the ThreadTable is created, then multi-threads can be >>> ???????? executing the for-loop to add their ThreadsList entries to >>> ???????? the ThreadTable. >> >> I guess, there is a confusion here. >> The lines 97-101 were recently added into the latest webrev: >> >> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ? 94?? if (!_is_initialized) { >> ? 95???? { >> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ? 97?????? if (_is_initialized) { >> ? 98???????? // There is no obvious benefits in allowing the thread >> table >> ? 99???????? // being concurently populated during the initalization. >> ?100???????? return; >> ?101?????? } >> >> It prevents multi-threads executing the for-loop to add their >> ThreadsList entries to the ThreadTable. >> Instead, these threads may not find the requested threads in the >> ThreadTable and so, will start linear search in their ThreadsList >> to add them into the ThreadTable. I should have read your comment more carefully... Yes, you are right about the early cut out on L97-100 causing the for-loop to be skipped by any thread that saw (!_is_initialized) and was blocked on the ThreadTableCreate_lock. So my response below is wrong... Sorry about the noise. Dan > > The ThreadTableCreate_lock is only held from here: > > ? L95: ??? { > ? L96: ????? MutexLocker ml(ThreadTableCreate_lock); > > to here: > > ? L103: ????? _is_initialized = true; > ? L104: ??? } > > and the for-loop starts here: > > ? L105: ??? for (uint i = 0; i < threads->length(); i++) { > > so the ThreadTable_Create_lock does not linearize the > for-loop. > > I'm quoting webrev.07... which I believe is the latest. > > Dan > > >> >> . . . >>> Short version: Thumbs up. >>> >>> Longer version: I don't think I've spotted anything other than nits >>> here. >>> Mostly I've just looked for multi-threaded races, proper usage of the >>> Thread-SMR stuff, and minimal impact in the case where the new >>> ThreadsTable is never needed. >>> >>> Dan >>> >>> P.S. >>> ThreadTable is a bit of misnomer. What you really have here is >>> a ThreadIdTable, but I'm really late to the code review flow >>> with that comment... >> >> Agreed, ThreadIdTable is better name for this table. >> >> Thanks, >> Serguei >> >> >>> >>> >>>> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> >>>> Thank you! >>>> --Daniil >>> >> > From david.holmes at oracle.com Sat Sep 21 02:20:18 2019 From: david.holmes at oracle.com (David Holmes) Date: Sat, 21 Sep 2019 12:20:18 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Message-ID: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Hi Serguei, On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Yes, the Threads_lock is still needed around thread->is_exiting() check > and add_thread(). > >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your snippet >> and then the nested Threads_lock around thread->is_exiting() and >> add_thread(java_tid, thread) lines then it will not work since the rank >> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. > > The ThreadTableCreate_lock is only used in the lazy_initialize() function. > It looks safe to adjust the ThreadTableCreate_lock to fix the above > problem. > Then the approach below will work as needed. > > ?void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??? if (_is_initialized) { > ????? return; > ??? } > ??? MutexLocker ml(ThreadTableCreate_lock); > ??? if (_is_initialized) { > ????? // There is no obvious benefits in allowing the thread table > ????? // being concurrently populated during the initalization. > ????? return; > ??? } > ??? create_table(threads->length()); > ??? _is_initialized = true; > > ??? for (uint i = 0; i < threads->length(); i++) { > ????? JavaThread* thread = threads->thread_at(i); > ????? oop tobj = thread->threadObj(); > ????? if (tobj != NULL) { > ??????? jlong java_tid = java_lang_Thread::thread_id(tobj); > ??????? MutexLocker ml(Threads_lock); > ??????? if (!thread->is_exiting()) { > ????????? // Must be inside the lock to ensure that we don't add the > thread to the table > ????????? // that has just passed the removal point in > ThreadsSMRSupport::remove_thread() > ????????? add_thread(java_tid, thread); > ??????? } > ????? } > ??? } > ? } I don't like holding a completely unrelated lock, whilst holding the Threads_lock, particularly as it seems unnecessary for correctness. We have to be certain that no M&M operation that holds the Threads_lock can try to initialize the thread table. Testing can only prove the presence of that problem, not the absence. I think Daniil's webrev.07 version is correct and less risky than what you propose. Sorry. David ----- > > If you rename ThreadTable to ThreadIdTable then the ThreadTableCreate_lock > has to be renamed to ThreadIdTableCreate_lock. > > Thanks, > Serguei > > On 9/20/19 8:42 AM, Daniil Titov wrote: >> Hi Serguei, >> >>> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ????? if (_is_initialized) { >>> ????? return; >> ? >?? } >>> ??? MutexLocker ml(ThreadTableCreate_lock); >> >> If I understood you correctly in the code snippet you sent you meant >> to? use >> Threads_lock, not ThreadTableCreate_lock, right? >> >> The original idea was to do a minimal amount of work while holding the >> lock and >> ? hold the lock for as short period of time as possible to not block >> other threads when it is not necessary. >> >> With the suggested approach no new threads could be started until the >> thread table is created and >> populated with all threads running inside a Java application and in >> case of large app there could be >> thousands of them. >> >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your >> snippet and then the nested Threads_lock around >> thread->is_exiting() and add_thread(java_tid, thread) lines then it >> will not work since the rank of Threads_lock >> is higher than the rank of ThreadTableCreate_lock. >> >> So choosing between blocking new threads from starting and potentially >> allowing >> some other monitoring thread? to do a one-time linear scan I think it >> makes sense to choose the latter. >> >> Thanks! >> >> Best regards, >> Daniil >> >> >> >> From: "serguei.spitsyn at oracle.com" >> Date: Thursday, September 19, 2019 at 10:30 PM >> To: Daniil Titov , Robbin Ehn >> , David Holmes , >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> , Claes Redestad >> Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> >> Hi Daniil, >> >> I think, it is better to grab the thread_lock just once at lazy >> initialization. >> It would look simpler, something, like this would work: >> ?? void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } >> >> Otherwise, concurrent executions of the find_JavaThread_from_java_tid() >> will sometimes do a linear search of threads that are not included yet to >> the ThreadTable from the ThreadsList (which is used for lazy >> initialization). >> Instead, it is better to wait for the lazy_initialization() to complete. >> Thanks, >> Serguei >> >> >> On 9/19/19 17:30, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes Serguei >> suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil >> >> ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com wrote: >> >> ???? Hi Daniil, >> ???? On 9/17/19 17:13, Daniil Titov wrote: >> ???? > Hi Serguei, >> ???? > >> ???? > Please find below my answers to the concerns you mentioned in >> the previous email. >> ???? > >> ???? > 1. >> ???? >?? > I have a concern about the checks for thread->is_exiting(). >> ???? >?? > - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >> It is interesting what might happen if an exiting thread is >> returned by the >> ???? >> ThreadsList::find_JavaThread_from_java_tid (). >> ???? >> Does it make sense to develop a test that would cover these >> cases? >> ???? > I agree, it doesn't really provide any protection so it makes >> sense just remove it. >> ???? Now, I'm not that confident about it. :) >> ???? >?? The current implementation >> ???? > find_JavaThread_from_java_tid()? doesn't provide such >> protection as well, since the thread could start exiting >> ???? > immediately after method find_JavaThread_from_java_tid() >> returns, so the assumption is that the callers of >> ???? > find_JavaThread_from_java_tid()? are expecting to deal with >> such threads and? looking on some of them shows that >> ???? > they usually try to retrieve threadObj or a thread statistic >> object and if it is NULL that just do nothing. >> ???? If I understand it correctly, the jt->threadObj() can remain >> non-NULL >> ???? for some time while jt->is_exiting() == true. >> ???? It is not clear how reliable is to use it. >> ???? But this is a pre-existing issue. It is not you who introduced >> it. :) >> ???? So, we can skip it for now. >> ???? But for the record, we may have a source of intermittent issues. >> ???? > I'm not sure we could cover this specific case with the test. >> The window between find_JavaThread_from_java_tid() returns and the caller >> ???? > continues the execution is too small. The window between the >> thread started exiting and removed itself from the thread table is >> very small as well. >> ???? Understand. >> ???? > 2. >> ???? >>?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? >?? I agree, it was missed, we need to wrap this code inside >> Thread_lock in the similar way as it is done >> find_JavaThread_from_java_tid() >> ???? Okay, thanks! >> ???? > 3. >> ???? >> I would suggest to rewrite this fragment in a safe way: >> ???? >>?? 95???? { >> ???? >>?? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>?? 97?????? if (!_is_initialized) { >> ???? >>?? 98???????? create_table(threads->length()); >> ???? >>?? 99???????? _is_initialized = true; >> ???? >> 100?????? } >> ???? >> 101???? } >> ???? >> as: >> ???? >>??????? { >> ???? >>????????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>????????? if (_is_initialized) { >> ???? >>??????????? return; >> ???? >?? >??????? } >> ???? >?? >??????? create_table(threads->length()); >> ???? >?? >??????? _is_initialized = true; >> ???? >?? >????? } >> ???? > >> ???? > It was an intension to not block? while populating the table >> with the threads from the current thread list. >> ???? > There is no needs to have other threads that call >> find_JavaThread_from_java_tid()? be blocked and waiting for >> ???? >?? it to complete since the requested thread could be not >> present in the thread list that triggers the thread table >> ???? >?? initialization. Plus in case of racing initialization it >> allows threads from not original? thread lists be added to the table >> ???? > and thus avoid the linear scan when these thread are looked up >> for the first time. >> ???? I've replied to David in another email. >> ???? Let's talk once more about it tomorrow. >> ???? > 4. >> ???? >>> The case you have described is exact the reason why we still >> have a code inside >> ???? >>> ThreadsList::find_JavaThread_from_java_tid() method that does >> a linear scan and adds >> ???? >>>??? the requested thread to the thread table if it is not >> there ( lines 614-613 below). >> ???? >> I disagree because it is easy to avoid concurrent ThreadTable >> ???? >> initialization (please, see my separate email). >> ???? >> The reason for this code is to cover a case of late/lazy >> ThreadTable >> ???? >> initialization. >> ???? > David Holmes replied to this in a separate email providing a >> very detailed >> ???? > explanation of the possible cases and how the proposed >> implementation satisfies them. >> ???? Yes. Please, see above. >> ???? Thanks, >> ???? Serguei >> ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > From: mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com >> ???? > Date: Tuesday, September 17, 2019 at 1:53 AM >> ???? > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn >> mailto:robbin.ehn at oracle.com, David Holmes >> mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, >> OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes >> Redestad mailto:claes.redestad at oracle.com >> ???? > Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> ???? > >> ???? > Hi Daniil, >> ???? > >> ???? > Thank you for you patience in working on this issue! >> ???? > Also, I like that the current thread related optimizations in >> management.cpp were factored out. >> ???? > It was a good idea to separate them. >> ???? > >> ???? > I have a concern about the checks for thread->is_exiting(). >> ???? > The threads are added to and removed from the ThreadTable under >> protection of Threads_lock. >> ???? > However, the thread->is_exiting() checks are not protected, and >> so, they are racy. >> ???? > >> ???? > There is a couple of such checks to mention: >> ???? >?? 611 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >?? 612?? ThreadTable::lazy_initialize(this); >> ???? >?? 613?? JavaThread* thread = >> ThreadTable::find_thread_by_tid(java_tid); >> ???? >?? 614?? if (thread == NULL) { >> ???? >?? 615???? // If the thread is not found in the table find it >> ???? >?? 616???? // with a linear search and add to the table. >> ???? >?? 617???? for (uint i = 0; i < length(); i++) { >> ???? >?? 618?????? thread = thread_at(i); >> ???? >?? 619?????? oop tobj = thread->threadObj(); >> ???? >?? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ???? >?? 621?????? // or is starting to exit. >> ???? >?? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? >?? 623???????? MutexLocker ml(Threads_lock); >> ???? >?? 624???????? // Must be inside the lock to ensure that we >> don't add the thread to the table >> ???? >?? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >?? 626???????? if (!thread->is_exiting()) { >> ???? >?? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? >?? 628?????????? return thread; >> ???? >?? 629???????? } >> ???? >?? 630?????? } >> ???? >?? 631???? } >> ???? >?? 632?? } else if (!thread->is_exiting()) { >> ???? >?? 633?????? return thread; >> ???? >?? 634?? } >> ???? >?? 635?? return NULL; >> ???? >?? 636 } >> ???? >??? ... >> ???? >??? 93 void ThreadTable::lazy_initialize(const ThreadsList >> *threads) { >> ???? >??? 94?? if (!_is_initialized) { >> ???? >??? 95???? { >> ???? >??? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >??? 97?????? if (!_is_initialized) { >> ???? >??? 98???????? create_table(threads->length()); >> ???? >??? 99???????? _is_initialized = true; >> ???? >?? 100?????? } >> ???? >?? 101???? } >> ???? >?? 102???? for (uint i = 0; i < threads->length(); i++) { >> ???? >?? 103?????? JavaThread* thread = threads->thread_at(i); >> ???? >?? 104?????? oop tobj = thread->threadObj(); >> ???? >?? 105?????? if (tobj != NULL && !thread->is_exiting()) { >> ???? >?? 106???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???? >?? 107???????? add_thread(java_tid, thread); >> ???? >?? 108?????? } >> ???? >?? 109???? } >> ???? >?? 110?? } >> ???? >?? 111 } >> ???? > >> ???? > A thread may start exiting right after the checks at the lines >> 626 and 105. >> ???? > So that: >> ???? >?? - the lines 632-633 are useless as they do not really protect >> from returning an exiting thread >> ???? >?? - the lines 105-108 can result in adding exiting threads into >> the ThreadTable >> ???? > >> ???? > Please, note, the lines 626-629 are safe in terms of addition >> to the ThreadTable as they >> ???? > are protected with the Threads_lock. But the returned thread >> still can exit after that. >> ???? > It is interesting what might happen if an exiting thread is >> returned by the >> ???? > ThreadsList::find_JavaThread_from_java_tid (). >> ???? > >> ???? > Does it make sense to develop a test that would cover these cases? >> ???? > >> ???? > Thanks, >> ???? > Serguei >> ???? > >> ???? > >> ???? > On 9/16/19 11:18, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on the >> performance (thanks a lot Claes for helping with it!) the conclusion >> was that the impact on the thread startup time is not a blocker for >> this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just to >> remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] >> minus changes in src/hotspot/share/services/management.cpp (that were >> factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code that >> adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev:? https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that may >> be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue if >> required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c]? jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333]? JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159]? ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me here. >> I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In case >> when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time the >> thread table >> ???? >????????? >????? >????? >?? is initialized with the threads from >> the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only initializing >> the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, but >> if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then that >> may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in that >> regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori or >> deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably originally >> suggested in the patch attached to the JBS issue, should be used.? It >> will make >> ???? >????????? >????? >????? > the backporting more complicated, >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? +?? def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324?????????? JavaThread* >> current_thread = (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not indent >> the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60???? ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = size_log >> > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? >????? DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong????????? _tid; >> ???? >????????? >????? >????? 181???? uintx???????? _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces before >> the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183???? ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201???? ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211??? assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" mailto:daniel.daugherty at oracle.com wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil Titov >> wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with an >> unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about the >> maintaining the thread table when it may never even be queried, one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it on >> and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as David >> H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and if >> the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> mailto:serguei.spitsyn at oracle.com >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability >> mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot c/c++ >> files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616???? if (java_thread == >> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? >?? 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? >?? 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? >?? 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? >?? 620???????? for (uint i = 0; >> i < length(); i++) { >> ???? >????????? >????? >????? >????? >?? 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? >?? 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? >?? 623 >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? >?? 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? >?? 625???????????? } >> ???? >????????? >????? >????? >????? >?? 626???????? } >> ???? >????????? >????? >????? >????? >?? 627???? } else if >> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? >?? 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629???? } >> ???? >????????? >????? >????? >????? >?? 630???? return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633???? oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634???? // Ignore the thread >> if it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635???? // or is starting to >> exit. >> ???? >????????? >????? >????? >????? >?? 636???? return (tobj != NULL >> && !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? >?? 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name 'is_valid_java_thread' >> looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? >????? implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Unfortunately I'm just >> about to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the mapping >> between the thread ids and JavaThread objects and replaces the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> >> >> >> >> > From serguei.spitsyn at oracle.com Sat Sep 21 06:11:34 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:11:34 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: Hi Dan, On 9/20/19 18:11, Daniel D. Daugherty wrote: > On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: >> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Dan, >>> >>> Please, find a minor correction below. >>> >>> >>> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>>> Daniil, >>>> >>>> Thanks for sticking with this project through the many versions. >>>> Sorry this review is late... >>>> >>>> >>>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>>> Hi David and Serguei, >>>>> >>>>> Please review new version of the fix that includes the changes >>>>> Serguei suggested: >>>>> ? 1. If racing threads initialize the thread table only one of >>>>> these threads will populate the table with the threads from the >>>>> thread list >>>>> ? 2. The code that adds the thread to the tread table is put >>>>> inside Threads_lock to ensure that we cannot accidentally add the >>>>> thread >>>>> ????? that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> >>>>> The changes are in ThreadTable::lazy_initialize() method only. >>>>> >>>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>>> successfully passed. >>>>> >>>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> . . . >>>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ??????? Re: discussion about lazy_initialize() racing with >>>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>>> >>>> ??????? There's a couple of aspects to these two pieces of code racing >>>> ??????? with each other and racing with new thread creation. Racing >>>> with >>>> ??????? new thread creation is the easy one: >>>> >>>> ????????? If a new thread isn't added to the ThreadTable by >>>> ????????? ThreadsSMRSupport::add_thread() calling >>>> ThreadTable::add_thread(), >>>> ????????? then the point in the future where someone calls >>>> ????????? find_JavaThread_from_java_tid() will add it to the table >>>> due to >>>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>>> ????????? returns NULL. >>>> >>>> ?????? As for multi-threads calling >>>> ThreadsList::find_JavaThread_from_java_tid() >>>> ?????? at the same time which results in multi-threads in >>>> lazy_initialize() >>>> ?????? at the same time... >>>> >>>> ?????? - ThreadTable creation will be linear due to >>>> ThreadTableCreate_lock. >>>> ???????? After _is_initialized is set to true, then no more callers to >>>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>>> block. >>>> ?????? - Once the ThreadTable is created, then multi-threads can be >>>> ???????? executing the for-loop to add their ThreadsList entries to >>>> ???????? the ThreadTable. >>> >>> I guess, there is a confusion here. >>> The lines 97-101 were recently added into the latest webrev: >>> >>> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ? 94?? if (!_is_initialized) { >>> ? 95???? { >>> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >>> ? 97?????? if (_is_initialized) { >>> ? 98???????? // There is no obvious benefits in allowing the thread >>> table >>> ? 99???????? // being concurently populated during the initalization. >>> ?100???????? return; >>> ?101?????? } >>> >>> It prevents multi-threads executing the for-loop to add their >>> ThreadsList entries to the ThreadTable. >>> Instead, these threads may not find the requested threads in the >>> ThreadTable and so, will start linear search in their ThreadsList >>> to add them into the ThreadTable. > > I should have read your comment more carefully... > > Yes, you are right about the early cut out on L97-100 causing > the for-loop to be skipped by any thread that saw (!_is_initialized) > and was blocked on the ThreadTableCreate_lock. > > So my response below is wrong... > > Sorry about the noise. It is hard to follow all moves on this email thread. :) Thanks, Serguei From serguei.spitsyn at oracle.com Sat Sep 21 06:21:16 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:21:16 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: Hi David, I do not want to insist in the area (Threads_lock use) where you have much more expertise. Also, I agreed that Daniil's webrev.07 version is correct and less risky. The only my concern was about possible performance impact of linear searches from the ThreadsList::find_JavaThread_from_java_tid() calls that can be executed concurrently with the ThreadTable::lazy_initialize() if the ThreadsList is big. But it should not be that big issue. So, I'm okay with the webrev.07 modulo ThreadTable renaming. Thanks, Serguei On 9/20/19 19:20, David Holmes wrote: > Hi Serguei, > > On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Yes, the Threads_lock is still needed around thread->is_exiting() >> check and add_thread(). >> >>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>> and then the nested Threads_lock around thread->is_exiting() and >>> add_thread(java_tid, thread) lines then it will not work since the rank >>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >> >> The ThreadTableCreate_lock is only used in the lazy_initialize() >> function. >> It looks safe to adjust the ThreadTableCreate_lock to fix the above >> problem. >> Then the approach below will work as needed. >> >> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? MutexLocker ml(Threads_lock); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } > > I don't like holding a completely unrelated lock, whilst holding the > Threads_lock, particularly as it seems unnecessary for correctness. We > have to be certain that no M&M operation that holds the Threads_lock > can try to initialize the thread table. Testing can only prove the > presence of that problem, not the absence. > > I think Daniil's webrev.07 version is correct and less risky than what > you propose. Sorry. > > David > ----- From serguei.spitsyn at oracle.com Sat Sep 21 06:24:42 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:24:42 -0700 (PDT) Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: <7e07188c-4190-cc59-57b6-e297f4197f80@oracle.com> Hi Dan, On 9/20/19 18:11, Daniel D. Daugherty wrote: > On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: >> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Dan, >>> >>> Please, find a minor correction below. >>> >>> >>> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>>> Daniil, >>>> >>>> Thanks for sticking with this project through the many versions. >>>> Sorry this review is late... >>>> >>>> >>>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>>> Hi David and Serguei, >>>>> >>>>> Please review new version of the fix that includes the changes >>>>> Serguei suggested: >>>>> ? 1. If racing threads initialize the thread table only one of >>>>> these threads will populate the table with the threads from the >>>>> thread list >>>>> ? 2. The code that adds the thread to the tread table is put >>>>> inside Threads_lock to ensure that we cannot accidentally add the >>>>> thread >>>>> ????? that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> >>>>> The changes are in ThreadTable::lazy_initialize() method only. >>>>> >>>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>>> successfully passed. >>>>> >>>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> . . . >>>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ??????? Re: discussion about lazy_initialize() racing with >>>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>>> >>>> ??????? There's a couple of aspects to these two pieces of code racing >>>> ??????? with each other and racing with new thread creation. Racing >>>> with >>>> ??????? new thread creation is the easy one: >>>> >>>> ????????? If a new thread isn't added to the ThreadTable by >>>> ????????? ThreadsSMRSupport::add_thread() calling >>>> ThreadTable::add_thread(), >>>> ????????? then the point in the future where someone calls >>>> ????????? find_JavaThread_from_java_tid() will add it to the table >>>> due to >>>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>>> ????????? returns NULL. >>>> >>>> ?????? As for multi-threads calling >>>> ThreadsList::find_JavaThread_from_java_tid() >>>> ?????? at the same time which results in multi-threads in >>>> lazy_initialize() >>>> ?????? at the same time... >>>> >>>> ?????? - ThreadTable creation will be linear due to >>>> ThreadTableCreate_lock. >>>> ???????? After _is_initialized is set to true, then no more callers to >>>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>>> block. >>>> ?????? - Once the ThreadTable is created, then multi-threads can be >>>> ???????? executing the for-loop to add their ThreadsList entries to >>>> ???????? the ThreadTable. >>> >>> I guess, there is a confusion here. >>> The lines 97-101 were recently added into the latest webrev: >>> >>> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ? 94?? if (!_is_initialized) { >>> ? 95???? { >>> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >>> ? 97?????? if (_is_initialized) { >>> ? 98???????? // There is no obvious benefits in allowing the thread >>> table >>> ? 99???????? // being concurently populated during the initalization. >>> ?100???????? return; >>> ?101?????? } >>> >>> It prevents multi-threads executing the for-loop to add their >>> ThreadsList entries to the ThreadTable. >>> Instead, these threads may not find the requested threads in the >>> ThreadTable and so, will start linear search in their ThreadsList >>> to add them into the ThreadTable. > > I should have read your comment more carefully... > > Yes, you are right about the early cut out on L97-100 causing > the for-loop to be skipped by any thread that saw (!_is_initialized) > and was blocked on the ThreadTableCreate_lock. > > So my response below is wrong... > > Sorry about the noise. It is hard to follow all moves on this email thread. :) Thanks, Serguei > > Dan From serguei.spitsyn at oracle.com Sat Sep 21 06:26:39 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:26:39 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: Hi David, I do not want to insist in the area (Threads_lock use) where you have much more expertise. Also, I agreed that Daniil's webrev.07 version is correct and less risky. The only my concern was about possible performance impact of linear searches from the ThreadsList::find_JavaThread_from_java_tid() calls that can be executed concurrently with the ThreadTable::lazy_initialize() when the ThreadsList is big. But it should not be that big issue. So, I'm okay with the webrev.07 modulo ThreadTable renaming. Thanks, Serguei On 9/20/19 19:20, David Holmes wrote: > Hi Serguei, > > On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Yes, the Threads_lock is still needed around thread->is_exiting() >> check and add_thread(). >> >>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>> and then the nested Threads_lock around thread->is_exiting() and >>> add_thread(java_tid, thread) lines then it will not work since the rank >>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >> >> The ThreadTableCreate_lock is only used in the lazy_initialize() >> function. >> It looks safe to adjust the ThreadTableCreate_lock to fix the above >> problem. >> Then the approach below will work as needed. >> >> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? MutexLocker ml(Threads_lock); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } > > I don't like holding a completely unrelated lock, whilst holding the > Threads_lock, particularly as it seems unnecessary for correctness. We > have to be certain that no M&M operation that holds the Threads_lock > can try to initialize the thread table. Testing can only prove the > presence of that problem, not the absence. > > I think Daniil's webrev.07 version is correct and less risky than what > you propose. Sorry. > > David > ----- From david.holmes at oracle.com Sat Sep 21 06:53:09 2019 From: david.holmes at oracle.com (David Holmes) Date: Sat, 21 Sep 2019 16:53:09 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> On 21/09/2019 4:21 pm, serguei.spitsyn at oracle.com wrote: > Hi David, > > I do not want to insist in the area (Threads_lock use) where you have > much more expertise. It's more conservatism than expertise :) I had a look through and there are not too many usages of the Threads_lock in the M&M code and they do appear safe. But I still prefer on principle to not hold nested locks if it is not necessary. > Also, I agreed that Daniil's webrev.07 version is correct and less risky. > The only my concern was about possible performance impact of linear > searches > from the ThreadsList::find_JavaThread_from_java_tid() calls that can be > executed > concurrently with the ThreadTable::lazy_initialize() if the ThreadsList > is big. > But it should not be that big issue. I think we can waste a lot of time wondering about what kinds of scenarios perform better or worse with which variant of the code. You will always be able to create a microbenchmark for which a particular code pattern works best. So I'd say stick to the simplest and least risky code and worry about performance issues like this if, or when, they turn up. > So, I'm okay with the webrev.07 modulo ThreadTable renaming. Okay. Thanks, David > Thanks, > Serguei > > > On 9/20/19 19:20, David Holmes wrote: >> Hi Serguei, >> >> On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >>> Hi Daniil, >>> >>> Yes, the Threads_lock is still needed around thread->is_exiting() >>> check and add_thread(). >>> >>>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>>> and then the nested Threads_lock around thread->is_exiting() and >>>> add_thread(java_tid, thread) lines then it will not work since the rank >>>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >>> >>> The ThreadTableCreate_lock is only used in the lazy_initialize() >>> function. >>> It looks safe to adjust the ThreadTableCreate_lock to fix the above >>> problem. >>> Then the approach below will work as needed. >>> >>> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ???? if (_is_initialized) { >>> ?????? return; >>> ???? } >>> ???? MutexLocker ml(ThreadTableCreate_lock); >>> ???? if (_is_initialized) { >>> ?????? // There is no obvious benefits in allowing the thread table >>> ?????? // being concurrently populated during the initalization. >>> ?????? return; >>> ???? } >>> ???? create_table(threads->length()); >>> ???? _is_initialized = true; >>> >>> ???? for (uint i = 0; i < threads->length(); i++) { >>> ?????? JavaThread* thread = threads->thread_at(i); >>> ?????? oop tobj = thread->threadObj(); >>> ?????? if (tobj != NULL) { >>> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >>> ???????? MutexLocker ml(Threads_lock); >>> ???????? if (!thread->is_exiting()) { >>> ?????????? // Must be inside the lock to ensure that we don't add the >>> thread to the table >>> ?????????? // that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> ?????????? add_thread(java_tid, thread); >>> ???????? } >>> ?????? } >>> ???? } >>> ?? } >> >> I don't like holding a completely unrelated lock, whilst holding the >> Threads_lock, particularly as it seems unnecessary for correctness. We >> have to be certain that no M&M operation that holds the Threads_lock >> can try to initialize the thread table. Testing can only prove the >> presence of that problem, not the absence. >> >> I think Daniil's webrev.07 version is correct and less risky than what >> you propose. Sorry. >> >> David >> ----- From serguei.spitsyn at oracle.com Sat Sep 21 06:59:45 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:59:45 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> Message-ID: On 9/20/19 23:53, David Holmes wrote: > On 21/09/2019 4:21 pm, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> I do not want to insist in the area (Threads_lock use) where you have >> much more expertise. > > It's more conservatism than expertise :) I had a look through and > there are not too many usages of the Threads_lock in the M&M code and > they do appear safe. I've figured the same. > But I still prefer on principle to not hold nested locks if it is not > necessary. Simplicity and conservatism rule! :) >> Also, I agreed that Daniil's webrev.07 version is correct and less >> risky. >> The only my concern was about possible performance impact of linear >> searches >> from the ThreadsList::find_JavaThread_from_java_tid() calls that can >> be executed >> concurrently with the ThreadTable::lazy_initialize() if the >> ThreadsList is big. >> But it should not be that big issue. > > I think we can waste a lot of time wondering about what kinds of > scenarios perform better or worse with which variant of the code. You > will always be able to create a microbenchmark for which a particular > code pattern works best. So I'd say stick to the simplest and least > risky code and worry about performance issues like this if, or when, > they turn up. Agreed. It is my usual approach too (which I've just tried to break). :) Thanks, Serguei >> So, I'm okay with the webrev.07 modulo ThreadTable renaming. > > Okay. > > Thanks, > David > >> Thanks, >> Serguei >> >> >> On 9/20/19 19:20, David Holmes wrote: >>> Hi Serguei, >>> >>> On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >>>> Hi Daniil, >>>> >>>> Yes, the Threads_lock is still needed around thread->is_exiting() >>>> check and add_thread(). >>>> >>>>> And if we try to use 2 locks, ThreadTableCreate_lock as in your >>>>> snippet >>>>> and then the nested Threads_lock around thread->is_exiting() and >>>>> add_thread(java_tid, thread) lines then it will not work since the >>>>> rank >>>>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >>>> >>>> The ThreadTableCreate_lock is only used in the lazy_initialize() >>>> function. >>>> It looks safe to adjust the ThreadTableCreate_lock to fix the above >>>> problem. >>>> Then the approach below will work as needed. >>>> >>>> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ???? if (_is_initialized) { >>>> ?????? return; >>>> ???? } >>>> ???? MutexLocker ml(ThreadTableCreate_lock); >>>> ???? if (_is_initialized) { >>>> ?????? // There is no obvious benefits in allowing the thread table >>>> ?????? // being concurrently populated during the initalization. >>>> ?????? return; >>>> ???? } >>>> ???? create_table(threads->length()); >>>> ???? _is_initialized = true; >>>> >>>> ???? for (uint i = 0; i < threads->length(); i++) { >>>> ?????? JavaThread* thread = threads->thread_at(i); >>>> ?????? oop tobj = thread->threadObj(); >>>> ?????? if (tobj != NULL) { >>>> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >>>> ???????? MutexLocker ml(Threads_lock); >>>> ???????? if (!thread->is_exiting()) { >>>> ?????????? // Must be inside the lock to ensure that we don't add >>>> the thread to the table >>>> ?????????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ?????????? add_thread(java_tid, thread); >>>> ???????? } >>>> ?????? } >>>> ???? } >>>> ?? } >>> >>> I don't like holding a completely unrelated lock, whilst holding the >>> Threads_lock, particularly as it seems unnecessary for correctness. >>> We have to be certain that no M&M operation that holds the >>> Threads_lock can try to initialize the thread table. Testing can >>> only prove the presence of that problem, not the absence. >>> >>> I think Daniil's webrev.07 version is correct and less risky than >>> what you propose. Sorry. >>> >>> David >>> ----- From serguei.spitsyn at oracle.com Sat Sep 21 07:13:24 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Sat, 21 Sep 2019 00:13:24 -0700 (PDT) Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> Message-ID: <2208a27a-f64e-5c15-3558-107024fd6463@oracle.com> Hi Christoph, The fix looks good to me. Thanks, Serguei On 9/20/19 14:13, Langer, Christoph wrote: > Thanks David! > > May I get another review? > > Best regards > Christoph > > >> -----Original Message----- >> From: David Holmes >> Sent: Donnerstag, 19. September 2019 13:56 >> To: Langer, Christoph ; Erik Joelsson >> ; Magnus Ihse Bursie >> ; OpenJDK Serviceability > dev at openjdk.java.net>; build-dev >> Subject: Re: RFR: 8230857: Avoid reflection in >> sun.tools.common.ProcessHelper >> >> Hi Christoph, >> >> On 19/09/2019 7:47 pm, Langer, Christoph wrote: >>> Hi, >>> >>> @Erik, Magnus: Thanks for stepping in to explain things. >>> >>> Now back to the actual change: Is this ok then (@David)? Any other >> reviews from somebody else? >>> http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ >> It seems okay. >> >> For the test I'm unclear on exactly how to ensure things are accessible, >> but presumably the +open is sufficient and works under all circumstances. >> >> Thanks, >> David >> >>> Thank you! >>> >>> Best regards >>> Christoph >>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Mittwoch, 18. September 2019 01:13 >>>> To: Erik Joelsson ; Magnus Ihse Bursie >>>> ; Langer, Christoph >>>> ; OpenJDK Serviceability >>> dev at openjdk.java.net>; build-dev >>>> Subject: Re: RFR: 8230857: Avoid reflection in >>>> sun.tools.common.ProcessHelper >>>> >>>> Hi Erik, >>>> >>>> Thanks for the additional details (I can't say I fully understand them :) ). >>>> >>>> David >>>> >>>> On 17/09/2019 11:39 pm, Erik Joelsson wrote: >>>>> Hello, >>>>> >>>>> On 2019-09-17 05:59, David Holmes wrote: >>>>>> Hi Magnus, >>>>>> >>>>>> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: >>>>>>> On 2019-09-17 01:01, David Holmes wrote: >>>>>>>> Hi Christoph, >>>>>>>> >>>>>>>> Sorry for the delay getting back you. >>>>>>>> >>>>>>>> cc'd build-dev to get some clarification on the below ... >>>>>>>> >>>>>>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: >>>>>>>>> Hi David, >>>>>>>>> >>>>>>>>>>> please review an enhancement which I've identified when >> working >>>> with >>>>>>>>>>> Processhelper for JDK-8230850. >>>>>>>>>>> >>>>>>>>>>> I noticed that ProcessHelper is an interface in common code with >> a >>>>>>>>>>> static method that would lookup the actual platform >>>>>>>>>>> implementation via >>>>>>>>>>> reflection. This seems a little cumbersome since we can have a >>>>>>>>>>> common >>>>>>>>>>> dummy for ProcessHelper and override it with the platform >> specific >>>>>>>>>>> implementation, leveraging the build system. >>>>>>>>>> I don't see you leveraging the build system. You have two source >>>>>>>>>> files >>>>>>>>>> that compile to the same destination class file. What is ensuring >> the >>>>>>>>>> platform specific version is compiled after the generic one? >>>>>>>>>> >>>>>>>>>> Service-provider patterns use reflection to instantiate the service >>>>>>>>>> implementation. I don't see any problem here that needs solving. >>>>>>>>> TL;DR: >>>>>>>>> There are two source files, one in share/classes and one in >>>>>>>>> linux/classes. The build system overrides the share/classes >>>>>>>>> implementation with the linux/classes implementation in the linux >>>>>>>>> build. This is not by coincidence and only one class is contained >>>>>>>>> in the generated jdk.jcmd module. Then there won't be a need for >>>>>>>>> having a service interface and a service implementation that is >>>>>>>>> looked up via reflection (which is not a bad pattern by itself). I >>>>>>>>> agree that it's not a big problem to be solved but still not "no >>>>>>>>> problem". >>>>>>>>> Here is some longer elaboration how the build system prefers >>>>>>>>> specific implementations of classes and filters generic duplicates: >>>>>>>>> The SetupJavaCompilation function from JavaCompilation.gmk [0] >> is >>>>>>>>> used to compile the java sources for JDK modules. In its >>>>>>>>> documentation, for argument SRC [1], it claims: "one or more >>>>>>>>> directories to search for sources. The order of the source roots is >>>>>>>>> significant. The first found file of a certain name has priority". >>>>>>>>> In its implementation the found files are first ordered [3] and >>>>>>>>> duplicates filtered out [4]. >>>>>>>>> The potential source files are handed to SetupJavaCompilation in >>>>>>>>> CompileJavaModules.gmk [5] and were collected by a call to >>>>>>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all >>>>>>>>> potential source dirs for Java classes in the module [7]. The >>>>>>>>> evaluated subdirs are (in that order) >>>> $(OPENJDK_TARGET_OS)/classes, >>>>>>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per >> [8]. >>>>>>>>> Hope that explains what I'm trying to leverage here. >>>>>>>> I'm not 100% certain that what you describe actually ensures what >>>>>>>> you want it to ensure. I can't reconcile "the first found file ... >>>>>>>> has priority" with the fact found files are sorted and duplicates >>>>>>>> eliminated. It is the sorting that concerns me as it suggests >>>>>>>> linux/Foo.java might replace shared/Foo.java, but if we're on >>>>>>>> Windows then we have a problem! That said there is also this >>>> comment: >>>>>>>> # Order src files according to the order of the src dirs. Correct >>>>>>>> odering is >>>>>>>> # needed for correct overriding between different source roots. >>>>>>>> >>>>>>>> I'd need the build team to clarify what "correct overriding" is >>>>>>>> actually defined as. >>>>>>> David, >>>>>>> >>>>>>> Christoph is correct. linux/Foo.java will override share/Foo.java. I >>>>>>> don't remember how the magic in JavaCompilation.gmk works >> anymore >>>>>>> :-), but we have relied on this behavior in other places for a long >>>>>>> time, so I'm pretty certain it is still working correctly. >>>>>>> Presumably, the $(sort ...) is there to remove (identical) >>>>>>> duplicates, which is a side-effect of sort. >>>>>> Thanks for confirming. I'd still like to understand exactly what these >>>>>> overriding rules are though. It's not a mechanism I was aware of. >>>>>> >>>>> SetupJavaCompilation is indeed behaving as Christoph describes and it is >>>>> by design. I implemented support for this behavior in: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8079344 >>>>> >>>>> The relevant parts of SetupJavaCompilation look like this: >>>>> >>>>> ? # Order src files according to the order of the src dirs. Correct >>>>> odering is >>>>> ? # needed for correct overriding between different source roots. >>>>> ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) >>>>> ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ >>>>> ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) >>>>> >>>>> The second line orders the src files by the src roots. (We used to just >>>>> call find for one src root at a time, but the above actually performs >>>>> better due only running 1 external process) >>>>> >>>>> Further down we have this: >>>>> >>>>> ? ifneq ($$($1_KEEP_DUPS), true) >>>>> ??? # Remove duplicate source files by keeping the first found of each >>>>> duplicate. >>>>> ??? # This allows for automatic overrides with custom or platform >>>>> specific versions >>>>> ??? # source files. >>>>> ??? # >>>>> ??? # For the smart javac wrapper case, add each removed file to an >>>>> extra exclude >>>>> ??? # file list to prevent sjavac from finding duplicate sources. >>>>> ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ >>>>> ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), >>>>> $$(s))) \ >>>>> ??????? $$(if $$($1_$$(relative_src)), \ >>>>> ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ >>>>> ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) >>>>> ? endif >>>>> >>>>> This loop is a bit hairy to wrap your head around. It's iterating over >>>>> all the src files, in the order of importance. The variable relative_src >>>>> is the path from the src root, the part that is common to all duplicate >>>>> src files. The variables on the form $1_$$(relative_src) basically act >>>>> as a hash map (string->boolean). So for each src file, if the relative >>>>> path for it has already been seen, add it to an exclude list, else mark >>>>> it as seen and add it to the return list. >>>>> >>>>> /Erik >>>>> From richard.reingruber at sap.com Sat Sep 21 21:06:55 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Sat, 21 Sep 2019 21:06:55 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi David, I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide optimizations from its client (Joe the Java developer). There has to be a way to debug an app without optimizations getting into your way. I like to debug my C/C++ program after I compiled it -O0 -g. There has to be something equivalent for Java debugging. And I don't mean -Xint. Since 1.4.1 we've got "Full Speed Debugging" [1]. So far JVMTI does an excellent job of making optimizations transparent to the user. If it didn't, I would like to file a JEP for an interface that shows the pure JVM state (at full speed of course). > > > You seem to have completely missed my point. If the object is local and > > > is synchronized upon then the synchronization can be elided (and should > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > > escape. If the synchronization cannot be elided then the object cannot > > > be considered local. That is how Escape Analysis should be operating > > > here IMHO. > > > > I presume we agree that it is the state of the abstract virtual machine that must be observed > > through JVMTI, right? > > > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > > abstract virtual machine. > > I don't agree. If the locking can be elided then it is completely > elided. If the state of the "abstract VM" had to be perfectly preserved > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > "Optimizing transformations of a program can be designed that reduce the > number of objects that are reachable to be less than those which would > naively be considered reachable." I'd like to write a few words about the relationship of specification and optimizations: It's the purpose of a specification to define an abstract model. Optimizations are techniques employed by an implementation to reduce resource consumption for better performance within the bounds set by the spec, meaning optimizations have to be transparent to the program. You can't write a program that detects an optimization. If you could then the implementation would violate rules given by the specification. (And I don't consider a benchmark a prove.) Very rarely you'll find specs mention optimizations. You just don't want to mix specification and implementation! If they do, they trade beauty and simplicity of an abstract model, ideally based on formal definitions, for performance. They do mention an optimization, if it is impossible to hide it. In the case of JLS 12.6 it is possible to detect that an object reference is removed from a local variable by means of finalizations and java.lang.ref. Implementors wanted it so badly, though, that they changed the spec. Another example in the future could be tail-call-optimization ... > I place lock elision in the same category of "optimising > transformations" that changes what would "naively" be expected. Now this > should be explicitly covered in the JLS/JVMS somewhere but I'm having > trouble finding exactly where. This article discusses lock elision: > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > and states: > > "It stands to reason, then, that if a thread enters a synchronized block > protected by a lock that no other thread will ever synchronize on, then > that synchronization has no effect and can therefore be removed by the > optimizer. (The Java Language Specification explicitly allows this > optimization.)" I doubt you'll find it in the JLS/JVMS. It's an implementation detail permitted by the memory model. That's what the article is saying. If a Java app has data races, then you would not reason about removed synchronization. You would argue with the means of the memory model: eg the synchronization action A1 (JLS 17.4.2 [2]) does not 'synchronized-with' (17.4.4 [3]) sync. action A2, so they are not ordered. > I'll have to ping Brian to see if he recalls exactly where this is > covered. :) Ok :) Please let me know! Richard. [1] https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/enhancements1.4.html#fsd [2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.2 [3] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.4 -----Original Message----- From: David Holmes Sent: Freitag, 20. September 2019 11:07 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Richard, On 20/09/2019 6:31 pm, Reingruber, Richard wrote: > Hi David, > > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: > > > Hi David, > > > > > > thanks for looking at this issue. And my appologies for the lengthy mail... > > > > > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and > > > > > optimizations like scalar replacement become invalid. > > > > > > > > What bothers me about this is that it seems like escape analysis is > > > > doing something wrong in this case. > > > > > > Yes it is. > > > > > > > If the object is thread-local but is > > > > being synchronized upon then either: > > > > > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis > > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual > > > machine, as observed through JVMTI, is wrong. See below... > > > > You seem to have completely missed my point. If the object is local and > > is synchronized upon then the synchronization can be elided (and should > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > escape. If the synchronization cannot be elided then the object cannot > > be considered local. That is how Escape Analysis should be operating > > here IMHO. > > I presume we agree that it is the state of the abstract virtual machine that must be observed > through JVMTI, right? > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > abstract virtual machine. I don't agree. If the locking can be elided then it is completely elided. If the state of the "abstract VM" had to be perfectly preserved then we wouldn't be able to do the optimisations referred to in JLS 12.6: "Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable." I place lock elision in the same category of "optimising transformations" that changes what would "naively" be expected. Now this should be explicitly covered in the JLS/JVMS somewhere but I'm having trouble finding exactly where. This article discusses lock elision: https://www.ibm.com/developerworks/library/j-jtp10185/index.html and states: "It stands to reason, then, that if a thread enters a synchronized block protected by a lock that no other thread will ever synchronize on, then that synchronization has no effect and can therefore be removed by the optimizer. (The Java Language Specification explicitly allows this optimization.)" I'll have to ping Brian to see if he recalls exactly where this is covered. :) David ----- > > Cheers, Richard. > > -----Original Message----- > From: David Holmes > Sent: Freitag, 20. September 2019 00:59 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >> Hi David, >> >> thanks for looking at this issue. And my appologies for the lengthy mail... >> >> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >> > > optimizations like scalar replacement become invalid. >> > >> > What bothers me about this is that it seems like escape analysis is >> > doing something wrong in this case. >> >> Yes it is. >> >> > If the object is thread-local but is >> > being synchronized upon then either: >> >> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >> machine, as observed through JVMTI, is wrong. See below... > > You seem to have completely missed my point. If the object is local and > is synchronized upon then the synchronization can be elided (and should > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > escape. If the synchronization cannot be elided then the object cannot > be considered local. That is how Escape Analysis should be operating > here IMHO. > > Cheers, > David > ----- > >> > a) the synchronization is elided and so the object will not appear in >> > the set of owned monitors; or >> > b) the fact synchronization occurs renders the object ineligible to be >> > considered thread-local, and so there is no problem with it appearing in >> > the set of owned monitors >> > >> > I think there is a bigger design philosophy issue here about the >> > interaction of escape analysis and debugging/management frameworks in >> > general. I'd like to see a very clear description on exactly how they >> > should interact. >> > >> >> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >> state of the virtual machine according to the spec. I think it fails to do so in this case. >> >> Please look again at the test: >> >> 172 public long dontinline_endlessLoop() { >> 173 long cs = checkSum; >> 174 while (doLoop && loopCount-- > 0) { >> 175 targetIsInLoop = true; >> 176 checkSum += checkSum % ++cs; >> 177 } >> 178 loopCount = 3; >> 179 targetIsInLoop = false; >> 180 return checkSum; >> 181 } >> >> 249 public void dontinline_testMethod() { >> 250 LockCls l1 = new LockCls(); // to be scalar replaced >> 251 synchronized (l1) { >> 252 inlinedTestMethodWithNestedLocking(l1); >> 253 } >> 254 } >> 255 >> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >> 257 synchronized (l1) { // nested >> 258 dontinline_endlessLoop(); >> 259 } >> 260 } >> >> This is the stack when the agent calls GetOwnedMonitorInfo() >> >> dontinline_endlessLoop() at line 176 >> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >> dontinline_testMethod() at line 252 // compiled frame >> >> The state of the _virtual_ machine at that point is obvious: >> >> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >> - That instance was locked by a monitorenter bytecode at line 251 >> >> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >> pencil (hope you won't make me do this, though ;)) >> >> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >> not. >> >> From a more philosophical point of view compiled code is free to change the state of the physical >> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >> and every machine instruction. But it must reach points in its execution trace, where it is actually >> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >> points are called safepoints. >> >> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >> state is presented. EA does not take into account that the lock state can be observed using >> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >> can_get_owned_monitor_info is taken. With the fix the test passes. >> >> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >> the JVMTI implementation cannot handout references to objects stored in local variables if they were >> scalar replaced. >> >> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >> and relock_objects() in class Deoptimization) >> >> Thanks and cheers, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> >> -----Original Message----- >> From: David Holmes >> Sent: Donnerstag, 19. September 2019 02:43 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> Hi Richard, >> >> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>> Hi, >>> >>> could I please get reviews for >>> >>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>> >>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> optimizations like scalar replacement become invalid. >> >> What bothers me about this is that it seems like escape analysis is >> doing something wrong in this case. If the object is thread-local but is >> being synchronized upon then either: >> a) the synchronization is elided and so the object will not appear in >> the set of owned monitors; or >> b) the fact synchronization occurs renders the object ineligible to be >> considered thread-local, and so there is no problem with it appearing in >> the set of owned monitors >> >> I think there is a bigger design philosophy issue here about the >> interaction of escape analysis and debugging/management frameworks in >> general. I'd like to see a very clear description on exactly how they >> should interact. >> >> Cheers, >> David >> >>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>> >>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>> objects escape through JVMTI. >>> >>> I've run tier1 tests. >>> >>> Thanks, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>> From david.holmes at oracle.com Mon Sep 23 06:59:01 2019 From: david.holmes at oracle.com (David Holmes) Date: Mon, 23 Sep 2019 16:59:01 +1000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi Richard, On 22/09/2019 7:06 am, Reingruber, Richard wrote: > Hi David, > > I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide > optimizations from its client (Joe the Java developer). I'm not sure I agree as a blanket statement. It may not be feasible to perform certain optimisations and at the same time keep around enough information to reconstruct the state as it would have been had the optimization not occurred. That's why I want to see a discussion of the broader issue of how something like escape analysis is supposed to interact with facilities like JVM TI. You've flagged a couple of conditions under which EA should be disabled, but are they sufficient? Give they are both dynamic facilities what is supposed to happen if you keep switching things on and off? Does enabling can_get_owned_monitor_info require previous EA actions to be undone? I don't know the answers here. I certainly hope these questions have been considered when EA was being developed - in which case I'd like to see the discussion. Otherwise this is a conversation that needs to be had, and the development of an overall approach for dealing with this, not trying to address things piece-meal as someone recognises a new problem. > There has to be a way to debug an app without optimizations getting into your way. I like to debug > my C/C++ program after I compiled it -O0 -g. There has to be something equivalent for Java > debugging. And I don't mean -Xint. Since 1.4.1 we've got "Full Speed Debugging" [1]. > > So far JVMTI does an excellent job of making optimizations transparent to the user. If it didn't, > I would like to file a JEP for an interface that shows the pure JVM state (at full speed of > course). > > > > > You seem to have completely missed my point. If the object is local and > > > > is synchronized upon then the synchronization can be elided (and should > > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > > > escape. If the synchronization cannot be elided then the object cannot > > > > be considered local. That is how Escape Analysis should be operating > > > > here IMHO. > > > > > > I presume we agree that it is the state of the abstract virtual machine that must be observed > > > through JVMTI, right? > > > > > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > > > > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > > > abstract virtual machine. > > > > I don't agree. If the locking can be elided then it is completely > > elided. If the state of the "abstract VM" had to be perfectly preserved > > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > > > "Optimizing transformations of a program can be designed that reduce the > > number of objects that are reachable to be less than those which would > > naively be considered reachable." > > I'd like to write a few words about the relationship of specification and optimizations: It's the purpose > of a specification to define an abstract model. Optimizations are techniques employed by an > implementation to reduce resource consumption for better performance within the bounds set by the > spec, meaning optimizations have to be transparent to the program. You can't write a program that > detects an optimization. If you could then the implementation would violate rules given by the > specification. (And I don't consider a benchmark a prove.) > > Very rarely you'll find specs mention optimizations. You just don't want to mix specification and > implementation! If they do, they trade beauty and simplicity of an abstract model, ideally based > on formal definitions, for performance. > > They do mention an optimization, if it is impossible to hide it. In the case of JLS 12.6 it is > possible to detect that an object reference is removed from a local variable by means of > finalizations and java.lang.ref. Implementors wanted it so badly, though, that they changed the > spec. Specs are imperfect and always evolving. Sometimes the consequences of specific optimizations may not be realised at the spec level yet need to be accounted for. That may or may not be the case with EA. I find it hard to see how to do lock elision yet at the same time track when the lock would have been taken, without negating the whole purpose of eliding the lock. Afterall if the object is thread-local then the locking would amount to only biased-locking and it could never be contended. So it doesn't seem like there was much to elide in the first place. Lock coarsening is also problematic. > Another example in the future could be tail-call-optimization ... > > > I place lock elision in the same category of "optimising > > transformations" that changes what would "naively" be expected. Now this > > should be explicitly covered in the JLS/JVMS somewhere but I'm having > > trouble finding exactly where. This article discusses lock elision: > > > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > > > and states: > > > > "It stands to reason, then, that if a thread enters a synchronized block > > protected by a lock that no other thread will ever synchronize on, then > > that synchronization has no effect and can therefore be removed by the > > optimizer. (The Java Language Specification explicitly allows this > > optimization.)" > > I doubt you'll find it in the JLS/JVMS. It's an implementation detail permitted by the memory > model. That's what the article is saying. If a Java app has data races, then you would not reason > about removed synchronization. You would argue with the means of the memory model: eg the > synchronization action A1 (JLS 17.4.2 [2]) does not 'synchronized-with' (17.4.4 [3]) sync. action > A2, so they are not ordered. > > > I'll have to ping Brian to see if he recalls exactly where this is > > covered. :) > > Ok :) > Please let me know! Brian conceded it may be more implicit in the JMM as you described above :) Cheers, David > Richard. > > [1] https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/enhancements1.4.html#fsd > [2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.2 > [3] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.4 > > -----Original Message----- > From: David Holmes > Sent: Freitag, 20. September 2019 11:07 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 20/09/2019 6:31 pm, Reingruber, Richard wrote: >> Hi David, >> >> > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >> > > Hi David, >> > > >> > > thanks for looking at this issue. And my appologies for the lengthy mail... >> > > >> > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >> > > > > optimizations like scalar replacement become invalid. >> > > > >> > > > What bothers me about this is that it seems like escape analysis is >> > > > doing something wrong in this case. >> > > >> > > Yes it is. >> > > >> > > > If the object is thread-local but is >> > > > being synchronized upon then either: >> > > >> > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >> > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >> > > machine, as observed through JVMTI, is wrong. See below... >> > >> > You seem to have completely missed my point. If the object is local and >> > is synchronized upon then the synchronization can be elided (and should >> > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> > escape. If the synchronization cannot be elided then the object cannot >> > be considered local. That is how Escape Analysis should be operating >> > here IMHO. >> >> I presume we agree that it is the state of the abstract virtual machine that must be observed >> through JVMTI, right? >> >> The locking state of an object O after a monitorenter on O is locked on the abstract vm. >> >> The JIT can still elide synchronization based on a prove that it is actually redundant for the >> computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >> abstract virtual machine. > > I don't agree. If the locking can be elided then it is completely > elided. If the state of the "abstract VM" had to be perfectly preserved > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > "Optimizing transformations of a program can be designed that reduce the > number of objects that are reachable to be less than those which would > naively be considered reachable." > > I place lock elision in the same category of "optimising > transformations" that changes what would "naively" be expected. Now this > should be explicitly covered in the JLS/JVMS somewhere but I'm having > trouble finding exactly where. This article discusses lock elision: > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > and states: > > "It stands to reason, then, that if a thread enters a synchronized block > protected by a lock that no other thread will ever synchronize on, then > that synchronization has no effect and can therefore be removed by the > optimizer. (The Java Language Specification explicitly allows this > optimization.)" > > I'll have to ping Brian to see if he recalls exactly where this is > covered. :) > > David > ----- > >> >> Cheers, Richard. >> >> -----Original Message----- >> From: David Holmes >> Sent: Freitag, 20. September 2019 00:59 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>> Hi David, >>> >>> thanks for looking at this issue. And my appologies for the lengthy mail... >>> >>> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> > > optimizations like scalar replacement become invalid. >>> > >>> > What bothers me about this is that it seems like escape analysis is >>> > doing something wrong in this case. >>> >>> Yes it is. >>> >>> > If the object is thread-local but is >>> > being synchronized upon then either: >>> >>> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>> machine, as observed through JVMTI, is wrong. See below... >> >> You seem to have completely missed my point. If the object is local and >> is synchronized upon then the synchronization can be elided (and should >> be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> escape. If the synchronization cannot be elided then the object cannot >> be considered local. That is how Escape Analysis should be operating >> here IMHO. >> >> Cheers, >> David >> ----- >> >>> > a) the synchronization is elided and so the object will not appear in >>> > the set of owned monitors; or >>> > b) the fact synchronization occurs renders the object ineligible to be >>> > considered thread-local, and so there is no problem with it appearing in >>> > the set of owned monitors >>> > >>> > I think there is a bigger design philosophy issue here about the >>> > interaction of escape analysis and debugging/management frameworks in >>> > general. I'd like to see a very clear description on exactly how they >>> > should interact. >>> > >>> >>> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >>> state of the virtual machine according to the spec. I think it fails to do so in this case. >>> >>> Please look again at the test: >>> >>> 172 public long dontinline_endlessLoop() { >>> 173 long cs = checkSum; >>> 174 while (doLoop && loopCount-- > 0) { >>> 175 targetIsInLoop = true; >>> 176 checkSum += checkSum % ++cs; >>> 177 } >>> 178 loopCount = 3; >>> 179 targetIsInLoop = false; >>> 180 return checkSum; >>> 181 } >>> >>> 249 public void dontinline_testMethod() { >>> 250 LockCls l1 = new LockCls(); // to be scalar replaced >>> 251 synchronized (l1) { >>> 252 inlinedTestMethodWithNestedLocking(l1); >>> 253 } >>> 254 } >>> 255 >>> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >>> 257 synchronized (l1) { // nested >>> 258 dontinline_endlessLoop(); >>> 259 } >>> 260 } >>> >>> This is the stack when the agent calls GetOwnedMonitorInfo() >>> >>> dontinline_endlessLoop() at line 176 >>> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >>> dontinline_testMethod() at line 252 // compiled frame >>> >>> The state of the _virtual_ machine at that point is obvious: >>> >>> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >>> - That instance was locked by a monitorenter bytecode at line 251 >>> >>> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >>> pencil (hope you won't make me do this, though ;)) >>> >>> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >>> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >>> not. >>> >>> From a more philosophical point of view compiled code is free to change the state of the physical >>> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >>> and every machine instruction. But it must reach points in its execution trace, where it is actually >>> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >>> points are called safepoints. >>> >>> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >>> state is presented. EA does not take into account that the lock state can be observed using >>> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >>> can_get_owned_monitor_info is taken. With the fix the test passes. >>> >>> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >>> the JVMTI implementation cannot handout references to objects stored in local variables if they were >>> scalar replaced. >>> >>> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >>> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >>> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >>> and relock_objects() in class Deoptimization) >>> >>> Thanks and cheers, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Donnerstag, 19. September 2019 02:43 >>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>> >>> Hi Richard, >>> >>> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>>> Hi, >>>> >>>> could I please get reviews for >>>> >>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>>> >>>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>> optimizations like scalar replacement become invalid. >>> >>> What bothers me about this is that it seems like escape analysis is >>> doing something wrong in this case. If the object is thread-local but is >>> being synchronized upon then either: >>> a) the synchronization is elided and so the object will not appear in >>> the set of owned monitors; or >>> b) the fact synchronization occurs renders the object ineligible to be >>> considered thread-local, and so there is no problem with it appearing in >>> the set of owned monitors >>> >>> I think there is a bigger design philosophy issue here about the >>> interaction of escape analysis and debugging/management frameworks in >>> general. I'd like to see a very clear description on exactly how they >>> should interact. >>> >>> Cheers, >>> David >>> >>>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>>> >>>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>>> objects escape through JVMTI. >>>> >>>> I've run tier1 tests. >>>> >>>> Thanks, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>> From daniel.fuchs at oracle.com Mon Sep 23 09:06:23 2019 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 23 Sep 2019 10:06:23 +0100 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <7de738c3-a9b8-5d49-870a-ae7392ead85a@oracle.com> Message-ID: <89babb17-5a0e-462c-e4c4-4bf2813d5689@oracle.com> Hi, On 21/09/2019 00:28, Hohensee, Paul wrote: > java.lang.management.ThreadMXBean has two default methods that throw UnsupportedOperationException: > > ThreadInfo[] getThreadInfo(long[], boolean, boolean, int) > ThreadInfo[] dumpAllThreads(boolean, boolean, int) > > Without actually testing it, is it safe to assume that since com.sun.management.ThreadMXBean extends j.l.m.ThreadMXBean, the new c.s.m.ThreadMXBean method > > public default long getCurrentThreadAllocatedBytes() { > throw new UnsupportedOperationException(); > } > > will behave the same way? Oh right, I'd forgotten about that. Looking at the history it seems I'd even checked that it did work at the time. duh. How time flies... Thanks Paul! best regards, -- daniel From christoph.langer at sap.com Mon Sep 23 10:34:09 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Mon, 23 Sep 2019 10:34:09 +0000 Subject: RFR: 8230857: Avoid reflection in sun.tools.common.ProcessHelper In-Reply-To: <2208a27a-f64e-5c15-3558-107024fd6463@oracle.com> References: <555a2cf2-e15e-abb6-5c0a-fb3ff4c0716f@oracle.com> <7cef2fd2-74cb-f069-d837-b5219924efc0@oracle.com> <105a9f1f-9b1e-c707-b65d-6e71db7c701d@oracle.com> <431b85fb-b131-b55b-f7a8-d7112b2c9fa4@oracle.com> <98c7eadb-3003-cea7-f5a3-c902d9a6f1db@oracle.com> <2208a27a-f64e-5c15-3558-107024fd6463@oracle.com> Message-ID: Thank you, Serguei. I pushed it now: http://hg.openjdk.java.net/jdk/jdk/rev/577e17cab93f > -----Original Message----- > From: serguei.spitsyn at oracle.com > Sent: Samstag, 21. September 2019 09:13 > To: Langer, Christoph ; David Holmes > ; OpenJDK Serviceability dev at openjdk.java.net> > Subject: Re: RFR: 8230857: Avoid reflection in > sun.tools.common.ProcessHelper > > Hi Christoph, > > The fix looks good to me. > > Thanks, > Serguei > > > On 9/20/19 14:13, Langer, Christoph wrote: > > Thanks David! > > > > May I get another review? > > > > Best regards > > Christoph > > > > > >> -----Original Message----- > >> From: David Holmes > >> Sent: Donnerstag, 19. September 2019 13:56 > >> To: Langer, Christoph ; Erik Joelsson > >> ; Magnus Ihse Bursie > >> ; OpenJDK Serviceability > >> dev at openjdk.java.net>; build-dev > >> Subject: Re: RFR: 8230857: Avoid reflection in > >> sun.tools.common.ProcessHelper > >> > >> Hi Christoph, > >> > >> On 19/09/2019 7:47 pm, Langer, Christoph wrote: > >>> Hi, > >>> > >>> @Erik, Magnus: Thanks for stepping in to explain things. > >>> > >>> Now back to the actual change: Is this ok then (@David)? Any other > >> reviews from somebody else? > >>> http://cr.openjdk.java.net/~clanger/webrevs/8230857.1/ > >> It seems okay. > >> > >> For the test I'm unclear on exactly how to ensure things are accessible, > >> but presumably the +open is sufficient and works under all circumstances. > >> > >> Thanks, > >> David > >> > >>> Thank you! > >>> > >>> Best regards > >>> Christoph > >>> > >>>> -----Original Message----- > >>>> From: David Holmes > >>>> Sent: Mittwoch, 18. September 2019 01:13 > >>>> To: Erik Joelsson ; Magnus Ihse Bursie > >>>> ; Langer, Christoph > >>>> ; OpenJDK Serviceability >>>> dev at openjdk.java.net>; build-dev > >>>> Subject: Re: RFR: 8230857: Avoid reflection in > >>>> sun.tools.common.ProcessHelper > >>>> > >>>> Hi Erik, > >>>> > >>>> Thanks for the additional details (I can't say I fully understand them :) ). > >>>> > >>>> David > >>>> > >>>> On 17/09/2019 11:39 pm, Erik Joelsson wrote: > >>>>> Hello, > >>>>> > >>>>> On 2019-09-17 05:59, David Holmes wrote: > >>>>>> Hi Magnus, > >>>>>> > >>>>>> On 17/09/2019 9:26 pm, Magnus Ihse Bursie wrote: > >>>>>>> On 2019-09-17 01:01, David Holmes wrote: > >>>>>>>> Hi Christoph, > >>>>>>>> > >>>>>>>> Sorry for the delay getting back you. > >>>>>>>> > >>>>>>>> cc'd build-dev to get some clarification on the below ... > >>>>>>>> > >>>>>>>> On 12/09/2019 7:30 pm, Langer, Christoph wrote: > >>>>>>>>> Hi David, > >>>>>>>>> > >>>>>>>>>>> please review an enhancement which I've identified when > >> working > >>>> with > >>>>>>>>>>> Processhelper for JDK-8230850. > >>>>>>>>>>> > >>>>>>>>>>> I noticed that ProcessHelper is an interface in common code > with > >> a > >>>>>>>>>>> static method that would lookup the actual platform > >>>>>>>>>>> implementation via > >>>>>>>>>>> reflection. This seems a little cumbersome since we can have > a > >>>>>>>>>>> common > >>>>>>>>>>> dummy for ProcessHelper and override it with the platform > >> specific > >>>>>>>>>>> implementation, leveraging the build system. > >>>>>>>>>> I don't see you leveraging the build system. You have two > source > >>>>>>>>>> files > >>>>>>>>>> that compile to the same destination class file. What is ensuring > >> the > >>>>>>>>>> platform specific version is compiled after the generic one? > >>>>>>>>>> > >>>>>>>>>> Service-provider patterns use reflection to instantiate the > service > >>>>>>>>>> implementation. I don't see any problem here that needs > solving. > >>>>>>>>> TL;DR: > >>>>>>>>> There are two source files, one in share/classes and one in > >>>>>>>>> linux/classes. The build system overrides the share/classes > >>>>>>>>> implementation with the linux/classes implementation in the > linux > >>>>>>>>> build. This is not by coincidence and only one class is contained > >>>>>>>>> in the generated jdk.jcmd module. Then there won't be a need > for > >>>>>>>>> having a service interface and a service implementation that is > >>>>>>>>> looked up via reflection (which is not a bad pattern by itself). I > >>>>>>>>> agree that it's not a big problem to be solved but still not "no > >>>>>>>>> problem". > >>>>>>>>> Here is some longer elaboration how the build system prefers > >>>>>>>>> specific implementations of classes and filters generic duplicates: > >>>>>>>>> The SetupJavaCompilation function from JavaCompilation.gmk > [0] > >> is > >>>>>>>>> used to compile the java sources for JDK modules. In its > >>>>>>>>> documentation, for argument SRC [1], it claims: "one or more > >>>>>>>>> directories to search for sources. The order of the source roots is > >>>>>>>>> significant. The first found file of a certain name has priority". > >>>>>>>>> In its implementation the found files are first ordered [3] and > >>>>>>>>> duplicates filtered out [4]. > >>>>>>>>> The potential source files are handed to SetupJavaCompilation in > >>>>>>>>> CompileJavaModules.gmk [5] and were collected by a call to > >>>>>>>>> FindModuleSrcDirs [6]. FindModuleSrcDirs iterates over all > >>>>>>>>> potential source dirs for Java classes in the module [7]. The > >>>>>>>>> evaluated subdirs are (in that order) > >>>> $(OPENJDK_TARGET_OS)/classes, > >>>>>>>>> $(OPENJDK_TARGET_OS_TYPE)/classes and share/classes, as per > >> [8]. > >>>>>>>>> Hope that explains what I'm trying to leverage here. > >>>>>>>> I'm not 100% certain that what you describe actually ensures what > >>>>>>>> you want it to ensure. I can't reconcile "the first found file ... > >>>>>>>> has priority" with the fact found files are sorted and duplicates > >>>>>>>> eliminated. It is the sorting that concerns me as it suggests > >>>>>>>> linux/Foo.java might replace shared/Foo.java, but if we're on > >>>>>>>> Windows then we have a problem! That said there is also this > >>>> comment: > >>>>>>>> # Order src files according to the order of the src dirs. Correct > >>>>>>>> odering is > >>>>>>>> # needed for correct overriding between different source roots. > >>>>>>>> > >>>>>>>> I'd need the build team to clarify what "correct overriding" is > >>>>>>>> actually defined as. > >>>>>>> David, > >>>>>>> > >>>>>>> Christoph is correct. linux/Foo.java will override share/Foo.java. I > >>>>>>> don't remember how the magic in JavaCompilation.gmk works > >> anymore > >>>>>>> :-), but we have relied on this behavior in other places for a long > >>>>>>> time, so I'm pretty certain it is still working correctly. > >>>>>>> Presumably, the $(sort ...) is there to remove (identical) > >>>>>>> duplicates, which is a side-effect of sort. > >>>>>> Thanks for confirming. I'd still like to understand exactly what these > >>>>>> overriding rules are though. It's not a mechanism I was aware of. > >>>>>> > >>>>> SetupJavaCompilation is indeed behaving as Christoph describes and > it is > >>>>> by design. I implemented support for this behavior in: > >>>>> > >>>>> https://bugs.openjdk.java.net/browse/JDK-8079344 > >>>>> > >>>>> The relevant parts of SetupJavaCompilation look like this: > >>>>> > >>>>> ? # Order src files according to the order of the src dirs. Correct > >>>>> odering is > >>>>> ? # needed for correct overriding between different source roots. > >>>>> ? $1_ALL_SRC_RAW := $$(call FindFiles, $$($1_SRC)) > >>>>> ? $1_ALL_SRCS := $$($1_EXTRA_FILES) \ > >>>>> ????? $$(foreach d, $$($1_SRC), $$(filter $$d%, $$($1_ALL_SRC_RAW))) > >>>>> > >>>>> The second line orders the src files by the src roots. (We used to just > >>>>> call find for one src root at a time, but the above actually performs > >>>>> better due only running 1 external process) > >>>>> > >>>>> Further down we have this: > >>>>> > >>>>> ? ifneq ($$($1_KEEP_DUPS), true) > >>>>> ??? # Remove duplicate source files by keeping the first found of each > >>>>> duplicate. > >>>>> ??? # This allows for automatic overrides with custom or platform > >>>>> specific versions > >>>>> ??? # source files. > >>>>> ??? # > >>>>> ??? # For the smart javac wrapper case, add each removed file to an > >>>>> extra exclude > >>>>> ??? # file list to prevent sjavac from finding duplicate sources. > >>>>> ??? $1_SRCS := $$(strip $$(foreach s, $$($1_SRCS), \ > >>>>> ??????? $$(eval relative_src := $$(call remove-prefixes, $$($1_SRC), > >>>>> $$(s))) \ > >>>>> ??????? $$(if $$($1_$$(relative_src)), \ > >>>>> ????????? $$(eval $1_SJAVAC_EXCLUDE_FILES += $$(s)), \ > >>>>> ????????? $$(eval $1_$$(relative_src) := 1) $$(s)))) > >>>>> ? endif > >>>>> > >>>>> This loop is a bit hairy to wrap your head around. It's iterating over > >>>>> all the src files, in the order of importance. The variable relative_src > >>>>> is the path from the src root, the part that is common to all duplicate > >>>>> src files. The variables on the form $1_$$(relative_src) basically act > >>>>> as a hash map (string->boolean). So for each src file, if the relative > >>>>> path for it has already been seen, add it to an exclude list, else mark > >>>>> it as seen and add it to the return list. > >>>>> > >>>>> /Erik > >>>>> From richard.reingruber at sap.com Mon Sep 23 15:12:41 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Mon, 23 Sep 2019 15:12:41 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: Hi David, The JVMTI documentation answers the question "What is the JVM Tool Interface?" as follows "It provides both a way to inspect the state and to control the execution of applications running in the JavaTM virtual machine (VM)" So it's the state on the _virtual_ machine. And then there's another section "Specification Context". I should have found it before, but it is the 16th section of the introduction. It should be the first, really. Specification Context Since this interface provides access to the state of applications running in the Java virtual machine; terminology refers to the Java platform and not the native platform (unless stated otherwise). For example: "thread" means Java programming language thread. "stack frame" means Java virtual machine stack frame. "class" means Java programming language class. "heap" means Java virtual machine heap. "monitor" means Java programming language object monitor. Based on this, in particular "...not the native platform...", I'd think we can finally agree that it is the state of the abstract virtual machine that's presented to JVMTI clients. I've answered more of your questions below. Please let me know, if you think this needs more attention and discussion. If so, I would like to start a new thread, because this one won't have to many followers anymore ;) Is mailing list discussion the right format at all? Richard. > > Hi David, > > > > I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide > > optimizations from its client (Joe the Java developer). > > I'm not sure I agree as a blanket statement. It may not be feasible to > perform certain optimisations and at the same time keep around enough > information to reconstruct the state as it would have been had the > optimization not occurred. It is feasible. And it is done already at every safepoint in a nmethod to be able to switch from compiled to interpreted execution. > That's why I want to see a discussion of the > broader issue of how something like escape analysis is supposed to > interact with facilities like JVM TI. You've flagged a couple of > conditions under which EA should be disabled, but are they sufficient? Well, I've created JDK-8227745. I claim it handles all EA issues. Vladimir Kozlov suggested to create separate bugs for cases when EA optimizations can be observed. That makes sense to be. And allows to downport the conservative (i.e. disabling EA) fixes. > Give they are both dynamic facilities what is supposed to happen if you > keep switching things on and off? Does enabling > can_get_owned_monitor_info require previous EA actions to be undone? can_get_owned_monitor_info is a capability that can be added only in the OnLoad phase, so nothing needs to be undone. can_tag_objects is dynamic (JDK-8230956), so disabling EA does not help in all cases. I have elaborated on this in the bug. With JDK-8227745, though, it does not matter how often the capability is added/removed. The EA base optimizations can be reverted as needed just-in-time. > I don't know the answers here. I certainly hope these questions have > been considered when EA was being developed - in which case I'd like to > see the discussion. Otherwise this is a conversation that needs to be > had, and the development of an overall approach for dealing with this, > not trying to address things piece-meal as someone recognises a new problem. > Specs are imperfect and always evolving. Sometimes the consequences of > specific optimizations may not be realised at the spec level yet need to > be accounted for. That may or may not be the case with EA. > > I find it hard to see how to do lock elision yet at the same time track > when the lock would have been taken, without negating the whole purpose > of eliding the lock. As stated above: this is done for deoptimization already. All tracking information is collected at compile time. There are no runtime costs, except size of debuginfo. JDK-8227745 does what deoptimization does: reallocating and relocking objects. Difference is that JDK-8227745 does it right before JVMTI access. Deoptimization does it when replacing a compiled frame by corresponding interpreted frames. -----Original Message----- From: David Holmes Sent: Montag, 23. September 2019 08:59 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Richard, On 22/09/2019 7:06 am, Reingruber, Richard wrote: > Hi David, > > I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide > optimizations from its client (Joe the Java developer). I'm not sure I agree as a blanket statement. It may not be feasible to perform certain optimisations and at the same time keep around enough information to reconstruct the state as it would have been had the optimization not occurred. That's why I want to see a discussion of the broader issue of how something like escape analysis is supposed to interact with facilities like JVM TI. You've flagged a couple of conditions under which EA should be disabled, but are they sufficient? Give they are both dynamic facilities what is supposed to happen if you keep switching things on and off? Does enabling can_get_owned_monitor_info require previous EA actions to be undone? I don't know the answers here. I certainly hope these questions have been considered when EA was being developed - in which case I'd like to see the discussion. Otherwise this is a conversation that needs to be had, and the development of an overall approach for dealing with this, not trying to address things piece-meal as someone recognises a new problem. > There has to be a way to debug an app without optimizations getting into your way. I like to debug > my C/C++ program after I compiled it -O0 -g. There has to be something equivalent for Java > debugging. And I don't mean -Xint. Since 1.4.1 we've got "Full Speed Debugging" [1]. > > So far JVMTI does an excellent job of making optimizations transparent to the user. If it didn't, > I would like to file a JEP for an interface that shows the pure JVM state (at full speed of > course). > > > > > You seem to have completely missed my point. If the object is local and > > > > is synchronized upon then the synchronization can be elided (and should > > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not > > > > escape. If the synchronization cannot be elided then the object cannot > > > > be considered local. That is how Escape Analysis should be operating > > > > here IMHO. > > > > > > I presume we agree that it is the state of the abstract virtual machine that must be observed > > > through JVMTI, right? > > > > > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. > > > > > > The JIT can still elide synchronization based on a prove that it is actually redundant for the > > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the > > > abstract virtual machine. > > > > I don't agree. If the locking can be elided then it is completely > > elided. If the state of the "abstract VM" had to be perfectly preserved > > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > > > "Optimizing transformations of a program can be designed that reduce the > > number of objects that are reachable to be less than those which would > > naively be considered reachable." > > I'd like to write a few words about the relationship of specification and optimizations: It's the purpose > of a specification to define an abstract model. Optimizations are techniques employed by an > implementation to reduce resource consumption for better performance within the bounds set by the > spec, meaning optimizations have to be transparent to the program. You can't write a program that > detects an optimization. If you could then the implementation would violate rules given by the > specification. (And I don't consider a benchmark a prove.) > > Very rarely you'll find specs mention optimizations. You just don't want to mix specification and > implementation! If they do, they trade beauty and simplicity of an abstract model, ideally based > on formal definitions, for performance. > > They do mention an optimization, if it is impossible to hide it. In the case of JLS 12.6 it is > possible to detect that an object reference is removed from a local variable by means of > finalizations and java.lang.ref. Implementors wanted it so badly, though, that they changed the > spec. Specs are imperfect and always evolving. Sometimes the consequences of specific optimizations may not be realised at the spec level yet need to be accounted for. That may or may not be the case with EA. I find it hard to see how to do lock elision yet at the same time track when the lock would have been taken, without negating the whole purpose of eliding the lock. Afterall if the object is thread-local then the locking would amount to only biased-locking and it could never be contended. So it doesn't seem like there was much to elide in the first place. Lock coarsening is also problematic. > Another example in the future could be tail-call-optimization ... > > > I place lock elision in the same category of "optimising > > transformations" that changes what would "naively" be expected. Now this > > should be explicitly covered in the JLS/JVMS somewhere but I'm having > > trouble finding exactly where. This article discusses lock elision: > > > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > > > and states: > > > > "It stands to reason, then, that if a thread enters a synchronized block > > protected by a lock that no other thread will ever synchronize on, then > > that synchronization has no effect and can therefore be removed by the > > optimizer. (The Java Language Specification explicitly allows this > > optimization.)" > > I doubt you'll find it in the JLS/JVMS. It's an implementation detail permitted by the memory > model. That's what the article is saying. If a Java app has data races, then you would not reason > about removed synchronization. You would argue with the means of the memory model: eg the > synchronization action A1 (JLS 17.4.2 [2]) does not 'synchronized-with' (17.4.4 [3]) sync. action > A2, so they are not ordered. > > > I'll have to ping Brian to see if he recalls exactly where this is > > covered. :) > > Ok :) > Please let me know! Brian conceded it may be more implicit in the JMM as you described above :) Cheers, David > Richard. > > [1] https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/enhancements1.4.html#fsd > [2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.2 > [3] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.4 > > -----Original Message----- > From: David Holmes > Sent: Freitag, 20. September 2019 11:07 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 20/09/2019 6:31 pm, Reingruber, Richard wrote: >> Hi David, >> >> > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >> > > Hi David, >> > > >> > > thanks for looking at this issue. And my appologies for the lengthy mail... >> > > >> > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >> > > > > optimizations like scalar replacement become invalid. >> > > > >> > > > What bothers me about this is that it seems like escape analysis is >> > > > doing something wrong in this case. >> > > >> > > Yes it is. >> > > >> > > > If the object is thread-local but is >> > > > being synchronized upon then either: >> > > >> > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >> > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >> > > machine, as observed through JVMTI, is wrong. See below... >> > >> > You seem to have completely missed my point. If the object is local and >> > is synchronized upon then the synchronization can be elided (and should >> > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> > escape. If the synchronization cannot be elided then the object cannot >> > be considered local. That is how Escape Analysis should be operating >> > here IMHO. >> >> I presume we agree that it is the state of the abstract virtual machine that must be observed >> through JVMTI, right? >> >> The locking state of an object O after a monitorenter on O is locked on the abstract vm. >> >> The JIT can still elide synchronization based on a prove that it is actually redundant for the >> computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >> abstract virtual machine. > > I don't agree. If the locking can be elided then it is completely > elided. If the state of the "abstract VM" had to be perfectly preserved > then we wouldn't be able to do the optimisations referred to in JLS 12.6: > > "Optimizing transformations of a program can be designed that reduce the > number of objects that are reachable to be less than those which would > naively be considered reachable." > > I place lock elision in the same category of "optimising > transformations" that changes what would "naively" be expected. Now this > should be explicitly covered in the JLS/JVMS somewhere but I'm having > trouble finding exactly where. This article discusses lock elision: > > https://www.ibm.com/developerworks/library/j-jtp10185/index.html > > and states: > > "It stands to reason, then, that if a thread enters a synchronized block > protected by a lock that no other thread will ever synchronize on, then > that synchronization has no effect and can therefore be removed by the > optimizer. (The Java Language Specification explicitly allows this > optimization.)" > > I'll have to ping Brian to see if he recalls exactly where this is > covered. :) > > David > ----- > >> >> Cheers, Richard. >> >> -----Original Message----- >> From: David Holmes >> Sent: Freitag, 20. September 2019 00:59 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>> Hi David, >>> >>> thanks for looking at this issue. And my appologies for the lengthy mail... >>> >>> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> > > optimizations like scalar replacement become invalid. >>> > >>> > What bothers me about this is that it seems like escape analysis is >>> > doing something wrong in this case. >>> >>> Yes it is. >>> >>> > If the object is thread-local but is >>> > being synchronized upon then either: >>> >>> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>> machine, as observed through JVMTI, is wrong. See below... >> >> You seem to have completely missed my point. If the object is local and >> is synchronized upon then the synchronization can be elided (and should >> be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> escape. If the synchronization cannot be elided then the object cannot >> be considered local. That is how Escape Analysis should be operating >> here IMHO. >> >> Cheers, >> David >> ----- >> >>> > a) the synchronization is elided and so the object will not appear in >>> > the set of owned monitors; or >>> > b) the fact synchronization occurs renders the object ineligible to be >>> > considered thread-local, and so there is no problem with it appearing in >>> > the set of owned monitors >>> > >>> > I think there is a bigger design philosophy issue here about the >>> > interaction of escape analysis and debugging/management frameworks in >>> > general. I'd like to see a very clear description on exactly how they >>> > should interact. >>> > >>> >>> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >>> state of the virtual machine according to the spec. I think it fails to do so in this case. >>> >>> Please look again at the test: >>> >>> 172 public long dontinline_endlessLoop() { >>> 173 long cs = checkSum; >>> 174 while (doLoop && loopCount-- > 0) { >>> 175 targetIsInLoop = true; >>> 176 checkSum += checkSum % ++cs; >>> 177 } >>> 178 loopCount = 3; >>> 179 targetIsInLoop = false; >>> 180 return checkSum; >>> 181 } >>> >>> 249 public void dontinline_testMethod() { >>> 250 LockCls l1 = new LockCls(); // to be scalar replaced >>> 251 synchronized (l1) { >>> 252 inlinedTestMethodWithNestedLocking(l1); >>> 253 } >>> 254 } >>> 255 >>> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >>> 257 synchronized (l1) { // nested >>> 258 dontinline_endlessLoop(); >>> 259 } >>> 260 } >>> >>> This is the stack when the agent calls GetOwnedMonitorInfo() >>> >>> dontinline_endlessLoop() at line 176 >>> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >>> dontinline_testMethod() at line 252 // compiled frame >>> >>> The state of the _virtual_ machine at that point is obvious: >>> >>> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >>> - That instance was locked by a monitorenter bytecode at line 251 >>> >>> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >>> pencil (hope you won't make me do this, though ;)) >>> >>> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >>> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >>> not. >>> >>> From a more philosophical point of view compiled code is free to change the state of the physical >>> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >>> and every machine instruction. But it must reach points in its execution trace, where it is actually >>> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >>> points are called safepoints. >>> >>> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >>> state is presented. EA does not take into account that the lock state can be observed using >>> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >>> can_get_owned_monitor_info is taken. With the fix the test passes. >>> >>> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >>> the JVMTI implementation cannot handout references to objects stored in local variables if they were >>> scalar replaced. >>> >>> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >>> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >>> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >>> and relock_objects() in class Deoptimization) >>> >>> Thanks and cheers, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Donnerstag, 19. September 2019 02:43 >>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>> >>> Hi Richard, >>> >>> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>>> Hi, >>>> >>>> could I please get reviews for >>>> >>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>>> >>>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>> optimizations like scalar replacement become invalid. >>> >>> What bothers me about this is that it seems like escape analysis is >>> doing something wrong in this case. If the object is thread-local but is >>> being synchronized upon then either: >>> a) the synchronization is elided and so the object will not appear in >>> the set of owned monitors; or >>> b) the fact synchronization occurs renders the object ineligible to be >>> considered thread-local, and so there is no problem with it appearing in >>> the set of owned monitors >>> >>> I think there is a bigger design philosophy issue here about the >>> interaction of escape analysis and debugging/management frameworks in >>> general. I'd like to see a very clear description on exactly how they >>> should interact. >>> >>> Cheers, >>> David >>> >>>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>>> >>>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>>> objects escape through JVMTI. >>>> >>>> I've run tier1 tests. >>>> >>>> Thanks, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>> From martinrb at google.com Mon Sep 23 17:43:18 2019 From: martinrb at google.com (Martin Buchholz) Date: Mon, 23 Sep 2019 10:43:18 -0700 Subject: RFR: jsr166 integration 2019-09 In-Reply-To: References: <517197737.1773850.1568278129180.JavaMail.zimbra@u-pem.fr> <738837959.2535658.1568387411851.JavaMail.zimbra@u-pem.fr> <16d7944e-4b61-7ed9-aeea-fca44e970016@cs.oswego.edu> <1150861893.2564720.1568393036106.JavaMail.zimbra@u-pem.fr> <1a4c7ebd-78bd-de5b-7053-cff63f13bdeb@oracle.com> Message-ID: We now have a fix-up integration that removes all the previously excluded tests from their exclude lists. https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/overview.html 8231031: runtime/ReservedStack/ReservedStackTest.java fails after jsr166 refresh https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/ReservedStackTest/index.html https://bugs.openjdk.java.net/browse/JDK-8231031 LockInfo objects now restored to their previous values (although David was looking at future improvements). 8231032: ThreadMXBean locking tests fail after JSR 166 refresh https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/wrong-blocker/index.html https://bugs.openjdk.java.net/browse/JDK-8231032 8231036: vmTestbase monitoring tests fail after JSR 166 refresh https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/SynchronizerLockingThreads/index.html https://bugs.openjdk.java.net/browse/JDK-8231036 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.menkov at oracle.com Mon Sep 23 18:11:26 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Mon, 23 Sep 2019 11:11:26 -0700 Subject: RFR(S): 8228625: [TESTBUG] sun/tools/jhsdb/JShellHeapDumpTest.java fails with RuntimeException 'JShellToolProvider' missing from stdout/stderr In-Reply-To: References: <4ea2ef0c-0530-41bf-683c-ca57def47beb@oracle.com> <51dd8f1e-1c3b-7688-53f4-9e1ccd5748fa@oracle.com> Message-ID: Looks good. --alex On 09/19/2019 22:31, Chris Plummer wrote: > Here's a new webrev: > > http://cr.openjdk.java.net/~cjplummer/8228625/webrev.01/ > > I decided to add a 2 second delay to make sure the test always passes > and avoids the issues in JDK-8230872. After some internal discussion > there seems to be some consensus that SA is probably not very stable > when attaching to an active process, and I'd rather not have those > issues disrupting this test. I filed a CR to write a new test without > the 2 second delay so we can still reproduce these issues. See: > > https://bugs.openjdk.java.net/browse/JDK-8231288 > > Besides now using ProcessTools.startProcess() with a line predicate > instead of Runtime.exec(), I also did a bit of cleaning up of variable > names and added a few more diagnostic printlns. > > thanks, > > Chris > > On 9/19/19 2:40 PM, Chris Plummer wrote: >> One thing I didn't mention before is that using "jshell> " as the line >> predicate does not work because jshell does not produce a \n after >> generating this prompt, thus it's not actually a line and no attempt >> will be made to match it, so eventually it times out. I fixed this by >> using a snippet from one of the previous 2 lines" >> >> ? Welcome to JShell -- Version 14-internal >> ? For an introduction type: /help intro >> >> I now search for "Welcome to JShell", which is working. >> >> I think this is related to additional issues I am having with this >> fix. I used to see JDK-8230872 happen about 1% of the time after the >> original fix I had for this CR (8228625). However, now it fails about >> 9% of the time. It seems that this new fix is making it more likely >> that jshell will be in a state that will uncover jmap bugs. If I add >> another 10 second delay to the test, all the jmap problems go away. >> >> This started to get me thinking that maybe "jmap --pid" is just not >> reliable. We do have a few other tests for this, but I believe they >> all wait until the target process has all threads blocked before >> issuing the jmap command. This might be why they don't see problems. >> In the case of this test, since I don't fully wait for the jshell> >> prompt, there is still probably some jshell activity when jmap >> attaches. Waiting an extra 10 seconds gets us past this activity, and >> likely jshell is blocked waiting for input. In fact even just waiting >> 1 additional second seems to be long enough. >> >> Chris >> >> On 9/18/19 9:44 PM, Chris Plummer wrote: >>> I got this to work, although it increased the test time from about >>> 30s to over 3m. I looked into it a bit and it appears to be due to >>> the size of the generate hprof file. It used to be about 300k, but >>> now is nearly 7mb. I guess that's because jshell has been run for >>> longer and probably allocated more data. >>> >>> I need to do some more testing and a bit of cleanup. I'll get another >>> review out tomorrow. >>> >>> Chris >>> >>> Suddenly the test time is taking >>> >>> On 9/18/19 5:43 PM, Chris Plummer wrote: >>>> Ok. It was a bit unclear to me why the author went with >>>> Runtime.exec() in the first place. I'll try ProcessTools. That will >>>> probably however, hide 8230872. I might need to write another test >>>> for it. >>>> >>>> Chris >>>> >>>> On 9/18/19 4:29 PM, Alex Menkov wrote: >>>>> You can use jdk.test.lib classes to simplify the things. >>>>> Something like >>>>> >>>>> ProcessBuilder pb = new >>>>> ProcessBuilder(JDKToolFinder.getTestJDKTool("jshell")); >>>>> Process p = ProcessTools.startProcess("JShell", pb, >>>>> ??? s -> {? // warm-up predicate >>>>> ??????? return s.contains(">jshell"); >>>>> ??? }); >>>>> >>>>> --alex >>>>> >>>>> On 09/18/2019 15:44, Chris Plummer wrote: >>>>>> Is there an easy way of doing this? Currently the jshell process >>>>>> is just spawned using Runtime.exec(). >>>>>> >>>>>> Chris >>>>>> >>>>>> On 9/18/19 3:01 PM, Alex Menkov wrote: >>>>>>> Hi Chris, >>>>>>> >>>>>>> Did you think about waiting for jshell prompt ("jshell>") before >>>>>>> run "jhsdb jmap" command instead of delay or re-tries? >>>>>>> >>>>>>> --alex >>>>>>> >>>>>>> On 09/18/2019 14:11, Chris Plummer wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> Please review the following changes: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~cjplummer/8228625/webrev.00/ >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8228625 >>>>>>>> >>>>>>>> There are actually numerous ways that JShellHeapDumpTest.java >>>>>>>> fails. One is a test bug, being addressed here, and the rest all >>>>>>>> seem to be SA bugs. Those are now being covered by JDK-8230872. >>>>>>>> All the issues seem to stem from the fact that the test spawns a >>>>>>>> jshell process, and then immediately does a "jhsdb jmap" on the >>>>>>>> process before jshell has fully started up. >>>>>>>> >>>>>>>> The test bug happens when the jmap succeeds, but jshell has not >>>>>>>> yet entered the main java thread. Thus the search for >>>>>>>> "JShellToolProvider" in the output fails. It expects >>>>>>>> "JShellToolProvider" to be in the output because it is part of a >>>>>>>> method name in the main thread, and the test dump all the thread >>>>>>>> stacks contained in the jmap generated hprof file. When the test >>>>>>>> fails in this way, you can see the stack dump in the output, but >>>>>>>> the main thread is missing. >>>>>>>> >>>>>>>> There's a couple of ways to fix this. One is to just add a delay >>>>>>>> (10s seems to be more than enough), and the other is to retry >>>>>>>> the "jhsdb jmap" command until the stack contains the >>>>>>>> JShellToolProvider symbol. I chose the later because doing a 10s >>>>>>>> delay masks the SA issues that are now covered by JDK-8230872. >>>>>>>> In a way the 10s delay is a better fix, because it makes this >>>>>>>> test pass every time, but I did not like that it also hid real >>>>>>>> SA problems in JDK-8230872. My plan for now is to do this retry >>>>>>>> fix, and then if there are too many failures due to JDK-8230872, >>>>>>>> then also add a 10s delay, with the intention of removing it >>>>>>>> once JDK-8230872 if fixed. From what I can see, JDK-8230872 >>>>>>>> failures happen on about 1% of the runs. >>>>>>>> >>>>>>>> I made a few of other changes. One was to no longer redirect >>>>>>>> stderr from the jmap process as was done from the following: >>>>>>>> >>>>>>>> processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); >>>>>>>> >>>>>>>> This causes the output not to appear in the OutputAnalyzer >>>>>>>> output, resulting in the following not working: >>>>>>>> >>>>>>>> ???????????? output.shouldNotContain("null"); >>>>>>>> >>>>>>>> Also I added code to dump the output of the jshell process so >>>>>>>> you can see if the jshell prompt was ever generated. >>>>>>>> >>>>>>>> thanks, >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >>> >> >> > > From david.holmes at oracle.com Tue Sep 24 00:16:32 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 10:16:32 +1000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: <2fc7826c-9d45-cd74-6b71-a172b0509dba@oracle.com> Hi Richard, On 24/09/2019 1:12 am, Reingruber, Richard wrote: > Hi David, > > The JVMTI documentation answers the question "What is the JVM Tool Interface?" as follows > > "It provides both a way to inspect the state and to control the execution of applications running > in the JavaTM virtual machine (VM)" > > So it's the state on the _virtual_ machine. You're placing a double-meaning on "virtual" there. The JVM is a "Java virtual machine". That doesn't equate to your 'abstract virtual machine' concept. > And then there's another section "Specification Context". I should have found it before, but it is > the 16th section of the introduction. It should be the first, really. > > Specification Context > > Since this interface provides access to the state of applications running in the Java virtual > machine; terminology refers to the Java platform and not the native platform (unless stated > otherwise). For example: > > "thread" means Java programming language thread. > "stack frame" means Java virtual machine stack frame. > "class" means Java programming language class. > "heap" means Java virtual machine heap. > "monitor" means Java programming language object monitor. > > Based on this, in particular "...not the native platform...", I'd think we can finally agree that it > is the state of the abstract virtual machine that's presented to JVMTI clients. I think you're reading far too much into these words that are simply allowing for usage of simple terms, like thread, without having to constantly re-state whether it means "Java programming language thread" or "JVM native thread" or some other kind of thread. I'm not saying this isn't how it should work, I'm saying it needs to be established/confirmed how it should work. > I've answered more of your questions below. Responses below too. > Please let me know, if you think this needs more attention and discussion. If so, I would like > to start a new thread, because this one won't have to many followers anymore ;) I certainly think this needs more attention and discussion - even if it is just the compiler folk confirming "hey this is all fine, we missed a couple of cases ....". That would set my mind at ease. I can see now that not all of the discussion has been cross-posted to the serviceability-dev list. > Is mailing list discussion the right format at all? It's the only medium we have available at present. > Richard. > > > > Hi David, > > > > > > I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide > > > optimizations from its client (Joe the Java developer). > > > > I'm not sure I agree as a blanket statement. It may not be feasible to > > perform certain optimisations and at the same time keep around enough > > information to reconstruct the state as it would have been had the > > optimization not occurred. > > It is feasible. And it is done already at every safepoint in a nmethod to be able to switch from > compiled to interpreted execution. Okay, so perhaps I am imagining scenarios that are actually impossible and things are much simpler in reality.. > > That's why I want to see a discussion of the > > broader issue of how something like escape analysis is supposed to > > interact with facilities like JVM TI. You've flagged a couple of > > conditions under which EA should be disabled, but are they sufficient? > > Well, I've created JDK-8227745. I claim it handles all EA issues. Vladimir Kozlov suggested to > create separate bugs for cases when EA optimizations can be observed. That makes sense to be. And > allows to downport the conservative (i.e. disabling EA) fixes. Okay. > > > Give they are both dynamic facilities what is supposed to happen if you > > keep switching things on and off? Does enabling > > can_get_owned_monitor_info require previous EA actions to be undone? > > can_get_owned_monitor_info is a capability that can be added only in the OnLoad phase, so nothing > needs to be undone. Okay. I see this discussed in the other emails now. > can_tag_objects is dynamic (JDK-8230956), so disabling EA does not help in all > cases. I have elaborated on this in the bug. With JDK-8227745, though, it does not matter how often > the capability is added/removed. The EA base optimizations can be reverted as needed just-in-time. > > > I don't know the answers here. I certainly hope these questions have > > been considered when EA was being developed - in which case I'd like to > > see the discussion. Otherwise this is a conversation that needs to be > > had, and the development of an overall approach for dealing with this, > > not trying to address things piece-meal as someone recognises a new problem. > > Specs are imperfect and always evolving. Sometimes the consequences of > > specific optimizations may not be realised at the spec level yet need to > > be accounted for. That may or may not be the case with EA. > > > > I find it hard to see how to do lock elision yet at the same time track > > when the lock would have been taken, without negating the whole purpose > > of eliding the lock. > > As stated above: this is done for deoptimization already. All tracking information is collected at > compile time. There are no runtime costs, except size of debuginfo. > > JDK-8227745 does what deoptimization does: reallocating and relocking objects. Difference is that > JDK-8227745 does it right before JVMTI access. Deoptimization does it when replacing a compiled > frame by corresponding interpreted frames. I haven't looked at your full proposal in detail as my concerns are not about the code as such but the higher-level model of what is supposed to happen here. Seems I'm the only one who has any concerns about this interaction - but that's fine, I get paid to think about such things. :) Thanks for the discussion. After all that I figure I at least owe you a code review to add to Sergeui's and Dean's. The changes as described look fine to me. I'm a little surprised EA is an all-or-nothing optimisation as I had envisaged something finer-grained that would have simply disabled EA for objects that are actually targets of synchronization. Thanks, David ----- > -----Original Message----- > From: David Holmes > Sent: Montag, 23. September 2019 08:59 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 22/09/2019 7:06 am, Reingruber, Richard wrote: >> Hi David, >> >> I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide >> optimizations from its client (Joe the Java developer). > > I'm not sure I agree as a blanket statement. It may not be feasible to > perform certain optimisations and at the same time keep around enough > information to reconstruct the state as it would have been had the > optimization not occurred. That's why I want to see a discussion of the > broader issue of how something like escape analysis is supposed to > interact with facilities like JVM TI. You've flagged a couple of > conditions under which EA should be disabled, but are they sufficient? > Give they are both dynamic facilities what is supposed to happen if you > keep switching things on and off? Does enabling > can_get_owned_monitor_info require previous EA actions to be undone? > > I don't know the answers here. I certainly hope these questions have > been considered when EA was being developed - in which case I'd like to > see the discussion. Otherwise this is a conversation that needs to be > had, and the development of an overall approach for dealing with this, > not trying to address things piece-meal as someone recognises a new problem. > >> There has to be a way to debug an app without optimizations getting into your way. I like to debug >> my C/C++ program after I compiled it -O0 -g. There has to be something equivalent for Java >> debugging. And I don't mean -Xint. Since 1.4.1 we've got "Full Speed Debugging" [1]. >> >> So far JVMTI does an excellent job of making optimizations transparent to the user. If it didn't, >> I would like to file a JEP for an interface that shows the pure JVM state (at full speed of >> course). >> >> > > > You seem to have completely missed my point. If the object is local and >> > > > is synchronized upon then the synchronization can be elided (and should >> > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> > > > escape. If the synchronization cannot be elided then the object cannot >> > > > be considered local. That is how Escape Analysis should be operating >> > > > here IMHO. >> > > >> > > I presume we agree that it is the state of the abstract virtual machine that must be observed >> > > through JVMTI, right? >> > > >> > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. >> > > >> > > The JIT can still elide synchronization based on a prove that it is actually redundant for the >> > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >> > > abstract virtual machine. >> > >> > I don't agree. If the locking can be elided then it is completely >> > elided. If the state of the "abstract VM" had to be perfectly preserved >> > then we wouldn't be able to do the optimisations referred to in JLS 12.6: >> > >> > "Optimizing transformations of a program can be designed that reduce the >> > number of objects that are reachable to be less than those which would >> > naively be considered reachable." >> >> I'd like to write a few words about the relationship of specification and optimizations: It's the purpose >> of a specification to define an abstract model. Optimizations are techniques employed by an >> implementation to reduce resource consumption for better performance within the bounds set by the >> spec, meaning optimizations have to be transparent to the program. You can't write a program that >> detects an optimization. If you could then the implementation would violate rules given by the >> specification. (And I don't consider a benchmark a prove.) >> >> Very rarely you'll find specs mention optimizations. You just don't want to mix specification and >> implementation! If they do, they trade beauty and simplicity of an abstract model, ideally based >> on formal definitions, for performance. >> >> They do mention an optimization, if it is impossible to hide it. In the case of JLS 12.6 it is >> possible to detect that an object reference is removed from a local variable by means of >> finalizations and java.lang.ref. Implementors wanted it so badly, though, that they changed the >> spec. > > Specs are imperfect and always evolving. Sometimes the consequences of > specific optimizations may not be realised at the spec level yet need to > be accounted for. That may or may not be the case with EA. > > I find it hard to see how to do lock elision yet at the same time track > when the lock would have been taken, without negating the whole purpose > of eliding the lock. Afterall if the object is thread-local then the > locking would amount to only biased-locking and it could never be > contended. So it doesn't seem like there was much to elide in the first > place. > > Lock coarsening is also problematic. > >> Another example in the future could be tail-call-optimization ... >> >> > I place lock elision in the same category of "optimising >> > transformations" that changes what would "naively" be expected. Now this >> > should be explicitly covered in the JLS/JVMS somewhere but I'm having >> > trouble finding exactly where. This article discusses lock elision: >> > >> > https://www.ibm.com/developerworks/library/j-jtp10185/index.html >> > >> > and states: >> > >> > "It stands to reason, then, that if a thread enters a synchronized block >> > protected by a lock that no other thread will ever synchronize on, then >> > that synchronization has no effect and can therefore be removed by the >> > optimizer. (The Java Language Specification explicitly allows this >> > optimization.)" >> >> I doubt you'll find it in the JLS/JVMS. It's an implementation detail permitted by the memory >> model. That's what the article is saying. If a Java app has data races, then you would not reason >> about removed synchronization. You would argue with the means of the memory model: eg the >> synchronization action A1 (JLS 17.4.2 [2]) does not 'synchronized-with' (17.4.4 [3]) sync. action >> A2, so they are not ordered. >> >> > I'll have to ping Brian to see if he recalls exactly where this is >> > covered. :) >> >> Ok :) >> Please let me know! > > Brian conceded it may be more implicit in the JMM as you described above :) > > Cheers, > David > >> Richard. >> >> [1] https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/enhancements1.4.html#fsd >> [2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.2 >> [3] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.4 >> >> -----Original Message----- >> From: David Holmes >> Sent: Freitag, 20. September 2019 11:07 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> Hi Richard, >> >> On 20/09/2019 6:31 pm, Reingruber, Richard wrote: >>> Hi David, >>> >>> > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>> > > Hi David, >>> > > >>> > > thanks for looking at this issue. And my appologies for the lengthy mail... >>> > > >>> > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> > > > > optimizations like scalar replacement become invalid. >>> > > > >>> > > > What bothers me about this is that it seems like escape analysis is >>> > > > doing something wrong in this case. >>> > > >>> > > Yes it is. >>> > > >>> > > > If the object is thread-local but is >>> > > > being synchronized upon then either: >>> > > >>> > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>> > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>> > > machine, as observed through JVMTI, is wrong. See below... >>> > >>> > You seem to have completely missed my point. If the object is local and >>> > is synchronized upon then the synchronization can be elided (and should >>> > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >>> > escape. If the synchronization cannot be elided then the object cannot >>> > be considered local. That is how Escape Analysis should be operating >>> > here IMHO. >>> >>> I presume we agree that it is the state of the abstract virtual machine that must be observed >>> through JVMTI, right? >>> >>> The locking state of an object O after a monitorenter on O is locked on the abstract vm. >>> >>> The JIT can still elide synchronization based on a prove that it is actually redundant for the >>> computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >>> abstract virtual machine. >> >> I don't agree. If the locking can be elided then it is completely >> elided. If the state of the "abstract VM" had to be perfectly preserved >> then we wouldn't be able to do the optimisations referred to in JLS 12.6: >> >> "Optimizing transformations of a program can be designed that reduce the >> number of objects that are reachable to be less than those which would >> naively be considered reachable." >> >> I place lock elision in the same category of "optimising >> transformations" that changes what would "naively" be expected. Now this >> should be explicitly covered in the JLS/JVMS somewhere but I'm having >> trouble finding exactly where. This article discusses lock elision: >> >> https://www.ibm.com/developerworks/library/j-jtp10185/index.html >> >> and states: >> >> "It stands to reason, then, that if a thread enters a synchronized block >> protected by a lock that no other thread will ever synchronize on, then >> that synchronization has no effect and can therefore be removed by the >> optimizer. (The Java Language Specification explicitly allows this >> optimization.)" >> >> I'll have to ping Brian to see if he recalls exactly where this is >> covered. :) >> >> David >> ----- >> >>> >>> Cheers, Richard. >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Freitag, 20. September 2019 00:59 >>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>> >>> On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>>> Hi David, >>>> >>>> thanks for looking at this issue. And my appologies for the lengthy mail... >>>> >>>> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>> > > optimizations like scalar replacement become invalid. >>>> > >>>> > What bothers me about this is that it seems like escape analysis is >>>> > doing something wrong in this case. >>>> >>>> Yes it is. >>>> >>>> > If the object is thread-local but is >>>> > being synchronized upon then either: >>>> >>>> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>>> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>>> machine, as observed through JVMTI, is wrong. See below... >>> >>> You seem to have completely missed my point. If the object is local and >>> is synchronized upon then the synchronization can be elided (and should >>> be) in which case it won't appear in GetOwnedMonitorInfo and so does not >>> escape. If the synchronization cannot be elided then the object cannot >>> be considered local. That is how Escape Analysis should be operating >>> here IMHO. >>> >>> Cheers, >>> David >>> ----- >>> >>>> > a) the synchronization is elided and so the object will not appear in >>>> > the set of owned monitors; or >>>> > b) the fact synchronization occurs renders the object ineligible to be >>>> > considered thread-local, and so there is no problem with it appearing in >>>> > the set of owned monitors >>>> > >>>> > I think there is a bigger design philosophy issue here about the >>>> > interaction of escape analysis and debugging/management frameworks in >>>> > general. I'd like to see a very clear description on exactly how they >>>> > should interact. >>>> > >>>> >>>> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >>>> state of the virtual machine according to the spec. I think it fails to do so in this case. >>>> >>>> Please look again at the test: >>>> >>>> 172 public long dontinline_endlessLoop() { >>>> 173 long cs = checkSum; >>>> 174 while (doLoop && loopCount-- > 0) { >>>> 175 targetIsInLoop = true; >>>> 176 checkSum += checkSum % ++cs; >>>> 177 } >>>> 178 loopCount = 3; >>>> 179 targetIsInLoop = false; >>>> 180 return checkSum; >>>> 181 } >>>> >>>> 249 public void dontinline_testMethod() { >>>> 250 LockCls l1 = new LockCls(); // to be scalar replaced >>>> 251 synchronized (l1) { >>>> 252 inlinedTestMethodWithNestedLocking(l1); >>>> 253 } >>>> 254 } >>>> 255 >>>> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >>>> 257 synchronized (l1) { // nested >>>> 258 dontinline_endlessLoop(); >>>> 259 } >>>> 260 } >>>> >>>> This is the stack when the agent calls GetOwnedMonitorInfo() >>>> >>>> dontinline_endlessLoop() at line 176 >>>> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >>>> dontinline_testMethod() at line 252 // compiled frame >>>> >>>> The state of the _virtual_ machine at that point is obvious: >>>> >>>> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >>>> - That instance was locked by a monitorenter bytecode at line 251 >>>> >>>> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >>>> pencil (hope you won't make me do this, though ;)) >>>> >>>> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >>>> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >>>> not. >>>> >>>> From a more philosophical point of view compiled code is free to change the state of the physical >>>> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >>>> and every machine instruction. But it must reach points in its execution trace, where it is actually >>>> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >>>> points are called safepoints. >>>> >>>> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >>>> state is presented. EA does not take into account that the lock state can be observed using >>>> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >>>> can_get_owned_monitor_info is taken. With the fix the test passes. >>>> >>>> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >>>> the JVMTI implementation cannot handout references to objects stored in local variables if they were >>>> scalar replaced. >>>> >>>> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >>>> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >>>> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >>>> and relock_objects() in class Deoptimization) >>>> >>>> Thanks and cheers, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Donnerstag, 19. September 2019 02:43 >>>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>>> >>>> Hi Richard, >>>> >>>> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>>>> Hi, >>>>> >>>>> could I please get reviews for >>>>> >>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>>>> >>>>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>>> optimizations like scalar replacement become invalid. >>>> >>>> What bothers me about this is that it seems like escape analysis is >>>> doing something wrong in this case. If the object is thread-local but is >>>> being synchronized upon then either: >>>> a) the synchronization is elided and so the object will not appear in >>>> the set of owned monitors; or >>>> b) the fact synchronization occurs renders the object ineligible to be >>>> considered thread-local, and so there is no problem with it appearing in >>>> the set of owned monitors >>>> >>>> I think there is a bigger design philosophy issue here about the >>>> interaction of escape analysis and debugging/management frameworks in >>>> general. I'd like to see a very clear description on exactly how they >>>> should interact. >>>> >>>> Cheers, >>>> David >>>> >>>>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>>>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>>>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>>>> >>>>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>>>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>>>> objects escape through JVMTI. >>>>> >>>>> I've run tier1 tests. >>>>> >>>>> Thanks, Richard. >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>>> From hohensee at amazon.com Tue Sep 24 00:42:52 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 00:42:52 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> Message-ID: <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> Update: Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ All test suites that reference getThreadAllocatedBytes pass. These are hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) jdk/com/sun/management jdk/jdk/jfr/event/runtime Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. Thanks, Paul ?On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: Hi Paul, I have almost the same comments as David: - the same two spots of changes identified - the addition of the default method was expected - the change in test is a surprise (I also doubt, it is really needed) - new CSR is needed Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( Thanks, Serguei On 9/19/19 16:06, David Holmes wrote: > Hi Paul, > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: >> More formally, >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > I'm assuming there are only two changes here: > > 1. The new method is now a default method that throws UOE. > > That seems fine. > > 2. You implemented the new method in the test class. > > I don't understand why you did that. The test can't be calling the new > method. Now that it is a default method we will get past the > compilation failure that caused the problem. So no change to the test > should be needed AFAICS. > > A new CSR request is needed. Just copy everything across from the old, > with the updated spec. But please also mention this is a > PlatformManagedObject in the compatibility discussion. > > Thanks, > David > >> Thanks, >> >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, >> Paul" > hohensee at amazon.com> wrote: >> >> Off by 2 error. Changed the subject to reflect 8231209. >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> Paul >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" >> wrote: >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> The redo bug is 8231209. 8231211 is closed as a dup >> of 8231210. >> Dan >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: >> > I'll have the default method throw UOE. That's the same as >> the other default methods do. >> > >> > The necessary test fix is in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, >> which needs a new getCurrentThreadAllocatedBytes method, defined as >> > >> > public long getCurrentThreadAllocatedBytes() { >> > return (Long) >> invokeMethod("getCurrentThreadAllocatedBytes", >> > new Object[] { }, >> > new String[] { }); >> > } >> > >> > With this fix, the 134 tests in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. >> Preliminary webrev at >> > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to >> the >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes >> set? >> > >> > Paul >> > >> > On 9/18/19, 8:16 PM, "David Holmes" >> wrote: >> > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: >> > >> They all implement >> com.sun.management.ThreadMXBean, so adding a >> > >> getCurrentThreadAllocatedBytes broke them. >> Potential fix is to give it >> > >> a default implementation, vis >> > >> >> > >> public default long >> getCurrentThreadAllocatedBytes() { >> > >> return -1; >> > >> } >> > >> >> > > >> > > com.sun.management.ThreadMXBean (and other platform >> MXBeans) is a >> > > "sealed" interface which should only be implemented >> by JDK. >> > >> > Didn't realize that. I don't recall knowing about >> PlatformManagedObject. >> > Sealed types will at least allow this to be enforced, >> though I have to >> > wonder what the tests are doing here. >> > >> > > Unfortunately we don't have the sealed type feature >> yet. Yes it needs >> > > to be a default method. I think it should throw UOE. >> > > >> > > * @implSpec >> > > * The default implementation throws {@code >> > > UnsupportedOperationException}. >> > > >> > > The @throw UOE can make it clear that it does not >> support current thread >> > > memory allocation measurement. >> > >> > Yes that seems a reasonable default if we don't want >> this to be >> > implemented outside the platform. >> > >> > Thanks, >> > David >> > >> > > Mandy >> > >> > >> From david.holmes at oracle.com Tue Sep 24 02:09:54 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 12:09:54 +1000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> Message-ID: <9bc52e50-8db4-bc53-2bbb-08ff0e071bf3@oracle.com> Hi Paul, That all seems fine to me. CSR reviewed as well. Thanks, David ----- On 24/09/2019 10:42 am, Hohensee, Paul wrote: > Update: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > All test suites that reference getThreadAllocatedBytes pass. These are > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > jdk/com/sun/management > jdk/jdk/jfr/event/runtime > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > Thanks, > > Paul > > ?On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > Hi Paul, > > I have almost the same comments as David: > - the same two spots of changes identified > - the addition of the default method was expected > - the change in test is a surprise (I also doubt, it is really needed) > - new CSR is needed > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > Thanks, > Serguei > > > On 9/19/19 16:06, David Holmes wrote: > > Hi Paul, > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > >> More formally, > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > I'm assuming there are only two changes here: > > > > 1. The new method is now a default method that throws UOE. > > > > That seems fine. > > > > 2. You implemented the new method in the test class. > > > > I don't understand why you did that. The test can't be calling the new > > method. Now that it is a default method we will get past the > > compilation failure that caused the problem. So no change to the test > > should be needed AFAICS. > > > > A new CSR request is needed. Just copy everything across from the old, > > with the updated spec. But please also mention this is a > > PlatformManagedObject in the compatibility discussion. > > > > Thanks, > > David > > > >> Thanks, > >> > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > >> Paul" >> hohensee at amazon.com> wrote: > >> > >> Off by 2 error. Changed the subject to reflect 8231209. > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > >> Paul > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > >> wrote: > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> The redo bug is 8231209. 8231211 is closed as a dup > >> of 8231210. > >> Dan > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > >> > I'll have the default method throw UOE. That's the same as > >> the other default methods do. > >> > > >> > The necessary test fix is in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > >> > > >> > public long getCurrentThreadAllocatedBytes() { > >> > return (Long) > >> invokeMethod("getCurrentThreadAllocatedBytes", > >> > new Object[] { }, > >> > new String[] { }); > >> > } > >> > > >> > With this fix, the 134 tests in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > >> Preliminary webrev at > >> > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > >> the > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > >> set? > >> > > >> > Paul > >> > > >> > On 9/18/19, 8:16 PM, "David Holmes" > >> wrote: > >> > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> > >> They all implement > >> com.sun.management.ThreadMXBean, so adding a > >> > >> getCurrentThreadAllocatedBytes broke them. > >> Potential fix is to give it > >> > >> a default implementation, vis > >> > >> > >> > >> public default long > >> getCurrentThreadAllocatedBytes() { > >> > >> return -1; > >> > >> } > >> > >> > >> > > > >> > > com.sun.management.ThreadMXBean (and other platform > >> MXBeans) is a > >> > > "sealed" interface which should only be implemented > >> by JDK. > >> > > >> > Didn't realize that. I don't recall knowing about > >> PlatformManagedObject. > >> > Sealed types will at least allow this to be enforced, > >> though I have to > >> > wonder what the tests are doing here. > >> > > >> > > Unfortunately we don't have the sealed type feature > >> yet. Yes it needs > >> > > to be a default method. I think it should throw UOE. > >> > > > >> > > * @implSpec > >> > > * The default implementation throws {@code > >> > > UnsupportedOperationException}. > >> > > > >> > > The @throw UOE can make it clear that it does not > >> support current thread > >> > > memory allocation measurement. > >> > > >> > Yes that seems a reasonable default if we don't want > >> this to be > >> > implemented outside the platform. > >> > > >> > Thanks, > >> > David > >> > > >> > > Mandy > >> > > >> > > >> > > > From david.holmes at oracle.com Tue Sep 24 02:20:47 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 12:20:47 +1000 Subject: RFR: jsr166 integration 2019-09 In-Reply-To: References: <517197737.1773850.1568278129180.JavaMail.zimbra@u-pem.fr> <738837959.2535658.1568387411851.JavaMail.zimbra@u-pem.fr> <16d7944e-4b61-7ed9-aeea-fca44e970016@cs.oswego.edu> <1150861893.2564720.1568393036106.JavaMail.zimbra@u-pem.fr> <1a4c7ebd-78bd-de5b-7053-cff63f13bdeb@oracle.com> Message-ID: <86366d20-1ae0-6694-4b41-ce21dfb8ddbb@oracle.com> Hi Martin, That all seems fine to me. Thanks, David On 24/09/2019 3:43 am, Martin Buchholz wrote: > We now have a fix-up integration that removes all the previously > excluded tests from their exclude lists. > > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/overview.html > > 8231031: runtime/ReservedStack/ReservedStackTest.java fails after jsr166 > refresh > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/ReservedStackTest/index.html > https://bugs.openjdk.java.net/browse/JDK-8231031 > > LockInfo objects now restored to their previous values (although David > was looking at future improvements). > > 8231032: ThreadMXBean locking tests fail after JSR 166 refresh > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/wrong-blocker/index.html > https://bugs.openjdk.java.net/browse/JDK-8231032 > > 8231036: vmTestbase monitoring tests fail after JSR 166 refresh > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/SynchronizerLockingThreads/index.html > https://bugs.openjdk.java.net/browse/JDK-8231036 From daniil.x.titov at oracle.com Tue Sep 24 02:54:44 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Mon, 23 Sep 2019 19:54:44 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> Message-ID: <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> Hi Paul, I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? Thank you, --Daniil ?On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: Update: Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ All test suites that reference getThreadAllocatedBytes pass. These are hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) jdk/com/sun/management jdk/jdk/jfr/event/runtime Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. Thanks, Paul ?On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: Hi Paul, I have almost the same comments as David: - the same two spots of changes identified - the addition of the default method was expected - the change in test is a surprise (I also doubt, it is really needed) - new CSR is needed Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( Thanks, Serguei On 9/19/19 16:06, David Holmes wrote: > Hi Paul, > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: >> More formally, >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > I'm assuming there are only two changes here: > > 1. The new method is now a default method that throws UOE. > > That seems fine. > > 2. You implemented the new method in the test class. > > I don't understand why you did that. The test can't be calling the new > method. Now that it is a default method we will get past the > compilation failure that caused the problem. So no change to the test > should be needed AFAICS. > > A new CSR request is needed. Just copy everything across from the old, > with the updated spec. But please also mention this is a > PlatformManagedObject in the compatibility discussion. > > Thanks, > David > >> Thanks, >> >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, >> Paul" > hohensee at amazon.com> wrote: >> >> Off by 2 error. Changed the subject to reflect 8231209. >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> Paul >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" >> wrote: >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> The redo bug is 8231209. 8231211 is closed as a dup >> of 8231210. >> Dan >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: >> > I'll have the default method throw UOE. That's the same as >> the other default methods do. >> > >> > The necessary test fix is in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, >> which needs a new getCurrentThreadAllocatedBytes method, defined as >> > >> > public long getCurrentThreadAllocatedBytes() { >> > return (Long) >> invokeMethod("getCurrentThreadAllocatedBytes", >> > new Object[] { }, >> > new String[] { }); >> > } >> > >> > With this fix, the 134 tests in >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. >> Preliminary webrev at >> > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to >> the >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes >> set? >> > >> > Paul >> > >> > On 9/18/19, 8:16 PM, "David Holmes" >> wrote: >> > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: >> > >> They all implement >> com.sun.management.ThreadMXBean, so adding a >> > >> getCurrentThreadAllocatedBytes broke them. >> Potential fix is to give it >> > >> a default implementation, vis >> > >> >> > >> public default long >> getCurrentThreadAllocatedBytes() { >> > >> return -1; >> > >> } >> > >> >> > > >> > > com.sun.management.ThreadMXBean (and other platform >> MXBeans) is a >> > > "sealed" interface which should only be implemented >> by JDK. >> > >> > Didn't realize that. I don't recall knowing about >> PlatformManagedObject. >> > Sealed types will at least allow this to be enforced, >> though I have to >> > wonder what the tests are doing here. >> > >> > > Unfortunately we don't have the sealed type feature >> yet. Yes it needs >> > > to be a default method. I think it should throw UOE. >> > > >> > > * @implSpec >> > > * The default implementation throws {@code >> > > UnsupportedOperationException}. >> > > >> > > The @throw UOE can make it clear that it does not >> support current thread >> > > memory allocation measurement. >> > >> > Yes that seems a reasonable default if we don't want >> this to be >> > implemented outside the platform. >> > >> > Thanks, >> > David >> > >> > > Mandy >> > >> > >> From mandy.chung at oracle.com Tue Sep 24 03:28:49 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 23 Sep 2019 20:28:49 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> Message-ID: Looks fine. I added myself to the CSR reviewer.? I don't categorize it as source incompatibility as other implementation outside JDK is not supported.? I took the liberty to edit the compatibility kind. Mandy On 9/23/19 5:42 PM, Hohensee, Paul wrote: > Update: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > All test suites that reference getThreadAllocatedBytes pass. These are > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > jdk/com/sun/management > jdk/jdk/jfr/event/runtime > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > Thanks, > > Paul > > ?On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > Hi Paul, > > I have almost the same comments as David: > - the same two spots of changes identified > - the addition of the default method was expected > - the change in test is a surprise (I also doubt, it is really needed) > - new CSR is needed > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > Thanks, > Serguei > > > On 9/19/19 16:06, David Holmes wrote: > > Hi Paul, > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > >> More formally, > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > I'm assuming there are only two changes here: > > > > 1. The new method is now a default method that throws UOE. > > > > That seems fine. > > > > 2. You implemented the new method in the test class. > > > > I don't understand why you did that. The test can't be calling the new > > method. Now that it is a default method we will get past the > > compilation failure that caused the problem. So no change to the test > > should be needed AFAICS. > > > > A new CSR request is needed. Just copy everything across from the old, > > with the updated spec. But please also mention this is a > > PlatformManagedObject in the compatibility discussion. > > > > Thanks, > > David > > > >> Thanks, > >> > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > >> Paul" >> hohensee at amazon.com> wrote: > >> > >> Off by 2 error. Changed the subject to reflect 8231209. > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > >> Paul > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > >> wrote: > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> The redo bug is 8231209. 8231211 is closed as a dup > >> of 8231210. > >> Dan > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > >> > I'll have the default method throw UOE. That's the same as > >> the other default methods do. > >> > > >> > The necessary test fix is in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > >> > > >> > public long getCurrentThreadAllocatedBytes() { > >> > return (Long) > >> invokeMethod("getCurrentThreadAllocatedBytes", > >> > new Object[] { }, > >> > new String[] { }); > >> > } > >> > > >> > With this fix, the 134 tests in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > >> Preliminary webrev at > >> > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > >> the > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > >> set? > >> > > >> > Paul > >> > > >> > On 9/18/19, 8:16 PM, "David Holmes" > >> wrote: > >> > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> > >> They all implement > >> com.sun.management.ThreadMXBean, so adding a > >> > >> getCurrentThreadAllocatedBytes broke them. > >> Potential fix is to give it > >> > >> a default implementation, vis > >> > >> > >> > >> public default long > >> getCurrentThreadAllocatedBytes() { > >> > >> return -1; > >> > >> } > >> > >> > >> > > > >> > > com.sun.management.ThreadMXBean (and other platform > >> MXBeans) is a > >> > > "sealed" interface which should only be implemented > >> by JDK. > >> > > >> > Didn't realize that. I don't recall knowing about > >> PlatformManagedObject. > >> > Sealed types will at least allow this to be enforced, > >> though I have to > >> > wonder what the tests are doing here. > >> > > >> > > Unfortunately we don't have the sealed type feature > >> yet. Yes it needs > >> > > to be a default method. I think it should throw UOE. > >> > > > >> > > * @implSpec > >> > > * The default implementation throws {@code > >> > > UnsupportedOperationException}. > >> > > > >> > > The @throw UOE can make it clear that it does not > >> support current thread > >> > > memory allocation measurement. > >> > > >> > Yes that seems a reasonable default if we don't want > >> this to be > >> > implemented outside the platform. > >> > > >> > Thanks, > >> > David > >> > > >> > > Mandy > >> > > >> > > >> > > > From mandy.chung at oracle.com Tue Sep 24 03:39:41 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 23 Sep 2019 20:39:41 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> Message-ID: <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> Good question. When HS express (mix-n-matched JDK and HS version) was supported, the JMM_VERSION was rev'ed to enable the version checking.? HS express is no longer supported.? JDK is supported to run with this version of HotSpot VM.? OTOH, this adds a new function in the middle of the function table.? I think it's a good convention to follow and bump the version number. Mandy On 9/23/19 7:54 PM, Daniil Titov wrote: > Hi Paul, > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > Thank you, > --Daniil > > ?On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > Update: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > All test suites that reference getThreadAllocatedBytes pass. These are > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > jdk/com/sun/management > jdk/jdk/jfr/event/runtime > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > Thanks, > > Paul > > ?On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > Hi Paul, > > I have almost the same comments as David: > - the same two spots of changes identified > - the addition of the default method was expected > - the change in test is a surprise (I also doubt, it is really needed) > - new CSR is needed > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > Thanks, > Serguei > > > On 9/19/19 16:06, David Holmes wrote: > > Hi Paul, > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > >> More formally, > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > I'm assuming there are only two changes here: > > > > 1. The new method is now a default method that throws UOE. > > > > That seems fine. > > > > 2. You implemented the new method in the test class. > > > > I don't understand why you did that. The test can't be calling the new > > method. Now that it is a default method we will get past the > > compilation failure that caused the problem. So no change to the test > > should be needed AFAICS. > > > > A new CSR request is needed. Just copy everything across from the old, > > with the updated spec. But please also mention this is a > > PlatformManagedObject in the compatibility discussion. > > > > Thanks, > > David > > > >> Thanks, > >> > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > >> Paul" >> hohensee at amazon.com> wrote: > >> > >> Off by 2 error. Changed the subject to reflect 8231209. > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > >> Paul > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > >> wrote: > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> The redo bug is 8231209. 8231211 is closed as a dup > >> of 8231210. > >> Dan > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > >> > I'll have the default method throw UOE. That's the same as > >> the other default methods do. > >> > > >> > The necessary test fix is in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > >> > > >> > public long getCurrentThreadAllocatedBytes() { > >> > return (Long) > >> invokeMethod("getCurrentThreadAllocatedBytes", > >> > new Object[] { }, > >> > new String[] { }); > >> > } > >> > > >> > With this fix, the 134 tests in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > >> Preliminary webrev at > >> > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > >> the > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > >> set? > >> > > >> > Paul > >> > > >> > On 9/18/19, 8:16 PM, "David Holmes" > >> wrote: > >> > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> > >> They all implement > >> com.sun.management.ThreadMXBean, so adding a > >> > >> getCurrentThreadAllocatedBytes broke them. > >> Potential fix is to give it > >> > >> a default implementation, vis > >> > >> > >> > >> public default long > >> getCurrentThreadAllocatedBytes() { > >> > >> return -1; > >> > >> } > >> > >> > >> > > > >> > > com.sun.management.ThreadMXBean (and other platform > >> MXBeans) is a > >> > > "sealed" interface which should only be implemented > >> by JDK. > >> > > >> > Didn't realize that. I don't recall knowing about > >> PlatformManagedObject. > >> > Sealed types will at least allow this to be enforced, > >> though I have to > >> > wonder what the tests are doing here. > >> > > >> > > Unfortunately we don't have the sealed type feature > >> yet. Yes it needs > >> > > to be a default method. I think it should throw UOE. > >> > > > >> > > * @implSpec > >> > > * The default implementation throws {@code > >> > > UnsupportedOperationException}. > >> > > > >> > > The @throw UOE can make it clear that it does not > >> support current thread > >> > > memory allocation measurement. > >> > > >> > Yes that seems a reasonable default if we don't want > >> this to be > >> > implemented outside the platform. > >> > > >> > Thanks, > >> > David > >> > > >> > > Mandy > >> > > >> > > >> > > > > > > > From david.holmes at oracle.com Tue Sep 24 05:09:24 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 15:09:24 +1000 Subject: RFR: 8231289: Disentangle JvmtiRawMonitor from ObjectMonitor and clean it up Message-ID: <40b43578-c5e6-4b1b-f94d-69a303cfeb51@oracle.com> Bug: https://bugs.openjdk.java.net/browse/JDK-8231289 webrev: http://cr.openjdk.java.net/~dholmes/8231289/webrev/ The earlier attempt to rewrite JvmtiRawMonitor as a simple wrapper around PlatformMonitor proved not so simple and ultimately had too many issues due to the need to support Thread.interrupt. I'd previously stated in the bug report: "In the worst-case I suppose we could just copy ObjectMonitor to a new class and have JvmtiRawMonitor continue to extend that (with some additional minor adjustments) - or even just inline it all as needed." but hadn't looked at it in detail. Richard Reingruber did look at it and pointed out that it is actually quite simple - we barely use any actual code from ObjectMonitor, mainly just the state. So thanks Richard! :) So this change basically copies or moves anything needed by JvmtiRawMonitor from ObjectMonitor, breaking the connection between the two. We also copy and simplify ObjectWaiter, turning it into a QNode internal class. There is then a lot of cleanup that was applied (and a lot more that could still be done): - Removed the never implemented/used PROPER_TRANSITIONS ifdefs - Fixed the disconnect between the types of non-JavaThreads expected by the upper layer code and lower layer code - cleaned up and simplified return codes - consolidated code that is identical for JavaThreads and non-JavaThreads (e.g. notify/notifyAll). - removed used of TRAPS/THREAD where not appropriate and replaced with "Thread * Self" in the style of the rest of the code - changed recursions to be int rather than intptr_t (a "fixme" in the ObjectMonitor code) I have not changed the many style flaws with this code: - Capitalized names - extra spaces before ; - ... but could do so if needed. I wanted to try and keep it more obvious that the fundamental functional code is actually unmodified. There is one aspect that requires further explanation: the notion of current pending monitor. The "current pending monitor" is stored in the Thread and used by a number of introspection APIs for things like finding monitors, doing deadlock detection, etc. The JvmtiRawMonitor code would also set/clear itself as "current pending monitor". Most uses of the current pending monitor actually, explicitly or implicitly, ignore the case when the monitor is a JvmtiRawMonitor (observed by the fact the mon->object() query returns NULL). The exception to that is deadlock detection where raw monitors are at least partially accounted for. To preserve that I added the notion of "current pending raw monitor" and updated the deadlock detection code to use that. The test: test/hotspot/jtreg/vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/rawmnwait005.cpp was updated because I'd noticed previously that it was the only test that used interrupt with raw monitors, but was in fact broken: the test thread is a daemon thread so the main thread could terminate the VM immediately after the interrupt() call, thus you would never know if the interruption actually worked as expected. Testing: - tiers 1 - 3 - vmTestbase/nsk/monitoring/ (for deadlock detection**) - vmTestbase/nsk/jdwp - vmTestbase/nsk/jdb/ - vmTestbase/nsk/jdi/ - vmTestbase/nsk/jvmti/ - serviceability/jvmti/ - serviceability/jdwp - JDK: java/lang/management ** There are no existing deadlock related tests involving JvmtiRawMonitor. It would be interesting/useful to add them to the existing nsk/monitoring tests that cover synchronized and JNI locking. But it's a non-trivial enhancement that I don't really have time to do. Thanks, David ----- From david.holmes at oracle.com Tue Sep 24 06:06:06 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 16:06:06 +1000 Subject: RFR: jsr166 integration 2019-09 In-Reply-To: <86366d20-1ae0-6694-4b41-ce21dfb8ddbb@oracle.com> References: <517197737.1773850.1568278129180.JavaMail.zimbra@u-pem.fr> <738837959.2535658.1568387411851.JavaMail.zimbra@u-pem.fr> <16d7944e-4b61-7ed9-aeea-fca44e970016@cs.oswego.edu> <1150861893.2564720.1568393036106.JavaMail.zimbra@u-pem.fr> <1a4c7ebd-78bd-de5b-7053-cff63f13bdeb@oracle.com> <86366d20-1ae0-6694-4b41-ce21dfb8ddbb@oracle.com> Message-ID: Except when I run it through our test system ReservedStackTest is still failing :( I tested it initially when Fred proposed it and that went fine. It also passes for me locally on Linux. David On 24/09/2019 12:20 pm, David Holmes wrote: > Hi Martin, > > That all seems fine to me. > > Thanks, > David > > On 24/09/2019 3:43 am, Martin Buchholz wrote: >> We now have a fix-up integration that removes all the previously >> excluded tests from their exclude lists. >> >> https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/overview.html >> >> >> 8231031: runtime/ReservedStack/ReservedStackTest.java fails after >> jsr166 refresh >> https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/ReservedStackTest/index.html >> >> https://bugs.openjdk.java.net/browse/JDK-8231031 >> >> LockInfo objects now restored to their previous values (although David >> was looking at future improvements). >> >> 8231032: ThreadMXBean locking tests fail after JSR 166 refresh >> https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/wrong-blocker/index.html >> >> https://bugs.openjdk.java.net/browse/JDK-8231032 >> >> 8231036: vmTestbase monitoring tests fail after JSR 166 refresh >> https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/SynchronizerLockingThreads/index.html >> >> https://bugs.openjdk.java.net/browse/JDK-8231036 From david.holmes at oracle.com Tue Sep 24 06:15:09 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Sep 2019 16:15:09 +1000 Subject: RFR: JDK-8199136: Dead code in src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java In-Reply-To: References: Message-ID: <4fc4d799-2e28-ac1b-beac-415ce0237aff@oracle.com> Hi Evgeny, I just spotted this review request languishing in the mailing list. On 26/08/2019 4:36 am, Evgeny Mandrikov wrote: > Hello! > > Please review patch [1] for JDK-8199136 [2]. Also it needs a sponsor > since I have only author status in OpenJDK Census [3]. I looked at the discussion in the bug report (which I've assigned to you) and looked at the code and agree that these API's are unused and so the code can be cleaned up. The patch will need to be rebased against latest jdk sources, so please do that. I'll see if I can drum up a second reviewer and get a sponsor. Thanks, David > After this change tier1 tests pass on my machine. > > > With best regards, > Evgeny Mandrikov > > [1] http://cr.openjdk.java.net/~godin/8199136/webrev.00/ > [2] https://bugs.openjdk.java.net/browse/JDK-8199136 > [3] https://openjdk.java.net/census#godin > From richard.reingruber at sap.com Tue Sep 24 07:38:39 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Tue, 24 Sep 2019 07:38:39 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: <2fc7826c-9d45-cd74-6b71-a172b0509dba@oracle.com> References: <2fc7826c-9d45-cd74-6b71-a172b0509dba@oracle.com> Message-ID: Hi David, thanks for the discussion so far. I certainly learned from it. I'll start a new thread were I'll ask for comments on the relationship of JVMTI and optimizations, especially those based on EA. And thanks for your Review too. I'll wait until there's some feedback on the new thread. Richard. -----Original Message----- From: David Holmes Sent: Dienstag, 24. September 2019 02:17 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Richard, On 24/09/2019 1:12 am, Reingruber, Richard wrote: > Hi David, > > The JVMTI documentation answers the question "What is the JVM Tool Interface?" as follows > > "It provides both a way to inspect the state and to control the execution of applications running > in the JavaTM virtual machine (VM)" > > So it's the state on the _virtual_ machine. You're placing a double-meaning on "virtual" there. The JVM is a "Java virtual machine". That doesn't equate to your 'abstract virtual machine' concept. > And then there's another section "Specification Context". I should have found it before, but it is > the 16th section of the introduction. It should be the first, really. > > Specification Context > > Since this interface provides access to the state of applications running in the Java virtual > machine; terminology refers to the Java platform and not the native platform (unless stated > otherwise). For example: > > "thread" means Java programming language thread. > "stack frame" means Java virtual machine stack frame. > "class" means Java programming language class. > "heap" means Java virtual machine heap. > "monitor" means Java programming language object monitor. > > Based on this, in particular "...not the native platform...", I'd think we can finally agree that it > is the state of the abstract virtual machine that's presented to JVMTI clients. I think you're reading far too much into these words that are simply allowing for usage of simple terms, like thread, without having to constantly re-state whether it means "Java programming language thread" or "JVM native thread" or some other kind of thread. I'm not saying this isn't how it should work, I'm saying it needs to be established/confirmed how it should work. > I've answered more of your questions below. Responses below too. > Please let me know, if you think this needs more attention and discussion. If so, I would like > to start a new thread, because this one won't have to many followers anymore ;) I certainly think this needs more attention and discussion - even if it is just the compiler folk confirming "hey this is all fine, we missed a couple of cases ....". That would set my mind at ease. I can see now that not all of the discussion has been cross-posted to the serviceability-dev list. > Is mailing list discussion the right format at all? It's the only medium we have available at present. > Richard. > > > > Hi David, > > > > > > I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide > > > optimizations from its client (Joe the Java developer). > > > > I'm not sure I agree as a blanket statement. It may not be feasible to > > perform certain optimisations and at the same time keep around enough > > information to reconstruct the state as it would have been had the > > optimization not occurred. > > It is feasible. And it is done already at every safepoint in a nmethod to be able to switch from > compiled to interpreted execution. Okay, so perhaps I am imagining scenarios that are actually impossible and things are much simpler in reality.. > > That's why I want to see a discussion of the > > broader issue of how something like escape analysis is supposed to > > interact with facilities like JVM TI. You've flagged a couple of > > conditions under which EA should be disabled, but are they sufficient? > > Well, I've created JDK-8227745. I claim it handles all EA issues. Vladimir Kozlov suggested to > create separate bugs for cases when EA optimizations can be observed. That makes sense to be. And > allows to downport the conservative (i.e. disabling EA) fixes. Okay. > > > Give they are both dynamic facilities what is supposed to happen if you > > keep switching things on and off? Does enabling > > can_get_owned_monitor_info require previous EA actions to be undone? > > can_get_owned_monitor_info is a capability that can be added only in the OnLoad phase, so nothing > needs to be undone. Okay. I see this discussed in the other emails now. > can_tag_objects is dynamic (JDK-8230956), so disabling EA does not help in all > cases. I have elaborated on this in the bug. With JDK-8227745, though, it does not matter how often > the capability is added/removed. The EA base optimizations can be reverted as needed just-in-time. > > > I don't know the answers here. I certainly hope these questions have > > been considered when EA was being developed - in which case I'd like to > > see the discussion. Otherwise this is a conversation that needs to be > > had, and the development of an overall approach for dealing with this, > > not trying to address things piece-meal as someone recognises a new problem. > > Specs are imperfect and always evolving. Sometimes the consequences of > > specific optimizations may not be realised at the spec level yet need to > > be accounted for. That may or may not be the case with EA. > > > > I find it hard to see how to do lock elision yet at the same time track > > when the lock would have been taken, without negating the whole purpose > > of eliding the lock. > > As stated above: this is done for deoptimization already. All tracking information is collected at > compile time. There are no runtime costs, except size of debuginfo. > > JDK-8227745 does what deoptimization does: reallocating and relocking objects. Difference is that > JDK-8227745 does it right before JVMTI access. Deoptimization does it when replacing a compiled > frame by corresponding interpreted frames. I haven't looked at your full proposal in detail as my concerns are not about the code as such but the higher-level model of what is supposed to happen here. Seems I'm the only one who has any concerns about this interaction - but that's fine, I get paid to think about such things. :) Thanks for the discussion. After all that I figure I at least owe you a code review to add to Sergeui's and Dean's. The changes as described look fine to me. I'm a little surprised EA is an all-or-nothing optimisation as I had envisaged something finer-grained that would have simply disabled EA for objects that are actually targets of synchronization. Thanks, David ----- > -----Original Message----- > From: David Holmes > Sent: Montag, 23. September 2019 08:59 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > Hi Richard, > > On 22/09/2019 7:06 am, Reingruber, Richard wrote: >> Hi David, >> >> I still think that JVMTI has to reflect the state of the abstract virtual machine and it must hide >> optimizations from its client (Joe the Java developer). > > I'm not sure I agree as a blanket statement. It may not be feasible to > perform certain optimisations and at the same time keep around enough > information to reconstruct the state as it would have been had the > optimization not occurred. That's why I want to see a discussion of the > broader issue of how something like escape analysis is supposed to > interact with facilities like JVM TI. You've flagged a couple of > conditions under which EA should be disabled, but are they sufficient? > Give they are both dynamic facilities what is supposed to happen if you > keep switching things on and off? Does enabling > can_get_owned_monitor_info require previous EA actions to be undone? > > I don't know the answers here. I certainly hope these questions have > been considered when EA was being developed - in which case I'd like to > see the discussion. Otherwise this is a conversation that needs to be > had, and the development of an overall approach for dealing with this, > not trying to address things piece-meal as someone recognises a new problem. > >> There has to be a way to debug an app without optimizations getting into your way. I like to debug >> my C/C++ program after I compiled it -O0 -g. There has to be something equivalent for Java >> debugging. And I don't mean -Xint. Since 1.4.1 we've got "Full Speed Debugging" [1]. >> >> So far JVMTI does an excellent job of making optimizations transparent to the user. If it didn't, >> I would like to file a JEP for an interface that shows the pure JVM state (at full speed of >> course). >> >> > > > You seem to have completely missed my point. If the object is local and >> > > > is synchronized upon then the synchronization can be elided (and should >> > > > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >> > > > escape. If the synchronization cannot be elided then the object cannot >> > > > be considered local. That is how Escape Analysis should be operating >> > > > here IMHO. >> > > >> > > I presume we agree that it is the state of the abstract virtual machine that must be observed >> > > through JVMTI, right? >> > > >> > > The locking state of an object O after a monitorenter on O is locked on the abstract vm. >> > > >> > > The JIT can still elide synchronization based on a prove that it is actually redundant for the >> > > computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >> > > abstract virtual machine. >> > >> > I don't agree. If the locking can be elided then it is completely >> > elided. If the state of the "abstract VM" had to be perfectly preserved >> > then we wouldn't be able to do the optimisations referred to in JLS 12.6: >> > >> > "Optimizing transformations of a program can be designed that reduce the >> > number of objects that are reachable to be less than those which would >> > naively be considered reachable." >> >> I'd like to write a few words about the relationship of specification and optimizations: It's the purpose >> of a specification to define an abstract model. Optimizations are techniques employed by an >> implementation to reduce resource consumption for better performance within the bounds set by the >> spec, meaning optimizations have to be transparent to the program. You can't write a program that >> detects an optimization. If you could then the implementation would violate rules given by the >> specification. (And I don't consider a benchmark a prove.) >> >> Very rarely you'll find specs mention optimizations. You just don't want to mix specification and >> implementation! If they do, they trade beauty and simplicity of an abstract model, ideally based >> on formal definitions, for performance. >> >> They do mention an optimization, if it is impossible to hide it. In the case of JLS 12.6 it is >> possible to detect that an object reference is removed from a local variable by means of >> finalizations and java.lang.ref. Implementors wanted it so badly, though, that they changed the >> spec. > > Specs are imperfect and always evolving. Sometimes the consequences of > specific optimizations may not be realised at the spec level yet need to > be accounted for. That may or may not be the case with EA. > > I find it hard to see how to do lock elision yet at the same time track > when the lock would have been taken, without negating the whole purpose > of eliding the lock. Afterall if the object is thread-local then the > locking would amount to only biased-locking and it could never be > contended. So it doesn't seem like there was much to elide in the first > place. > > Lock coarsening is also problematic. > >> Another example in the future could be tail-call-optimization ... >> >> > I place lock elision in the same category of "optimising >> > transformations" that changes what would "naively" be expected. Now this >> > should be explicitly covered in the JLS/JVMS somewhere but I'm having >> > trouble finding exactly where. This article discusses lock elision: >> > >> > https://www.ibm.com/developerworks/library/j-jtp10185/index.html >> > >> > and states: >> > >> > "It stands to reason, then, that if a thread enters a synchronized block >> > protected by a lock that no other thread will ever synchronize on, then >> > that synchronization has no effect and can therefore be removed by the >> > optimizer. (The Java Language Specification explicitly allows this >> > optimization.)" >> >> I doubt you'll find it in the JLS/JVMS. It's an implementation detail permitted by the memory >> model. That's what the article is saying. If a Java app has data races, then you would not reason >> about removed synchronization. You would argue with the means of the memory model: eg the >> synchronization action A1 (JLS 17.4.2 [2]) does not 'synchronized-with' (17.4.4 [3]) sync. action >> A2, so they are not ordered. >> >> > I'll have to ping Brian to see if he recalls exactly where this is >> > covered. :) >> >> Ok :) >> Please let me know! > > Brian conceded it may be more implicit in the JMM as you described above :) > > Cheers, > David > >> Richard. >> >> [1] https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/enhancements1.4.html#fsd >> [2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.2 >> [3] https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.4 >> >> -----Original Message----- >> From: David Holmes >> Sent: Freitag, 20. September 2019 11:07 >> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >> >> Hi Richard, >> >> On 20/09/2019 6:31 pm, Reingruber, Richard wrote: >>> Hi David, >>> >>> > On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>> > > Hi David, >>> > > >>> > > thanks for looking at this issue. And my appologies for the lengthy mail... >>> > > >>> > > > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>> > > > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>> > > > > optimizations like scalar replacement become invalid. >>> > > > >>> > > > What bothers me about this is that it seems like escape analysis is >>> > > > doing something wrong in this case. >>> > > >>> > > Yes it is. >>> > > >>> > > > If the object is thread-local but is >>> > > > being synchronized upon then either: >>> > > >>> > > The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>> > > does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>> > > machine, as observed through JVMTI, is wrong. See below... >>> > >>> > You seem to have completely missed my point. If the object is local and >>> > is synchronized upon then the synchronization can be elided (and should >>> > be) in which case it won't appear in GetOwnedMonitorInfo and so does not >>> > escape. If the synchronization cannot be elided then the object cannot >>> > be considered local. That is how Escape Analysis should be operating >>> > here IMHO. >>> >>> I presume we agree that it is the state of the abstract virtual machine that must be observed >>> through JVMTI, right? >>> >>> The locking state of an object O after a monitorenter on O is locked on the abstract vm. >>> >>> The JIT can still elide synchronization based on a prove that it is actually redundant for the >>> computation. But at a safepoint JVMTI must report O as locked, because that's its state on the >>> abstract virtual machine. >> >> I don't agree. If the locking can be elided then it is completely >> elided. If the state of the "abstract VM" had to be perfectly preserved >> then we wouldn't be able to do the optimisations referred to in JLS 12.6: >> >> "Optimizing transformations of a program can be designed that reduce the >> number of objects that are reachable to be less than those which would >> naively be considered reachable." >> >> I place lock elision in the same category of "optimising >> transformations" that changes what would "naively" be expected. Now this >> should be explicitly covered in the JLS/JVMS somewhere but I'm having >> trouble finding exactly where. This article discusses lock elision: >> >> https://www.ibm.com/developerworks/library/j-jtp10185/index.html >> >> and states: >> >> "It stands to reason, then, that if a thread enters a synchronized block >> protected by a lock that no other thread will ever synchronize on, then >> that synchronization has no effect and can therefore be removed by the >> optimizer. (The Java Language Specification explicitly allows this >> optimization.)" >> >> I'll have to ping Brian to see if he recalls exactly where this is >> covered. :) >> >> David >> ----- >> >>> >>> Cheers, Richard. >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Freitag, 20. September 2019 00:59 >>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>> >>> On 20/09/2019 2:42 am, Reingruber, Richard wrote: >>>> Hi David, >>>> >>>> thanks for looking at this issue. And my appologies for the lengthy mail... >>>> >>>> > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>> > > retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>> > > optimizations like scalar replacement become invalid. >>>> > >>>> > What bothers me about this is that it seems like escape analysis is >>>> > doing something wrong in this case. >>>> >>>> Yes it is. >>>> >>>> > If the object is thread-local but is >>>> > being synchronized upon then either: >>>> >>>> The object is not local, because it can escape through JVMTI GetOwnedMonitorInfo(). Escape analysis >>>> does not recognize this. That's what it is doing wrong. Consequently the state of the virtual >>>> machine, as observed through JVMTI, is wrong. See below... >>> >>> You seem to have completely missed my point. If the object is local and >>> is synchronized upon then the synchronization can be elided (and should >>> be) in which case it won't appear in GetOwnedMonitorInfo and so does not >>> escape. If the synchronization cannot be elided then the object cannot >>> be considered local. That is how Escape Analysis should be operating >>> here IMHO. >>> >>> Cheers, >>> David >>> ----- >>> >>>> > a) the synchronization is elided and so the object will not appear in >>>> > the set of owned monitors; or >>>> > b) the fact synchronization occurs renders the object ineligible to be >>>> > considered thread-local, and so there is no problem with it appearing in >>>> > the set of owned monitors >>>> > >>>> > I think there is a bigger design philosophy issue here about the >>>> > interaction of escape analysis and debugging/management frameworks in >>>> > general. I'd like to see a very clear description on exactly how they >>>> > should interact. >>>> > >>>> >>>> I don't see too many design alternatives here. The JVMTI implementation has to present the correct >>>> state of the virtual machine according to the spec. I think it fails to do so in this case. >>>> >>>> Please look again at the test: >>>> >>>> 172 public long dontinline_endlessLoop() { >>>> 173 long cs = checkSum; >>>> 174 while (doLoop && loopCount-- > 0) { >>>> 175 targetIsInLoop = true; >>>> 176 checkSum += checkSum % ++cs; >>>> 177 } >>>> 178 loopCount = 3; >>>> 179 targetIsInLoop = false; >>>> 180 return checkSum; >>>> 181 } >>>> >>>> 249 public void dontinline_testMethod() { >>>> 250 LockCls l1 = new LockCls(); // to be scalar replaced >>>> 251 synchronized (l1) { >>>> 252 inlinedTestMethodWithNestedLocking(l1); >>>> 253 } >>>> 254 } >>>> 255 >>>> 256 public void inlinedTestMethodWithNestedLocking(LockCls l1) { >>>> 257 synchronized (l1) { // nested >>>> 258 dontinline_endlessLoop(); >>>> 259 } >>>> 260 } >>>> >>>> This is the stack when the agent calls GetOwnedMonitorInfo() >>>> >>>> dontinline_endlessLoop() at line 176 >>>> inlinedTestMethodWithNestedLocking() at line 258 // inlined into caller frame >>>> dontinline_testMethod() at line 252 // compiled frame >>>> >>>> The state of the _virtual_ machine at that point is obvious: >>>> >>>> - An instance of LockCls must exist. It was allocated by a new bytecode at line 250. >>>> - That instance was locked by a monitorenter bytecode at line 251 >>>> >>>> This could be proven by interpreting the execution trace bytecode by bytecode using paper and >>>> pencil (hope you won't make me do this, though ;)) >>>> >>>> JVMTI is used to examine the state of the virtual machine. The result of the JVMTI call >>>> GetOwnedMonitorInfo() must include that locked instance of LockCls. It is obviously a bug if it does >>>> not. >>>> >>>> From a more philosophical point of view compiled code is free to change the state of the physical >>>> machine in a way such that it cannot be mapped to a valid state of the virtual machine after each >>>> and every machine instruction. But it must reach points in its execution trace, where it is actually >>>> possible to present a valid state of the virtual machine to observers, e.g. JVMTI agents. These >>>> points are called safepoints. >>>> >>>> The test is a prove that compiled code fails to do so, as it reaches a safepoint where an invalid vm >>>> state is presented. EA does not take into account that the lock state can be observed using >>>> GetOwnedMonitorInfo(). As a fix EA is disabled if the corresponding capability >>>> can_get_owned_monitor_info is taken. With the fix the test passes. >>>> >>>> Note that for the very same reason EA is disabled if can_access_local_variables is taken, because >>>> the JVMTI implementation cannot handout references to objects stored in local variables if they were >>>> scalar replaced. >>>> >>>> With the proposed enhancement JDK-8227745 it is not necessary to disable EA. It allows to revert EA >>>> based optimizations just-in-time before local objects escape. Note that EA opts are already reverted >>>> today if a compiled frame gets replaced by corresponding interpreted frames (see realloc_objects() >>>> and relock_objects() in class Deoptimization) >>>> >>>> Thanks and cheers, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Donnerstag, 19. September 2019 02:43 >>>> To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken >>>> >>>> Hi Richard, >>>> >>>> On 7/09/2019 12:24 am, Reingruber, Richard wrote: >>>>> Hi, >>>>> >>>>> could I please get reviews for >>>>> >>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >>>>> >>>>> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >>>>> retrieve objects locked by a thread. In terms of escape analysis those references escape and >>>>> optimizations like scalar replacement become invalid. >>>> >>>> What bothers me about this is that it seems like escape analysis is >>>> doing something wrong in this case. If the object is thread-local but is >>>> being synchronized upon then either: >>>> a) the synchronization is elided and so the object will not appear in >>>> the set of owned monitors; or >>>> b) the fact synchronization occurs renders the object ineligible to be >>>> considered thread-local, and so there is no problem with it appearing in >>>> the set of owned monitors >>>> >>>> I think there is a bigger design philosophy issue here about the >>>> interaction of escape analysis and debugging/management frameworks in >>>> general. I'd like to see a very clear description on exactly how they >>>> should interact. >>>> >>>> Cheers, >>>> David >>>> >>>>> The runtime currently cannot cope with objects escaping through JVMTI (try included >>>>> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >>>>> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >>>>> >>>>> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >>>>> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >>>>> objects escape through JVMTI. >>>>> >>>>> I've run tier1 tests. >>>>> >>>>> Thanks, Richard. >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >>>>> From mandrikov at gmail.com Tue Sep 24 08:30:00 2019 From: mandrikov at gmail.com (Evgeny Mandrikov) Date: Tue, 24 Sep 2019 10:30:00 +0200 Subject: RFR: JDK-8199136: Dead code in src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java In-Reply-To: <4fc4d799-2e28-ac1b-beac-415ce0237aff@oracle.com> References: <4fc4d799-2e28-ac1b-beac-415ce0237aff@oracle.com> Message-ID: On Tue, Sep 24, 2019 at 8:15 AM David Holmes wrote: > I just spotted this review request languishing in the mailing list. > > The patch will need to be rebased against latest jdk sources, so please > do that. I'll see if I can drum up a second reviewer and get a sponsor. > Hi David, No pb, here is rebased patch - http://cr.openjdk.java.net/~godin/8199136/webrev.01/ Regards, Evgeny -------------- next part -------------- An HTML attachment was scrubbed... URL: From hohensee at amazon.com Tue Sep 24 15:45:22 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 15:45:22 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> Message-ID: <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> Good idea. The current definition is enum { JMM_VERSION_1 = 0x20010000, JMM_VERSION_1_0 = 0x20010000, JMM_VERSION_1_1 = 0x20010100, // JDK 6 JMM_VERSION_1_2 = 0x20010200, // JDK 7 JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA JMM_VERSION_1_2_2 = 0x20010202, JMM_VERSION_2 = 0x20020000, // JDK 10 JMM_VERSION = 0x20020000 }; Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? Absent any other additions, would this work? It creates a minor version for 14. JMM_VERSION_2_1 = 0x20020100, // JDK 14 JMM_VERSION = JMM_VERSION_2_1 Thanks, Paul ?On 9/23/19, 8:42 PM, "Mandy Chung" wrote: Good question. When HS express (mix-n-matched JDK and HS version) was supported, the JMM_VERSION was rev'ed to enable the version checking. HS express is no longer supported. JDK is supported to run with this version of HotSpot VM. OTOH, this adds a new function in the middle of the function table. I think it's a good convention to follow and bump the version number. Mandy On 9/23/19 7:54 PM, Daniil Titov wrote: > Hi Paul, > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > Thank you, > --Daniil > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > Update: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > All test suites that reference getThreadAllocatedBytes pass. These are > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > jdk/com/sun/management > jdk/jdk/jfr/event/runtime > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > Thanks, > > Paul > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > Hi Paul, > > I have almost the same comments as David: > - the same two spots of changes identified > - the addition of the default method was expected > - the change in test is a surprise (I also doubt, it is really needed) > - new CSR is needed > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > Thanks, > Serguei > > > On 9/19/19 16:06, David Holmes wrote: > > Hi Paul, > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > >> More formally, > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > I'm assuming there are only two changes here: > > > > 1. The new method is now a default method that throws UOE. > > > > That seems fine. > > > > 2. You implemented the new method in the test class. > > > > I don't understand why you did that. The test can't be calling the new > > method. Now that it is a default method we will get past the > > compilation failure that caused the problem. So no change to the test > > should be needed AFAICS. > > > > A new CSR request is needed. Just copy everything across from the old, > > with the updated spec. But please also mention this is a > > PlatformManagedObject in the compatibility discussion. > > > > Thanks, > > David > > > >> Thanks, > >> > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > >> Paul" >> hohensee at amazon.com> wrote: > >> > >> Off by 2 error. Changed the subject to reflect 8231209. > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > >> Paul > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > >> wrote: > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> The redo bug is 8231209. 8231211 is closed as a dup > >> of 8231210. > >> Dan > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > >> > I'll have the default method throw UOE. That's the same as > >> the other default methods do. > >> > > >> > The necessary test fix is in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > >> > > >> > public long getCurrentThreadAllocatedBytes() { > >> > return (Long) > >> invokeMethod("getCurrentThreadAllocatedBytes", > >> > new Object[] { }, > >> > new String[] { }); > >> > } > >> > > >> > With this fix, the 134 tests in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > >> Preliminary webrev at > >> > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > >> the > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > >> set? > >> > > >> > Paul > >> > > >> > On 9/18/19, 8:16 PM, "David Holmes" > >> wrote: > >> > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> > >> They all implement > >> com.sun.management.ThreadMXBean, so adding a > >> > >> getCurrentThreadAllocatedBytes broke them. > >> Potential fix is to give it > >> > >> a default implementation, vis > >> > >> > >> > >> public default long > >> getCurrentThreadAllocatedBytes() { > >> > >> return -1; > >> > >> } > >> > >> > >> > > > >> > > com.sun.management.ThreadMXBean (and other platform > >> MXBeans) is a > >> > > "sealed" interface which should only be implemented > >> by JDK. > >> > > >> > Didn't realize that. I don't recall knowing about > >> PlatformManagedObject. > >> > Sealed types will at least allow this to be enforced, > >> though I have to > >> > wonder what the tests are doing here. > >> > > >> > > Unfortunately we don't have the sealed type feature > >> yet. Yes it needs > >> > > to be a default method. I think it should throw UOE. > >> > > > >> > > * @implSpec > >> > > * The default implementation throws {@code > >> > > UnsupportedOperationException}. > >> > > > >> > > The @throw UOE can make it clear that it does not > >> support current thread > >> > > memory allocation measurement. > >> > > >> > Yes that seems a reasonable default if we don't want > >> this to be > >> > implemented outside the platform. > >> > > >> > Thanks, > >> > David > >> > > >> > > Mandy > >> > > >> > > >> > > > > > > > From hohensee at amazon.com Tue Sep 24 15:54:55 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 15:54:55 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <4D2D61B8-C207-4B5D-A2B3-181DB83985B2@amazon.com> Message-ID: Thank you, Mandy. Paul ?On 9/23/19, 8:29 PM, "Mandy Chung" wrote: Looks fine. I added myself to the CSR reviewer. I don't categorize it as source incompatibility as other implementation outside JDK is not supported. I took the liberty to edit the compatibility kind. Mandy On 9/23/19 5:42 PM, Hohensee, Paul wrote: > Update: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > All test suites that reference getThreadAllocatedBytes pass. These are > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > jdk/com/sun/management > jdk/jdk/jfr/event/runtime > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > Thanks, > > Paul > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > Hi Paul, > > I have almost the same comments as David: > - the same two spots of changes identified > - the addition of the default method was expected > - the change in test is a surprise (I also doubt, it is really needed) > - new CSR is needed > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > Thanks, > Serguei > > > On 9/19/19 16:06, David Holmes wrote: > > Hi Paul, > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > >> More formally, > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > I'm assuming there are only two changes here: > > > > 1. The new method is now a default method that throws UOE. > > > > That seems fine. > > > > 2. You implemented the new method in the test class. > > > > I don't understand why you did that. The test can't be calling the new > > method. Now that it is a default method we will get past the > > compilation failure that caused the problem. So no change to the test > > should be needed AFAICS. > > > > A new CSR request is needed. Just copy everything across from the old, > > with the updated spec. But please also mention this is a > > PlatformManagedObject in the compatibility discussion. > > > > Thanks, > > David > > > >> Thanks, > >> > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > >> Paul" >> hohensee at amazon.com> wrote: > >> > >> Off by 2 error. Changed the subject to reflect 8231209. > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > >> Paul > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > >> wrote: > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> The redo bug is 8231209. 8231211 is closed as a dup > >> of 8231210. > >> Dan > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > >> > I'll have the default method throw UOE. That's the same as > >> the other default methods do. > >> > > >> > The necessary test fix is in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > >> > > >> > public long getCurrentThreadAllocatedBytes() { > >> > return (Long) > >> invokeMethod("getCurrentThreadAllocatedBytes", > >> > new Object[] { }, > >> > new String[] { }); > >> > } > >> > > >> > With this fix, the 134 tests in > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > >> Preliminary webrev at > >> > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > >> > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > >> the > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > >> set? > >> > > >> > Paul > >> > > >> > On 9/18/19, 8:16 PM, "David Holmes" > >> wrote: > >> > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > >> > >> They all implement > >> com.sun.management.ThreadMXBean, so adding a > >> > >> getCurrentThreadAllocatedBytes broke them. > >> Potential fix is to give it > >> > >> a default implementation, vis > >> > >> > >> > >> public default long > >> getCurrentThreadAllocatedBytes() { > >> > >> return -1; > >> > >> } > >> > >> > >> > > > >> > > com.sun.management.ThreadMXBean (and other platform > >> MXBeans) is a > >> > > "sealed" interface which should only be implemented > >> by JDK. > >> > > >> > Didn't realize that. I don't recall knowing about > >> PlatformManagedObject. > >> > Sealed types will at least allow this to be enforced, > >> though I have to > >> > wonder what the tests are doing here. > >> > > >> > > Unfortunately we don't have the sealed type feature > >> yet. Yes it needs > >> > > to be a default method. I think it should throw UOE. > >> > > > >> > > * @implSpec > >> > > * The default implementation throws {@code > >> > > UnsupportedOperationException}. > >> > > > >> > > The @throw UOE can make it clear that it does not > >> support current thread > >> > > memory allocation measurement. > >> > > >> > Yes that seems a reasonable default if we don't want > >> this to be > >> > implemented outside the platform. > >> > > >> > Thanks, > >> > David > >> > > >> > > Mandy > >> > > >> > > >> > > > From daniel.daugherty at oracle.com Tue Sep 24 16:04:12 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 24 Sep 2019 12:04:12 -0400 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> Message-ID: $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 47592:?? JMM_VERSION_2? = 0x20020000,? // JDK 10 $ hg log -r 47592 changeset:?? 47592:68d46cb9be45 user:??????? uvangapally date:??????? Thu Oct 05 01:31:53 2017 -0700 summary:???? 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads with a maxDepth argument 8185003 was fixed in JDK10-B31. Dunno about the other releases... Dan On 9/24/19 11:45 AM, Hohensee, Paul wrote: > Good idea. The current definition is > > enum { > JMM_VERSION_1 = 0x20010000, > JMM_VERSION_1_0 = 0x20010000, > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > JMM_VERSION_1_2_2 = 0x20010202, > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION = 0x20020000 > }; > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > Absent any other additions, would this work? It creates a minor version for 14. > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > JMM_VERSION = JMM_VERSION_2_1 > > Thanks, > > Paul > > ?On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > Good question. > > When HS express (mix-n-matched JDK and HS version) was supported, the > JMM_VERSION was rev'ed to enable the version checking. HS express is no > longer supported. JDK is supported to run with this version of HotSpot > VM. OTOH, this adds a new function in the middle of the function > table. I think it's a good convention to follow and bump the version > number. > > Mandy > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > Hi Paul, > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > Thank you, > > --Daniil > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > Update: > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > jdk/com/sun/management > > jdk/jdk/jfr/event/runtime > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > Thanks, > > > > Paul > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > Hi Paul, > > > > I have almost the same comments as David: > > - the same two spots of changes identified > > - the addition of the default method was expected > > - the change in test is a surprise (I also doubt, it is really needed) > > - new CSR is needed > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > Thanks, > > Serguei > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > Hi Paul, > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > >> More formally, > > >> > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > I'm assuming there are only two changes here: > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > That seems fine. > > > > > > 2. You implemented the new method in the test class. > > > > > > I don't understand why you did that. The test can't be calling the new > > > method. Now that it is a default method we will get past the > > > compilation failure that caused the problem. So no change to the test > > > should be needed AFAICS. > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > with the updated spec. But please also mention this is a > > > PlatformManagedObject in the compatibility discussion. > > > > > > Thanks, > > > David > > > > > >> Thanks, > > >> > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > >> Paul" > >> hohensee at amazon.com> wrote: > > >> > > >> Off by 2 error. Changed the subject to reflect 8231209. > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > >> Paul > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > >> wrote: > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> The redo bug is 8231209. 8231211 is closed as a dup > > >> of 8231210. > > >> Dan > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > >> > I'll have the default method throw UOE. That's the same as > > >> the other default methods do. > > >> > > > >> > The necessary test fix is in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > >> > > > >> > public long getCurrentThreadAllocatedBytes() { > > >> > return (Long) > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > >> > new Object[] { }, > > >> > new String[] { }); > > >> > } > > >> > > > >> > With this fix, the 134 tests in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > >> Preliminary webrev at > > >> > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > >> the > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > >> set? > > >> > > > >> > Paul > > >> > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > >> wrote: > > >> > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > >> > >> They all implement > > >> com.sun.management.ThreadMXBean, so adding a > > >> > >> getCurrentThreadAllocatedBytes broke them. > > >> Potential fix is to give it > > >> > >> a default implementation, vis > > >> > >> > > >> > >> public default long > > >> getCurrentThreadAllocatedBytes() { > > >> > >> return -1; > > >> > >> } > > >> > >> > > >> > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > >> MXBeans) is a > > >> > > "sealed" interface which should only be implemented > > >> by JDK. > > >> > > > >> > Didn't realize that. I don't recall knowing about > > >> PlatformManagedObject. > > >> > Sealed types will at least allow this to be enforced, > > >> though I have to > > >> > wonder what the tests are doing here. > > >> > > > >> > > Unfortunately we don't have the sealed type feature > > >> yet. Yes it needs > > >> > > to be a default method. I think it should throw UOE. > > >> > > > > >> > > * @implSpec > > >> > > * The default implementation throws {@code > > >> > > UnsupportedOperationException}. > > >> > > > > >> > > The @throw UOE can make it clear that it does not > > >> support current thread > > >> > > memory allocation measurement. > > >> > > > >> > Yes that seems a reasonable default if we don't want > > >> this to be > > >> > implemented outside the platform. > > >> > > > >> > Thanks, > > >> > David > > >> > > > >> > > Mandy > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Tue Sep 24 16:36:49 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 24 Sep 2019 09:36:49 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Message-ID: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Hi Daniel, David and Serguei, Please review a new version of the fix (webrev.08) that as Daniel suggested renames ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and corrects formatting issues. There are no other changes in this webrev.08 comparing to the previous version webrev.07. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! Best regards, Daniil ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: Daniil, Thanks for sticking with this project through the many versions. Sorry this review is late... On 9/19/19 8:30 PM, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ src/hotspot/share/runtime/mutexLocker.hpp No comments. src/hotspot/share/runtime/mutexLocker.cpp No comments. src/hotspot/share/runtime/threadSMR.cpp L623: MutexLocker ml(Threads_lock); L626: if (!thread->is_exiting()) { Re: discussion about is_exiting() The header comment is pretty clear: src/hotspot/share/runtime/thread.hpp: // thread has called JavaThread::exit() or is terminated bool is_exiting() const; is_exiting() might become true right after you have called it, but its purpose is to ask the question and not prevent the condition from becoming true. As David said, you should consider it an optimization. If you happen to see the condition is true, then you know that the JavaThread isn't going to be around much longer and should act accordingly. The is_exiting() implementation is: inline bool JavaThread::is_exiting() const { // Use load-acquire so that setting of _terminated by // JavaThread::exit() is seen more quickly. TerminatedTypes l_terminated = (TerminatedTypes) OrderAccess::load_acquire((volatile jint *) &_terminated); return l_terminated == _thread_exiting || check_is_terminated(l_terminated); } and it depends on the JavaThread's _terminated field value. // JavaThread termination support enum TerminatedTypes { _not_terminated = 0xDEAD - 2, _thread_exiting, // JavaThread::exit() has been called for this thread _thread_terminated, // JavaThread is removed from thread list _vm_exited // JavaThread is still executing native code, but VM is terminated // only VM_Exit can set _vm_exited }; so the JavaThread's _terminated field can get set to _thread_exiting independent of the Threads_lock, but it can't get set to _thread_terminated without the Threads_lock. So by grabbing the Threads_lock on L623, you make sure that ThreadTable::add_thread(java_tid, thread) does not add a JavaThread that's not on the ThreadsList. It might still become is_exiting() == true right after your L626 if (!thread->is_exiting()) { but it will still be on the main ThreadsList. And that means that when the JavaThread is removed from the main ThreadsList, you'll still call: L931: ThreadTable::remove_thread(tid); L624: // Must be inside the lock to ensure that we don't add the thread to the table typo: s/the thread/a thread/ L633: return thread; nit - L633 - indented too far (should be 2 spaces) src/hotspot/share/services/threadTable.hpp L42: static void lazy_initialize(const ThreadsList *threads); nit - put space between '*' the variable: static void lazy_initialize(const ThreadsList* threads); like you do in your other decls. L45: // Lookup and inserts Perhaps: // Lookup and list management L60-61 - nit - please delete these blank lines. src/hotspot/share/services/threadTable.cpp L28: #include "runtime/timerTrace.hpp" nit - This should be after threadSMR.hpp... (alpha sorted order) L39: static const size_t DefaultThreadTableSizeLog = 8; nit - your other 'static const' are not CamelCase. Why is this one? L45: static ThreadTableHash* volatile _local_table = NULL; L50: static volatile size_t _current_size = 0; L51: static volatile size_t _items_count = 0; nit - can you group the file statics together? (up with L41). L60: _tid(tid),_java_thread(java_thread) {} nit - space after ',' L62 jlong tid() const { return _tid;} L63 JavaThread* thread() const {return _java_thread;} nit - space before '}' nit - space after '{' on L63. L70: static uintx get_hash(Value const& value, bool* is_dead) { Parameter 'is_dead' is not used. L74: static void* allocate_node(size_t size, Value const& value) { Parameter 'value' is not used. L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { Re: discussion about lazy_initialize() racing with ThreadsList::find_JavaThread_from_java_tid() There's a couple of aspects to these two pieces of code racing with each other and racing with new thread creation. Racing with new thread creation is the easy one: If a new thread isn't added to the ThreadTable by ThreadsSMRSupport::add_thread() calling ThreadTable::add_thread(), then the point in the future where someone calls find_JavaThread_from_java_tid() will add it to the table due to the linear search when ThreadTable::find_thread_by_tid() returns NULL. As for multi-threads calling ThreadsList::find_JavaThread_from_java_tid() at the same time which results in multi-threads in lazy_initialize() at the same time... - ThreadTable creation will be linear due to ThreadTableCreate_lock. After _is_initialized is set to true, then no more callers to lazy_initialize() will be in the "if (!_is_initialized)" block. - Once the ThreadTable is created, then multi-threads can be executing the for-loop to add their ThreadsList entries to the ThreadTable. There will be a bit of Threads_lock contention as each of the multi-threads tries to add their entries and there will be some wasted work since the multi-threads will likely have similar ThreadLists. Of course, once _is_initialized is set to true, then any caller to lazy_initialize() will return quickly and ThreadsList::find_JavaThread_from_java_tid() will call ThreadTable::find_thread_by_tid(). If the target java_tid isn't found, then we do the linear search thing here and add the the entry if we find a match in our current ThreadsList. Since we're only adding the one here, we only contend for the Threads_lock here if we find it. If ThreadsList::find_JavaThread_from_java_tid() is called with a target java_tid for a JavaThread that was created after the ThreadsList object that the caller has in hand for the find_JavaThread_from_java_tid() call, then, of course, that target 'java_tid' won't be found because the JavaThread was added the main ThreadsList _after_ the ThreadsList object was created by the caller. Of course, you have to ask where the target java_tid value came from since the JavaThread wasn't around when the ThreadsList::find_JavaThread_from_java_tid() call was made with that target java_tid value... L99: // being concurently populated during the initalization. Typos? Perhaps: // to be concurrently populated during initialization. But I think those two comment lines are more appropriate above this line: L96: MutexLocker ml(ThreadTableCreate_lock); L112: // Must be inside the lock to ensure that we don't add the thread to the table typo: s/the thread/a thread/ L141: return ((double)_items_count)/_current_size; nit - need spaces around '/'. L177: bool equals(ThreadTableEntry **value, bool* is_dead) { nit - put space between '**' the variable: bool equals(ThreadTableEntry** value, Parameter 'is_dead' is not used. L214: while(true) { nit - space before '('. Short version: Thumbs up. Longer version: I don't think I've spotted anything other than nits here. Mostly I've just looked for multi-threaded races, proper usage of the Thread-SMR stuff, and minimal impact in the case where the new ThreadsTable is never needed. Dan P.S. ThreadTable is a bit of misnomer. What you really have here is a ThreadIdTable, but I'm really late to the code review flow with that comment... > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil From mandy.chung at oracle.com Tue Sep 24 16:38:21 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 24 Sep 2019 09:38:21 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> Message-ID: <456ccb4a-a6ce-a4f0-4ae1-1798b5651abc@oracle.com> On 9/24/19 8:45 AM, Hohensee, Paul wrote: > Good idea. The current definition is > > enum { > JMM_VERSION_1 = 0x20010000, > JMM_VERSION_1_0 = 0x20010000, > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > JMM_VERSION_1_2_2 = 0x20010202, > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION = 0x20020000 > }; > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? A new API ThreadMXBean.dumpAllThreads taking a maxDepth argument [1] was added in 10.? It changed an existing function signature. ? I don't think any change was made to jmm.h since 10. The JMM version was originally designed to follow the same convention as the old JDK versioning where major version revision indicates a major release with potential incompatible change whereas minor version revision indicates that no incompatible change to existing APIs.? This explains why major version was bumped in 10. > Absent any other additions, would this work? It creates a minor version for 14. > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > JMM_VERSION = JMM_VERSION_2_1 Perhaps it's time to simplify the scheme to bump the major number when there is change in a JDK release. JMM_VERSION_14 = 0x20040000, Mandy > Thanks, > > Paul > > ?On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > Good question. > > When HS express (mix-n-matched JDK and HS version) was supported, the > JMM_VERSION was rev'ed to enable the version checking. HS express is no > longer supported. JDK is supported to run with this version of HotSpot > VM. OTOH, this adds a new function in the middle of the function > table. I think it's a good convention to follow and bump the version > number. > > Mandy > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > Hi Paul, > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > Thank you, > > --Daniil > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > Update: > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > jdk/com/sun/management > > jdk/jdk/jfr/event/runtime > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > Thanks, > > > > Paul > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > Hi Paul, > > > > I have almost the same comments as David: > > - the same two spots of changes identified > > - the addition of the default method was expected > > - the change in test is a surprise (I also doubt, it is really needed) > > - new CSR is needed > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > Thanks, > > Serguei > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > Hi Paul, > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > >> More formally, > > >> > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > I'm assuming there are only two changes here: > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > That seems fine. > > > > > > 2. You implemented the new method in the test class. > > > > > > I don't understand why you did that. The test can't be calling the new > > > method. Now that it is a default method we will get past the > > > compilation failure that caused the problem. So no change to the test > > > should be needed AFAICS. > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > with the updated spec. But please also mention this is a > > > PlatformManagedObject in the compatibility discussion. > > > > > > Thanks, > > > David > > > > > >> Thanks, > > >> > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > >> Paul" > >> hohensee at amazon.com> wrote: > > >> > > >> Off by 2 error. Changed the subject to reflect 8231209. > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > >> Paul > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > >> wrote: > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> The redo bug is 8231209. 8231211 is closed as a dup > > >> of 8231210. > > >> Dan > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > >> > I'll have the default method throw UOE. That's the same as > > >> the other default methods do. > > >> > > > >> > The necessary test fix is in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > >> > > > >> > public long getCurrentThreadAllocatedBytes() { > > >> > return (Long) > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > >> > new Object[] { }, > > >> > new String[] { }); > > >> > } > > >> > > > >> > With this fix, the 134 tests in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > >> Preliminary webrev at > > >> > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > >> the > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > >> set? > > >> > > > >> > Paul > > >> > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > >> wrote: > > >> > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > >> > >> They all implement > > >> com.sun.management.ThreadMXBean, so adding a > > >> > >> getCurrentThreadAllocatedBytes broke them. > > >> Potential fix is to give it > > >> > >> a default implementation, vis > > >> > >> > > >> > >> public default long > > >> getCurrentThreadAllocatedBytes() { > > >> > >> return -1; > > >> > >> } > > >> > >> > > >> > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > >> MXBeans) is a > > >> > > "sealed" interface which should only be implemented > > >> by JDK. > > >> > > > >> > Didn't realize that. I don't recall knowing about > > >> PlatformManagedObject. > > >> > Sealed types will at least allow this to be enforced, > > >> though I have to > > >> > wonder what the tests are doing here. > > >> > > > >> > > Unfortunately we don't have the sealed type feature > > >> yet. Yes it needs > > >> > > to be a default method. I think it should throw UOE. > > >> > > > > >> > > * @implSpec > > >> > > * The default implementation throws {@code > > >> > > UnsupportedOperationException}. > > >> > > > > >> > > The @throw UOE can make it clear that it does not > > >> support current thread > > >> > > memory allocation measurement. > > >> > > > >> > Yes that seems a reasonable default if we don't want > > >> this to be > > >> > implemented outside the platform. > > >> > > > >> > Thanks, > > >> > David > > >> > > > >> > > Mandy > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > From hohensee at amazon.com Tue Sep 24 16:51:17 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 16:51:17 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> Message-ID: I did JBS queries for, e.g., Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 For 11, I got https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> I didn't find any spec changes for 8 and 9, so we would have JMM_VERSION_2_1 = 0x20020100, // JDK 11 JMM_VERSION_2_2 = 0x20020200, // JDK 14 JMM_VERSION = JMM_VERSION_2_2 Reasonable? ?On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 $ hg log -r 47592 changeset: 47592:68d46cb9be45 user: uvangapally date: Thu Oct 05 01:31:53 2017 -0700 summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads with a maxDepth argument 8185003 was fixed in JDK10-B31. Dunno about the other releases... Dan On 9/24/19 11:45 AM, Hohensee, Paul wrote: > Good idea. The current definition is > > enum { > JMM_VERSION_1 = 0x20010000, > JMM_VERSION_1_0 = 0x20010000, > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > JMM_VERSION_1_2_2 = 0x20010202, > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION = 0x20020000 > }; > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > Absent any other additions, would this work? It creates a minor version for 14. > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > JMM_VERSION = JMM_VERSION_2_1 > > Thanks, > > Paul > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > Good question. > > When HS express (mix-n-matched JDK and HS version) was supported, the > JMM_VERSION was rev'ed to enable the version checking. HS express is no > longer supported. JDK is supported to run with this version of HotSpot > VM. OTOH, this adds a new function in the middle of the function > table. I think it's a good convention to follow and bump the version > number. > > Mandy > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > Hi Paul, > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > Thank you, > > --Daniil > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > Update: > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > jdk/com/sun/management > > jdk/jdk/jfr/event/runtime > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > Thanks, > > > > Paul > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > Hi Paul, > > > > I have almost the same comments as David: > > - the same two spots of changes identified > > - the addition of the default method was expected > > - the change in test is a surprise (I also doubt, it is really needed) > > - new CSR is needed > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > Thanks, > > Serguei > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > Hi Paul, > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > >> More formally, > > >> > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > I'm assuming there are only two changes here: > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > That seems fine. > > > > > > 2. You implemented the new method in the test class. > > > > > > I don't understand why you did that. The test can't be calling the new > > > method. Now that it is a default method we will get past the > > > compilation failure that caused the problem. So no change to the test > > > should be needed AFAICS. > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > with the updated spec. But please also mention this is a > > > PlatformManagedObject in the compatibility discussion. > > > > > > Thanks, > > > David > > > > > >> Thanks, > > >> > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > >> Paul" > >> hohensee at amazon.com> wrote: > > >> > > >> Off by 2 error. Changed the subject to reflect 8231209. > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > >> Paul > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > >> wrote: > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> The redo bug is 8231209. 8231211 is closed as a dup > > >> of 8231210. > > >> Dan > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > >> > I'll have the default method throw UOE. That's the same as > > >> the other default methods do. > > >> > > > >> > The necessary test fix is in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > >> > > > >> > public long getCurrentThreadAllocatedBytes() { > > >> > return (Long) > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > >> > new Object[] { }, > > >> > new String[] { }); > > >> > } > > >> > > > >> > With this fix, the 134 tests in > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > >> Preliminary webrev at > > >> > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > >> > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > >> the > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > >> set? > > >> > > > >> > Paul > > >> > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > >> wrote: > > >> > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > >> > >> They all implement > > >> com.sun.management.ThreadMXBean, so adding a > > >> > >> getCurrentThreadAllocatedBytes broke them. > > >> Potential fix is to give it > > >> > >> a default implementation, vis > > >> > >> > > >> > >> public default long > > >> getCurrentThreadAllocatedBytes() { > > >> > >> return -1; > > >> > >> } > > >> > >> > > >> > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > >> MXBeans) is a > > >> > > "sealed" interface which should only be implemented > > >> by JDK. > > >> > > > >> > Didn't realize that. I don't recall knowing about > > >> PlatformManagedObject. > > >> > Sealed types will at least allow this to be enforced, > > >> though I have to > > >> > wonder what the tests are doing here. > > >> > > > >> > > Unfortunately we don't have the sealed type feature > > >> yet. Yes it needs > > >> > > to be a default method. I think it should throw UOE. > > >> > > > > >> > > * @implSpec > > >> > > * The default implementation throws {@code > > >> > > UnsupportedOperationException}. > > >> > > > > >> > > The @throw UOE can make it clear that it does not > > >> support current thread > > >> > > memory allocation measurement. > > >> > > > >> > Yes that seems a reasonable default if we don't want > > >> this to be > > >> > implemented outside the platform. > > >> > > > >> > Thanks, > > >> > David > > >> > > > >> > > Mandy > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > From mandy.chung at oracle.com Tue Sep 24 16:53:21 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 24 Sep 2019 09:53:21 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> Message-ID: <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> On 9/24/19 9:51 AM, Hohensee, Paul wrote: > I did JBS queries for, e.g., > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > For 11, I got > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes These two issues did not change jmm interface and no need to rev the version. Mandy > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > I didn't find any spec changes for 8 and 9, so we would have > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > JMM_VERSION = JMM_VERSION_2_2 > > Reasonable? > > ?On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > $ hg log -r 47592 > changeset: 47592:68d46cb9be45 > user: uvangapally > date: Thu Oct 05 01:31:53 2017 -0700 > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > with a maxDepth argument > > 8185003 was fixed in JDK10-B31. > > Dunno about the other releases... > > Dan > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > Good idea. The current definition is > > > > enum { > > JMM_VERSION_1 = 0x20010000, > > JMM_VERSION_1_0 = 0x20010000, > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > JMM_VERSION_1_2_2 = 0x20010202, > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > JMM_VERSION = 0x20020000 > > }; > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_1 > > > > Thanks, > > > > Paul > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > Good question. > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > longer supported. JDK is supported to run with this version of HotSpot > > VM. OTOH, this adds a new function in the middle of the function > > table. I think it's a good convention to follow and bump the version > > number. > > > > Mandy > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > Hi Paul, > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > Thank you, > > > --Daniil > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > Update: > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > jdk/com/sun/management > > > jdk/jdk/jfr/event/runtime > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > Hi Paul, > > > > > > I have almost the same comments as David: > > > - the same two spots of changes identified > > > - the addition of the default method was expected > > > - the change in test is a surprise (I also doubt, it is really needed) > > > - new CSR is needed > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > Thanks, > > > Serguei > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > Hi Paul, > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > >> More formally, > > > >> > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > That seems fine. > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > method. Now that it is a default method we will get past the > > > > compilation failure that caused the problem. So no change to the test > > > > should be needed AFAICS. > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > with the updated spec. But please also mention this is a > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > Thanks, > > > > David > > > > > > > >> Thanks, > > > >> > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > >> Paul" > > >> hohensee at amazon.com> wrote: > > > >> > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > >> Paul > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > >> wrote: > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > >> of 8231210. > > > >> Dan > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > >> > I'll have the default method throw UOE. That's the same as > > > >> the other default methods do. > > > >> > > > > >> > The necessary test fix is in > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > >> > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > >> > return (Long) > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > >> > new Object[] { }, > > > >> > new String[] { }); > > > >> > } > > > >> > > > > >> > With this fix, the 134 tests in > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > >> Preliminary webrev at > > > >> > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > >> > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > >> the > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > >> set? > > > >> > > > > >> > Paul > > > >> > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > >> wrote: > > > >> > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > >> > >> They all implement > > > >> com.sun.management.ThreadMXBean, so adding a > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > >> Potential fix is to give it > > > >> > >> a default implementation, vis > > > >> > >> > > > >> > >> public default long > > > >> getCurrentThreadAllocatedBytes() { > > > >> > >> return -1; > > > >> > >> } > > > >> > >> > > > >> > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > >> MXBeans) is a > > > >> > > "sealed" interface which should only be implemented > > > >> by JDK. > > > >> > > > > >> > Didn't realize that. I don't recall knowing about > > > >> PlatformManagedObject. > > > >> > Sealed types will at least allow this to be enforced, > > > >> though I have to > > > >> > wonder what the tests are doing here. > > > >> > > > > >> > > Unfortunately we don't have the sealed type feature > > > >> yet. Yes it needs > > > >> > > to be a default method. I think it should throw UOE. > > > >> > > > > > >> > > * @implSpec > > > >> > > * The default implementation throws {@code > > > >> > > UnsupportedOperationException}. > > > >> > > > > > >> > > The @throw UOE can make it clear that it does not > > > >> support current thread > > > >> > > memory allocation measurement. > > > >> > > > > >> > Yes that seems a reasonable default if we don't want > > > >> this to be > > > >> > implemented outside the platform. > > > >> > > > > >> > Thanks, > > > >> > David > > > >> > > > > >> > > Mandy > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hohensee at amazon.com Tue Sep 24 16:58:10 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 16:58:10 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> Message-ID: <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have JMM_VERSION_2 = 0x20020000, // JDK 10 JMM_VERSION_3 = 0x20030000, // JDK 14 JMM_VERSION = JMM_VERSION_3 ?On 9/24/19, 9:54 AM, "Mandy Chung" wrote: On 9/24/19 9:51 AM, Hohensee, Paul wrote: > I did JBS queries for, e.g., > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > For 11, I got > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes These two issues did not change jmm interface and no need to rev the version. Mandy > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > I didn't find any spec changes for 8 and 9, so we would have > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > JMM_VERSION = JMM_VERSION_2_2 > > Reasonable? > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > $ hg log -r 47592 > changeset: 47592:68d46cb9be45 > user: uvangapally > date: Thu Oct 05 01:31:53 2017 -0700 > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > with a maxDepth argument > > 8185003 was fixed in JDK10-B31. > > Dunno about the other releases... > > Dan > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > Good idea. The current definition is > > > > enum { > > JMM_VERSION_1 = 0x20010000, > > JMM_VERSION_1_0 = 0x20010000, > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > JMM_VERSION_1_2_2 = 0x20010202, > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > JMM_VERSION = 0x20020000 > > }; > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_1 > > > > Thanks, > > > > Paul > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > Good question. > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > longer supported. JDK is supported to run with this version of HotSpot > > VM. OTOH, this adds a new function in the middle of the function > > table. I think it's a good convention to follow and bump the version > > number. > > > > Mandy > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > Hi Paul, > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > Thank you, > > > --Daniil > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > Update: > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > jdk/com/sun/management > > > jdk/jdk/jfr/event/runtime > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > Hi Paul, > > > > > > I have almost the same comments as David: > > > - the same two spots of changes identified > > > - the addition of the default method was expected > > > - the change in test is a surprise (I also doubt, it is really needed) > > > - new CSR is needed > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > Thanks, > > > Serguei > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > Hi Paul, > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > >> More formally, > > > >> > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > That seems fine. > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > method. Now that it is a default method we will get past the > > > > compilation failure that caused the problem. So no change to the test > > > > should be needed AFAICS. > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > with the updated spec. But please also mention this is a > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > Thanks, > > > > David > > > > > > > >> Thanks, > > > >> > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > >> Paul" > > >> hohensee at amazon.com> wrote: > > > >> > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > >> Paul > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > >> wrote: > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > >> of 8231210. > > > >> Dan > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > >> > I'll have the default method throw UOE. That's the same as > > > >> the other default methods do. > > > >> > > > > >> > The necessary test fix is in > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > >> > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > >> > return (Long) > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > >> > new Object[] { }, > > > >> > new String[] { }); > > > >> > } > > > >> > > > > >> > With this fix, the 134 tests in > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > >> Preliminary webrev at > > > >> > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > >> > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > >> the > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > >> set? > > > >> > > > > >> > Paul > > > >> > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > >> wrote: > > > >> > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > >> > >> They all implement > > > >> com.sun.management.ThreadMXBean, so adding a > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > >> Potential fix is to give it > > > >> > >> a default implementation, vis > > > >> > >> > > > >> > >> public default long > > > >> getCurrentThreadAllocatedBytes() { > > > >> > >> return -1; > > > >> > >> } > > > >> > >> > > > >> > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > >> MXBeans) is a > > > >> > > "sealed" interface which should only be implemented > > > >> by JDK. > > > >> > > > > >> > Didn't realize that. I don't recall knowing about > > > >> PlatformManagedObject. > > > >> > Sealed types will at least allow this to be enforced, > > > >> though I have to > > > >> > wonder what the tests are doing here. > > > >> > > > > >> > > Unfortunately we don't have the sealed type feature > > > >> yet. Yes it needs > > > >> > > to be a default method. I think it should throw UOE. > > > >> > > > > > >> > > * @implSpec > > > >> > > * The default implementation throws {@code > > > >> > > UnsupportedOperationException}. > > > >> > > > > > >> > > The @throw UOE can make it clear that it does not > > > >> support current thread > > > >> > > memory allocation measurement. > > > >> > > > > >> > Yes that seems a reasonable default if we don't want > > > >> this to be > > > >> > implemented outside the platform. > > > >> > > > > >> > Thanks, > > > >> > David > > > >> > > > > >> > > Mandy > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mandy.chung at oracle.com Tue Sep 24 17:02:55 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 24 Sep 2019 10:02:55 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> Message-ID: It's fine by me. I have no strong opinion. Mandy On 9/24/19 9:58 AM, Hohensee, Paul wrote: > Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have > > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION_3 = 0x20030000, // JDK 14 > JMM_VERSION = JMM_VERSION_3 > > ?On 9/24/19, 9:54 AM, "Mandy Chung" wrote: > > On 9/24/19 9:51 AM, Hohensee, Paul wrote: > > I did JBS queries for, e.g., > > > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > > > For 11, I got > > > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes > > These two issues did not change jmm interface and no need to rev the > version. > > Mandy > > > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > > > I didn't find any spec changes for 8 and 9, so we would have > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_2 > > > > Reasonable? > > > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > > > $ hg log -r 47592 > > changeset: 47592:68d46cb9be45 > > user: uvangapally > > date: Thu Oct 05 01:31:53 2017 -0700 > > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > > with a maxDepth argument > > > > 8185003 was fixed in JDK10-B31. > > > > Dunno about the other releases... > > > > Dan > > > > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > > Good idea. The current definition is > > > > > > enum { > > > JMM_VERSION_1 = 0x20010000, > > > JMM_VERSION_1_0 = 0x20010000, > > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > > JMM_VERSION_1_2_2 = 0x20010202, > > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > > JMM_VERSION = 0x20020000 > > > }; > > > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > > JMM_VERSION = JMM_VERSION_2_1 > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > > > Good question. > > > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > > longer supported. JDK is supported to run with this version of HotSpot > > > VM. OTOH, this adds a new function in the middle of the function > > > table. I think it's a good convention to follow and bump the version > > > number. > > > > > > Mandy > > > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > > Hi Paul, > > > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > > > Thank you, > > > > --Daniil > > > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > Update: > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > > jdk/com/sun/management > > > > jdk/jdk/jfr/event/runtime > > > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > > > Hi Paul, > > > > > > > > I have almost the same comments as David: > > > > - the same two spots of changes identified > > > > - the addition of the default method was expected > > > > - the change in test is a surprise (I also doubt, it is really needed) > > > > - new CSR is needed > > > > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > > > Thanks, > > > > Serguei > > > > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > > Hi Paul, > > > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > > >> More formally, > > > > >> > > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > > > That seems fine. > > > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > > method. Now that it is a default method we will get past the > > > > > compilation failure that caused the problem. So no change to the test > > > > > should be needed AFAICS. > > > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > > with the updated spec. But please also mention this is a > > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > >> Thanks, > > > > >> > > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > > >> Paul" > > > >> hohensee at amazon.com> wrote: > > > > >> > > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > >> Paul > > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > > >> wrote: > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > > >> of 8231210. > > > > >> Dan > > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > > >> > I'll have the default method throw UOE. That's the same as > > > > >> the other default methods do. > > > > >> > > > > > >> > The necessary test fix is in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > >> > > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > > >> > return (Long) > > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > > >> > new Object[] { }, > > > > >> > new String[] { }); > > > > >> > } > > > > >> > > > > > >> > With this fix, the 134 tests in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > > >> Preliminary webrev at > > > > >> > > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> > > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > > >> the > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > > >> set? > > > > >> > > > > > >> > Paul > > > > >> > > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > > >> wrote: > > > > >> > > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > > >> > >> They all implement > > > > >> com.sun.management.ThreadMXBean, so adding a > > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > > >> Potential fix is to give it > > > > >> > >> a default implementation, vis > > > > >> > >> > > > > >> > >> public default long > > > > >> getCurrentThreadAllocatedBytes() { > > > > >> > >> return -1; > > > > >> > >> } > > > > >> > >> > > > > >> > > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > > >> MXBeans) is a > > > > >> > > "sealed" interface which should only be implemented > > > > >> by JDK. > > > > >> > > > > > >> > Didn't realize that. I don't recall knowing about > > > > >> PlatformManagedObject. > > > > >> > Sealed types will at least allow this to be enforced, > > > > >> though I have to > > > > >> > wonder what the tests are doing here. > > > > >> > > > > > >> > > Unfortunately we don't have the sealed type feature > > > > >> yet. Yes it needs > > > > >> > > to be a default method. I think it should throw UOE. > > > > >> > > > > > > >> > > * @implSpec > > > > >> > > * The default implementation throws {@code > > > > >> > > UnsupportedOperationException}. > > > > >> > > > > > > >> > > The @throw UOE can make it clear that it does not > > > > >> support current thread > > > > >> > > memory allocation measurement. > > > > >> > > > > > >> > Yes that seems a reasonable default if we don't want > > > > >> this to be > > > > >> > implemented outside the platform. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > Mandy > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hohensee at amazon.com Tue Sep 24 17:32:57 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 17:32:57 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> Message-ID: <91C9C9F3-1F54-4C13-9DAE-36D6E63B57A3@amazon.com> Excellent. Updated webrev at http://cr.openjdk.java.net/~phh/8231209/webrev.02/ ?On 9/24/19, 10:03 AM, "Mandy Chung" wrote: It's fine by me. I have no strong opinion. Mandy On 9/24/19 9:58 AM, Hohensee, Paul wrote: > Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have > > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION_3 = 0x20030000, // JDK 14 > JMM_VERSION = JMM_VERSION_3 > > On 9/24/19, 9:54 AM, "Mandy Chung" wrote: > > On 9/24/19 9:51 AM, Hohensee, Paul wrote: > > I did JBS queries for, e.g., > > > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > > > For 11, I got > > > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes > > These two issues did not change jmm interface and no need to rev the > version. > > Mandy > > > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > > > I didn't find any spec changes for 8 and 9, so we would have > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_2 > > > > Reasonable? > > > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > > > $ hg log -r 47592 > > changeset: 47592:68d46cb9be45 > > user: uvangapally > > date: Thu Oct 05 01:31:53 2017 -0700 > > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > > with a maxDepth argument > > > > 8185003 was fixed in JDK10-B31. > > > > Dunno about the other releases... > > > > Dan > > > > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > > Good idea. The current definition is > > > > > > enum { > > > JMM_VERSION_1 = 0x20010000, > > > JMM_VERSION_1_0 = 0x20010000, > > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > > JMM_VERSION_1_2_2 = 0x20010202, > > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > > JMM_VERSION = 0x20020000 > > > }; > > > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > > JMM_VERSION = JMM_VERSION_2_1 > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > > > Good question. > > > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > > longer supported. JDK is supported to run with this version of HotSpot > > > VM. OTOH, this adds a new function in the middle of the function > > > table. I think it's a good convention to follow and bump the version > > > number. > > > > > > Mandy > > > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > > Hi Paul, > > > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > > > Thank you, > > > > --Daniil > > > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > Update: > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > > jdk/com/sun/management > > > > jdk/jdk/jfr/event/runtime > > > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > > > Hi Paul, > > > > > > > > I have almost the same comments as David: > > > > - the same two spots of changes identified > > > > - the addition of the default method was expected > > > > - the change in test is a surprise (I also doubt, it is really needed) > > > > - new CSR is needed > > > > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > > > Thanks, > > > > Serguei > > > > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > > Hi Paul, > > > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > > >> More formally, > > > > >> > > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > > > That seems fine. > > > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > > method. Now that it is a default method we will get past the > > > > > compilation failure that caused the problem. So no change to the test > > > > > should be needed AFAICS. > > > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > > with the updated spec. But please also mention this is a > > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > >> Thanks, > > > > >> > > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > > >> Paul" > > > >> hohensee at amazon.com> wrote: > > > > >> > > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > >> Paul > > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > > >> wrote: > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > > >> of 8231210. > > > > >> Dan > > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > > >> > I'll have the default method throw UOE. That's the same as > > > > >> the other default methods do. > > > > >> > > > > > >> > The necessary test fix is in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > >> > > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > > >> > return (Long) > > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > > >> > new Object[] { }, > > > > >> > new String[] { }); > > > > >> > } > > > > >> > > > > > >> > With this fix, the 134 tests in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > > >> Preliminary webrev at > > > > >> > > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> > > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > > >> the > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > > >> set? > > > > >> > > > > > >> > Paul > > > > >> > > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > > >> wrote: > > > > >> > > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > > >> > >> They all implement > > > > >> com.sun.management.ThreadMXBean, so adding a > > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > > >> Potential fix is to give it > > > > >> > >> a default implementation, vis > > > > >> > >> > > > > >> > >> public default long > > > > >> getCurrentThreadAllocatedBytes() { > > > > >> > >> return -1; > > > > >> > >> } > > > > >> > >> > > > > >> > > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > > >> MXBeans) is a > > > > >> > > "sealed" interface which should only be implemented > > > > >> by JDK. > > > > >> > > > > > >> > Didn't realize that. I don't recall knowing about > > > > >> PlatformManagedObject. > > > > >> > Sealed types will at least allow this to be enforced, > > > > >> though I have to > > > > >> > wonder what the tests are doing here. > > > > >> > > > > > >> > > Unfortunately we don't have the sealed type feature > > > > >> yet. Yes it needs > > > > >> > > to be a default method. I think it should throw UOE. > > > > >> > > > > > > >> > > * @implSpec > > > > >> > > * The default implementation throws {@code > > > > >> > > UnsupportedOperationException}. > > > > >> > > > > > > >> > > The @throw UOE can make it clear that it does not > > > > >> support current thread > > > > >> > > memory allocation measurement. > > > > >> > > > > > >> > Yes that seems a reasonable default if we don't want > > > > >> this to be > > > > >> > implemented outside the platform. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > Mandy > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From aph at redhat.com Tue Sep 24 18:08:31 2019 From: aph at redhat.com (Andrew Haley) Date: Tue, 24 Sep 2019 19:08:31 +0100 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: <8944ad22-2d07-b4d7-671c-45796275d69d@redhat.com> On 9/23/19 4:12 PM, Reingruber, Richard wrote: > Please let me know, if you think this needs more attention and discussion. If so, I would like > to start a new thread, because this one won't have to many followers anymore ;) > > Is mailing list discussion the right format at all? Yes. This is important to many of us. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From serguei.spitsyn at oracle.com Tue Sep 24 18:35:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 24 Sep 2019 11:35:35 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Hi Daniil, This version looks good to me. Thank you for the update! Just one question about ThreadIdTable::remove_thread(jlong tid). What happens if there is no thread with the specified tid in ThreadIdTable? Is it possible? Thanks, Serguei On 9/24/19 9:36 AM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From vladimir.kozlov at oracle.com Tue Sep 24 19:04:18 2019 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 24 Sep 2019 12:04:18 -0700 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: Message-ID: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> I read discussion and this change looks good to me. May be add comment that it is onload capability and can't be changed during execution. Thanks, Vladimir On 9/6/19 7:24 AM, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape analysis those references escape and > optimizations like scalar replacement become invalid. > > The runtime currently cannot cope with objects escaping through JVMTI (try included > tests). Therefore escape analysis should be disabled if an agent requests the capabilities > can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > From daniel.daugherty at oracle.com Tue Sep 24 19:43:16 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 24 Sep 2019 15:43:16 -0400 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <246ece16-045f-3651-bf8c-6f5122ef017a@oracle.com> On 9/24/19 12:36 PM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ Did a "jfilemerge -r" compare between the v07 and v07 patches. Thumbs up! Dan > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From serguei.spitsyn at oracle.com Tue Sep 24 19:45:41 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 24 Sep 2019 12:45:41 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> Message-ID: <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> Hi Paul and Mandy, There was a move from Mikael Vidsted to unify the versioning. The idea is to keep versions in sync with the JDK versions. Now, for JVMTI we have (see, https://bugs.openjdk.java.net/browse/JDK-8219023): enum { ??? JVMTI_VERSION_1?? = 0x30010000, ??? JVMTI_VERSION_1_0 = 0x30010000, ??? JVMTI_VERSION_1_1 = 0x30010100, ??? JVMTI_VERSION_1_2 = 0x30010200, ??? JVMTI_VERSION_9?? = 0x30090000, ??? JVMTI_VERSION_11? = 0x300B0000, ??? JVMTI_VERSION = 0x30000000 + (14 * 0x10000) + ( 0 * 0x100) + 0 /* version: 14.0.0 */ }; Also, it was decided to auto bump the JVMTI and JDWP versions for new releases no matter the spec was changed or not. I've added Mikael to the cc-list. Thanks, Serguei On 9/24/19 9:58 AM, Hohensee, Paul wrote: > Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have > > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION_3 = 0x20030000, // JDK 14 > JMM_VERSION = JMM_VERSION_3 > > ?On 9/24/19, 9:54 AM, "Mandy Chung" wrote: > > On 9/24/19 9:51 AM, Hohensee, Paul wrote: > > I did JBS queries for, e.g., > > > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > > > For 11, I got > > > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes > > These two issues did not change jmm interface and no need to rev the > version. > > Mandy > > > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > > > I didn't find any spec changes for 8 and 9, so we would have > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_2 > > > > Reasonable? > > > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > > > $ hg log -r 47592 > > changeset: 47592:68d46cb9be45 > > user: uvangapally > > date: Thu Oct 05 01:31:53 2017 -0700 > > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > > with a maxDepth argument > > > > 8185003 was fixed in JDK10-B31. > > > > Dunno about the other releases... > > > > Dan > > > > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > > Good idea. The current definition is > > > > > > enum { > > > JMM_VERSION_1 = 0x20010000, > > > JMM_VERSION_1_0 = 0x20010000, > > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > > JMM_VERSION_1_2_2 = 0x20010202, > > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > > JMM_VERSION = 0x20020000 > > > }; > > > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > > JMM_VERSION = JMM_VERSION_2_1 > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > > > Good question. > > > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > > longer supported. JDK is supported to run with this version of HotSpot > > > VM. OTOH, this adds a new function in the middle of the function > > > table. I think it's a good convention to follow and bump the version > > > number. > > > > > > Mandy > > > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > > Hi Paul, > > > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > > > Thank you, > > > > --Daniil > > > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > Update: > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > > jdk/com/sun/management > > > > jdk/jdk/jfr/event/runtime > > > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > > > Hi Paul, > > > > > > > > I have almost the same comments as David: > > > > - the same two spots of changes identified > > > > - the addition of the default method was expected > > > > - the change in test is a surprise (I also doubt, it is really needed) > > > > - new CSR is needed > > > > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > > > Thanks, > > > > Serguei > > > > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > > Hi Paul, > > > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > > >> More formally, > > > > >> > > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > > > That seems fine. > > > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > > method. Now that it is a default method we will get past the > > > > > compilation failure that caused the problem. So no change to the test > > > > > should be needed AFAICS. > > > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > > with the updated spec. But please also mention this is a > > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > >> Thanks, > > > > >> > > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > > >> Paul" > > > >> hohensee at amazon.com> wrote: > > > > >> > > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > >> Paul > > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > > >> wrote: > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > > >> of 8231210. > > > > >> Dan > > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > > >> > I'll have the default method throw UOE. That's the same as > > > > >> the other default methods do. > > > > >> > > > > > >> > The necessary test fix is in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > >> > > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > > >> > return (Long) > > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > > >> > new Object[] { }, > > > > >> > new String[] { }); > > > > >> > } > > > > >> > > > > > >> > With this fix, the 134 tests in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > > >> Preliminary webrev at > > > > >> > > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> > > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > > >> the > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > > >> set? > > > > >> > > > > > >> > Paul > > > > >> > > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > > >> wrote: > > > > >> > > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > > >> > >> They all implement > > > > >> com.sun.management.ThreadMXBean, so adding a > > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > > >> Potential fix is to give it > > > > >> > >> a default implementation, vis > > > > >> > >> > > > > >> > >> public default long > > > > >> getCurrentThreadAllocatedBytes() { > > > > >> > >> return -1; > > > > >> > >> } > > > > >> > >> > > > > >> > > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > > >> MXBeans) is a > > > > >> > > "sealed" interface which should only be implemented > > > > >> by JDK. > > > > >> > > > > > >> > Didn't realize that. I don't recall knowing about > > > > >> PlatformManagedObject. > > > > >> > Sealed types will at least allow this to be enforced, > > > > >> though I have to > > > > >> > wonder what the tests are doing here. > > > > >> > > > > > >> > > Unfortunately we don't have the sealed type feature > > > > >> yet. Yes it needs > > > > >> > > to be a default method. I think it should throw UOE. > > > > >> > > > > > > >> > > * @implSpec > > > > >> > > * The default implementation throws {@code > > > > >> > > UnsupportedOperationException}. > > > > >> > > > > > > >> > > The @throw UOE can make it clear that it does not > > > > >> support current thread > > > > >> > > memory allocation measurement. > > > > >> > > > > > >> > Yes that seems a reasonable default if we don't want > > > > >> this to be > > > > >> > implemented outside the platform. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > Mandy > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Tue Sep 24 19:46:07 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 24 Sep 2019 12:46:07 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: Hi Serguei, Thank you for reviewing this version of the fix. > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? It could be possible when the thread that was started while the thread table was initializing exits. At this point the thread table is initialized and the thread tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable is a correct operation that just leaves the table unchanged. src/hotspot/share/services/threadIdTable.cpp 233 bool ThreadIdTable::remove_thread(jlong tid) { 234 assert(_is_initialized, "Thread table is not initialized"); 235 Thread* thread = Thread::current(); 236 ThreadIdTableLookup lookup(tid); 237 return _local_table->remove(thread, lookup); 238 } src/hotspot/share/utilities/concurrentHashTable.hpp 422 // Returns true if items was deleted matching LOOKUP_FUNC and 423 // prior to destruction DELETE_FUNC is called. 424 template 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { 426 return internal_remove(thread, lookup_f, del_f); 427 } 428 429 // Same without DELETE_FUNC. 430 template 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { 432 return internal_remove(thread, lookup_f, noOp); 433 } src/hotspot/share/utilities/concurrentHashTable.inline.hpp 446 inline bool ConcurrentHashTable:: 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) 448 { 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); 450 assert(bucket->is_locked(), "Must be locked."); 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); 452 Node* rem_n = bucket->first(); 453 bool have_dead = false; 454 while (rem_n != NULL) { 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); 457 break; 458 } else { 459 rem_n_prev = rem_n->next_ptr(); 460 rem_n = rem_n->next(); 461 } 462 } 463 464 bucket->unlock(); 465 466 if (rem_n == NULL) { 467 return false; 468 } Best regards, Daniil ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, This version looks good to me. Thank you for the update! Just one question about ThreadIdTable::remove_thread(jlong tid). What happens if there is no thread with the specified tid in ThreadIdTable? Is it possible? Thanks, Serguei On 9/24/19 9:36 AM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From hohensee at amazon.com Tue Sep 24 20:29:48 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Tue, 24 Sep 2019 20:29:48 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> Message-ID: I can change 0x20030000 to 0x200E0000 to conform to that. Let me know what's wanted. Paul ?On 9/24/19, 12:46 PM, "serguei.spitsyn at oracle.com" wrote: Hi Paul and Mandy, There was a move from Mikael Vidsted to unify the versioning. The idea is to keep versions in sync with the JDK versions. Now, for JVMTI we have (see, https://bugs.openjdk.java.net/browse/JDK-8219023): enum { JVMTI_VERSION_1 = 0x30010000, JVMTI_VERSION_1_0 = 0x30010000, JVMTI_VERSION_1_1 = 0x30010100, JVMTI_VERSION_1_2 = 0x30010200, JVMTI_VERSION_9 = 0x30090000, JVMTI_VERSION_11 = 0x300B0000, JVMTI_VERSION = 0x30000000 + (14 * 0x10000) + ( 0 * 0x100) + 0 /* version: 14.0.0 */ }; Also, it was decided to auto bump the JVMTI and JDWP versions for new releases no matter the spec was changed or not. I've added Mikael to the cc-list. Thanks, Serguei On 9/24/19 9:58 AM, Hohensee, Paul wrote: > Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have > > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION_3 = 0x20030000, // JDK 14 > JMM_VERSION = JMM_VERSION_3 > > On 9/24/19, 9:54 AM, "Mandy Chung" wrote: > > On 9/24/19 9:51 AM, Hohensee, Paul wrote: > > I did JBS queries for, e.g., > > > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > > > For 11, I got > > > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes > > These two issues did not change jmm interface and no need to rev the > version. > > Mandy > > > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > > > I didn't find any spec changes for 8 and 9, so we would have > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_2 > > > > Reasonable? > > > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > > > $ hg log -r 47592 > > changeset: 47592:68d46cb9be45 > > user: uvangapally > > date: Thu Oct 05 01:31:53 2017 -0700 > > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > > with a maxDepth argument > > > > 8185003 was fixed in JDK10-B31. > > > > Dunno about the other releases... > > > > Dan > > > > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > > Good idea. The current definition is > > > > > > enum { > > > JMM_VERSION_1 = 0x20010000, > > > JMM_VERSION_1_0 = 0x20010000, > > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > > JMM_VERSION_1_2_2 = 0x20010202, > > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > > JMM_VERSION = 0x20020000 > > > }; > > > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > > JMM_VERSION = JMM_VERSION_2_1 > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > > > Good question. > > > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > > longer supported. JDK is supported to run with this version of HotSpot > > > VM. OTOH, this adds a new function in the middle of the function > > > table. I think it's a good convention to follow and bump the version > > > number. > > > > > > Mandy > > > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > > Hi Paul, > > > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > > > Thank you, > > > > --Daniil > > > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > Update: > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > > jdk/com/sun/management > > > > jdk/jdk/jfr/event/runtime > > > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > > > Hi Paul, > > > > > > > > I have almost the same comments as David: > > > > - the same two spots of changes identified > > > > - the addition of the default method was expected > > > > - the change in test is a surprise (I also doubt, it is really needed) > > > > - new CSR is needed > > > > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > > > Thanks, > > > > Serguei > > > > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > > Hi Paul, > > > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > > >> More formally, > > > > >> > > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > > > That seems fine. > > > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > > method. Now that it is a default method we will get past the > > > > > compilation failure that caused the problem. So no change to the test > > > > > should be needed AFAICS. > > > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > > with the updated spec. But please also mention this is a > > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > >> Thanks, > > > > >> > > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > > >> Paul" > > > >> hohensee at amazon.com> wrote: > > > > >> > > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > >> Paul > > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > > >> wrote: > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > > >> of 8231210. > > > > >> Dan > > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > > >> > I'll have the default method throw UOE. That's the same as > > > > >> the other default methods do. > > > > >> > > > > > >> > The necessary test fix is in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > >> > > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > > >> > return (Long) > > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > > >> > new Object[] { }, > > > > >> > new String[] { }); > > > > >> > } > > > > >> > > > > > >> > With this fix, the 134 tests in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > > >> Preliminary webrev at > > > > >> > > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> > > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > > >> the > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > > >> set? > > > > >> > > > > > >> > Paul > > > > >> > > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > > >> wrote: > > > > >> > > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > > >> > >> They all implement > > > > >> com.sun.management.ThreadMXBean, so adding a > > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > > >> Potential fix is to give it > > > > >> > >> a default implementation, vis > > > > >> > >> > > > > >> > >> public default long > > > > >> getCurrentThreadAllocatedBytes() { > > > > >> > >> return -1; > > > > >> > >> } > > > > >> > >> > > > > >> > > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > > >> MXBeans) is a > > > > >> > > "sealed" interface which should only be implemented > > > > >> by JDK. > > > > >> > > > > > >> > Didn't realize that. I don't recall knowing about > > > > >> PlatformManagedObject. > > > > >> > Sealed types will at least allow this to be enforced, > > > > >> though I have to > > > > >> > wonder what the tests are doing here. > > > > >> > > > > > >> > > Unfortunately we don't have the sealed type feature > > > > >> yet. Yes it needs > > > > >> > > to be a default method. I think it should throw UOE. > > > > >> > > > > > > >> > > * @implSpec > > > > >> > > * The default implementation throws {@code > > > > >> > > UnsupportedOperationException}. > > > > >> > > > > > > >> > > The @throw UOE can make it clear that it does not > > > > >> support current thread > > > > >> > > memory allocation measurement. > > > > >> > > > > > >> > Yes that seems a reasonable default if we don't want > > > > >> this to be > > > > >> > implemented outside the platform. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > Mandy > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From richard.reingruber at sap.com Tue Sep 24 21:28:57 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Tue, 24 Sep 2019 21:28:57 +0000 Subject: Should optimizations be observable for JVMTI agents? Message-ID: Hi, I would like to get comments on the following questions: 1. Should optimizations be observable for JVMTI agents in general? 2. Should in particular optimzations based on escape analysis be observable? (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? Currently it does [1] (b) Should GetLocalObject() return null for scalar replaced objects? Currently it does not. EA is disabled if agents can access locals. (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the object graph. Currently they are omitted [2] (d) Should memory optimizations (caching of reads) be observable? Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in a compiled method that optimized field accesses based on EA? Currently memory optimizations can be observed, if the agent uses heap functions only to acquire a reference to the local object, which was not scalar replaced. ...and in the end hopefully an agreement :) 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the discussion and to draw more attention to it to get answers we can rely on. Thanks, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8230677 [2] https://bugs.openjdk.java.net/browse/JDK-8230956 [3] Prev. discussion https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html From richard.reingruber at sap.com Tue Sep 24 21:45:53 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Tue, 24 Sep 2019 21:45:53 +0000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: Message-ID: To get started: my own answers to the questions: Optimizations should _not_ be observable for JVMTI agents. They should be as transparent as they are for Java programs. Therefore EA should be disabled appropriately or JDK-8227745 should be realized [3] This is why: Documentation states "[...] JVMTI [...] provides a way to inspect the state and to control the execution of applications running in the JavaTM virtual machine (VM). " [1] It clarifies again that it refers to the Java platform and not to the native platform. Eg '"monitor" means Java programming language object monitor.' [2] 1. + 2.: No In essence the JVMTI Doc says a JVMTI agent observes what the app itself observes. Namely the state of the app. Optimizations that are transparent to the application must be transparent to JVMTI agents also. In other and more words: I see the JVM as an abstract model created by JLS/JVM specs. Concrete implementations use the degrees of freedom the model has for optimizations to reduce resource consumption for better performance. Optimizations are implementation details. They are transparent to Java programs, as they don't have any additional effect on the application state in the JVM(*). According to the documentation above JVMTI agents have to observe the very same state and optimizations therefore are transparent for them too. (*) should be true at least for compiler optimizations. 2a: No So let's look at some thread T which has successfully completed a monitorenter bytecode on an object O. The state of the program on the JVM is then T owns O's monitor. It may be, though, that the monitorenter has no ordering effect, as specified by the Java memory model, because no other thread S will ever execute a monitorenter on O. This allows to eliminate the locking instructions, but even then O should be reported by GetOwnedMonitorInfo(), because that's the state of the program on the JVM. 2b: No This one is a little bit more complicated, because JLS 12.6.1 [4] states "Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable. For example, a Java compiler or code generator may choose to set a variable or parameter that will no longer be used to null to cause the storage for such an object to be potentially reclaimable sooner. Another example of this occurs if the values in an object's fields are stored in registers. The program may then access the registers instead of the object, and never access the object again. This would imply that the object is garbage. Note that this sort of optimization is only allowed if references are on the stack, not stored in the heap." This is tailored exactly to an optimization that aims to remove heap references as soon as possible from the stack roots of the object graph in order to make them unreachable and reclaim their memory. This change in reachability can be detected, though, by means of finalization and java.lang.ref. Therefore it was "legalized" in the JLS. The cited sections match scalar replacement very well also. Would have been nice and sufficient if the statements were restricted to instances with finalization or subclasses of java.lang.ref.Reference. They are considered to escape globally and are never scalar replaced. Finalization and notification/nullification for java.lang.ref.Reference, though, are the only means by which a Java program can observe the optimization. For other objects it is impossible. Therefore scalar replacement is transparent to the Java app and consequently to JVMTI agents also. Note also that C2 can scalar replace objects even if they do escape globally (but only after uncommon traps). Then 12.6.1 does not apply anyway. 2c: No. For the same reasons as for 2b. 2d: No, but not sure :) JVMTI doc above says "... provides a way ... to control the execution of applications...". According to this I would think a modification by a JVMTI agent should have an effect. There might be cases (loop without call but with safepoint), where reads get cached, even without EA and modifications have no effect. Thanks, Richard. [1] https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#whatIs [2] https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#context [3] JDK-8227745 in a nutshell will do what's already done when a thread traps from compiled execution to the interpreter: it reallocates and relocks objects upon access through JVMTI. This allows to enable EA and keep it transparent for agents. [4] https://docs.oracle.com/javase/specs/jls/se13/html/jls-12.html#jls-12.6.1 -----Original Message----- From: serviceability-dev On Behalf Of Reingruber, Richard Sent: Dienstag, 24. September 2019 23:29 To: hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: [CAUTION] Should optimizations be observable for JVMTI agents? Hi, I would like to get comments on the following questions: 1. Should optimizations be observable for JVMTI agents in general? 2. Should in particular optimzations based on escape analysis be observable? (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? Currently it does [1] (b) Should GetLocalObject() return null for scalar replaced objects? Currently it does not. EA is disabled if agents can access locals. (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the object graph. Currently they are omitted [2] (d) Should memory optimizations (caching of reads) be observable? Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in a compiled method that optimized field accesses based on EA? Currently memory optimizations can be observed, if the agent uses heap functions only to acquire a reference to the local object, which was not scalar replaced. ...and in the end hopefully an agreement :) 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the discussion and to draw more attention to it to get answers we can rely on. Thanks, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8230677 [2] https://bugs.openjdk.java.net/browse/JDK-8230956 [3] Prev. discussion https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html From martinrb at google.com Tue Sep 24 22:15:19 2019 From: martinrb at google.com (Martin Buchholz) Date: Tue, 24 Sep 2019 15:15:19 -0700 Subject: RFR: jsr166 integration 2019-09 In-Reply-To: References: <517197737.1773850.1568278129180.JavaMail.zimbra@u-pem.fr> <738837959.2535658.1568387411851.JavaMail.zimbra@u-pem.fr> <16d7944e-4b61-7ed9-aeea-fca44e970016@cs.oswego.edu> <1150861893.2564720.1568393036106.JavaMail.zimbra@u-pem.fr> <1a4c7ebd-78bd-de5b-7053-cff63f13bdeb@oracle.com> <86366d20-1ae0-6694-4b41-ce21dfb8ddbb@oracle.com> Message-ID: Frederic, could you figure out how to resolve 8231031: runtime/ReservedStack/ReservedStackTest.java fails after jsr166 refresh https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/ReservedStackTest/index.html https://bugs.openjdk.java.net/browse/JDK-8231031 On Mon, Sep 23, 2019 at 11:08 PM David Holmes wrote: > Except when I run it through our test system ReservedStackTest is still > failing :( > > I tested it initially when Fred proposed it and that went fine. It also > passes for me locally on Linux. > > David > > On 24/09/2019 12:20 pm, David Holmes wrote: > > Hi Martin, > > > > That all seems fine to me. > > > > Thanks, > > David > > > > On 24/09/2019 3:43 am, Martin Buchholz wrote: > >> We now have a fix-up integration that removes all the previously > >> excluded tests from their exclude lists. > >> > >> > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/overview.html > >> > >> > >> 8231031: runtime/ReservedStack/ReservedStackTest.java fails after > >> jsr166 refresh > >> > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/ReservedStackTest/index.html > >> > >> https://bugs.openjdk.java.net/browse/JDK-8231031 > >> > >> LockInfo objects now restored to their previous values (although David > >> was looking at future improvements). > >> > >> 8231032: ThreadMXBean locking tests fail after JSR 166 refresh > >> > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/wrong-blocker/index.html > >> > >> https://bugs.openjdk.java.net/browse/JDK-8231032 > >> > >> 8231036: vmTestbase monitoring tests fail after JSR 166 refresh > >> > https://cr.openjdk.java.net/~martin/webrevs/jdk/jsr166-integration/SynchronizerLockingThreads/index.html > >> > >> https://bugs.openjdk.java.net/browse/JDK-8231036 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.kozlov at oracle.com Tue Sep 24 22:37:49 2019 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 24 Sep 2019 15:37:49 -0700 Subject: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken In-Reply-To: References: Message-ID: can_tag_objects is "always" capability. If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad. Am I missing something? Thanks, Vladimir On 9/13/19 7:12 AM, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230956 > > JVMTI provides functions to follow references beginning at the roots of the object graph and it > provides functions to iterate all objects on the heap[1][2]. These functions are means to access > objects which are otherwise local to a Java thread. In terms of escape analysis these local objects > escape through these JVMTI functions invalidating optimizations based on escape analysis. > > Example: > > - Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is > scalar replaced. > > - JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all > instances of C. > > - A uses GetObjectsWithTags() to retrieve the tagged instances of C. > > - Error: I is missing because its allocation was eliminated / scalar replaced. > > Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap > functions that let objects escape. Currently it is not possible to revert EA based optimizations > just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the > JVMTI capability can_tag_objects is taken. > > But this is not sufficient, because there may be compiled frames on stack with EA based > optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then > it does not help to disable escape analysis or invalidate compiled methods with ea based > optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete > solution to the issue. > > An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets > added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after > calling JVMTI AddCapabilities(), even with an empty set of capabilities, > JvmtiExport::can_walk_any_space() will return true. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap > [2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0 > From david.holmes at oracle.com Tue Sep 24 22:45:10 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 25 Sep 2019 08:45:10 +1000 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: Looks good to me. Thanks, David On 25/09/2019 2:36 am, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From vladimir.kozlov at oracle.com Wed Sep 25 01:04:11 2019 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 24 Sep 2019 18:04:11 -0700 Subject: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken In-Reply-To: References: Message-ID: It is also not clear to me that it is bug. Based on all description this functionality is used to catch leaks in Java heap. But scalar replaced objects do not exists. JVMTI should not even see them. Thanks, Vladimir On 9/24/19 3:37 PM, Vladimir Kozlov wrote: > can_tag_objects is "always" capability. > > If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad. > > Am I missing something? > > Thanks, > Vladimir > > On 9/13/19 7:12 AM, Reingruber, Richard wrote: >> Hi, >> >> could I please get reviews for >> >> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/ >> Bug:??? https://bugs.openjdk.java.net/browse/JDK-8230956 >> >> JVMTI provides functions to follow references beginning at the roots of the object graph and it >> provides functions to iterate all objects on the heap[1][2]. These functions are means to access >> objects which are otherwise local to a Java thread. In terms of escape analysis these local objects >> escape through these JVMTI functions invalidating optimizations based on escape analysis. >> >> Example: >> >> - Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is >> ?? scalar replaced. >> >> - JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all >> ?? instances of C. >> >> - A uses GetObjectsWithTags() to retrieve the tagged instances of C. >> >> - Error: I is missing because its allocation was eliminated / scalar replaced. >> >> Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap >> functions that let objects escape.? Currently it is not possible to revert EA based optimizations >> just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the >> JVMTI capability can_tag_objects is taken. >> >> But this is not sufficient, because there may be compiled frames on stack with EA based >> optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then >> it does not help to disable escape analysis or invalidate compiled methods with ea based >> optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete >> solution to the issue. >> >> An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets >> added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after >> calling JVMTI AddCapabilities(), even with an empty set of capabilities, >> JvmtiExport::can_walk_any_space() will return true. >> >> I've run tier1 tests. >> >> Thanks, Richard. >> >> [1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap >> [2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0 >> From david.holmes at oracle.com Wed Sep 25 02:07:49 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 25 Sep 2019 12:07:49 +1000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: Message-ID: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> Hi Richard, Thanks for continuing the discussion. Some responses below. On 25/09/2019 7:28 am, Reingruber, Richard wrote: > Hi, > > I would like to get comments on the following questions: > > 1. Should optimizations be observable for JVMTI agents in general? Not a yes/no question IMO. I certainly don't subscribe to your view that JVM TI must always expose the "abstract virtual machine" regardless of what may have been done in the real VM. This seems potentially too limiting and proscriptive to me. The JLS, as you note, already allows for optimisations that may surprise the "naive". That is currently only stated in terms of reachability and finalization, but it establishes the precedent of being pragmatic about these things. Unfortunately the JLS doesn't define its own interactions with specifications like JVM TI, so we can't expect to find all the answers in the JLS, again IMO. The JVMS also doesn't cover interactions between various specifications. If we look at the JVM TI specification it is defined as: "It provides both a way to inspect the state and to control the execution of applications running in the JavaTM virtual machine (VM). " That's nice and broad to me - its about what is running in the JVM. It explicitly does not say anything about observability or control of things that are visible in the Java source code (which seems to be what your 'abstract virtual machine' maps to). Obviously there's is a lot of correlation between the runtime entities and source code entities, but no requirement to exactly replicate what may be present in the source code. Indeed given there is no specification for how to transform source code to bytecode, it would be very difficult for it to state that. So I think we have to consider each optimisation (or group thereof) on its own merits, considering: - how important it would be to accurately interact with things as expressed at the source level - the cost of disabling or reversing the optimisation when needed; and - the overall cost and complexity it adds to the development and maintenance of the JVM > 2. Should in particular optimzations based on escape analysis be observable? > > (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? > Currently it does [1] I certainly don't have a problem with lock-elision really meaning lock-elision. But nor do I insist that is the way it should be. My concern with the current behaviour(?) and proposals in this area is that the approaches are far too coarse-grained**. If we have the can_get_monitor_info capability then we disable all of Escape Analysis for everything! That's harsh. With your proposal for JDK-8227745 IIUC if we are looking for monitors and hit an optimized frame then we deopt incase it may contained an elided monitor. Less harsh but still more coarse-grained than I would like. I would expect EA to be a bit smarter here: - if an object is thread-confined but subject to synchronization, and we can_get_monitor_info, then EA does not perform lock elision (or lock coarsening) That way JVM TI just works as expected without disabling anything unnecessarily, and without needing to preemptively deopt things "just in case". Now full disclosure I am not a JIT person so while what I wrote seems eminently sensible to me, there may be practical reasons why this can't be done (or can't be done at a reasonable cost). ** This is historical. The typical first-level approach has always been "debug on -> optimisation off". At least today we have "fastdebug" VMs, but still we have "specific debug capability on -> optimisation off". > (b) Should GetLocalObject() return null for scalar replaced objects? > Currently it does not. EA is disabled if agents can access locals. > (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the > object graph. > Currently they are omitted [2] > (d) Should memory optimizations (caching of reads) be observable? > Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in > a compiled method that optimized field accesses based on EA? > Currently memory optimizations can be observed, if the agent uses heap functions only to > acquire a reference to the local object, which was not scalar replaced. I haven't thought enough about the above cases, or know enough about the related code to really comment. Cheers, David ----- > ...and in the end hopefully an agreement :) > > 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the > discussion and to draw more attention to it to get answers we can rely on. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8230677 > > [2] https://bugs.openjdk.java.net/browse/JDK-8230956 > > [3] Prev. discussion > https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html > From daniil.x.titov at oracle.com Wed Sep 25 04:46:31 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 24 Sep 2019 21:46:31 -0700 Subject: JDK-8199136: Dead code in src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java In-Reply-To: References: <4fc4d799-2e28-ac1b-beac-415ce0237aff@oracle.com> Message-ID: <66A8280E-B4EB-40F6-98BC-DABF66EF2107@oracle.com> Hi Evgeny, The change looks good to me. Thanks! --Daniil From: serviceability-dev on behalf of Evgeny Mandrikov Date: Tuesday, September 24, 2019 at 1:32 AM To: David Holmes Cc: Subject: Re: RFR: JDK-8199136: Dead code in src/jdk.jcmd/share/classes/sun/tools/common/ProcessArgumentMatcher.java On Tue, Sep 24, 2019 at 8:15 AM David Holmes wrote: I just spotted this review request languishing in the mailing list. The patch will need to be rebased against latest jdk sources, so please do that. I'll see if I can drum up a second reviewer and get a sponsor. Hi David, No pb, here is rebased patch - http://cr.openjdk.java.net/~godin/8199136/webrev.01/ Regards, Evgeny -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbin.ehn at oracle.com Wed Sep 25 06:45:37 2019 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 25 Sep 2019 08:45:37 +0200 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: Hi Daniil, Looks good, thanks! /Robbin On 9/25/19 12:45 AM, David Holmes wrote: > Looks good to me. > > Thanks, > David > > On 25/09/2019 2:36 am, Daniil Titov wrote: >> Hi Daniel, David and Serguei, >> >> Please review a new version of the fix (webrev.08) that as Daniel suggested >> renames >> ThreadTable to ThreadIdTable (related classes and variables are renamed as >> well) and >> corrects formatting issues. There are no other changes in this webrev.08 >> comparing >> to the previous version webrev.07. >> >> Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> >> Best regards, >> Daniil >> >> ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: >> >> ???? Daniil, >> ???? Thanks for sticking with this project through the many versions. >> ???? Sorry this review is late... >> ???? On 9/19/19 8:30 PM, Daniil Titov wrote: >> ???? > Hi David and Serguei, >> ???? > >> ???? > Please review new version of the fix that includes the changes Serguei >> suggested: >> ???? >?? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ???? >?? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ???? >?????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > >> ???? > The changes are in ThreadTable::lazy_initialize() method only. >> ???? > >> ???? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> ???? > >> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> ???? src/hotspot/share/runtime/mutexLocker.hpp >> ????????? No comments. >> ???? src/hotspot/share/runtime/mutexLocker.cpp >> ????????? No comments. >> ???? src/hotspot/share/runtime/threadSMR.cpp >> ????????? L623:???????? MutexLocker ml(Threads_lock); >> ????????? L626:???????? if (!thread->is_exiting()) { >> ????????????? Re: discussion about is_exiting() >> ????????????? The header comment is pretty clear: >> ??????????????? src/hotspot/share/runtime/thread.hpp: >> ????????????????? // thread has called JavaThread::exit() or is terminated >> ????????????????? bool is_exiting() const; >> ????????????? is_exiting() might become true right after you have called it, >> ????????????? but its purpose is to ask the question and not prevent the >> ????????????? condition from becoming true. As David said, you should consider >> ????????????? it an optimization. If you happen to see the condition is true, >> ????????????? then you know that the JavaThread isn't going to be around much >> ????????????? longer and should act accordingly. >> ????????????? The is_exiting() implementation is: >> ??????????????? inline bool JavaThread::is_exiting() const { >> ????????????????? // Use load-acquire so that setting of _terminated by >> ????????????????? // JavaThread::exit() is seen more quickly. >> ????????????????? TerminatedTypes l_terminated = (TerminatedTypes) >> ????????????????????? OrderAccess::load_acquire((volatile jint *) &_terminated); >> ????????????????? return l_terminated == _thread_exiting || >> ???? check_is_terminated(l_terminated); >> ??????????????? } >> ????????????? and it depends on the JavaThread's _terminated field value. >> ??????????????? // JavaThread termination support >> ??????????????? enum TerminatedTypes { >> ???????????????? _not_terminated = 0xDEAD - 2, >> ???????????????? _thread_exiting,???????????????????????????? // >> ???? JavaThread::exit() has been called for this thread >> ???????????????? _thread_terminated,????????????????????????? // JavaThread >> ???? is removed from thread list >> ???????????????? _vm_exited?????????????????????????????????? // JavaThread >> ???? is still executing native code, but VM is terminated >> ????????????????????????????????????????????????????????????? // only VM_Exit >> ???? can set _vm_exited >> ??????????????? }; >> ????????????? so the JavaThread's _terminated field can get set to >> ????????????? _thread_exiting independent of the Threads_lock, but >> ????????????? it can't get set to _thread_terminated without the >> ????????????? Threads_lock. >> ????????????? So by grabbing the Threads_lock on L623, you make sure >> ????????????? that ThreadTable::add_thread(java_tid, thread) does not >> ????????????? add a JavaThread that's not on the ThreadsList. It might >> ????????????? still become is_exiting() == true right after your >> ??????????????? L626???????? if (!thread->is_exiting()) { >> ????????????? but it will still be on the main ThreadsList. And that >> ????????????? means that when the JavaThread is removed from the main >> ????????????? ThreadsList, you'll still call: >> ??????????????? L931:???? ThreadTable::remove_thread(tid); >> ????????? L624:???????? // Must be inside the lock to ensure that we don't >> ???? add the thread to the table >> ????????????? typo: s/the thread/a thread/ >> ????????? L633:?????? return thread; >> ????????????? nit - L633 - indented too far (should be 2 spaces) >> ???? src/hotspot/share/services/threadTable.hpp >> ????????? L42:?? static void lazy_initialize(const ThreadsList *threads); >> ????????????? nit - put space between '*' the variable: >> ??????????????? static void lazy_initialize(const ThreadsList* threads); >> ????????????? like you do in your other decls. >> ????????? L45:?? // Lookup and inserts >> ????????????? Perhaps:? // Lookup and list management >> ????????? L60-61 - nit - please delete these blank lines. >> ???? src/hotspot/share/services/threadTable.cpp >> ????????? L28: #include "runtime/timerTrace.hpp" >> ????????????? nit - This should be after threadSMR.hpp... (alpha sorted order) >> ????????? L39: static const size_t DefaultThreadTableSizeLog = 8; >> ????????????? nit - your other 'static const' are not CamelCase. Why is this one? >> ????????? L45: static ThreadTableHash* volatile _local_table = NULL; >> ????????? L50: static volatile size_t _current_size = 0; >> ????????? L51: static volatile size_t _items_count = 0; >> ????????????? nit - can you group the file statics together? (up with L41). >> ????????? L60:???? _tid(tid),_java_thread(java_thread) {} >> ????????????? nit - space after ',' >> ????????? L62?? jlong tid() const { return _tid;} >> ????????? L63?? JavaThread* thread() const {return _java_thread;} >> ????????????? nit - space before '}' >> ????????????? nit - space after '{' on L63. >> ????????? L70:???? static uintx get_hash(Value const& value, bool* is_dead) { >> ????????????? Parameter 'is_dead' is not used. >> ????????? L74:???? static void* allocate_node(size_t size, Value const& value) { >> ????????????? Parameter 'value' is not used. >> ????????? L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ????????????? Re: discussion about lazy_initialize() racing with >> ????????????????? ThreadsList::find_JavaThread_from_java_tid() >> ????????????? There's a couple of aspects to these two pieces of code racing >> ????????????? with each other and racing with new thread creation. Racing with >> ????????????? new thread creation is the easy one: >> ??????????????? If a new thread isn't added to the ThreadTable by >> ??????????????? ThreadsSMRSupport::add_thread() calling >> ???? ThreadTable::add_thread(), >> ??????????????? then the point in the future where someone calls >> ??????????????? find_JavaThread_from_java_tid() will add it to the table due to >> ??????????????? the linear search when ThreadTable::find_thread_by_tid() >> ??????????????? returns NULL. >> ???????????? As for multi-threads calling >> ???? ThreadsList::find_JavaThread_from_java_tid() >> ???????????? at the same time which results in multi-threads in lazy_initialize() >> ???????????? at the same time... >> ???????????? - ThreadTable creation will be linear due to ThreadTableCreate_lock. >> ?????????????? After _is_initialized is set to true, then no more callers to >> ?????????????? lazy_initialize() will be in the "if (!_is_initialized)" block. >> ???????????? - Once the ThreadTable is created, then multi-threads can be >> ?????????????? executing the for-loop to add their ThreadsList entries to >> ?????????????? the ThreadTable. There will be a bit of Threads_lock contention >> ?????????????? as each of the multi-threads tries to add their entries and >> ?????????????? there will be some wasted work since the multi-threads will >> ?????????????? likely have similar ThreadLists. >> ???????????? Of course, once _is_initialized is set to true, then any caller >> ???????????? to lazy_initialize() will return quickly and >> ???????????? ThreadsList::find_JavaThread_from_java_tid() will call >> ???????????? ThreadTable::find_thread_by_tid(). If the target java_tid isn't >> ???????????? found, then we do the linear search thing here and add the >> ???????????? the entry if we find a match in our current ThreadsList. Since >> ???????????? we're only adding the one here, we only contend for the Threads_lock >> ???????????? here if we find it. >> ???????????? If ThreadsList::find_JavaThread_from_java_tid() is called with a >> ???????????? target java_tid for a JavaThread that was created after the >> ???????????? ThreadsList object that the caller has in hand for the >> ???????????? find_JavaThread_from_java_tid() call, then, of course, that >> ???????????? target 'java_tid' won't be found because the JavaThread was >> ???????????? added the main ThreadsList _after_ the ThreadsList object was >> ???????????? created by the caller. Of course, you have to ask where the >> ???????????? target java_tid value came from since the JavaThread wasn't >> ???????????? around when the ThreadsList::find_JavaThread_from_java_tid() >> ???????????? call was made with that target java_tid value... >> ????????? L99:???????? // being concurently populated during the initalization. >> ????????????? Typos? Perhaps: >> ?????????????????????? // to be concurrently populated during initialization. >> ????????????? But I think those two comment lines are more appropriate above >> ????????????? this line: >> ????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >> ????????? L112:?????????? // Must be inside the lock to ensure that we don't >> ???? add the thread to the table >> ????????????? typo: s/the thread/a thread/ >> ????????? L141:?? return ((double)_items_count)/_current_size; >> ????????????? nit - need spaces around '/'. >> ????????? L177:?? bool equals(ThreadTableEntry **value, bool* is_dead) { >> ????????????? nit - put space between '**' the variable: >> ????????????????? bool equals(ThreadTableEntry** value, >> ????????????? Parameter 'is_dead' is not used. >> ????????? L214:?? while(true) { >> ????????????? nit - space before '('. >> ???? Short version: Thumbs up. >> ???? Longer version: I don't think I've spotted anything other than nits here. >> ???? Mostly I've just looked for multi-threaded races, proper usage of the >> ???? Thread-SMR stuff, and minimal impact in the case where the new >> ???? ThreadsTable is never needed. >> ???? Dan >> ???? P.S. >> ???? ThreadTable is a bit of misnomer. What you really have here is >> ???? a ThreadIdTable, but I'm really late to the code review flow >> ???? with that comment... >> ???? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > >> ???? > Thank you! >> ???? > --Daniil >> >> From richard.reingruber at sap.com Wed Sep 25 09:46:25 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 25 Sep 2019 09:46:25 +0000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> Message-ID: Hi David, thanks for taking part in the discussion. > Hi Richard, > > Thanks for continuing the discussion. Some responses below. > > On 25/09/2019 7:28 am, Reingruber, Richard wrote: > > Hi, > > > > I would like to get comments on the following questions: > > > > 1. Should optimizations be observable for JVMTI agents in general? > > Not a yes/no question IMO. I certainly don't subscribe to your view that > JVM TI must always expose the "abstract virtual machine" regardless of > what may have been done in the real VM. That's what what the documentation says. Everything refers to the Java VM not to the native platform. An object is allocated on the JVM heap which is distinct from native memory. JVM TI must report an object on the JVM heap regardless wether that virtual allocation was mapped to x86 registers, linux thread stack, linux process heap, or to nv ram. This is just an example for the fact that it is the state of the virtual machine which is inspected. Please read again https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#context 'Since this interface provides access to the state of applications running in the Java virtual machine; terminology refers to the Java platform and not the native platform (unless stated otherwise). For example:' "thread" means Java programming language thread. "stack frame" means Java virtual machine stack frame. "class" means Java programming language class. "heap" means Java virtual machine heap. <- note this "monitor" means Java programming language object monitor. These sum up to *the state* to be shown. You'll see that it is all about the _virtual_ machine. Maybe there is room for additional information about the native machine... but first JVMTI must show the virtual state to agents. > This seems potentially too > limiting and proscriptive to me. The JLS, as you note, already allows > for optimisations that may surprise the "naive". That is currently only > stated in terms of reachability and finalization, but it establishes the > precedent of being pragmatic about these things. Unfortunately the JLS > doesn't define its own interactions with specifications like JVM TI, so > we can't expect to find all the answers in the JLS, again IMO. The JVMS > also doesn't cover interactions between various specifications. If we > look at the JVM TI specification it is defined as: > > "It provides both a way to inspect the state and to control the > execution of applications running in the JavaTM virtual machine (VM). " > > That's nice and broad to me - its about what is running in the JVM. It > explicitly does not say anything about observability or control of > things that are visible in the Java source code (which seems to be what > your 'abstract virtual machine' maps to). Obviously there's is a lot of > correlation between the runtime entities and source code entities, but > no requirement to exactly replicate what may be present in the source > code. Indeed given there is no specification for how to transform source > code to bytecode, it would be very difficult for it to state that. > > So I think we have to consider each optimisation (or group thereof) on > its own merits, considering: > - how important it would be to accurately interact with things as > expressed at the source level > - the cost of disabling or reversing the optimisation when needed; and > - the overall cost and complexity it adds to the development and > maintenance of the JVM > > > 2. Should in particular optimzations based on escape analysis be observable? > > > > (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? > > Currently it does [1] > > I certainly don't have a problem with lock-elision really meaning > lock-elision. But nor do I insist that is the way it should be. > > My concern with the current behaviour(?) and proposals in this area is > that the approaches are far too coarse-grained**. If we have the > can_get_monitor_info capability then we disable all of Escape Analysis > for everything! That's harsh. With your proposal for JDK-8227745 IIUC if > we are looking for monitors and hit an optimized frame then we deopt > incase it may contained an elided monitor. Less harsh but still more > coarse-grained than I would like. I would expect EA to be a bit smarter > here: > > - if an object is thread-confined but subject to synchronization, and we > can_get_monitor_info, then EA does not perform lock elision (or lock > coarsening) > > That way JVM TI just works as expected without disabling anything > unnecessarily, and without needing to preemptively deopt things "just in > case". Now full disclosure I am not a JIT person so while what I wrote > seems eminently sensible to me, there may be practical reasons why this > can't be done (or can't be done at a reasonable cost). Just disabling lock elimination is not sufficient, scalar replacement needs to be disabled too, because otherwise it is currently impossible to return the eliminated owner of a locked monitor. Cheers, Richard. -----Original Message----- From: David Holmes Sent: Mittwoch, 25. September 2019 04:08 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: Should optimizations be observable for JVMTI agents? Hi Richard, Thanks for continuing the discussion. Some responses below. On 25/09/2019 7:28 am, Reingruber, Richard wrote: > Hi, > > I would like to get comments on the following questions: > > 1. Should optimizations be observable for JVMTI agents in general? Not a yes/no question IMO. I certainly don't subscribe to your view that JVM TI must always expose the "abstract virtual machine" regardless of what may have been done in the real VM. This seems potentially too limiting and proscriptive to me. The JLS, as you note, already allows for optimisations that may surprise the "naive". That is currently only stated in terms of reachability and finalization, but it establishes the precedent of being pragmatic about these things. Unfortunately the JLS doesn't define its own interactions with specifications like JVM TI, so we can't expect to find all the answers in the JLS, again IMO. The JVMS also doesn't cover interactions between various specifications. If we look at the JVM TI specification it is defined as: "It provides both a way to inspect the state and to control the execution of applications running in the JavaTM virtual machine (VM). " That's nice and broad to me - its about what is running in the JVM. It explicitly does not say anything about observability or control of things that are visible in the Java source code (which seems to be what your 'abstract virtual machine' maps to). Obviously there's is a lot of correlation between the runtime entities and source code entities, but no requirement to exactly replicate what may be present in the source code. Indeed given there is no specification for how to transform source code to bytecode, it would be very difficult for it to state that. So I think we have to consider each optimisation (or group thereof) on its own merits, considering: - how important it would be to accurately interact with things as expressed at the source level - the cost of disabling or reversing the optimisation when needed; and - the overall cost and complexity it adds to the development and maintenance of the JVM > 2. Should in particular optimzations based on escape analysis be observable? > > (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? > Currently it does [1] I certainly don't have a problem with lock-elision really meaning lock-elision. But nor do I insist that is the way it should be. My concern with the current behaviour(?) and proposals in this area is that the approaches are far too coarse-grained**. If we have the can_get_monitor_info capability then we disable all of Escape Analysis for everything! That's harsh. With your proposal for JDK-8227745 IIUC if we are looking for monitors and hit an optimized frame then we deopt incase it may contained an elided monitor. Less harsh but still more coarse-grained than I would like. I would expect EA to be a bit smarter here: - if an object is thread-confined but subject to synchronization, and we can_get_monitor_info, then EA does not perform lock elision (or lock coarsening) That way JVM TI just works as expected without disabling anything unnecessarily, and without needing to preemptively deopt things "just in case". Now full disclosure I am not a JIT person so while what I wrote seems eminently sensible to me, there may be practical reasons why this can't be done (or can't be done at a reasonable cost). ** This is historical. The typical first-level approach has always been "debug on -> optimisation off". At least today we have "fastdebug" VMs, but still we have "specific debug capability on -> optimisation off". > (b) Should GetLocalObject() return null for scalar replaced objects? > Currently it does not. EA is disabled if agents can access locals. > (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the > object graph. > Currently they are omitted [2] > (d) Should memory optimizations (caching of reads) be observable? > Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in > a compiled method that optimized field accesses based on EA? > Currently memory optimizations can be observed, if the agent uses heap functions only to > acquire a reference to the local object, which was not scalar replaced. I haven't thought enough about the above cases, or know enough about the related code to really comment. Cheers, David ----- > ...and in the end hopefully an agreement :) > > 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the > discussion and to draw more attention to it to get answers we can rely on. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8230677 > > [2] https://bugs.openjdk.java.net/browse/JDK-8230956 > > [3] Prev. discussion > https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html > From richard.reingruber at sap.com Wed Sep 25 10:20:07 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 25 Sep 2019 10:20:07 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> Message-ID: Thank you Vladimir and also David and Serguei for your Reviews. > May be add comment that it is onload capability and can't be changed during execution. Done. I'll be out-of-office next week. Will push when coming back. Thanks, Richard. -----Original Message----- From: Vladimir Kozlov Sent: Dienstag, 24. September 2019 21:04 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken I read discussion and this change looks good to me. May be add comment that it is onload capability and can't be changed during execution. Thanks, Vladimir On 9/6/19 7:24 AM, Reingruber, Richard wrote: > Hi, > > could I please get reviews for > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape analysis those references escape and > optimizations like scalar replacement become invalid. > > The runtime currently cannot cope with objects escaping through JVMTI (try included > tests). Therefore escape analysis should be disabled if an agent requests the capabilities > can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > From david.holmes at oracle.com Wed Sep 25 10:34:29 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 25 Sep 2019 20:34:29 +1000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> Message-ID: On 25/09/2019 7:46 pm, Reingruber, Richard wrote: > Hi David, > > thanks for taking part in the discussion. > > > Hi Richard, > > > > Thanks for continuing the discussion. Some responses below. > > > > On 25/09/2019 7:28 am, Reingruber, Richard wrote: > > > Hi, > > > > > > I would like to get comments on the following questions: > > > > > > 1. Should optimizations be observable for JVMTI agents in general? > > > > Not a yes/no question IMO. I certainly don't subscribe to your view that > > JVM TI must always expose the "abstract virtual machine" regardless of > > what may have been done in the real VM. > > That's what what the documentation says. Everything refers to the Java VM not to the native > platform. An object is allocated on the JVM heap which is distinct from native memory. JVM TI must > report an object on the JVM heap regardless wether that virtual allocation was mapped to x86 > registers, linux thread stack, linux process heap, or to nv ram. This is just an example for > the fact that it is the state of the virtual machine which is inspected. > > Please read again https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#context > > 'Since this interface provides access to the state of applications running in the Java virtual > machine; terminology refers to the Java platform and not the native platform (unless stated > otherwise). For example:' > > "thread" means Java programming language thread. > "stack frame" means Java virtual machine stack frame. > "class" means Java programming language class. > "heap" means Java virtual machine heap. <- note this > "monitor" means Java programming language object monitor. > > These sum up to *the state* to be shown. The terminology clarification is simply that - a clarification so that when the spec says "heap" it means "Java heap", when it says "Thread" it means "Java thread" etc without having to spell it out each time. I do not read this as meaning anything about an "abstract virtual machine" state, it is about the Java Virtual Machine state - which to mean can be the concrete state as implemented by Hotspot, not some abstract conceptual model state. We'll just have to agree to disagree here. > > You'll see that it is all about the _virtual_ machine. > Maybe there is room for additional information about the native machine... but first JVMTI must > show the virtual state to agents. > > > This seems potentially too > > limiting and proscriptive to me. The JLS, as you note, already allows > > for optimisations that may surprise the "naive". That is currently only > > stated in terms of reachability and finalization, but it establishes the > > precedent of being pragmatic about these things. Unfortunately the JLS > > doesn't define its own interactions with specifications like JVM TI, so > > we can't expect to find all the answers in the JLS, again IMO. The JVMS > > also doesn't cover interactions between various specifications. If we > > look at the JVM TI specification it is defined as: > > > > "It provides both a way to inspect the state and to control the > > execution of applications running in the JavaTM virtual machine (VM). " > > > > That's nice and broad to me - its about what is running in the JVM. It > > explicitly does not say anything about observability or control of > > things that are visible in the Java source code (which seems to be what > > your 'abstract virtual machine' maps to). Obviously there's is a lot of > > correlation between the runtime entities and source code entities, but > > no requirement to exactly replicate what may be present in the source > > code. Indeed given there is no specification for how to transform source > > code to bytecode, it would be very difficult for it to state that. > > > > So I think we have to consider each optimisation (or group thereof) on > > its own merits, considering: > > - how important it would be to accurately interact with things as > > expressed at the source level > > - the cost of disabling or reversing the optimisation when needed; and > > - the overall cost and complexity it adds to the development and > > maintenance of the JVM > > > > > 2. Should in particular optimzations based on escape analysis be observable? > > > > > > (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? > > > Currently it does [1] > > > > I certainly don't have a problem with lock-elision really meaning > > lock-elision. But nor do I insist that is the way it should be. > > > > My concern with the current behaviour(?) and proposals in this area is > > that the approaches are far too coarse-grained**. If we have the > > can_get_monitor_info capability then we disable all of Escape Analysis > > for everything! That's harsh. With your proposal for JDK-8227745 IIUC if > > we are looking for monitors and hit an optimized frame then we deopt > > incase it may contained an elided monitor. Less harsh but still more > > coarse-grained than I would like. I would expect EA to be a bit smarter > > here: > > > > - if an object is thread-confined but subject to synchronization, and we > > can_get_monitor_info, then EA does not perform lock elision (or lock > > coarsening) > > > > That way JVM TI just works as expected without disabling anything > > unnecessarily, and without needing to preemptively deopt things "just in > > case". Now full disclosure I am not a JIT person so while what I wrote > > seems eminently sensible to me, there may be practical reasons why this > > can't be done (or can't be done at a reasonable cost). > > Just disabling lock elimination is not sufficient, scalar replacement needs to be disabled too, > because otherwise it is currently impossible to return the eliminated owner of a locked monitor. Fine - if the object is used for synchronization it is not a candidate for EA. The point is that EA can be far more selective in how it is applied so that these problems with JVM TI don't arise. Cheers, David ----- > Cheers, Richard. > > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 25. September 2019 04:08 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: Should optimizations be observable for JVMTI agents? > > Hi Richard, > > Thanks for continuing the discussion. Some responses below. > > On 25/09/2019 7:28 am, Reingruber, Richard wrote: >> Hi, >> >> I would like to get comments on the following questions: >> >> 1. Should optimizations be observable for JVMTI agents in general? > > Not a yes/no question IMO. I certainly don't subscribe to your view that > JVM TI must always expose the "abstract virtual machine" regardless of > what may have been done in the real VM. This seems potentially too > limiting and proscriptive to me. The JLS, as you note, already allows > for optimisations that may surprise the "naive". That is currently only > stated in terms of reachability and finalization, but it establishes the > precedent of being pragmatic about these things. Unfortunately the JLS > doesn't define its own interactions with specifications like JVM TI, so > we can't expect to find all the answers in the JLS, again IMO. The JVMS > also doesn't cover interactions between various specifications. If we > look at the JVM TI specification it is defined as: > > "It provides both a way to inspect the state and to control the > execution of applications running in the JavaTM virtual machine (VM). " > > That's nice and broad to me - its about what is running in the JVM. It > explicitly does not say anything about observability or control of > things that are visible in the Java source code (which seems to be what > your 'abstract virtual machine' maps to). Obviously there's is a lot of > correlation between the runtime entities and source code entities, but > no requirement to exactly replicate what may be present in the source > code. Indeed given there is no specification for how to transform source > code to bytecode, it would be very difficult for it to state that. > > So I think we have to consider each optimisation (or group thereof) on > its own merits, considering: > - how important it would be to accurately interact with things as > expressed at the source level > - the cost of disabling or reversing the optimisation when needed; and > - the overall cost and complexity it adds to the development and > maintenance of the JVM > >> 2. Should in particular optimzations based on escape analysis be observable? >> >> (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? >> Currently it does [1] > > I certainly don't have a problem with lock-elision really meaning > lock-elision. But nor do I insist that is the way it should be. > > My concern with the current behaviour(?) and proposals in this area is > that the approaches are far too coarse-grained**. If we have the > can_get_monitor_info capability then we disable all of Escape Analysis > for everything! That's harsh. With your proposal for JDK-8227745 IIUC if > we are looking for monitors and hit an optimized frame then we deopt > incase it may contained an elided monitor. Less harsh but still more > coarse-grained than I would like. I would expect EA to be a bit smarter > here: > > - if an object is thread-confined but subject to synchronization, and we > can_get_monitor_info, then EA does not perform lock elision (or lock > coarsening) > > That way JVM TI just works as expected without disabling anything > unnecessarily, and without needing to preemptively deopt things "just in > case". Now full disclosure I am not a JIT person so while what I wrote > seems eminently sensible to me, there may be practical reasons why this > can't be done (or can't be done at a reasonable cost). > > ** This is historical. The typical first-level approach has always been > "debug on -> optimisation off". At least today we have "fastdebug" VMs, > but still we have "specific debug capability on -> optimisation off". > >> (b) Should GetLocalObject() return null for scalar replaced objects? >> Currently it does not. EA is disabled if agents can access locals. >> (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the >> object graph. >> Currently they are omitted [2] >> (d) Should memory optimizations (caching of reads) be observable? >> Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in >> a compiled method that optimized field accesses based on EA? >> Currently memory optimizations can be observed, if the agent uses heap functions only to >> acquire a reference to the local object, which was not scalar replaced. > > I haven't thought enough about the above cases, or know enough about the > related code to really comment. > > Cheers, > David > ----- > >> ...and in the end hopefully an agreement :) >> >> 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the >> discussion and to draw more attention to it to get answers we can rely on. >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8230677 >> >> [2] https://bugs.openjdk.java.net/browse/JDK-8230956 >> >> [3] Prev. discussion >> https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html >> From richard.reingruber at sap.com Wed Sep 25 12:31:29 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 25 Sep 2019 12:31:29 +0000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> Message-ID: > > > Not a yes/no question IMO. I certainly don't subscribe to your view that > > > JVM TI must always expose the "abstract virtual machine" regardless of > > > what may have been done in the real VM. > > > > That's what what the documentation says. Everything refers to the Java VM not to the native > > platform. An object is allocated on the JVM heap which is distinct from native memory. JVM TI must > > report an object on the JVM heap regardless wether that virtual allocation was mapped to x86 > > registers, linux thread stack, linux process heap, or to nv ram. This is just an example for > > the fact that it is the state of the virtual machine which is inspected. > > > > Please read again https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#context > > > > 'Since this interface provides access to the state of applications running in the Java virtual > > machine; terminology refers to the Java platform and not the native platform (unless stated > > otherwise). For example:' > > > > "thread" means Java programming language thread. > > "stack frame" means Java virtual machine stack frame. > > "class" means Java programming language class. > > "heap" means Java virtual machine heap. <- note this > > "monitor" means Java programming language object monitor. > > > > These sum up to *the state* to be shown. > > The terminology clarification is simply that - a clarification so that > when the spec says "heap" it means "Java heap", when it says "Thread" it > means "Java thread" etc without having to spell it out each time. I do > not read this as meaning anything about an "abstract virtual machine" > state, it is about the Java Virtual Machine state - which to mean can be > the concrete state as implemented by Hotspot, not some abstract > conceptual model state. It is more than that. It is the glue to the JVM and Java Language specifications. This is important to accept. Otherwise I would suggest to do your java debugging with gdb and Intel (or AMD??) x86 manuals at your hand. > We'll just have to agree to disagree here. Agreed. All I do is offer my points to our audience hoping for a few '+1' ;) Richard. -----Original Message----- From: David Holmes Sent: Mittwoch, 25. September 2019 12:34 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: Should optimizations be observable for JVMTI agents? On 25/09/2019 7:46 pm, Reingruber, Richard wrote: > Hi David, > > thanks for taking part in the discussion. > > > Hi Richard, > > > > Thanks for continuing the discussion. Some responses below. > > > > On 25/09/2019 7:28 am, Reingruber, Richard wrote: > > > Hi, > > > > > > I would like to get comments on the following questions: > > > > > > 1. Should optimizations be observable for JVMTI agents in general? > > > > Not a yes/no question IMO. I certainly don't subscribe to your view that > > JVM TI must always expose the "abstract virtual machine" regardless of > > what may have been done in the real VM. > > That's what what the documentation says. Everything refers to the Java VM not to the native > platform. An object is allocated on the JVM heap which is distinct from native memory. JVM TI must > report an object on the JVM heap regardless wether that virtual allocation was mapped to x86 > registers, linux thread stack, linux process heap, or to nv ram. This is just an example for > the fact that it is the state of the virtual machine which is inspected. > > Please read again https://docs.oracle.com/en/java/javase/13/docs/specs/jvmti.html#context > > 'Since this interface provides access to the state of applications running in the Java virtual > machine; terminology refers to the Java platform and not the native platform (unless stated > otherwise). For example:' > > "thread" means Java programming language thread. > "stack frame" means Java virtual machine stack frame. > "class" means Java programming language class. > "heap" means Java virtual machine heap. <- note this > "monitor" means Java programming language object monitor. > > These sum up to *the state* to be shown. The terminology clarification is simply that - a clarification so that when the spec says "heap" it means "Java heap", when it says "Thread" it means "Java thread" etc without having to spell it out each time. I do not read this as meaning anything about an "abstract virtual machine" state, it is about the Java Virtual Machine state - which to mean can be the concrete state as implemented by Hotspot, not some abstract conceptual model state. We'll just have to agree to disagree here. > > You'll see that it is all about the _virtual_ machine. > Maybe there is room for additional information about the native machine... but first JVMTI must > show the virtual state to agents. > > > This seems potentially too > > limiting and proscriptive to me. The JLS, as you note, already allows > > for optimisations that may surprise the "naive". That is currently only > > stated in terms of reachability and finalization, but it establishes the > > precedent of being pragmatic about these things. Unfortunately the JLS > > doesn't define its own interactions with specifications like JVM TI, so > > we can't expect to find all the answers in the JLS, again IMO. The JVMS > > also doesn't cover interactions between various specifications. If we > > look at the JVM TI specification it is defined as: > > > > "It provides both a way to inspect the state and to control the > > execution of applications running in the JavaTM virtual machine (VM). " > > > > That's nice and broad to me - its about what is running in the JVM. It > > explicitly does not say anything about observability or control of > > things that are visible in the Java source code (which seems to be what > > your 'abstract virtual machine' maps to). Obviously there's is a lot of > > correlation between the runtime entities and source code entities, but > > no requirement to exactly replicate what may be present in the source > > code. Indeed given there is no specification for how to transform source > > code to bytecode, it would be very difficult for it to state that. > > > > So I think we have to consider each optimisation (or group thereof) on > > its own merits, considering: > > - how important it would be to accurately interact with things as > > expressed at the source level > > - the cost of disabling or reversing the optimisation when needed; and > > - the overall cost and complexity it adds to the development and > > maintenance of the JVM > > > > > 2. Should in particular optimzations based on escape analysis be observable? > > > > > > (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? > > > Currently it does [1] > > > > I certainly don't have a problem with lock-elision really meaning > > lock-elision. But nor do I insist that is the way it should be. > > > > My concern with the current behaviour(?) and proposals in this area is > > that the approaches are far too coarse-grained**. If we have the > > can_get_monitor_info capability then we disable all of Escape Analysis > > for everything! That's harsh. With your proposal for JDK-8227745 IIUC if > > we are looking for monitors and hit an optimized frame then we deopt > > incase it may contained an elided monitor. Less harsh but still more > > coarse-grained than I would like. I would expect EA to be a bit smarter > > here: > > > > - if an object is thread-confined but subject to synchronization, and we > > can_get_monitor_info, then EA does not perform lock elision (or lock > > coarsening) > > > > That way JVM TI just works as expected without disabling anything > > unnecessarily, and without needing to preemptively deopt things "just in > > case". Now full disclosure I am not a JIT person so while what I wrote > > seems eminently sensible to me, there may be practical reasons why this > > can't be done (or can't be done at a reasonable cost). > > Just disabling lock elimination is not sufficient, scalar replacement needs to be disabled too, > because otherwise it is currently impossible to return the eliminated owner of a locked monitor. Fine - if the object is used for synchronization it is not a candidate for EA. The point is that EA can be far more selective in how it is applied so that these problems with JVM TI don't arise. Cheers, David ----- > Cheers, Richard. > > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 25. September 2019 04:08 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: Should optimizations be observable for JVMTI agents? > > Hi Richard, > > Thanks for continuing the discussion. Some responses below. > > On 25/09/2019 7:28 am, Reingruber, Richard wrote: >> Hi, >> >> I would like to get comments on the following questions: >> >> 1. Should optimizations be observable for JVMTI agents in general? > > Not a yes/no question IMO. I certainly don't subscribe to your view that > JVM TI must always expose the "abstract virtual machine" regardless of > what may have been done in the real VM. This seems potentially too > limiting and proscriptive to me. The JLS, as you note, already allows > for optimisations that may surprise the "naive". That is currently only > stated in terms of reachability and finalization, but it establishes the > precedent of being pragmatic about these things. Unfortunately the JLS > doesn't define its own interactions with specifications like JVM TI, so > we can't expect to find all the answers in the JLS, again IMO. The JVMS > also doesn't cover interactions between various specifications. If we > look at the JVM TI specification it is defined as: > > "It provides both a way to inspect the state and to control the > execution of applications running in the JavaTM virtual machine (VM). " > > That's nice and broad to me - its about what is running in the JVM. It > explicitly does not say anything about observability or control of > things that are visible in the Java source code (which seems to be what > your 'abstract virtual machine' maps to). Obviously there's is a lot of > correlation between the runtime entities and source code entities, but > no requirement to exactly replicate what may be present in the source > code. Indeed given there is no specification for how to transform source > code to bytecode, it would be very difficult for it to state that. > > So I think we have to consider each optimisation (or group thereof) on > its own merits, considering: > - how important it would be to accurately interact with things as > expressed at the source level > - the cost of disabling or reversing the optimisation when needed; and > - the overall cost and complexity it adds to the development and > maintenance of the JVM > >> 2. Should in particular optimzations based on escape analysis be observable? >> >> (a) Should GetOwnedMonitorInfo() exclude objects with eliminated locking? >> Currently it does [1] > > I certainly don't have a problem with lock-elision really meaning > lock-elision. But nor do I insist that is the way it should be. > > My concern with the current behaviour(?) and proposals in this area is > that the approaches are far too coarse-grained**. If we have the > can_get_monitor_info capability then we disable all of Escape Analysis > for everything! That's harsh. With your proposal for JDK-8227745 IIUC if > we are looking for monitors and hit an optimized frame then we deopt > incase it may contained an elided monitor. Less harsh but still more > coarse-grained than I would like. I would expect EA to be a bit smarter > here: > > - if an object is thread-confined but subject to synchronization, and we > can_get_monitor_info, then EA does not perform lock elision (or lock > coarsening) > > That way JVM TI just works as expected without disabling anything > unnecessarily, and without needing to preemptively deopt things "just in > case". Now full disclosure I am not a JIT person so while what I wrote > seems eminently sensible to me, there may be practical reasons why this > can't be done (or can't be done at a reasonable cost). > > ** This is historical. The typical first-level approach has always been > "debug on -> optimisation off". At least today we have "fastdebug" VMs, > but still we have "specific debug capability on -> optimisation off". > >> (b) Should GetLocalObject() return null for scalar replaced objects? >> Currently it does not. EA is disabled if agents can access locals. >> (c) Should scalar replaced objects be omitted when agents iterate the heap / traverse the >> object graph. >> Currently they are omitted [2] >> (d) Should memory optimizations (caching of reads) be observable? >> Eg. should field modifications of a NoEscape object O by a JVMTI agent have an effect in >> a compiled method that optimized field accesses based on EA? >> Currently memory optimizations can be observed, if the agent uses heap functions only to >> acquire a reference to the local object, which was not scalar replaced. > > I haven't thought enough about the above cases, or know enough about the > related code to really comment. > > Cheers, > David > ----- > >> ...and in the end hopefully an agreement :) >> >> 2a was discussed in the code review of JDK-8230677 [3]. David Holmes wished to generalize the >> discussion and to draw more attention to it to get answers we can rely on. >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8230677 >> >> [2] https://bugs.openjdk.java.net/browse/JDK-8230956 >> >> [3] Prev. discussion >> https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-September/029224.html >> From richard.reingruber at sap.com Wed Sep 25 13:43:00 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 25 Sep 2019 13:43:00 +0000 Subject: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken In-Reply-To: References: Message-ID: Hello Vladimir, thanks for looking at this. > can_tag_objects is "always" capability. That's correct. > If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad. > > Am I missing something? No that's correct too. If you include jvmti as hotspot feature in your build then you should disable escape analysis by default. That's really too hard. My proposed improvement has the effect that escape analysis is disabled if any agent calls AddCapabilities() during the OnLoad phase. Because calling AddCapabilities(), even with an empty set of capabilities, has the effect that JvmtiExport::can_walk_any_space() will return always true after that. Not sure if this is a bug or feature though. But only disabling EA by default or JDK-8227745 would be a real fix. > It is also not clear to me that it is bug. Based on all description this functionality is used to > catch leaks in Java heap. But scalar replaced objects do not exists. JVMTI should not even see them. I do think, JVMTI heap functions must report scalar replaced objects for formal and for practical reasons. JVM spec defines for the new bytecode [1] Operation Create new object Description [...] Memory for a new instance of that class is allocated from the garbage-collected heap, [...] So I think JVMTI heap functions have to report that new instance on the virtual heap. If they don't do it you can get contradicting data, too, from agents that employ bytecode instrumentation to count instances. Also there can be a leak rooted at a chain of scalar replaced objects. This is currently not reported either. If an agent traverses references it won't arrive at the leaked objects. If it itereates objects on the heap then the leaking objects are found, but it will remain unknown what's keeping them alive. Thanks, Richard. [1] https://docs.oracle.com/javase/specs/jvms/se13/html/jvms-6.html#jvms-6.5.new -----Original Message----- From: Vladimir Kozlov Sent: Mittwoch, 25. September 2019 03:04 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; OpenJDK Serviceability Subject: Re: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken It is also not clear to me that it is bug. Based on all description this functionality is used to catch leaks in Java heap. But scalar replaced objects do not exists. JVMTI should not even see them. Thanks, Vladimir On 9/24/19 3:37 PM, Vladimir Kozlov wrote: > can_tag_objects is "always" capability. > > If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad. > > Am I missing something? > > Thanks, > Vladimir > > On 9/13/19 7:12 AM, Reingruber, Richard wrote: >> Hi, >> >> could I please get reviews for >> >> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/ >> Bug:??? https://bugs.openjdk.java.net/browse/JDK-8230956 >> >> JVMTI provides functions to follow references beginning at the roots of the object graph and it >> provides functions to iterate all objects on the heap[1][2]. These functions are means to access >> objects which are otherwise local to a Java thread. In terms of escape analysis these local objects >> escape through these JVMTI functions invalidating optimizations based on escape analysis. >> >> Example: >> >> - Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is >> ?? scalar replaced. >> >> - JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all >> ?? instances of C. >> >> - A uses GetObjectsWithTags() to retrieve the tagged instances of C. >> >> - Error: I is missing because its allocation was eliminated / scalar replaced. >> >> Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap >> functions that let objects escape.? Currently it is not possible to revert EA based optimizations >> just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the >> JVMTI capability can_tag_objects is taken. >> >> But this is not sufficient, because there may be compiled frames on stack with EA based >> optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then >> it does not help to disable escape analysis or invalidate compiled methods with ea based >> optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete >> solution to the issue. >> >> An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets >> added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after >> calling JVMTI AddCapabilities(), even with an empty set of capabilities, >> JvmtiExport::can_walk_any_space() will return true. >> >> I've run tier1 tests. >> >> Thanks, Richard. >> >> [1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap >> [2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0 >> From daniel.daugherty at oracle.com Wed Sep 25 14:32:26 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 25 Sep 2019 10:32:26 -0400 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> Message-ID: <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> Based on the review thread, it looks like Richard has run Tier1 tests on this change. I don't think there are any JVM/TI tests in Tier1. I'm not sure how much compiler testing is done in Tier1, but I do know that the compiler stress testing doesn't kick in until the later tiers (Tier5 or Tier6)... Serguei, with your JVM/TI hat on, what kind of additional testing (if any) do you think we need here? Dan On 9/25/19 6:20 AM, Reingruber, Richard wrote: > Thank you Vladimir and also David and Serguei for your Reviews. > > > May be add comment that it is onload capability and can't be changed during execution. > > Done. > > I'll be out-of-office next week. Will push when coming back. > > Thanks, Richard. > > -----Original Message----- > From: Vladimir Kozlov > Sent: Dienstag, 24. September 2019 21:04 > To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken > > I read discussion and this change looks good to me. > > May be add comment that it is onload capability and can't be changed during execution. > > Thanks, > Vladimir > > On 9/6/19 7:24 AM, Reingruber, Richard wrote: >> Hi, >> >> could I please get reviews for >> >> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 >> >> The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to >> retrieve objects locked by a thread. In terms of escape analysis those references escape and >> optimizations like scalar replacement become invalid. >> >> The runtime currently cannot cope with objects escaping through JVMTI (try included >> tests). Therefore escape analysis should be disabled if an agent requests the capabilities >> can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. >> >> This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to >> disable escape analysis, instead optimizations based on escape analysis will be reverted just before >> objects escape through JVMTI. >> >> I've run tier1 tests. >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8227745 >> From adinn at redhat.com Wed Sep 25 14:32:07 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 25 Sep 2019 15:32:07 +0100 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> Message-ID: <7d0bb4b7-dad7-e7ec-e60a-accea3ce43a0@redhat.com> On 25/09/2019 13:31, Reingruber, Richard wrote: > > The terminology clarification is simply that - a clarification so that > > when the spec says "heap" it means "Java heap", when it says "Thread" it > > means "Java thread" etc without having to spell it out each time. I do > > not read this as meaning anything about an "abstract virtual machine" > > state, it is about the Java Virtual Machine state - which to mean can be > > the concrete state as implemented by Hotspot, not some abstract > > conceptual model state. > > It is more than that. It is the glue to the JVM and Java Language specifications. This is important > to accept. Otherwise I would suggest to do your java debugging with gdb and Intel (or AMD??) x86 > manuals at your hand. > > > We'll just have to agree to disagree here. > > Agreed. All I do is offer my points to our audience hoping for a few '+1' ;) Sorry, but I'm going to throw in a -1 here, not on any grounds of textual exegesis of the JVM spec rather those of simple pragmatism -- grounds that David already alluded to in his mention of finalization. The path you want to go down is to force disabling of optimizations as needed in order to ensure that JVMTI agents see a 'pure' version of the /abstract/ machine state as defined in the Java Virtual Machine spec. This is in many ways laudable: 1) such a 'pure' view is one that can be mapped fairly directly back to the original Java application code state (assuming you factor in enough knowledge of how javac translates code to bytecode and relying on the fact that javac does not do anything very tricksy by way of optimization during that translation) i.e. it is clear. 2) this pure view is going to be unchanged by whatever tricks the JIT and/or JVM might have up their sleeves i.e. it is consistent/repeatable. That latter point of consistency is, perhaps, more significant than the former one of clarity since for most programmers bytecode is only ever observed 'through a glass darkly'. The downside comes when you consider why you might want to use an agent. Primarily, JVMTI was provided for monitoring actual program execution. Agents can also be used to modify program execution, making program transformations that result in locally and/or globally variant behaviour. Well, actually, I guess some monitoring operations falls into that latter camp. In order to pursue your proposed path it is going to be necessary to disable, either piecemeal or wholesale, a variety of JIT optimizations. This would mean that use of an agent might involve a cost that is prohibitive. Primarily, that's a cost in performance (whether measured in space or time). When it comes to monitoring it is also the extra cost in the accuracy of any report of what is being monitored -- if installation of an agent or use of a montoring capability means you run different code in a different way then you can end up measuring apples not pears. That possible inaccuracy has to be weighed against other inaccuracies that arise when the JIT invalidates or changes notifications of all the events the idealized version of the abstract machine would imply. Where the balance lies depends on the specifics of which optimizations are disabled and the needs/behaviour of the app. I believe that was what David was talking about when he mentioned as an example the case of finalizers being run 'early' for non-reachable objects. The optimization here allows early reclamation of objects and reuse of stack slots. The cost is that object appear to go out of scope 'before' their fields are read -- at least as far as the source/abstract machine view of 'before' is defined. How much that cost imposes on users is unclear but the cost was discounted because finalization is already a highly unreliable mechanism -- i.e. who cares? So, I agree with David's general argument that we have to be pragmatic about these things as he said taking into account " - how important it would be to accurately interact with things as expressed at the source level - the cost of disabling or reversing the optimisation when needed; and - the overall cost and complexity it adds to the development and maintenance of the JVM" regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From sam.thomas at broadcom.com Fri Sep 20 21:08:38 2019 From: sam.thomas at broadcom.com (Sam Thomas) Date: Fri, 20 Sep 2019 14:08:38 -0700 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Message-ID: cool thanks Thanks ./Sam On Fri, Sep 20, 2019 at 5:05 AM Michael Rasmussen < Michael.Rasmussen at roguewave.com> wrote: > On 9/18/19 2:47 PM, Sam Thomas wrote: > > Hi, > > > > I'm trying to understand if a class will load as soon as all the > transformers return. The aim is to get a class reference of a class I have > seen in my transformer. > > Short answer: no. > > Trying to define a class can cause other classes to try to be loaded > before (for instance the super types), meaning when you're done > transforming class A, the same thread might try to load multiple other > classes before it finishes defining A. > And if any of these fail, A will also fail to be defined. > > > Also if there is way to get the same without triggering class loading - > if the class is not loaded return a null reference. > > ClassLoader has a protected method findLoadedClass that checks if a class > is loaded by that classloader and returns the Class instance if so, or null > otherwise. > But as mentioned, the method is protected so normally not accessible > outside the classloader itself. > > /Michael > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.thomas at broadcom.com Tue Sep 24 23:11:00 2019 From: sam.thomas at broadcom.com (Sam Thomas) Date: Tue, 24 Sep 2019 16:11:00 -0700 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Message-ID: Hi Michael, Returning question: I understand that the method findLoadedClass is protected. But say it was public, how would you find out loaded classes on the bootstrap classloader? Since from instrumentation perspective when a loader is null its the bootstrap classloader. Thanks ./Sam On Fri, Sep 20, 2019 at 2:08 PM Sam Thomas wrote: > cool thanks > > Thanks > ./Sam > > > On Fri, Sep 20, 2019 at 5:05 AM Michael Rasmussen < > Michael.Rasmussen at roguewave.com> wrote: > >> On 9/18/19 2:47 PM, Sam Thomas wrote: >> > Hi, >> > >> > I'm trying to understand if a class will load as soon as all the >> transformers return. The aim is to get a class reference of a class I have >> seen in my transformer. >> >> Short answer: no. >> >> Trying to define a class can cause other classes to try to be loaded >> before (for instance the super types), meaning when you're done >> transforming class A, the same thread might try to load multiple other >> classes before it finishes defining A. >> And if any of these fail, A will also fail to be defined. >> >> > Also if there is way to get the same without triggering class loading - >> if the class is not loaded return a null reference. >> >> ClassLoader has a protected method findLoadedClass that checks if a class >> is loaded by that classloader and returns the Class instance if so, or null >> otherwise. >> But as mentioned, the method is protected so normally not accessible >> outside the classloader itself. >> >> /Michael >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.kozlov at oracle.com Wed Sep 25 16:52:57 2019 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 25 Sep 2019 09:52:57 -0700 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: <7d0bb4b7-dad7-e7ec-e60a-accea3ce43a0@redhat.com> References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> <7d0bb4b7-dad7-e7ec-e60a-accea3ce43a0@redhat.com> Message-ID: I had very good discussion yesterday with Serguei Spitsyn from serviceability. He said that we may not need one solution. Both cases, monitoring and debugging, are important and we can use different solutions for them. As Andrew said, for monitoring you want to observe the same behavior as in production. I think we should not restrict JIT and others optimizations in this case. I also agree with Richard that JVMTI may report scalar replaced objects and elided monitors in this mode if requested - it is also interesting information. For debugging we may want to see locals and monitors present in compiled code and it is reasonable to avoid optimizations which remove them. But even for debugging we may want to use optimized code as we do when debugging optimized C code (we don't expect to see all locals in such case). May be we should check not which JVMTI capabilities are switched on but for what JVMTI is used for. Or add API for user to ask what he wants. As Richard mentioned in some email compiled frame can't be deoptimized (to get objects and monitors back) if it is not top frame on stack. So we can't rely on deoptimization here. JIT and Runtime need to know what to do with optimizations when agent is attached to JVM. I don't like suggestion for 8230956 to switch off EA when can_tag_objects is on. Based on declaration, can_tag_objects is always on. Which means we will have EA off in all cases including monitoring tools (JFR, managment, profiling) which is not acceptable for me. I share David's concern about missing cases for some JVMTI capabilities which are not checked when JIT decides to use optimizations. And I think it will be troublesome to go through all capabilities old and new to see if we need to add checks. For me as JIT compiler engineer it would be much simpler to ask JVMTI should optimizations be switched off or not and let JVMTI find in which case it is used and make decision instead of current case when JIT checks some JVMTI capabilities. I agree to provide information in compiled code for JVMTI - we do provide information about scalar replaced object already. And I think it would be much easier to document such relation between JIT and JVMTI. Thanks, Vladimir On 9/25/19 7:32 AM, Andrew Dinn wrote: > On 25/09/2019 13:31, Reingruber, Richard wrote: >> > The terminology clarification is simply that - a clarification so that >> > when the spec says "heap" it means "Java heap", when it says "Thread" it >> > means "Java thread" etc without having to spell it out each time. I do >> > not read this as meaning anything about an "abstract virtual machine" >> > state, it is about the Java Virtual Machine state - which to mean can be >> > the concrete state as implemented by Hotspot, not some abstract >> > conceptual model state. >> >> It is more than that. It is the glue to the JVM and Java Language specifications. This is important >> to accept. Otherwise I would suggest to do your java debugging with gdb and Intel (or AMD??) x86 >> manuals at your hand. >> >> > We'll just have to agree to disagree here. >> >> Agreed. All I do is offer my points to our audience hoping for a few '+1' ;) > Sorry, but I'm going to throw in a -1 here, not on any grounds of > textual exegesis of the JVM spec rather those of simple pragmatism -- > grounds that David already alluded to in his mention of finalization. > > The path you want to go down is to force disabling of optimizations as > needed in order to ensure that JVMTI agents see a 'pure' version of the > /abstract/ machine state as defined in the Java Virtual Machine spec. > This is in many ways laudable: > > 1) such a 'pure' view is one that can be mapped fairly directly back to > the original Java application code state (assuming you factor in enough > knowledge of how javac translates code to bytecode and relying on the > fact that javac does not do anything very tricksy by way of optimization > during that translation) i.e. it is clear. > > 2) this pure view is going to be unchanged by whatever tricks the JIT > and/or JVM might have up their sleeves i.e. it is consistent/repeatable. > > That latter point of consistency is, perhaps, more significant than the > former one of clarity since for most programmers bytecode is only ever > observed 'through a glass darkly'. > > The downside comes when you consider why you might want to use an agent. > Primarily, JVMTI was provided for monitoring actual program execution. > Agents can also be used to modify program execution, making program > transformations that result in locally and/or globally variant > behaviour. Well, actually, I guess some monitoring operations falls into > that latter camp. > > In order to pursue your proposed path it is going to be necessary to > disable, either piecemeal or wholesale, a variety of JIT optimizations. > This would mean that use of an agent might involve a cost that is > prohibitive. > > Primarily, that's a cost in performance (whether measured in space or > time). When it comes to monitoring it is also the extra cost in the > accuracy of any report of what is being monitored -- if installation of > an agent or use of a montoring capability means you run different code > in a different way then you can end up measuring apples not pears. > > That possible inaccuracy has to be weighed against other inaccuracies > that arise when the JIT invalidates or changes notifications of all the > events the idealized version of the abstract machine would imply. Where > the balance lies depends on the specifics of which optimizations are > disabled and the needs/behaviour of the app. > > I believe that was what David was talking about when he mentioned as an > example the case of finalizers being run 'early' for non-reachable > objects. The optimization here allows early reclamation of objects and > reuse of stack slots. The cost is that object appear to go out of scope > 'before' their fields are read -- at least as far as the source/abstract > machine view of 'before' is defined. How much that cost imposes on users > is unclear but the cost was discounted because finalization is already a > highly unreliable mechanism -- i.e. who cares? > > So, I agree with David's general argument that we have to be pragmatic > about these things as he said taking into account > > " - how important it would be to accurately interact with things as > expressed at the source level > - the cost of disabling or reversing the optimisation when needed; and > - the overall cost and complexity it adds to the development and > maintenance of the JVM" > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From serguei.spitsyn at oracle.com Wed Sep 25 17:09:48 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 25 Sep 2019 10:09:48 -0700 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From hohensee at amazon.com Wed Sep 25 17:51:29 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Wed, 25 Sep 2019 17:51:29 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> Message-ID: <4039406E-2936-45A5-AEE0-46658B6CF40C@amazon.com> In the interest of getting this patch pushed, can I get confirmation to leave it as is with JMM_VERSION_3 = 0x2003000? It's not used anywhere yet. Imo best to file a new issue to change to the new scheme, which would redefine JMM_VERSION_3 = JMM_VERSION_14 = 0x200E0000. JMM_VERSION_2 would get changed to JMM_VERSION_10 = 0x200A0000. Mikael, I'll file an issue upon your request. ?On 9/24/19, 1:32 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: I can change 0x20030000 to 0x200E0000 to conform to that. Let me know what's wanted. Paul On 9/24/19, 12:46 PM, "serguei.spitsyn at oracle.com" wrote: Hi Paul and Mandy, There was a move from Mikael Vidsted to unify the versioning. The idea is to keep versions in sync with the JDK versions. Now, for JVMTI we have (see, https://bugs.openjdk.java.net/browse/JDK-8219023): enum { JVMTI_VERSION_1 = 0x30010000, JVMTI_VERSION_1_0 = 0x30010000, JVMTI_VERSION_1_1 = 0x30010100, JVMTI_VERSION_1_2 = 0x30010200, JVMTI_VERSION_9 = 0x30090000, JVMTI_VERSION_11 = 0x300B0000, JVMTI_VERSION = 0x30000000 + (14 * 0x10000) + ( 0 * 0x100) + 0 /* version: 14.0.0 */ }; Also, it was decided to auto bump the JVMTI and JDWP versions for new releases no matter the spec was changed or not. I've added Mikael to the cc-list. Thanks, Serguei On 9/24/19 9:58 AM, Hohensee, Paul wrote: > Do we want to increment the JMM version by one whenever we change the spec regardless of JDK version? If so, we'd have > > JMM_VERSION_2 = 0x20020000, // JDK 10 > JMM_VERSION_3 = 0x20030000, // JDK 14 > JMM_VERSION = JMM_VERSION_3 > > On 9/24/19, 9:54 AM, "Mandy Chung" wrote: > > On 9/24/19 9:51 AM, Hohensee, Paul wrote: > > I did JBS queries for, e.g., > > > > Issuetype=CSR AND Subcomponent="java.lang.management" AND fixVersion=11 > > > > For 11, I got > > > > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP agent > > https://bugs.openjdk.java.net/browse/JDK-8198732: ThreadInfo.from(CompositeData) does not throw IAE when the given CompositeData with missing attributes > > These two issues did not change jmm interface and no need to rev the > version. > > Mandy > > > 8199295 seems to me to justify a minor version for 11. There weren't any management CSRs for 12 and 13 (I also checked for 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked > > > > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 and their updates> > > > > I didn't find any spec changes for 8 and 9, so we would have > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 11 > > JMM_VERSION_2_2 = 0x20020200, // JDK 14 > > JMM_VERSION = JMM_VERSION_2_2 > > > > Reasonable? > > > > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" wrote: > > > > $ hg annot src/hotspot/share/include/jmm.h | grep JMM_VERSION_2 > > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 > > > > $ hg log -r 47592 > > changeset: 47592:68d46cb9be45 > > user: uvangapally > > date: Thu Oct 05 01:31:53 2017 -0700 > > summary: 8185003: JMX: Add a version of ThreadMXBean.dumpAllThreads > > with a maxDepth argument > > > > 8185003 was fixed in JDK10-B31. > > > > Dunno about the other releases... > > > > Dan > > > > > > On 9/24/19 11:45 AM, Hohensee, Paul wrote: > > > Good idea. The current definition is > > > > > > enum { > > > JMM_VERSION_1 = 0x20010000, > > > JMM_VERSION_1_0 = 0x20010000, > > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 > > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 > > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA > > > JMM_VERSION_1_2_2 = 0x20010202, > > > JMM_VERSION_2 = 0x20020000, // JDK 10 > > > JMM_VERSION = 0x20020000 > > > }; > > > > > > Were there changes in 11, 12, and 13 to justify adding major/minor versions for any of them? What was changed in 10 to justify a new major version? > > > > > > Absent any other additions, would this work? It creates a minor version for 14. > > > > > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 > > > JMM_VERSION = JMM_VERSION_2_1 > > > > > > Thanks, > > > > > > Paul > > > > > > On 9/23/19, 8:42 PM, "Mandy Chung" wrote: > > > > > > Good question. > > > > > > When HS express (mix-n-matched JDK and HS version) was supported, the > > > JMM_VERSION was rev'ed to enable the version checking. HS express is no > > > longer supported. JDK is supported to run with this version of HotSpot > > > VM. OTOH, this adds a new function in the middle of the function > > > table. I think it's a good convention to follow and bump the version > > > number. > > > > > > Mandy > > > > > > On 9/23/19 7:54 PM, Daniil Titov wrote: > > > > Hi Paul, > > > > > > > > I have a question about JMM_VERSION. Since the changeset introduces a new method in the interface > > > > should not JMM_VERSION declared in src/hotspot/share/include/jmm.h be bumped? > > > > > > > > Thank you, > > > > --Daniil > > > > > > > > On 9/23/19, 5:43 PM, "serviceability-dev on behalf of Hohensee, Paul" wrote: > > > > > > > > Update: > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > CSR: https://bugs.openjdk.java.net/browse/JDK-8231374 > > > > Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.01/ > > > > > > > > All test suites that reference getThreadAllocatedBytes pass. These are > > > > > > > > hotspot/jtreg/vmTestbase/nsk/monitoring (contained the failing test) > > > > jdk/com/sun/management > > > > jdk/jdk/jfr/event/runtime > > > > > > > > Per Mandy, the default getCurrentThreadAllocatedBytes implementation throws a UOE. > > > > > > > > The CSR is a copy of the original, and in addition points out that ThreadMXBean is a PlatformManagedObject, why that's important, and why a default getCurrentThreadAllocatedBytes implementation is necessary. > > > > > > > > I changed the nsk test to make sure that the approach it uses will work with getCurrentThreadAllocatedBytes, which per Mandy is defined as a property. Though I'm happy to remove it if there's a consensus it isn't needed. > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > On 9/19/19, 11:03 PM, "serguei.spitsyn at oracle.com" wrote: > > > > > > > > Hi Paul, > > > > > > > > I have almost the same comments as David: > > > > - the same two spots of changes identified > > > > - the addition of the default method was expected > > > > - the change in test is a surprise (I also doubt, it is really needed) > > > > - new CSR is needed > > > > > > > > > > > > Sorry, I forgot to remind about running the vmTestbase monitoring tests. :( > > > > > > > > Thanks, > > > > Serguei > > > > > > > > > > > > On 9/19/19 16:06, David Holmes wrote: > > > > > Hi Paul, > > > > > > > > > > On 20/09/2019 2:52 am, Hohensee, Paul wrote: > > > > >> More formally, > > > > >> > > > > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231209 > > > > >> Webrev: http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > > > > > > > I'm assuming there are only two changes here: > > > > > > > > > > 1. The new method is now a default method that throws UOE. > > > > > > > > > > That seems fine. > > > > > > > > > > 2. You implemented the new method in the test class. > > > > > > > > > > I don't understand why you did that. The test can't be calling the new > > > > > method. Now that it is a default method we will get past the > > > > > compilation failure that caused the problem. So no change to the test > > > > > should be needed AFAICS. > > > > > > > > > > A new CSR request is needed. Just copy everything across from the old, > > > > > with the updated spec. But please also mention this is a > > > > > PlatformManagedObject in the compatibility discussion. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > >> Thanks, > > > > >> > > > > >> On 9/19/19, 9:44 AM, "serviceability-dev on behalf of Hohensee, > > > > >> Paul" > > > >> hohensee at amazon.com> wrote: > > > > >> > > > > >> Off by 2 error. Changed the subject to reflect 8231209. > > > > >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ > > > > >> Paul > > > > >> On 9/19/19, 6:31 AM, "Daniel D. Daugherty" > > > > >> wrote: > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> The redo bug is 8231209. 8231211 is closed as a dup > > > > >> of 8231210. > > > > >> Dan > > > > >> On 9/19/19 9:17 AM, Hohensee, Paul wrote: > > > > >> > I'll have the default method throw UOE. That's the same as > > > > >> the other default methods do. > > > > >> > > > > > >> > The necessary test fix is in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, > > > > >> which needs a new getCurrentThreadAllocatedBytes method, defined as > > > > >> > > > > > >> > public long getCurrentThreadAllocatedBytes() { > > > > >> > return (Long) > > > > >> invokeMethod("getCurrentThreadAllocatedBytes", > > > > >> > new Object[] { }, > > > > >> > new String[] { }); > > > > >> > } > > > > >> > > > > > >> > With this fix, the 134 tests in > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. > > > > >> Preliminary webrev at > > > > >> > > > > > >> > http://cr.openjdk.java.net/~phh/8231211/webrev.00/ > > > > >> > > > > > >> > Is it worth adding getCurrentThreadAllocatedBytes tests to > > > > >> the > > > > >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes > > > > >> set? > > > > >> > > > > > >> > Paul > > > > >> > > > > > >> > On 9/18/19, 8:16 PM, "David Holmes" > > > > >> wrote: > > > > >> > > > > > >> > On 19/09/2019 12:57 pm, Mandy Chung wrote: > > > > >> > > On 9/18/19 5:00 PM, Hohensee, Paul wrote: > > > > >> > >> They all implement > > > > >> com.sun.management.ThreadMXBean, so adding a > > > > >> > >> getCurrentThreadAllocatedBytes broke them. > > > > >> Potential fix is to give it > > > > >> > >> a default implementation, vis > > > > >> > >> > > > > >> > >> public default long > > > > >> getCurrentThreadAllocatedBytes() { > > > > >> > >> return -1; > > > > >> > >> } > > > > >> > >> > > > > >> > > > > > > >> > > com.sun.management.ThreadMXBean (and other platform > > > > >> MXBeans) is a > > > > >> > > "sealed" interface which should only be implemented > > > > >> by JDK. > > > > >> > > > > > >> > Didn't realize that. I don't recall knowing about > > > > >> PlatformManagedObject. > > > > >> > Sealed types will at least allow this to be enforced, > > > > >> though I have to > > > > >> > wonder what the tests are doing here. > > > > >> > > > > > >> > > Unfortunately we don't have the sealed type feature > > > > >> yet. Yes it needs > > > > >> > > to be a default method. I think it should throw UOE. > > > > >> > > > > > > >> > > * @implSpec > > > > >> > > * The default implementation throws {@code > > > > >> > > UnsupportedOperationException}. > > > > >> > > > > > > >> > > The @throw UOE can make it clear that it does not > > > > >> support current thread > > > > >> > > memory allocation measurement. > > > > >> > > > > > >> > Yes that seems a reasonable default if we don't want > > > > >> this to be > > > > >> > implemented outside the platform. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > Mandy > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Wed Sep 25 18:22:46 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Wed, 25 Sep 2019 11:22:46 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <745A2116-EDB8-4F27-974D-F52F15A89477@oracle.com> Thank you, David, Daniel, Serguei, and Robbin, for reviewing this change! Best regards, Daniil ?On 9/24/19, 11:45 PM, "Robbin Ehn" wrote: Hi Daniil, Looks good, thanks! /Robbin On 9/25/19 12:45 AM, David Holmes wrote: > Looks good to me. > > Thanks, > David > > On 25/09/2019 2:36 am, Daniil Titov wrote: >> Hi Daniel, David and Serguei, >> >> Please review a new version of the fix (webrev.08) that as Daniel suggested >> renames >> ThreadTable to ThreadIdTable (related classes and variables are renamed as >> well) and >> corrects formatting issues. There are no other changes in this webrev.08 >> comparing >> to the previous version webrev.07. >> >> Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> >> Best regards, >> Daniil >> >> ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: >> >> Daniil, >> Thanks for sticking with this project through the many versions. >> Sorry this review is late... >> On 9/19/19 8:30 PM, Daniil Titov wrote: >> > Hi David and Serguei, >> > >> > Please review new version of the fix that includes the changes Serguei >> suggested: >> > 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> > 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> > that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> > >> > The changes are in ThreadTable::lazy_initialize() method only. >> > >> > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> > >> > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> src/hotspot/share/runtime/mutexLocker.hpp >> No comments. >> src/hotspot/share/runtime/mutexLocker.cpp >> No comments. >> src/hotspot/share/runtime/threadSMR.cpp >> L623: MutexLocker ml(Threads_lock); >> L626: if (!thread->is_exiting()) { >> Re: discussion about is_exiting() >> The header comment is pretty clear: >> src/hotspot/share/runtime/thread.hpp: >> // thread has called JavaThread::exit() or is terminated >> bool is_exiting() const; >> is_exiting() might become true right after you have called it, >> but its purpose is to ask the question and not prevent the >> condition from becoming true. As David said, you should consider >> it an optimization. If you happen to see the condition is true, >> then you know that the JavaThread isn't going to be around much >> longer and should act accordingly. >> The is_exiting() implementation is: >> inline bool JavaThread::is_exiting() const { >> // Use load-acquire so that setting of _terminated by >> // JavaThread::exit() is seen more quickly. >> TerminatedTypes l_terminated = (TerminatedTypes) >> OrderAccess::load_acquire((volatile jint *) &_terminated); >> return l_terminated == _thread_exiting || >> check_is_terminated(l_terminated); >> } >> and it depends on the JavaThread's _terminated field value. >> // JavaThread termination support >> enum TerminatedTypes { >> _not_terminated = 0xDEAD - 2, >> _thread_exiting, // >> JavaThread::exit() has been called for this thread >> _thread_terminated, // JavaThread >> is removed from thread list >> _vm_exited // JavaThread >> is still executing native code, but VM is terminated >> // only VM_Exit >> can set _vm_exited >> }; >> so the JavaThread's _terminated field can get set to >> _thread_exiting independent of the Threads_lock, but >> it can't get set to _thread_terminated without the >> Threads_lock. >> So by grabbing the Threads_lock on L623, you make sure >> that ThreadTable::add_thread(java_tid, thread) does not >> add a JavaThread that's not on the ThreadsList. It might >> still become is_exiting() == true right after your >> L626 if (!thread->is_exiting()) { >> but it will still be on the main ThreadsList. And that >> means that when the JavaThread is removed from the main >> ThreadsList, you'll still call: >> L931: ThreadTable::remove_thread(tid); >> L624: // Must be inside the lock to ensure that we don't >> add the thread to the table >> typo: s/the thread/a thread/ >> L633: return thread; >> nit - L633 - indented too far (should be 2 spaces) >> src/hotspot/share/services/threadTable.hpp >> L42: static void lazy_initialize(const ThreadsList *threads); >> nit - put space between '*' the variable: >> static void lazy_initialize(const ThreadsList* threads); >> like you do in your other decls. >> L45: // Lookup and inserts >> Perhaps: // Lookup and list management >> L60-61 - nit - please delete these blank lines. >> src/hotspot/share/services/threadTable.cpp >> L28: #include "runtime/timerTrace.hpp" >> nit - This should be after threadSMR.hpp... (alpha sorted order) >> L39: static const size_t DefaultThreadTableSizeLog = 8; >> nit - your other 'static const' are not CamelCase. Why is this one? >> L45: static ThreadTableHash* volatile _local_table = NULL; >> L50: static volatile size_t _current_size = 0; >> L51: static volatile size_t _items_count = 0; >> nit - can you group the file statics together? (up with L41). >> L60: _tid(tid),_java_thread(java_thread) {} >> nit - space after ',' >> L62 jlong tid() const { return _tid;} >> L63 JavaThread* thread() const {return _java_thread;} >> nit - space before '}' >> nit - space after '{' on L63. >> L70: static uintx get_hash(Value const& value, bool* is_dead) { >> Parameter 'is_dead' is not used. >> L74: static void* allocate_node(size_t size, Value const& value) { >> Parameter 'value' is not used. >> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> Re: discussion about lazy_initialize() racing with >> ThreadsList::find_JavaThread_from_java_tid() >> There's a couple of aspects to these two pieces of code racing >> with each other and racing with new thread creation. Racing with >> new thread creation is the easy one: >> If a new thread isn't added to the ThreadTable by >> ThreadsSMRSupport::add_thread() calling >> ThreadTable::add_thread(), >> then the point in the future where someone calls >> find_JavaThread_from_java_tid() will add it to the table due to >> the linear search when ThreadTable::find_thread_by_tid() >> returns NULL. >> As for multi-threads calling >> ThreadsList::find_JavaThread_from_java_tid() >> at the same time which results in multi-threads in lazy_initialize() >> at the same time... >> - ThreadTable creation will be linear due to ThreadTableCreate_lock. >> After _is_initialized is set to true, then no more callers to >> lazy_initialize() will be in the "if (!_is_initialized)" block. >> - Once the ThreadTable is created, then multi-threads can be >> executing the for-loop to add their ThreadsList entries to >> the ThreadTable. There will be a bit of Threads_lock contention >> as each of the multi-threads tries to add their entries and >> there will be some wasted work since the multi-threads will >> likely have similar ThreadLists. >> Of course, once _is_initialized is set to true, then any caller >> to lazy_initialize() will return quickly and >> ThreadsList::find_JavaThread_from_java_tid() will call >> ThreadTable::find_thread_by_tid(). If the target java_tid isn't >> found, then we do the linear search thing here and add the >> the entry if we find a match in our current ThreadsList. Since >> we're only adding the one here, we only contend for the Threads_lock >> here if we find it. >> If ThreadsList::find_JavaThread_from_java_tid() is called with a >> target java_tid for a JavaThread that was created after the >> ThreadsList object that the caller has in hand for the >> find_JavaThread_from_java_tid() call, then, of course, that >> target 'java_tid' won't be found because the JavaThread was >> added the main ThreadsList _after_ the ThreadsList object was >> created by the caller. Of course, you have to ask where the >> target java_tid value came from since the JavaThread wasn't >> around when the ThreadsList::find_JavaThread_from_java_tid() >> call was made with that target java_tid value... >> L99: // being concurently populated during the initalization. >> Typos? Perhaps: >> // to be concurrently populated during initialization. >> But I think those two comment lines are more appropriate above >> this line: >> L96: MutexLocker ml(ThreadTableCreate_lock); >> L112: // Must be inside the lock to ensure that we don't >> add the thread to the table >> typo: s/the thread/a thread/ >> L141: return ((double)_items_count)/_current_size; >> nit - need spaces around '/'. >> L177: bool equals(ThreadTableEntry **value, bool* is_dead) { >> nit - put space between '**' the variable: >> bool equals(ThreadTableEntry** value, >> Parameter 'is_dead' is not used. >> L214: while(true) { >> nit - space before '('. >> Short version: Thumbs up. >> Longer version: I don't think I've spotted anything other than nits here. >> Mostly I've just looked for multi-threaded races, proper usage of the >> Thread-SMR stuff, and minimal impact in the case where the new >> ThreadsTable is never needed. >> Dan >> P.S. >> ThreadTable is a bit of misnomer. What you really have here is >> a ThreadIdTable, but I'm really late to the code review flow >> with that comment... >> > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> > >> > Thank you! >> > --Daniil >> >> From mandy.chung at oracle.com Wed Sep 25 18:36:58 2019 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 25 Sep 2019 11:36:58 -0700 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> Message-ID: One point to note is that JVM TI, JDI and JDWP are supported interfaces. jmm.h (and jvm.h and possibly others) are private interface between VM and libraries and it has the freedom to make any change (even dropping JMM_VERSION - this can be done in the future). For this patch to move forward, I have no objection if it's preferred to use the same versioning scheme.? Bumping JMM_VERSION for every release is not necessary IMO. Mandy On 9/24/19 12:45 PM, serguei.spitsyn at oracle.com wrote: > Hi Paul and Mandy, > > There was a move from Mikael Vidsted to unify the versioning. > The idea is to keep versions in sync with the JDK versions. > Now, for JVMTI we have (see, > https://bugs.openjdk.java.net/browse/JDK-8219023): > > enum { > ??? JVMTI_VERSION_1?? = 0x30010000, > ??? JVMTI_VERSION_1_0 = 0x30010000, > ??? JVMTI_VERSION_1_1 = 0x30010100, > ??? JVMTI_VERSION_1_2 = 0x30010200, > ??? JVMTI_VERSION_9?? = 0x30090000, > ??? JVMTI_VERSION_11? = 0x300B0000, > > ??? JVMTI_VERSION = 0x30000000 + (14 * 0x10000) + ( 0 * 0x100) + 0 /* > version: 14.0.0 */ > }; > > Also, it was decided to auto bump the JVMTI and JDWP versions > for new releases no matter the spec was changed or not. > > I've added Mikael to the cc-list. > > Thanks, > Serguei > > > On 9/24/19 9:58 AM, Hohensee, Paul wrote: >> Do we want to increment the JMM version by one whenever we change the >> spec regardless of JDK version? If so, we'd have >> >> JMM_VERSION_2? = 0x20020000,? // JDK 10 >> JMM_VERSION_3 = 0x20030000, // JDK 14 >> JMM_VERSION = JMM_VERSION_3 >> >> ?On 9/24/19, 9:54 AM, "Mandy Chung" wrote: >> >> ???? On 9/24/19 9:51 AM, Hohensee, Paul wrote: >> ???? > I did JBS queries for, e.g., >> ???? > >> ???? > Issuetype=CSR AND Subcomponent="java.lang.management" AND >> fixVersion=11 >> ???? > >> ???? > For 11, I got >> ???? > >> ???? > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP >> agent >> ???? > https://bugs.openjdk.java.net/browse/JDK-8198732: >> ThreadInfo.from(CompositeData) does not throw IAE when the given >> CompositeData with missing attributes >> ???? ???? These two issues did not change jmm interface and no need >> to rev the >> ???? version. >> ???? ???? Mandy >> ???? ???? > 8199295 seems to me to justify a minor version for 11. >> There weren't any management CSRs for 12 and 13 (I also checked for >> 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked >> ???? > >> ???? > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 >> and their updates> >> ???? > >> ???? > I didn't find any spec changes for 8 and 9, so we would have >> ???? > >> ???? > JMM_VERSION_2_1 = 0x20020100, // JDK 11 >> ???? > JMM_VERSION_2_2 = 0x20020200, // JDK 14 >> ???? > JMM_VERSION = JMM_VERSION_2_2 >> ???? > >> ???? > Reasonable? >> ???? > >> ???? > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" >> wrote: >> ???? > >> ???? >????? $ hg annot src/hotspot/share/include/jmm.h | grep >> JMM_VERSION_2 >> ???? >????? 47592:?? JMM_VERSION_2? = 0x20020000,? // JDK 10 >> ???? > >> ???? >????? $ hg log -r 47592 >> ???? >????? changeset:?? 47592:68d46cb9be45 >> ???? >????? user:??????? uvangapally >> ???? >????? date:??????? Thu Oct 05 01:31:53 2017 -0700 >> ???? >????? summary:???? 8185003: JMX: Add a version of >> ThreadMXBean.dumpAllThreads >> ???? >????? with a maxDepth argument >> ???? > >> ???? >????? 8185003 was fixed in JDK10-B31. >> ???? > >> ???? >????? Dunno about the other releases... >> ???? > >> ???? >????? Dan >> ???? > >> ???? > >> ???? >????? On 9/24/19 11:45 AM, Hohensee, Paul wrote: >> ???? >????? > Good idea. The current definition is >> ???? >????? > >> ???? >????? > enum { >> ???? >????? >??? JMM_VERSION_1?? = 0x20010000, >> ???? >????? >??? JMM_VERSION_1_0 = 0x20010000, >> ???? >????? >??? JMM_VERSION_1_1 = 0x20010100, // JDK 6 >> ???? >????? >??? JMM_VERSION_1_2 = 0x20010200, // JDK 7 >> ???? >????? >??? JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA >> ???? >????? >??? JMM_VERSION_1_2_2 = 0x20010202, >> ???? >????? >??? JMM_VERSION_2? = 0x20020000,? // JDK 10 >> ???? >????? >??? JMM_VERSION???? = 0x20020000 >> ???? >????? > }; >> ???? >????? > >> ???? >????? > Were there changes in 11, 12, and 13 to justify adding >> major/minor versions for any of them? What was changed in 10 to >> justify a new major version? >> ???? >????? > >> ???? >????? > Absent any other additions, would this work? It creates >> a minor version for 14. >> ???? >????? > >> ???? >????? > JMM_VERSION_2_1 = 0x20020100, // JDK 14 >> ???? >????? > JMM_VERSION = JMM_VERSION_2_1 >> ???? >????? > >> ???? >????? > Thanks, >> ???? >????? > >> ???? >????? > Paul >> ???? >????? > >> ???? >????? > On 9/23/19, 8:42 PM, "Mandy Chung" >> wrote: >> ???? >????? > >> ???? >????? >????? Good question. >> ???? >????? > >> ???? >????? >????? When HS express (mix-n-matched JDK and HS version) >> was supported, the >> ???? >????? >????? JMM_VERSION was rev'ed to enable the version >> checking.? HS express is no >> ???? >????? >????? longer supported.? JDK is supported to run with >> this version of HotSpot >> ???? >????? >????? VM.? OTOH, this adds a new function in the middle >> of the function >> ???? >????? >????? table.? I think it's a good convention to follow >> and bump the version >> ???? >????? >????? number. >> ???? >????? > >> ???? >????? >????? Mandy >> ???? >????? > >> ???? >????? >????? On 9/23/19 7:54 PM, Daniil Titov wrote: >> ???? >????? >????? > Hi Paul, >> ???? >????? >????? > >> ???? >????? >????? > I have a question about JMM_VERSION. Since the >> changeset introduces a new method in the interface >> ???? >????? >????? > should not JMM_VERSION? declared in >> src/hotspot/share/include/jmm.h? be bumped? >> ???? >????? >????? > >> ???? >????? >????? > Thank you, >> ???? >????? >????? > --Daniil >> ???? >????? >????? > >> ???? >????? >????? > On 9/23/19, 5:43 PM, "serviceability-dev on >> behalf of Hohensee, Paul" >> > hohensee at amazon.com> wrote: >> ???? >????? >????? > >> ???? >????? >????? >????? Update: >> ???? >????? >????? > >> ???? >????? >????? >????? Bug: >> https://bugs.openjdk.java.net/browse/JDK-8231209 >> ???? >????? >????? >????? CSR: >> https://bugs.openjdk.java.net/browse/JDK-8231374 >> ???? >????? >????? >????? Webrev: >> http://cr.openjdk.java.net/~phh/8231209/webrev.01/ >> ???? >????? >????? > >> ???? >????? >????? >????? All test suites that reference >> getThreadAllocatedBytes pass. These are >> ???? >????? >????? > >> ???? >????? >????? > hotspot/jtreg/vmTestbase/nsk/monitoring >> (contained the failing test) >> ???? >????? >????? >????? jdk/com/sun/management >> ???? >????? >????? >????? jdk/jdk/jfr/event/runtime >> ???? >????? >????? > >> ???? >????? >????? >????? Per Mandy, the default >> getCurrentThreadAllocatedBytes implementation throws a UOE. >> ???? >????? >????? > >> ???? >????? >????? >????? The CSR is a copy of the original, and in >> addition points out that ThreadMXBean is a PlatformManagedObject, why >> that's important, and why a default getCurrentThreadAllocatedBytes >> implementation is necessary. >> ???? >????? >????? > >> ???? >????? >????? >????? I changed the nsk test to make sure that >> the approach it uses will work with getCurrentThreadAllocatedBytes, >> which per Mandy is defined as a property. Though I'm happy to remove >> it if there's a consensus it isn't needed. >> ???? >????? >????? > >> ???? >????? >????? >????? Thanks, >> ???? >????? >????? > >> ???? >????? >????? >????? Paul >> ???? >????? >????? > >> ???? >????? >????? >????? On 9/19/19, 11:03 PM, >> "serguei.spitsyn at oracle.com" wrote: >> ???? >????? >????? > >> ???? >????? >????? >????????? Hi Paul, >> ???? >????? >????? > >> ???? >????? >????? >????????? I have almost the same comments as David: >> ???? >????? >????? >??????????? - the same two spots of changes >> identified >> ???? >????? >????? >??????????? - the addition of the default method >> was expected >> ???? >????? >????? >??????????? - the change in test is a surprise (I >> also doubt, it is really needed) >> ???? >????? >????? >??????????? - new CSR is needed >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? >????????? Sorry, I forgot to remind about running >> the vmTestbase monitoring tests. :( >> ???? >????? >????? > >> ???? >????? >????? >????????? Thanks, >> ???? >????? >????? >????????? Serguei >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? >????????? On 9/19/19 16:06, David Holmes wrote: >> ???? >????? >????? >????????? > Hi Paul, >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > On 20/09/2019 2:52 am, Hohensee, Paul >> wrote: >> ???? >????? >????? >????????? >> More formally, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8231209 >> ???? >????? >????? >????????? >> Webrev: >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > I'm assuming there are only two >> changes here: >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > 1. The new method is now a default >> method that throws UOE. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > That seems fine. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > 2. You implemented the new method in >> the test class. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > I don't understand why you did that. >> The test can't be calling the new >> ???? >????? >????? >????????? > method. Now that it is a default >> method we will get past the >> ???? >????? >????? >????????? > compilation failure that caused the >> problem. So no change to the test >> ???? >????? >????? >????????? > should be needed AFAICS. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > A new CSR request is needed. Just >> copy everything across from the old, >> ???? >????? >????? >????????? > with the updated spec. But please >> also mention this is a >> ???? >????? >????? >????????? > PlatformManagedObject in the >> compatibility discussion. >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? > Thanks, >> ???? >????? >????? >????????? > David >> ???? >????? >????? >????????? > >> ???? >????? >????? >????????? >> Thanks, >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >> On 9/19/19, 9:44 AM, >> "serviceability-dev on behalf of Hohensee, >> ???? >????? >????? >????????? >> Paul" >> > ???? >????? >????? >????????? >> hohensee at amazon.com> wrote: >> ???? >????? >????? >????????? >> >> ???? >????? >????? >????????? >>????? Off by 2 error. Changed the >> subject to reflect 8231209. >> ???? >????? >????? >????????? >> >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> ???? >????? >????? >????????? >>?????????? Paul >> ???? >????? >????? >????????? >>?????????? On 9/19/19, 6:31 AM, >> "Daniel D. Daugherty" >> ???? >????? >????? >????????? >> wrote: >> ???? >????? >????? >????????? >> > >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> ???? >????? >????? > >>?????????????????? The redo bug is 8231209. >> 8231211 is closed as a dup >> ???? >????? >????? >????????? >> of 8231210. >> ???? >????? >????? > >>?????????????????? Dan >> ???? >????? >????? > >>??????????????????????????? On 9/19/19 9:17 >> AM, Hohensee, Paul wrote: >> ???? >????? >????? >????????? >>????????? > I'll have the default >> method throw UOE. That's the same as >> ???? >????? >????? >????????? >> the other default methods do. >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > The necessary test fix is in >> ???? >????? >????? >????????? >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, >> ???? >????? >????? >????????? >> which needs a new >> getCurrentThreadAllocatedBytes method, defined as >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? public long >> getCurrentThreadAllocatedBytes() { >> ???? >????? >????? >????????? >> >????????? return (Long) >> ???? >????? >????? >????????? >> >> invokeMethod("getCurrentThreadAllocatedBytes", >> ???? >????? >????? >????????? >> >????????????? new Object[] { }, >> ???? >????? >????? >????????? >> >????????????? new String[] { }); >> ???? >????? >????? >????????? >> >????? } >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > With this fix, the 134 >> tests in >> ???? >????? >????? >????????? >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. >> ???? >????? >????? >????????? >> Preliminary webrev at >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > Is it worth adding >> getCurrentThreadAllocatedBytes tests to >> ???? >????? >????? >????????? >> the >> ???? >????? >????? >????????? >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes >> ???? >????? >????? >????????? >> set? >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > Paul >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > On 9/18/19, 8:16 PM, >> "David Holmes" >> ???? >????? >????? >????????? >> wrote: >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? On 19/09/2019 12:57 pm, Mandy >> Chung wrote: >> ???? >????? >????? >????????? >> >????? > On 9/18/19 5:00 PM, >> Hohensee, Paul wrote: >> ???? >????? >????? >????????? >> >????? >> They all implement >> ???? >????? >????? >????????? >> com.sun.management.ThreadMXBean, so >> adding a >> ???? >????? >????? >????????? >> >????? >> >> getCurrentThreadAllocatedBytes broke them. >> ???? >????? >????? >????????? >> Potential fix is to give it >> ???? >????? >????? >????????? >> >????? >> a default implementation, vis >> ???? >????? >????? >????????? >> >????? >> >> ???? >????? >????? >????????? >> >????? >>????? public default long >> ???? >????? >????? >????????? >> getCurrentThreadAllocatedBytes() { >> ???? >????? >????? >????????? >> >????? >>????????? return -1; >> ???? >????? >????? >????????? >> >????? >>????? } >> ???? >????? >????? >????????? >> >????? >> >> ???? >????? >????? >????????? >> >????? > >> ???? >????? >????? >????????? >> >????? > >> com.sun.management.ThreadMXBean (and other platform >> ???? >????? >????? >????????? >> MXBeans) is a >> ???? >????? >????? >????????? >> >????? > "sealed" interface which >> should only be implemented >> ???? >????? >????? >????????? >> by JDK. >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? Didn't realize that. I don't >> recall knowing about >> ???? >????? >????? >????????? >> PlatformManagedObject. >> ???? >????? >????? >????????? >> >????? Sealed types will at least >> allow this to be enforced, >> ???? >????? >????? >????????? >> though I have to >> ???? >????? >????? >????????? >> >????? wonder what the tests are >> doing here. >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? > Unfortunately we don't have >> the sealed type feature >> ???? >????? >????? >????????? >> yet.? Yes it needs >> ???? >????? >????? >????????? >> >????? > to be a default method.? I >> think it should throw UOE. >> ???? >????? >????? >????????? >> >????? > >> ???? >????? >????? >????????? >> >????? >?????? * @implSpec >> ???? >????? >????? >????????? >> >????? >?????? * The default >> implementation throws {@code >> ???? >????? >????? >????????? >> >????? > >> UnsupportedOperationException}. >> ???? >????? >????? >????????? >> >????? > >> ???? >????? >????? >????????? >> >????? > The @throw UOE can make it >> clear that it does not >> ???? >????? >????? >????????? >> support current thread >> ???? >????? >????? >????????? >> >????? > memory allocation measurement. >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? Yes that seems a reasonable >> default if we don't want >> ???? >????? >????? >????????? >> this to be >> ???? >????? >????? >????????? >> >????? implemented outside the >> platform. >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? Thanks, >> ???? >????? >????? >????????? >> >????? David >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >????? > Mandy >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >>????????? > >> ???? >????? >????? >????????? >> >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? >????? > >> ???? >????? > >> ???? >????? > >> ???? >????? > >> ???? > >> ???? > >> ???? > >> > From hohensee at amazon.com Wed Sep 25 21:23:45 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Wed, 25 Sep 2019 21:23:45 +0000 Subject: RFR (M): 8231209: [REDO] ThreadMXBean::getThreadAllocatedBytes() can be quicker for self thread In-Reply-To: References: <189D112E-1706-46F3-AC72-30DAFFCE9BB7@amazon.com> <3276c2d1-b189-4387-0a3a-2381454401de@oracle.com> <47FEF1B9-2C49-47D5-AE83-23DC2D2503A3@amazon.com> <7914DA97-E385-4FC1-B0DC-9C2FCBCCE91A@oracle.com> <30ee49ba-8be0-6453-0e1e-8c61c1b990a6@oracle.com> <41E3F6CB-8FF5-4C52-8452-CC6E660FF45D@amazon.com> <5e943f6c-d6e7-7878-31af-867c96fe07d8@oracle.com> <0F8CA728-FEBB-43C9-8E45-411FDAC8A6D5@amazon.com> <8c5e0ebe-6fca-bef8-8f17-0744129b0f74@oracle.com> Message-ID: <89E3E1E9-F0A6-431A-B931-32BDD6C73095@amazon.com> Excellent! David and Mandy have formally approved, so I'll push the current version of the patch. Paul ?On 9/25/19, 11:39 AM, "Mandy Chung" wrote: One point to note is that JVM TI, JDI and JDWP are supported interfaces. jmm.h (and jvm.h and possibly others) are private interface between VM and libraries and it has the freedom to make any change (even dropping JMM_VERSION - this can be done in the future). For this patch to move forward, I have no objection if it's preferred to use the same versioning scheme. Bumping JMM_VERSION for every release is not necessary IMO. Mandy On 9/24/19 12:45 PM, serguei.spitsyn at oracle.com wrote: > Hi Paul and Mandy, > > There was a move from Mikael Vidsted to unify the versioning. > The idea is to keep versions in sync with the JDK versions. > Now, for JVMTI we have (see, > https://bugs.openjdk.java.net/browse/JDK-8219023): > > enum { > JVMTI_VERSION_1 = 0x30010000, > JVMTI_VERSION_1_0 = 0x30010000, > JVMTI_VERSION_1_1 = 0x30010100, > JVMTI_VERSION_1_2 = 0x30010200, > JVMTI_VERSION_9 = 0x30090000, > JVMTI_VERSION_11 = 0x300B0000, > > JVMTI_VERSION = 0x30000000 + (14 * 0x10000) + ( 0 * 0x100) + 0 /* > version: 14.0.0 */ > }; > > Also, it was decided to auto bump the JVMTI and JDWP versions > for new releases no matter the spec was changed or not. > > I've added Mikael to the cc-list. > > Thanks, > Serguei > > > On 9/24/19 9:58 AM, Hohensee, Paul wrote: >> Do we want to increment the JMM version by one whenever we change the >> spec regardless of JDK version? If so, we'd have >> >> JMM_VERSION_2 = 0x20020000, // JDK 10 >> JMM_VERSION_3 = 0x20030000, // JDK 14 >> JMM_VERSION = JMM_VERSION_3 >> >> On 9/24/19, 9:54 AM, "Mandy Chung" wrote: >> >> On 9/24/19 9:51 AM, Hohensee, Paul wrote: >> > I did JBS queries for, e.g., >> > >> > Issuetype=CSR AND Subcomponent="java.lang.management" AND >> fixVersion=11 >> > >> > For 11, I got >> > >> > https://bugs.openjdk.java.net/browse/JDK-8199295: Remove SNMP >> agent >> > https://bugs.openjdk.java.net/browse/JDK-8198732: >> ThreadInfo.from(CompositeData) does not throw IAE when the given >> CompositeData with missing attributes >> These two issues did not change jmm interface and no need >> to rev the >> version. >> Mandy >> > 8199295 seems to me to justify a minor version for 11. >> There weren't any management CSRs for 12 and 13 (I also checked for >> 'component="core-svc"'). CSRs don't exist for 8 and 9, but I checked >> > >> > Subcomponent="java.lang.management" AND fixVersion=<8 and 9 >> and their updates> >> > >> > I didn't find any spec changes for 8 and 9, so we would have >> > >> > JMM_VERSION_2_1 = 0x20020100, // JDK 11 >> > JMM_VERSION_2_2 = 0x20020200, // JDK 14 >> > JMM_VERSION = JMM_VERSION_2_2 >> > >> > Reasonable? >> > >> > On 9/24/19, 9:05 AM, "Daniel D. Daugherty" >> wrote: >> > >> > $ hg annot src/hotspot/share/include/jmm.h | grep >> JMM_VERSION_2 >> > 47592: JMM_VERSION_2 = 0x20020000, // JDK 10 >> > >> > $ hg log -r 47592 >> > changeset: 47592:68d46cb9be45 >> > user: uvangapally >> > date: Thu Oct 05 01:31:53 2017 -0700 >> > summary: 8185003: JMX: Add a version of >> ThreadMXBean.dumpAllThreads >> > with a maxDepth argument >> > >> > 8185003 was fixed in JDK10-B31. >> > >> > Dunno about the other releases... >> > >> > Dan >> > >> > >> > On 9/24/19 11:45 AM, Hohensee, Paul wrote: >> > > Good idea. The current definition is >> > > >> > > enum { >> > > JMM_VERSION_1 = 0x20010000, >> > > JMM_VERSION_1_0 = 0x20010000, >> > > JMM_VERSION_1_1 = 0x20010100, // JDK 6 >> > > JMM_VERSION_1_2 = 0x20010200, // JDK 7 >> > > JMM_VERSION_1_2_1 = 0x20010201, // JDK 7 GA >> > > JMM_VERSION_1_2_2 = 0x20010202, >> > > JMM_VERSION_2 = 0x20020000, // JDK 10 >> > > JMM_VERSION = 0x20020000 >> > > }; >> > > >> > > Were there changes in 11, 12, and 13 to justify adding >> major/minor versions for any of them? What was changed in 10 to >> justify a new major version? >> > > >> > > Absent any other additions, would this work? It creates >> a minor version for 14. >> > > >> > > JMM_VERSION_2_1 = 0x20020100, // JDK 14 >> > > JMM_VERSION = JMM_VERSION_2_1 >> > > >> > > Thanks, >> > > >> > > Paul >> > > >> > > On 9/23/19, 8:42 PM, "Mandy Chung" >> wrote: >> > > >> > > Good question. >> > > >> > > When HS express (mix-n-matched JDK and HS version) >> was supported, the >> > > JMM_VERSION was rev'ed to enable the version >> checking. HS express is no >> > > longer supported. JDK is supported to run with >> this version of HotSpot >> > > VM. OTOH, this adds a new function in the middle >> of the function >> > > table. I think it's a good convention to follow >> and bump the version >> > > number. >> > > >> > > Mandy >> > > >> > > On 9/23/19 7:54 PM, Daniil Titov wrote: >> > > > Hi Paul, >> > > > >> > > > I have a question about JMM_VERSION. Since the >> changeset introduces a new method in the interface >> > > > should not JMM_VERSION declared in >> src/hotspot/share/include/jmm.h be bumped? >> > > > >> > > > Thank you, >> > > > --Daniil >> > > > >> > > > On 9/23/19, 5:43 PM, "serviceability-dev on >> behalf of Hohensee, Paul" >> > hohensee at amazon.com> wrote: >> > > > >> > > > Update: >> > > > >> > > > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8231209 >> > > > CSR: >> https://bugs.openjdk.java.net/browse/JDK-8231374 >> > > > Webrev: >> http://cr.openjdk.java.net/~phh/8231209/webrev.01/ >> > > > >> > > > All test suites that reference >> getThreadAllocatedBytes pass. These are >> > > > >> > > > hotspot/jtreg/vmTestbase/nsk/monitoring >> (contained the failing test) >> > > > jdk/com/sun/management >> > > > jdk/jdk/jfr/event/runtime >> > > > >> > > > Per Mandy, the default >> getCurrentThreadAllocatedBytes implementation throws a UOE. >> > > > >> > > > The CSR is a copy of the original, and in >> addition points out that ThreadMXBean is a PlatformManagedObject, why >> that's important, and why a default getCurrentThreadAllocatedBytes >> implementation is necessary. >> > > > >> > > > I changed the nsk test to make sure that >> the approach it uses will work with getCurrentThreadAllocatedBytes, >> which per Mandy is defined as a property. Though I'm happy to remove >> it if there's a consensus it isn't needed. >> > > > >> > > > Thanks, >> > > > >> > > > Paul >> > > > >> > > > On 9/19/19, 11:03 PM, >> "serguei.spitsyn at oracle.com" wrote: >> > > > >> > > > Hi Paul, >> > > > >> > > > I have almost the same comments as David: >> > > > - the same two spots of changes >> identified >> > > > - the addition of the default method >> was expected >> > > > - the change in test is a surprise (I >> also doubt, it is really needed) >> > > > - new CSR is needed >> > > > >> > > > >> > > > Sorry, I forgot to remind about running >> the vmTestbase monitoring tests. :( >> > > > >> > > > Thanks, >> > > > Serguei >> > > > >> > > > >> > > > On 9/19/19 16:06, David Holmes wrote: >> > > > > Hi Paul, >> > > > > >> > > > > On 20/09/2019 2:52 am, Hohensee, Paul >> wrote: >> > > > >> More formally, >> > > > >> >> > > > >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8231209 >> > > > >> Webrev: >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> > > > > >> > > > > I'm assuming there are only two >> changes here: >> > > > > >> > > > > 1. The new method is now a default >> method that throws UOE. >> > > > > >> > > > > That seems fine. >> > > > > >> > > > > 2. You implemented the new method in >> the test class. >> > > > > >> > > > > I don't understand why you did that. >> The test can't be calling the new >> > > > > method. Now that it is a default >> method we will get past the >> > > > > compilation failure that caused the >> problem. So no change to the test >> > > > > should be needed AFAICS. >> > > > > >> > > > > A new CSR request is needed. Just >> copy everything across from the old, >> > > > > with the updated spec. But please >> also mention this is a >> > > > > PlatformManagedObject in the >> compatibility discussion. >> > > > > >> > > > > Thanks, >> > > > > David >> > > > > >> > > > >> Thanks, >> > > > >> >> > > > >> On 9/19/19, 9:44 AM, >> "serviceability-dev on behalf of Hohensee, >> > > > >> Paul" >> > > > > >> hohensee at amazon.com> wrote: >> > > > >> >> > > > >> Off by 2 error. Changed the >> subject to reflect 8231209. >> > > > >> >> http://cr.openjdk.java.net/~phh/8231209/webrev.00/ >> > > > >> Paul >> > > > >> On 9/19/19, 6:31 AM, >> "Daniel D. Daugherty" >> > > > >> wrote: >> > > > >> > >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> > > > >> The redo bug is 8231209. >> 8231211 is closed as a dup >> > > > >> of 8231210. >> > > > >> Dan >> > > > >> On 9/19/19 9:17 >> AM, Hohensee, Paul wrote: >> > > > >> > I'll have the default >> method throw UOE. That's the same as >> > > > >> the other default methods do. >> > > > >> > >> > > > >> > The necessary test fix is in >> > > > >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/share/server/ServerThreadMXBeanNew.java, >> > > > >> which needs a new >> getCurrentThreadAllocatedBytes method, defined as >> > > > >> > >> > > > >> > public long >> getCurrentThreadAllocatedBytes() { >> > > > >> > return (Long) >> > > > >> >> invokeMethod("getCurrentThreadAllocatedBytes", >> > > > >> > new Object[] { }, >> > > > >> > new String[] { }); >> > > > >> > } >> > > > >> > >> > > > >> > With this fix, the 134 >> tests in >> > > > >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean pass. >> > > > >> Preliminary webrev at >> > > > >> > >> > > > >> > >> http://cr.openjdk.java.net/~phh/8231211/webrev.00/ >> > > > >> > >> > > > >> > Is it worth adding >> getCurrentThreadAllocatedBytes tests to >> > > > >> the >> > > > >> >> test/hotspot/jtreg/vmTestbase/nsk/monitoring/ThreadMXBean/GetThreadAllocatedBytes >> > > > >> set? >> > > > >> > >> > > > >> > Paul >> > > > >> > >> > > > >> > On 9/18/19, 8:16 PM, >> "David Holmes" >> > > > >> wrote: >> > > > >> > >> > > > >> > On 19/09/2019 12:57 pm, Mandy >> Chung wrote: >> > > > >> > > On 9/18/19 5:00 PM, >> Hohensee, Paul wrote: >> > > > >> > >> They all implement >> > > > >> com.sun.management.ThreadMXBean, so >> adding a >> > > > >> > >> >> getCurrentThreadAllocatedBytes broke them. >> > > > >> Potential fix is to give it >> > > > >> > >> a default implementation, vis >> > > > >> > >> >> > > > >> > >> public default long >> > > > >> getCurrentThreadAllocatedBytes() { >> > > > >> > >> return -1; >> > > > >> > >> } >> > > > >> > >> >> > > > >> > > >> > > > >> > > >> com.sun.management.ThreadMXBean (and other platform >> > > > >> MXBeans) is a >> > > > >> > > "sealed" interface which >> should only be implemented >> > > > >> by JDK. >> > > > >> > >> > > > >> > Didn't realize that. I don't >> recall knowing about >> > > > >> PlatformManagedObject. >> > > > >> > Sealed types will at least >> allow this to be enforced, >> > > > >> though I have to >> > > > >> > wonder what the tests are >> doing here. >> > > > >> > >> > > > >> > > Unfortunately we don't have >> the sealed type feature >> > > > >> yet. Yes it needs >> > > > >> > > to be a default method. I >> think it should throw UOE. >> > > > >> > > >> > > > >> > > * @implSpec >> > > > >> > > * The default >> implementation throws {@code >> > > > >> > > >> UnsupportedOperationException}. >> > > > >> > > >> > > > >> > > The @throw UOE can make it >> clear that it does not >> > > > >> support current thread >> > > > >> > > memory allocation measurement. >> > > > >> > >> > > > >> > Yes that seems a reasonable >> default if we don't want >> > > > >> this to be >> > > > >> > implemented outside the >> platform. >> > > > >> > >> > > > >> > Thanks, >> > > > >> > David >> > > > >> > >> > > > >> > > Mandy >> > > > >> > >> > > > >> > >> > > > >> >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > >> > >> > From coleen.phillimore at oracle.com Wed Sep 25 21:28:15 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 17:28:15 -0400 Subject: RFR (S) 8225681: vmTestbase/nsk/jvmti/RedefineClasses/StressRedefine fails due a) MT-unsafe modification of inline cache In-Reply-To: <8cf70e41-46a0-27d3-1b74-ef0b94563e4a@oracle.com> References: <8cf70e41-46a0-27d3-1b74-ef0b94563e4a@oracle.com> Message-ID: <19865f51-596b-5782-a763-5dfda29a65e8@oracle.com> Adding serviceability-dev. Coleen On 9/25/19 5:22 PM, coleen.phillimore at oracle.com wrote: > Summary: allow old methods in > CompiledStaticDirectCall::set_to_interpreted > > This is the comment in the bug that describes this race and this fix: > https://bugs.openjdk.java.net/browse/JDK-8225681?focusedCommentId=14278441&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14278441 > > > The rest of the bug and sightings are actually caused by > https://bugs.openjdk.java.net/browse/JDK-8226690, > and this one might have been caused by it also, but the race that Erik > describes is possible as well. > > The s390 code had an exception for callee->is_compiled_lambda_form() > which should probably apply to all the platforms, so the code is the > same on all the platforms with this change. > > Tested with tier1-6. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8225681.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8225681 > > Thanks, > Coleen From coleen.phillimore at oracle.com Wed Sep 25 21:29:07 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 17:29:07 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> Message-ID: Adding serviceability-dev. Coleen On 9/25/19 10:33 AM, coleen.phillimore at oracle.com wrote: > Summary: Dont create nmethod if classes have been redefined since > compilation start. > > The bug was caused by a new nmethod created with an old Method in the > metadata section.? Added verification (which hit on windows) and NSV > in the other place where the method can be replaced in the nmethod > metadata section. > > There are some jvmci changes (to vmStructs_jvmci.cpp) that might be > needed also in the graal compiler. > > Tested with tier1-6 and failing test 100 times. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8226690 > > Thanks, > Coleen From serguei.spitsyn at oracle.com Wed Sep 25 21:56:04 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 25 Sep 2019 14:56:04 -0700 Subject: RFR (S) 8225681: vmTestbase/nsk/jvmti/RedefineClasses/StressRedefine fails due a) MT-unsafe modification of inline cache In-Reply-To: <19865f51-596b-5782-a763-5dfda29a65e8@oracle.com> References: <8cf70e41-46a0-27d3-1b74-ef0b94563e4a@oracle.com> <19865f51-596b-5782-a763-5dfda29a65e8@oracle.com> Message-ID: <6db3af24-251b-283b-3278-a1dd5b9a6326@oracle.com> Hi Coleen, This looks fine to me. Nice unification for all platforms. Thank you for taking care about this issue! Thanks, Serguei On 9/25/19 2:28 PM, coleen.phillimore at oracle.com wrote: > Adding serviceability-dev. > Coleen > > On 9/25/19 5:22 PM, coleen.phillimore at oracle.com wrote: >> Summary: allow old methods in >> CompiledStaticDirectCall::set_to_interpreted >> >> This is the comment in the bug that describes this race and this fix: >> https://bugs.openjdk.java.net/browse/JDK-8225681?focusedCommentId=14278441&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14278441 >> >> >> The rest of the bug and sightings are actually caused by >> https://bugs.openjdk.java.net/browse/JDK-8226690, >> and this one might have been caused by it also, but the race that >> Erik describes is possible as well. >> >> The s390 code had an exception for callee->is_compiled_lambda_form() >> which should probably apply to all the platforms, so the code is the >> same on all the platforms with this change. >> >> Tested with tier1-6. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2019/8225681.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8225681 >> >> Thanks, >> Coleen > From dean.long at oracle.com Wed Sep 25 22:18:46 2019 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 25 Sep 2019 15:18:46 -0700 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> Message-ID: <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> Saving and restoring redefinition_count for compiler replay doesn't make sense to me.? It won't affect the generated code, and we probably shouldn't even be installing/registering replay nmethods. I would remove the ciEnv::dump_replay_data_unsafe() and process_JvmtiExport() changes. dl On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: > Summary: Dont create nmethod if classes have been redefined since > compilation start. > > The bug was caused by a new nmethod created with an old Method in the > metadata section.? Added verification (which hit on windows) and NSV > in the other place where the method can be replaced in the nmethod > metadata section. > > There are some jvmci changes (to vmStructs_jvmci.cpp) that might be > needed also in the graal compiler. > > Tested with tier1-6 and failing test 100 times. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8226690 > > Thanks, > Coleen From serguei.spitsyn at oracle.com Thu Sep 26 00:54:12 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 25 Sep 2019 17:54:12 -0700 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> Message-ID: <842f7b3c-f9a1-9acf-320d-9c1383554b92@oracle.com> Hi Coleen, It looks pretty good to me. I'm not aware much about reply. Thanks, Serguei On 9/25/19 2:29 PM, coleen.phillimore at oracle.com wrote: > Adding serviceability-dev. > Coleen > > On 9/25/19 10:33 AM, coleen.phillimore at oracle.com wrote: >> Summary: Dont create nmethod if classes have been redefined since >> compilation start. >> >> The bug was caused by a new nmethod created with an old Method in the >> metadata section.? Added verification (which hit on windows) and NSV >> in the other place where the method can be replaced in the nmethod >> metadata section. >> >> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >> needed also in the graal compiler. >> >> Tested with tier1-6 and failing test 100 times. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >> >> Thanks, >> Coleen > From coleen.phillimore at oracle.com Thu Sep 26 01:21:20 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 21:21:20 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> Message-ID: <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> I see.? I dumped the redefinition count in the replay data because I saw the other fields were dumped there.? Would they also not affect the generated code? I can remove these changes. Thanks, Coleen On 9/25/19 6:18 PM, dean.long at oracle.com wrote: > Saving and restoring redefinition_count for compiler replay doesn't > make sense to me.? It won't affect the generated code, and we probably > shouldn't even be installing/registering replay nmethods. I would > remove the ciEnv::dump_replay_data_unsafe() and process_JvmtiExport() > changes. > > dl > > On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >> Summary: Dont create nmethod if classes have been redefined since >> compilation start. >> >> The bug was caused by a new nmethod created with an old Method in the >> metadata section.? Added verification (which hit on windows) and NSV >> in the other place where the method can be replaced in the nmethod >> metadata section. >> >> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >> needed also in the graal compiler. >> >> Tested with tier1-6 and failing test 100 times. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >> >> Thanks, >> Coleen > From coleen.phillimore at oracle.com Thu Sep 26 01:23:39 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 21:23:39 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <842f7b3c-f9a1-9acf-320d-9c1383554b92@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <842f7b3c-f9a1-9acf-320d-9c1383554b92@oracle.com> Message-ID: <34860f1a-41a1-82a8-0fa0-904c5cec196d@oracle.com> Thanks Serguei!? I removed the replay code. Coleen On 9/25/19 8:54 PM, serguei.spitsyn at oracle.com wrote: > Hi Coleen, > > It looks pretty good to me. > I'm not aware much about reply. > > Thanks, > Serguei > > > On 9/25/19 2:29 PM, coleen.phillimore at oracle.com wrote: >> Adding serviceability-dev. >> Coleen >> >> On 9/25/19 10:33 AM, coleen.phillimore at oracle.com wrote: >>> Summary: Dont create nmethod if classes have been redefined since >>> compilation start. >>> >>> The bug was caused by a new nmethod created with an old Method in >>> the metadata section.? Added verification (which hit on windows) and >>> NSV in the other place where the method can be replaced in the >>> nmethod metadata section. >>> >>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>> needed also in the graal compiler. >>> >>> Tested with tier1-6 and failing test 100 times. >>> >>> open webrev at >>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>> >>> Thanks, >>> Coleen >> > From coleen.phillimore at oracle.com Thu Sep 26 01:24:19 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 21:24:19 -0400 Subject: RFR (S) 8225681: vmTestbase/nsk/jvmti/RedefineClasses/StressRedefine fails due a) MT-unsafe modification of inline cache In-Reply-To: <6db3af24-251b-283b-3278-a1dd5b9a6326@oracle.com> References: <8cf70e41-46a0-27d3-1b74-ef0b94563e4a@oracle.com> <19865f51-596b-5782-a763-5dfda29a65e8@oracle.com> <6db3af24-251b-283b-3278-a1dd5b9a6326@oracle.com> Message-ID: <56a45e1a-852f-2fa0-d67e-17645d44a49b@oracle.com> Thanks Serguei! Coleen On 9/25/19 5:56 PM, serguei.spitsyn at oracle.com wrote: > Hi Coleen, > > This looks fine to me. > Nice unification for all platforms. > Thank you for taking care about this issue! > > Thanks, > Serguei > > > On 9/25/19 2:28 PM, coleen.phillimore at oracle.com wrote: >> Adding serviceability-dev. >> Coleen >> >> On 9/25/19 5:22 PM, coleen.phillimore at oracle.com wrote: >>> Summary: allow old methods in >>> CompiledStaticDirectCall::set_to_interpreted >>> >>> This is the comment in the bug that describes this race and this fix: >>> https://bugs.openjdk.java.net/browse/JDK-8225681?focusedCommentId=14278441&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14278441 >>> >>> >>> The rest of the bug and sightings are actually caused by >>> https://bugs.openjdk.java.net/browse/JDK-8226690, >>> and this one might have been caused by it also, but the race that >>> Erik describes is possible as well. >>> >>> The s390 code had an exception for callee->is_compiled_lambda_form() >>> which should probably apply to all the platforms, so the code is the >>> same on all the platforms with this change. >>> >>> Tested with tier1-6. >>> >>> open webrev at >>> http://cr.openjdk.java.net/~coleenp/2019/8225681.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8225681 >>> >>> Thanks, >>> Coleen >> > From coleen.phillimore at oracle.com Thu Sep 26 01:29:32 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 25 Sep 2019 21:29:32 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> Message-ID: <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: > > I see.? I dumped the redefinition count in the replay data because I > saw the other fields were dumped there.? Would they also not affect > the generated code? > > I can remove these changes. open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev Rebuilt and retested with the ciReplay tests.? Thank you for looking at the change. Coleen > > Thanks, > Coleen > > On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >> Saving and restoring redefinition_count for compiler replay doesn't >> make sense to me.? It won't affect the generated code, and we >> probably shouldn't even be installing/registering replay nmethods. I >> would remove the ciEnv::dump_replay_data_unsafe() and >> process_JvmtiExport() changes. >> >> dl >> >> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>> Summary: Dont create nmethod if classes have been redefined since >>> compilation start. >>> >>> The bug was caused by a new nmethod created with an old Method in >>> the metadata section.? Added verification (which hit on windows) and >>> NSV in the other place where the method can be replaced in the >>> nmethod metadata section. >>> >>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>> needed also in the graal compiler. >>> >>> Tested with tier1-6 and failing test 100 times. >>> >>> open webrev at >>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>> >>> Thanks, >>> Coleen >> > From dean.long at oracle.com Thu Sep 26 04:01:35 2019 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 25 Sep 2019 21:01:35 -0700 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> Message-ID: On 9/25/19 6:21 PM, coleen.phillimore at oracle.com wrote: > > I see.? I dumped the redefinition count in the replay data because I > saw the other fields were dumped there.? Would they also not affect > the generated code? > I know some like _jvmti_can_access_local_variables can affect the generated code.? See https://github.com/openjdk/jdk/blob/c83c8515ebb4e49fb63c9b896581c9f056268aa0/src/hotspot/share/ci/ciEnv.hpp#L344 dl > I can remove these changes. > > Thanks, > Coleen > > On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >> Saving and restoring redefinition_count for compiler replay doesn't >> make sense to me.? It won't affect the generated code, and we >> probably shouldn't even be installing/registering replay nmethods. I >> would remove the ciEnv::dump_replay_data_unsafe() and >> process_JvmtiExport() changes. >> >> dl >> >> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>> Summary: Dont create nmethod if classes have been redefined since >>> compilation start. >>> >>> The bug was caused by a new nmethod created with an old Method in >>> the metadata section.? Added verification (which hit on windows) and >>> NSV in the other place where the method can be replaced in the >>> nmethod metadata section. >>> >>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>> needed also in the graal compiler. >>> >>> Tested with tier1-6 and failing test 100 times. >>> >>> open webrev at >>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>> >>> Thanks, >>> Coleen >> > From dean.long at oracle.com Thu Sep 26 04:02:25 2019 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 25 Sep 2019 21:02:25 -0700 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> Message-ID: Looks good. dl On 9/25/19 6:29 PM, coleen.phillimore at oracle.com wrote: > > > On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: >> >> I see.? I dumped the redefinition count in the replay data because I >> saw the other fields were dumped there.? Would they also not affect >> the generated code? >> >> I can remove these changes. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev > > Rebuilt and retested with the ciReplay tests.? Thank you for looking > at the change. > Coleen >> >> Thanks, >> Coleen >> >> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>> Saving and restoring redefinition_count for compiler replay doesn't >>> make sense to me.? It won't affect the generated code, and we >>> probably shouldn't even be installing/registering replay nmethods. I >>> would remove the ciEnv::dump_replay_data_unsafe() and >>> process_JvmtiExport() changes. >>> >>> dl >>> >>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>> Summary: Dont create nmethod if classes have been redefined since >>>> compilation start. >>>> >>>> The bug was caused by a new nmethod created with an old Method in >>>> the metadata section.? Added verification (which hit on windows) >>>> and NSV in the other place where the method can be replaced in the >>>> nmethod metadata section. >>>> >>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>>> needed also in the graal compiler. >>>> >>>> Tested with tier1-6 and failing test 100 times. >>>> >>>> open webrev at >>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>> >>>> Thanks, >>>> Coleen >>> >> > From richard.reingruber at sap.com Thu Sep 26 07:36:11 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Thu, 26 Sep 2019 07:36:11 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> Message-ID: Hi Dan and Serguei, The change went through our nightly testing a few times, which includes these tests and many more on all platforms. Thanks, Richard. From: serguei.spitsyn at oracle.com Sent: Mittwoch, 25. September 2019 19:10 To: daniel.daugherty at oracle.com; Reingruber, Richard ; Vladimir Kozlov ; David Holmes ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Dan and Richard, The JVMTI and JDI tests are: vmTestbase_nsk_jvmti, vmTestbase_nsk_jdi and jdk_jdi The tests locations are: open/test/hotspot/jtreg/vmTestbase/nsk/jvmti open/test/hotspot/jtreg/vmTestbase/nsk/jdi open/test/jdk/com/sun/jdi I think, they all have to be in the hs-tier5-rt. Thanks, Serguei On 9/25/19 07:32, Daniel D. Daugherty wrote: Based on the review thread, it looks like Richard has run Tier1 tests on this change. I don't think there are any JVM/TI tests in Tier1. I'm not sure how much compiler testing is done in Tier1, but I do know that the compiler stress testing doesn't kick in until the later tiers (Tier5 or Tier6)... Serguei, with your JVM/TI hat on, what kind of additional testing (if any) do you think we need here? Dan On 9/25/19 6:20 AM, Reingruber, Richard wrote: Thank you Vladimir and also David and Serguei for your Reviews. > May be add comment that it is onload capability and can't be changed during execution. Done. I'll be out-of-office next week. Will push when coming back. Thanks, Richard. -----Original Message----- From: Vladimir Kozlov Sent: Dienstag, 24. September 2019 21:04 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken I read discussion and this change looks good to me. May be add comment that it is onload capability and can't be changed during execution. Thanks, Vladimir On 9/6/19 7:24 AM, Reingruber, Richard wrote: Hi, could I please get reviews for Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to retrieve objects locked by a thread. In terms of escape analysis those references escape and optimizations like scalar replacement become invalid. The runtime currently cannot cope with objects escaping through JVMTI (try included tests). Therefore escape analysis should be disabled if an agent requests the capabilities can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to disable escape analysis, instead optimizations based on escape analysis will be reverted just before objects escape through JVMTI. I've run tier1 tests. Thanks, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8227745 -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Thu Sep 26 08:47:05 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Sep 2019 01:47:05 -0700 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> Message-ID: Hi Coleen, The .02/webrev looks good. You also removed changes in the src/hotspot/share/jvmci/vmStructs_jvmci.cpp. Was it intentional? Thanks, Serguei On 9/25/19 18:29, coleen.phillimore at oracle.com wrote: > > > On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: >> >> I see.? I dumped the redefinition count in the replay data because I >> saw the other fields were dumped there.? Would they also not affect >> the generated code? >> >> I can remove these changes. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev > > Rebuilt and retested with the ciReplay tests.? Thank you for looking > at the change. > Coleen >> >> Thanks, >> Coleen >> >> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>> Saving and restoring redefinition_count for compiler replay doesn't >>> make sense to me.? It won't affect the generated code, and we >>> probably shouldn't even be installing/registering replay nmethods. I >>> would remove the ciEnv::dump_replay_data_unsafe() and >>> process_JvmtiExport() changes. >>> >>> dl >>> >>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>> Summary: Dont create nmethod if classes have been redefined since >>>> compilation start. >>>> >>>> The bug was caused by a new nmethod created with an old Method in >>>> the metadata section.? Added verification (which hit on windows) >>>> and NSV in the other place where the method can be replaced in the >>>> nmethod metadata section. >>>> >>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>>> needed also in the graal compiler. >>>> >>>> Tested with tier1-6 and failing test 100 times. >>>> >>>> open webrev at >>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>> >>>> Thanks, >>>> Coleen >>> >> > From erik.osterlund at oracle.com Thu Sep 26 09:43:43 2019 From: erik.osterlund at oracle.com (erik.osterlund at oracle.com) Date: Thu, 26 Sep 2019 11:43:43 +0200 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> Message-ID: Hi Coleen, Could you please make the counter uint64_t instead? We usually use 64 bit unsigned counters when we don't want to think about overflow. Otherwise I like the approach. Don't need another webrev... This looks good. Thanks, /Erik On 9/26/19 3:29 AM, coleen.phillimore at oracle.com wrote: > > > On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: >> >> I see.? I dumped the redefinition count in the replay data because I >> saw the other fields were dumped there.? Would they also not affect >> the generated code? >> >> I can remove these changes. > > open webrev at http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev > > Rebuilt and retested with the ciReplay tests.? Thank you for looking > at the change. > Coleen >> >> Thanks, >> Coleen >> >> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>> Saving and restoring redefinition_count for compiler replay doesn't >>> make sense to me.? It won't affect the generated code, and we >>> probably shouldn't even be installing/registering replay nmethods. I >>> would remove the ciEnv::dump_replay_data_unsafe() and >>> process_JvmtiExport() changes. >>> >>> dl >>> >>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>> Summary: Dont create nmethod if classes have been redefined since >>>> compilation start. >>>> >>>> The bug was caused by a new nmethod created with an old Method in >>>> the metadata section.? Added verification (which hit on windows) >>>> and NSV in the other place where the method can be replaced in the >>>> nmethod metadata section. >>>> >>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>>> needed also in the graal compiler. >>>> >>>> Tested with tier1-6 and failing test 100 times. >>>> >>>> open webrev at >>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>> >>>> Thanks, >>>> Coleen >>> >> > From coleen.phillimore at oracle.com Thu Sep 26 11:27:17 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 26 Sep 2019 07:27:17 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> Message-ID: <3d6ade0e-82bb-b645-a5ec-1b38bda2b907@oracle.com> On 9/26/19 4:47 AM, serguei.spitsyn at oracle.com wrote: > Hi Coleen, > > The .02/webrev looks good. > You also removed changes in the > src/hotspot/share/jvmci/vmStructs_jvmci.cpp. > Was it intentional? Yes, Gilles said that graal doesn't need the field. Thanks for reviewing. Coleen > > Thanks, > Serguei > > > On 9/25/19 18:29, coleen.phillimore at oracle.com wrote: >> >> >> On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: >>> >>> I see.? I dumped the redefinition count in the replay data because I >>> saw the other fields were dumped there.? Would they also not affect >>> the generated code? >>> >>> I can remove these changes. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev >> >> Rebuilt and retested with the ciReplay tests.? Thank you for looking >> at the change. >> Coleen >>> >>> Thanks, >>> Coleen >>> >>> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>>> Saving and restoring redefinition_count for compiler replay doesn't >>>> make sense to me.? It won't affect the generated code, and we >>>> probably shouldn't even be installing/registering replay nmethods. >>>> I would remove the ciEnv::dump_replay_data_unsafe() and >>>> process_JvmtiExport() changes. >>>> >>>> dl >>>> >>>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>>> Summary: Dont create nmethod if classes have been redefined since >>>>> compilation start. >>>>> >>>>> The bug was caused by a new nmethod created with an old Method in >>>>> the metadata section.? Added verification (which hit on windows) >>>>> and NSV in the other place where the method can be replaced in the >>>>> nmethod metadata section. >>>>> >>>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might >>>>> be needed also in the graal compiler. >>>>> >>>>> Tested with tier1-6 and failing test 100 times. >>>>> >>>>> open webrev at >>>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>>> >>>>> Thanks, >>>>> Coleen >>>> >>> >> > From coleen.phillimore at oracle.com Thu Sep 26 11:42:51 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 26 Sep 2019 07:42:51 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> <71903cf1-d532-ea9e-5c00-cf89652b11ac@oracle.com> Message-ID: <88246737-2bce-8066-b25f-249129afc5c9@oracle.com> On 9/26/19 5:43 AM, erik.osterlund at oracle.com wrote: > Hi Coleen, > > Could you please make the counter uint64_t instead? We usually use 64 > bit unsigned counters when we don't want to think about overflow. > Otherwise I like the approach. Don't need another webrev... This looks > good. You are right. I don't want to think about overflow for int for number of redefinitions.? I changed it to uint64_t and am rerunning tier1 on Oracle platforms just to be sure. http://cr.openjdk.java.net/~coleenp/2019/8226690.03/webrev Thanks, Coleen > > Thanks, > /Erik > > On 9/26/19 3:29 AM, coleen.phillimore at oracle.com wrote: >> >> >> On 9/25/19 9:21 PM, coleen.phillimore at oracle.com wrote: >>> >>> I see.? I dumped the redefinition count in the replay data because I >>> saw the other fields were dumped there.? Would they also not affect >>> the generated code? >>> >>> I can remove these changes. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2019/8226690.02/webrev >> >> Rebuilt and retested with the ciReplay tests.? Thank you for looking >> at the change. >> Coleen >>> >>> Thanks, >>> Coleen >>> >>> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>>> Saving and restoring redefinition_count for compiler replay doesn't >>>> make sense to me.? It won't affect the generated code, and we >>>> probably shouldn't even be installing/registering replay nmethods. >>>> I would remove the ciEnv::dump_replay_data_unsafe() and >>>> process_JvmtiExport() changes. >>>> >>>> dl >>>> >>>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>>> Summary: Dont create nmethod if classes have been redefined since >>>>> compilation start. >>>>> >>>>> The bug was caused by a new nmethod created with an old Method in >>>>> the metadata section.? Added verification (which hit on windows) >>>>> and NSV in the other place where the method can be replaced in the >>>>> nmethod metadata section. >>>>> >>>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might >>>>> be needed also in the graal compiler. >>>>> >>>>> Tested with tier1-6 and failing test 100 times. >>>>> >>>>> open webrev at >>>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>>> >>>>> Thanks, >>>>> Coleen >>>> >>> >> From coleen.phillimore at oracle.com Thu Sep 26 11:43:31 2019 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 26 Sep 2019 07:43:31 -0400 Subject: RFR 8226690: SIGSEGV in MetadataOnStackClosure::do_metadata In-Reply-To: References: <4a945b40-0bf1-5990-ae51-f34bbd54bb0c@oracle.com> <65a6d23a-95d3-a040-f7af-21c6e989ffb1@oracle.com> <0238524d-f13f-8a4a-04c8-195f1d8f6820@oracle.com> Message-ID: On 9/26/19 12:01 AM, dean.long at oracle.com wrote: > On 9/25/19 6:21 PM, coleen.phillimore at oracle.com wrote: >> >> I see.? I dumped the redefinition count in the replay data because I >> saw the other fields were dumped there.? Would they also not affect >> the generated code? >> > > I know some like _jvmti_can_access_local_variables can affect the > generated code.? See > > https://github.com/openjdk/jdk/blob/c83c8515ebb4e49fb63c9b896581c9f056268aa0/src/hotspot/share/ci/ciEnv.hpp#L344 > I see.? Thank you for reviewing the change and your comments. Coleen > > dl >> I can remove these changes. >> >> Thanks, >> Coleen >> >> On 9/25/19 6:18 PM, dean.long at oracle.com wrote: >>> Saving and restoring redefinition_count for compiler replay doesn't >>> make sense to me.? It won't affect the generated code, and we >>> probably shouldn't even be installing/registering replay nmethods. I >>> would remove the ciEnv::dump_replay_data_unsafe() and >>> process_JvmtiExport() changes. >>> >>> dl >>> >>> On 9/25/19 7:33 AM, coleen.phillimore at oracle.com wrote: >>>> Summary: Dont create nmethod if classes have been redefined since >>>> compilation start. >>>> >>>> The bug was caused by a new nmethod created with an old Method in >>>> the metadata section.? Added verification (which hit on windows) >>>> and NSV in the other place where the method can be replaced in the >>>> nmethod metadata section. >>>> >>>> There are some jvmci changes (to vmStructs_jvmci.cpp) that might be >>>> needed also in the graal compiler. >>>> >>>> Tested with tier1-6 and failing test 100 times. >>>> >>>> open webrev at >>>> http://cr.openjdk.java.net/~coleenp/2019/8226690.01/webrev >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8226690 >>>> >>>> Thanks, >>>> Coleen >>> >> > From andrew_m_leonard at uk.ibm.com Thu Sep 26 12:35:08 2019 From: andrew_m_leonard at uk.ibm.com (Andrew Leonard) Date: Thu, 26 Sep 2019 13:35:08 +0100 Subject: RFR JDK-8225474: JDI connector accept fails "Address already in use" with concurrent listeners Message-ID: I think given the complexities involved in trying to make the JDI connector's "thread-safe", i'm thinking now it's probably best leaving this as-is. As Alan pointed out the java doc does not indicate it is meant to be thread safe, and we will review the testcases that we have. Thanks for all the discussion on this. Cheers Andrew Andrew Leonard Java Runtimes Development IBM Hursley IBM United Kingdom Ltd internet email: andrew_m_leonard at uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.reingruber at sap.com Thu Sep 26 14:10:53 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Thu, 26 Sep 2019 14:10:53 +0000 Subject: Should optimizations be observable for JVMTI agents? In-Reply-To: References: <186684c7-ddc4-a298-f11a-a3b425a0122b@oracle.com> <7d0bb4b7-dad7-e7ec-e60a-accea3ce43a0@redhat.com> Message-ID: Vladimir and Andrew, thanks for the feedback. Even if it is -1 so far ;) I think I should clarify what my intend and motivation are. Clearly it is not my intend to hinder optimization in the (potential) presence of JVMTI agents. It probably looks like it, though, as I'm opening all those bugs requesting to disable EA. All the bugs are split off the RFE JDK-8227745. Vladimir has asked me to do so. This way we can discuss the issues I see one-by-one and we're also able to selectively downport conservative fixes, i.e. disabling EA in some scenarios. **My actual intend** is to enable EA independently of JVMTI agents, reconstructing the high level state just before it is inspected. **Main motivation** behind RFE JDK-8227745 is to be able to start a production system in a mode that allows to initiate a debugging session anytime later if necessary or desired. In most cases debugging will never be activated and the production systems should run at the best possible performance while still being ready for debugging. Besides this the enhancement will improve performance when a debugger has attached to the vm. Note that the cost for this is virtually zero when you don't make JVMTI calls. You'll get the full speed-up of EA. Note also that most of the required debuginfo is there already. It is needed to switch from compiled to interpreted execution. If your agent actually performs JVMTI calls, then I would estimate the costs of reconstructing the state, i.e. reallocating/relocking and saving the objects for later deoptimization as relatively small. Also you have to do that just once. The next time the agent inspects the same part of the app state it will be there already. Thanks, Richard. PS: I'll be out-of-office for a week and therefore unresponsive. -----Original Message----- From: Vladimir Kozlov Sent: Mittwoch, 25. September 2019 18:53 To: Andrew Dinn ; Reingruber, Richard ; David Holmes ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: Should optimizations be observable for JVMTI agents? I had very good discussion yesterday with Serguei Spitsyn from serviceability. He said that we may not need one solution. Both cases, monitoring and debugging, are important and we can use different solutions for them. As Andrew said, for monitoring you want to observe the same behavior as in production. I think we should not restrict JIT and others optimizations in this case. I also agree with Richard that JVMTI may report scalar replaced objects and elided monitors in this mode if requested - it is also interesting information. For debugging we may want to see locals and monitors present in compiled code and it is reasonable to avoid optimizations which remove them. But even for debugging we may want to use optimized code as we do when debugging optimized C code (we don't expect to see all locals in such case). May be we should check not which JVMTI capabilities are switched on but for what JVMTI is used for. Or add API for user to ask what he wants. As Richard mentioned in some email compiled frame can't be deoptimized (to get objects and monitors back) if it is not top frame on stack. So we can't rely on deoptimization here. JIT and Runtime need to know what to do with optimizations when agent is attached to JVM. I don't like suggestion for 8230956 to switch off EA when can_tag_objects is on. Based on declaration, can_tag_objects is always on. Which means we will have EA off in all cases including monitoring tools (JFR, managment, profiling) which is not acceptable for me. I share David's concern about missing cases for some JVMTI capabilities which are not checked when JIT decides to use optimizations. And I think it will be troublesome to go through all capabilities old and new to see if we need to add checks. For me as JIT compiler engineer it would be much simpler to ask JVMTI should optimizations be switched off or not and let JVMTI find in which case it is used and make decision instead of current case when JIT checks some JVMTI capabilities. I agree to provide information in compiled code for JVMTI - we do provide information about scalar replaced object already. And I think it would be much easier to document such relation between JIT and JVMTI. Thanks, Vladimir On 9/25/19 7:32 AM, Andrew Dinn wrote: > On 25/09/2019 13:31, Reingruber, Richard wrote: >> > The terminology clarification is simply that - a clarification so that >> > when the spec says "heap" it means "Java heap", when it says "Thread" it >> > means "Java thread" etc without having to spell it out each time. I do >> > not read this as meaning anything about an "abstract virtual machine" >> > state, it is about the Java Virtual Machine state - which to mean can be >> > the concrete state as implemented by Hotspot, not some abstract >> > conceptual model state. >> >> It is more than that. It is the glue to the JVM and Java Language specifications. This is important >> to accept. Otherwise I would suggest to do your java debugging with gdb and Intel (or AMD??) x86 >> manuals at your hand. >> >> > We'll just have to agree to disagree here. >> >> Agreed. All I do is offer my points to our audience hoping for a few '+1' ;) > Sorry, but I'm going to throw in a -1 here, not on any grounds of > textual exegesis of the JVM spec rather those of simple pragmatism -- > grounds that David already alluded to in his mention of finalization. > > The path you want to go down is to force disabling of optimizations as > needed in order to ensure that JVMTI agents see a 'pure' version of the > /abstract/ machine state as defined in the Java Virtual Machine spec. > This is in many ways laudable: > > 1) such a 'pure' view is one that can be mapped fairly directly back to > the original Java application code state (assuming you factor in enough > knowledge of how javac translates code to bytecode and relying on the > fact that javac does not do anything very tricksy by way of optimization > during that translation) i.e. it is clear. > > 2) this pure view is going to be unchanged by whatever tricks the JIT > and/or JVM might have up their sleeves i.e. it is consistent/repeatable. > > That latter point of consistency is, perhaps, more significant than the > former one of clarity since for most programmers bytecode is only ever > observed 'through a glass darkly'. > > The downside comes when you consider why you might want to use an agent. > Primarily, JVMTI was provided for monitoring actual program execution. > Agents can also be used to modify program execution, making program > transformations that result in locally and/or globally variant > behaviour. Well, actually, I guess some monitoring operations falls into > that latter camp. > > In order to pursue your proposed path it is going to be necessary to > disable, either piecemeal or wholesale, a variety of JIT optimizations. > This would mean that use of an agent might involve a cost that is > prohibitive. > > Primarily, that's a cost in performance (whether measured in space or > time). When it comes to monitoring it is also the extra cost in the > accuracy of any report of what is being monitored -- if installation of > an agent or use of a montoring capability means you run different code > in a different way then you can end up measuring apples not pears. > > That possible inaccuracy has to be weighed against other inaccuracies > that arise when the JIT invalidates or changes notifications of all the > events the idealized version of the abstract machine would imply. Where > the balance lies depends on the specifics of which optimizations are > disabled and the needs/behaviour of the app. > > I believe that was what David was talking about when he mentioned as an > example the case of finalizers being run 'early' for non-reachable > objects. The optimization here allows early reclamation of objects and > reuse of stack slots. The cost is that object appear to go out of scope > 'before' their fields are read -- at least as far as the source/abstract > machine view of 'before' is defined. How much that cost imposes on users > is unclear but the cost was discounted because finalization is already a > highly unreliable mechanism -- i.e. who cares? > > So, I agree with David's general argument that we have to be pragmatic > about these things as he said taking into account > > " - how important it would be to accurately interact with things as > expressed at the source level > - the cost of disabling or reversing the optimisation when needed; and > - the overall cost and complexity it adds to the development and > maintenance of the JVM" > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From daniel.daugherty at oracle.com Thu Sep 26 14:28:28 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 26 Sep 2019 10:28:28 -0400 Subject: RFR: 8231294: ZGC: vmTestbase/nsk/jvmti/ResourceExhausted/resexhausted002 fails In-Reply-To: <7af66362-44da-a192-5688-ae1fc1987061@oracle.com> References: <7af66362-44da-a192-5688-ae1fc1987061@oracle.com> Message-ID: <6c76ecbe-75dc-2b50-64ab-5a75167c49cd@oracle.com> Just for the record... JVM/TI belongs to the Serviceability Team so I've added serviceability-dev at ... However, the Runtime team and Serviceability team quite often tag team on JVM/TI issues and lurk on each other's aliases... Dan On 9/26/19 3:14 AM, Per Liden wrote: > (CC:ing hotspot-runtime-dev, as the test belongs to runtime) > > On 9/25/19 10:57 AM, Per Liden wrote: >> The resexhausted002 test uses an 8M heap, which is a bit too small >> for ZGC. Bumping it to 128M to allow the test to pass on ZGC too. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231294 >> Webrev: http://cr.openjdk.java.net/~pliden/8231294/webrev.0 >> >> Testing: Manual testing with G1 and ZGC >> >> /Per From daniel.daugherty at oracle.com Thu Sep 26 14:30:31 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 26 Sep 2019 10:30:31 -0400 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> Message-ID: <06c2643d-4a8e-4b17-db0c-5d0ae5334bbf@oracle.com> I'm good with the testing that you've done. Thanks for closing the loop. Serguei? Dan On 9/26/19 3:36 AM, Reingruber, Richard wrote: > > Hi Dan and Serguei, > > The change went through our nightly testing a few times, which > includes these tests and many more on all platforms. > > Thanks, Richard. > > *From:*serguei.spitsyn at oracle.com > *Sent:* Mittwoch, 25. September 2019 19:10 > *To:* daniel.daugherty at oracle.com; Reingruber, Richard > ; Vladimir Kozlov > ; David Holmes ; > hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > *Subject:* Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI > capability can_get_owned_monitor_info was taken > > Hi Dan and Richard, > > The JVMTI and JDI tests are: > ? vmTestbase_nsk_jvmti, vmTestbase_nsk_jdi and jdk_jdi > > The tests locations are: > open/test/hotspot/jtreg/vmTestbase/nsk/jvmti > open/test/hotspot/jtreg/vmTestbase/nsk/jdi > ? open/test/jdk/com/sun/jdi > > I think, they all have to be in the hs-tier5-rt. > > Thanks, > Serguei > > > On 9/25/19 07:32, Daniel D. Daugherty wrote: > > Based on the review thread, it looks like Richard has run Tier1 > tests on > this change. I don't think there are any JVM/TI tests in Tier1. > I'm not > sure how much compiler testing is done in Tier1, but I do know > that the > compiler stress testing doesn't kick in until the later tiers > (Tier5 or > Tier6)... > > Serguei, with your JVM/TI hat on, what kind of additional testing > (if any) > do you think we need here? > > Dan > > > On 9/25/19 6:20 AM, Reingruber, Richard wrote: > > Thank you Vladimir and also David and Serguei for your Reviews. > > ?? > May be add comment that it is onload capability and can't > be changed during execution. > > Done. > > I'll be out-of-office next week. Will push when coming back. > > Thanks, Richard. > > -----Original Message----- > From: Vladimir Kozlov > > Sent: Dienstag, 24. September 2019 21:04 > To: Reingruber, Richard > ; > hotspot-compiler-dev at openjdk.java.net > ; > serviceability-dev at openjdk.java.net > > Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if > JVMTI capability can_get_owned_monitor_info was taken > > I read discussion and this change looks good to me. > > May be add comment that it is onload capability and can't be > changed during execution. > > Thanks, > Vladimir > > On 9/6/19 7:24 AM, Reingruber, Richard wrote: > > Hi, > > could I please get reviews for > > Webrev: > http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 > > The JVMTI functions GetOwnedMonitorInfo() and > GetOwnedMonitorStackDepthInfo() can be used to > retrieve objects locked by a thread. In terms of escape > analysis those references escape and > optimizations like scalar replacement become invalid. > > The runtime currently cannot cope with objects escaping > through JVMTI (try included > tests). Therefore escape analysis should be disabled if an > agent requests the capabilities > can_get_owned_monitor_info or > can_get_owned_monitor_stack_depth_info. > > This was taken out of JDK-8227745 [1] to make it smaller. > With JDK-8227745 there's no need to > disable escape analysis, instead optimizations based on > escape analysis will be reverted just before > objects escape through JVMTI. > > I've run tier1 tests. > > Thanks, Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8227745 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Thu Sep 26 14:32:49 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Sep 2019 07:32:49 -0700 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: <06c2643d-4a8e-4b17-db0c-5d0ae5334bbf@oracle.com> References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> <06c2643d-4a8e-4b17-db0c-5d0ae5334bbf@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From richard.reingruber at sap.com Thu Sep 26 14:40:53 2019 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Thu, 26 Sep 2019 14:40:53 +0000 Subject: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken In-Reply-To: References: <6a4d8cfc-37c0-cb37-6008-ff9307cbc480@oracle.com> <0829dea0-8bd8-4b70-d190-dac694aa0a25@oracle.com> <06c2643d-4a8e-4b17-db0c-5d0ae5334bbf@oracle.com> Message-ID: Thank you, Dan and Serguei. Richard. From: serguei.spitsyn at oracle.com Sent: Donnerstag, 26. September 2019 16:33 To: daniel.daugherty at oracle.com; Reingruber, Richard ; Vladimir Kozlov ; David Holmes ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken I'm okay with that too. Thanks, Serguei On 9/26/19 07:30, Daniel D. Daugherty wrote: I'm good with the testing that you've done. Thanks for closing the loop. Serguei? Dan On 9/26/19 3:36 AM, Reingruber, Richard wrote: Hi Dan and Serguei, The change went through our nightly testing a few times, which includes these tests and many more on all platforms. Thanks, Richard. From: serguei.spitsyn at oracle.com Sent: Mittwoch, 25. September 2019 19:10 To: daniel.daugherty at oracle.com; Reingruber, Richard ; Vladimir Kozlov ; David Holmes ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken Hi Dan and Richard, The JVMTI and JDI tests are: vmTestbase_nsk_jvmti, vmTestbase_nsk_jdi and jdk_jdi The tests locations are: open/test/hotspot/jtreg/vmTestbase/nsk/jvmti open/test/hotspot/jtreg/vmTestbase/nsk/jdi open/test/jdk/com/sun/jdi I think, they all have to be in the hs-tier5-rt. Thanks, Serguei On 9/25/19 07:32, Daniel D. Daugherty wrote: Based on the review thread, it looks like Richard has run Tier1 tests on this change. I don't think there are any JVM/TI tests in Tier1. I'm not sure how much compiler testing is done in Tier1, but I do know that the compiler stress testing doesn't kick in until the later tiers (Tier5 or Tier6)... Serguei, with your JVM/TI hat on, what kind of additional testing (if any) do you think we need here? Dan On 9/25/19 6:20 AM, Reingruber, Richard wrote: Thank you Vladimir and also David and Serguei for your Reviews. > May be add comment that it is onload capability and can't be changed during execution. Done. I'll be out-of-office next week. Will push when coming back. Thanks, Richard. -----Original Message----- From: Vladimir Kozlov Sent: Dienstag, 24. September 2019 21:04 To: Reingruber, Richard ; hotspot-compiler-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Subject: Re: RFR(S) 8230677: Should disable Escape Analysis if JVMTI capability can_get_owned_monitor_info was taken I read discussion and this change looks good to me. May be add comment that it is onload capability and can't be changed during execution. Thanks, Vladimir On 9/6/19 7:24 AM, Reingruber, Richard wrote: Hi, could I please get reviews for Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230677/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8230677 The JVMTI functions GetOwnedMonitorInfo() and GetOwnedMonitorStackDepthInfo() can be used to retrieve objects locked by a thread. In terms of escape analysis those references escape and optimizations like scalar replacement become invalid. The runtime currently cannot cope with objects escaping through JVMTI (try included tests). Therefore escape analysis should be disabled if an agent requests the capabilities can_get_owned_monitor_info or can_get_owned_monitor_stack_depth_info. This was taken out of JDK-8227745 [1] to make it smaller. With JDK-8227745 there's no need to disable escape analysis, instead optimizations based on escape analysis will be reverted just before objects escape through JVMTI. I've run tier1 tests. Thanks, Richard. [1] https://bugs.openjdk.java.net/browse/JDK-8227745 -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Thu Sep 26 18:06:38 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Sep 2019 11:06:38 -0700 Subject: RFR: 8231294: ZGC: vmTestbase/nsk/jvmti/ResourceExhausted/resexhausted002 fails In-Reply-To: <6c76ecbe-75dc-2b50-64ab-5a75167c49cd@oracle.com> References: <7af66362-44da-a192-5688-ae1fc1987061@oracle.com> <6c76ecbe-75dc-2b50-64ab-5a75167c49cd@oracle.com> Message-ID: <1452ee17-815c-396b-c121-c709c631836e@oracle.com> Dan, thank you for adding the serviceability-dev mailing list. It looks good but has been already pushed. :) Thanks, Serguei On 9/26/19 07:28, Daniel D. Daugherty wrote: > Just for the record... JVM/TI belongs to the Serviceability Team > so I've added serviceability-dev at ... > > However, the Runtime team and Serviceability team quite often tag > team on JVM/TI issues and lurk on each other's aliases... > > Dan > > > On 9/26/19 3:14 AM, Per Liden wrote: >> (CC:ing hotspot-runtime-dev, as the test belongs to runtime) >> >> On 9/25/19 10:57 AM, Per Liden wrote: >>> The resexhausted002 test uses an 8M heap, which is a bit too small >>> for ZGC. Bumping it to 128M to allow the test to pass on ZGC too. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231294 >>> Webrev: http://cr.openjdk.java.net/~pliden/8231294/webrev.0 >>> >>> Testing: Manual testing with G1 and ZGC >>> >>> /Per > From daniel.daugherty at oracle.com Thu Sep 26 18:20:23 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 26 Sep 2019 14:20:23 -0400 Subject: RFR: 8231294: ZGC: vmTestbase/nsk/jvmti/ResourceExhausted/resexhausted002 fails In-Reply-To: <1452ee17-815c-396b-c121-c709c631836e@oracle.com> References: <7af66362-44da-a192-5688-ae1fc1987061@oracle.com> <6c76ecbe-75dc-2b50-64ab-5a75167c49cd@oracle.com> <1452ee17-815c-396b-c121-c709c631836e@oracle.com> Message-ID: <585efd40-6870-743b-d1b3-ed176c3a73c4@oracle.com> Just figured you might want to know that the heap size change was happening... :-) Dan On 9/26/19 2:06 PM, serguei.spitsyn at oracle.com wrote: > Dan, thank you for adding the serviceability-dev mailing list. > It looks good but has been already pushed. :) > > Thanks, > Serguei > > > On 9/26/19 07:28, Daniel D. Daugherty wrote: >> Just for the record... JVM/TI belongs to the Serviceability Team >> so I've added serviceability-dev at ... >> >> However, the Runtime team and Serviceability team quite often tag >> team on JVM/TI issues and lurk on each other's aliases... >> >> Dan >> >> >> On 9/26/19 3:14 AM, Per Liden wrote: >>> (CC:ing hotspot-runtime-dev, as the test belongs to runtime) >>> >>> On 9/25/19 10:57 AM, Per Liden wrote: >>>> The resexhausted002 test uses an 8M heap, which is a bit too small >>>> for ZGC. Bumping it to 128M to allow the test to pass on ZGC too. >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231294 >>>> Webrev: http://cr.openjdk.java.net/~pliden/8231294/webrev.0 >>>> >>>> Testing: Manual testing with G1 and ZGC >>>> >>>> /Per >> > From per.liden at oracle.com Thu Sep 26 20:39:01 2019 From: per.liden at oracle.com (Per Liden) Date: Thu, 26 Sep 2019 22:39:01 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails Message-ID: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 /Per From igor.ignatyev at oracle.com Thu Sep 26 20:58:28 2019 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 26 Sep 2019 13:58:28 -0700 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> References: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> Message-ID: <553F158A-E36E-4F9D-8D21-EE8FC63D0057@oracle.com> Hi Per, wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? Thanks, -- Igor > On Sep 26, 2019, at 1:39 PM, Per Liden wrote: > > Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 > Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 > > /Per From leonid.mesnik at oracle.com Thu Sep 26 21:18:24 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Thu, 26 Sep 2019 14:18:24 -0700 Subject: RFR: 8230942: Support compressed cores in SA tests Message-ID: Hi Some hosts used for JDK testing have customized core dump settings. They compress core files saved in current directory on-the-fly to reduce required disk space. This fix adopt several SA tests, trying to unpack core.pid.gz before test process it with jhsdb. It affects only execution in the case if core.pid.gz files are actually generated. Verified that tests are passed and not skipped anymore on default and new configurations. webrev: http://cr.openjdk.java.net/~lmesnik/8230942/webrev.00/ bug: https://bugs.openjdk.java.net/browse/JDK-8230942 Leonid -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.liden at oracle.com Thu Sep 26 21:32:46 2019 From: per.liden at oracle.com (Per Liden) Date: Thu, 26 Sep 2019 23:32:46 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <553F158A-E36E-4F9D-8D21-EE8FC63D0057@oracle.com> References: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> <553F158A-E36E-4F9D-8D21-EE8FC63D0057@oracle.com> Message-ID: Hi Igor, I don't think it belongs in the problem list, for two reasons: 1) The test doesn't fail because of a bug. It fails because ZGC doesn't currently support that use case. In other words, the test shouldn't be testing something that isn't supposed to work. 2) As far as I know, the test in question is part of rt-tiers, where it's executed with the default GC (G1). So, the problem is typically only visible when doing non-CI testing with ZGC enabled. cheers, Per On 9/26/19 10:58 PM, Igor Ignatyev wrote: > Hi Per, > > wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? > > Thanks, > -- Igor > >> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >> >> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >> >> /Per > From serguei.spitsyn at oracle.com Thu Sep 26 22:20:36 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Sep 2019 15:20:36 -0700 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> References: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> Message-ID: <9d8db88e-f7dc-89ca-a827-c3fa57ce0d16@oracle.com> Hi Per, It looks good and trivial. Thanks, Serguei On 9/26/19 1:39 PM, Per Liden wrote: > Please review this one-liner to disable > vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root > problem is that ZGC doesn't properly respect address space limitations > (RLIMIT_AS). This test will be re-enabled again as part of fixing > https://bugs.openjdk.java.net/browse/JDK-8231552 > > Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 > Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 > > /Per From igor.ignatyev at oracle.com Fri Sep 27 01:49:49 2019 From: igor.ignatyev at oracle.com (Igor Ignatev) Date: Thu, 26 Sep 2019 18:49:49 -0700 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: References: Message-ID: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> Hi Per, > On Sep 26, 2019, at 2:32 PM, Per Liden wrote: > > ?Hi Igor, > > I don't think it belongs in the problem list, for two reasons: > > 1) The test doesn't fail because of a bug. It fails because ZGC doesn't currently support that use case. In other words, the test shouldn't be testing something that isn't supposed to work. Problem lists aren?t only to exclude tests which fail due to bugs, they can be and are used for use cases you described as well. > > 2) As far as I know, the test in question is part of rt-tiers, where it's executed with the default GC (G1). So, the problem is typically only visible when doing non-CI testing with ZGC enabled. I agree that necessity to pass extra make arts whenever you are doing adhoc testing, esp. on a localhost, is cumbersome, but you are expected to do that anyway as some of tests are known to fail only w/ zGC and are in ProblemList-zgc.txt. The main advantage of problem list over @requiers is semi-automatic reminding to re-enable tests when a defect is fixed/feature is implemented, it also sends a clear message that we plan to make a test pass. In compiler, we 1st use @requiers to temporary exclude tests from graal testing, but soon we realized that there are two different meanings:1) test isn't able to pass w/ graal and will never be able to and 2) test can?t pass w/ graal now (b/c of test / product bugs and/or some features aren?t there), but we want to fix product/test and reenable test. we now try to use @requiers for 1 and problem list for 2, so the reasons and expectations are clearer, but there are still some tests which have @requiers which actually should be in problem list, and it?s expensive to go thru all of them to dig out which it should be. So I just don?t want you guys to get the same problems we got. > > cheers, > Per > >> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >> Hi Per, >> wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? >> Thanks, >> -- Igor >>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>> >>> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>> >>> /Per From david.holmes at oracle.com Fri Sep 27 02:00:45 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 27 Sep 2019 12:00:45 +1000 Subject: RFR: 8230942: Support compressed cores in SA tests In-Reply-To: References: Message-ID: Hi Leonid, On 27/09/2019 7:18 am, Leonid Mesnik wrote: > Hi > > Some hosts used for JDK testing have customized core dump settings. They > compress core files saved in current directory on-the-fly to reduce > required disk space. > This fix adopt several SA tests, trying to unpack core.pid.gz before > test process it with jhsdb. It affects only execution in the case if > core.pid.gz files are actually generated. > > Verified that tests are passed and not skipped anymore on default and > new configurations. > > webrev: http://cr.openjdk.java.net/~lmesnik/8230942/webrev.00/ > bug: https://bugs.openjdk.java.net/browse/JDK-8230942 Overall seems fine. I hope it doesn't take too long to do the unzipping. :) A few minor items test/lib/jdk/test/lib/SA/SATestUtils.java + for(File gzCore : gzCores) { Nit: add space after for + } catch (IOException e) { + throw new SkippedException("Not able to unzip core file."); + } Please add the IOException as a cause for the SkippedException so that we have some diagnostics on why it couldn't be unzipped. --- test/hotspot/jtreg/serviceability/sa/TestJmapCore.java 32 import java.io.File; File is already imported at line 46. + SATestUtils.unzipCores(new File(".")); ... File[] cores = new File(".").listFiles((dir, name) -> name.matches(pattern)); Suggest: File pwd = new File("."); SATestUtils.unzipCores(pwd); ... File[] cores = pwd.listFiles((dir, name) -> name.matches(pattern)); and also at line 117: + ": " + String.join(",", pwd.list()) + "."); Thanks, David ----- > Leonid From serguei.spitsyn at oracle.com Fri Sep 27 05:09:58 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Sep 2019 22:09:58 -0700 Subject: RFR JDK-8225474: JDI connector accept fails "Address already in use" with concurrent listeners In-Reply-To: References: Message-ID: Hi Andrew, That is Okay. Thank you for sharing it! Thanks, Serguei On 9/26/19 05:35, Andrew Leonard wrote: > I think given the complexities involved in trying to make the JDI > connector's "thread-safe", i'm thinking now it's probably best leaving > this as-is. As Alan pointed out the java doc does not indicate it is > meant to be thread safe, and we will review the testcases that we have. > Thanks for all the discussion on this. > Cheers > Andrew > > > Andrew Leonard > Java Runtimes Development > IBM Hursley > IBM United Kingdom Ltd > internet email: andrew_m_leonard at uk.ibm.com > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with > number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From per.liden at oracle.com Fri Sep 27 06:16:19 2019 From: per.liden at oracle.com (Per Liden) Date: Fri, 27 Sep 2019 08:16:19 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> Message-ID: <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> Hi Igor, On 9/27/19 3:49 AM, Igor Ignatev wrote: > Hi Per, > >> On Sep 26, 2019, at 2:32 PM, Per Liden wrote: >> >> ?Hi Igor, >> >> I don't think it belongs in the problem list, for two reasons: >> >> 1) The test doesn't fail because of a bug. It fails because ZGC doesn't currently support that use case. In other words, the test shouldn't be testing something that isn't supposed to work. > Problem lists aren?t only to exclude tests which fail due to bugs, they can be and are used for use cases you described as well. >> >> 2) As far as I know, the test in question is part of rt-tiers, where it's executed with the default GC (G1). So, the problem is typically only visible when doing non-CI testing with ZGC enabled. > I agree that necessity to pass extra make arts whenever you are doing adhoc testing, esp. on a localhost, is cumbersome, but you are expected to do that anyway as some of tests are known to fail only w/ zGC and are in ProblemList-zgc.txt. > > The main advantage of problem list over @requiers is semi-automatic reminding to re-enable tests when a defect is fixed/feature is implemented, it also sends a clear message that we plan to make a test pass. > > In compiler, we 1st use @requiers to temporary exclude tests from graal testing, but soon we realized that there are two different meanings:1) test isn't able to pass w/ graal and will never be able to and 2) test can?t pass w/ graal now (b/c of test / product bugs and/or some features aren?t there), but we want to fix product/test and reenable test. we now try to use @requiers for 1 and problem list for 2, so the reasons and expectations are clearer, but there are still some tests which have @requiers which actually should be in problem list, and it?s expensive to go thru all of them to dig out which it should be. So I just don?t want you guys to get the same problems we got. Ok, thanks. I think I see the problem list more of a temporary measure to avoid noise in the CI pipeline, i.e. when the test should work but doesn't. In cases where a test isn't expected to work, I think @requires is better. However, I do see your point, it's not always a black or white call. cheers, Per >> >> cheers, >> Per >> >>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>> Hi Per, >>> wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>> Thanks, >>> -- Igor >>>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>>> >>>> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>> >>>> /Per > From leonid.mesnik at oracle.com Fri Sep 27 06:28:53 2019 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Thu, 26 Sep 2019 23:28:53 -0700 Subject: RFR: 8230942: Support compressed cores in SA tests In-Reply-To: References: Message-ID: <85EA401F-84ED-48F1-B125-539C80EC0B5F@oracle.com> Thanks for feedback. I checked time execution. Tests takes several seconds longer on the hosts which compress cores. Also, tests TestJmapCore.java, TestJmapCoreMetaspace.java are slow and not included in tier1_serviceability anyway. Updated version here. I fixed it accordingly with your comments: http://cr.openjdk.java.net/~lmesnik/8230942/webrev.01/ Leonid > On Sep 26, 2019, at 7:00 PM, David Holmes wrote: > > Hi Leonid, > > On 27/09/2019 7:18 am, Leonid Mesnik wrote: >> Hi >> Some hosts used for JDK testing have customized core dump settings. They compress core files saved in current directory on-the-fly to reduce required disk space. >> This fix adopt several SA tests, trying to unpack core.pid.gz before test process it with jhsdb. It affects only execution in the case if core.pid.gz files are actually generated. >> Verified that tests are passed and not skipped anymore on default and new configurations. >> webrev: http://cr.openjdk.java.net/~lmesnik/8230942/webrev.00/ >> bug: https://bugs.openjdk.java.net/browse/JDK-8230942 > > Overall seems fine. I hope it doesn't take too long to do the unzipping. :) > > A few minor items > > test/lib/jdk/test/lib/SA/SATestUtils.java > > + for(File gzCore : gzCores) { > > Nit: add space after for > > + } catch (IOException e) { > + throw new SkippedException("Not able to unzip core file."); > + } > > Please add the IOException as a cause for the SkippedException so that we have some diagnostics on why it couldn't be unzipped. > > --- > > test/hotspot/jtreg/serviceability/sa/TestJmapCore.java > > 32 import java.io.File; > > File is already imported at line 46. > > + SATestUtils.unzipCores(new File(".")); > ... > File[] cores = new File(".").listFiles((dir, name) -> name.matches(pattern)); > > Suggest: > > File pwd = new File("."); > SATestUtils.unzipCores(pwd); > ... > File[] cores = pwd.listFiles((dir, name) -> name.matches(pattern)); > > and also at line 117: > + ": " + String.join(",", pwd.list()) + "."); > > Thanks, > David > ----- > >> Leonid -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Fri Sep 27 07:04:03 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 27 Sep 2019 17:04:03 +1000 Subject: RFR: 8230942: Support compressed cores in SA tests In-Reply-To: <85EA401F-84ED-48F1-B125-539C80EC0B5F@oracle.com> References: <85EA401F-84ED-48F1-B125-539C80EC0B5F@oracle.com> Message-ID: Looks good Leonid! Thanks, David On 27/09/2019 4:28 pm, Leonid Mesnik wrote: > Thanks for feedback. > > I checked time execution. Tests takes several seconds longer on the > hosts which compress cores. Also, tests > TestJmapCore.java,?TestJmapCoreMetaspace.java?are slow and not included > in tier1_serviceability anyway. > > Updated version here. I fixed it accordingly with your comments: > http://cr.openjdk.java.net/~lmesnik/8230942/webrev.01/ > > Leonid > >> On Sep 26, 2019, at 7:00 PM, David Holmes > > wrote: >> >> Hi Leonid, >> >> On 27/09/2019 7:18 am, Leonid Mesnik wrote: >>> Hi >>> Some hosts used for JDK testing have customized core dump settings. >>> They compress core files saved in current directory on-the-fly to >>> reduce required disk space. >>> This fix adopt several SA tests, trying to unpack core.pid.gz before >>> test process it with jhsdb. It affects only execution in the case if >>> core.pid.gz files are actually generated. >>> Verified that tests are passed and not skipped anymore on default and >>> new configurations. >>> webrev: http://cr.openjdk.java.net/~lmesnik/8230942/webrev.00/ >>> bug: https://bugs.openjdk.java.net/browse/JDK-8230942 >> >> Overall seems fine. I hope it doesn't take too long to do the >> unzipping. :) >> >> A few minor items >> >> test/lib/jdk/test/lib/SA/SATestUtils.java >> >> + ????????for(File gzCore : gzCores) { >> >> Nit: add space after for >> >> + ????????????} catch (IOException e) { >> + ????????????????throw new SkippedException("Not able to unzip core >> file."); >> + ????????????} >> >> Please add the IOException as a cause for the SkippedException so that >> we have some diagnostics on why it couldn't be unzipped. >> >> --- >> >> test/hotspot/jtreg/serviceability/sa/TestJmapCore.java >> >> 32 import java.io.File; >> >> File is already imported at line 46. >> >> + ????????SATestUtils.unzipCores(new File(".")); >> ... >> ?????????File[] cores = new File(".").listFiles((dir, name) -> >> name.matches(pattern)); >> >> Suggest: >> >> File pwd = new File("."); >> SATestUtils.unzipCores(pwd); >> ... >> File[] cores = pwd.listFiles((dir, name) -> name.matches(pattern)); >> >> and also at line 117: >> ????????????????????+ ": " + String.join(",", pwd.list()) + "."); >> >> Thanks, >> David >> ----- >> >>> Leonid > From per.liden at oracle.com Fri Sep 27 08:58:55 2019 From: per.liden at oracle.com (Per Liden) Date: Fri, 27 Sep 2019 10:58:55 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <9d8db88e-f7dc-89ca-a827-c3fa57ce0d16@oracle.com> References: <0564cbfb-7a67-d0b8-2b51-824c4b93166b@oracle.com> <9d8db88e-f7dc-89ca-a827-c3fa57ce0d16@oracle.com> Message-ID: Thanks Serguei! /Per On 9/27/19 12:20 AM, serguei.spitsyn at oracle.com wrote: > Hi Per, > > It looks good and trivial. > > Thanks, > Serguei > > On 9/26/19 1:39 PM, Per Liden wrote: >> Please review this one-liner to disable >> vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root >> problem is that ZGC doesn't properly respect address space limitations >> (RLIMIT_AS). This test will be re-enabled again as part of fixing >> https://bugs.openjdk.java.net/browse/JDK-8231552 >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >> >> /Per > From serguei.spitsyn at oracle.com Fri Sep 27 10:22:02 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 03:22:02 -0700 Subject: RFR: 8230942: Support compressed cores in SA tests In-Reply-To: <85EA401F-84ED-48F1-B125-539C80EC0B5F@oracle.com> References: <85EA401F-84ED-48F1-B125-539C80EC0B5F@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From sgehwolf at redhat.com Fri Sep 27 12:12:29 2019 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 27 Sep 2019 14:12:29 +0200 Subject: [RFR] 8196969: JTreg Failure: serviceability/sa/ClhsdbJstack.java causes NPE Message-ID: <43692bed6880866c06f875a829a9778fbd1fafb1.camel@redhat.com> Hi, Could I please get reviews for this SA fix? The issue only happens intermittently and with -Xcomp. The new regression test reproduces the issue somewhat reliably. I got 10/10 fails for unpatched, but I've seen it pass as well. When the issue happens, PCDesc's getScopeDecodeOffset() returs 0 (DebugInformationRecorder.SERIALIZED_NULL). The current SA code doesn't handle this case and goes on and tries to read ScopeDesc from the DebugInfoReadStream at the bogus offset. From then on, bad things happen. A NPE in StackTrace could be one symptom. The same code in hotspot deals with serialized null differently. It doesn't read from the debug info stream, and manually sets up a reasonable frame. Note decode_body is called from ScopeDesc's constructor where decode_offset might have been set to 0: void ScopeDesc::decode_body() { if (decode_offset() == DebugInformationRecorder::serialized_null) { // This is a sentinel record, which is only relevant to // approximate queries. Decode a reasonable frame. _sender_decode_offset = DebugInformationRecorder::serialized_null; _method = _code->method(); _bci = InvocationEntryBci; _locals_decode_offset = DebugInformationRecorder::serialized_null; _expressions_decode_offset = DebugInformationRecorder::serialized_null; _monitors_decode_offset = DebugInformationRecorder::serialized_null; } else { // decode header DebugInfoReadStream* stream = stream_at(decode_offset()); _sender_decode_offset = stream->read_int(); _method = stream->read_method(); _bci = stream->read_bci(); // decode offsets for body and sender _locals_decode_offset = stream->read_int(); _expressions_decode_offset = stream->read_int(); _monitors_decode_offset = stream->read_int(); } } The proposed patch handles serialized null scopes similar to the hotspot side of things, by returning a null scope. CompiledVFrame already deals with null scopes when in debugging mode. Bug: https://bugs.openjdk.java.net/browse/JDK-8196969 webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8196969/03/webrev/ Testing: tier 1 tests on Linux x86_64 (release/fastdebug). jdk-submit and ran various reproducer tests including 1000 interations of the added regression test. All pass. Thoughts? Thanks, Severin From daniel.daugherty at oracle.com Fri Sep 27 12:48:48 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 27 Sep 2019 08:48:48 -0400 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> Message-ID: <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> On 9/27/19 2:16 AM, Per Liden wrote: > Hi Igor, > > On 9/27/19 3:49 AM, Igor Ignatev wrote: >> Hi Per, >> >>> On Sep 26, 2019, at 2:32 PM, Per Liden wrote: >>> >>> ?Hi Igor, >>> >>> I don't think it belongs in the problem list, for two reasons: >>> >>> 1) The test doesn't fail because of a bug. It fails because ZGC >>> doesn't currently support that use case. In other words, the test >>> shouldn't be testing something that isn't supposed to work. >> Problem lists aren?t only to exclude tests which fail due to bugs, >> they can be and are used for use cases you described as well. >>> >>> 2) As far as I know, the test in question is part of rt-tiers, where >>> it's executed with the default GC (G1). So, the problem is typically >>> only visible when doing non-CI testing with ZGC enabled. >> I agree that necessity to pass extra make arts whenever you are doing >> adhoc testing, esp. on a localhost, is cumbersome, but you are >> expected to do that anyway as some of tests are known to fail only w/ >> zGC and are in ProblemList-zgc.txt. >> >> The main advantage of problem list over @requiers is semi-automatic >> reminding to re-enable tests when a defect is fixed/feature is >> implemented, it also sends a clear message that we plan to make a >> test pass. >> >> In compiler, we 1st use @requiers to temporary exclude tests from >> graal testing, but soon we realized that there are two different >> meanings:1) test isn't able to pass w/ graal and will never be able >> to and 2) test can?t pass w/ graal now (b/c of test / product bugs >> and/or some features aren?t there), but we want to fix product/test >> and reenable test. we now try to use @requiers for 1 and problem list >> for 2, so the reasons and expectations are clearer, but there are >> still some tests which have @requiers which actually should be in >> problem list, and it?s expensive to go thru all of them to dig out >> which it should be. So I just don?t want you guys to get the same >> problems we got. > > Ok, thanks. I think I see the problem list more of a temporary measure > to avoid noise in the CI pipeline, i.e. when the test should work but > doesn't. In cases where a test isn't expected to work, I think > @requires is better. However, I do see your point, it's not always a > black or white call. In your code review invite you say this: > The root problem is that ZGC doesn't properly respect address space > limitations (RLIMIT_AS). To me that says the current behavior is temporary and then you say this: > This test will be re-enabled again as part of fixing > https://bugs.openjdk.java.net/browse/JDK-8231552 and that confirms that the issue is planned to be resolved. I'm not sure that I agree that 8231552 is an RFE rather than a bug, but that's not the point here... To me, both of those sentences would lead to a ProblemListing and not an @requires. Your call... Dan > > cheers, > Per > >>> >>> cheers, >>> Per >>> >>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>> Hi Per, >>>> wouldn't it be better to put this test into zgc-specific problem >>>> list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>> Thanks, >>>> -- Igor >>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>>>> >>>>> Please review this one-liner to disable >>>>> vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root >>>>> problem is that ZGC doesn't properly respect address space >>>>> limitations (RLIMIT_AS). This test will be re-enabled again as >>>>> part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>> >>>>> /Per >> From per.liden at oracle.com Fri Sep 27 13:39:26 2019 From: per.liden at oracle.com (Per Liden) Date: Fri, 27 Sep 2019 15:39:26 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> Message-ID: <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> Hi Dan, The background here is that ZGC previously (by design) required a certain amount of address space to run. So it was a deliberate decision to not support RLIMIT_AS. However, as ZGC's design have progressed, we're now in a position where we could reconsider this (which is why the RFE was filed). In _theory_, we could come to the conclusion that we still don't want to support this and just close the RFE as Won't fix. That's (very) unlikely, but I think that line of thinking shows why sticking this in the problem list is the wrong thing to do here, because that problem list entry would then potentially stay there forever. Until that RFE is pushed, that test should not even attempt to test it with ZGC, because it's not a supported use case. Anyways... ;) cheers, Per On 9/27/19 2:48 PM, Daniel D. Daugherty wrote: [...] > > In your code review invite you say this: > >> The root problem is that ZGC doesn't properly respect address space >> limitations (RLIMIT_AS). > > To me that says the current behavior is temporary and then you say this: > >> This test will be re-enabled again as part of fixing >> https://bugs.openjdk.java.net/browse/JDK-8231552 > > and that confirms that the issue is planned to be resolved. I'm not > sure that I agree that 8231552 is an RFE rather than a bug, but that's > not the point here... > > To me, both of those sentences would lead to a ProblemListing > and not an @requires. Your call... > > Dan > > >> >> cheers, >> Per >> >>>> >>>> cheers, >>>> Per >>>> >>>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>>> Hi Per, >>>>> wouldn't it be better to put this test into zgc-specific problem >>>>> list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>>> Thanks, >>>>> -- Igor >>>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>>>>> >>>>>> Please review this one-liner to disable >>>>>> vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root >>>>>> problem is that ZGC doesn't properly respect address space >>>>>> limitations (RLIMIT_AS). This test will be re-enabled again as >>>>>> part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>>> >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>>> >>>>>> /Per >>> > From daniel.daugherty at oracle.com Fri Sep 27 13:43:56 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 27 Sep 2019 09:43:56 -0400 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> Message-ID: <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> Thanks for the background info. I think that's a more convincing explanation and I'm okay with the change as is. Please add a comment to JDK-8231552 making it clear that vmTestbase/nsk/jvmti/Allocate/alloc001 should be reenabled if/when the bug^H^H^H RFE is fixed. :-) Dan On 9/27/19 9:39 AM, Per Liden wrote: > Hi Dan, > > The background here is that ZGC previously (by design) required a > certain amount of address space to run. So it was a deliberate > decision to not support RLIMIT_AS. However, as ZGC's design have > progressed, we're now in a position where we could reconsider this > (which is why the RFE was filed). In _theory_, we could come to the > conclusion that we still don't want to support this and just close the > RFE as Won't fix. That's (very) unlikely, but I think that line of > thinking shows why sticking this in the problem list is the wrong > thing to do here, because that problem list entry would then > potentially stay there forever. Until that RFE is pushed, that test > should not even attempt to test it with ZGC, because it's not a > supported use case. > > Anyways... ;) > > cheers, > Per > > On 9/27/19 2:48 PM, Daniel D. Daugherty wrote: > [...] >> >> In your code review invite you say this: >> >>> The root problem is that ZGC doesn't properly respect address space >>> limitations (RLIMIT_AS). >> >> To me that says the current behavior is temporary and then you say this: >> >>> This test will be re-enabled again as part of fixing >>> https://bugs.openjdk.java.net/browse/JDK-8231552 >> >> and that confirms that the issue is planned to be resolved. I'm not >> sure that I agree that 8231552 is an RFE rather than a bug, but that's >> not the point here... >> >> To me, both of those sentences would lead to a ProblemListing >> and not an @requires. Your call... >> >> Dan >> >> >>> >>> cheers, >>> Per >>> >>>>> >>>>> cheers, >>>>> Per >>>>> >>>>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>>>> Hi Per, >>>>>> wouldn't it be better to put this test into zgc-specific problem >>>>>> list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>>>> Thanks, >>>>>> -- Igor >>>>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden >>>>>>>> wrote: >>>>>>> >>>>>>> Please review this one-liner to disable >>>>>>> vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root >>>>>>> problem is that ZGC doesn't properly respect address space >>>>>>> limitations (RLIMIT_AS). This test will be re-enabled again as >>>>>>> part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>>>> >>>>>>> /Per >>>> >> From serguei.spitsyn at oracle.com Fri Sep 27 17:58:23 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 10:58:23 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: Hi Daniil, Just notice I did not reply to you. Thank you for the explanation! Have you already pushed this one? Thanks, Serguei On 9/24/19 12:46, Daniil Titov wrote: > Hi Serguei, > > Thank you for reviewing this version of the fix. > >> Just one question about ThreadIdTable::remove_thread(jlong tid). >> What happens if there is no thread with the specified tid in ThreadIdTable? >> Is it possible? > It could be possible when the thread that was started while the thread table > was initializing exits. At this point the thread table is initialized and the thread > tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable > is a correct operation that just leaves the table unchanged. > > src/hotspot/share/services/threadIdTable.cpp > > 233 bool ThreadIdTable::remove_thread(jlong tid) { > 234 assert(_is_initialized, "Thread table is not initialized"); > 235 Thread* thread = Thread::current(); > 236 ThreadIdTableLookup lookup(tid); > 237 return _local_table->remove(thread, lookup); > 238 } > > src/hotspot/share/utilities/concurrentHashTable.hpp > > 422 // Returns true if items was deleted matching LOOKUP_FUNC and > 423 // prior to destruction DELETE_FUNC is called. > 424 template > 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { > 426 return internal_remove(thread, lookup_f, del_f); > 427 } > 428 > 429 // Same without DELETE_FUNC. > 430 template > 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { > 432 return internal_remove(thread, lookup_f, noOp); > 433 } > > src/hotspot/share/utilities/concurrentHashTable.inline.hpp > > 446 inline bool ConcurrentHashTable:: > 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) > 448 { > 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); > 450 assert(bucket->is_locked(), "Must be locked."); > 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); > 452 Node* rem_n = bucket->first(); > 453 bool have_dead = false; > 454 while (rem_n != NULL) { > 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { > 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); > 457 break; > 458 } else { > 459 rem_n_prev = rem_n->next_ptr(); > 460 rem_n = rem_n->next(); > 461 } > 462 } > 463 > 464 bucket->unlock(); > 465 > 466 if (rem_n == NULL) { > 467 return false; > 468 } > > Best regards, > Daniil > > > ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > This version looks good to me. > Thank you for the update! > > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? > > Thanks, > Serguei > > On 9/24/19 9:36 AM, Daniil Titov wrote: > > Hi Daniel, David and Serguei, > > > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > > corrects formatting issues. There are no other changes in this webrev.08 comparing > > to the previous version webrev.07. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > > > Best regards, > > Daniil > > > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > > > Daniil, > > > > Thanks for sticking with this project through the many versions. > > Sorry this review is late... > > > > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > > Hi David and Serguei, > > > > > > Please review new version of the fix that includes the changes Serguei suggested: > > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > > > src/hotspot/share/runtime/mutexLocker.hpp > > No comments. > > > > src/hotspot/share/runtime/mutexLocker.cpp > > No comments. > > > > src/hotspot/share/runtime/threadSMR.cpp > > L623: MutexLocker ml(Threads_lock); > > L626: if (!thread->is_exiting()) { > > Re: discussion about is_exiting() > > > > The header comment is pretty clear: > > > > src/hotspot/share/runtime/thread.hpp: > > > > // thread has called JavaThread::exit() or is terminated > > bool is_exiting() const; > > > > is_exiting() might become true right after you have called it, > > but its purpose is to ask the question and not prevent the > > condition from becoming true. As David said, you should consider > > it an optimization. If you happen to see the condition is true, > > then you know that the JavaThread isn't going to be around much > > longer and should act accordingly. > > > > The is_exiting() implementation is: > > > > inline bool JavaThread::is_exiting() const { > > // Use load-acquire so that setting of _terminated by > > // JavaThread::exit() is seen more quickly. > > TerminatedTypes l_terminated = (TerminatedTypes) > > OrderAccess::load_acquire((volatile jint *) &_terminated); > > return l_terminated == _thread_exiting || > > check_is_terminated(l_terminated); > > } > > > > and it depends on the JavaThread's _terminated field value. > > > > // JavaThread termination support > > enum TerminatedTypes { > > _not_terminated = 0xDEAD - 2, > > _thread_exiting, // > > JavaThread::exit() has been called for this thread > > _thread_terminated, // JavaThread > > is removed from thread list > > _vm_exited // JavaThread > > is still executing native code, but VM is terminated > > // only VM_Exit > > can set _vm_exited > > }; > > > > so the JavaThread's _terminated field can get set to > > _thread_exiting independent of the Threads_lock, but > > it can't get set to _thread_terminated without the > > Threads_lock. > > > > So by grabbing the Threads_lock on L623, you make sure > > that ThreadTable::add_thread(java_tid, thread) does not > > add a JavaThread that's not on the ThreadsList. It might > > still become is_exiting() == true right after your > > > > L626 if (!thread->is_exiting()) { > > > > but it will still be on the main ThreadsList. And that > > means that when the JavaThread is removed from the main > > ThreadsList, you'll still call: > > > > L931: ThreadTable::remove_thread(tid); > > > > L624: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L633: return thread; > > nit - L633 - indented too far (should be 2 spaces) > > > > src/hotspot/share/services/threadTable.hpp > > L42: static void lazy_initialize(const ThreadsList *threads); > > nit - put space between '*' the variable: > > > > static void lazy_initialize(const ThreadsList* threads); > > > > like you do in your other decls. > > > > L45: // Lookup and inserts > > Perhaps: // Lookup and list management > > > > L60-61 - nit - please delete these blank lines. > > > > src/hotspot/share/services/threadTable.cpp > > L28: #include "runtime/timerTrace.hpp" > > nit - This should be after threadSMR.hpp... (alpha sorted order) > > > > L39: static const size_t DefaultThreadTableSizeLog = 8; > > nit - your other 'static const' are not CamelCase. Why is this one? > > > > L45: static ThreadTableHash* volatile _local_table = NULL; > > L50: static volatile size_t _current_size = 0; > > L51: static volatile size_t _items_count = 0; > > nit - can you group the file statics together? (up with L41). > > > > L60: _tid(tid),_java_thread(java_thread) {} > > nit - space after ',' > > > > L62 jlong tid() const { return _tid;} > > L63 JavaThread* thread() const {return _java_thread;} > > nit - space before '}' > > nit - space after '{' on L63. > > > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > > Parameter 'is_dead' is not used. > > > > L74: static void* allocate_node(size_t size, Value const& value) { > > Parameter 'value' is not used. > > > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > Re: discussion about lazy_initialize() racing with > > ThreadsList::find_JavaThread_from_java_tid() > > > > There's a couple of aspects to these two pieces of code racing > > with each other and racing with new thread creation. Racing with > > new thread creation is the easy one: > > > > If a new thread isn't added to the ThreadTable by > > ThreadsSMRSupport::add_thread() calling > > ThreadTable::add_thread(), > > then the point in the future where someone calls > > find_JavaThread_from_java_tid() will add it to the table due to > > the linear search when ThreadTable::find_thread_by_tid() > > returns NULL. > > > > As for multi-threads calling > > ThreadsList::find_JavaThread_from_java_tid() > > at the same time which results in multi-threads in lazy_initialize() > > at the same time... > > > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > > After _is_initialized is set to true, then no more callers to > > lazy_initialize() will be in the "if (!_is_initialized)" block. > > - Once the ThreadTable is created, then multi-threads can be > > executing the for-loop to add their ThreadsList entries to > > the ThreadTable. There will be a bit of Threads_lock contention > > as each of the multi-threads tries to add their entries and > > there will be some wasted work since the multi-threads will > > likely have similar ThreadLists. > > > > Of course, once _is_initialized is set to true, then any caller > > to lazy_initialize() will return quickly and > > ThreadsList::find_JavaThread_from_java_tid() will call > > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > > found, then we do the linear search thing here and add the > > the entry if we find a match in our current ThreadsList. Since > > we're only adding the one here, we only contend for the Threads_lock > > here if we find it. > > > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > > target java_tid for a JavaThread that was created after the > > ThreadsList object that the caller has in hand for the > > find_JavaThread_from_java_tid() call, then, of course, that > > target 'java_tid' won't be found because the JavaThread was > > added the main ThreadsList _after_ the ThreadsList object was > > created by the caller. Of course, you have to ask where the > > target java_tid value came from since the JavaThread wasn't > > around when the ThreadsList::find_JavaThread_from_java_tid() > > call was made with that target java_tid value... > > > > L99: // being concurently populated during the initalization. > > Typos? Perhaps: > > // to be concurrently populated during initialization. > > > > But I think those two comment lines are more appropriate above > > this line: > > > > L96: MutexLocker ml(ThreadTableCreate_lock); > > > > L112: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L141: return ((double)_items_count)/_current_size; > > nit - need spaces around '/'. > > > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > > nit - put space between '**' the variable: > > bool equals(ThreadTableEntry** value, > > > > Parameter 'is_dead' is not used. > > > > L214: while(true) { > > nit - space before '('. > > > > > > Short version: Thumbs up. > > > > Longer version: I don't think I've spotted anything other than nits here. > > Mostly I've just looked for multi-threaded races, proper usage of the > > Thread-SMR stuff, and minimal impact in the case where the new > > ThreadsTable is never needed. > > > > Dan > > > > P.S. > > ThreadTable is a bit of misnomer. What you really have here is > > a ThreadIdTable, but I'm really late to the code review flow > > with that comment... > > > > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thank you! > > > --Daniil > > > > > > > > > > > > From daniil.x.titov at oracle.com Fri Sep 27 18:07:51 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Fri, 27 Sep 2019 11:07:51 -0700 Subject: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: <7DB8CB3A-33CF-4C1D-8C7F-0324FB269A3E@oracle.com> Hi Serguei, Yes, it is already pushed in the repository [1]. [1] https://hg.openjdk.java.net/jdk/jdk/rev/f4abe950c3b0 Best regards, Daniil ?On 9/27/19, 10:58 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, Just notice I did not reply to you. Thank you for the explanation! Have you already pushed this one? Thanks, Serguei On 9/24/19 12:46, Daniil Titov wrote: > Hi Serguei, > > Thank you for reviewing this version of the fix. > >> Just one question about ThreadIdTable::remove_thread(jlong tid). >> What happens if there is no thread with the specified tid in ThreadIdTable? >> Is it possible? > It could be possible when the thread that was started while the thread table > was initializing exits. At this point the thread table is initialized and the thread > tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable > is a correct operation that just leaves the table unchanged. > > src/hotspot/share/services/threadIdTable.cpp > > 233 bool ThreadIdTable::remove_thread(jlong tid) { > 234 assert(_is_initialized, "Thread table is not initialized"); > 235 Thread* thread = Thread::current(); > 236 ThreadIdTableLookup lookup(tid); > 237 return _local_table->remove(thread, lookup); > 238 } > > src/hotspot/share/utilities/concurrentHashTable.hpp > > 422 // Returns true if items was deleted matching LOOKUP_FUNC and > 423 // prior to destruction DELETE_FUNC is called. > 424 template > 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { > 426 return internal_remove(thread, lookup_f, del_f); > 427 } > 428 > 429 // Same without DELETE_FUNC. > 430 template > 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { > 432 return internal_remove(thread, lookup_f, noOp); > 433 } > > src/hotspot/share/utilities/concurrentHashTable.inline.hpp > > 446 inline bool ConcurrentHashTable:: > 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) > 448 { > 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); > 450 assert(bucket->is_locked(), "Must be locked."); > 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); > 452 Node* rem_n = bucket->first(); > 453 bool have_dead = false; > 454 while (rem_n != NULL) { > 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { > 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); > 457 break; > 458 } else { > 459 rem_n_prev = rem_n->next_ptr(); > 460 rem_n = rem_n->next(); > 461 } > 462 } > 463 > 464 bucket->unlock(); > 465 > 466 if (rem_n == NULL) { > 467 return false; > 468 } > > Best regards, > Daniil > > > ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > This version looks good to me. > Thank you for the update! > > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? > > Thanks, > Serguei > > On 9/24/19 9:36 AM, Daniil Titov wrote: > > Hi Daniel, David and Serguei, > > > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > > corrects formatting issues. There are no other changes in this webrev.08 comparing > > to the previous version webrev.07. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > > > Best regards, > > Daniil > > > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > > > Daniil, > > > > Thanks for sticking with this project through the many versions. > > Sorry this review is late... > > > > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > > Hi David and Serguei, > > > > > > Please review new version of the fix that includes the changes Serguei suggested: > > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > > > src/hotspot/share/runtime/mutexLocker.hpp > > No comments. > > > > src/hotspot/share/runtime/mutexLocker.cpp > > No comments. > > > > src/hotspot/share/runtime/threadSMR.cpp > > L623: MutexLocker ml(Threads_lock); > > L626: if (!thread->is_exiting()) { > > Re: discussion about is_exiting() > > > > The header comment is pretty clear: > > > > src/hotspot/share/runtime/thread.hpp: > > > > // thread has called JavaThread::exit() or is terminated > > bool is_exiting() const; > > > > is_exiting() might become true right after you have called it, > > but its purpose is to ask the question and not prevent the > > condition from becoming true. As David said, you should consider > > it an optimization. If you happen to see the condition is true, > > then you know that the JavaThread isn't going to be around much > > longer and should act accordingly. > > > > The is_exiting() implementation is: > > > > inline bool JavaThread::is_exiting() const { > > // Use load-acquire so that setting of _terminated by > > // JavaThread::exit() is seen more quickly. > > TerminatedTypes l_terminated = (TerminatedTypes) > > OrderAccess::load_acquire((volatile jint *) &_terminated); > > return l_terminated == _thread_exiting || > > check_is_terminated(l_terminated); > > } > > > > and it depends on the JavaThread's _terminated field value. > > > > // JavaThread termination support > > enum TerminatedTypes { > > _not_terminated = 0xDEAD - 2, > > _thread_exiting, // > > JavaThread::exit() has been called for this thread > > _thread_terminated, // JavaThread > > is removed from thread list > > _vm_exited // JavaThread > > is still executing native code, but VM is terminated > > // only VM_Exit > > can set _vm_exited > > }; > > > > so the JavaThread's _terminated field can get set to > > _thread_exiting independent of the Threads_lock, but > > it can't get set to _thread_terminated without the > > Threads_lock. > > > > So by grabbing the Threads_lock on L623, you make sure > > that ThreadTable::add_thread(java_tid, thread) does not > > add a JavaThread that's not on the ThreadsList. It might > > still become is_exiting() == true right after your > > > > L626 if (!thread->is_exiting()) { > > > > but it will still be on the main ThreadsList. And that > > means that when the JavaThread is removed from the main > > ThreadsList, you'll still call: > > > > L931: ThreadTable::remove_thread(tid); > > > > L624: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L633: return thread; > > nit - L633 - indented too far (should be 2 spaces) > > > > src/hotspot/share/services/threadTable.hpp > > L42: static void lazy_initialize(const ThreadsList *threads); > > nit - put space between '*' the variable: > > > > static void lazy_initialize(const ThreadsList* threads); > > > > like you do in your other decls. > > > > L45: // Lookup and inserts > > Perhaps: // Lookup and list management > > > > L60-61 - nit - please delete these blank lines. > > > > src/hotspot/share/services/threadTable.cpp > > L28: #include "runtime/timerTrace.hpp" > > nit - This should be after threadSMR.hpp... (alpha sorted order) > > > > L39: static const size_t DefaultThreadTableSizeLog = 8; > > nit - your other 'static const' are not CamelCase. Why is this one? > > > > L45: static ThreadTableHash* volatile _local_table = NULL; > > L50: static volatile size_t _current_size = 0; > > L51: static volatile size_t _items_count = 0; > > nit - can you group the file statics together? (up with L41). > > > > L60: _tid(tid),_java_thread(java_thread) {} > > nit - space after ',' > > > > L62 jlong tid() const { return _tid;} > > L63 JavaThread* thread() const {return _java_thread;} > > nit - space before '}' > > nit - space after '{' on L63. > > > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > > Parameter 'is_dead' is not used. > > > > L74: static void* allocate_node(size_t size, Value const& value) { > > Parameter 'value' is not used. > > > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > Re: discussion about lazy_initialize() racing with > > ThreadsList::find_JavaThread_from_java_tid() > > > > There's a couple of aspects to these two pieces of code racing > > with each other and racing with new thread creation. Racing with > > new thread creation is the easy one: > > > > If a new thread isn't added to the ThreadTable by > > ThreadsSMRSupport::add_thread() calling > > ThreadTable::add_thread(), > > then the point in the future where someone calls > > find_JavaThread_from_java_tid() will add it to the table due to > > the linear search when ThreadTable::find_thread_by_tid() > > returns NULL. > > > > As for multi-threads calling > > ThreadsList::find_JavaThread_from_java_tid() > > at the same time which results in multi-threads in lazy_initialize() > > at the same time... > > > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > > After _is_initialized is set to true, then no more callers to > > lazy_initialize() will be in the "if (!_is_initialized)" block. > > - Once the ThreadTable is created, then multi-threads can be > > executing the for-loop to add their ThreadsList entries to > > the ThreadTable. There will be a bit of Threads_lock contention > > as each of the multi-threads tries to add their entries and > > there will be some wasted work since the multi-threads will > > likely have similar ThreadLists. > > > > Of course, once _is_initialized is set to true, then any caller > > to lazy_initialize() will return quickly and > > ThreadsList::find_JavaThread_from_java_tid() will call > > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > > found, then we do the linear search thing here and add the > > the entry if we find a match in our current ThreadsList. Since > > we're only adding the one here, we only contend for the Threads_lock > > here if we find it. > > > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > > target java_tid for a JavaThread that was created after the > > ThreadsList object that the caller has in hand for the > > find_JavaThread_from_java_tid() call, then, of course, that > > target 'java_tid' won't be found because the JavaThread was > > added the main ThreadsList _after_ the ThreadsList object was > > created by the caller. Of course, you have to ask where the > > target java_tid value came from since the JavaThread wasn't > > around when the ThreadsList::find_JavaThread_from_java_tid() > > call was made with that target java_tid value... > > > > L99: // being concurently populated during the initalization. > > Typos? Perhaps: > > // to be concurrently populated during initialization. > > > > But I think those two comment lines are more appropriate above > > this line: > > > > L96: MutexLocker ml(ThreadTableCreate_lock); > > > > L112: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L141: return ((double)_items_count)/_current_size; > > nit - need spaces around '/'. > > > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > > nit - put space between '**' the variable: > > bool equals(ThreadTableEntry** value, > > > > Parameter 'is_dead' is not used. > > > > L214: while(true) { > > nit - space before '('. > > > > > > Short version: Thumbs up. > > > > Longer version: I don't think I've spotted anything other than nits here. > > Mostly I've just looked for multi-threaded races, proper usage of the > > Thread-SMR stuff, and minimal impact in the case where the new > > ThreadsTable is never needed. > > > > Dan > > > > P.S. > > ThreadTable is a bit of misnomer. What you really have here is > > a ThreadIdTable, but I'm really late to the code review flow > > with that comment... > > > > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thank you! > > > --Daniil > > > > > > > > > > > > From serguei.spitsyn at oracle.com Fri Sep 27 18:08:17 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 11:08:17 -0700 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> Message-ID: <35780c4b-1399-f585-cc41-b4c06a93c45d@oracle.com> On 9/27/19 11:06, Daniel D. Daugherty wrote: > On 9/27/19 1:58 PM, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Just notice I did not reply to you. >> Thank you for the explanation! >> >> Have you already pushed this one? > > Pushed on 2019.09.25 at 1416 ET. It has made it thru Tier7 testing > as of yesterday... Nice. Thanks, Dan! Serguei > > Dan > > >> >> Thanks, >> Serguei >> >> >> On 9/24/19 12:46, Daniil Titov wrote: >>> Hi Serguei, >>> >>> Thank you for reviewing this version of the fix. >>> >>>> ??? Just one question about ThreadIdTable::remove_thread(jlong tid). >>>> ??? What happens if there is no thread with the specified tid in >>>> ThreadIdTable? >>>> ??? Is it possible? >>> It could be possible when the thread that was started while the >>> thread table >>> was initializing exits.? At this point the thread table is >>> initialized and the thread >>> tries to remove itself from it. Removing non-existing? entry from >>> ConcurrentHashTable >>> is a correct operation that just leaves the table unchanged. >>> >>> src/hotspot/share/services/threadIdTable.cpp >>> >>> ??? 233??? bool ThreadIdTable::remove_thread(jlong tid) { >>> ??? 234????? assert(_is_initialized, "Thread table is not >>> initialized"); >>> ??? 235????? Thread* thread = Thread::current(); >>> ??? 236????? ThreadIdTableLookup lookup(tid); >>> ??? 237????? return _local_table->remove(thread, lookup); >>> ??? 238??? } >>> >>> src/hotspot/share/utilities/concurrentHashTable.hpp >>> >>> ?? 422????? // Returns true if items was deleted matching >>> LOOKUP_FUNC and >>> ??? 423????? // prior to destruction DELETE_FUNC is called. >>> ??? 424????? template >>> ??? 425????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, >>> DELETE_FUNC& del_f) { >>> ??? 426??????? return internal_remove(thread, lookup_f, del_f); >>> ??? 427????? } >>> ??? 428 >>> ??? 429????? // Same without DELETE_FUNC. >>> ??? 430????? template >>> ??? 431????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { >>> ??? 432??????? return internal_remove(thread, lookup_f, noOp); >>> ??? 433????? } >>> >>> src/hotspot/share/utilities/concurrentHashTable.inline.hpp >>> >>> ??? 446??? inline bool ConcurrentHashTable:: >>> ??? 447????? internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, >>> DELETE_FUNC& delete_f) >>> ??? 448??? { >>> ??? 449????? Bucket* bucket = get_bucket_locked(thread, >>> lookup_f.get_hash()); >>> ??? 450????? assert(bucket->is_locked(), "Must be locked."); >>> ??? 451????? Node* const volatile * rem_n_prev = bucket->first_ptr(); >>> ??? 452????? Node* rem_n = bucket->first(); >>> ??? 453????? bool have_dead = false; >>> ??? 454????? while (rem_n != NULL) { >>> ??? 455??????? if (lookup_f.equals(rem_n->value(), &have_dead)) { >>> ??? 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); >>> ??? 457????????? break; >>> ??? 458??????? } else { >>> ??? 459????????? rem_n_prev = rem_n->next_ptr(); >>> ??? 460????????? rem_n = rem_n->next(); >>> ??? 461??????? } >>> ??? 462????? } >>> ??? 463 >>> ??? 464????? bucket->unlock(); >>> ??? 465 >>> ??? 466????? if (rem_n == NULL) { >>> ??? 467??????? return false; >>> ??? 468????? } >>> >>> Best regards, >>> Daniil >>> >>> >>> ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" >>> wrote: >>> >>> ???? Hi Daniil, >>> ???? ???? This version looks good to me. >>> ???? Thank you for the update! >>> ???? ???? Just one question about ThreadIdTable::remove_thread(jlong >>> tid). >>> ???? What happens if there is no thread with the specified tid in >>> ThreadIdTable? >>> ???? Is it possible? >>> ???? ???? Thanks, >>> ???? Serguei >>> ???? ???? On 9/24/19 9:36 AM, Daniil Titov wrote: >>> ???? > Hi Daniel, David and Serguei, >>> ???? > >>> ???? > Please review a new version of the fix (webrev.08) that as >>> Daniel suggested renames >>> ???? > ThreadTable to ThreadIdTable (related classes and variables >>> are renamed as well) and >>> ???? > corrects formatting issues. There are no other changes in >>> this webrev.08 comparing >>> ???? > to the previous version webrev.07. >>> ???? > >>> ???? > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>> successfully passed. >>> ???? > >>> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >>> ???? > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> ???? > >>> ???? > Thank you! >>> ???? > >>> ???? > Best regards, >>> ???? > Daniil >>> ???? > >>> ???? > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" >>> wrote: >>> ???? > >>> ???? >????? Daniil, >>> ???? > >>> ???? >????? Thanks for sticking with this project through the many >>> versions. >>> ???? >????? Sorry this review is late... >>> ???? > >>> ???? > >>> ???? >????? On 9/19/19 8:30 PM, Daniil Titov wrote: >>> ???? >????? > Hi David and Serguei, >>> ???? >????? > >>> ???? >????? > Please review new version of the fix that includes the >>> changes Serguei suggested: >>> ???? >????? >?? 1. If racing threads initialize the thread table >>> only one of these threads will populate the table with the threads >>> from the thread list >>> ???? >????? >?? 2. The code that adds the thread to the tread table >>> is put inside Threads_lock to ensure that we cannot accidentally add >>> the thread >>> ???? >????? >?????? that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> ???? >????? > >>> ???? >????? > The changes are in ThreadTable::lazy_initialize() >>> method only. >>> ???? >????? > >>> ???? >????? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 >>> tests successfully passed. >>> ???? >????? > >>> ???? >????? > Webrev: >>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> ???? > >>> ???? >????? src/hotspot/share/runtime/mutexLocker.hpp >>> ???? >?????????? No comments. >>> ???? > >>> ???? >????? src/hotspot/share/runtime/mutexLocker.cpp >>> ???? >?????????? No comments. >>> ???? > >>> ???? >????? src/hotspot/share/runtime/threadSMR.cpp >>> ???? >?????????? L623:???????? MutexLocker ml(Threads_lock); >>> ???? >?????????? L626:???????? if (!thread->is_exiting()) { >>> ???? >?????????????? Re: discussion about is_exiting() >>> ???? > >>> ???? >?????????????? The header comment is pretty clear: >>> ???? > >>> ???? > src/hotspot/share/runtime/thread.hpp: >>> ???? > >>> ???? >?????????????????? // thread has called JavaThread::exit() or >>> is terminated >>> ???? >?????????????????? bool is_exiting() const; >>> ???? > >>> ???? >?????????????? is_exiting() might become true right after you >>> have called it, >>> ???? >?????????????? but its purpose is to ask the question and not >>> prevent the >>> ???? >?????????????? condition from becoming true. As David said, >>> you should consider >>> ???? >?????????????? it an optimization. If you happen to see the >>> condition is true, >>> ???? >?????????????? then you know that the JavaThread isn't going >>> to be around much >>> ???? >?????????????? longer and should act accordingly. >>> ???? > >>> ???? >?????????????? The is_exiting() implementation is: >>> ???? > >>> ???? >???????????????? inline bool JavaThread::is_exiting() const { >>> ???? >?????????????????? // Use load-acquire so that setting of >>> _terminated by >>> ???? >?????????????????? // JavaThread::exit() is seen more quickly. >>> ???? >?????????????????? TerminatedTypes l_terminated = >>> (TerminatedTypes) >>> ???? > OrderAccess::load_acquire((volatile jint *) &_terminated); >>> ???? >?????????????????? return l_terminated == _thread_exiting || >>> ???? >????? check_is_terminated(l_terminated); >>> ???? >???????????????? } >>> ???? > >>> ???? >?????????????? and it depends on the JavaThread's _terminated >>> field value. >>> ???? > >>> ???? >???????????????? // JavaThread termination support >>> ???? >???????????????? enum TerminatedTypes { >>> ???? >????????????????? _not_terminated = 0xDEAD - 2, >>> ???? > _thread_exiting,???????????????????????????? // >>> ???? >????? JavaThread::exit() has been called for this thread >>> ???? > _thread_terminated,????????????????????????? // JavaThread >>> ???? >????? is removed from thread list >>> ???? > _vm_exited?????????????????????????????????? // JavaThread >>> ???? >????? is still executing native code, but VM is terminated >>> > // only VM_Exit >>> ???? >????? can set _vm_exited >>> ???? >???????????????? }; >>> ???? > >>> ???? >?????????????? so the JavaThread's _terminated field can get >>> set to >>> ???? >?????????????? _thread_exiting independent of the >>> Threads_lock, but >>> ???? >?????????????? it can't get set to _thread_terminated without the >>> ???? >?????????????? Threads_lock. >>> ???? > >>> ???? >?????????????? So by grabbing the Threads_lock on L623, you >>> make sure >>> ???? >?????????????? that ThreadTable::add_thread(java_tid, thread) >>> does not >>> ???? >?????????????? add a JavaThread that's not on the ThreadsList. >>> It might >>> ???? >?????????????? still become is_exiting() == true right after your >>> ???? > >>> ???? >???????????????? L626???????? if (!thread->is_exiting()) { >>> ???? > >>> ???? >?????????????? but it will still be on the main ThreadsList. >>> And that >>> ???? >?????????????? means that when the JavaThread is removed from >>> the main >>> ???? >?????????????? ThreadsList, you'll still call: >>> ???? > >>> ???? >???????????????? L931: ThreadTable::remove_thread(tid); >>> ???? > >>> ???? >?????????? L624:???????? // Must be inside the lock to ensure >>> that we don't >>> ???? >????? add the thread to the table >>> ???? >?????????????? typo: s/the thread/a thread/ >>> ???? > >>> ???? >?????????? L633:?????? return thread; >>> ???? >?????????????? nit - L633 - indented too far (should be 2 spaces) >>> ???? > >>> ???? >????? src/hotspot/share/services/threadTable.hpp >>> ???? >?????????? L42:?? static void lazy_initialize(const >>> ThreadsList *threads); >>> ???? >?????????????? nit - put space between '*' the variable: >>> ???? > >>> ???? >???????????????? static void lazy_initialize(const >>> ThreadsList* threads); >>> ???? > >>> ???? >?????????????? like you do in your other decls. >>> ???? > >>> ???? >?????????? L45:?? // Lookup and inserts >>> ???? >?????????????? Perhaps:? // Lookup and list management >>> ???? > >>> ???? >?????????? L60-61 - nit - please delete these blank lines. >>> ???? > >>> ???? >????? src/hotspot/share/services/threadTable.cpp >>> ???? >?????????? L28: #include "runtime/timerTrace.hpp" >>> ???? >?????????????? nit - This should be after threadSMR.hpp... >>> (alpha sorted order) >>> ???? > >>> ???? >?????????? L39: static const size_t DefaultThreadTableSizeLog >>> = 8; >>> ???? >?????????????? nit - your other 'static const' are not >>> CamelCase. Why is this one? >>> ???? > >>> ???? >?????????? L45: static ThreadTableHash* volatile _local_table >>> = NULL; >>> ???? >?????????? L50: static volatile size_t _current_size = 0; >>> ???? >?????????? L51: static volatile size_t _items_count = 0; >>> ???? >?????????????? nit - can you group the file statics together? >>> (up with L41). >>> ???? > >>> ???? >?????????? L60: _tid(tid),_java_thread(java_thread) {} >>> ???? >?????????????? nit - space after ',' >>> ???? > >>> ???? >?????????? L62?? jlong tid() const { return _tid;} >>> ???? >?????????? L63?? JavaThread* thread() const {return >>> _java_thread;} >>> ???? >?????????????? nit - space before '}' >>> ???? >?????????????? nit - space after '{' on L63. >>> ???? > >>> ???? >?????????? L70:???? static uintx get_hash(Value const& value, >>> bool* is_dead) { >>> ???? >?????????????? Parameter 'is_dead' is not used. >>> ???? > >>> ???? >?????????? L74:???? static void* allocate_node(size_t size, >>> Value const& value) { >>> ???? >?????????????? Parameter 'value' is not used. >>> ???? > >>> ???? >?????????? L93: void ThreadTable::lazy_initialize(const >>> ThreadsList *threads) { >>> ???? >?????????????? Re: discussion about lazy_initialize() racing with >>> ???? > ThreadsList::find_JavaThread_from_java_tid() >>> ???? > >>> ???? >?????????????? There's a couple of aspects to these two pieces >>> of code racing >>> ???? >?????????????? with each other and racing with new thread >>> creation. Racing with >>> ???? >?????????????? new thread creation is the easy one: >>> ???? > >>> ???? >???????????????? If a new thread isn't added to the >>> ThreadTable by >>> ???? >???????????????? ThreadsSMRSupport::add_thread() calling >>> ???? >????? ThreadTable::add_thread(), >>> ???? >???????????????? then the point in the future where someone calls >>> ???? >???????????????? find_JavaThread_from_java_tid() will add it >>> to the table due to >>> ???? >???????????????? the linear search when >>> ThreadTable::find_thread_by_tid() >>> ???? >???????????????? returns NULL. >>> ???? > >>> ???? >????????????? As for multi-threads calling >>> ???? >????? ThreadsList::find_JavaThread_from_java_tid() >>> ???? >????????????? at the same time which results in multi-threads >>> in lazy_initialize() >>> ???? >????????????? at the same time... >>> ???? > >>> ???? >????????????? - ThreadTable creation will be linear due to >>> ThreadTableCreate_lock. >>> ???? >??????????????? After _is_initialized is set to true, then no >>> more callers to >>> ???? >??????????????? lazy_initialize() will be in the "if >>> (!_is_initialized)" block. >>> ???? >????????????? - Once the ThreadTable is created, then >>> multi-threads can be >>> ???? >??????????????? executing the for-loop to add their >>> ThreadsList entries to >>> ???? >??????????????? the ThreadTable. There will be a bit of >>> Threads_lock contention >>> ???? >??????????????? as each of the multi-threads tries to add >>> their entries and >>> ???? >??????????????? there will be some wasted work since the >>> multi-threads will >>> ???? >??????????????? likely have similar ThreadLists. >>> ???? > >>> ???? >????????????? Of course, once _is_initialized is set to true, >>> then any caller >>> ???? >????????????? to lazy_initialize() will return quickly and >>> ???? > ThreadsList::find_JavaThread_from_java_tid() will call >>> ???? >????????????? ThreadTable::find_thread_by_tid(). If the target >>> java_tid isn't >>> ???? >????????????? found, then we do the linear search thing here >>> and add the >>> ???? >????????????? the entry if we find a match in our current >>> ThreadsList. Since >>> ???? >????????????? we're only adding the one here, we only contend >>> for the Threads_lock >>> ???? >????????????? here if we find it. >>> ???? > >>> ???? >????????????? If ThreadsList::find_JavaThread_from_java_tid() >>> is called with a >>> ???? >????????????? target java_tid for a JavaThread that was >>> created after the >>> ???? >????????????? ThreadsList object that the caller has in hand >>> for the >>> ???? >????????????? find_JavaThread_from_java_tid() call, then, of >>> course, that >>> ???? >????????????? target 'java_tid' won't be found because the >>> JavaThread was >>> ???? >????????????? added the main ThreadsList _after_ the >>> ThreadsList object was >>> ???? >????????????? created by the caller. Of course, you have to >>> ask where the >>> ???? >????????????? target java_tid value came from since the >>> JavaThread wasn't >>> ???? >????????????? around when the >>> ThreadsList::find_JavaThread_from_java_tid() >>> ???? >????????????? call was made with that target java_tid value... >>> ???? > >>> ???? >?????????? L99:???????? // being concurently populated during >>> the initalization. >>> ???? >?????????????? Typos? Perhaps: >>> ???? >??????????????????????? // to be concurrently populated during >>> initialization. >>> ???? > >>> ???? >?????????????? But I think those two comment lines are more >>> appropriate above >>> ???? >?????????????? this line: >>> ???? > >>> ???? >?????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >>> ???? > >>> ???? >?????????? L112:?????????? // Must be inside the lock to >>> ensure that we don't >>> ???? >????? add the thread to the table >>> ???? >?????????????? typo: s/the thread/a thread/ >>> ???? > >>> ???? >?????????? L141:?? return ((double)_items_count)/_current_size; >>> ???? >?????????????? nit - need spaces around '/'. >>> ???? > >>> ???? >?????????? L177:?? bool equals(ThreadTableEntry **value, bool* >>> is_dead) { >>> ???? >?????????????? nit - put space between '**' the variable: >>> ???? >?????????????????? bool equals(ThreadTableEntry** value, >>> ???? > >>> ???? >?????????????? Parameter 'is_dead' is not used. >>> ???? > >>> ???? >?????????? L214:?? while(true) { >>> ???? >?????????????? nit - space before '('. >>> ???? > >>> ???? > >>> ???? >????? Short version: Thumbs up. >>> ???? > >>> ???? >????? Longer version: I don't think I've spotted anything >>> other than nits here. >>> ???? >????? Mostly I've just looked for multi-threaded races, proper >>> usage of the >>> ???? >????? Thread-SMR stuff, and minimal impact in the case where >>> the new >>> ???? >????? ThreadsTable is never needed. >>> ???? > >>> ???? >????? Dan >>> ???? > >>> ???? >????? P.S. >>> ???? >????? ThreadTable is a bit of misnomer. What you really have >>> here is >>> ???? >????? a ThreadIdTable, but I'm really late to the code review >>> flow >>> ???? >????? with that comment... >>> ???? > >>> ???? > >>> ???? >????? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> ???? >????? > >>> ???? >????? > Thank you! >>> ???? >????? > --Daniil >>> ???? > >>> ???? > >>> ???? > >>> ???? > >>> >>> >> >> > From daniel.daugherty at oracle.com Fri Sep 27 18:06:12 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 27 Sep 2019 14:06:12 -0400 Subject: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> On 9/27/19 1:58 PM, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Just notice I did not reply to you. > Thank you for the explanation! > > Have you already pushed this one? Pushed on 2019.09.25 at 1416 ET. It has made it thru Tier7 testing as of yesterday... Dan > > Thanks, > Serguei > > > On 9/24/19 12:46, Daniil Titov wrote: >> Hi Serguei, >> >> Thank you for reviewing this version of the fix. >> >>> ??? Just one question about ThreadIdTable::remove_thread(jlong tid). >>> ??? What happens if there is no thread with the specified tid in >>> ThreadIdTable? >>> ??? Is it possible? >> It could be possible when the thread that was started while the >> thread table >> was initializing exits.? At this point the thread table is >> initialized and the thread >> tries to remove itself from it. Removing non-existing? entry from >> ConcurrentHashTable >> is a correct operation that just leaves the table unchanged. >> >> src/hotspot/share/services/threadIdTable.cpp >> >> ??? 233??? bool ThreadIdTable::remove_thread(jlong tid) { >> ??? 234????? assert(_is_initialized, "Thread table is not initialized"); >> ??? 235????? Thread* thread = Thread::current(); >> ??? 236????? ThreadIdTableLookup lookup(tid); >> ??? 237????? return _local_table->remove(thread, lookup); >> ??? 238??? } >> >> src/hotspot/share/utilities/concurrentHashTable.hpp >> >> ?? 422????? // Returns true if items was deleted matching LOOKUP_FUNC >> and >> ??? 423????? // prior to destruction DELETE_FUNC is called. >> ??? 424????? template >> ??? 425????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, >> DELETE_FUNC& del_f) { >> ??? 426??????? return internal_remove(thread, lookup_f, del_f); >> ??? 427????? } >> ??? 428 >> ??? 429????? // Same without DELETE_FUNC. >> ??? 430????? template >> ??? 431????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { >> ??? 432??????? return internal_remove(thread, lookup_f, noOp); >> ??? 433????? } >> >> src/hotspot/share/utilities/concurrentHashTable.inline.hpp >> >> ??? 446??? inline bool ConcurrentHashTable:: >> ??? 447????? internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, >> DELETE_FUNC& delete_f) >> ??? 448??? { >> ??? 449????? Bucket* bucket = get_bucket_locked(thread, >> lookup_f.get_hash()); >> ??? 450????? assert(bucket->is_locked(), "Must be locked."); >> ??? 451????? Node* const volatile * rem_n_prev = bucket->first_ptr(); >> ??? 452????? Node* rem_n = bucket->first(); >> ??? 453????? bool have_dead = false; >> ??? 454????? while (rem_n != NULL) { >> ??? 455??????? if (lookup_f.equals(rem_n->value(), &have_dead)) { >> ??? 456????????? bucket->release_assign_node_ptr(rem_n_prev, >> rem_n->next()); >> ??? 457????????? break; >> ??? 458??????? } else { >> ??? 459????????? rem_n_prev = rem_n->next_ptr(); >> ??? 460????????? rem_n = rem_n->next(); >> ??? 461??????? } >> ??? 462????? } >> ??? 463 >> ??? 464????? bucket->unlock(); >> ??? 465 >> ??? 466????? if (rem_n == NULL) { >> ??? 467??????? return false; >> ??? 468????? } >> >> Best regards, >> Daniil >> >> >> ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" >> wrote: >> >> ???? Hi Daniil, >> ???? ???? This version looks good to me. >> ???? Thank you for the update! >> ???? ???? Just one question about ThreadIdTable::remove_thread(jlong >> tid). >> ???? What happens if there is no thread with the specified tid in >> ThreadIdTable? >> ???? Is it possible? >> ???? ???? Thanks, >> ???? Serguei >> ???? ???? On 9/24/19 9:36 AM, Daniil Titov wrote: >> ???? > Hi Daniel, David and Serguei, >> ???? > >> ???? > Please review a new version of the fix (webrev.08) that as >> Daniel suggested renames >> ???? > ThreadTable to ThreadIdTable (related classes and variables >> are renamed as well) and >> ???? > corrects formatting issues. There are no other changes in this >> webrev.08 comparing >> ???? > to the previous version webrev.07. >> ???? > >> ???? > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> ???? > >> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> ???? > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > >> ???? > Thank you! >> ???? > >> ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" >> wrote: >> ???? > >> ???? >????? Daniil, >> ???? > >> ???? >????? Thanks for sticking with this project through the many >> versions. >> ???? >????? Sorry this review is late... >> ???? > >> ???? > >> ???? >????? On 9/19/19 8:30 PM, Daniil Titov wrote: >> ???? >????? > Hi David and Serguei, >> ???? >????? > >> ???? >????? > Please review new version of the fix that includes the >> changes Serguei suggested: >> ???? >????? >?? 1. If racing threads initialize the thread table only >> one of these threads will populate the table with the threads from >> the thread list >> ???? >????? >?? 2. The code that adds the thread to the tread table >> is put inside Threads_lock to ensure that we cannot accidentally add >> the thread >> ???? >????? >?????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >????? > >> ???? >????? > The changes are in ThreadTable::lazy_initialize() >> method only. >> ???? >????? > >> ???? >????? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 >> tests successfully passed. >> ???? >????? > >> ???? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> ???? > >> ???? >????? src/hotspot/share/runtime/mutexLocker.hpp >> ???? >?????????? No comments. >> ???? > >> ???? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >?????????? No comments. >> ???? > >> ???? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >?????????? L623:???????? MutexLocker ml(Threads_lock); >> ???? >?????????? L626:???????? if (!thread->is_exiting()) { >> ???? >?????????????? Re: discussion about is_exiting() >> ???? > >> ???? >?????????????? The header comment is pretty clear: >> ???? > >> ???? >???????????????? src/hotspot/share/runtime/thread.hpp: >> ???? > >> ???? >?????????????????? // thread has called JavaThread::exit() or >> is terminated >> ???? >?????????????????? bool is_exiting() const; >> ???? > >> ???? >?????????????? is_exiting() might become true right after you >> have called it, >> ???? >?????????????? but its purpose is to ask the question and not >> prevent the >> ???? >?????????????? condition from becoming true. As David said, you >> should consider >> ???? >?????????????? it an optimization. If you happen to see the >> condition is true, >> ???? >?????????????? then you know that the JavaThread isn't going to >> be around much >> ???? >?????????????? longer and should act accordingly. >> ???? > >> ???? >?????????????? The is_exiting() implementation is: >> ???? > >> ???? >???????????????? inline bool JavaThread::is_exiting() const { >> ???? >?????????????????? // Use load-acquire so that setting of >> _terminated by >> ???? >?????????????????? // JavaThread::exit() is seen more quickly. >> ???? >?????????????????? TerminatedTypes l_terminated = >> (TerminatedTypes) >> ???? > OrderAccess::load_acquire((volatile jint *) &_terminated); >> ???? >?????????????????? return l_terminated == _thread_exiting || >> ???? >????? check_is_terminated(l_terminated); >> ???? >???????????????? } >> ???? > >> ???? >?????????????? and it depends on the JavaThread's _terminated >> field value. >> ???? > >> ???? >???????????????? // JavaThread termination support >> ???? >???????????????? enum TerminatedTypes { >> ???? >????????????????? _not_terminated = 0xDEAD - 2, >> ???? > _thread_exiting,???????????????????????????? // >> ???? >????? JavaThread::exit() has been called for this thread >> ???? > _thread_terminated,????????????????????????? // JavaThread >> ???? >????? is removed from thread list >> ???? > _vm_exited?????????????????????????????????? // JavaThread >> ???? >????? is still executing native code, but VM is terminated >> > // only VM_Exit >> ???? >????? can set _vm_exited >> ???? >???????????????? }; >> ???? > >> ???? >?????????????? so the JavaThread's _terminated field can get >> set to >> ???? >?????????????? _thread_exiting independent of the Threads_lock, >> but >> ???? >?????????????? it can't get set to _thread_terminated without the >> ???? >?????????????? Threads_lock. >> ???? > >> ???? >?????????????? So by grabbing the Threads_lock on L623, you >> make sure >> ???? >?????????????? that ThreadTable::add_thread(java_tid, thread) >> does not >> ???? >?????????????? add a JavaThread that's not on the ThreadsList. >> It might >> ???? >?????????????? still become is_exiting() == true right after your >> ???? > >> ???? >???????????????? L626???????? if (!thread->is_exiting()) { >> ???? > >> ???? >?????????????? but it will still be on the main ThreadsList. >> And that >> ???? >?????????????? means that when the JavaThread is removed from >> the main >> ???? >?????????????? ThreadsList, you'll still call: >> ???? > >> ???? >???????????????? L931: ThreadTable::remove_thread(tid); >> ???? > >> ???? >?????????? L624:???????? // Must be inside the lock to ensure >> that we don't >> ???? >????? add the thread to the table >> ???? >?????????????? typo: s/the thread/a thread/ >> ???? > >> ???? >?????????? L633:?????? return thread; >> ???? >?????????????? nit - L633 - indented too far (should be 2 spaces) >> ???? > >> ???? >????? src/hotspot/share/services/threadTable.hpp >> ???? >?????????? L42:?? static void lazy_initialize(const ThreadsList >> *threads); >> ???? >?????????????? nit - put space between '*' the variable: >> ???? > >> ???? >???????????????? static void lazy_initialize(const ThreadsList* >> threads); >> ???? > >> ???? >?????????????? like you do in your other decls. >> ???? > >> ???? >?????????? L45:?? // Lookup and inserts >> ???? >?????????????? Perhaps:? // Lookup and list management >> ???? > >> ???? >?????????? L60-61 - nit - please delete these blank lines. >> ???? > >> ???? >????? src/hotspot/share/services/threadTable.cpp >> ???? >?????????? L28: #include "runtime/timerTrace.hpp" >> ???? >?????????????? nit - This should be after threadSMR.hpp... >> (alpha sorted order) >> ???? > >> ???? >?????????? L39: static const size_t DefaultThreadTableSizeLog = 8; >> ???? >?????????????? nit - your other 'static const' are not >> CamelCase. Why is this one? >> ???? > >> ???? >?????????? L45: static ThreadTableHash* volatile _local_table = >> NULL; >> ???? >?????????? L50: static volatile size_t _current_size = 0; >> ???? >?????????? L51: static volatile size_t _items_count = 0; >> ???? >?????????????? nit - can you group the file statics together? >> (up with L41). >> ???? > >> ???? >?????????? L60:???? _tid(tid),_java_thread(java_thread) {} >> ???? >?????????????? nit - space after ',' >> ???? > >> ???? >?????????? L62?? jlong tid() const { return _tid;} >> ???? >?????????? L63?? JavaThread* thread() const {return _java_thread;} >> ???? >?????????????? nit - space before '}' >> ???? >?????????????? nit - space after '{' on L63. >> ???? > >> ???? >?????????? L70:???? static uintx get_hash(Value const& value, >> bool* is_dead) { >> ???? >?????????????? Parameter 'is_dead' is not used. >> ???? > >> ???? >?????????? L74:???? static void* allocate_node(size_t size, >> Value const& value) { >> ???? >?????????????? Parameter 'value' is not used. >> ???? > >> ???? >?????????? L93: void ThreadTable::lazy_initialize(const >> ThreadsList *threads) { >> ???? >?????????????? Re: discussion about lazy_initialize() racing with >> ???? > ThreadsList::find_JavaThread_from_java_tid() >> ???? > >> ???? >?????????????? There's a couple of aspects to these two pieces >> of code racing >> ???? >?????????????? with each other and racing with new thread >> creation. Racing with >> ???? >?????????????? new thread creation is the easy one: >> ???? > >> ???? >???????????????? If a new thread isn't added to the ThreadTable by >> ???? >???????????????? ThreadsSMRSupport::add_thread() calling >> ???? >????? ThreadTable::add_thread(), >> ???? >???????????????? then the point in the future where someone calls >> ???? >???????????????? find_JavaThread_from_java_tid() will add it to >> the table due to >> ???? >???????????????? the linear search when >> ThreadTable::find_thread_by_tid() >> ???? >???????????????? returns NULL. >> ???? > >> ???? >????????????? As for multi-threads calling >> ???? >????? ThreadsList::find_JavaThread_from_java_tid() >> ???? >????????????? at the same time which results in multi-threads >> in lazy_initialize() >> ???? >????????????? at the same time... >> ???? > >> ???? >????????????? - ThreadTable creation will be linear due to >> ThreadTableCreate_lock. >> ???? >??????????????? After _is_initialized is set to true, then no >> more callers to >> ???? >??????????????? lazy_initialize() will be in the "if >> (!_is_initialized)" block. >> ???? >????????????? - Once the ThreadTable is created, then >> multi-threads can be >> ???? >??????????????? executing the for-loop to add their ThreadsList >> entries to >> ???? >??????????????? the ThreadTable. There will be a bit of >> Threads_lock contention >> ???? >??????????????? as each of the multi-threads tries to add their >> entries and >> ???? >??????????????? there will be some wasted work since the >> multi-threads will >> ???? >??????????????? likely have similar ThreadLists. >> ???? > >> ???? >????????????? Of course, once _is_initialized is set to true, >> then any caller >> ???? >????????????? to lazy_initialize() will return quickly and >> ???? > ThreadsList::find_JavaThread_from_java_tid() will call >> ???? >????????????? ThreadTable::find_thread_by_tid(). If the target >> java_tid isn't >> ???? >????????????? found, then we do the linear search thing here >> and add the >> ???? >????????????? the entry if we find a match in our current >> ThreadsList. Since >> ???? >????????????? we're only adding the one here, we only contend >> for the Threads_lock >> ???? >????????????? here if we find it. >> ???? > >> ???? >????????????? If ThreadsList::find_JavaThread_from_java_tid() >> is called with a >> ???? >????????????? target java_tid for a JavaThread that was created >> after the >> ???? >????????????? ThreadsList object that the caller has in hand >> for the >> ???? >????????????? find_JavaThread_from_java_tid() call, then, of >> course, that >> ???? >????????????? target 'java_tid' won't be found because the >> JavaThread was >> ???? >????????????? added the main ThreadsList _after_ the >> ThreadsList object was >> ???? >????????????? created by the caller. Of course, you have to ask >> where the >> ???? >????????????? target java_tid value came from since the >> JavaThread wasn't >> ???? >????????????? around when the >> ThreadsList::find_JavaThread_from_java_tid() >> ???? >????????????? call was made with that target java_tid value... >> ???? > >> ???? >?????????? L99:???????? // being concurently populated during >> the initalization. >> ???? >?????????????? Typos? Perhaps: >> ???? >??????????????????????? // to be concurrently populated during >> initialization. >> ???? > >> ???? >?????????????? But I think those two comment lines are more >> appropriate above >> ???? >?????????????? this line: >> ???? > >> ???? >?????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? > >> ???? >?????????? L112:?????????? // Must be inside the lock to ensure >> that we don't >> ???? >????? add the thread to the table >> ???? >?????????????? typo: s/the thread/a thread/ >> ???? > >> ???? >?????????? L141:?? return ((double)_items_count)/_current_size; >> ???? >?????????????? nit - need spaces around '/'. >> ???? > >> ???? >?????????? L177:?? bool equals(ThreadTableEntry **value, bool* >> is_dead) { >> ???? >?????????????? nit - put space between '**' the variable: >> ???? >?????????????????? bool equals(ThreadTableEntry** value, >> ???? > >> ???? >?????????????? Parameter 'is_dead' is not used. >> ???? > >> ???? >?????????? L214:?? while(true) { >> ???? >?????????????? nit - space before '('. >> ???? > >> ???? > >> ???? >????? Short version: Thumbs up. >> ???? > >> ???? >????? Longer version: I don't think I've spotted anything other >> than nits here. >> ???? >????? Mostly I've just looked for multi-threaded races, proper >> usage of the >> ???? >????? Thread-SMR stuff, and minimal impact in the case where >> the new >> ???? >????? ThreadsTable is never needed. >> ???? > >> ???? >????? Dan >> ???? > >> ???? >????? P.S. >> ???? >????? ThreadTable is a bit of misnomer. What you really have >> here is >> ???? >????? a ThreadIdTable, but I'm really late to the code review flow >> ???? >????? with that comment... >> ???? > >> ???? > >> ???? >????? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????? > >> ???? >????? > Thank you! >> ???? >????? > --Daniil >> ???? > >> ???? > >> ???? > >> ???? > >> >> > > From martinrb at google.com Fri Sep 27 19:33:48 2019 From: martinrb at google.com (Martin Buchholz) Date: Fri, 27 Sep 2019 12:33:48 -0700 Subject: RFR: jsr166 integration 2019-09 In-Reply-To: References: <517197737.1773850.1568278129180.JavaMail.zimbra@u-pem.fr> <738837959.2535658.1568387411851.JavaMail.zimbra@u-pem.fr> <16d7944e-4b61-7ed9-aeea-fca44e970016@cs.oswego.edu> <1150861893.2564720.1568393036106.JavaMail.zimbra@u-pem.fr> <1a4c7ebd-78bd-de5b-7053-cff63f13bdeb@oracle.com> <86366d20-1ae0-6694-4b41-ce21dfb8ddbb@oracle.com> Message-ID: On Mon, Sep 23, 2019 at 11:08 PM David Holmes wrote: > Except when I run it through our test system ReservedStackTest is still > failing :( > > I tested it initially when Fred proposed it and that went fine. It also > passes for me locally on Linux. > I committed the changes other than for @ReservedStackAccess, which remains in limbo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Fri Sep 27 20:29:34 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 27 Sep 2019 13:29:34 -0700 Subject: RFR(S): JDK-8231287: JMap should do a better job of reporting exception it catches Message-ID: <73aa3635-8f54-eb4a-bf65-77437871e97b@oracle.com> Hello, Please review the following: https://bugs.openjdk.java.net/browse/JDK-8231287 http://cr.openjdk.java.net/~cjplummer/8231287/webrev.00 Basically no longer have jmap consume and hide exceptions that are thrown, and have all tools catch and print any exception that is propagated to Tools.execute(). The CR contains more details. Tested with the following: ? :hotspot_serviceability ? :jdk_svc ? :vmTestbase_nsk_jdi ? :vmTestbase_nsk_jvmti ? :vmTestbase_nsk_jdwp ? :vmTestbase_nsk_jdb thanks, Chris From serguei.spitsyn at oracle.com Fri Sep 27 21:04:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 14:04:35 -0700 Subject: RFR(S): JDK-8231287: JMap should do a better job of reporting exception it catches In-Reply-To: <73aa3635-8f54-eb4a-bf65-77437871e97b@oracle.com> References: <73aa3635-8f54-eb4a-bf65-77437871e97b@oracle.com> Message-ID: Hi Chris, It looks good. Thanks, Serguei On 9/27/19 1:29 PM, Chris Plummer wrote: > Hello, > > Please review the following: > > https://bugs.openjdk.java.net/browse/JDK-8231287 > http://cr.openjdk.java.net/~cjplummer/8231287/webrev.00 > > Basically no longer have jmap consume and hide exceptions that are > thrown, and have all tools catch and print any exception that is > propagated to Tools.execute(). The CR contains more details. > > Tested with the following: > > ? :hotspot_serviceability > ? :jdk_svc > ? :vmTestbase_nsk_jdi > ? :vmTestbase_nsk_jvmti > ? :vmTestbase_nsk_jdwp > ? :vmTestbase_nsk_jdb > > thanks, > > Chris > From hohensee at amazon.com Fri Sep 27 21:49:02 2019 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 27 Sep 2019 21:49:02 +0000 Subject: RFR(S): JDK-8231287: JMap should do a better job of reporting exception it catches In-Reply-To: References: <73aa3635-8f54-eb4a-bf65-77437871e97b@oracle.com> Message-ID: <7CE26CEC-7447-4C28-B102-F2D54C038654@amazon.com> +1. Paul ?On 9/27/19, 2:06 PM, "serviceability-dev on behalf of serguei.spitsyn at oracle.com" wrote: Hi Chris, It looks good. Thanks, Serguei On 9/27/19 1:29 PM, Chris Plummer wrote: > Hello, > > Please review the following: > > https://bugs.openjdk.java.net/browse/JDK-8231287 > http://cr.openjdk.java.net/~cjplummer/8231287/webrev.00 > > Basically no longer have jmap consume and hide exceptions that are > thrown, and have all tools catch and print any exception that is > propagated to Tools.execute(). The CR contains more details. > > Tested with the following: > > :hotspot_serviceability > :jdk_svc > :vmTestbase_nsk_jdi > :vmTestbase_nsk_jvmti > :vmTestbase_nsk_jdwp > :vmTestbase_nsk_jdb > > thanks, > > Chris > From chris.plummer at oracle.com Fri Sep 27 23:54:42 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 27 Sep 2019 16:54:42 -0700 Subject: RFR(S): JDK-8231287: JMap should do a better job of reporting exception it catches In-Reply-To: <7CE26CEC-7447-4C28-B102-F2D54C038654@amazon.com> References: <73aa3635-8f54-eb4a-bf65-77437871e97b@oracle.com> <7CE26CEC-7447-4C28-B102-F2D54C038654@amazon.com> Message-ID: <4064e799-85fc-51e9-894c-9a35f395c422@oracle.com> Thanks for the quick reviews! Chris On 9/27/19 2:49 PM, Hohensee, Paul wrote: > +1. > > Paul > > ?On 9/27/19, 2:06 PM, "serviceability-dev on behalf of serguei.spitsyn at oracle.com" wrote: > > Hi Chris, > > It looks good. > > Thanks, > Serguei > > On 9/27/19 1:29 PM, Chris Plummer wrote: > > Hello, > > > > Please review the following: > > > > https://bugs.openjdk.java.net/browse/JDK-8231287 > > http://cr.openjdk.java.net/~cjplummer/8231287/webrev.00 > > > > Basically no longer have jmap consume and hide exceptions that are > > thrown, and have all tools catch and print any exception that is > > propagated to Tools.execute(). The CR contains more details. > > > > Tested with the following: > > > > :hotspot_serviceability > > :jdk_svc > > :vmTestbase_nsk_jdi > > :vmTestbase_nsk_jvmti > > :vmTestbase_nsk_jdwp > > :vmTestbase_nsk_jdb > > > > thanks, > > > > Chris > > > > > From serguei.spitsyn at oracle.com Sat Sep 28 01:25:49 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 18:25:49 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread Message-ID: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> Please, review fix for test enhancement: ? https://bugs.openjdk.java.net/browse/JDK-8231595 Webrev: http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ Summary: ? New test is a coverage for the JVMTI bug: ??? https://bugs.openjdk.java.net/browse/JDK-8217762 ????? SuspendThreadList won't work correctly if the current thread is not last in the list ? It provides a prove the bug JDK-8217762 does not exist ? as the test is passed with the current implementation. Thanks, Serguei From fweimer at redhat.com Sat Sep 28 09:09:25 2019 From: fweimer at redhat.com (Florian Weimer) Date: Sat, 28 Sep 2019 11:09:25 +0200 Subject: RFR (XS): 8231600: Avoid GCC 8 strncpy error in jdk.jdwp.agent Message-ID: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> Bug: https://bugs.openjdk.java.net/browse/JDK-8231600 Webrev: http://cr.openjdk.java.net/~fweimer/8231600/webrev.01/ We know the size of the string, so we can use memcpy instead of strncpy. Thanks, Florian From igor.ignatyev at oracle.com Sat Sep 28 16:47:52 2019 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Sat, 28 Sep 2019 09:47:52 -0700 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> Message-ID: <58EB6707-22C7-44BB-9F0A-D73DD4E764EF@oracle.com> Hi Per, if you decide to go w/ @requires, I'd encourage you to also put an explaining comment in the test descriptor using jtreg's tag @comment, so at least it will be easier for future readers to understand why the test can't / shouldn't be run w/ ZGC. Thanks, -- Igor > On Sep 27, 2019, at 6:43 AM, Daniel D. Daugherty wrote: > > Thanks for the background info. I think that's a more convincing > explanation and I'm okay with the change as is. > > Please add a comment to JDK-8231552 making it clear that > vmTestbase/nsk/jvmti/Allocate/alloc001 should be reenabled > if/when the bug^H^H^H RFE is fixed. :-) > > Dan > > > On 9/27/19 9:39 AM, Per Liden wrote: >> Hi Dan, >> >> The background here is that ZGC previously (by design) required a certain amount of address space to run. So it was a deliberate decision to not support RLIMIT_AS. However, as ZGC's design have progressed, we're now in a position where we could reconsider this (which is why the RFE was filed). In _theory_, we could come to the conclusion that we still don't want to support this and just close the RFE as Won't fix. That's (very) unlikely, but I think that line of thinking shows why sticking this in the problem list is the wrong thing to do here, because that problem list entry would then potentially stay there forever. Until that RFE is pushed, that test should not even attempt to test it with ZGC, because it's not a supported use case. >> >> Anyways... ;) >> >> cheers, >> Per >> >> On 9/27/19 2:48 PM, Daniel D. Daugherty wrote: >> [...] >>> >>> In your code review invite you say this: >>> >>>> The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). >>> >>> To me that says the current behavior is temporary and then you say this: >>> >>>> This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>> >>> and that confirms that the issue is planned to be resolved. I'm not >>> sure that I agree that 8231552 is an RFE rather than a bug, but that's >>> not the point here... >>> >>> To me, both of those sentences would lead to a ProblemListing >>> and not an @requires. Your call... >>> >>> Dan >>> >>> >>>> >>>> cheers, >>>> Per >>>> >>>>>> >>>>>> cheers, >>>>>> Per >>>>>> >>>>>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>>>>> Hi Per, >>>>>>> wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>>>>> Thanks, >>>>>>> -- Igor >>>>>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>>>>>>> >>>>>>>> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>>>>> >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>>>>> >>>>>>>> /Per >>>>> >>> > From chris.plummer at oracle.com Sat Sep 28 19:12:33 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Sat, 28 Sep 2019 12:12:33 -0700 Subject: RFR (XS): 8231600: Avoid GCC 8 strncpy error in jdk.jdwp.agent In-Reply-To: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> References: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> Message-ID: <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> Fix looks good. thanks, Chris On 9/28/19 2:09 AM, Florian Weimer wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8231600 > Webrev: http://cr.openjdk.java.net/~fweimer/8231600/webrev.01/ > > We know the size of the string, so we can use memcpy instead of strncpy. > > Thanks, > Florian From chris.plummer at oracle.com Sat Sep 28 19:33:10 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Sat, 28 Sep 2019 12:33:10 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> Message-ID: <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> Hi Serguei, Overall looks good. A few questions: I don't understand the need for all the 'i' and 'n' theatrics in the shouldFinish loop. Can you explain and also add a comment? Is this comment right? ?193???? // set thread is not ready again ?194???? public void setAllThreadsReady() { ?195???????? allThreadsReady = true; ?196???? } Also, shouldn't "setAllThreadsReady()" be static? Do you think it would be useful to also run the test with the last thread in the list being the suspender thread? thanks, Chris On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: > Please, review fix for test enhancement: > ? https://bugs.openjdk.java.net/browse/JDK-8231595 > > Webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ > > > Summary: > ? New test is a coverage for the JVMTI bug: > ??? https://bugs.openjdk.java.net/browse/JDK-8217762 > ????? SuspendThreadList won't work correctly if the current thread is > not last in the list > > ? It provides a prove the bug JDK-8217762 does not exist > ? as the test is passed with the current implementation. > > Thanks, > Serguei From david.holmes at oracle.com Sat Sep 28 23:53:35 2019 From: david.holmes at oracle.com (David Holmes) Date: Sun, 29 Sep 2019 09:53:35 +1000 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> Message-ID: <2a0b6dc8-ab2b-f7df-f65f-e3b883e75b90@oracle.com> Hi Serguei, On 28/09/2019 11:25 am, serguei.spitsyn at oracle.com wrote: > Please, review fix for test enhancement: > ? https://bugs.openjdk.java.net/browse/JDK-8231595 > > Webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ Sorry but I could barely follow the logic of that test at all. Can you please write a comment block that clearly explains what the test is doing. I still don't understand how JDK-8217762 is "not an issue". Possibly the suspend request on the current thread doesn't actually take effect until it tries to return to Java code, but I'm unsure. Thanks, David > > Summary: > ? New test is a coverage for the JVMTI bug: > ??? https://bugs.openjdk.java.net/browse/JDK-8217762 > ????? SuspendThreadList won't work correctly if the current thread is > not last in the list > > ? It provides a prove the bug JDK-8217762 does not exist > ? as the test is passed with the current implementation. > > Thanks, > Serguei From fweimer at redhat.com Sun Sep 29 14:13:52 2019 From: fweimer at redhat.com (Florian Weimer) Date: Sun, 29 Sep 2019 16:13:52 +0200 Subject: RFR (XS): 8231600: Avoid GCC 8 strncpy error in jdk.jdwp.agent In-Reply-To: <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> (Chris Plummer's message of "Sat, 28 Sep 2019 12:12:33 -0700") References: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> Message-ID: <878sq75gwv.fsf@oldenburg2.str.redhat.com> * Chris Plummer: > Fix looks good. Thanks. Would you be able to sponsor this change for me? Florian > On 9/28/19 2:09 AM, Florian Weimer wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231600 >> Webrev: http://cr.openjdk.java.net/~fweimer/8231600/webrev.01/ >> >> We know the size of the string, so we can use memcpy instead of strncpy. >> >> Thanks, >> Florian From chris.plummer at oracle.com Sun Sep 29 16:46:59 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Sun, 29 Sep 2019 09:46:59 -0700 Subject: RFR (XS): 8231600: Avoid GCC 8 strncpy error in jdk.jdwp.agent In-Reply-To: <878sq75gwv.fsf@oldenburg2.str.redhat.com> References: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> <878sq75gwv.fsf@oldenburg2.str.redhat.com> Message-ID: <32ccde5d-008a-6e79-475f-40efadc4cfdb@oracle.com> On 9/29/19 7:13 AM, Florian Weimer wrote: > * Chris Plummer: > >> Fix looks good. > Thanks. Would you be able to sponsor this change for me? Yes. Just send me the changeset patch once you get a second review. Chris > > Florian > >> On 9/28/19 2:09 AM, Florian Weimer wrote: >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231600 >>> Webrev: http://cr.openjdk.java.net/~fweimer/8231600/webrev.01/ >>> >>> We know the size of the string, so we can use memcpy instead of strncpy. >>> >>> Thanks, >>> Florian From Michael.Rasmussen at roguewave.com Mon Sep 30 07:21:16 2019 From: Michael.Rasmussen at roguewave.com (Michael Rasmussen) Date: Mon, 30 Sep 2019 07:21:16 +0000 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> , Message-ID: There is a similar protected method on ClassLoader for that: ClassLoader.findBootstrapClassOrNull /Michael ________________________________ From: Sam Thomas Sent: 25 September 2019 02:11 To: Michael Rasmussen Cc: serviceability-dev at openjdk.java.net ; daniel.daugherty at oracle.com Subject: Re: Bytecode Instrumentation and Class Loading. Hi Michael, Returning question: I understand that the method findLoadedClass is protected. But say it was public, how would you find out loaded classes on the bootstrap classloader? Since from instrumentation perspective when a loader is null its the bootstrap classloader. Thanks ./Sam On Fri, Sep 20, 2019 at 2:08 PM Sam Thomas > wrote: cool thanks Thanks ./Sam On Fri, Sep 20, 2019 at 5:05 AM Michael Rasmussen > wrote: On 9/18/19 2:47 PM, Sam Thomas wrote: > Hi, > > I'm trying to understand if a class will load as soon as all the transformers return. The aim is to get a class reference of a class I have seen in my transformer. Short answer: no. Trying to define a class can cause other classes to try to be loaded before (for instance the super types), meaning when you're done transforming class A, the same thread might try to load multiple other classes before it finishes defining A. And if any of these fail, A will also fail to be defined. > Also if there is way to get the same without triggering class loading - if the class is not loaded return a null reference. ClassLoader has a protected method findLoadedClass that checks if a class is loaded by that classloader and returns the Class instance if so, or null otherwise. But as mentioned, the method is protected so normally not accessible outside the classloader itself. /Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.liden at oracle.com Mon Sep 30 13:17:56 2019 From: per.liden at oracle.com (Per Liden) Date: Mon, 30 Sep 2019 15:17:56 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> Message-ID: <91ae07aa-9004-fd38-e891-b1aafebc6db5@oracle.com> Hi Dan, On 9/27/19 3:43 PM, Daniel D. Daugherty wrote: > Thanks for the background info. I think that's a more convincing > explanation and I'm okay with the change as is. Thanks! > > Please add a comment to JDK-8231552 making it clear that > vmTestbase/nsk/jvmti/Allocate/alloc001 should be reenabled > if/when the bug^H^H^H RFE is fixed. :-) Done ;) > > Dan > > > On 9/27/19 9:39 AM, Per Liden wrote: >> Hi Dan, >> >> The background here is that ZGC previously (by design) required a >> certain amount of address space to run. So it was a deliberate >> decision to not support RLIMIT_AS. However, as ZGC's design have >> progressed, we're now in a position where we could reconsider this >> (which is why the RFE was filed). In _theory_, we could come to the >> conclusion that we still don't want to support this and just close the >> RFE as Won't fix. That's (very) unlikely, but I think that line of >> thinking shows why sticking this in the problem list is the wrong >> thing to do here, because that problem list entry would then >> potentially stay there forever. Until that RFE is pushed, that test >> should not even attempt to test it with ZGC, because it's not a >> supported use case. >> >> Anyways... ;) >> >> cheers, >> Per >> >> On 9/27/19 2:48 PM, Daniel D. Daugherty wrote: >> [...] >>> >>> In your code review invite you say this: >>> >>>> The root problem is that ZGC doesn't properly respect address space >>>> limitations (RLIMIT_AS). >>> >>> To me that says the current behavior is temporary and then you say this: >>> >>>> This test will be re-enabled again as part of fixing >>>> https://bugs.openjdk.java.net/browse/JDK-8231552 >>> >>> and that confirms that the issue is planned to be resolved. I'm not >>> sure that I agree that 8231552 is an RFE rather than a bug, but that's >>> not the point here... >>> >>> To me, both of those sentences would lead to a ProblemListing >>> and not an @requires. Your call... >>> >>> Dan >>> >>> >>>> >>>> cheers, >>>> Per >>>> >>>>>> >>>>>> cheers, >>>>>> Per >>>>>> >>>>>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>>>>> Hi Per, >>>>>>> wouldn't it be better to put this test into zgc-specific problem >>>>>>> list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>>>>> Thanks, >>>>>>> -- Igor >>>>>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden >>>>>>>>> wrote: >>>>>>>> >>>>>>>> Please review this one-liner to disable >>>>>>>> vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root >>>>>>>> problem is that ZGC doesn't properly respect address space >>>>>>>> limitations (RLIMIT_AS). This test will be re-enabled again as >>>>>>>> part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>>>>> >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>>>>> >>>>>>>> /Per >>>>> >>> > From per.liden at oracle.com Mon Sep 30 13:18:39 2019 From: per.liden at oracle.com (Per Liden) Date: Mon, 30 Sep 2019 15:18:39 +0200 Subject: RFR: 8231296: ZGC: vmTestbase/nsk/jvmti/Allocate/alloc001/ fails In-Reply-To: <58EB6707-22C7-44BB-9F0A-D73DD4E764EF@oracle.com> References: <3B6004E4-D5EC-411E-845E-1D8712B3CC4D@oracle.com> <9d34ee27-e581-6255-5796-85082a7a5a4b@oracle.com> <383ff735-b7de-2bdb-9add-cfb1320ccd9c@oracle.com> <096bb4aa-f4fb-c369-b5b9-461a60ad9c55@oracle.com> <7c0124c3-ead6-05d1-e15a-5023752753cd@oracle.com> <58EB6707-22C7-44BB-9F0A-D73DD4E764EF@oracle.com> Message-ID: <13de14f2-a6c0-1eaf-56bd-ffff3d5f2da7@oracle.com> Hi Igor, On 9/28/19 6:47 PM, Igor Ignatyev wrote: > Hi Per, > > if you decide to go w/ @requires, I'd encourage you to also put an explaining comment in the test descriptor using jtreg's tag @comment, so at least it will be easier for future readers to understand why the test can't / shouldn't be run w/ ZGC. Will do! cheers, Per > > Thanks, > -- Igor > >> On Sep 27, 2019, at 6:43 AM, Daniel D. Daugherty wrote: >> >> Thanks for the background info. I think that's a more convincing >> explanation and I'm okay with the change as is. >> >> Please add a comment to JDK-8231552 making it clear that >> vmTestbase/nsk/jvmti/Allocate/alloc001 should be reenabled >> if/when the bug^H^H^H RFE is fixed. :-) >> >> Dan >> >> >> On 9/27/19 9:39 AM, Per Liden wrote: >>> Hi Dan, >>> >>> The background here is that ZGC previously (by design) required a certain amount of address space to run. So it was a deliberate decision to not support RLIMIT_AS. However, as ZGC's design have progressed, we're now in a position where we could reconsider this (which is why the RFE was filed). In _theory_, we could come to the conclusion that we still don't want to support this and just close the RFE as Won't fix. That's (very) unlikely, but I think that line of thinking shows why sticking this in the problem list is the wrong thing to do here, because that problem list entry would then potentially stay there forever. Until that RFE is pushed, that test should not even attempt to test it with ZGC, because it's not a supported use case. >>> >>> Anyways... ;) >>> >>> cheers, >>> Per >>> >>> On 9/27/19 2:48 PM, Daniel D. Daugherty wrote: >>> [...] >>>> >>>> In your code review invite you say this: >>>> >>>>> The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). >>>> >>>> To me that says the current behavior is temporary and then you say this: >>>> >>>>> This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>> >>>> and that confirms that the issue is planned to be resolved. I'm not >>>> sure that I agree that 8231552 is an RFE rather than a bug, but that's >>>> not the point here... >>>> >>>> To me, both of those sentences would lead to a ProblemListing >>>> and not an @requires. Your call... >>>> >>>> Dan >>>> >>>> >>>>> >>>>> cheers, >>>>> Per >>>>> >>>>>>> >>>>>>> cheers, >>>>>>> Per >>>>>>> >>>>>>>> On 9/26/19 10:58 PM, Igor Ignatyev wrote: >>>>>>>> Hi Per, >>>>>>>> wouldn't it be better to put this test into zgc-specific problem list (test/hotspot/jtreg/ProblemList-zgc.txt)? >>>>>>>> Thanks, >>>>>>>> -- Igor >>>>>>>>>> On Sep 26, 2019, at 1:39 PM, Per Liden wrote: >>>>>>>>> >>>>>>>>> Please review this one-liner to disable vmTestbase/nsk/jvmti/Allocate/alloc001 when using ZGC. The root problem is that ZGC doesn't properly respect address space limitations (RLIMIT_AS). This test will be re-enabled again as part of fixing https://bugs.openjdk.java.net/browse/JDK-8231552 >>>>>>>>> >>>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8231296 >>>>>>>>> Webrev: http://cr.openjdk.java.net/~pliden/8231296/webrev.0 >>>>>>>>> >>>>>>>>> /Per >>>>>> >>>> >> > From sam.thomas at broadcom.com Mon Sep 30 18:12:27 2019 From: sam.thomas at broadcom.com (Sam Thomas) Date: Mon, 30 Sep 2019 11:12:27 -0700 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Message-ID: Hi Michael, I saw this method, but was unsure if this method only looked for loaded classes in the bootstrap class loader or if it proceeded to try load the class. I saw this being used in the ClassLoader.loadClass and the code and comments was what made me unsure. I'll hunt for the native code to see what is actually being done. Meanwhile, a google search made me notice that some implementations had this method named findBootstrapClass0. I can't tell if this is vendor specific or was it renamed at some point - Any idea? Thanks ./Sam On Mon, Sep 30, 2019 at 12:21 AM Michael Rasmussen < Michael.Rasmussen at roguewave.com> wrote: > There is a similar protected method on ClassLoader for that: ClassLoader. > findBootstrapClassOrNull > > /Michael > ------------------------------ > *From:* Sam Thomas > *Sent:* 25 September 2019 02:11 > *To:* Michael Rasmussen > *Cc:* serviceability-dev at openjdk.java.net < > serviceability-dev at openjdk.java.net>; daniel.daugherty at oracle.com < > daniel.daugherty at oracle.com> > *Subject:* Re: Bytecode Instrumentation and Class Loading. > > Hi Michael, > > Returning question: I understand that the method findLoadedClass is > protected. But say it was public, how would you find out loaded classes on > the bootstrap classloader? Since from instrumentation perspective when a > loader is null its the bootstrap classloader. > > Thanks > ./Sam > > > On Fri, Sep 20, 2019 at 2:08 PM Sam Thomas > wrote: > > cool thanks > > Thanks > ./Sam > > > On Fri, Sep 20, 2019 at 5:05 AM Michael Rasmussen < > Michael.Rasmussen at roguewave.com> wrote: > > On 9/18/19 2:47 PM, Sam Thomas wrote: > > Hi, > > > > I'm trying to understand if a class will load as soon as all the > transformers return. The aim is to get a class reference of a class I have > seen in my transformer. > > Short answer: no. > > Trying to define a class can cause other classes to try to be loaded > before (for instance the super types), meaning when you're done > transforming class A, the same thread might try to load multiple other > classes before it finishes defining A. > And if any of these fail, A will also fail to be defined. > > > Also if there is way to get the same without triggering class loading - > if the class is not loaded return a null reference. > > ClassLoader has a protected method findLoadedClass that checks if a class > is loaded by that classloader and returns the Class instance if so, or null > otherwise. > But as mentioned, the method is protected so normally not accessible > outside the classloader itself. > > /Michael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Sep 30 18:28:05 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 30 Sep 2019 19:28:05 +0100 Subject: Bytecode Instrumentation and Class Loading. In-Reply-To: References: <02f974fc-37cb-ca65-0596-30b2a1fd3feb@oracle.com> Message-ID: <5b68f0d5-795d-e62d-93fb-197842e06229@oracle.com> On 30/09/2019 19:12, Sam Thomas wrote: > Hi Michael, > > I saw this method, but was unsure if this method only looked for > loaded classes in the bootstrap class loader or if it proceeded to try > load the class. I saw this being used in the ClassLoader.loadClass and > the code and comments was what made me unsure. I'll hunt for the > native code to see what is actually being done. Meanwhile, a google > search made me notice that some implementations had this method named > findBootstrapClass0. I can't tell if this is vendor specific or was it > renamed at some point - Any idea? It's a non-public method so not intended to be used directly and way too fragile to depend on it as it could change or be removed at any time. -Alan From chris.plummer at oracle.com Mon Sep 30 19:59:40 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 30 Sep 2019 12:59:40 -0700 Subject: RFR(S): 8231288: "jhsdb jmap" test needed to reproduce issues that used to be reproduced by JShellHeapDumpTest Message-ID: Hello, Please review the following: https://bugs.openjdk.java.net/browse/JDK-8231288 http://cr.openjdk.java.net/~cjplummer/8231288/webrev.00/index.html There were a number of jmap issues that JShellHeapDumpTest.java used to reproduce that went away with the fixes for JDK-8228625 because these fixes bring the jshell process to an idle state. This new test gets rid of the Thread.sleep(2000) that JShellHeapDumpTest.java does, making it so the jshell process is still active during the jmap heap dump. The helps to reproduce 4 other jmap bugs for which there are open CRs already filed. This new test has to be problem listed as a results. Note I left mention of JDK-8196969 out of the problemlist because a fix for it is about to be pushed by someone else, and I don't want either of us to have to deal with timing issues for which CR gets pushed first. thanks, Chris From serguei.spitsyn at oracle.com Mon Sep 30 20:21:42 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 30 Sep 2019 13:21:42 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> Message-ID: <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> Hi Chris, Thank you for reviewing this! On 9/28/19 12:33, Chris Plummer wrote: > Hi Serguei, > > Overall looks good. A few questions: > > I don't understand the need for all the 'i' and 'n' theatrics in the > shouldFinish loop. Can you explain and also add a comment? I used this part from one of the old SuspendThreadList nsk tests in the vmTestbase. As I understand it, the point was to make sure the JVMTI SuspendThreadList works well wen the top frames executed on tested threads have been compiled. These code is needed to make the run() method hot. I can add a comment if you think it is not clear. > > Is this comment right? > > ?193???? // set thread is not ready again > ?194???? public void setAllThreadsReady() { > ?195???????? allThreadsReady = true; > ?196???? } Nice catch. The comment is not needed anymore. Is a leftover from previous test version. > > Also, shouldn't "setAllThreadsReady()" be static? Right. It has to be static. Will fix it. > Do you think it would be useful to also run the test with the last > thread in the list being the suspender thread? I'm not sure it is worth to do it. It'd add more complexity into the test. We could try to make the suspender thread to be random though. Thanks, Serguei > > thanks, > > Chris > > On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: >> Please, review fix for test enhancement: >> ? https://bugs.openjdk.java.net/browse/JDK-8231595 >> >> Webrev: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >> >> >> Summary: >> ? New test is a coverage for the JVMTI bug: >> ??? https://bugs.openjdk.java.net/browse/JDK-8217762 >> ????? SuspendThreadList won't work correctly if the current thread is >> not last in the list >> >> ? It provides a prove the bug JDK-8217762 does not exist >> ? as the test is passed with the current implementation. >> >> Thanks, >> Serguei > > From alexey.menkov at oracle.com Mon Sep 30 20:22:53 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Mon, 30 Sep 2019 13:22:53 -0700 Subject: RFR (XS): 8231600: Avoid GCC 8 strncpy error in jdk.jdwp.agent In-Reply-To: <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> References: <87lfu86b3u.fsf@oldenburg2.str.redhat.com> <5d8fb350-46ec-68a6-20e9-9001176f7b62@oracle.com> Message-ID: +1 --alex On 09/28/2019 12:12, Chris Plummer wrote: > Fix looks good. > > thanks, > > Chris > > On 9/28/19 2:09 AM, Florian Weimer wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8231600 >> Webrev: http://cr.openjdk.java.net/~fweimer/8231600/webrev.01/ >> >> We know the size of the string, so we can use memcpy instead of strncpy. >> >> Thanks, >> Florian > From chris.plummer at oracle.com Mon Sep 30 20:25:25 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 30 Sep 2019 13:25:25 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> Message-ID: On 9/30/19 1:21 PM, serguei.spitsyn at oracle.com wrote: > Hi Chris, > > Thank you for reviewing this! > > > On 9/28/19 12:33, Chris Plummer wrote: >> Hi Serguei, >> >> Overall looks good. A few questions: >> >> I don't understand the need for all the 'i' and 'n' theatrics in the >> shouldFinish loop. Can you explain and also add a comment? > > I used this part from one of the old SuspendThreadList nsk tests in > the vmTestbase. > As I understand it, the point was to make sure the JVMTI > SuspendThreadList works well > wen the top frames executed on tested threads have been compiled. > These code is needed to make the run() method hot. > I can add a comment if you think it is not clear. > > >> >> Is this comment right? >> >> ?193???? // set thread is not ready again >> ?194???? public void setAllThreadsReady() { >> ?195???????? allThreadsReady = true; >> ?196???? } > > Nice catch. > The comment is not needed anymore. > Is a leftover from previous test version. > >> >> Also, shouldn't "setAllThreadsReady()" be static? > > Right. It has to be static. Will fix it. > > >> Do you think it would be useful to also run the test with the last >> thread in the list being the suspender thread? > > I'm not sure it is worth to do it. > It'd add more complexity into the test. > We could try to make the suspender thread to be random though. I don't think random is good. Makes it hard to reproduce an issue if it turns up. I was thinking just rerun the test for each possible suspender. thanks, Chris > > Thanks, > Serguei > >> >> thanks, >> >> Chris >> >> On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: >>> Please, review fix for test enhancement: >>> ? https://bugs.openjdk.java.net/browse/JDK-8231595 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >>> >>> >>> Summary: >>> ? New test is a coverage for the JVMTI bug: >>> ??? https://bugs.openjdk.java.net/browse/JDK-8217762 >>> ????? SuspendThreadList won't work correctly if the current thread >>> is not last in the list >>> >>> ? It provides a prove the bug JDK-8217762 does not exist >>> ? as the test is passed with the current implementation. >>> >>> Thanks, >>> Serguei >> >> > From serguei.spitsyn at oracle.com Mon Sep 30 20:30:15 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 30 Sep 2019 13:30:15 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> Message-ID: On 9/30/19 13:25, Chris Plummer wrote: > On 9/30/19 1:21 PM, serguei.spitsyn at oracle.com wrote: >> Hi Chris, >> >> Thank you for reviewing this! >> >> >> On 9/28/19 12:33, Chris Plummer wrote: >>> Hi Serguei, >>> >>> Overall looks good. A few questions: >>> >>> I don't understand the need for all the 'i' and 'n' theatrics in the >>> shouldFinish loop. Can you explain and also add a comment? >> >> I used this part from one of the old SuspendThreadList nsk tests in >> the vmTestbase. >> As I understand it, the point was to make sure the JVMTI >> SuspendThreadList works well >> wen the top frames executed on tested threads have been compiled. >> These code is needed to make the run() method hot. >> I can add a comment if you think it is not clear. >> >> >>> >>> Is this comment right? >>> >>> ?193???? // set thread is not ready again >>> ?194???? public void setAllThreadsReady() { >>> ?195???????? allThreadsReady = true; >>> ?196???? } >> >> Nice catch. >> The comment is not needed anymore. >> Is a leftover from previous test version. >> >>> >>> Also, shouldn't "setAllThreadsReady()" be static? >> >> Right. It has to be static. Will fix it. >> >> >>> Do you think it would be useful to also run the test with the last >>> thread in the list being the suspender thread? >> >> I'm not sure it is worth to do it. >> It'd add more complexity into the test. >> We could try to make the suspender thread to be random though. > I don't think random is good. Makes it hard to reproduce an issue if > it turns up. I was thinking just rerun the test for each possible > suspender. Good idea to rerun the test and pass the suspender thread index in the arguments. Do you think, two runs would be good enough or we also need an index somewhere in the middle? Thanks, Serguei > > thanks, > > Chris >> >> Thanks, >> Serguei >> >>> >>> thanks, >>> >>> Chris >>> >>> On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: >>>> Please, review fix for test enhancement: >>>> ? https://bugs.openjdk.java.net/browse/JDK-8231595 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >>>> >>>> >>>> Summary: >>>> ? New test is a coverage for the JVMTI bug: >>>> ??? https://bugs.openjdk.java.net/browse/JDK-8217762 >>>> ????? SuspendThreadList won't work correctly if the current thread >>>> is not last in the list >>>> >>>> ? It provides a prove the bug JDK-8217762 does not exist >>>> ? as the test is passed with the current implementation. >>>> >>>> Thanks, >>>> Serguei >>> >>> >> > > From alexey.menkov at oracle.com Mon Sep 30 20:41:17 2019 From: alexey.menkov at oracle.com (Alex Menkov) Date: Mon, 30 Sep 2019 13:41:17 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> Message-ID: Name of the test doesn't look good (it's too general). Maybe SuspendWithCurrentThread? 72 throw new RuntimeException("Man: unable to prepare tested thread: " + threads[i]); Do you mean "Main" (instead of "Man")? In "run" method you don't actually need "boolean success" 109 success = checkSuspendedStatus(threads); 110 if (!success) { 111 throw new RuntimeException("Main: FAILED status returned from checkTestedThreadsSuspended"); 112 } ==> if (!checkSuspendedStatus(threads)) { throw new RuntimeException(...); } 115 success = resumeTestedThreads(); 116 if (!success) { 117 throw new RuntimeException("Main: FAILED status returned from resumeTestedThreads"); 118 } ==> if (!resumeTestedThreads()) { throw new RuntimeException(...); } Looks like all native methods (except checkTestedThreadsSuspended) can return only JNI_TRUE (on error they call jni->FatalError) So they can be void (and don't need to test returned value) --alex On 09/27/2019 18:25, serguei.spitsyn at oracle.com wrote: > Please, review fix for test enhancement: > ? https://bugs.openjdk.java.net/browse/JDK-8231595 > > Webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ > > > Summary: > ? New test is a coverage for the JVMTI bug: > ??? https://bugs.openjdk.java.net/browse/JDK-8217762 > ????? SuspendThreadList won't work correctly if the current thread is > not last in the list > > ? It provides a prove the bug JDK-8217762 does not exist > ? as the test is passed with the current implementation. > > Thanks, > Serguei From chris.plummer at oracle.com Mon Sep 30 20:49:44 2019 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 30 Sep 2019 13:49:44 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> Message-ID: On 9/30/19 1:30 PM, serguei.spitsyn at oracle.com wrote: > On 9/30/19 13:25, Chris Plummer wrote: >> On 9/30/19 1:21 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Chris, >>> >>> Thank you for reviewing this! >>> >>> >>> On 9/28/19 12:33, Chris Plummer wrote: >>>> Hi Serguei, >>>> >>>> Overall looks good. A few questions: >>>> >>>> I don't understand the need for all the 'i' and 'n' theatrics in >>>> the shouldFinish loop. Can you explain and also add a comment? >>> >>> I used this part from one of the old SuspendThreadList nsk tests in >>> the vmTestbase. >>> As I understand it, the point was to make sure the JVMTI >>> SuspendThreadList works well >>> wen the top frames executed on tested threads have been compiled. >>> These code is needed to make the run() method hot. >>> I can add a comment if you think it is not clear. >>> >>> >>>> >>>> Is this comment right? >>>> >>>> ?193???? // set thread is not ready again >>>> ?194???? public void setAllThreadsReady() { >>>> ?195???????? allThreadsReady = true; >>>> ?196???? } >>> >>> Nice catch. >>> The comment is not needed anymore. >>> Is a leftover from previous test version. >>> >>>> >>>> Also, shouldn't "setAllThreadsReady()" be static? >>> >>> Right. It has to be static. Will fix it. >>> >>> >>>> Do you think it would be useful to also run the test with the last >>>> thread in the list being the suspender thread? >>> >>> I'm not sure it is worth to do it. >>> It'd add more complexity into the test. >>> We could try to make the suspender thread to be random though. >> I don't think random is good. Makes it hard to reproduce an issue if >> it turns up. I was thinking just rerun the test for each possible >> suspender. > > Good idea to rerun the test and pass the suspender thread index in the > arguments. > Do you think, two runs would be good enough or we also need an index > somewhere in the middle? Maybe just first and last indices would be good. I'm not sure something in the middle helps any. Chris > > Thanks, > Serguei > >> >> thanks, >> >> Chris >>> >>> Thanks, >>> Serguei >>> >>>> >>>> thanks, >>>> >>>> Chris >>>> >>>> On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: >>>>> Please, review fix for test enhancement: >>>>> ? https://bugs.openjdk.java.net/browse/JDK-8231595 >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >>>>> >>>>> >>>>> Summary: >>>>> ? New test is a coverage for the JVMTI bug: >>>>> ??? https://bugs.openjdk.java.net/browse/JDK-8217762 >>>>> ????? SuspendThreadList won't work correctly if the current thread >>>>> is not last in the list >>>>> >>>>> ? It provides a prove the bug JDK-8217762 does not exist >>>>> ? as the test is passed with the current implementation. >>>>> >>>>> Thanks, >>>>> Serguei >>>> >>>> >>> >> >> > From serguei.spitsyn at oracle.com Mon Sep 30 20:57:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 30 Sep 2019 13:57:35 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> Message-ID: <9403cee8-7925-4328-876a-363dac425c94@oracle.com> Hi Alex, Thank you for reviewing this! BTW, I forgot to say that Alex already did a pre-review of this, and I've resolved many of his early comments/suggestions. On 9/30/19 13:41, Alex Menkov wrote: > Name of the test doesn't look good (it's too general). > Maybe SuspendWithCurrentThread? Good suggestion, thanks! > > > 72???????????????? throw new RuntimeException("Man: unable to prepare > tested thread: " + threads[i]); > > Do you mean "Main" (instead of "Man")? Yes, it is a typo - nice catch! > > In "run" method you don't actually need "boolean success" > > 109???????????? success = checkSuspendedStatus(threads); > ?110???????????? if (!success) { > ?111???????????????? throw new RuntimeException("Main: FAILED status > returned from checkTestedThreadsSuspended"); > ?112???????????? } > > ==> > ???????????? if (!checkSuspendedStatus(threads)) { > ???????????????? throw new RuntimeException(...); > ???????????? } > > ?115???????????? success = resumeTestedThreads(); > ?116???????????? if (!success) { > ?117???????????????? throw new RuntimeException("Main: FAILED status > returned from resumeTestedThreads"); > ?118???????????? } > > ==> > > ???????????? if (!resumeTestedThreads()) { > ???????????????? throw new RuntimeException(...); > ???????????? } This local was to make the code more readable. But I can get rid of it if you think it is not important. > Looks like all native methods (except checkTestedThreadsSuspended) can > return only JNI_TRUE (on error they call jni->FatalError) > So they can be void (and don't need to test returned value) Okay, I'll double check if we can do this simplification. Thanks, Serguei > > --alex > > On 09/27/2019 18:25, serguei.spitsyn at oracle.com wrote: >> Please, review fix for test enhancement: >> ?? https://bugs.openjdk.java.net/browse/JDK-8231595 >> >> Webrev: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >> >> >> Summary: >> ?? New test is a coverage for the JVMTI bug: >> ???? https://bugs.openjdk.java.net/browse/JDK-8217762 >> ?????? SuspendThreadList won't work correctly if the current thread >> is not last in the list >> >> ?? It provides a prove the bug JDK-8217762 does not exist >> ?? as the test is passed with the current implementation. >> >> Thanks, >> Serguei From serguei.spitsyn at oracle.com Mon Sep 30 20:58:27 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 30 Sep 2019 13:58:27 -0700 Subject: RFR (M): 8231595 [TEST] develop a test case for SuspendThreadList including current thread In-Reply-To: References: <2e8b3d82-6c71-c955-8867-dbf05edabd44@oracle.com> <46f625be-c0bc-751d-ebfb-457b7e872c65@oracle.com> <2d72e476-c392-2213-d962-44a1535160e8@oracle.com> Message-ID: On 9/30/19 13:49, Chris Plummer wrote: > On 9/30/19 1:30 PM, serguei.spitsyn at oracle.com wrote: >> On 9/30/19 13:25, Chris Plummer wrote: >>> On 9/30/19 1:21 PM, serguei.spitsyn at oracle.com wrote: >>>> Hi Chris, >>>> >>>> Thank you for reviewing this! >>>> >>>> >>>> On 9/28/19 12:33, Chris Plummer wrote: >>>>> Hi Serguei, >>>>> >>>>> Overall looks good. A few questions: >>>>> >>>>> I don't understand the need for all the 'i' and 'n' theatrics in >>>>> the shouldFinish loop. Can you explain and also add a comment? >>>> >>>> I used this part from one of the old SuspendThreadList nsk tests in >>>> the vmTestbase. >>>> As I understand it, the point was to make sure the JVMTI >>>> SuspendThreadList works well >>>> wen the top frames executed on tested threads have been compiled. >>>> These code is needed to make the run() method hot. >>>> I can add a comment if you think it is not clear. >>>> >>>> >>>>> >>>>> Is this comment right? >>>>> >>>>> ?193???? // set thread is not ready again >>>>> ?194???? public void setAllThreadsReady() { >>>>> ?195???????? allThreadsReady = true; >>>>> ?196???? } >>>> >>>> Nice catch. >>>> The comment is not needed anymore. >>>> Is a leftover from previous test version. >>>> >>>>> >>>>> Also, shouldn't "setAllThreadsReady()" be static? >>>> >>>> Right. It has to be static. Will fix it. >>>> >>>> >>>>> Do you think it would be useful to also run the test with the last >>>>> thread in the list being the suspender thread? >>>> >>>> I'm not sure it is worth to do it. >>>> It'd add more complexity into the test. >>>> We could try to make the suspender thread to be random though. >>> I don't think random is good. Makes it hard to reproduce an issue if >>> it turns up. I was thinking just rerun the test for each possible >>> suspender. >> >> Good idea to rerun the test and pass the suspender thread index in >> the arguments. >> Do you think, two runs would be good enough or we also need an index >> somewhere in the middle? > Maybe just first and last indices would be good. I'm not sure > something in the middle helps any. Agreed, thanks! Serguei > > Chris >> >> Thanks, >> Serguei >> >>> >>> thanks, >>> >>> Chris >>>> >>>> Thanks, >>>> Serguei >>>> >>>>> >>>>> thanks, >>>>> >>>>> Chris >>>>> >>>>> On 9/27/19 6:25 PM, serguei.spitsyn at oracle.com wrote: >>>>>> Please, review fix for test enhancement: >>>>>> ? https://bugs.openjdk.java.net/browse/JDK-8231595 >>>>>> >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2019/8231595-jvmti-susp-tlist.1/ >>>>>> >>>>>> >>>>>> Summary: >>>>>> ? New test is a coverage for the JVMTI bug: >>>>>> ??? https://bugs.openjdk.java.net/browse/JDK-8217762 >>>>>> ????? SuspendThreadList won't work correctly if the current >>>>>> thread is not last in the list >>>>>> >>>>>> ? It provides a prove the bug JDK-8217762 does not exist >>>>>> ? as the test is passed with the current implementation. >>>>>> >>>>>> Thanks, >>>>>> Serguei >>>>> >>>>> >>>> >>> >>> >> > > From david.holmes at oracle.com Mon Sep 30 22:52:02 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 1 Oct 2019 08:52:02 +1000 Subject: RFR: 8231289: Disentangle JvmtiRawMonitor from ObjectMonitor and clean it up In-Reply-To: <40b43578-c5e6-4b1b-f94d-69a303cfeb51@oracle.com> References: <40b43578-c5e6-4b1b-f94d-69a303cfeb51@oracle.com> Message-ID: ping! Thanks, David On 24/09/2019 3:09 pm, David Holmes wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8231289 > webrev: http://cr.openjdk.java.net/~dholmes/8231289/webrev/ > > The earlier attempt to rewrite JvmtiRawMonitor as a simple wrapper > around PlatformMonitor proved not so simple and ultimately had too many > issues due to the need to support Thread.interrupt. > > I'd previously stated in the bug report: > > "In the worst-case I suppose we could just copy ObjectMonitor to a new > class and have JvmtiRawMonitor continue to extend that (with some > additional minor adjustments) - or even just inline it all as needed." > > but hadn't looked at it in detail. Richard Reingruber did look at it and > pointed out that it is actually quite simple - we barely use any actual > code from ObjectMonitor, mainly just the state. So thanks Richard! :) > > So this change basically copies or moves anything needed by > JvmtiRawMonitor from ObjectMonitor, breaking the connection between the > two. We also copy and simplify ObjectWaiter, turning it into a QNode > internal class. There is then a lot of cleanup that was applied (and a > lot more that could still be done): > > - Removed the never implemented/used PROPER_TRANSITIONS ifdefs > - Fixed the disconnect between the types of non-JavaThreads expected by > the upper layer code and lower layer code > - cleaned up and simplified return codes > - consolidated code that is identical for JavaThreads and > non-JavaThreads (e.g. notify/notifyAll). > - removed used of TRAPS/THREAD where not appropriate and replaced with > "Thread * Self" in the style of the rest of the code > - changed recursions to be int rather than intptr_t (a "fixme" in the > ObjectMonitor code) > > > I have not changed the many style flaws with this code: > - Capitalized names > - extra spaces before ; > - ... > > but could do so if needed. I wanted to try and keep it more obvious that > the fundamental functional code is actually unmodified. > > There is one aspect that requires further explanation: the notion of > current pending monitor. The "current pending monitor" is stored in the > Thread and used by a number of introspection APIs for things like > finding monitors, doing deadlock detection, etc. The JvmtiRawMonitor > code would also set/clear itself as "current pending monitor". Most uses > of the current pending monitor actually, explicitly or implicitly, > ignore the case when the monitor is a JvmtiRawMonitor (observed by the > fact the mon->object() query returns NULL). The exception to that is > deadlock detection where raw monitors are at least partially accounted > for. To preserve that I added the notion of "current pending raw > monitor" and updated the deadlock detection code to use that. > > The test: > > > test/hotspot/jtreg/vmTestbase/nsk/jvmti/RawMonitorWait/rawmnwait005/rawmnwait005.cpp > > > was updated because I'd noticed previously that it was the only test > that used interrupt with raw monitors, but was in fact broken: the test > thread is a daemon thread so the main thread could terminate the VM > immediately after the interrupt() call, thus you would never know if the > interruption actually worked as expected. > > Testing: > ?- tiers 1 - 3 > ?- vmTestbase/nsk/monitoring/? (for deadlock detection**) > ?- vmTestbase/nsk/jdwp > ?- vmTestbase/nsk/jdb/ > ?- vmTestbase/nsk/jdi/ > ?- vmTestbase/nsk/jvmti/ > ?- serviceability/jvmti/ > ?- serviceability/jdwp > ?- JDK: java/lang/management > > ** There are no existing deadlock related tests involving > JvmtiRawMonitor. It would be interesting/useful to add them to the > existing nsk/monitoring tests that cover synchronized and JNI locking. > But it's a non-trivial enhancement that I don't really have time to do. > > Thanks, > David > -----