From daniil.x.titov at oracle.com Mon Sep 16 18:18:49 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Mon, 16 Sep 2019 11:18:49 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> Message-ID: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Hello, After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. src/hotspot/share/runtime/threadSMR.cpp 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 [4] https://bugs.openjdk.java.net/browse/JDK-8229391 ?Thank you, Daniil > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > Hi Daniil, > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > Hi David, > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > the changes you suggested: > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > - ThreadTableCreate_lock is made _safepoint_check_always; > > Okay. > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > the thread table is changed to grow on demand by the thread that is doing the addition; > > Okay - I'm happy to get the serviceThread out of the picture here. > > > - fixed nits and formatting issues. > > Okay. > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > >>> as Daniel suggested. > >> Not sure it's best to combine these, but if they are limited to the > >> changes in management.cpp only then that may be okay. > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > limited to management.cpp (plus a new test) so I left them in the webrev but > > I also could move it in the separate issue if required. > > I'd prefer this part of be separated out, but won't insist. Let's see if > Dan or Serguei have a strong opinion. > > > > src/hotspot/share/runtime/threadSMR.cpp > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > I think it cleaner/better to just use > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > I had to leave this code unchanged since it turned out the threadObj is null > > when VM is destroyed: > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > C [libjli.so+0x4333] JavaMain+0x2c3 > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > This is actually nothing to do with the VM being destroyed, but is an > issue with JNI_AttachCurrentThread and its interaction with the > ThreadSMR iterators. The attach process is: > - create JavaThread > - mark as "is attaching via jni" > - add to ThreadsList > - create java.lang.Thread object (you can only execute Java code after > you are attached) > - mark as "attach completed" > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > iterator but will have a NULL java.lang.Thread object. > > We special-case attaching threads in a number of places in the VM and I > think we should be explicitly doing something here to filter out > attaching threads, rather than just being tolerant of a NULL j.l.Thread > object. Specifically in ThreadsSMRSupport::add_thread: > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > ThreadTable::add_thread(tid, thread); > } > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > which covers the case the JNI attach encountered an error trying to > create the j.l.Thread object. > > >> src/hotspot/share/services/threadTable.cpp > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > >> The is_dead parameter still bothers me here. I can't make enough sense > >> out of the template code in ConcurrentHashtable to see why we have to > >> have it, but I'm concerned that its very existence means we perhaps > >> should not be trying to extend CHT in this context. ?? > > > > My understanding is that is_dead parameter provides a mechanism for > > ConcurrentHashtable to remove stale entries that were not explicitly > > removed by calling ConcurrentHashTable::remove() method. > > I think that just because in our case we don't use this mechanism doesn't > > mean we should not use ConcurrentHashTable. > > Can you confirm that this usage is okay with Robbin Ehn please. He's > back from vacation this week. > > >> I would still want to see what impact this has on thread > >> startup cost, both with and without the table being initialized. > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > 100,000 threads are created and started for about 15200 ms. If the thread table > > is off the test takes about 14800 ms. Based on this information the enabled > > thread table makes the thread startup about 2.7% slower. > > That doesn't sound very good. I think we may need to Claes involved to > help investigate overall performance impact here. > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > No further code comments. > > I didn't look at the test in detail. > > Thanks, > David > > > Thanks! > > --Daniil > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > Hi Daniil, > > > > Overall I think this is a reasonable approach but I would still like to > > see some performance and footprint numbers, both to verify it fixes the > > problem reported, and that we are not getting penalized elsewhere. > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > Hi David, Daniel, and Serguei, > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > The initialization allows the created but unpopulated, or partially > > populated, table to be seen by other threads - is that your intention? > > It seems it should be okay as the other threads will then race with the > > initializing thread to add specific entries, and this is a concurrent > > map so that should be functionally correct. But if so then I think you > > can also reduce the scope of the ThreadTableCreate_lock so that it > > covers creation of the table only, not the initial population of the table. > > > > I like the approach of only initializing the table when needed and using > > that to control when the add/remove-thread code needs to update the > > table. But I would still want to see what impact this has on thread > > startup cost, both with and without the table being initialized. > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > as Daniel suggested. > > > > Not sure it's best to combine these, but if they are limited to the > > changes in management.cpp only then that may be okay. It helps to be > > able to focus on the table related changes without being distracted by > > other optimizations. > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > to strip it of the all functionality that is not required in the thread table case. > > > > The revised version seems better in that regard. But I still have a > > concern, see below. > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > growing the thread table when required. > > > > Yes but why? Why can't this table be grown on demand by the thread that > > is doing the addition? For other tables we may have to delegate to the > > service thread because the current thread cannot perform the action, or > > it doesn't want to perform it at the time the need for the resize is > > detected (e.g. its detected at a safepoint and you want the resize to > > happen later outside the safepoint). It's not apparent to me that such > > restrictions apply here. > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > Ok. > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > Some specific code comments: > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > false, Monitor::_safepoint_check_never); > > > > I think this needs to be a _safepoint_check_always lock. The table will > > be created by regular JavaThreads and they should (nearly) always be > > checking for safepoints if they are going to block acquiring the lock. > > And it isn't at all obvious that the thread doing the creation can't go > > to a safepoint whilst this lock is held. > > > > --- > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > Nit: > > > > 618 JavaThread* thread = thread_at(i); > > > > you could reuse the new java_thread local you introduced at line 613 and > > just rename that "new" variable to "thread" so you don't have to change > > all other uses. > > > > 628 } else if (java_thread != NULL && ... > > > > You don't need to check != NULL here as you only get here when > > java_thread is not NULL. > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > non-null threadObj. > > > > --- > > > > src/hotspot/share/services/management.cpp > > > > 1323 if (THREAD->is_Java_thread()) { > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > These calls can only be made on a JavaThread so this be simplified to > > remove the is_Java_thread() call. Similarly in other places. > > > > --- > > > > src/hotspot/share/services/threadTable.cpp > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > I believe hotspot style is to not indent the access modifiers in C++ > > class declarations, so the above would just be: > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > etc. > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > 61 _tid(tid),_java_thread(java_thread) {} > > > > line 61 should be indented as it continues line 60. > > > > 67 class ThreadTableConfig : public AllStatic { > > ... > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > The is_dead parameter still bothers me here. I can't make enough sense > > out of the template code in ConcurrentHashtable to see why we have to > > have it, but I'm concerned that its very existence means we perhaps > > should not be trying to extend CHT in this context. ?? > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > line 116 should be indented, though in this case I think a better layout > > would be: > > > > 115 size_t start_size_log = > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > DefaultThreadTableSizeLog; > > > > 131 double ThreadTable::get_load_factor() { > > 132 return (double)_items_count/_current_size; > > 133 } > > > > Not sure that is doing what you want/expect. It will perform integer > > division and then cast that whole integer to a double. If you want > > double arithmetic you need: > > > > return ((double)_items_count)/_current_size; > > > > 180 jlong _tid; > > 181 uintx _hash; > > > > Nit: no need for all those spaces before the variable name. > > > > 183 ThreadTableLookup(jlong tid) > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > line 184 should be indented. > > > > 201 ThreadGet():_return(NULL) {} > > > > Nit: need space after : > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > 212 _has_work = false; > > > > line 211 is indented one space too far. > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > Nit: need space after , > > > > 252 return _local_table->remove(thread,lookup); > > > > Nit: need space after , > > > > Thanks, > > David > > ------ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > Hi Serguei and David, > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > in src/hotspot/share/services/management.cpp so I think that table > > > needs to enabled and populated only if it is going to be used. > > > > > > I've taken a look at the webrev below and I see that David has > > > followed up with additional comments. Before I do a crawl through > > > code review for this, I would like to see the ThreadTable stuff > > > made optional and David's other comments addressed. > > > > > > Another possible optimization is for callers of > > > find_JavaThread_from_java_tid() to save the calling thread's > > > tid value before they loop and if the current tid == saved_tid > > > then use the current JavaThread* instead of calling > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > Dan > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > From: > > > > Organization: Oracle Corporation > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > Hi Daniil, > > > > > > > > I have several quick comments. > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > 619 // to the thread table. > > > > 620 for (uint i = 0; i < length(); i++) { > > > > 621 JavaThread* thread = thread_at(i); > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > 624 return thread; > > > > 625 } > > > > 626 } > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > 628 return java_thread; > > > > 629 } > > > > 630 return NULL; > > > > 631 } > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > 633 oop tobj = java_thread->threadObj(); > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > 635 // or is starting to exit. > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > 638 } > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > A space is missed after the comma: > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > An empty line is needed before L632. > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > It'd better to list parameters in the opposite order. > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > Thanks, > > > > Serguei > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > implementation!) will need careful examination. We have to be concerned > > > > about the cost of maintaining it when it may never even be queried. You > > > > would need to look at footprint cost and performance impact. > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > next few days. I will try to look at this asap next week, but we will > > > > need a lot more data on it. > > > > > > > > Thanks, > > > > David > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > in the thread table. > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > > > > > Best regards, > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Tue Sep 17 02:26:48 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 12:26:48 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: Hi Daniil, Thanks again for your perseverance on this one. I think there is a problem with initialization of the thread table. Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid and has commenced execution of ThreadTable::lazy_initialize, but not yet marked _is_initialized as true. Now two new threads (T2 and T3) are created and start running - they aren't added to the ThreadTable yet because it isn't initialized. Now T0 also calls ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If _is_initialized is still false T0 will attempt initialization but once it gets the lock it will see the table has now been initialized by T1. It will then proceed to update the table with its own ThreadList content - adding T2 and T3. That is all fine. But now suppose T0 initially sees _is_initialized as true, it will do nothing in lazy_initialize and simply return to find_JavaThread_from_java_tid. But now T2 and T3 are missing from the ThreadTable and nothing will cause them to be added. More generally any ThreadsList that is created after the ThreadsList that will be used for initialization, may contain threads that will not be added to the table. Thanks, David On 17/09/2019 4:18 am, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Tue Sep 17 04:36:01 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Mon, 16 Sep 2019 21:36:01 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Hi David, The case you have described is exact the reason why we still have a code inside ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds the requested thread to the thread table if it is not there ( lines 614-613 below). The assumption is that it's quite uncommon and even if this is the case the linear scan happens only once per such thread. 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { 612 ThreadTable::lazy_initialize(this); 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); 614 if (thread == NULL) { 615 // If the thread is not found in the table find it 616 // with a linear search and add to the table. 617 for (uint i = 0; i < length(); i++) { 618 thread = thread_at(i); 619 oop tobj = thread->threadObj(); 620 // Ignore the thread if it hasn't run yet, has exited 621 // or is starting to exit. 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } 631 } 632 } else if (!thread->is_exiting()) { 633 return thread; 634 } 635 return NULL; 636 } Thanks, Daniil ?On 9/16/19, 7:27 PM, "David Holmes" wrote: Hi Daniil, Thanks again for your perseverance on this one. I think there is a problem with initialization of the thread table. Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid and has commenced execution of ThreadTable::lazy_initialize, but not yet marked _is_initialized as true. Now two new threads (T2 and T3) are created and start running - they aren't added to the ThreadTable yet because it isn't initialized. Now T0 also calls ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If _is_initialized is still false T0 will attempt initialization but once it gets the lock it will see the table has now been initialized by T1. It will then proceed to update the table with its own ThreadList content - adding T2 and T3. That is all fine. But now suppose T0 initially sees _is_initialized as true, it will do nothing in lazy_initialize and simply return to find_JavaThread_from_java_tid. But now T2 and T3 are missing from the ThreadTable and nothing will cause them to be added. More generally any ThreadsList that is created after the ThreadsList that will be used for initialization, may contain threads that will not be added to the table. Thanks, David On 17/09/2019 4:18 am, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Tue Sep 17 05:17:03 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 15:17:03 +1000 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Message-ID: Hi Daniil, On 17/09/2019 2:36 pm, Daniil Titov wrote: > Hi David, > > The case you have described is exact the reason why we still have a code inside > ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > the requested thread to the thread table if it is not there ( lines 614-613 below). The > assumption is that it's quite uncommon and even if this is the case the linear scan happens > only once per such thread. But that is a linear scan of the current threadslist which doesn't contain the new threads ... hmmm need to think more on this. If the tid I'm looking for is valid it should belong to a thread that existed before the current ThreadsList was created to process that tid. So it would only be a problem if the tid was speculatively produced and matched one of those new threads. But that is a race anyway. Okay. Thanks for clarifying. David ----- > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > > Thanks, > Daniil > > ?On 9/16/19, 7:27 PM, "David Holmes" wrote: > > Hi Daniil, > > Thanks again for your perseverance on this one. > > I think there is a problem with initialization of the thread table. > Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid > and has commenced execution of ThreadTable::lazy_initialize, but not yet > marked _is_initialized as true. Now two new threads (T2 and T3) are > created and start running - they aren't added to the ThreadTable yet > because it isn't initialized. Now T0 also calls > ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList > that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If > _is_initialized is still false T0 will attempt initialization but once > it gets the lock it will see the table has now been initialized by T1. > It will then proceed to update the table with its own ThreadList content > - adding T2 and T3. That is all fine. But now suppose T0 initially sees > _is_initialized as true, it will do nothing in lazy_initialize and > simply return to find_JavaThread_from_java_tid. But now T2 and T3 are > missing from the ThreadTable and nothing will cause them to be added. > > More generally any ThreadsList that is created after the ThreadsList > that will be used for initialization, may contain threads that will not > be added to the table. > > Thanks, > David > > On 17/09/2019 4:18 am, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Tue Sep 17 08:53:54 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 01:53:54 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Tue Sep 17 09:04:42 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 02:04:42 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Tue Sep 17 09:10:10 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 17 Sep 2019 02:10:10 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> Message-ID: <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Hi Daniil, On 9/16/19 21:36, Daniil Titov wrote: > Hi David, > > The case you have described is exact the reason why we still have a code inside > ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > the requested thread to the thread table if it is not there ( lines 614-613 below). I disagree because it is easy to avoid concurrent ThreadTable initialization (please, see my separate email). The reason for this code is to cover a case of late/lazy ThreadTable initialization. Thanks, Serguei > The > assumption is that it's quite uncommon and even if this is the case the linear scan happens > only once per such thread. > > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > > Thanks, > Daniil > > ?On 9/16/19, 7:27 PM, "David Holmes" wrote: > > Hi Daniil, > > Thanks again for your perseverance on this one. > > I think there is a problem with initialization of the thread table. > Suppose thread T1 has called ThreadsList::find_JavaThread_from_java_tid > and has commenced execution of ThreadTable::lazy_initialize, but not yet > marked _is_initialized as true. Now two new threads (T2 and T3) are > created and start running - they aren't added to the ThreadTable yet > because it isn't initialized. Now T0 also calls > ThreadsList::find_JavaThread_from_java_tid using an updated ThreadsList > that contains T2 and T3. It also calls ThreadTable::lazy_initialize. If > _is_initialized is still false T0 will attempt initialization but once > it gets the lock it will see the table has now been initialized by T1. > It will then proceed to update the table with its own ThreadList content > - adding T2 and T3. That is all fine. But now suppose T0 initially sees > _is_initialized as true, it will do nothing in lazy_initialize and > simply return to find_JavaThread_from_java_tid. But now T2 and T3 are > missing from the ThreadTable and nothing will cause them to be added. > > More generally any ThreadsList that is created after the ThreadsList > that will be used for initialization, may contain threads that will not > be added to the table. > > Thanks, > David > > On 17/09/2019 4:18 am, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Tue Sep 17 10:46:07 2019 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Sep 2019 20:46:07 +1000 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Message-ID: Hi Serguei, On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > > On 9/16/19 21:36, Daniil Titov wrote: >> Hi David, >> >> The case you have described is exact the reason why we still have a >> code inside >> ThreadsList::find_JavaThread_from_java_tid() method that does a linear >> scan and adds >> ? the requested thread to the thread table if it is not there ( lines >> 614-613 below). > > I disagree because it is easy to avoid concurrent ThreadTable > initialization (please, see my separate email). > The reason for this code is to cover a case of late/lazy ThreadTable > initialization. I'm not sure I follow. With the current code if two threads are racing to initialize the ThreadTable with ThreadsLists that contain a different set of threads then there are two possibilities with regards to the interleaving. Assume T1 initializes the table with its set of threads and so finds the tid it is looking for in the table. Meanwhile T2 is racing with the initialization logic: - If T2 sees _is_initialized then lazy_initialization does nothing for T2, and the additional threads in its ThreadsList (say T3 and T4) are not added to the table. But the specific thread associated with the tid (say T3) will be found by linear search of the ThreadsList and then added. If any other threads come searching for T4 they too will not find it in the ThreadTable but instead perform the linear search of their ThreadsList (and add it). - if T2 doesn't see _is_initialized at first it will try to acquire the lock, and eventually see _is_initialized is true, at which point it will try to add all of its thread's to the table (so T3 and T4 will be added). When lazy_initialize returns, T3 will be found in the table and returned. If any other threads come searching for T4 they will also find it in the table. With your suggested code change this second case is not possible so for any racing initialization the lookup of any threads not in the original ThreadsList will always result in using the linear search before adding to the table. Both seem correct to me. Which one is more efficient will totally depend on the number of differences between the ThreadsLists and whether the code ever tries to look up those additional threads. If we assume racing initialization is likely to be rare anyway (because generally one thread is in charge of doing the monitoring) then the choice seems somewhat arbitrary. Cheers, David ----- > Thanks, > Serguei > >> ??? The >> assumption is that it's quite uncommon and even if this is the case >> the linear scan happens >> only once per such thread. >> >> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >> java_tid) const { >> ? 612?? ThreadTable::lazy_initialize(this); >> ? 613?? JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); >> ? 614?? if (thread == NULL) { >> ? 615???? // If the thread is not found in the table find it >> ? 616???? // with a linear search and add to the table. >> ? 617???? for (uint i = 0; i < length(); i++) { >> ? 618?????? thread = thread_at(i); >> ? 619?????? oop tobj = thread->threadObj(); >> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ? 621?????? // or is starting to exit. >> ? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ? 623???????? MutexLocker ml(Threads_lock); >> ? 624???????? // Must be inside the lock to ensure that we don't add >> the thread to the table >> ? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ? 626???????? if (!thread->is_exiting()) { >> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ? 628?????????? return thread; >> ? 629???????? } >> ? 630?????? } >> ? 631???? } >> ? 632?? } else if (!thread->is_exiting()) { >> ? 633?????? return thread; >> ? 634?? } >> ? 635?? return NULL; >> ? 636 } >> >> Thanks, >> Daniil >> >> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >> >> ???? Hi Daniil, >> ???? Thanks again for your perseverance on this one. >> ???? I think there is a problem with initialization of the thread table. >> ???? Suppose thread T1 has called >> ThreadsList::find_JavaThread_from_java_tid >> ???? and has commenced execution of ThreadTable::lazy_initialize, but >> not yet >> ???? marked _is_initialized as true. Now two new threads (T2 and T3) are >> ???? created and start running - they aren't added to the ThreadTable yet >> ???? because it isn't initialized. Now T0 also calls >> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >> ThreadsList >> ???? that contains T2 and T3. It also calls >> ThreadTable::lazy_initialize. If >> ???? _is_initialized is still false T0 will attempt initialization but >> once >> ???? it gets the lock it will see the table has now been initialized >> by T1. >> ???? It will then proceed to update the table with its own ThreadList >> content >> ???? - adding T2 and T3. That is all fine. But now suppose T0 >> initially sees >> ???? _is_initialized as true, it will do nothing in lazy_initialize and >> ???? simply return to find_JavaThread_from_java_tid. But now T2 and T3 >> are >> ???? missing from the ThreadTable and nothing will cause them to be >> added. >> ???? More generally any ThreadsList that is created after the ThreadsList >> ???? that will be used for initialization, may contain threads that >> will not >> ???? be added to the table. >> ???? Thanks, >> ???? David >> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on the >> performance (thanks a lot Claes for helping with it!) the conclusion >> was that the impact on the thread startup time is not a blocker for >> this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just to >> remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] >> minus changes in src/hotspot/share/services/management.cpp (that were >> factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code that >> adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev:? https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that may >> be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue if >> required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c]? jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333]? JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159]? ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me here. >> I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In case >> when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time the >> thread table >> ???? >????????? >????? >????? >?? is initialized with the threads from >> the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only initializing >> the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, but >> if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then that >> may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in that >> regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori or >> deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably originally >> suggested in the patch attached to the JBS issue, should be used.? It >> will make >> ???? >????????? >????? >????? > the backporting more complicated, >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? +?? def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324?????????? JavaThread* >> current_thread = (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not indent >> the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60???? ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = size_log >> > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? >????? DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong????????? _tid; >> ???? >????????? >????? >????? 181???? uintx???????? _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces before >> the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183???? ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201???? ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211??? assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil Titov >> wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with an >> unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about the >> maintaining the thread table when it may never even be queried, one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it on >> and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as David >> H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and if >> the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot c/c++ >> files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616???? if (java_thread == >> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? >?? 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? >?? 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? >?? 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? >?? 620???????? for (uint i = 0; >> i < length(); i++) { >> ???? >????????? >????? >????? >????? >?? 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? >?? 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? >?? 623 >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? >?? 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? >?? 625???????????? } >> ???? >????????? >????? >????? >????? >?? 626???????? } >> ???? >????????? >????? >????? >????? >?? 627???? } else if >> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? >?? 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629???? } >> ???? >????????? >????? >????? >????? >?? 630???? return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633???? oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634???? // Ignore the thread >> if it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635???? // or is starting to >> exit. >> ???? >????????? >????? >????? >????? >?? 636???? return (tobj != NULL >> && !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? >?? 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name 'is_valid_java_thread' >> looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? >????? implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Unfortunately I'm just >> about to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the mapping >> between the thread ids and JavaThread objects and replaces the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> > From daniil.x.titov at oracle.com Wed Sep 18 00:13:57 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 17 Sep 2019 17:13:57 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> Message-ID: <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Hi Serguei, Please find below my answers to the concerns you mentioned in the previous email. 1. > I have a concern about the checks for thread->is_exiting(). > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > Does it make sense to develop a test that would cover these cases? I agree, it doesn't really provide any protection so it makes sense just remove it. The current implementation find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. 2. > - the lines 105-108 can result in adding exiting threads into the ThreadTable I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() 3. > I would suggest to rewrite this fragment in a safe way: > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > as: > { > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > return; > } > create_table(threads->length()); > _is_initialized = true; > } It was an intension to not block while populating the table with the threads from the current thread list. There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for it to complete since the requested thread could be not present in the thread list that triggers the thread table initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table and thus avoid the linear scan when these thread are looked up for the first time. 4. >> The case you have described is exact the reason why we still have a code inside >> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >> the requested thread to the thread table if it is not there ( lines 614-613 below). > I disagree because it is easy to avoid concurrent ThreadTable > initialization (please, see my separate email). > The reason for this code is to cover a case of late/lazy ThreadTable > initialization. David Holmes replied to this in a separate email providing a very detailed explanation of the possible cases and how the proposed implementation satisfies them. Best regards, Daniil From: "serguei.spitsyn at oracle.com" Date: Tuesday, September 17, 2019 at 1:53 AM To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) Hi Daniil, Thank you for you patience in working on this issue! Also, I like that the current thread related optimizations in management.cpp were factored out. It was a good idea to separate them. I have a concern about the checks for thread->is_exiting(). The threads are added to and removed from the ThreadTable under protection of Threads_lock. However, the thread->is_exiting() checks are not protected, and so, they are racy. There is a couple of such checks to mention: 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { 612 ThreadTable::lazy_initialize(this); 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); 614 if (thread == NULL) { 615 // If the thread is not found in the table find it 616 // with a linear search and add to the table. 617 for (uint i = 0; i < length(); i++) { 618 thread = thread_at(i); 619 oop tobj = thread->threadObj(); 620 // Ignore the thread if it hasn't run yet, has exited 621 // or is starting to exit. 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } 631 } 632 } else if (!thread->is_exiting()) { 633 return thread; 634 } 635 return NULL; 636 } ? ... 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { 94 if (!_is_initialized) { 95 { 96 MutexLocker ml(ThreadTableCreate_lock); 97 if (!_is_initialized) { 98 create_table(threads->length()); 99 _is_initialized = true; 100 } 101 } 102 for (uint i = 0; i < threads->length(); i++) { 103 JavaThread* thread = threads->thread_at(i); 104 oop tobj = thread->threadObj(); 105 if (tobj != NULL && !thread->is_exiting()) { 106 jlong java_tid = java_lang_Thread::thread_id(tobj); 107 add_thread(java_tid, thread); 108 } 109 } 110 } 111 } A thread may start exiting right after the checks at the lines 626 and 105. So that: ?- the lines 632-633 are useless as they do not really protect from returning an exiting thread ?- the lines 105-108 can result in adding exiting threads into the ThreadTable Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they are protected with the Threads_lock. But the returned thread still can exit after that. It is interesting what might happen if an exiting thread is returned by the ThreadsList::find_JavaThread_from_java_tid (). Does it make sense to develop a test that would cover these cases? Thanks, Serguei On 9/16/19 11:18, Daniil Titov wrote: Hello, After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. src/hotspot/share/runtime/threadSMR.cpp 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { 623 MutexLocker ml(Threads_lock); 624 // Must be inside the lock to ensure that we don't add the thread to the table 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() 626 if (!thread->is_exiting()) { 627 ThreadTable::add_thread(java_tid, thread); 628 return thread; 629 } 630 } [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 [4] https://bugs.openjdk.java.net/browse/JDK-8229391 ?Thank you, Daniil > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > Hi Daniil, > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > Hi David, > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > the changes you suggested: > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > - ThreadTableCreate_lock is made _safepoint_check_always; > > Okay. > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > the thread table is changed to grow on demand by the thread that is doing the addition; > > Okay - I'm happy to get the serviceThread out of the picture here. > > > - fixed nits and formatting issues. > > Okay. > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > >>> as Daniel suggested. > >> Not sure it's best to combine these, but if they are limited to the > >> changes in management.cpp only then that may be okay. > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > limited to management.cpp (plus a new test) so I left them in the webrev but > > I also could move it in the separate issue if required. > > I'd prefer this part of be separated out, but won't insist. Let's see if > Dan or Serguei have a strong opinion. > > > > src/hotspot/share/runtime/threadSMR.cpp > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > I think it cleaner/better to just use > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > I had to leave this code unchanged since it turned out the threadObj is null > > when VM is destroyed: > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > C [libjli.so+0x4333] JavaMain+0x2c3 > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > This is actually nothing to do with the VM being destroyed, but is an > issue with JNI_AttachCurrentThread and its interaction with the > ThreadSMR iterators. The attach process is: > - create JavaThread > - mark as "is attaching via jni" > - add to ThreadsList > - create java.lang.Thread object (you can only execute Java code after > you are attached) > - mark as "attach completed" > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > iterator but will have a NULL java.lang.Thread object. > > We special-case attaching threads in a number of places in the VM and I > think we should be explicitly doing something here to filter out > attaching threads, rather than just being tolerant of a NULL j.l.Thread > object. Specifically in ThreadsSMRSupport::add_thread: > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > ThreadTable::add_thread(tid, thread); > } > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > which covers the case the JNI attach encountered an error trying to > create the j.l.Thread object. > > >> src/hotspot/share/services/threadTable.cpp > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > >> The is_dead parameter still bothers me here. I can't make enough sense > >> out of the template code in ConcurrentHashtable to see why we have to > >> have it, but I'm concerned that its very existence means we perhaps > >> should not be trying to extend CHT in this context. ?? > > > > My understanding is that is_dead parameter provides a mechanism for > > ConcurrentHashtable to remove stale entries that were not explicitly > > removed by calling ConcurrentHashTable::remove() method. > > I think that just because in our case we don't use this mechanism doesn't > > mean we should not use ConcurrentHashTable. > > Can you confirm that this usage is okay with Robbin Ehn please. He's > back from vacation this week. > > >> I would still want to see what impact this has on thread > >> startup cost, both with and without the table being initialized. > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > 100,000 threads are created and started for about 15200 ms. If the thread table > > is off the test takes about 14800 ms. Based on this information the enabled > > thread table makes the thread startup about 2.7% slower. > > That doesn't sound very good. I think we may need to Claes involved to > help investigate overall performance impact here. > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > No further code comments. > > I didn't look at the test in detail. > > Thanks, > David > > > Thanks! > > --Daniil > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > Overall I think this is a reasonable approach but I would still like to > > see some performance and footprint numbers, both to verify it fixes the > > problem reported, and that we are not getting penalized elsewhere. > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > Hi David, Daniel, and Serguei, > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > The initialization allows the created but unpopulated, or partially > > populated, table to be seen by other threads - is that your intention? > > It seems it should be okay as the other threads will then race with the > > initializing thread to add specific entries, and this is a concurrent > > map so that should be functionally correct. But if so then I think you > > can also reduce the scope of the ThreadTableCreate_lock so that it > > covers creation of the table only, not the initial population of the table. > > > > I like the approach of only initializing the table when needed and using > > that to control when the add/remove-thread code needs to update the > > table. But I would still want to see what impact this has on thread > > startup cost, both with and without the table being initialized. > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > as Daniel suggested. > > > > Not sure it's best to combine these, but if they are limited to the > > changes in management.cpp only then that may be okay. It helps to be > > able to focus on the table related changes without being distracted by > > other optimizations. > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > to strip it of the all functionality that is not required in the thread table case. > > > > The revised version seems better in that regard. But I still have a > > concern, see below. > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > growing the thread table when required. > > > > Yes but why? Why can't this table be grown on demand by the thread that > > is doing the addition? For other tables we may have to delegate to the > > service thread because the current thread cannot perform the action, or > > it doesn't want to perform it at the time the need for the resize is > > detected (e.g. its detected at a safepoint and you want the resize to > > happen later outside the safepoint). It's not apparent to me that such > > restrictions apply here. > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > Ok. > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > Some specific code comments: > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > false, Monitor::_safepoint_check_never); > > > > I think this needs to be a _safepoint_check_always lock. The table will > > be created by regular JavaThreads and they should (nearly) always be > > checking for safepoints if they are going to block acquiring the lock. > > And it isn't at all obvious that the thread doing the creation can't go > > to a safepoint whilst this lock is held. > > > > --- > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > Nit: > > > > 618 JavaThread* thread = thread_at(i); > > > > you could reuse the new java_thread local you introduced at line 613 and > > just rename that "new" variable to "thread" so you don't have to change > > all other uses. > > > > 628 } else if (java_thread != NULL && ... > > > > You don't need to check != NULL here as you only get here when > > java_thread is not NULL. > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > non-null threadObj. > > > > --- > > > > src/hotspot/share/services/management.cpp > > > > 1323 if (THREAD->is_Java_thread()) { > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > These calls can only be made on a JavaThread so this be simplified to > > remove the is_Java_thread() call. Similarly in other places. > > > > --- > > > > src/hotspot/share/services/threadTable.cpp > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > I believe hotspot style is to not indent the access modifiers in C++ > > class declarations, so the above would just be: > > > > 55 class ThreadTableEntry : public CHeapObj { > > 56 private: > > 57 jlong _tid; > > > > etc. > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > 61 _tid(tid),_java_thread(java_thread) {} > > > > line 61 should be indented as it continues line 60. > > > > 67 class ThreadTableConfig : public AllStatic { > > ... > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > The is_dead parameter still bothers me here. I can't make enough sense > > out of the template code in ConcurrentHashtable to see why we have to > > have it, but I'm concerned that its very existence means we perhaps > > should not be trying to extend CHT in this context. ?? > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > line 116 should be indented, though in this case I think a better layout > > would be: > > > > 115 size_t start_size_log = > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > DefaultThreadTableSizeLog; > > > > 131 double ThreadTable::get_load_factor() { > > 132 return (double)_items_count/_current_size; > > 133 } > > > > Not sure that is doing what you want/expect. It will perform integer > > division and then cast that whole integer to a double. If you want > > double arithmetic you need: > > > > return ((double)_items_count)/_current_size; > > > > 180 jlong _tid; > > 181 uintx _hash; > > > > Nit: no need for all those spaces before the variable name. > > > > 183 ThreadTableLookup(jlong tid) > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > line 184 should be indented. > > > > 201 ThreadGet():_return(NULL) {} > > > > Nit: need space after : > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > 212 _has_work = false; > > > > line 211 is indented one space too far. > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > Nit: need space after , > > > > 252 return _local_table->remove(thread,lookup); > > > > Nit: need space after , > > > > Thanks, > > David > > ------ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > Hi Serguei and David, > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > in src/hotspot/share/services/management.cpp so I think that table > > > needs to enabled and populated only if it is going to be used. > > > > > > I've taken a look at the webrev below and I see that David has > > > followed up with additional comments. Before I do a crawl through > > > code review for this, I would like to see the ThreadTable stuff > > > made optional and David's other comments addressed. > > > > > > Another possible optimization is for callers of > > > find_JavaThread_from_java_tid() to save the calling thread's > > > tid value before they loop and if the current tid == saved_tid > > > then use the current JavaThread* instead of calling > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > Dan > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > Organization: Oracle Corporation > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > Hi Daniil, > > > > > > > > I have several quick comments. > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > 619 // to the thread table. > > > > 620 for (uint i = 0; i < length(); i++) { > > > > 621 JavaThread* thread = thread_at(i); > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > 624 return thread; > > > > 625 } > > > > 626 } > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > 628 return java_thread; > > > > 629 } > > > > 630 return NULL; > > > > 631 } > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > 633 oop tobj = java_thread->threadObj(); > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > 635 // or is starting to exit. > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > 638 } > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > A space is missed after the comma: > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > An empty line is needed before L632. > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > It'd better to list parameters in the opposite order. > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > Thanks, > > > > Serguei > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > implementation!) will need careful examination. We have to be concerned > > > > about the cost of maintaining it when it may never even be queried. You > > > > would need to look at footprint cost and performance impact. > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > next few days. I will try to look at this asap next week, but we will > > > > need a lot more data on it. > > > > > > > > Thanks, > > > > David > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > in the thread table. > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > > > > > Best regards, > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Wed Sep 18 07:25:29 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Sep 2019 17:25:29 +1000 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> Message-ID: <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> Hi Serguei, In the interests of full disclosure I was the one who told Daniil to not hold the lock while populating the table. I'll leave you two to work out which way you want to go there. Thanks, David On 18/09/2019 5:13 pm, serguei.spitsyn at oracle.com wrote: > Hi David, > > > On 9/17/19 03:46, David Holmes wrote: >> Hi Serguei, >> >> On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: >>> Hi Daniil, >>> >>> >>> On 9/16/19 21:36, Daniil Titov wrote: >>>> Hi David, >>>> >>>> The case you have described is exact the reason why we still have a >>>> code inside >>>> ThreadsList::find_JavaThread_from_java_tid() method that does a >>>> linear scan and adds >>>> ? the requested thread to the thread table if it is not there ( >>>> lines 614-613 below). >>> >>> I disagree because it is easy to avoid concurrent ThreadTable >>> initialization (please, see my separate email). >>> The reason for this code is to cover a case of late/lazy ThreadTable >>> initialization. >> >> I'm not sure I follow. With the current code if two threads are racing >> to initialize the ThreadTable with ThreadsLists that contain a >> different set of threads then there are two possibilities with regards >> to the interleaving. Assume T1 initializes the table with its set of >> threads and so finds the tid it is looking for in the table. Meanwhile >> T2 is racing with the initialization logic: >> >> - If T2 sees _is_initialized then lazy_initialization does nothing for >> T2, and the additional threads in its ThreadsList (say T3 and T4) are >> not added to the table. But the specific thread associated with the >> tid (say T3) will be found by linear search of the ThreadsList and >> then added. If any other threads come searching for T4 they too will >> not find it in the ThreadTable but instead perform the linear search >> of their ThreadsList (and add it). >> >> - if T2 doesn't see _is_initialized at first it will try to acquire >> the lock, and eventually see _is_initialized is true, at which point >> it will try to add all of its thread's to the table (so T3 and T4 will >> be added). When lazy_initialize returns, T3 will be found in the table >> and returned. If any other threads come searching for T4 they will >> also find it in the table. > > My main concerns are simplicity and reliability. > I do no care much about extra overhead at the ThreadTable initialization. > A probability of the ThreadTable::lazy_initialize() being called > concurrently is low. > Also, it might happen only once for the whole VM process execution. > > I was wrong by thinking that adding new threads to the ThreadTable after > its initialization > will result in thread linear search as well. So my conclusion was that > we should not care > if it happens once more at lazy initialization point. > But now I see that after ThreadTable::is_initialized() returns true the > ThreadsSMRSupport::add_thread() > makes a call to the ThreadTable::add_thread(). So, no more linear search > will happen. > > > However, it seems to me, a possible concurrent lazy initialization in > the webrev.06 introduces > its own extra overhead - competing threads (suppose, we have just two of > them) will do > the same amount of work concurrently: > ?? - all indirect memory readings, lookup/comparisons and other checks > will be performed twice > ?? - new ThreadTableEntry can be allocated twice for some threads in > the list > ???? (depending on how ConcurrentHashTable is implemented, there can be > a potential a memory leak) > > So, I doubt we win much in performance here but can loose in reliability. > > > I'd suggest to simplify the lazy initialization code and make it more > reliable this way: > > if (!_is_initialized) { > MutexLocker ml(ThreadTableCreate_lock); > if (!_is_initialized) { > create_table(threads->length()); > _is_initialized = true; > } > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL && !thread->is_exiting()) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > add_thread(java_tid, thread); > } > } > } > >> With your suggested code change this second case is not possible so >> for any racing initialization the lookup of any threads not in the >> original ThreadsList will always result in using the linear search >> before adding to the table. > > Yes, but I did not care about this. > The overhead is expected to be lower than the lazy initialization cost, > especially because the probability of concurrent initialization is low. > >> Both seem correct to me. Which one is more efficient will totally >> depend on the number of differences between the ThreadsLists and >> whether the code ever tries to look up those additional threads. If we >> assume racing initialization is likely to be rare anyway (because >> generally one thread is in charge of doing the monitoring) then the >> choice seems somewhat arbitrary. > > I agree, but have a little preference in favor of simplicity. > It was a good discussion though. :) > > Thanks, > Serguei > >> Cheers, >> David >> ----- >> >>> Thanks, >>> Serguei >>> >>>> ??? The >>>> assumption is that it's quite uncommon and even if this is the case >>>> the linear scan happens >>>> only once per such thread. >>>> >>>> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >>>> java_tid) const { >>>> ? 612?? ThreadTable::lazy_initialize(this); >>>> ? 613?? JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); >>>> ? 614?? if (thread == NULL) { >>>> ? 615???? // If the thread is not found in the table find it >>>> ? 616???? // with a linear search and add to the table. >>>> ? 617???? for (uint i = 0; i < length(); i++) { >>>> ? 618?????? thread = thread_at(i); >>>> ? 619?????? oop tobj = thread->threadObj(); >>>> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >>>> ? 621?????? // or is starting to exit. >>>> ? 622?????? if (tobj != NULL && java_tid == >>>> java_lang_Thread::thread_id(tobj)) { >>>> ? 623???????? MutexLocker ml(Threads_lock); >>>> ? 624???????? // Must be inside the lock to ensure that we don't add >>>> the thread to the table >>>> ? 625???????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ? 626???????? if (!thread->is_exiting()) { >>>> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >>>> ? 628?????????? return thread; >>>> ? 629???????? } >>>> ? 630?????? } >>>> ? 631???? } >>>> ? 632?? } else if (!thread->is_exiting()) { >>>> ? 633?????? return thread; >>>> ? 634?? } >>>> ? 635?? return NULL; >>>> ? 636 } >>>> >>>> Thanks, >>>> Daniil >>>> >>>> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >>>> >>>> ???? Hi Daniil, >>>> ???? Thanks again for your perseverance on this one. >>>> ???? I think there is a problem with initialization of the thread >>>> table. >>>> ???? Suppose thread T1 has called >>>> ThreadsList::find_JavaThread_from_java_tid >>>> ???? and has commenced execution of ThreadTable::lazy_initialize, >>>> but not yet >>>> ???? marked _is_initialized as true. Now two new threads (T2 and T3) >>>> are >>>> ???? created and start running - they aren't added to the >>>> ThreadTable yet >>>> ???? because it isn't initialized. Now T0 also calls >>>> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >>>> ThreadsList >>>> ???? that contains T2 and T3. It also calls >>>> ThreadTable::lazy_initialize. If >>>> ???? _is_initialized is still false T0 will attempt initialization >>>> but once >>>> ???? it gets the lock it will see the table has now been initialized >>>> by T1. >>>> ???? It will then proceed to update the table with its own >>>> ThreadList content >>>> ???? - adding T2 and T3. That is all fine. But now suppose T0 >>>> initially sees >>>> ???? _is_initialized as true, it will do nothing in lazy_initialize and >>>> ???? simply return to find_JavaThread_from_java_tid. But now T2 and >>>> T3 are >>>> ???? missing from the ThreadTable and nothing will cause them to be >>>> added. >>>> ???? More generally any ThreadsList that is created after the >>>> ThreadsList >>>> ???? that will be used for initialization, may contain threads that >>>> will not >>>> ???? be added to the table. >>>> ???? Thanks, >>>> ???? David >>>> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >>>> ???? > Hello, >>>> ???? > >>>> ???? > After investigating with Claes the impact of this change on >>>> the performance (thanks a lot Claes for helping with it!) the >>>> conclusion was that the impact on the thread startup time is not a >>>> blocker for this change. >>>> ???? > >>>> ???? > I also measured the memory footprint using Native Memory >>>> Tracking and results showed around 40 bytes per live thread. >>>> ???? > >>>> ???? > Please review a new version of the fix, webrev.06 [1].? Just >>>> to remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 >>>> [3] minus changes in src/hotspot/share/services/management.cpp (that >>>> were factored out to a separate issue [4]) and plus a change in >>>> ThreadsList::find_JavaThread_from_java_tid() method (please, see >>>> below)? that addresses the problem Robbin found and puts the code >>>> that adds a new thread to the thread table inside Threads_lock. >>>> ???? > >>>> ???? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? > >>>> ???? > 622?????? if (tobj != NULL && java_tid == >>>> java_lang_Thread::thread_id(tobj)) { >>>> ???? > 623???????? MutexLocker ml(Threads_lock); >>>> ???? > 624???????? // Must be inside the lock to ensure that we >>>> don't add the thread to the table >>>> ???? > 625???????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ???? > 626???????? if (!thread->is_exiting()) { >>>> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >>>> ???? > 628?????????? return thread; >>>> ???? > 629???????? } >>>> ???? > 630?????? } >>>> ???? > >>>> ???? > [1] Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >>>> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >>>> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >>>> ???? > >>>> ???? > ?Thank you, >>>> ???? > Daniil >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? >????????? > >>>> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >>>> wrote: >>>> ???? >????????? > >>>> ???? >????????? >????? Hi Daniil, >>>> ???? >????????? > >>>> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >>>> ???? >????????? >????? > Hi David, >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > Thank you for your detailed review. Please >>>> review a new version of the fix that includes >>>> ???? >????????? >????? > the changes you suggested: >>>> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced >>>> to cover the creation of the table only; >>>> ???? >????????? >????? > - ThreadTableCreate_lock is made >>>> _safepoint_check_always; >>>> ???? >????????? > >>>> ???? >????????? >????? Okay. >>>> ???? >????????? > >>>> ???? >????????? >????? > - ServiceThread is no longer responsible >>>> for the resizing of the thread table, instead, >>>> ???? >????????? >????? >??? the thread table is changed to grow on >>>> demand by the thread that is doing the addition; >>>> ???? >????????? > >>>> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >>>> of the picture here. >>>> ???? >????????? > >>>> ???? >????????? >????? > - fixed nits and formatting issues. >>>> ???? >????????? > >>>> ???? >????????? >????? Okay. >>>> ???? >????????? > >>>> ???? >????????? >????? >>> The change also includes additional >>>> optimization for some callers of find_JavaThread_from_java_tid() >>>> ???? >????????? >????? >>>?? as Daniel suggested. >>>> ???? >????????? >????? >> Not sure it's best to combine these, but >>>> if they are limited to the >>>> ???? >????????? >????? >> changes in management.cpp only then that >>>> may be okay. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > The additional optimization for some >>>> callers of find_JavaThread_from_java_tid() is >>>> ???? >????????? >????? > limited to management.cpp (plus a new test) >>>> so I left them in the webrev? but >>>> ???? >????????? >????? > I also could move it in the separate issue >>>> if required. >>>> ???? >????????? > >>>> ???? >????????? >????? I'd prefer this part of be separated out, but >>>> won't insist. Let's see if >>>> ???? >????????? >????? Dan or Serguei have a strong opinion. >>>> ???? >????????? > >>>> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? >????????? >????? >??? >755???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >??? > 926???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >?? >? I think it cleaner/better to just use >>>> ???? >????????? >????? >?? > jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >>>> JavaThread and it has to have a >>>> ???? >????????? >????? >?? > non-null threadObj. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > I had to leave this code unchanged since it >>>> turned out the threadObj is null >>>> ???? >????????? >????? > when VM is destroyed: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >>>> oopDesc::long_field(int) const+0x67 >>>> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >>>> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >>>> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >>>> Threads::add(JavaThread*, bool)+0x82 >>>> ???? >????????? >????? > V? [libjvm.so+0xef8369] >>>> attach_current_thread.part.197+0xc9 >>>> ???? >????????? >????? > V? [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c >>>> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >>>> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >>>> ???? >????????? > >>>> ???? >????????? >????? This is actually nothing to do with the VM >>>> being destroyed, but is an >>>> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >>>> interaction with the >>>> ???? >????????? >????? ThreadSMR iterators. The attach process is: >>>> ???? >????????? >????? - create JavaThread >>>> ???? >????????? >????? - mark as "is attaching via jni" >>>> ???? >????????? >????? - add to ThreadsList >>>> ???? >????????? >????? - create java.lang.Thread object (you can >>>> only execute Java code after >>>> ???? >????????? >????? you are attached) >>>> ???? >????????? >????? - mark as "attach completed" >>>> ???? >????????? > >>>> ???? >????????? >????? So while a thread "is attaching" it will be >>>> seen by the ThreadSMR thread >>>> ???? >????????? >????? iterator but will have a NULL >>>> java.lang.Thread object. >>>> ???? >????????? > >>>> ???? >????????? >????? We special-case attaching threads in a number >>>> of places in the VM and I >>>> ???? >????????? >????? think we should be explicitly doing something >>>> here to filter out >>>> ???? >????????? >????? attaching threads, rather than just being >>>> tolerant of a NULL j.l.Thread >>>> ???? >????????? >????? object. Specifically in >>>> ThreadsSMRSupport::add_thread: >>>> ???? >????????? > >>>> ???? >????????? >????? if (ThreadTable::is_initialized() && >>>> !thread->is_attaching_via_jni()) { >>>> ???? >????????? >???????? jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >>>> ???? >????????? >????? } >>>> ???? >????????? > >>>> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >>>> we can use the same guard, >>>> ???? >????????? >????? which covers the case the JNI attach >>>> encountered an error trying to >>>> ???? >????????? >????? create the j.l.Thread object. >>>> ???? >????????? > >>>> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >>>> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >>>> value, bool* is_dead) { >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >> The is_dead parameter still bothers me >>>> here. I can't make enough sense >>>> ???? >????????? >????? >> out of the template code in >>>> ConcurrentHashtable to see why we have to >>>> ???? >????????? >????? >> have it, but I'm concerned that its very >>>> existence means we perhaps >>>> ???? >????????? >????? >> should not be trying to extend CHT in this >>>> context. ?? >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > My understanding is that is_dead parameter >>>> provides a mechanism for >>>> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >>>> that were not explicitly >>>> ???? >????????? >????? > removed by calling >>>> ConcurrentHashTable::remove() method. >>>> ???? >????????? >????? > I think that just because in our case we >>>> don't use this mechanism doesn't >>>> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >>>> ???? >????????? > >>>> ???? >????????? >????? Can you confirm that this usage is okay with >>>> Robbin Ehn please. He's >>>> ???? >????????? >????? back from vacation this week. >>>> ???? >????????? > >>>> ???? >????????? >????? >> I would still want to see what impact this >>>> has on thread >>>> ???? >????????? >????? >> startup cost, both with and without the >>>> table being initialized. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > I run a test that initializes the table by >>>> calling ThreadMXBean.get getThreadInfo(), >>>> ???? >????????? >????? > starts some threads as a worm-up, and then >>>> creates and starts 100,000 threads >>>> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >>>> case when the thread table is enabled >>>> ???? >????????? >????? > 100,000 threads are created and started >>>> for about 15200 ms. If the thread table >>>> ???? >????????? >????? > is off the test takes about 14800 ms. Based >>>> on this information the enabled >>>> ???? >????????? >????? > thread table makes the thread startup about >>>> 2.7% slower. >>>> ???? >????????? > >>>> ???? >????????? >????? That doesn't sound very good. I think we may >>>> need to Claes involved to >>>> ???? >????????? >????? help investigate overall performance impact >>>> here. >>>> ???? >????????? > >>>> ???? >????????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >>>> ???? >????????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? > >>>> ???? >????????? >????? No further code comments. >>>> ???? >????????? > >>>> ???? >????????? >????? I didn't look at the test in detail. >>>> ???? >????????? > >>>> ???? >????????? >????? Thanks, >>>> ???? >????????? >????? David >>>> ???? >????????? > >>>> ???? >????????? >????? > Thanks! >>>> ???? >????????? >????? > --Daniil >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >>>> wrote: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Hi Daniil, >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Overall I think this is a reasonable >>>> approach but I would still like to >>>> ???? >????????? >????? >????? see some performance and footprint >>>> numbers, both to verify it fixes the >>>> ???? >????????? >????? >????? problem reported, and that we are not >>>> getting penalized elsewhere. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov >>>> wrote: >>>> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > Please review the new version of the >>>> fix, that makes the thread table initialization on demand and >>>> ???? >????????? >????? >????? > moves it inside >>>> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >>>> the thread table >>>> ???? >????????? >????? >????? >?? is initialized with the threads >>>> from the current thread list. We don't want to hold Threads_lock >>>> ???? >????????? >????? >????? > inside >>>> find_JavaThread_from_java_tid(),? thus new threads still could be >>>> created? while the thread >>>> ???? >????????? >????? >????? > table is being initialized . Such >>>> threads will be found by the linear search and added to the thread >>>> table >>>> ???? >????????? >????? >????? > later, in >>>> ThreadsList::find_JavaThread_from_java_tid(). >>>> ????? >????????? >????? > >>>> ???? >????????? >????? >????? The initialization allows the created >>>> but unpopulated, or partially >>>> ???? >????????? >????? >????? populated, table to be seen by other >>>> threads - is that your intention? >>>> ???? >????????? >????? >????? It seems it should be okay as the >>>> other threads will then race with the >>>> ???? >????????? >????? >????? initializing thread to add specific >>>> entries, and this is a concurrent >>>> ???? >????????? >????? >????? map so that should be functionally >>>> correct. But if so then I think you >>>> ???? >????????? >????? >????? can also reduce the scope of the >>>> ThreadTableCreate_lock so that it >>>> ???? >????????? >????? >????? covers creation of the table only, not >>>> the initial population of the table. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I like the approach of only >>>> initializing the table when needed and using >>>> ???? >????????? >????? >????? that to control when the >>>> add/remove-thread code needs to update the >>>> ???? >????????? >????? >????? table. But I would still want to see >>>> what impact this has on thread >>>> ???? >????????? >????? >????? startup cost, both with and without >>>> the table being initialized. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > The change also includes additional >>>> optimization for some callers of find_JavaThread_from_java_tid() >>>> ???? >????????? >????? >????? > as Daniel suggested. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Not sure it's best to combine these, >>>> but if they are limited to the >>>> ???? >????????? >????? >????? changes in management.cpp only then >>>> that may be okay. It helps to be >>>> ???? >????????? >????? >????? able to focus on the table related >>>> changes without being distracted by >>>> ???? >????????? >????? >????? other optimizations. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > That is correct that >>>> ResolvedMethodTable was used as a blueprint for the thread table, >>>> however, I tried >>>> ???? >????????? >????? >????? > to strip it of the all functionality >>>> that is not required in the thread table case. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? The revised version seems better in >>>> that regard. But I still have a >>>> ???? >????????? >????? >????? concern, see below. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > We need to have the thread table >>>> resizable and allow it to grow as the number of threads increases to >>>> avoid >>>> ???? >????????? >????? >????? > reserving excessive memory a-priori >>>> or deteriorating lookup times. The ServiceThread is responsible for >>>> ???? >????????? >????? >????? > growing the thread table when required. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Yes but why? Why can't this table be >>>> grown on demand by the thread that >>>> ???? >????????? >????? >????? is doing the addition? For other >>>> tables we may have to delegate to the >>>> ???? >????????? >????? >????? service thread because the current >>>> thread cannot perform the action, or >>>> ???? >????????? >????? >????? it doesn't want to perform it at the >>>> time the need for the resize is >>>> ???? >????????? >????? >????? detected (e.g. its detected at a >>>> safepoint and you want the resize to >>>> ???? >????????? >????? >????? happen later outside the safepoint). >>>> It's not apparent to me that such >>>> ???? >????????? >????? >????? restrictions apply here. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > There is no ConcurrentHashTable >>>> available in Java 8 and for backporting this fix to Java 8 another >>>> implementation >>>> ???? >????????? >????? >????? > of the hash table, probably >>>> originally suggested in the patch attached to the JBS issue, should >>>> be used.? It will make >>>> ???? >????????? >????? >????? > the backporting more complicated, >>>> however, adding a new Implementation of the hash table in Java 14 >>>> while it >>>> ???? >????????? >????? >????? > already has ConcurrentHashTable >>>> doesn't seem? reasonable for me. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Ok. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > Webrev: >>>> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Some specific code comments: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >>>> PaddedMutex? , special, >>>> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I think this needs to be a >>>> _safepoint_check_always lock. The table will >>>> ???? >????????? >????? >????? be created by regular JavaThreads and >>>> they should (nearly) always be >>>> ???? >????????? >????? >????? checking for safepoints if they are >>>> going to block acquiring the lock. >>>> ???? >????????? >????? >????? And it isn't at all obvious that the >>>> thread doing the creation can't go >>>> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >>>> thread_at(i); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? you could reuse the new java_thread >>>> local you introduced at line 613 and >>>> ???? >????????? >????? >????? just rename that "new" variable to >>>> "thread" so you don't have to change >>>> ???? >????????? >????? >????? all other uses. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL >>>> && ... >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? You don't need to check != NULL here >>>> as you only get here when >>>> ???? >????????? >????? >????? java_thread is not NULL. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 755???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? >??????? 926???? jlong tid = >>>> SharedRuntime::get_java_tid(thread); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I think it cleaner/better to just use >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? jlong tid = >>>> java_lang_Thread::thread_id(thread->threadObj()); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? as we know thread is not NULL, it is a >>>> JavaThread and it has to have a >>>> ???? >????????? >????? >????? non-null threadObj. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/services/management.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 1323???????? if >>>> (THREAD->is_Java_thread()) { >>>> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >>>> (JavaThread*)THREAD; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? These calls can only be made on a >>>> JavaThread so this be simplified to >>>> ???? >????????? >????? >????? remove the is_Java_thread() call. >>>> Similarly in other places. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? --- >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >>>> CHeapObj { >>>> ???? >????????? >????? >???????? 56?? private: >>>> ???? >????????? >????? >???????? 57???? jlong _tid; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? I believe hotspot style is to not >>>> indent the access modifiers in C++ >>>> ???? >????????? >????? >????? class declarations, so the above would >>>> just be: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >>>> CHeapObj { >>>> ???? >????????? >????? >???????? 56 private: >>>> ???? >????????? >????? >???????? 57?? jlong _tid; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? etc. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >>>> JavaThread* java_thread) : >>>> ???? >????????? >????? >??????? 61 >>>> _tid(tid),_java_thread(java_thread) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 61 should be indented as it >>>> continues line 60. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >>>> AllStatic { >>>> ???? >????????? >????? >???????? ... >>>> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >>>> const& value, bool* is_dead) { >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? The is_dead parameter still bothers me >>>> here. I can't make enough sense >>>> ???? >????????? >????? >????? out of the template code in >>>> ConcurrentHashtable to see why we have to >>>> ???? >????????? >????? >????? have it, but I'm concerned that its >>>> very existence means we perhaps >>>> ???? >????????? >????? >????? should not be trying to extend CHT in >>>> this context. ?? >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>> size_log > DefaultThreadTableSizeLog >>>> ???? >????????? >????? >??????? 116?? ? size_log : >>>> DefaultThreadTableSizeLog; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 116 should be indented, though in >>>> this case I think a better layout >>>> ???? >????????? >????? >????? would be: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>> ???? >????????? >????? >??????? 116?????? size_log > >>>> DefaultThreadTableSizeLog ? size_log : >>>> ???? >????????? >????? > DefaultThreadTableSizeLog; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 131 double >>>> ThreadTable::get_load_factor() { >>>> ???? >????????? >????? >??????? 132?? return >>>> (double)_items_count/_current_size; >>>> ???? >????????? >????? >??????? 133 } >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Not sure that is doing what you >>>> want/expect. It will perform integer >>>> ???? >????????? >????? >????? division and then cast that whole >>>> integer to a double. If you want >>>> ???? >????????? >????? >????? double arithmetic you need: >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? return >>>> ((double)_items_count)/_current_size; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 180???? jlong _tid; >>>> ???? >????????? >????? >????? 181???? uintx _hash; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: no need for all those spaces >>>> before the variable name. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >>>> ???? >????????? >????? >??????? 184???? : _tid(tid), >>>> _hash(primitive_hash(tid)) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 184 should be indented. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after : >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >??????? 211 assert(_is_initialized, "Thread >>>> table is not initialized"); >>>> ???? >????????? >????? >??????? 212?? _has_work = false; >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? line 211 is indented one space too far. >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 229 ThreadTableEntry* entry = new >>>> ThreadTableEntry(tid,java_thread); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after , >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? 252?? return >>>> _local_table->remove(thread,lookup); >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Nit: need space after , >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? Thanks, >>>> ???? >????????? >????? >????? David >>>> ???? >????????? >????? >????? ------ >>>> ???? >????????? >????? > >>>> ???? >????????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? > --Daniil >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >>>> Daugherty" wrote: >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >>>> Titov wrote: >>>> ???? >????????? >????? >????? >????? > Hi Serguei and David, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Serguei is right, >>>> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with >>>> an unmatched java_tid. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Please find a new version of >>>> the fix that includes the changes Serguei suggested. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Regarding the concern about >>>> the maintaining the thread table when it may never even be queried, >>>> one of >>>> ???? >????????? >????? >????? >????? > the options could be to add >>>> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >>>> the calls to the thread table >>>> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >>>> add_thread() and remove_thread() methods to check this flag. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > When >>>> ThreadsList::find_JavaThread_from_java_tid() is called for the first >>>> time it could check if ThreadTable ::isEnabled >>>> ???? >????????? >????? >????? >????? > Is on and if not then set it >>>> on and populate the thread table with all existing threads from the >>>> thread list. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? I have the same concerns as >>>> David H. about this new ThreadTable. >>>> ???? >????????? >????? >????? > >>>> ThreadsList::find_JavaThread_from_java_tid() is only called from code >>>> ???? >????????? >????? >????? >????? in >>>> src/hotspot/share/services/management.cpp so I think that table >>>> ???? >????????? >????? >????? >????? needs to enabled and populated >>>> only if it is going to be used. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? I've taken a look at the webrev >>>> below and I see that David has >>>> ???? >????????? >????? >????? >????? followed up with additional >>>> comments. Before I do a crawl through >>>> ???? >????????? >????? >????? >????? code review for this, I would >>>> like to see the ThreadTable stuff >>>> ???? >????????? >????? >????? >????? made optional and David's other >>>> comments addressed. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? Another possible optimization >>>> is for callers of >>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>> save the calling thread's >>>> ???? >????????? >????? >????? >????? tid value before they loop and >>>> if the current tid == saved_tid >>>> ???? >????????? >????? >????? >????? then use the current >>>> JavaThread* instead of calling >>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>> get the JavaThread*. >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? Dan >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >>>> ???? >????????? >????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? >????? > --Daniil >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > From: >>>> >>>> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >>>> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 >>>> at 7:56 PM >>>> ???? >????????? >????? >????? >????? > To: Daniil Titov >>>> , OpenJDK Serviceability >>>> , >>>> "hotspot-runtime-dev at openjdk.java.net" >>>> , "jmx-dev at openjdk.java.net" >>>> >>>> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >>>> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >>>> maxDepth) >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > I have several quick comments. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The indent in the hotspot >>>> c/c++ files has to be 2, not 4. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >>>> >>>> ???? >????????? >????? >????? >????? > 614 JavaThread* >>>> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>> java_thread = ThreadTable::find_thread(java_tid); >>>> ???? >????????? >????? >????? >????? > 616???? if (java_thread == >>>> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >>>> ???? >????????? >????? >????? >????? > 617???????? // >>>> ThreadsSMRSupport::add_thread() is not called for the primordial >>>> ???? >????????? >????? >????? >????? > 618???????? // thread. Thus, >>>> we find this thread with a linear search and add it >>>> ???? >????????? >????? >????? >????? > 619???????? // to the thread >>>> table. >>>> ???? >????????? >????? >????? >????? > 620???????? for (uint i = 0; >>>> i < length(); i++) { >>>> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >>>> thread = thread_at(i); >>>> ???? >????????? >????? >????? >????? > 622???????????? if >>>> (is_valid_java_thread(java_tid,thread)) { >>>> ???? >????????? >????? >????? >????? > 623 >>>> ThreadTable::add_thread(java_tid, thread); >>>> ???? >????????? >????? >????? >????? > 624???????????????? return >>>> thread; >>>> ???? >????????? >????? >????? >????? > 625???????????? } >>>> ???? >????????? >????? >????? >????? > 626???????? } >>>> ???? >????????? >????? >????? >????? > 627???? } else if >>>> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>> ???? >????????? >????? >????? >????? > 628???????? return java_thread; >>>> ???? >????????? >????? >????? >????? > 629???? } >>>> ???? >????????? >????? >????? >????? > 630???? return NULL; >>>> ???? >????????? >????? >????? >????? >?? 631 } >>>> ???? >????????? >????? >????? >????? >?? 632 bool >>>> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >>>> java_thread) { >>>> ???? >????????? >????? >????? >????? > 633???? oop tobj = >>>> java_thread->threadObj(); >>>> ???? >????????? >????? >????? >????? > 634???? // Ignore the thread >>>> if it hasn't run yet, has exited >>>> ???? >????????? >????? >????? >????? > 635???? // or is starting to >>>> exit. >>>> ???? >????????? >????? >????? >????? > 636???? return (tobj != NULL >>>> && !java_thread->is_exiting() && >>>> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >>>> java_lang_Thread::thread_id(tobj)); >>>> ???? >????????? >????? >????? >????? >?? 638 } >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>> java_thread = ThreadTable::find_thread(java_tid); >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >>>> find_thread() to find_thread_by_tid(). >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > A space is missed after the >>>> comma: >>>> ???? >????????? >????? >????? >????? >??? 622 if >>>> (is_valid_java_thread(java_tid,thread)) { >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > An empty line is needed >>>> before L632. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The name >>>> 'is_valid_java_thread' looks wrong (or confusing) to me. >>>> ???? >????????? >????? >????? >????? > Something like >>>> 'is_alive_java_thread_with_tid()' would be better. >>>> ???? >????????? >????? >????? >????? > It'd better to list >>>> parameters in the opposite order. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > The call to >>>> is_valid_java_thread() is confusing: >>>> ???? >????????? >????? >????? >????? >???? 627 } else if >>>> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Why would the call >>>> ThreadTable::find_thread(java_tid) return a JavaThread with an >>>> unmatched java_tid? >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks, >>>> ???? >????????? >????? >????? >????? > Serguei >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >>>> Holmes" wrote: >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >????? Hi Daniil, >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? >????? The definition and use >>>> of this hashtable (yet another hashtable >>>> ???? >????????? >????? >????? >????? > implementation!) will need >>>> careful examination. We have to be concerned >>>> ???? >????????? >????? >????? >????? > about the cost of maintaining >>>> it when it may never even be queried. You >>>> ???? >????????? >????? >????? >????? > would need to look at >>>> footprint cost and performance impact. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Unfortunately I'm just about >>>> to board a plane and will be out for the >>>> ???? >????????? >????? >????? >????? > next few days. I will try to >>>> look at this asap next week, but we will >>>> ???? >????????? >????? >????? >????? > need a lot more data on it. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks, >>>> ???? >????????? >????? >????? >????? > David >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >>>> Titov wrote: >>>> ???? >????????? >????? >????? >????? > Please review the change that >>>> improves performance of ThreadMXBean MXBean methods returning the >>>> ???? >????????? >????? >????? >????? > information for specific >>>> threads. The change introduces the thread table that uses >>>> ConcurrentHashTable >>>> ???? >????????? >????? >????? >????? > to store one-to-one the >>>> mapping between the thread ids and JavaThread objects and replaces >>>> the linear >>>> ???? >????????? >????? >????? >????? > search over the thread list >>>> in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with >>>> the lookup >>>> ???? >????????? >????? >????? >????? > in the thread table. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 >>>> and tier3 tests successfully passed. >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Webrev: >>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >>>> ???? >????????? >????? >????? >????? > Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Thanks! >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > Best regards, >>>> ???? >????????? >????? >????? >????? > Daniil >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? >????? > >>>> ???? >????????? > >>>> ???? >????????? > >>>> ???? >????????? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> ???? > >>>> >>>> >>> > From serguei.spitsyn at oracle.com Wed Sep 18 07:34:55 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 00:34:55 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> <48c3ef87-a171-18d1-fbe8-ed4dcb622193@oracle.com> Message-ID: Hi David, On 9/18/19 00:25, David Holmes wrote: > Hi Serguei, > > In the interests of full disclosure I was the one who told Daniil to > not hold the lock while populating the table. I well understand your point to not hold the lock while populating the table. > > I'll leave you two to work out which way you want to go there. In fact, I have no strong opinion here, just my gut feelings. :) And I'm open for any solution. Just wanted to share my thoughts with you, guys, to have all reasoning on the table. Let me briefly talk to Daniil tomorrow. Thanks, Serguei > Thanks, > David > > On 18/09/2019 5:13 pm, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> >> On 9/17/19 03:46, David Holmes wrote: >>> Hi Serguei, >>> >>> On 17/09/2019 7:10 pm, serguei.spitsyn at oracle.com wrote: >>>> Hi Daniil, >>>> >>>> >>>> On 9/16/19 21:36, Daniil Titov wrote: >>>>> Hi David, >>>>> >>>>> The case you have described is exact the reason why we still have >>>>> a code inside >>>>> ThreadsList::find_JavaThread_from_java_tid() method that does a >>>>> linear scan and adds >>>>> ? the requested thread to the thread table if it is not there ( >>>>> lines 614-613 below). >>>> >>>> I disagree because it is easy to avoid concurrent ThreadTable >>>> initialization (please, see my separate email). >>>> The reason for this code is to cover a case of late/lazy >>>> ThreadTable initialization. >>> >>> I'm not sure I follow. With the current code if two threads are >>> racing to initialize the ThreadTable with ThreadsLists that contain >>> a different set of threads then there are two possibilities with >>> regards to the interleaving. Assume T1 initializes the table with >>> its set of threads and so finds the tid it is looking for in the >>> table. Meanwhile T2 is racing with the initialization logic: >>> >>> - If T2 sees _is_initialized then lazy_initialization does nothing >>> for T2, and the additional threads in its ThreadsList (say T3 and >>> T4) are not added to the table. But the specific thread associated >>> with the tid (say T3) will be found by linear search of the >>> ThreadsList and then added. If any other threads come searching for >>> T4 they too will not find it in the ThreadTable but instead perform >>> the linear search of their ThreadsList (and add it). >>> >>> - if T2 doesn't see _is_initialized at first it will try to acquire >>> the lock, and eventually see _is_initialized is true, at which point >>> it will try to add all of its thread's to the table (so T3 and T4 >>> will be added). When lazy_initialize returns, T3 will be found in >>> the table and returned. If any other threads come searching for T4 >>> they will also find it in the table. >> >> My main concerns are simplicity and reliability. >> I do no care much about extra overhead at the ThreadTable >> initialization. >> A probability of the ThreadTable::lazy_initialize() being called >> concurrently is low. >> Also, it might happen only once for the whole VM process execution. >> >> I was wrong by thinking that adding new threads to the ThreadTable >> after its initialization >> will result in thread linear search as well. So my conclusion was >> that we should not care >> if it happens once more at lazy initialization point. >> But now I see that after ThreadTable::is_initialized() returns true >> the ThreadsSMRSupport::add_thread() >> makes a call to the ThreadTable::add_thread(). So, no more linear >> search will happen. >> >> >> However, it seems to me, a possible concurrent lazy initialization in >> the webrev.06 introduces >> its own extra overhead - competing threads (suppose, we have just two >> of them) will do >> the same amount of work concurrently: >> ??? - all indirect memory readings, lookup/comparisons and other >> checks will be performed twice >> ??? - new ThreadTableEntry can be allocated twice for some threads in >> the list >> ????? (depending on how ConcurrentHashTable is implemented, there can >> be a potential a memory leak) >> >> So, I doubt we win much in performance here but can loose in >> reliability. >> >> >> I'd suggest to simplify the lazy initialization code and make it more >> reliable this way: >> >> ????? if (!_is_initialized) { >> ??????? MutexLocker ml(ThreadTableCreate_lock); >> ??????? if (!_is_initialized) { >> ????????? create_table(threads->length()); >> ????????? _is_initialized = true; >> ??????? } >> ??????? for (uint i = 0; i < threads->length(); i++) { >> ????????? JavaThread* thread = threads->thread_at(i); >> ????????? oop tobj = thread->threadObj(); >> ????????? if (tobj != NULL && !thread->is_exiting()) { >> ??????????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ??????????? add_thread(java_tid, thread); >> ????????? } >> ??????? } >> ????? } >> >>> With your suggested code change this second case is not possible so >>> for any racing initialization the lookup of any threads not in the >>> original ThreadsList will always result in using the linear search >>> before adding to the table. >> >> Yes, but I did not care about this. >> The overhead is expected to be lower than the lazy initialization cost, >> especially because the probability of concurrent initialization is low. >> >>> Both seem correct to me. Which one is more efficient will totally >>> depend on the number of differences between the ThreadsLists and >>> whether the code ever tries to look up those additional threads. If >>> we assume racing initialization is likely to be rare anyway (because >>> generally one thread is in charge of doing the monitoring) then the >>> choice seems somewhat arbitrary. >> >> I agree, but have a little preference in favor of simplicity. >> It was a good discussion though. :) >> >> Thanks, >> Serguei >> >>> Cheers, >>> David >>> ----- >>> >>>> Thanks, >>>> Serguei >>>> >>>>> ??? The >>>>> assumption is that it's quite uncommon and even if this is the >>>>> case the linear scan happens >>>>> only once per such thread. >>>>> >>>>> ? 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong >>>>> java_tid) const { >>>>> ? 612?? ThreadTable::lazy_initialize(this); >>>>> ? 613?? JavaThread* thread = >>>>> ThreadTable::find_thread_by_tid(java_tid); >>>>> ? 614?? if (thread == NULL) { >>>>> ? 615???? // If the thread is not found in the table find it >>>>> ? 616???? // with a linear search and add to the table. >>>>> ? 617???? for (uint i = 0; i < length(); i++) { >>>>> ? 618?????? thread = thread_at(i); >>>>> ? 619?????? oop tobj = thread->threadObj(); >>>>> ? 620?????? // Ignore the thread if it hasn't run yet, has exited >>>>> ? 621?????? // or is starting to exit. >>>>> ? 622?????? if (tobj != NULL && java_tid == >>>>> java_lang_Thread::thread_id(tobj)) { >>>>> ? 623???????? MutexLocker ml(Threads_lock); >>>>> ? 624???????? // Must be inside the lock to ensure that we don't >>>>> add the thread to the table >>>>> ? 625???????? // that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> ? 626???????? if (!thread->is_exiting()) { >>>>> ? 627?????????? ThreadTable::add_thread(java_tid, thread); >>>>> ? 628?????????? return thread; >>>>> ? 629???????? } >>>>> ? 630?????? } >>>>> ? 631???? } >>>>> ? 632?? } else if (!thread->is_exiting()) { >>>>> ? 633?????? return thread; >>>>> ? 634?? } >>>>> ? 635?? return NULL; >>>>> ? 636 } >>>>> >>>>> Thanks, >>>>> Daniil >>>>> >>>>> ?On 9/16/19, 7:27 PM, "David Holmes" wrote: >>>>> >>>>> ???? Hi Daniil, >>>>> ???? Thanks again for your perseverance on this one. >>>>> ???? I think there is a problem with initialization of the thread >>>>> table. >>>>> ???? Suppose thread T1 has called >>>>> ThreadsList::find_JavaThread_from_java_tid >>>>> ???? and has commenced execution of ThreadTable::lazy_initialize, >>>>> but not yet >>>>> ???? marked _is_initialized as true. Now two new threads (T2 and >>>>> T3) are >>>>> ???? created and start running - they aren't added to the >>>>> ThreadTable yet >>>>> ???? because it isn't initialized. Now T0 also calls >>>>> ???? ThreadsList::find_JavaThread_from_java_tid using an updated >>>>> ThreadsList >>>>> ???? that contains T2 and T3. It also calls >>>>> ThreadTable::lazy_initialize. If >>>>> ???? _is_initialized is still false T0 will attempt initialization >>>>> but once >>>>> ???? it gets the lock it will see the table has now been >>>>> initialized by T1. >>>>> ???? It will then proceed to update the table with its own >>>>> ThreadList content >>>>> ???? - adding T2 and T3. That is all fine. But now suppose T0 >>>>> initially sees >>>>> ???? _is_initialized as true, it will do nothing in >>>>> lazy_initialize and >>>>> ???? simply return to find_JavaThread_from_java_tid. But now T2 >>>>> and T3 are >>>>> ???? missing from the ThreadTable and nothing will cause them to >>>>> be added. >>>>> ???? More generally any ThreadsList that is created after the >>>>> ThreadsList >>>>> ???? that will be used for initialization, may contain threads >>>>> that will not >>>>> ???? be added to the table. >>>>> ???? Thanks, >>>>> ???? David >>>>> ???? On 17/09/2019 4:18 am, Daniil Titov wrote: >>>>> ???? > Hello, >>>>> ???? > >>>>> ???? > After investigating with Claes the impact of this change on >>>>> the performance (thanks a lot Claes for helping with it!) the >>>>> conclusion was that the impact on the thread startup time is not a >>>>> blocker for this change. >>>>> ???? > >>>>> ???? > I also measured the memory footprint using Native Memory >>>>> Tracking and results showed around 40 bytes per live thread. >>>>> ???? > >>>>> ???? > Please review a new version of the fix, webrev.06 [1].? >>>>> Just to remind,? webrev.05 was abandoned and webrev.06 [1] is >>>>> webrev.04 [3] minus changes in >>>>> src/hotspot/share/services/management.cpp (that were factored out >>>>> to a separate issue [4]) and plus a change in >>>>> ThreadsList::find_JavaThread_from_java_tid() method (please, see >>>>> below)? that addresses the problem Robbin found and puts the code >>>>> that adds a new thread to the thread table inside Threads_lock. >>>>> ???? > >>>>> ???? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? > >>>>> ???? > 622?????? if (tobj != NULL && java_tid == >>>>> java_lang_Thread::thread_id(tobj)) { >>>>> ???? > 623???????? MutexLocker ml(Threads_lock); >>>>> ???? > 624???????? // Must be inside the lock to ensure that we >>>>> don't add the thread to the table >>>>> ???? > 625???????? // that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> ???? > 626???????? if (!thread->is_exiting()) { >>>>> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >>>>> ???? > 628?????????? return thread; >>>>> ???? > 629???????? } >>>>> ???? > 630?????? } >>>>> ???? > >>>>> ???? > [1] Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >>>>> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >>>>> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >>>>> ???? > >>>>> ???? > ?Thank you, >>>>> ???? > Daniil >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? >????????? > >>>>> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >>>>> wrote: >>>>> ???? >????????? > >>>>> ???? >????????? >????? Hi Daniil, >>>>> ???? >????????? > >>>>> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >>>>> ???? >????????? >????? > Hi David, >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > Thank you for your detailed review. >>>>> Please review a new version of the fix that includes >>>>> ???? >????????? >????? > the changes you suggested: >>>>> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced >>>>> to cover the creation of the table only; >>>>> ???? >????????? >????? > - ThreadTableCreate_lock is made >>>>> _safepoint_check_always; >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > - ServiceThread is no longer responsible >>>>> for the resizing of the thread table, instead, >>>>> ???? >????????? >????? >??? the thread table is changed to grow on >>>>> demand by the thread that is doing the addition; >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay - I'm happy to get the serviceThread >>>>> out of the picture here. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > - fixed nits and formatting issues. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Okay. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >>> The change also includes additional >>>>> optimization for some callers of find_JavaThread_from_java_tid() >>>>> ???? >????????? >????? >>>?? as Daniel suggested. >>>>> ???? >????????? >????? >> Not sure it's best to combine these, but >>>>> if they are limited to the >>>>> ???? >????????? >????? >> changes in management.cpp only then that >>>>> may be okay. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > The additional optimization for some >>>>> callers of find_JavaThread_from_java_tid() is >>>>> ???? >????????? >????? > limited to management.cpp (plus a new >>>>> test) so I left them in the webrev? but >>>>> ???? >????????? >????? > I also could move it in the separate >>>>> issue if required. >>>>> ???? >????????? > >>>>> ???? >????????? >????? I'd prefer this part of be separated out, >>>>> but won't insist. Let's see if >>>>> ???? >????????? >????? Dan or Serguei have a strong opinion. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? >????????? >????? >??? >755???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >??? > 926???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >?? >? I think it cleaner/better to just use >>>>> ???? >????????? >????? >?? > jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? >????? >?? > as we know thread is not NULL, it is >>>>> a JavaThread and it has to have a >>>>> ???? >????????? >????? >?? > non-null threadObj. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > I had to leave this code unchanged since >>>>> it turned out the threadObj is null >>>>> ???? >????????? >????? > when VM is destroyed: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >>>>> oopDesc::long_field(int) const+0x67 >>>>> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >>>>> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >>>>> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >>>>> Threads::add(JavaThread*, bool)+0x82 >>>>> ???? >????????? >????? > V? [libjvm.so+0xef8369] >>>>> attach_current_thread.part.197+0xc9 >>>>> ???? >????????? >????? > V? [libjvm.so+0xec136c] >>>>> jni_DestroyJavaVM+0x6c >>>>> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >>>>> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >>>>> ???? >????????? > >>>>> ???? >????????? >????? This is actually nothing to do with the VM >>>>> being destroyed, but is an >>>>> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >>>>> interaction with the >>>>> ???? >????????? >????? ThreadSMR iterators. The attach process is: >>>>> ???? >????????? >????? - create JavaThread >>>>> ???? >????????? >????? - mark as "is attaching via jni" >>>>> ???? >????????? >????? - add to ThreadsList >>>>> ???? >????????? >????? - create java.lang.Thread object (you can >>>>> only execute Java code after >>>>> ???? >????????? >????? you are attached) >>>>> ???? >????????? >????? - mark as "attach completed" >>>>> ???? >????????? > >>>>> ???? >????????? >????? So while a thread "is attaching" it will be >>>>> seen by the ThreadSMR thread >>>>> ???? >????????? >????? iterator but will have a NULL >>>>> java.lang.Thread object. >>>>> ???? >????????? > >>>>> ???? >????????? >????? We special-case attaching threads in a >>>>> number of places in the VM and I >>>>> ???? >????????? >????? think we should be explicitly doing >>>>> something here to filter out >>>>> ???? >????????? >????? attaching threads, rather than just being >>>>> tolerant of a NULL j.l.Thread >>>>> ???? >????????? >????? object. Specifically in >>>>> ThreadsSMRSupport::add_thread: >>>>> ???? >????????? > >>>>> ???? >????????? >????? if (ThreadTable::is_initialized() && >>>>> !thread->is_attaching_via_jni()) { >>>>> ???? >????????? >???????? jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? > ThreadTable::add_thread(tid, thread); >>>>> ???? >????????? >????? } >>>>> ???? >????????? > >>>>> ???? >????????? >????? Note that in >>>>> ThreadsSMRSupport::remove_thread we can use the same guard, >>>>> ???? >????????? >????? which covers the case the JNI attach >>>>> encountered an error trying to >>>>> ???? >????????? >????? create the j.l.Thread object. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >>>>> ???? >????????? >????? >> 71???? static uintx get_hash(Value >>>>> const& value, bool* is_dead) { >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >> The is_dead parameter still bothers me >>>>> here. I can't make enough sense >>>>> ???? >????????? >????? >> out of the template code in >>>>> ConcurrentHashtable to see why we have to >>>>> ???? >????????? >????? >> have it, but I'm concerned that its very >>>>> existence means we perhaps >>>>> ???? >????????? >????? >> should not be trying to extend CHT in >>>>> this context. ?? >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > My understanding is that is_dead >>>>> parameter provides a mechanism for >>>>> ???? >????????? >????? > ConcurrentHashtable to remove stale >>>>> entries that were not explicitly >>>>> ???? >????????? >????? > removed by calling >>>>> ConcurrentHashTable::remove() method. >>>>> ???? >????????? >????? > I think that just because in our case we >>>>> don't use this mechanism doesn't >>>>> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Can you confirm that this usage is okay >>>>> with Robbin Ehn please. He's >>>>> ???? >????????? >????? back from vacation this week. >>>>> ???? >????????? > >>>>> ???? >????????? >????? >> I would still want to see what impact >>>>> this has on thread >>>>> ???? >????????? >????? >> startup cost, both with and without the >>>>> table being initialized. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > I run a test that initializes the table >>>>> by calling ThreadMXBean.get getThreadInfo(), >>>>> ???? >????????? >????? > starts some threads as a worm-up, and >>>>> then creates and starts 100,000 threads >>>>> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >>>>> case when the thread table is enabled >>>>> ???? >????????? >????? > 100,000 threads are created and started? >>>>> for about 15200 ms. If the thread table >>>>> ???? >????????? >????? > is off the test takes about 14800 ms. >>>>> Based on this information the enabled >>>>> ???? >????????? >????? > thread table makes the thread startup >>>>> about 2.7% slower. >>>>> ???? >????????? > >>>>> ???? >????????? >????? That doesn't sound very good. I think we >>>>> may need to Claes involved to >>>>> ???? >????????? >????? help investigate overall performance impact >>>>> here. >>>>> ???? >????????? > >>>>> ???? >????????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >>>>> ???? >????????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? > >>>>> ???? >????????? >????? No further code comments. >>>>> ???? >????????? > >>>>> ???? >????????? >????? I didn't look at the test in detail. >>>>> ???? >????????? > >>>>> ???? >????????? >????? Thanks, >>>>> ???? >????????? >????? David >>>>> ???? >????????? > >>>>> ???? >????????? >????? > Thanks! >>>>> ???? >????????? >????? > --Daniil >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >>>>> wrote: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Hi Daniil, >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Overall I think this is a reasonable >>>>> approach but I would still like to >>>>> ???? >????????? >????? >????? see some performance and footprint >>>>> numbers, both to verify it fixes the >>>>> ???? >????????? >????? >????? problem reported, and that we are >>>>> not getting penalized elsewhere. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov >>>>> wrote: >>>>> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > Please review the new version of >>>>> the fix, that makes the thread table initialization on demand and >>>>> ???? >????????? >????? >????? > moves it inside >>>>> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >>>>> the thread table >>>>> ???? >????????? >????? >????? >?? is initialized with the threads >>>>> from the current thread list. We don't want to hold Threads_lock >>>>> ???? >????????? >????? >????? > inside >>>>> find_JavaThread_from_java_tid(),? thus new threads still could be >>>>> created? while the thread >>>>> ???? >????????? >????? >????? > table is being initialized . Such >>>>> threads will be found by the linear search and added to the thread >>>>> table >>>>> ???? >????????? >????? >????? > later, in >>>>> ThreadsList::find_JavaThread_from_java_tid(). >>>>> ????? >????????? >????? > >>>>> ???? >????????? >????? >????? The initialization allows the >>>>> created but unpopulated, or partially >>>>> ???? >????????? >????? >????? populated, table to be seen by other >>>>> threads - is that your intention? >>>>> ???? >????????? >????? >????? It seems it should be okay as the >>>>> other threads will then race with the >>>>> ???? >????????? >????? >????? initializing thread to add specific >>>>> entries, and this is a concurrent >>>>> ???? >????????? >????? >????? map so that should be functionally >>>>> correct. But if so then I think you >>>>> ???? >????????? >????? >????? can also reduce the scope of the >>>>> ThreadTableCreate_lock so that it >>>>> ???? >????????? >????? >????? covers creation of the table only, >>>>> not the initial population of the table. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I like the approach of only >>>>> initializing the table when needed and using >>>>> ???? >????????? >????? >????? that to control when the >>>>> add/remove-thread code needs to update the >>>>> ???? >????????? >????? >????? table. But I would still want to see >>>>> what impact this has on thread >>>>> ???? >????????? >????? >????? startup cost, both with and without >>>>> the table being initialized. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > The change also includes >>>>> additional optimization for some callers of >>>>> find_JavaThread_from_java_tid() >>>>> ???? >????????? >????? >????? > as Daniel suggested. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Not sure it's best to combine these, >>>>> but if they are limited to the >>>>> ???? >????????? >????? >????? changes in management.cpp only then >>>>> that may be okay. It helps to be >>>>> ???? >????????? >????? >????? able to focus on the table related >>>>> changes without being distracted by >>>>> ???? >????????? >????? >????? other optimizations. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > That is correct that >>>>> ResolvedMethodTable was used as a blueprint for the thread table, >>>>> however, I tried >>>>> ???? >????????? >????? >????? > to strip it of the all >>>>> functionality that is not required in the thread table case. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? The revised version seems better in >>>>> that regard. But I still have a >>>>> ???? >????????? >????? >????? concern, see below. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > We need to have the thread table >>>>> resizable and allow it to grow as the number of threads increases >>>>> to avoid >>>>> ???? >????????? >????? >????? > reserving excessive memory >>>>> a-priori or deteriorating lookup times. The ServiceThread is >>>>> responsible for >>>>> ???? >????????? >????? >????? > growing the thread table when >>>>> required. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Yes but why? Why can't this table be >>>>> grown on demand by the thread that >>>>> ???? >????????? >????? >????? is doing the addition? For other >>>>> tables we may have to delegate to the >>>>> ???? >????????? >????? >????? service thread because the current >>>>> thread cannot perform the action, or >>>>> ???? >????????? >????? >????? it doesn't want to perform it at the >>>>> time the need for the resize is >>>>> ???? >????????? >????? >????? detected (e.g. its detected at a >>>>> safepoint and you want the resize to >>>>> ???? >????????? >????? >????? happen later outside the safepoint). >>>>> It's not apparent to me that such >>>>> ???? >????????? >????? >????? restrictions apply here. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > There is no ConcurrentHashTable >>>>> available in Java 8 and for backporting this fix to Java 8 another >>>>> implementation >>>>> ???? >????????? >????? >????? > of the hash table, probably >>>>> originally suggested in the patch attached to the JBS issue, >>>>> should be used.? It will make >>>>> ???? >????????? >????? >????? > the backporting more complicated,? >>>>> however, adding a new Implementation of the hash table in Java 14 >>>>> while it >>>>> ???? >????????? >????? >????? > already has ConcurrentHashTable >>>>> doesn't seem? reasonable for me. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Ok. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > Webrev: >>>>> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Some specific code comments: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >>>>> PaddedMutex? , special, >>>>> ???? >????????? >????? >????? false, >>>>> Monitor::_safepoint_check_never); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I think this needs to be a >>>>> _safepoint_check_always lock. The table will >>>>> ???? >????????? >????? >????? be created by regular JavaThreads >>>>> and they should (nearly) always be >>>>> ???? >????????? >????? >????? checking for safepoints if they are >>>>> going to block acquiring the lock. >>>>> ???? >????????? >????? >????? And it isn't at all obvious that the >>>>> thread doing the creation can't go >>>>> ???? >????????? >????? >????? to a safepoint whilst this lock is >>>>> held. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 618 JavaThread* thread = >>>>> thread_at(i); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? you could reuse the new java_thread >>>>> local you introduced at line 613 and >>>>> ???? >????????? >????? >????? just rename that "new" variable to >>>>> "thread" so you don't have to change >>>>> ???? >????????? >????? >????? all other uses. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL >>>>> && ... >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? You don't need to check != NULL here >>>>> as you only get here when >>>>> ???? >????????? >????? >????? java_thread is not NULL. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 755???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? >??????? 926???? jlong tid = >>>>> SharedRuntime::get_java_tid(thread); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I think it cleaner/better to just use >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? jlong tid = >>>>> java_lang_Thread::thread_id(thread->threadObj()); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? as we know thread is not NULL, it is >>>>> a JavaThread and it has to have a >>>>> ???? >????????? >????? >????? non-null threadObj. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/services/management.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 1323???????? if >>>>> (THREAD->is_Java_thread()) { >>>>> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >>>>> (JavaThread*)THREAD; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? These calls can only be made on a >>>>> JavaThread so this be simplified to >>>>> ???? >????????? >????? >????? remove the is_Java_thread() call. >>>>> Similarly in other places. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? --- >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : >>>>> public CHeapObj { >>>>> ???? >????????? >????? >???????? 56?? private: >>>>> ???? >????????? >????? >???????? 57???? jlong _tid; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? I believe hotspot style is to not >>>>> indent the access modifiers in C++ >>>>> ???? >????????? >????? >????? class declarations, so the above >>>>> would just be: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 55 class ThreadTableEntry : >>>>> public CHeapObj { >>>>> ???? >????????? >????? >???????? 56 private: >>>>> ???? >????????? >????? >???????? 57?? jlong _tid; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? etc. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >>>>> JavaThread* java_thread) : >>>>> ???? >????????? >????? >??????? 61 >>>>> _tid(tid),_java_thread(java_thread) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 61 should be indented as it >>>>> continues line 60. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >???????? 67 class ThreadTableConfig : >>>>> public AllStatic { >>>>> ???? >????????? >????? >???????? ... >>>>> ???? >????????? >????? >???????? 71???? static uintx >>>>> get_hash(Value const& value, bool* is_dead) { >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? The is_dead parameter still bothers >>>>> me here. I can't make enough sense >>>>> ???? >????????? >????? >????? out of the template code in >>>>> ConcurrentHashtable to see why we have to >>>>> ???? >????????? >????? >????? have it, but I'm concerned that its >>>>> very existence means we perhaps >>>>> ???? >????????? >????? >????? should not be trying to extend CHT >>>>> in this context. ?? >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>>> size_log > DefaultThreadTableSizeLog >>>>> ???? >????????? >????? >??????? 116?? ? size_log : >>>>> DefaultThreadTableSizeLog; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 116 should be indented, though >>>>> in this case I think a better layout >>>>> ???? >????????? >????? >????? would be: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 115?? size_t start_size_log = >>>>> ???? >????????? >????? >??????? 116 size_log > >>>>> DefaultThreadTableSizeLog ? size_log : >>>>> ???? >????????? >????? > DefaultThreadTableSizeLog; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 131 double >>>>> ThreadTable::get_load_factor() { >>>>> ???? >????????? >????? >??????? 132?? return >>>>> (double)_items_count/_current_size; >>>>> ???? >????????? >????? >??????? 133 } >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Not sure that is doing what you >>>>> want/expect. It will perform integer >>>>> ???? >????????? >????? >????? division and then cast that whole >>>>> integer to a double. If you want >>>>> ???? >????????? >????? >????? double arithmetic you need: >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? return >>>>> ((double)_items_count)/_current_size; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 180???? jlong _tid; >>>>> ???? >????????? >????? >????? 181???? uintx _hash; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: no need for all those spaces >>>>> before the variable name. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >>>>> ???? >????????? >????? >??????? 184???? : _tid(tid), >>>>> _hash(primitive_hash(tid)) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 184 should be indented. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after : >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >??????? 211 assert(_is_initialized, >>>>> "Thread table is not initialized"); >>>>> ???? >????????? >????? >??????? 212?? _has_work = false; >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? line 211 is indented one space too far. >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 229 ThreadTableEntry* entry = new >>>>> ThreadTableEntry(tid,java_thread); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after , >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? 252?? return >>>>> _local_table->remove(thread,lookup); >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Nit: need space after , >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? Thanks, >>>>> ???? >????????? >????? >????? David >>>>> ???? >????????? >????? >????? ------ >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? > --Daniil >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >>>>> Daugherty" wrote: >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >>>>> Titov wrote: >>>>> ???? >????????? >????? >????? >????? > Hi Serguei and David, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Serguei is right, >>>>> ThreadTable::find_thread(java_tid) cannot? return a JavaThread >>>>> with an unmatched java_tid. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Please find a new version >>>>> of the fix that includes the changes Serguei suggested. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Regarding the concern about >>>>> the maintaining the thread table when it may never even be >>>>> queried, one of >>>>> ???? >????????? >????? >????? >????? > the options could be to add >>>>> ThreadTable ::isEnabled flag, set it to "false" by default, and >>>>> wrap the calls to the thread table >>>>> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >>>>> add_thread() and remove_thread() methods to check this flag. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > When >>>>> ThreadsList::find_JavaThread_from_java_tid() is called for the >>>>> first time it could check if ThreadTable ::isEnabled >>>>> ???? >????????? >????? >????? >????? > Is on and if not then set >>>>> it on and populate the thread table with all existing threads from >>>>> the thread list. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? I have the same concerns as >>>>> David H. about this new ThreadTable. >>>>> ???? >????????? >????? >????? > >>>>> ThreadsList::find_JavaThread_from_java_tid() is only called from code >>>>> ???? >????????? >????? >????? >????? in >>>>> src/hotspot/share/services/management.cpp so I think that table >>>>> ???? >????????? >????? >????? >????? needs to enabled and >>>>> populated only if it is going to be used. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? I've taken a look at the >>>>> webrev below and I see that David has >>>>> ???? >????????? >????? >????? >????? followed up with additional >>>>> comments. Before I do a crawl through >>>>> ???? >????????? >????? >????? >????? code review for this, I would >>>>> like to see the ThreadTable stuff >>>>> ???? >????????? >????? >????? >????? made optional and David's >>>>> other comments addressed. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? Another possible optimization >>>>> is for callers of >>>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>>> save the calling thread's >>>>> ???? >????????? >????? >????? >????? tid value before they loop >>>>> and if the current tid == saved_tid >>>>> ???? >????????? >????? >????? >????? then use the current >>>>> JavaThread* instead of calling >>>>> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >>>>> get the JavaThread*. >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? Dan >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >>>>> ???? >????????? >????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? >????? > --Daniil >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > From: >>>>> >>>>> ???? >????????? >????? >????? >????? > Organization: Oracle >>>>> Corporation >>>>> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 >>>>> at 7:56 PM >>>>> ???? >????????? >????? >????? >????? > To: Daniil Titov >>>>> , OpenJDK Serviceability >>>>> , >>>>> "hotspot-runtime-dev at openjdk.java.net" >>>>> , "jmx-dev at openjdk.java.net" >>>>> >>>>> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >>>>> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >>>>> maxDepth) >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > I have several quick comments. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The indent in the hotspot >>>>> c/c++ files has to be 2, not 4. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >>>>> >>>>> ???? >????????? >????? >????? >????? > 614 JavaThread* >>>>> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >>>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>>> java_thread = ThreadTable::find_thread(java_tid); >>>>> ???? >????????? >????? >????? >????? > 616???? if (java_thread == >>>>> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >>>>> ???? >????????? >????? >????? >????? > 617???????? // >>>>> ThreadsSMRSupport::add_thread() is not called for the primordial >>>>> ???? >????????? >????? >????? >????? > 618???????? // thread. >>>>> Thus, we find this thread with a linear search and add it >>>>> ???? >????????? >????? >????? >????? > 619???????? // to the >>>>> thread table. >>>>> ???? >????????? >????? >????? >????? > 620???????? for (uint i = >>>>> 0; i < length(); i++) { >>>>> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >>>>> thread = thread_at(i); >>>>> ???? >????????? >????? >????? >????? > 622???????????? if >>>>> (is_valid_java_thread(java_tid,thread)) { >>>>> ???? >????????? >????? >????? >????? > 623???????????????? >>>>> ThreadTable::add_thread(java_tid, thread); >>>>> ???? >????????? >????? >????? >????? > 624???????????????? return >>>>> thread; >>>>> ???? >????????? >????? >????? >????? > 625???????????? } >>>>> ???? >????????? >????? >????? >????? > 626???????? } >>>>> ???? >????????? >????? >????? >????? > 627???? } else if >>>>> (java_thread != NULL && is_valid_java_thread(java_tid, >>>>> java_thread)) { >>>>> ???? >????????? >????? >????? >????? > 628???????? return >>>>> java_thread; >>>>> ???? >????????? >????? >????? >????? > 629???? } >>>>> ???? >????????? >????? >????? >????? > 630???? return NULL; >>>>> ???? >????????? >????? >????? >????? > 631 } >>>>> ???? >????????? >????? >????? >????? > 632 bool >>>>> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >>>>> java_thread) { >>>>> ???? >????????? >????? >????? >????? > 633???? oop tobj = >>>>> java_thread->threadObj(); >>>>> ???? >????????? >????? >????? >????? > 634???? // Ignore the >>>>> thread if it hasn't run yet, has exited >>>>> ???? >????????? >????? >????? >????? > 635???? // or is starting >>>>> to exit. >>>>> ???? >????????? >????? >????? >????? > 636???? return (tobj != >>>>> NULL && !java_thread->is_exiting() && >>>>> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >>>>> java_lang_Thread::thread_id(tobj)); >>>>> ???? >????????? >????? >????? >????? > 638 } >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > 615???? JavaThread* >>>>> java_thread = ThreadTable::find_thread(java_tid); >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > I'd suggest to rename >>>>> find_thread() to find_thread_by_tid(). >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > A space is missed after the >>>>> comma: >>>>> ???? >????????? >????? >????? >????? > 622 if >>>>> (is_valid_java_thread(java_tid,thread)) { >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > An empty line is needed >>>>> before L632. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The name >>>>> 'is_valid_java_thread' looks wrong (or confusing) to me. >>>>> ???? >????????? >????? >????? >????? > Something like >>>>> 'is_alive_java_thread_with_tid()' would be better. >>>>> ???? >????????? >????? >????? >????? > It'd better to list >>>>> parameters in the opposite order. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The call to >>>>> is_valid_java_thread() is confusing: >>>>> ???? >????????? >????? >????? >????? > 627 } else if (java_thread >>>>> != NULL && is_valid_java_thread(java_tid, java_thread)) { >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Why would the call >>>>> ThreadTable::find_thread(java_tid) return a JavaThread with an >>>>> unmatched java_tid? >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks, >>>>> ???? >????????? >????? >????? >????? > Serguei >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >>>>> Holmes" wrote: >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Hi Daniil, >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > The definition and use of >>>>> this hashtable (yet another hashtable >>>>> ???? >????????? >????? >????? >????? > implementation!) will need >>>>> careful examination. We have to be concerned >>>>> ???? >????????? >????? >????? >????? > about the cost of >>>>> maintaining it when it may never even be queried. You >>>>> ???? >????????? >????? >????? >????? > would need to look at >>>>> footprint cost and performance impact. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Unfortunately I'm just >>>>> about to board a plane and will be out for the >>>>> ???? >????????? >????? >????? >????? > next few days. I will try >>>>> to look at this asap next week, but we will >>>>> ???? >????????? >????? >????? >????? > need a lot more data on it. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks, >>>>> ???? >????????? >????? >????? >????? > David >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >>>>> Titov wrote: >>>>> ???? >????????? >????? >????? >????? > Please review the change >>>>> that improves performance of ThreadMXBean MXBean methods returning >>>>> the >>>>> ???? >????????? >????? >????? >????? > information for specific >>>>> threads. The change introduces the thread table that uses >>>>> ConcurrentHashTable >>>>> ???? >????????? >????? >????? >????? > to store one-to-one the >>>>> mapping between the thread ids and JavaThread objects and replaces >>>>> the linear >>>>> ???? >????????? >????? >????? >????? > search over the thread list >>>>> in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method >>>>> with the lookup >>>>> ???? >????????? >????? >????? >????? > in the thread table. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 >>>>> and tier3 tests successfully passed. >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Webrev: >>>>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >>>>> ???? >????????? >????? >????? >????? > Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8185005 >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Thanks! >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > Best regards, >>>>> ???? >????????? >????? >????? >????? > Daniil >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? >????? > >>>>> ???? >????????? > >>>>> ???? >????????? > >>>>> ???? >????????? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> ???? > >>>>> >>>>> >>>> >> From serguei.spitsyn at oracle.com Wed Sep 18 08:01:25 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 01:01:25 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: "serguei.spitsyn at oracle.com" > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ? ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > ?- the lines 632-633 are useless as they do not really protect from returning an exiting thread > ?- the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Wed Sep 18 07:13:15 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 18 Sep 2019 00:13:15 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <9A125A5A-3904-4E3B-9650-308B56E15F20@oracle.com> <0b307a92-5b5d-bd36-a128-99af6d0f3b1b@oracle.com> Message-ID: <88ec033f-a216-3e0a-8e27-b82fa4728055@oracle.com> An HTML attachment was scrubbed... URL: From daniil.x.titov at oracle.com Fri Sep 20 00:30:02 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Thu, 19 Sep 2019 17:30:02 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: Hi David and Serguei, Please review new version of the fix that includes the changes Serguei suggested: 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread that has just passed the removal point in ThreadsSMRSupport::remove_thread() The changes are in ThreadTable::lazy_initialize() method only. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! --Daniil ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: "serguei.spitsyn at oracle.com" > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Fri Sep 20 01:15:25 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 11:15:25 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> Hi Daniil, On 20/09/2019 10:30 am, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list Ok. > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() That seems good too. I agree with previous comments about the general race with is_exiting in terms of how this API behaves. But there's no change in that behaviour with your changes AFAICS. The main concern, now addressed, is that you mustn't be able to add a thread that never gets removed. Thanks, David ----- > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: "serguei.spitsyn at oracle.com" > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniil.x.titov at oracle.com Fri Sep 20 02:54:31 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Thu, 19 Sep 2019 19:54:31 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> Message-ID: <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> Hi David, Thank you for reviewing this version of the fix. > I agree with previous comments about the general race with is_exiting in > terms of how this API behaves. But there's no change in that behaviour > with your changes AFAICS. Could you please say am I right that you are referring here to the discussion about the fact that find_JavaThread_from_java_tid() still could return the thread that is exiting? If so then, yes, this behavior was not changed. We discussed it with Serguei and the conclusion was that since the current implementation behaves exact the same way then even if decide somehow address this then it makes sense to do it in a separate issue. It also is not clear at this moment what the possible solution could be. The obvious one, just to hold Thread_locks in the callers of find_JavaThread_from_java_tid() (management.cpp) is too expensive and doesn't look as acceptable. Thanks! Best regards, Daniil ?On 9/19/19, 6:15 PM, "David Holmes" wrote: Hi Daniil, On 20/09/2019 10:30 am, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list Ok. > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() That seems good too. I agree with previous comments about the general race with is_exiting in terms of how this API behaves. But there's no change in that behaviour with your changes AFAICS. The main concern, now addressed, is that you mustn't be able to add a thread that never gets removed. Thanks, David ----- > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: "serguei.spitsyn at oracle.com" > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From david.holmes at oracle.com Fri Sep 20 03:24:03 2019 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Sep 2019 13:24:03 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <076f78a4-007b-54a9-bfef-a336222fc058@oracle.com> <23D9AFB5-A978-4368-8C55-D360A296E2FE@oracle.com> Message-ID: <0810ce57-ed4f-0156-c5ee-be4de87c11ab@oracle.com> On 20/09/2019 12:54 pm, Daniil Titov wrote: > Hi David, > > Thank you for reviewing this version of the fix. > >> I agree with previous comments about the general race with is_exiting in >> terms of how this API behaves. But there's no change in that behaviour >> with your changes AFAICS. > > Could you please say am I right that you are referring here to the discussion about > the fact that find_JavaThread_from_java_tid() still could return the thread that > is exiting? Yes you are right. > If so then, yes, this behavior was not changed. We discussed it with Serguei > and the conclusion was that since the current implementation behaves exact the same way > then even if decide somehow address this then it makes sense to do it in a separate issue. > It also is not clear at this moment what the possible solution could be. The obvious one, > just to hold Thread_locks in the callers of find_JavaThread_from_java_tid() (management.cpp) > is too expensive and doesn't look as acceptable. As far as I am concerned it is just an optimization to try to avoid returning a JavaThread that is in the process of exiting. It is an unavoidable race. Even holding the Threads_lock doesn't really help as the thread can still be "exited" for all intents and purposes even if the JavaThread can't truly terminate. The ThreadSMR now ensures that it is always safe to inspect a JavaThread. If we catch such a thread late in its termination sequence then its threadObj will be NULL and we'll just skip it; or we'll access it in a VM op and see it is_exiting, or some combination of such checks. So I don't see any bug here that needs to be fixed. Cheers, David > Thanks! > > Best regards, > Daniil > > > ?On 9/19/19, 6:15 PM, "David Holmes" wrote: > > Hi Daniil, > > On 20/09/2019 10:30 am, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > Ok. > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > That seems good too. > > I agree with previous comments about the general race with is_exiting in > terms of how this API behaves. But there's no change in that behaviour > with your changes AFAICS. The main concern, now addressed, is that you > mustn't be able to add a thread that never gets removed. > > Thanks, > David > ----- > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > ?On 9/18/19, 1:01 AM, "serguei.spitsyn at oracle.com" wrote: > > > > Hi Daniil, > > > > On 9/17/19 17:13, Daniil Titov wrote: > > > Hi Serguei, > > > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > > > 1. > > > > I have a concern about the checks for thread->is_exiting(). > > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > >> It is interesting what might happen if an exiting thread is returned by the > > >> ThreadsList::find_JavaThread_from_java_tid (). > > >> Does it make sense to develop a test that would cover these cases? > > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > > > Now, I'm not that confident about it. :) > > > > > The current implementation > > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > > > If I understand it correctly, the jt->threadObj() can remain non-NULL > > for some time while jt->is_exiting() == true. > > It is not clear how reliable is to use it. > > But this is a pre-existing issue. It is not you who introduced it. :) > > > > So, we can skip it for now. > > But for the record, we may have a source of intermittent issues. > > > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > > > Understand. > > > > > 2. > > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > > > > Okay, thanks! > > > > > 3. > > >> I would suggest to rewrite this fragment in a safe way: > > >> 95 { > > >> 96 MutexLocker ml(ThreadTableCreate_lock); > > >> 97 if (!_is_initialized) { > > >> 98 create_table(threads->length()); > > >> 99 _is_initialized = true; > > >> 100 } > > >> 101 } > > >> as: > > >> { > > >> MutexLocker ml(ThreadTableCreate_lock); > > >> if (_is_initialized) { > > >> return; > > > > } > > > > create_table(threads->length()); > > > > _is_initialized = true; > > > > } > > > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > > and thus avoid the linear scan when these thread are looked up for the first time. > > > > > > I've replied to David in another email. > > Let's talk once more about it tomorrow. > > > > > > > 4. > > >>> The case you have described is exact the reason why we still have a code inside > > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > > >> I disagree because it is easy to avoid concurrent ThreadTable > > >> initialization (please, see my separate email). > > >> The reason for this code is to cover a case of late/lazy ThreadTable > > >> initialization. > > > David Holmes replied to this in a separate email providing a very detailed > > > explanation of the possible cases and how the proposed implementation satisfies them. > > > > Yes. Please, see above. > > > > Thanks, > > Serguei > > > > > Best regards, > > > Daniil > > > > > > From: "serguei.spitsyn at oracle.com" > > > Date: Tuesday, September 17, 2019 at 1:53 AM > > > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > Hi Daniil, > > > > > > Thank you for you patience in working on this issue! > > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > > It was a good idea to separate them. > > > > > > I have a concern about the checks for thread->is_exiting(). > > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > > > There is a couple of such checks to mention: > > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > 612 ThreadTable::lazy_initialize(this); > > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > > 614 if (thread == NULL) { > > > 615 // If the thread is not found in the table find it > > > 616 // with a linear search and add to the table. > > > 617 for (uint i = 0; i < length(); i++) { > > > 618 thread = thread_at(i); > > > 619 oop tobj = thread->threadObj(); > > > 620 // Ignore the thread if it hasn't run yet, has exited > > > 621 // or is starting to exit. > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > > 623 MutexLocker ml(Threads_lock); > > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > 626 if (!thread->is_exiting()) { > > > 627 ThreadTable::add_thread(java_tid, thread); > > > 628 return thread; > > > 629 } > > > 630 } > > > 631 } > > > 632 } else if (!thread->is_exiting()) { > > > 633 return thread; > > > 634 } > > > 635 return NULL; > > > 636 } > > > ... > > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > > 94 if (!_is_initialized) { > > > 95 { > > > 96 MutexLocker ml(ThreadTableCreate_lock); > > > 97 if (!_is_initialized) { > > > 98 create_table(threads->length()); > > > 99 _is_initialized = true; > > > 100 } > > > 101 } > > > 102 for (uint i = 0; i < threads->length(); i++) { > > > 103 JavaThread* thread = threads->thread_at(i); > > > 104 oop tobj = thread->threadObj(); > > > 105 if (tobj != NULL && !thread->is_exiting()) { > > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > > 107 add_thread(java_tid, thread); > > > 108 } > > > 109 } > > > 110 } > > > 111 } > > > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > > So that: > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > > are protected with the Threads_lock. But the returned thread still can exit after that. > > > It is interesting what might happen if an exiting thread is returned by the > > > ThreadsList::find_JavaThread_from_java_tid (). > > > > > > Does it make sense to develop a test that would cover these cases? > > > > > > Thanks, > > > Serguei > > > > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > > Hello, > > > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > > 623 MutexLocker ml(Threads_lock); > > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > 626 if (!thread->is_exiting()) { > > > 627 ThreadTable::add_thread(java_tid, thread); > > > 628 return thread; > > > 629 } > > > 630 } > > > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > > > ?Thank you, > > > Daniil > > > > > > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > > Hi David, > > > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > > the changes you suggested: > > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > > > Okay. > > > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > > > - fixed nits and formatting issues. > > > > > > > > Okay. > > > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > >>> as Daniel suggested. > > > > >> Not sure it's best to combine these, but if they are limited to the > > > > >> changes in management.cpp only then that may be okay. > > > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > > I also could move it in the separate issue if required. > > > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > > Dan or Serguei have a strong opinion. > > > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > > non-null threadObj. > > > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > > when VM is destroyed: > > > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > > issue with JNI_AttachCurrentThread and its interaction with the > > > > ThreadSMR iterators. The attach process is: > > > > - create JavaThread > > > > - mark as "is attaching via jni" > > > > - add to ThreadsList > > > > - create java.lang.Thread object (you can only execute Java code after > > > > you are attached) > > > > - mark as "attach completed" > > > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > > iterator but will have a NULL java.lang.Thread object. > > > > > > > > We special-case attaching threads in a number of places in the VM and I > > > > think we should be explicitly doing something here to filter out > > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > ThreadTable::add_thread(tid, thread); > > > > } > > > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > > which covers the case the JNI attach encountered an error trying to > > > > create the j.l.Thread object. > > > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > > >> have it, but I'm concerned that its very existence means we perhaps > > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > > removed by calling ConcurrentHashTable::remove() method. > > > > > I think that just because in our case we don't use this mechanism doesn't > > > > > mean we should not use ConcurrentHashTable. > > > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > > back from vacation this week. > > > > > > > > >> I would still want to see what impact this has on thread > > > > >> startup cost, both with and without the table being initialized. > > > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > > thread table makes the thread startup about 2.7% slower. > > > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > > help investigate overall performance impact here. > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > No further code comments. > > > > > > > > I didn't look at the test in detail. > > > > > > > > Thanks, > > > > David > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > > see some performance and footprint numbers, both to verify it fixes the > > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > > populated, table to be seen by other threads - is that your intention? > > > > > It seems it should be okay as the other threads will then race with the > > > > > initializing thread to add specific entries, and this is a concurrent > > > > > map so that should be functionally correct. But if so then I think you > > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > > > I like the approach of only initializing the table when needed and using > > > > > that to control when the add/remove-thread code needs to update the > > > > > table. But I would still want to see what impact this has on thread > > > > > startup cost, both with and without the table being initialized. > > > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > > as Daniel suggested. > > > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > > changes in management.cpp only then that may be okay. It helps to be > > > > > able to focus on the table related changes without being distracted by > > > > > other optimizations. > > > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > > > The revised version seems better in that regard. But I still have a > > > > > concern, see below. > > > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > > growing the thread table when required. > > > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > > is doing the addition? For other tables we may have to delegate to the > > > > > service thread because the current thread cannot perform the action, or > > > > > it doesn't want to perform it at the time the need for the resize is > > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > > happen later outside the safepoint). It's not apparent to me that such > > > > > restrictions apply here. > > > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > > > Ok. > > > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > > > Some specific code comments: > > > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > > false, Monitor::_safepoint_check_never); > > > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > > be created by regular JavaThreads and they should (nearly) always be > > > > > checking for safepoints if they are going to block acquiring the lock. > > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > > to a safepoint whilst this lock is held. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > > > Nit: > > > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > > just rename that "new" variable to "thread" so you don't have to change > > > > > all other uses. > > > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > > > You don't need to check != NULL here as you only get here when > > > > > java_thread is not NULL. > > > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > > > I think it cleaner/better to just use > > > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > > > --- > > > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > > 56 private: > > > > > 57 jlong _tid; > > > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > > class declarations, so the above would just be: > > > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > > 56 private: > > > > > 57 jlong _tid; > > > > > > > > > > etc. > > > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > > ... > > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > > have it, but I'm concerned that its very existence means we perhaps > > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > > would be: > > > > > > > > > > 115 size_t start_size_log = > > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > > DefaultThreadTableSizeLog; > > > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > > 132 return (double)_items_count/_current_size; > > > > > 133 } > > > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > > division and then cast that whole integer to a double. If you want > > > > > double arithmetic you need: > > > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > > > 180 jlong _tid; > > > > > 181 uintx _hash; > > > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > > > line 184 should be indented. > > > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > > > Nit: need space after : > > > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > > 212 _has_work = false; > > > > > > > > > > line 211 is indented one space too far. > > > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > > > Nit: need space after , > > > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > > > Nit: need space after , > > > > > > > > > > Thanks, > > > > > David > > > > > ------ > > > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > > Hi Serguei and David, > > > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > > followed up with additional comments. Before I do a crawl through > > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > > made optional and David's other comments addressed. > > > > > > > > > > > > Another possible optimization is for callers of > > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > > tid value before they loop and if the current tid == saved_tid > > > > > > then use the current JavaThread* instead of calling > > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > > > Thanks! > > > > > > > --Daniil > > > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > > Organization: Oracle Corporation > > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > > 619 // to the thread table. > > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > > 624 return thread; > > > > > > > 625 } > > > > > > > 626 } > > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > 628 return java_thread; > > > > > > > 629 } > > > > > > > 630 return NULL; > > > > > > > 631 } > > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > > 635 // or is starting to exit. > > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > > 638 } > > > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > > > A space is missed after the comma: > > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Serguei > > > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > > need a lot more data on it. > > > > > > > > > > > > > > Thanks, > > > > > > > David > > > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > > in the thread table. > > > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Best regards, > > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Fri Sep 20 05:30:39 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 19 Sep 2019 22:30:39 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From daniil.x.titov at oracle.com Fri Sep 20 15:42:15 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Fri, 20 Sep 2019 08:42:15 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Hi Serguei, > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); If I understood you correctly in the code snippet you sent you meant to use Threads_lock, not ThreadTableCreate_lock, right? The original idea was to do a minimal amount of work while holding the lock and hold the lock for as short period of time as possible to not block other threads when it is not necessary. With the suggested approach no new threads could be started until the thread table is created and populated with all threads running inside a Java application and in case of large app there could be thousands of them. And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock is higher than the rank of ThreadTableCreate_lock. So choosing between blocking new threads from starting and potentially allowing some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. Thanks! Best regards, Daniil From: "serguei.spitsyn at oracle.com" Date: Thursday, September 19, 2019 at 10:30 PM To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) Hi Daniil, I think, it is better to grab the thread_lock just once at lazy initialization. It would look simpler, something, like this would work: void ThreadTable::lazy_initialize(const ThreadsList *threads) { if (_is_initialized) { return; } MutexLocker ml(ThreadTableCreate_lock); if (_is_initialized) { // There is no obvious benefits in allowing the thread table // being concurrently populated during the initalization. return; } create_table(threads->length()); _is_initialized = true; for (uint i = 0; i < threads->length(); i++) { JavaThread* thread = threads->thread_at(i); oop tobj = thread->threadObj(); if (tobj != NULL) { jlong java_tid = java_lang_Thread::thread_id(tobj); if (!thread->is_exiting()) { // Must be inside the lock to ensure that we don't add the thread to the table // that has just passed the removal point in ThreadsSMRSupport::remove_thread() add_thread(java_tid, thread); } } } } Otherwise, concurrent executions of the find_JavaThread_from_java_tid() will sometimes do a linear search of threads that are not included yet to the ThreadTable from the ThreadsList (which is used for lazy initialization). Instead, it is better to wait for the lazy_initialization() to complete. Thanks, Serguei On 9/19/19 17:30, Daniil Titov wrote: Hi David and Serguei, Please review new version of the fix that includes the changes Serguei suggested: 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread that has just passed the removal point in ThreadsSMRSupport::remove_thread() The changes are in ThreadTable::lazy_initialize() method only. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! --Daniil ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: Hi Daniil, On 9/17/19 17:13, Daniil Titov wrote: > Hi Serguei, > > Please find below my answers to the concerns you mentioned in the previous email. > > 1. > > I have a concern about the checks for thread->is_exiting(). > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread >> It is interesting what might happen if an exiting thread is returned by the >> ThreadsList::find_JavaThread_from_java_tid (). >> Does it make sense to develop a test that would cover these cases? > I agree, it doesn't really provide any protection so it makes sense just remove it. Now, I'm not that confident about it. :) > The current implementation > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. If I understand it correctly, the jt->threadObj() can remain non-NULL for some time while jt->is_exiting() == true. It is not clear how reliable is to use it. But this is a pre-existing issue. It is not you who introduced it. :) So, we can skip it for now. But for the record, we may have a source of intermittent issues. > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. Understand. > 2. >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() Okay, thanks! > 3. >> I would suggest to rewrite this fragment in a safe way: >> 95 { >> 96 MutexLocker ml(ThreadTableCreate_lock); >> 97 if (!_is_initialized) { >> 98 create_table(threads->length()); >> 99 _is_initialized = true; >> 100 } >> 101 } >> as: >> { >> MutexLocker ml(ThreadTableCreate_lock); >> if (_is_initialized) { >> return; > > } > > create_table(threads->length()); > > _is_initialized = true; > > } > > It was an intension to not block while populating the table with the threads from the current thread list. > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > it to complete since the requested thread could be not present in the thread list that triggers the thread table > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > and thus avoid the linear scan when these thread are looked up for the first time. I've replied to David in another email. Let's talk once more about it tomorrow. > 4. >>> The case you have described is exact the reason why we still have a code inside >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds >>> the requested thread to the thread table if it is not there ( lines 614-613 below). >> I disagree because it is easy to avoid concurrent ThreadTable >> initialization (please, see my separate email). >> The reason for this code is to cover a case of late/lazy ThreadTable >> initialization. > David Holmes replied to this in a separate email providing a very detailed > explanation of the possible cases and how the proposed implementation satisfies them. Yes. Please, see above. Thanks, Serguei > Best regards, > Daniil > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > Date: Tuesday, September 17, 2019 at 1:53 AM > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > Thank you for you patience in working on this issue! > Also, I like that the current thread related optimizations in management.cpp were factored out. > It was a good idea to separate them. > > I have a concern about the checks for thread->is_exiting(). > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > There is a couple of such checks to mention: > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > 612 ThreadTable::lazy_initialize(this); > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > 614 if (thread == NULL) { > 615 // If the thread is not found in the table find it > 616 // with a linear search and add to the table. > 617 for (uint i = 0; i < length(); i++) { > 618 thread = thread_at(i); > 619 oop tobj = thread->threadObj(); > 620 // Ignore the thread if it hasn't run yet, has exited > 621 // or is starting to exit. > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > 631 } > 632 } else if (!thread->is_exiting()) { > 633 return thread; > 634 } > 635 return NULL; > 636 } > ... > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > 94 if (!_is_initialized) { > 95 { > 96 MutexLocker ml(ThreadTableCreate_lock); > 97 if (!_is_initialized) { > 98 create_table(threads->length()); > 99 _is_initialized = true; > 100 } > 101 } > 102 for (uint i = 0; i < threads->length(); i++) { > 103 JavaThread* thread = threads->thread_at(i); > 104 oop tobj = thread->threadObj(); > 105 if (tobj != NULL && !thread->is_exiting()) { > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > 107 add_thread(java_tid, thread); > 108 } > 109 } > 110 } > 111 } > > A thread may start exiting right after the checks at the lines 626 and 105. > So that: > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > are protected with the Threads_lock. But the returned thread still can exit after that. > It is interesting what might happen if an exiting thread is returned by the > ThreadsList::find_JavaThread_from_java_tid (). > > Does it make sense to develop a test that would cover these cases? > > Thanks, > Serguei > > > On 9/16/19 11:18, Daniil Titov wrote: > Hello, > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > src/hotspot/share/runtime/threadSMR.cpp > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > 623 MutexLocker ml(Threads_lock); > 624 // Must be inside the lock to ensure that we don't add the thread to the table > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > 626 if (!thread->is_exiting()) { > 627 ThreadTable::add_thread(java_tid, thread); > 628 return thread; > 629 } > 630 } > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > ?Thank you, > Daniil > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > Hi Daniil, > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > Hi David, > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > the changes you suggested: > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > Okay. > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > - fixed nits and formatting issues. > > > > Okay. > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > >>> as Daniel suggested. > > >> Not sure it's best to combine these, but if they are limited to the > > >> changes in management.cpp only then that may be okay. > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > I also could move it in the separate issue if required. > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > Dan or Serguei have a strong opinion. > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > I think it cleaner/better to just use > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > when VM is destroyed: > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > This is actually nothing to do with the VM being destroyed, but is an > > issue with JNI_AttachCurrentThread and its interaction with the > > ThreadSMR iterators. The attach process is: > > - create JavaThread > > - mark as "is attaching via jni" > > - add to ThreadsList > > - create java.lang.Thread object (you can only execute Java code after > > you are attached) > > - mark as "attach completed" > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > iterator but will have a NULL java.lang.Thread object. > > > > We special-case attaching threads in a number of places in the VM and I > > think we should be explicitly doing something here to filter out > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > ThreadTable::add_thread(tid, thread); > > } > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > which covers the case the JNI attach encountered an error trying to > > create the j.l.Thread object. > > > > >> src/hotspot/share/services/threadTable.cpp > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > >> out of the template code in ConcurrentHashtable to see why we have to > > >> have it, but I'm concerned that its very existence means we perhaps > > >> should not be trying to extend CHT in this context. ?? > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > removed by calling ConcurrentHashTable::remove() method. > > > I think that just because in our case we don't use this mechanism doesn't > > > mean we should not use ConcurrentHashTable. > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > back from vacation this week. > > > > >> I would still want to see what impact this has on thread > > >> startup cost, both with and without the table being initialized. > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > is off the test takes about 14800 ms. Based on this information the enabled > > > thread table makes the thread startup about 2.7% slower. > > > > That doesn't sound very good. I think we may need to Claes involved to > > help investigate overall performance impact here. > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > No further code comments. > > > > I didn't look at the test in detail. > > > > Thanks, > > David > > > > > Thanks! > > > --Daniil > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > Overall I think this is a reasonable approach but I would still like to > > > see some performance and footprint numbers, both to verify it fixes the > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > Hi David, Daniel, and Serguei, > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > The initialization allows the created but unpopulated, or partially > > > populated, table to be seen by other threads - is that your intention? > > > It seems it should be okay as the other threads will then race with the > > > initializing thread to add specific entries, and this is a concurrent > > > map so that should be functionally correct. But if so then I think you > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > covers creation of the table only, not the initial population of the table. > > > > > > I like the approach of only initializing the table when needed and using > > > that to control when the add/remove-thread code needs to update the > > > table. But I would still want to see what impact this has on thread > > > startup cost, both with and without the table being initialized. > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > as Daniel suggested. > > > > > > Not sure it's best to combine these, but if they are limited to the > > > changes in management.cpp only then that may be okay. It helps to be > > > able to focus on the table related changes without being distracted by > > > other optimizations. > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > The revised version seems better in that regard. But I still have a > > > concern, see below. > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > growing the thread table when required. > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > is doing the addition? For other tables we may have to delegate to the > > > service thread because the current thread cannot perform the action, or > > > it doesn't want to perform it at the time the need for the resize is > > > detected (e.g. its detected at a safepoint and you want the resize to > > > happen later outside the safepoint). It's not apparent to me that such > > > restrictions apply here. > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > Ok. > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > Some specific code comments: > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > false, Monitor::_safepoint_check_never); > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > be created by regular JavaThreads and they should (nearly) always be > > > checking for safepoints if they are going to block acquiring the lock. > > > And it isn't at all obvious that the thread doing the creation can't go > > > to a safepoint whilst this lock is held. > > > > > > --- > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > Nit: > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > just rename that "new" variable to "thread" so you don't have to change > > > all other uses. > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > You don't need to check != NULL here as you only get here when > > > java_thread is not NULL. > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > I think it cleaner/better to just use > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > non-null threadObj. > > > > > > --- > > > > > > src/hotspot/share/services/management.cpp > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > --- > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > class declarations, so the above would just be: > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > 56 private: > > > 57 jlong _tid; > > > > > > etc. > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > line 61 should be indented as it continues line 60. > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > ... > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > out of the template code in ConcurrentHashtable to see why we have to > > > have it, but I'm concerned that its very existence means we perhaps > > > should not be trying to extend CHT in this context. ?? > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > line 116 should be indented, though in this case I think a better layout > > > would be: > > > > > > 115 size_t start_size_log = > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > DefaultThreadTableSizeLog; > > > > > > 131 double ThreadTable::get_load_factor() { > > > 132 return (double)_items_count/_current_size; > > > 133 } > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > division and then cast that whole integer to a double. If you want > > > double arithmetic you need: > > > > > > return ((double)_items_count)/_current_size; > > > > > > 180 jlong _tid; > > > 181 uintx _hash; > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > 183 ThreadTableLookup(jlong tid) > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > line 184 should be indented. > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > Nit: need space after : > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > 212 _has_work = false; > > > > > > line 211 is indented one space too far. > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > Nit: need space after , > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > Nit: need space after , > > > > > > Thanks, > > > David > > > ------ > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > Hi Serguei and David, > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > followed up with additional comments. Before I do a crawl through > > > > code review for this, I would like to see the ThreadTable stuff > > > > made optional and David's other comments addressed. > > > > > > > > Another possible optimization is for callers of > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > tid value before they loop and if the current tid == saved_tid > > > > then use the current JavaThread* instead of calling > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > Dan > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > Organization: Oracle Corporation > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > Hi Daniil, > > > > > > > > > > I have several quick comments. > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > 619 // to the thread table. > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > 621 JavaThread* thread = thread_at(i); > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > 624 return thread; > > > > > 625 } > > > > > 626 } > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > 628 return java_thread; > > > > > 629 } > > > > > 630 return NULL; > > > > > 631 } > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > 635 // or is starting to exit. > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > 638 } > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > A space is missed after the comma: > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > Thanks, > > > > > Serguei > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > Hi Daniil, > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > implementation!) will need careful examination. We have to be concerned > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > next few days. I will try to look at this asap next week, but we will > > > > > need a lot more data on it. > > > > > > > > > > Thanks, > > > > > David > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > in the thread table. > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Fri Sep 20 18:06:45 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 11:06:45 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Message-ID: <53347b95-0a5d-7143-1a74-4951b4097489@oracle.com> Hi Daniil, You are right that we also have to grab the Threads_lock around the add_thread call to avoid adding exiting threads. Let me think a little bit more here. Thanks, Serguei On 9/20/19 08:42, Daniil Titov wrote: > Hi Serguei, > >> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> if (_is_initialized) { >> return; > > } >> MutexLocker ml(ThreadTableCreate_lock); > > If I understood you correctly in the code snippet you sent you meant to use > Threads_lock, not ThreadTableCreate_lock, right? > > The original idea was to do a minimal amount of work while holding the lock and > hold the lock for as short period of time as possible to not block other threads when it is not necessary. > > With the suggested approach no new threads could be started until the thread table is created and > populated with all threads running inside a Java application and in case of large app there could be > thousands of them. > > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around > thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock > is higher than the rank of ThreadTableCreate_lock. > > So choosing between blocking new threads from starting and potentially allowing > some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. > > Thanks! > > Best regards, > Daniil > > > > From: "serguei.spitsyn at oracle.com" > Date: Thursday, September 19, 2019 at 10:30 PM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > I think, it is better to grab the thread_lock just once at lazy initialization. > It would look simpler, something, like this would work: > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > // There is no obvious benefits in allowing the thread table > // being concurrently populated during the initalization. > return; > } > create_table(threads->length()); > _is_initialized = true; > > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > if (!thread->is_exiting()) { > // Must be inside the lock to ensure that we don't add the thread to the table > // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > add_thread(java_tid, thread); > } > } > } > } > > Otherwise, concurrent executions of the find_JavaThread_from_java_tid() > will sometimes do a linear search of threads that are not included yet to > the ThreadTable from the ThreadsList (which is used for lazy initialization). > Instead, it is better to wait for the lazy_initialization() to complete. > Thanks, > Serguei > > > On 9/19/19 17:30, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From daniel.daugherty at oracle.com Fri Sep 20 21:59:09 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 17:59:09 -0400 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> Message-ID: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Daniil, Thanks for sticking with this project through the many versions. Sorry this review is late... On 9/19/19 8:30 PM, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ src/hotspot/share/runtime/mutexLocker.hpp ??? No comments. src/hotspot/share/runtime/mutexLocker.cpp ??? No comments. src/hotspot/share/runtime/threadSMR.cpp ??? L623: ??????? MutexLocker ml(Threads_lock); ??? L626: ??????? if (!thread->is_exiting()) { ??????? Re: discussion about is_exiting() ??????? The header comment is pretty clear: ????????? src/hotspot/share/runtime/thread.hpp: ??????????? // thread has called JavaThread::exit() or is terminated ??????????? bool is_exiting() const; ??????? is_exiting() might become true right after you have called it, ??????? but its purpose is to ask the question and not prevent the ??????? condition from becoming true. As David said, you should consider ??????? it an optimization. If you happen to see the condition is true, ??????? then you know that the JavaThread isn't going to be around much ??????? longer and should act accordingly. ??????? The is_exiting() implementation is: ????????? inline bool JavaThread::is_exiting() const { ? ? ? ? ? ? // Use load-acquire so that setting of _terminated by ? ? ? ? ? ? // JavaThread::exit() is seen more quickly. ? ? ? ? ? ? TerminatedTypes l_terminated = (TerminatedTypes) ? ? ? ? ?? ???? OrderAccess::load_acquire((volatile jint *) &_terminated); ? ? ? ? ??? return l_terminated == _thread_exiting || check_is_terminated(l_terminated); ????????? } ??????? and it depends on the JavaThread's _terminated field value. ? ? ? ? ? // JavaThread termination support ? ? ? ? ? enum TerminatedTypes { ? ? ? ? ?? _not_terminated = 0xDEAD - 2, ? ? ? ? ?? _thread_exiting,???????????????????????????? // JavaThread::exit() has been called for this thread ? ? ? ? ?? _thread_terminated,????????????????????????? // JavaThread is removed from thread list ? ? ? ? ?? _vm_exited?????????????????????????????????? // JavaThread is still executing native code, but VM is terminated ????????????????????????????????? ? ? ? ? ????????????? // only VM_Exit can set _vm_exited ? ? ? ??? }; ??????? so the JavaThread's _terminated field can get set to ??????? _thread_exiting independent of the Threads_lock, but ??????? it can't get set to _thread_terminated without the ??????? Threads_lock. ??????? So by grabbing the Threads_lock on L623, you make sure ??????? that ThreadTable::add_thread(java_tid, thread) does not ??????? add a JavaThread that's not on the ThreadsList. It might ??????? still become is_exiting() == true right after your ????????? L626???????? if (!thread->is_exiting()) { ??????? but it will still be on the main ThreadsList. And that ??????? means that when the JavaThread is removed from the main ??????? ThreadsList, you'll still call: ????????? L931: ??? ThreadTable::remove_thread(tid); ??? L624: ??????? // Must be inside the lock to ensure that we don't add the thread to the table ??????? typo: s/the thread/a thread/ ??? L633: ????? return thread; ??????? nit - L633 - indented too far (should be 2 spaces) src/hotspot/share/services/threadTable.hpp ??? L42: ? static void lazy_initialize(const ThreadsList *threads); ??????? nit - put space between '*' the variable: ????????? static void lazy_initialize(const ThreadsList* threads); ??????? like you do in your other decls. ??? L45: ? // Lookup and inserts ??????? Perhaps:? // Lookup and list management ??? L60-61 - nit - please delete these blank lines. src/hotspot/share/services/threadTable.cpp ??? L28: #include "runtime/timerTrace.hpp" ??????? nit - This should be after threadSMR.hpp... (alpha sorted order) ??? L39: static const size_t DefaultThreadTableSizeLog = 8; ??????? nit - your other 'static const' are not CamelCase. Why is this one? ??? L45: static ThreadTableHash* volatile _local_table = NULL; ??? L50: static volatile size_t _current_size = 0; ??? L51: static volatile size_t _items_count = 0; ??????? nit - can you group the file statics together? (up with L41). ??? L60:???? _tid(tid),_java_thread(java_thread) {} ??????? nit - space after ',' ??? L62?? jlong tid() const { return _tid;} ??? L63?? JavaThread* thread() const {return _java_thread;} ??????? nit - space before '}' ??????? nit - space after '{' on L63. ??? L70: ??? static uintx get_hash(Value const& value, bool* is_dead) { ??????? Parameter 'is_dead' is not used. ??? L74: ??? static void* allocate_node(size_t size, Value const& value) { ??????? Parameter 'value' is not used. ??? L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { ??????? Re: discussion about lazy_initialize() racing with ??????????? ThreadsList::find_JavaThread_from_java_tid() ??????? There's a couple of aspects to these two pieces of code racing ??????? with each other and racing with new thread creation. Racing with ??????? new thread creation is the easy one: ????????? If a new thread isn't added to the ThreadTable by ????????? ThreadsSMRSupport::add_thread() calling ThreadTable::add_thread(), ????????? then the point in the future where someone calls ????????? find_JavaThread_from_java_tid() will add it to the table due to ????????? the linear search when ThreadTable::find_thread_by_tid() ????????? returns NULL. ?????? As for multi-threads calling ThreadsList::find_JavaThread_from_java_tid() ?????? at the same time which results in multi-threads in lazy_initialize() ?????? at the same time... ?????? - ThreadTable creation will be linear due to ThreadTableCreate_lock. ???????? After _is_initialized is set to true, then no more callers to ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. ?????? - Once the ThreadTable is created, then multi-threads can be ???????? executing the for-loop to add their ThreadsList entries to ???????? the ThreadTable. There will be a bit of Threads_lock contention ???????? as each of the multi-threads tries to add their entries and ???????? there will be some wasted work since the multi-threads will ???????? likely have similar ThreadLists. ?????? Of course, once _is_initialized is set to true, then any caller ?????? to lazy_initialize() will return quickly and ?????? ThreadsList::find_JavaThread_from_java_tid() will call ?????? ThreadTable::find_thread_by_tid(). If the target java_tid isn't ?????? found, then we do the linear search thing here and add the ?????? the entry if we find a match in our current ThreadsList. Since ?????? we're only adding the one here, we only contend for the Threads_lock ?????? here if we find it. ?????? If ThreadsList::find_JavaThread_from_java_tid() is called with a ?????? target java_tid for a JavaThread that was created after the ?????? ThreadsList object that the caller has in hand for the ?????? find_JavaThread_from_java_tid() call, then, of course, that ?????? target 'java_tid' won't be found because the JavaThread was ?????? added the main ThreadsList _after_ the ThreadsList object was ?????? created by the caller. Of course, you have to ask where the ?????? target java_tid value came from since the JavaThread wasn't ?????? around when the ThreadsList::find_JavaThread_from_java_tid() ?????? call was made with that target java_tid value... ??? L99: ??????? // being concurently populated during the initalization. ??????? Typos? Perhaps: ???????????????? // to be concurrently populated during initialization. ??????? But I think those two comment lines are more appropriate above ??????? this line: ??????? L96: ????? MutexLocker ml(ThreadTableCreate_lock); ??? L112: ????????? // Must be inside the lock to ensure that we don't add the thread to the table ??????? typo: s/the thread/a thread/ ??? L141: ? return ((double)_items_count)/_current_size; ??????? nit - need spaces around '/'. ??? L177: ? bool equals(ThreadTableEntry **value, bool* is_dead) { ??????? nit - put space between '**' the variable: ??????????? bool equals(ThreadTableEntry** value, ??????? Parameter 'is_dead' is not used. ??? L214: ? while(true) { ??????? nit - space before '('. Short version: Thumbs up. Longer version: I don't think I've spotted anything other than nits here. Mostly I've just looked for multi-threaded races, proper usage of the Thread-SMR stuff, and minimal impact in the case where the new ThreadsTable is never needed. Dan P.S. ThreadTable is a bit of misnomer. What you really have here is a ThreadIdTable, but I'm really late to the code review flow with that comment... > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil From serguei.spitsyn at oracle.com Fri Sep 20 23:15:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 16:15:35 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Message-ID: <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> Hi Dan, Please, find a minor correction below. On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes >> Serguei suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ . . . > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??????? Re: discussion about lazy_initialize() racing with > ??????????? ThreadsList::find_JavaThread_from_java_tid() > > ??????? There's a couple of aspects to these two pieces of code racing > ??????? with each other and racing with new thread creation. Racing with > ??????? new thread creation is the easy one: > > ????????? If a new thread isn't added to the ThreadTable by > ????????? ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > ????????? then the point in the future where someone calls > ????????? find_JavaThread_from_java_tid() will add it to the table due to > ????????? the linear search when ThreadTable::find_thread_by_tid() > ????????? returns NULL. > > ?????? As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > ?????? at the same time which results in multi-threads in > lazy_initialize() > ?????? at the same time... > > ?????? - ThreadTable creation will be linear due to > ThreadTableCreate_lock. > ???????? After _is_initialized is set to true, then no more callers to > ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. > ?????? - Once the ThreadTable is created, then multi-threads can be > ???????? executing the for-loop to add their ThreadsList entries to > ???????? the ThreadTable. I guess, there is a confusion here. The lines 97-101 were recently added into the latest webrev: 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { 94 if (!_is_initialized) { 95 { 96 MutexLocker ml(ThreadTableCreate_lock); 97 if (_is_initialized) { 98 // There is no obvious benefits in allowing the thread table 99 // being concurently populated during the initalization. 100 return; 101 } It prevents multi-threads executing the for-loop to add their ThreadsList entries to the ThreadTable. Instead, these threads may not find the requested threads in the ThreadTable and so, will start linear search in their ThreadsList to add them into the ThreadTable. . . . > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... Agreed, ThreadIdTable is better name for this table. Thanks, Serguei > > >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil > From serguei.spitsyn at oracle.com Fri Sep 20 23:50:48 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 16:50:48 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> Message-ID: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Hi Daniil, Yes, the Threads_lock is still needed around thread->is_exiting() check and add_thread(). > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet > and then the nested Threads_lock around thread->is_exiting() and > add_thread(java_tid, thread) lines then it will not work since the rank > of Threads_lock is higher than the rank of ThreadTableCreate_lock. The ThreadTableCreate_lock is only used in the lazy_initialize() function. It looks safe to adjust the ThreadTableCreate_lock to fix the above problem. Then the approach below will work as needed. void ThreadTable::lazy_initialize(const ThreadsList *threads) { if (_is_initialized) { return; } MutexLocker ml(ThreadTableCreate_lock); if (_is_initialized) { // There is no obvious benefits in allowing the thread table // being concurrently populated during the initalization. return; } create_table(threads->length()); _is_initialized = true; for (uint i = 0; i < threads->length(); i++) { JavaThread* thread = threads->thread_at(i); oop tobj = thread->threadObj(); if (tobj != NULL) { jlong java_tid = java_lang_Thread::thread_id(tobj); MutexLocker ml(Threads_lock); if (!thread->is_exiting()) { // Must be inside the lock to ensure that we don't add the thread to the table // that has just passed the removal point in ThreadsSMRSupport::remove_thread() add_thread(java_tid, thread); } } } } If you rename ThreadTable to ThreadIdTable then the ThreadTableCreate_lock has to be renamed to ThreadIdTableCreate_lock. Thanks, Serguei On 9/20/19 8:42 AM, Daniil Titov wrote: > Hi Serguei, > >> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> if (_is_initialized) { >> return; > > } >> MutexLocker ml(ThreadTableCreate_lock); > > If I understood you correctly in the code snippet you sent you meant to use > Threads_lock, not ThreadTableCreate_lock, right? > > The original idea was to do a minimal amount of work while holding the lock and > hold the lock for as short period of time as possible to not block other threads when it is not necessary. > > With the suggested approach no new threads could be started until the thread table is created and > populated with all threads running inside a Java application and in case of large app there could be > thousands of them. > > And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet and then the nested Threads_lock around > thread->is_exiting() and add_thread(java_tid, thread) lines then it will not work since the rank of Threads_lock > is higher than the rank of ThreadTableCreate_lock. > > So choosing between blocking new threads from starting and potentially allowing > some other monitoring thread to do a one-time linear scan I think it makes sense to choose the latter. > > Thanks! > > Best regards, > Daniil > > > > From: "serguei.spitsyn at oracle.com" > Date: Thursday, September 19, 2019 at 10:30 PM > To: Daniil Titov , Robbin Ehn , David Holmes , , OpenJDK Serviceability , "hotspot-runtime-dev at openjdk.java.net" , "jmx-dev at openjdk.java.net" , Claes Redestad > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > Hi Daniil, > > I think, it is better to grab the thread_lock just once at lazy initialization. > It would look simpler, something, like this would work: > void ThreadTable::lazy_initialize(const ThreadsList *threads) { > if (_is_initialized) { > return; > } > MutexLocker ml(ThreadTableCreate_lock); > if (_is_initialized) { > // There is no obvious benefits in allowing the thread table > // being concurrently populated during the initalization. > return; > } > create_table(threads->length()); > _is_initialized = true; > > for (uint i = 0; i < threads->length(); i++) { > JavaThread* thread = threads->thread_at(i); > oop tobj = thread->threadObj(); > if (tobj != NULL) { > jlong java_tid = java_lang_Thread::thread_id(tobj); > if (!thread->is_exiting()) { > // Must be inside the lock to ensure that we don't add the thread to the table > // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > add_thread(java_tid, thread); > } > } > } > } > > Otherwise, concurrent executions of the find_JavaThread_from_java_tid() > will sometimes do a linear search of threads that are not included yet to > the ThreadTable from the ThreadsList (which is used for lazy initialization). > Instead, it is better to wait for the lazy_initialization() to complete. > Thanks, > Serguei > > > On 9/19/19 17:30, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil > > ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com wrote: > > Hi Daniil, > > On 9/17/19 17:13, Daniil Titov wrote: > > Hi Serguei, > > > > Please find below my answers to the concerns you mentioned in the previous email. > > > > 1. > > > I have a concern about the checks for thread->is_exiting(). > > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > >> It is interesting what might happen if an exiting thread is returned by the > >> ThreadsList::find_JavaThread_from_java_tid (). > >> Does it make sense to develop a test that would cover these cases? > > I agree, it doesn't really provide any protection so it makes sense just remove it. > > Now, I'm not that confident about it. :) > > > The current implementation > > find_JavaThread_from_java_tid() doesn't provide such protection as well, since the thread could start exiting > > immediately after method find_JavaThread_from_java_tid() returns, so the assumption is that the callers of > > find_JavaThread_from_java_tid() are expecting to deal with such threads and looking on some of them shows that > > they usually try to retrieve threadObj or a thread statistic object and if it is NULL that just do nothing. > > If I understand it correctly, the jt->threadObj() can remain non-NULL > for some time while jt->is_exiting() == true. > It is not clear how reliable is to use it. > But this is a pre-existing issue. It is not you who introduced it. :) > > So, we can skip it for now. > But for the record, we may have a source of intermittent issues. > > > I'm not sure we could cover this specific case with the test. The window between find_JavaThread_from_java_tid() returns and the caller > > continues the execution is too small. The window between the thread started exiting and removed itself from the thread table is very small as well. > > Understand. > > > 2. > >> - the lines 105-108 can result in adding exiting threads into the ThreadTable > > I agree, it was missed, we need to wrap this code inside Thread_lock in the similar way as it is done find_JavaThread_from_java_tid() > > > Okay, thanks! > > > 3. > >> I would suggest to rewrite this fragment in a safe way: > >> 95 { > >> 96 MutexLocker ml(ThreadTableCreate_lock); > >> 97 if (!_is_initialized) { > >> 98 create_table(threads->length()); > >> 99 _is_initialized = true; > >> 100 } > >> 101 } > >> as: > >> { > >> MutexLocker ml(ThreadTableCreate_lock); > >> if (_is_initialized) { > >> return; > > > } > > > create_table(threads->length()); > > > _is_initialized = true; > > > } > > > > It was an intension to not block while populating the table with the threads from the current thread list. > > There is no needs to have other threads that call find_JavaThread_from_java_tid() be blocked and waiting for > > it to complete since the requested thread could be not present in the thread list that triggers the thread table > > initialization. Plus in case of racing initialization it allows threads from not original thread lists be added to the table > > and thus avoid the linear scan when these thread are looked up for the first time. > > > I've replied to David in another email. > Let's talk once more about it tomorrow. > > > > 4. > >>> The case you have described is exact the reason why we still have a code inside > >>> ThreadsList::find_JavaThread_from_java_tid() method that does a linear scan and adds > >>> the requested thread to the thread table if it is not there ( lines 614-613 below). > >> I disagree because it is easy to avoid concurrent ThreadTable > >> initialization (please, see my separate email). > >> The reason for this code is to cover a case of late/lazy ThreadTable > >> initialization. > > David Holmes replied to this in a separate email providing a very detailed > > explanation of the possible cases and how the proposed implementation satisfies them. > > Yes. Please, see above. > > Thanks, > Serguei > > > Best regards, > > Daniil > > > > From: mailto:serguei.spitsyn at oracle.com mailto:serguei.spitsyn at oracle.com > > Date: Tuesday, September 17, 2019 at 1:53 AM > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn mailto:robbin.ehn at oracle.com, David Holmes mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes Redestad mailto:claes.redestad at oracle.com > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > Hi Daniil, > > > > Thank you for you patience in working on this issue! > > Also, I like that the current thread related optimizations in management.cpp were factored out. > > It was a good idea to separate them. > > > > I have a concern about the checks for thread->is_exiting(). > > The threads are added to and removed from the ThreadTable under protection of Threads_lock. > > However, the thread->is_exiting() checks are not protected, and so, they are racy. > > > > There is a couple of such checks to mention: > > 611 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > 612 ThreadTable::lazy_initialize(this); > > 613 JavaThread* thread = ThreadTable::find_thread_by_tid(java_tid); > > 614 if (thread == NULL) { > > 615 // If the thread is not found in the table find it > > 616 // with a linear search and add to the table. > > 617 for (uint i = 0; i < length(); i++) { > > 618 thread = thread_at(i); > > 619 oop tobj = thread->threadObj(); > > 620 // Ignore the thread if it hasn't run yet, has exited > > 621 // or is starting to exit. > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > 631 } > > 632 } else if (!thread->is_exiting()) { > > 633 return thread; > > 634 } > > 635 return NULL; > > 636 } > > ... > > 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > 94 if (!_is_initialized) { > > 95 { > > 96 MutexLocker ml(ThreadTableCreate_lock); > > 97 if (!_is_initialized) { > > 98 create_table(threads->length()); > > 99 _is_initialized = true; > > 100 } > > 101 } > > 102 for (uint i = 0; i < threads->length(); i++) { > > 103 JavaThread* thread = threads->thread_at(i); > > 104 oop tobj = thread->threadObj(); > > 105 if (tobj != NULL && !thread->is_exiting()) { > > 106 jlong java_tid = java_lang_Thread::thread_id(tobj); > > 107 add_thread(java_tid, thread); > > 108 } > > 109 } > > 110 } > > 111 } > > > > A thread may start exiting right after the checks at the lines 626 and 105. > > So that: > > - the lines 632-633 are useless as they do not really protect from returning an exiting thread > > - the lines 105-108 can result in adding exiting threads into the ThreadTable > > > > Please, note, the lines 626-629 are safe in terms of addition to the ThreadTable as they > > are protected with the Threads_lock. But the returned thread still can exit after that. > > It is interesting what might happen if an exiting thread is returned by the > > ThreadsList::find_JavaThread_from_java_tid (). > > > > Does it make sense to develop a test that would cover these cases? > > > > Thanks, > > Serguei > > > > > > On 9/16/19 11:18, Daniil Titov wrote: > > Hello, > > > > After investigating with Claes the impact of this change on the performance (thanks a lot Claes for helping with it!) the conclusion was that the impact on the thread startup time is not a blocker for this change. > > > > I also measured the memory footprint using Native Memory Tracking and results showed around 40 bytes per live thread. > > > > Please review a new version of the fix, webrev.06 [1]. Just to remind, webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] minus changes in src/hotspot/share/services/management.cpp (that were factored out to a separate issue [4]) and plus a change in ThreadsList::find_JavaThread_from_java_tid() method (please, see below) that addresses the problem Robbin found and puts the code that adds a new thread to the thread table inside Threads_lock. > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > 622 if (tobj != NULL && java_tid == java_lang_Thread::thread_id(tobj)) { > > 623 MutexLocker ml(Threads_lock); > > 624 // Must be inside the lock to ensure that we don't add the thread to the table > > 625 // that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > 626 if (!thread->is_exiting()) { > > 627 ThreadTable::add_thread(java_tid, thread); > > 628 return thread; > > 629 } > > 630 } > > > > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 > > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 > > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 > > > > ?Thank you, > > Daniil > > > > > > > > > > > > ?On 8/4/19, 7:54 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > Hi Daniil, > > > > > > On 3/08/2019 8:16 am, Daniil Titov wrote: > > > > Hi David, > > > > > > > > Thank you for your detailed review. Please review a new version of the fix that includes > > > > the changes you suggested: > > > > - ThreadTableCreate_lock scope is reduced to cover the creation of the table only; > > > > - ThreadTableCreate_lock is made _safepoint_check_always; > > > > > > Okay. > > > > > > > - ServiceThread is no longer responsible for the resizing of the thread table, instead, > > > > the thread table is changed to grow on demand by the thread that is doing the addition; > > > > > > Okay - I'm happy to get the serviceThread out of the picture here. > > > > > > > - fixed nits and formatting issues. > > > > > > Okay. > > > > > > >>> The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > >>> as Daniel suggested. > > > >> Not sure it's best to combine these, but if they are limited to the > > > >> changes in management.cpp only then that may be okay. > > > > > > > > The additional optimization for some callers of find_JavaThread_from_java_tid() is > > > > limited to management.cpp (plus a new test) so I left them in the webrev but > > > > I also could move it in the separate issue if required. > > > > > > I'd prefer this part of be separated out, but won't insist. Let's see if > > > Dan or Serguei have a strong opinion. > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > >755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > I think it cleaner/better to just use > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > > non-null threadObj. > > > > > > > > I had to leave this code unchanged since it turned out the threadObj is null > > > > when VM is destroyed: > > > > > > > > V [libjvm.so+0xe165d7] oopDesc::long_field(int) const+0x67 > > > > V [libjvm.so+0x16e06c6] ThreadsSMRSupport::add_thread(JavaThread*)+0x116 > > > > V [libjvm.so+0x16d1302] Threads::add(JavaThread*, bool)+0x82 > > > > V [libjvm.so+0xef8369] attach_current_thread.part.197+0xc9 > > > > V [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c > > > > C [libjli.so+0x4333] JavaMain+0x2c3 > > > > C [libjli.so+0x8159] ThreadJavaMain+0x9 > > > > > > This is actually nothing to do with the VM being destroyed, but is an > > > issue with JNI_AttachCurrentThread and its interaction with the > > > ThreadSMR iterators. The attach process is: > > > - create JavaThread > > > - mark as "is attaching via jni" > > > - add to ThreadsList > > > - create java.lang.Thread object (you can only execute Java code after > > > you are attached) > > > - mark as "attach completed" > > > > > > So while a thread "is attaching" it will be seen by the ThreadSMR thread > > > iterator but will have a NULL java.lang.Thread object. > > > > > > We special-case attaching threads in a number of places in the VM and I > > > think we should be explicitly doing something here to filter out > > > attaching threads, rather than just being tolerant of a NULL j.l.Thread > > > object. Specifically in ThreadsSMRSupport::add_thread: > > > > > > if (ThreadTable::is_initialized() && !thread->is_attaching_via_jni()) { > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > ThreadTable::add_thread(tid, thread); > > > } > > > > > > Note that in ThreadsSMRSupport::remove_thread we can use the same guard, > > > which covers the case the JNI attach encountered an error trying to > > > create the j.l.Thread object. > > > > > > >> src/hotspot/share/services/threadTable.cpp > > > >> 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > >> The is_dead parameter still bothers me here. I can't make enough sense > > > >> out of the template code in ConcurrentHashtable to see why we have to > > > >> have it, but I'm concerned that its very existence means we perhaps > > > >> should not be trying to extend CHT in this context. ?? > > > > > > > > My understanding is that is_dead parameter provides a mechanism for > > > > ConcurrentHashtable to remove stale entries that were not explicitly > > > > removed by calling ConcurrentHashTable::remove() method. > > > > I think that just because in our case we don't use this mechanism doesn't > > > > mean we should not use ConcurrentHashTable. > > > > > > Can you confirm that this usage is okay with Robbin Ehn please. He's > > > back from vacation this week. > > > > > > >> I would still want to see what impact this has on thread > > > >> startup cost, both with and without the table being initialized. > > > > > > > > I run a test that initializes the table by calling ThreadMXBean.get getThreadInfo(), > > > > starts some threads as a worm-up, and then creates and starts 100,000 threads > > > > (each thread just sleeps for 100 ms). In case when the thread table is enabled > > > > 100,000 threads are created and started for about 15200 ms. If the thread table > > > > is off the test takes about 14800 ms. Based on this information the enabled > > > > thread table makes the thread startup about 2.7% slower. > > > > > > That doesn't sound very good. I think we may need to Claes involved to > > > help investigate overall performance impact here. > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > No further code comments. > > > > > > I didn't look at the test in detail. > > > > > > Thanks, > > > David > > > > > > > Thanks! > > > > --Daniil > > > > > > > > > > > > ?On 7/29/19, 12:53 AM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > Hi Daniil, > > > > > > > > Overall I think this is a reasonable approach but I would still like to > > > > see some performance and footprint numbers, both to verify it fixes the > > > > problem reported, and that we are not getting penalized elsewhere. > > > > > > > > On 25/07/2019 3:21 am, Daniil Titov wrote: > > > > > Hi David, Daniel, and Serguei, > > > > > > > > > > Please review the new version of the fix, that makes the thread table initialization on demand and > > > > > moves it inside ThreadsList::find_JavaThread_from_java_tid(). At the creation time the thread table > > > > > is initialized with the threads from the current thread list. We don't want to hold Threads_lock > > > > > inside find_JavaThread_from_java_tid(), thus new threads still could be created while the thread > > > > > table is being initialized . Such threads will be found by the linear search and added to the thread table > > > > > later, in ThreadsList::find_JavaThread_from_java_tid(). > > > > > > > > The initialization allows the created but unpopulated, or partially > > > > populated, table to be seen by other threads - is that your intention? > > > > It seems it should be okay as the other threads will then race with the > > > > initializing thread to add specific entries, and this is a concurrent > > > > map so that should be functionally correct. But if so then I think you > > > > can also reduce the scope of the ThreadTableCreate_lock so that it > > > > covers creation of the table only, not the initial population of the table. > > > > > > > > I like the approach of only initializing the table when needed and using > > > > that to control when the add/remove-thread code needs to update the > > > > table. But I would still want to see what impact this has on thread > > > > startup cost, both with and without the table being initialized. > > > > > > > > > The change also includes additional optimization for some callers of find_JavaThread_from_java_tid() > > > > > as Daniel suggested. > > > > > > > > Not sure it's best to combine these, but if they are limited to the > > > > changes in management.cpp only then that may be okay. It helps to be > > > > able to focus on the table related changes without being distracted by > > > > other optimizations. > > > > > > > > > That is correct that ResolvedMethodTable was used as a blueprint for the thread table, however, I tried > > > > > to strip it of the all functionality that is not required in the thread table case. > > > > > > > > The revised version seems better in that regard. But I still have a > > > > concern, see below. > > > > > > > > > We need to have the thread table resizable and allow it to grow as the number of threads increases to avoid > > > > > reserving excessive memory a-priori or deteriorating lookup times. The ServiceThread is responsible for > > > > > growing the thread table when required. > > > > > > > > Yes but why? Why can't this table be grown on demand by the thread that > > > > is doing the addition? For other tables we may have to delegate to the > > > > service thread because the current thread cannot perform the action, or > > > > it doesn't want to perform it at the time the need for the resize is > > > > detected (e.g. its detected at a safepoint and you want the resize to > > > > happen later outside the safepoint). It's not apparent to me that such > > > > restrictions apply here. > > > > > > > > > There is no ConcurrentHashTable available in Java 8 and for backporting this fix to Java 8 another implementation > > > > > of the hash table, probably originally suggested in the patch attached to the JBS issue, should be used. It will make > > > > > the backporting more complicated, however, adding a new Implementation of the hash table in Java 14 while it > > > > > already has ConcurrentHashTable doesn't seem reasonable for me. > > > > > > > > Ok. > > > > > > > > > Webrev: http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 > > > > > > > > Some specific code comments: > > > > > > > > src/hotspot/share/runtime/mutexLocker.cpp > > > > > > > > + def(ThreadTableCreate_lock , PaddedMutex , special, > > > > false, Monitor::_safepoint_check_never); > > > > > > > > I think this needs to be a _safepoint_check_always lock. The table will > > > > be created by regular JavaThreads and they should (nearly) always be > > > > checking for safepoints if they are going to block acquiring the lock. > > > > And it isn't at all obvious that the thread doing the creation can't go > > > > to a safepoint whilst this lock is held. > > > > > > > > --- > > > > > > > > src/hotspot/share/runtime/threadSMR.cpp > > > > > > > > Nit: > > > > > > > > 618 JavaThread* thread = thread_at(i); > > > > > > > > you could reuse the new java_thread local you introduced at line 613 and > > > > just rename that "new" variable to "thread" so you don't have to change > > > > all other uses. > > > > > > > > 628 } else if (java_thread != NULL && ... > > > > > > > > You don't need to check != NULL here as you only get here when > > > > java_thread is not NULL. > > > > > > > > 755 jlong tid = SharedRuntime::get_java_tid(thread); > > > > 926 jlong tid = SharedRuntime::get_java_tid(thread); > > > > > > > > I think it cleaner/better to just use > > > > > > > > jlong tid = java_lang_Thread::thread_id(thread->threadObj()); > > > > > > > > as we know thread is not NULL, it is a JavaThread and it has to have a > > > > non-null threadObj. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/management.cpp > > > > > > > > 1323 if (THREAD->is_Java_thread()) { > > > > 1324 JavaThread* current_thread = (JavaThread*)THREAD; > > > > > > > > These calls can only be made on a JavaThread so this be simplified to > > > > remove the is_Java_thread() call. Similarly in other places. > > > > > > > > --- > > > > > > > > src/hotspot/share/services/threadTable.cpp > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > I believe hotspot style is to not indent the access modifiers in C++ > > > > class declarations, so the above would just be: > > > > > > > > 55 class ThreadTableEntry : public CHeapObj { > > > > 56 private: > > > > 57 jlong _tid; > > > > > > > > etc. > > > > > > > > 60 ThreadTableEntry(jlong tid, JavaThread* java_thread) : > > > > 61 _tid(tid),_java_thread(java_thread) {} > > > > > > > > line 61 should be indented as it continues line 60. > > > > > > > > 67 class ThreadTableConfig : public AllStatic { > > > > ... > > > > 71 static uintx get_hash(Value const& value, bool* is_dead) { > > > > > > > > The is_dead parameter still bothers me here. I can't make enough sense > > > > out of the template code in ConcurrentHashtable to see why we have to > > > > have it, but I'm concerned that its very existence means we perhaps > > > > should not be trying to extend CHT in this context. ?? > > > > > > > > 115 size_t start_size_log = size_log > DefaultThreadTableSizeLog > > > > 116 ? size_log : DefaultThreadTableSizeLog; > > > > > > > > line 116 should be indented, though in this case I think a better layout > > > > would be: > > > > > > > > 115 size_t start_size_log = > > > > 116 size_log > DefaultThreadTableSizeLog ? size_log : > > > > DefaultThreadTableSizeLog; > > > > > > > > 131 double ThreadTable::get_load_factor() { > > > > 132 return (double)_items_count/_current_size; > > > > 133 } > > > > > > > > Not sure that is doing what you want/expect. It will perform integer > > > > division and then cast that whole integer to a double. If you want > > > > double arithmetic you need: > > > > > > > > return ((double)_items_count)/_current_size; > > > > > > > > 180 jlong _tid; > > > > 181 uintx _hash; > > > > > > > > Nit: no need for all those spaces before the variable name. > > > > > > > > 183 ThreadTableLookup(jlong tid) > > > > 184 : _tid(tid), _hash(primitive_hash(tid)) {} > > > > > > > > line 184 should be indented. > > > > > > > > 201 ThreadGet():_return(NULL) {} > > > > > > > > Nit: need space after : > > > > > > > > 211 assert(_is_initialized, "Thread table is not initialized"); > > > > 212 _has_work = false; > > > > > > > > line 211 is indented one space too far. > > > > > > > > 229 ThreadTableEntry* entry = new ThreadTableEntry(tid,java_thread); > > > > > > > > Nit: need space after , > > > > > > > > 252 return _local_table->remove(thread,lookup); > > > > > > > > Nit: need space after , > > > > > > > > Thanks, > > > > David > > > > ------ > > > > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > Thanks! > > > > > --Daniil > > > > > > > > > > > > > > > ?On 7/8/19, 3:24 PM, "Daniel D. Daugherty" mailto:daniel.daugherty at oracle.com wrote: > > > > > > > > > > On 6/29/19 12:06 PM, Daniil Titov wrote: > > > > > > Hi Serguei and David, > > > > > > > > > > > > Serguei is right, ThreadTable::find_thread(java_tid) cannot return a JavaThread with an unmatched java_tid. > > > > > > > > > > > > Please find a new version of the fix that includes the changes Serguei suggested. > > > > > > > > > > > > Regarding the concern about the maintaining the thread table when it may never even be queried, one of > > > > > > the options could be to add ThreadTable ::isEnabled flag, set it to "false" by default, and wrap the calls to the thread table > > > > > > in ThreadsSMRSupport add_thread() and remove_thread() methods to check this flag. > > > > > > > > > > > > When ThreadsList::find_JavaThread_from_java_tid() is called for the first time it could check if ThreadTable ::isEnabled > > > > > > Is on and if not then set it on and populate the thread table with all existing threads from the thread list. > > > > > > > > > > I have the same concerns as David H. about this new ThreadTable. > > > > > ThreadsList::find_JavaThread_from_java_tid() is only called from code > > > > > in src/hotspot/share/services/management.cpp so I think that table > > > > > needs to enabled and populated only if it is going to be used. > > > > > > > > > > I've taken a look at the webrev below and I see that David has > > > > > followed up with additional comments. Before I do a crawl through > > > > > code review for this, I would like to see the ThreadTable stuff > > > > > made optional and David's other comments addressed. > > > > > > > > > > Another possible optimization is for callers of > > > > > find_JavaThread_from_java_tid() to save the calling thread's > > > > > tid value before they loop and if the current tid == saved_tid > > > > > then use the current JavaThread* instead of calling > > > > > find_JavaThread_from_java_tid() to get the JavaThread*. > > > > > > > > > > Dan > > > > > > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > --Daniil > > > > > > > > > > > > From: mailto:serguei.spitsyn at oracle.com > > > > > > Organization: Oracle Corporation > > > > > > Date: Friday, June 28, 2019 at 7:56 PM > > > > > > To: Daniil Titov mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, mailto:hotspot-runtime-dev at openjdk.java.net mailto:hotspot-runtime-dev at openjdk.java.net, mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net > > > > > > Subject: Re: RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > I have several quick comments. > > > > > > > > > > > > The indent in the hotspot c/c++ files has to be 2, not 4. > > > > > > > > > > > > https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html > > > > > > 614 JavaThread* ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > 616 if (java_thread == NULL && java_tid == PMIMORDIAL_JAVA_TID) { > > > > > > 617 // ThreadsSMRSupport::add_thread() is not called for the primordial > > > > > > 618 // thread. Thus, we find this thread with a linear search and add it > > > > > > 619 // to the thread table. > > > > > > 620 for (uint i = 0; i < length(); i++) { > > > > > > 621 JavaThread* thread = thread_at(i); > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > 623 ThreadTable::add_thread(java_tid, thread); > > > > > > 624 return thread; > > > > > > 625 } > > > > > > 626 } > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > 628 return java_thread; > > > > > > 629 } > > > > > > 630 return NULL; > > > > > > 631 } > > > > > > 632 bool ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* java_thread) { > > > > > > 633 oop tobj = java_thread->threadObj(); > > > > > > 634 // Ignore the thread if it hasn't run yet, has exited > > > > > > 635 // or is starting to exit. > > > > > > 636 return (tobj != NULL && !java_thread->is_exiting() && > > > > > > 637 java_tid == java_lang_Thread::thread_id(tobj)); > > > > > > 638 } > > > > > > > > > > > > 615 JavaThread* java_thread = ThreadTable::find_thread(java_tid); > > > > > > > > > > > > I'd suggest to rename find_thread() to find_thread_by_tid(). > > > > > > > > > > > > A space is missed after the comma: > > > > > > 622 if (is_valid_java_thread(java_tid,thread)) { > > > > > > > > > > > > An empty line is needed before L632. > > > > > > > > > > > > The name 'is_valid_java_thread' looks wrong (or confusing) to me. > > > > > > Something like 'is_alive_java_thread_with_tid()' would be better. > > > > > > It'd better to list parameters in the opposite order. > > > > > > > > > > > > The call to is_valid_java_thread() is confusing: > > > > > > 627 } else if (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { > > > > > > > > > > > > Why would the call ThreadTable::find_thread(java_tid) return a JavaThread with an unmatched java_tid? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Serguei > > > > > > > > > > > > On 6/28/19, 9:40 PM, "David Holmes" mailto:david.holmes at oracle.com wrote: > > > > > > > > > > > > Hi Daniil, > > > > > > > > > > > > The definition and use of this hashtable (yet another hashtable > > > > > > implementation!) will need careful examination. We have to be concerned > > > > > > about the cost of maintaining it when it may never even be queried. You > > > > > > would need to look at footprint cost and performance impact. > > > > > > > > > > > > Unfortunately I'm just about to board a plane and will be out for the > > > > > > next few days. I will try to look at this asap next week, but we will > > > > > > need a lot more data on it. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > On 6/28/19 3:31 PM, Daniil Titov wrote: > > > > > > Please review the change that improves performance of ThreadMXBean MXBean methods returning the > > > > > > information for specific threads. The change introduces the thread table that uses ConcurrentHashTable > > > > > > to store one-to-one the mapping between the thread ids and JavaThread objects and replaces the linear > > > > > > search over the thread list in ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the lookup > > > > > > in the thread table. > > > > > > > > > > > > Testing: Mach5 tier1,tier2 and tier3 tests successfully passed. > > > > > > > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Daniil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From serguei.spitsyn at oracle.com Sat Sep 21 00:07:01 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 17:07:01 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Message-ID: <43f900b8-538f-6270-033b-cf71748e7f23@oracle.com> On 9/20/19 4:50 PM, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Yes, the Threads_lock is still needed around thread->is_exiting() > check and add_thread(). > >> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >> and then the nested Threads_lock around thread->is_exiting() and >> add_thread(java_tid, thread) lines then it will not work since the rank >> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. > > The ThreadTableCreate_lock is only used in the lazy_initialize() > function. > It looks safe to adjust the ThreadTableCreate_lock to fix the above > problem. Sorry, I wanted to say: "to adjust the ThreadTableCreate_lock rank". Thanks, Serguei > Then the approach below will work as needed. > > ?void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??? if (_is_initialized) { > ????? return; > ??? } > ??? MutexLocker ml(ThreadTableCreate_lock); > ??? if (_is_initialized) { > ????? // There is no obvious benefits in allowing the thread table > ????? // being concurrently populated during the initalization. > ????? return; > ??? } > ??? create_table(threads->length()); > ??? _is_initialized = true; > > ??? for (uint i = 0; i < threads->length(); i++) { > ????? JavaThread* thread = threads->thread_at(i); > ????? oop tobj = thread->threadObj(); > ????? if (tobj != NULL) { > ??????? jlong java_tid = java_lang_Thread::thread_id(tobj); > ??????? MutexLocker ml(Threads_lock); > ??????? if (!thread->is_exiting()) { > ????????? // Must be inside the lock to ensure that we don't add the > thread to the table > ????????? // that has just passed the removal point in > ThreadsSMRSupport::remove_thread() > ????????? add_thread(java_tid, thread); > ??????? } > ????? } > ??? } > ? } > > > If you rename ThreadTable to ThreadIdTable then the > ThreadTableCreate_lock > has to be renamed to ThreadIdTableCreate_lock. > > Thanks, > Serguei > > On 9/20/19 8:42 AM, Daniil Titov wrote: >> Hi Serguei, >> >>> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ????? if (_is_initialized) { >>> ????? return; >> ? >?? } >>> ??? MutexLocker ml(ThreadTableCreate_lock); >> >> If I understood you correctly in the code snippet you sent you meant >> to? use >> Threads_lock, not ThreadTableCreate_lock, right? >> >> The original idea was to do a minimal amount of work while holding >> the lock and >> ? hold the lock for as short period of time as possible to not block >> other threads when it is not necessary. >> >> With the suggested approach no new threads could be started until the >> thread table is created and >> populated with all threads running inside a Java application and in >> case of large app there could be >> thousands of them. >> >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your >> snippet and then the nested Threads_lock around >> thread->is_exiting() and add_thread(java_tid, thread) lines then it >> will not work since the rank of Threads_lock >> is higher than the rank of ThreadTableCreate_lock. >> >> So choosing between blocking new threads from starting and >> potentially allowing >> some other monitoring thread? to do a one-time linear scan I think it >> makes sense to choose the latter. >> >> Thanks! >> >> Best regards, >> Daniil >> >> >> >> From: "serguei.spitsyn at oracle.com" >> Date: Thursday, September 19, 2019 at 10:30 PM >> To: Daniil Titov , Robbin Ehn >> , David Holmes , >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> , Claes Redestad >> Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> >> Hi Daniil, >> >> I think, it is better to grab the thread_lock just once at lazy >> initialization. >> It would look simpler, something, like this would work: >> ?? void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } >> >> Otherwise, concurrent executions of the find_JavaThread_from_java_tid() >> will sometimes do a linear search of threads that are not included >> yet to >> the ThreadTable from the ThreadsList (which is used for lazy >> initialization). >> Instead, it is better to wait for the lazy_initialization() to complete. >> Thanks, >> Serguei >> >> >> On 9/19/19 17:30, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes >> Serguei suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil >> >> ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com wrote: >> >> ???? Hi Daniil, >> ???? ???? On 9/17/19 17:13, Daniil Titov wrote: >> ???? > Hi Serguei, >> ???? > >> ???? > Please find below my answers to the concerns you mentioned in >> the previous email. >> ???? > >> ???? > 1. >> ???? >?? > I have a concern about the checks for thread->is_exiting(). >> ???? >?? > - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >> It is interesting what might happen if an exiting thread is >> returned by the >> ???? >> ThreadsList::find_JavaThread_from_java_tid (). >> ???? >> Does it make sense to develop a test that would cover these >> cases? >> ???? > I agree, it doesn't really provide any protection so it makes >> sense just remove it. >> ???? ???? Now, I'm not that confident about it. :) >> ???? ???? >?? The current implementation >> ???? > find_JavaThread_from_java_tid()? doesn't provide such >> protection as well, since the thread could start exiting >> ???? > immediately after method find_JavaThread_from_java_tid() >> returns, so the assumption is that the callers of >> ???? > find_JavaThread_from_java_tid()? are expecting to deal with >> such threads and? looking on some of them shows that >> ???? > they usually try to retrieve threadObj or a thread statistic >> object and if it is NULL that just do nothing. >> ???? ???? If I understand it correctly, the jt->threadObj() can >> remain non-NULL >> ???? for some time while jt->is_exiting() == true. >> ???? It is not clear how reliable is to use it. >> ???? But this is a pre-existing issue. It is not you who introduced >> it. :) >> ???? ???? So, we can skip it for now. >> ???? But for the record, we may have a source of intermittent issues. >> ???? ???? > I'm not sure we could cover this specific case with the >> test. The window between find_JavaThread_from_java_tid() returns and >> the caller >> ???? > continues the execution is too small. The window between the >> thread started exiting and removed itself from the thread table is >> very small as well. >> ???? ???? Understand. >> ???? ???? > 2. >> ???? >>?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? >?? I agree, it was missed, we need to wrap this code inside >> Thread_lock in the similar way as it is done >> find_JavaThread_from_java_tid() >> ???? ???? ???? Okay, thanks! >> ???? ???? > 3. >> ???? >> I would suggest to rewrite this fragment in a safe way: >> ???? >>?? 95???? { >> ???? >>?? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>?? 97?????? if (!_is_initialized) { >> ???? >>?? 98???????? create_table(threads->length()); >> ???? >>?? 99???????? _is_initialized = true; >> ???? >> 100?????? } >> ???? >> 101???? } >> ???? >> as: >> ???? >>??????? { >> ???? >>????????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>????????? if (_is_initialized) { >> ???? >>??????????? return; >> ???? >?? >??????? } >> ???? >?? >??????? create_table(threads->length()); >> ???? >?? >??????? _is_initialized = true; >> ???? >?? >????? } >> ???? > >> ???? > It was an intension to not block? while populating the table >> with the threads from the current thread list. >> ???? > There is no needs to have other threads that call >> find_JavaThread_from_java_tid()? be blocked and waiting for >> ???? >?? it to complete since the requested thread could be not >> present in the thread list that triggers the thread table >> ???? >?? initialization. Plus in case of racing initialization it >> allows threads from not original? thread lists be added to the table >> ???? > and thus avoid the linear scan when these thread are looked up >> for the first time. >> ???? ???? ???? I've replied to David in another email. >> ???? Let's talk once more about it tomorrow. >> ???? ???? ???? > 4. >> ???? >>> The case you have described is exact the reason why we still >> have a code inside >> ???? >>> ThreadsList::find_JavaThread_from_java_tid() method that >> does a linear scan and adds >> ???? >>>??? the requested thread to the thread table if it is not >> there ( lines 614-613 below). >> ???? >> I disagree because it is easy to avoid concurrent ThreadTable >> ???? >> initialization (please, see my separate email). >> ???? >> The reason for this code is to cover a case of late/lazy >> ThreadTable >> ???? >> initialization. >> ???? > David Holmes replied to this in a separate email providing a >> very detailed >> ???? > explanation of the possible cases and how the proposed >> implementation satisfies them. >> ???? ???? Yes. Please, see above. >> ???? ???? Thanks, >> ???? Serguei >> ???? ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > From: mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com >> ???? > Date: Tuesday, September 17, 2019 at 1:53 AM >> ???? > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn >> mailto:robbin.ehn at oracle.com, David Holmes >> mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, >> OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, >> Claes Redestad mailto:claes.redestad at oracle.com >> ???? > Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> ???? > >> ???? > Hi Daniil, >> ???? > >> ???? > Thank you for you patience in working on this issue! >> ???? > Also, I like that the current thread related optimizations in >> management.cpp were factored out. >> ???? > It was a good idea to separate them. >> ???? > >> ???? > I have a concern about the checks for thread->is_exiting(). >> ???? > The threads are added to and removed from the ThreadTable >> under protection of Threads_lock. >> ???? > However, the thread->is_exiting() checks are not protected, >> and so, they are racy. >> ???? > >> ???? > There is a couple of such checks to mention: >> ???? >?? 611 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >?? 612?? ThreadTable::lazy_initialize(this); >> ???? >?? 613?? JavaThread* thread = >> ThreadTable::find_thread_by_tid(java_tid); >> ???? >?? 614?? if (thread == NULL) { >> ???? >?? 615???? // If the thread is not found in the table find it >> ???? >?? 616???? // with a linear search and add to the table. >> ???? >?? 617???? for (uint i = 0; i < length(); i++) { >> ???? >?? 618?????? thread = thread_at(i); >> ???? >?? 619?????? oop tobj = thread->threadObj(); >> ???? >?? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ???? >?? 621?????? // or is starting to exit. >> ???? >?? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? >?? 623???????? MutexLocker ml(Threads_lock); >> ???? >?? 624???????? // Must be inside the lock to ensure that we >> don't add the thread to the table >> ???? >?? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >?? 626???????? if (!thread->is_exiting()) { >> ???? >?? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? >?? 628?????????? return thread; >> ???? >?? 629???????? } >> ???? >?? 630?????? } >> ???? >?? 631???? } >> ???? >?? 632?? } else if (!thread->is_exiting()) { >> ???? >?? 633?????? return thread; >> ???? >?? 634?? } >> ???? >?? 635?? return NULL; >> ???? >?? 636 } >> ???? >??? ... >> ???? >??? 93 void ThreadTable::lazy_initialize(const ThreadsList >> *threads) { >> ???? >??? 94?? if (!_is_initialized) { >> ???? >??? 95???? { >> ???? >??? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >??? 97?????? if (!_is_initialized) { >> ???? >??? 98???????? create_table(threads->length()); >> ???? >??? 99???????? _is_initialized = true; >> ???? >?? 100?????? } >> ???? >?? 101???? } >> ???? >?? 102???? for (uint i = 0; i < threads->length(); i++) { >> ???? >?? 103?????? JavaThread* thread = threads->thread_at(i); >> ???? >?? 104?????? oop tobj = thread->threadObj(); >> ???? >?? 105?????? if (tobj != NULL && !thread->is_exiting()) { >> ???? >?? 106???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???? >?? 107???????? add_thread(java_tid, thread); >> ???? >?? 108?????? } >> ???? >?? 109???? } >> ???? >?? 110?? } >> ???? >?? 111 } >> ???? > >> ???? > A thread may start exiting right after the checks at the lines >> 626 and 105. >> ???? > So that: >> ???? >?? - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? > >> ???? > Please, note, the lines 626-629 are safe in terms of addition >> to the ThreadTable as they >> ???? > are protected with the Threads_lock. But the returned thread >> still can exit after that. >> ???? > It is interesting what might happen if an exiting thread is >> returned by the >> ???? > ThreadsList::find_JavaThread_from_java_tid (). >> ???? > >> ???? > Does it make sense to develop a test that would cover these >> cases? >> ???? > >> ???? > Thanks, >> ???? > Serguei >> ???? > >> ???? > >> ???? > On 9/16/19 11:18, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on >> the performance (thanks a lot Claes for helping with it!) the >> conclusion was that the impact on the thread startup time is not a >> blocker for this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just >> to remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 >> [3] minus changes in src/hotspot/share/services/management.cpp (that >> were factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code >> that adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that >> may be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue >> if required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c] jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333] JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159] ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In >> case when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time >> the thread table >> ???? >????????? >????? >????? >?? is initialized with the threads >> from the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only >> initializing the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, >> but if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then >> that may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in >> that regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori >> or deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably >> originally suggested in the patch attached to the JBS issue, should >> be used.? It will make >> ???? >????????? >????? >????? > the backporting more complicated,? >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? + def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && >> ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324 JavaThread* current_thread = >> (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? > src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not >> indent the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60 ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its >> very existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> size_log > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? > DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong _tid; >> ???? >????????? >????? >????? 181???? uintx _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces >> before the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183 ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201 ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211 assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" mailto:daniel.daugherty at oracle.com wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with >> an unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about >> the maintaining the thread table when it may never even be queried, >> one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it >> on and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as >> David H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >> save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and >> if the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? > find_JavaThread_from_java_tid() to >> get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> mailto:serguei.spitsyn at oracle.com >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability >> mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot >> c/c++ files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615 JavaThread* java_thread >> = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616 if (java_thread == NULL >> && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? > 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? > 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? > 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? > 620???????? for (uint i = 0; i >> < length(); i++) { >> ???? >????????? >????? >????? >????? > 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? > 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > 623???????????????? >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? > 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? > 625???????????? } >> ???? >????????? >????? >????? >????? > 626???????? } >> ???? >????????? >????? >????? >????? >?? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629 } >> ???? >????????? >????? >????? >????? >?? 630 return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633 oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634 // Ignore the thread if >> it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635 // or is starting to exit. >> ???? >????????? >????? >????? >????? >?? 636 return (tobj != NULL && >> !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? > 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615 JavaThread* java_thread >> = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the >> comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name >> 'is_valid_java_thread' looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? > implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Unfortunately I'm just about >> to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the >> mapping between the thread ids and JavaThread objects and replaces >> the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> >> >> >> >> > From daniel.daugherty at oracle.com Sat Sep 21 01:07:29 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 21:07:29 -0400 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> Message-ID: <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: > Hi Dan, > > Please, find a minor correction below. > > > On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >> Daniil, >> >> Thanks for sticking with this project through the many versions. >> Sorry this review is late... >> >> >> On 9/19/19 8:30 PM, Daniil Titov wrote: >>> Hi David and Serguei, >>> >>> Please review new version of the fix that includes the changes >>> Serguei suggested: >>> ? 1. If racing threads initialize the thread table only one of these >>> threads will populate the table with the threads from the thread list >>> ? 2. The code that adds the thread to the tread table is put inside >>> Threads_lock to ensure that we cannot accidentally add the thread >>> ????? that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> >>> The changes are in ThreadTable::lazy_initialize() method only. >>> >>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>> successfully passed. >>> >>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > . . . >> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ??????? Re: discussion about lazy_initialize() racing with >> ??????????? ThreadsList::find_JavaThread_from_java_tid() >> >> ??????? There's a couple of aspects to these two pieces of code racing >> ??????? with each other and racing with new thread creation. Racing with >> ??????? new thread creation is the easy one: >> >> ????????? If a new thread isn't added to the ThreadTable by >> ????????? ThreadsSMRSupport::add_thread() calling >> ThreadTable::add_thread(), >> ????????? then the point in the future where someone calls >> ????????? find_JavaThread_from_java_tid() will add it to the table >> due to >> ????????? the linear search when ThreadTable::find_thread_by_tid() >> ????????? returns NULL. >> >> ?????? As for multi-threads calling >> ThreadsList::find_JavaThread_from_java_tid() >> ?????? at the same time which results in multi-threads in >> lazy_initialize() >> ?????? at the same time... >> >> ?????? - ThreadTable creation will be linear due to >> ThreadTableCreate_lock. >> ???????? After _is_initialized is set to true, then no more callers to >> ???????? lazy_initialize() will be in the "if (!_is_initialized)" block. >> ?????? - Once the ThreadTable is created, then multi-threads can be >> ???????? executing the for-loop to add their ThreadsList entries to >> ???????? the ThreadTable. > > I guess, there is a confusion here. > The lines 97-101 were recently added into the latest webrev: > > ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ? 94?? if (!_is_initialized) { > ? 95???? { > ? 96?????? MutexLocker ml(ThreadTableCreate_lock); > ? 97?????? if (_is_initialized) { > ? 98???????? // There is no obvious benefits in allowing the thread table > ? 99???????? // being concurently populated during the initalization. > ?100???????? return; > ?101?????? } > > It prevents multi-threads executing the for-loop to add their > ThreadsList entries to the ThreadTable. > Instead, these threads may not find the requested threads in the > ThreadTable and so, will start linear search in their ThreadsList > to add them into the ThreadTable. The ThreadTableCreate_lock is only held from here: ? L95: ??? { ? L96: ????? MutexLocker ml(ThreadTableCreate_lock); to here: ? L103: ????? _is_initialized = true; ? L104: ??? } and the for-loop starts here: ? L105: ??? for (uint i = 0; i < threads->length(); i++) { so the ThreadTable_Create_lock does not linearize the for-loop. I'm quoting webrev.07... which I believe is the latest. Dan > > . . . >> Short version: Thumbs up. >> >> Longer version: I don't think I've spotted anything other than nits >> here. >> Mostly I've just looked for multi-threaded races, proper usage of the >> Thread-SMR stuff, and minimal impact in the case where the new >> ThreadsTable is never needed. >> >> Dan >> >> P.S. >> ThreadTable is a bit of misnomer. What you really have here is >> a ThreadIdTable, but I'm really late to the code review flow >> with that comment... > > Agreed, ThreadIdTable is better name for this table. > > Thanks, > Serguei > > >> >> >>> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> >>> Thank you! >>> --Daniil >> > From daniel.daugherty at oracle.com Sat Sep 21 01:11:27 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 20 Sep 2019 21:11:27 -0400 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: > On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >> Hi Dan, >> >> Please, find a minor correction below. >> >> >> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>> Daniil, >>> >>> Thanks for sticking with this project through the many versions. >>> Sorry this review is late... >>> >>> >>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>> Hi David and Serguei, >>>> >>>> Please review new version of the fix that includes the changes >>>> Serguei suggested: >>>> ? 1. If racing threads initialize the thread table only one of >>>> these threads will populate the table with the threads from the >>>> thread list >>>> ? 2. The code that adds the thread to the tread table is put inside >>>> Threads_lock to ensure that we cannot accidentally add the thread >>>> ????? that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> >>>> The changes are in ThreadTable::lazy_initialize() method only. >>>> >>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>> successfully passed. >>>> >>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> . . . >>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ??????? Re: discussion about lazy_initialize() racing with >>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>> >>> ??????? There's a couple of aspects to these two pieces of code racing >>> ??????? with each other and racing with new thread creation. Racing >>> with >>> ??????? new thread creation is the easy one: >>> >>> ????????? If a new thread isn't added to the ThreadTable by >>> ????????? ThreadsSMRSupport::add_thread() calling >>> ThreadTable::add_thread(), >>> ????????? then the point in the future where someone calls >>> ????????? find_JavaThread_from_java_tid() will add it to the table >>> due to >>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>> ????????? returns NULL. >>> >>> ?????? As for multi-threads calling >>> ThreadsList::find_JavaThread_from_java_tid() >>> ?????? at the same time which results in multi-threads in >>> lazy_initialize() >>> ?????? at the same time... >>> >>> ?????? - ThreadTable creation will be linear due to >>> ThreadTableCreate_lock. >>> ???????? After _is_initialized is set to true, then no more callers to >>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>> block. >>> ?????? - Once the ThreadTable is created, then multi-threads can be >>> ???????? executing the for-loop to add their ThreadsList entries to >>> ???????? the ThreadTable. >> >> I guess, there is a confusion here. >> The lines 97-101 were recently added into the latest webrev: >> >> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ? 94?? if (!_is_initialized) { >> ? 95???? { >> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ? 97?????? if (_is_initialized) { >> ? 98???????? // There is no obvious benefits in allowing the thread >> table >> ? 99???????? // being concurently populated during the initalization. >> ?100???????? return; >> ?101?????? } >> >> It prevents multi-threads executing the for-loop to add their >> ThreadsList entries to the ThreadTable. >> Instead, these threads may not find the requested threads in the >> ThreadTable and so, will start linear search in their ThreadsList >> to add them into the ThreadTable. I should have read your comment more carefully... Yes, you are right about the early cut out on L97-100 causing the for-loop to be skipped by any thread that saw (!_is_initialized) and was blocked on the ThreadTableCreate_lock. So my response below is wrong... Sorry about the noise. Dan > > The ThreadTableCreate_lock is only held from here: > > ? L95: ??? { > ? L96: ????? MutexLocker ml(ThreadTableCreate_lock); > > to here: > > ? L103: ????? _is_initialized = true; > ? L104: ??? } > > and the for-loop starts here: > > ? L105: ??? for (uint i = 0; i < threads->length(); i++) { > > so the ThreadTable_Create_lock does not linearize the > for-loop. > > I'm quoting webrev.07... which I believe is the latest. > > Dan > > >> >> . . . >>> Short version: Thumbs up. >>> >>> Longer version: I don't think I've spotted anything other than nits >>> here. >>> Mostly I've just looked for multi-threaded races, proper usage of the >>> Thread-SMR stuff, and minimal impact in the case where the new >>> ThreadsTable is never needed. >>> >>> Dan >>> >>> P.S. >>> ThreadTable is a bit of misnomer. What you really have here is >>> a ThreadIdTable, but I'm really late to the code review flow >>> with that comment... >> >> Agreed, ThreadIdTable is better name for this table. >> >> Thanks, >> Serguei >> >> >>> >>> >>>> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>>> >>>> Thank you! >>>> --Daniil >>> >> > From david.holmes at oracle.com Sat Sep 21 02:20:18 2019 From: david.holmes at oracle.com (David Holmes) Date: Sat, 21 Sep 2019 12:20:18 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> Message-ID: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Hi Serguei, On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Yes, the Threads_lock is still needed around thread->is_exiting() check > and add_thread(). > >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your snippet >> and then the nested Threads_lock around thread->is_exiting() and >> add_thread(java_tid, thread) lines then it will not work since the rank >> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. > > The ThreadTableCreate_lock is only used in the lazy_initialize() function. > It looks safe to adjust the ThreadTableCreate_lock to fix the above > problem. > Then the approach below will work as needed. > > ?void ThreadTable::lazy_initialize(const ThreadsList *threads) { > ??? if (_is_initialized) { > ????? return; > ??? } > ??? MutexLocker ml(ThreadTableCreate_lock); > ??? if (_is_initialized) { > ????? // There is no obvious benefits in allowing the thread table > ????? // being concurrently populated during the initalization. > ????? return; > ??? } > ??? create_table(threads->length()); > ??? _is_initialized = true; > > ??? for (uint i = 0; i < threads->length(); i++) { > ????? JavaThread* thread = threads->thread_at(i); > ????? oop tobj = thread->threadObj(); > ????? if (tobj != NULL) { > ??????? jlong java_tid = java_lang_Thread::thread_id(tobj); > ??????? MutexLocker ml(Threads_lock); > ??????? if (!thread->is_exiting()) { > ????????? // Must be inside the lock to ensure that we don't add the > thread to the table > ????????? // that has just passed the removal point in > ThreadsSMRSupport::remove_thread() > ????????? add_thread(java_tid, thread); > ??????? } > ????? } > ??? } > ? } I don't like holding a completely unrelated lock, whilst holding the Threads_lock, particularly as it seems unnecessary for correctness. We have to be certain that no M&M operation that holds the Threads_lock can try to initialize the thread table. Testing can only prove the presence of that problem, not the absence. I think Daniil's webrev.07 version is correct and less risky than what you propose. Sorry. David ----- > > If you rename ThreadTable to ThreadIdTable then the ThreadTableCreate_lock > has to be renamed to ThreadIdTableCreate_lock. > > Thanks, > Serguei > > On 9/20/19 8:42 AM, Daniil Titov wrote: >> Hi Serguei, >> >>> void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ????? if (_is_initialized) { >>> ????? return; >> ? >?? } >>> ??? MutexLocker ml(ThreadTableCreate_lock); >> >> If I understood you correctly in the code snippet you sent you meant >> to? use >> Threads_lock, not ThreadTableCreate_lock, right? >> >> The original idea was to do a minimal amount of work while holding the >> lock and >> ? hold the lock for as short period of time as possible to not block >> other threads when it is not necessary. >> >> With the suggested approach no new threads could be started until the >> thread table is created and >> populated with all threads running inside a Java application and in >> case of large app there could be >> thousands of them. >> >> And if we try to use 2 locks,? ThreadTableCreate_lock as in your >> snippet and then the nested Threads_lock around >> thread->is_exiting() and add_thread(java_tid, thread) lines then it >> will not work since the rank of Threads_lock >> is higher than the rank of ThreadTableCreate_lock. >> >> So choosing between blocking new threads from starting and potentially >> allowing >> some other monitoring thread? to do a one-time linear scan I think it >> makes sense to choose the latter. >> >> Thanks! >> >> Best regards, >> Daniil >> >> >> >> From: "serguei.spitsyn at oracle.com" >> Date: Thursday, September 19, 2019 at 10:30 PM >> To: Daniil Titov , Robbin Ehn >> , David Holmes , >> , OpenJDK Serviceability >> , >> "hotspot-runtime-dev at openjdk.java.net" >> , "jmx-dev at openjdk.java.net" >> , Claes Redestad >> Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> >> Hi Daniil, >> >> I think, it is better to grab the thread_lock just once at lazy >> initialization. >> It would look simpler, something, like this would work: >> ?? void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } >> >> Otherwise, concurrent executions of the find_JavaThread_from_java_tid() >> will sometimes do a linear search of threads that are not included yet to >> the ThreadTable from the ThreadsList (which is used for lazy >> initialization). >> Instead, it is better to wait for the lazy_initialization() to complete. >> Thanks, >> Serguei >> >> >> On 9/19/19 17:30, Daniil Titov wrote: >> Hi David and Serguei, >> >> Please review new version of the fix that includes the changes Serguei >> suggested: >> ? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> >> The changes are in ThreadTable::lazy_initialize() method only. >> >> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> --Daniil >> >> ?On 9/18/19, 1:01 AM, mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com wrote: >> >> ???? Hi Daniil, >> ???? On 9/17/19 17:13, Daniil Titov wrote: >> ???? > Hi Serguei, >> ???? > >> ???? > Please find below my answers to the concerns you mentioned in >> the previous email. >> ???? > >> ???? > 1. >> ???? >?? > I have a concern about the checks for thread->is_exiting(). >> ???? >?? > - the lines 632-633 are useless as they do not really >> protect from returning an exiting thread >> ???? >> It is interesting what might happen if an exiting thread is >> returned by the >> ???? >> ThreadsList::find_JavaThread_from_java_tid (). >> ???? >> Does it make sense to develop a test that would cover these >> cases? >> ???? > I agree, it doesn't really provide any protection so it makes >> sense just remove it. >> ???? Now, I'm not that confident about it. :) >> ???? >?? The current implementation >> ???? > find_JavaThread_from_java_tid()? doesn't provide such >> protection as well, since the thread could start exiting >> ???? > immediately after method find_JavaThread_from_java_tid() >> returns, so the assumption is that the callers of >> ???? > find_JavaThread_from_java_tid()? are expecting to deal with >> such threads and? looking on some of them shows that >> ???? > they usually try to retrieve threadObj or a thread statistic >> object and if it is NULL that just do nothing. >> ???? If I understand it correctly, the jt->threadObj() can remain >> non-NULL >> ???? for some time while jt->is_exiting() == true. >> ???? It is not clear how reliable is to use it. >> ???? But this is a pre-existing issue. It is not you who introduced >> it. :) >> ???? So, we can skip it for now. >> ???? But for the record, we may have a source of intermittent issues. >> ???? > I'm not sure we could cover this specific case with the test. >> The window between find_JavaThread_from_java_tid() returns and the caller >> ???? > continues the execution is too small. The window between the >> thread started exiting and removed itself from the thread table is >> very small as well. >> ???? Understand. >> ???? > 2. >> ???? >>?? - the lines 105-108 can result in adding exiting threads >> into the ThreadTable >> ???? >?? I agree, it was missed, we need to wrap this code inside >> Thread_lock in the similar way as it is done >> find_JavaThread_from_java_tid() >> ???? Okay, thanks! >> ???? > 3. >> ???? >> I would suggest to rewrite this fragment in a safe way: >> ???? >>?? 95???? { >> ???? >>?? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>?? 97?????? if (!_is_initialized) { >> ???? >>?? 98???????? create_table(threads->length()); >> ???? >>?? 99???????? _is_initialized = true; >> ???? >> 100?????? } >> ???? >> 101???? } >> ???? >> as: >> ???? >>??????? { >> ???? >>????????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >>????????? if (_is_initialized) { >> ???? >>??????????? return; >> ???? >?? >??????? } >> ???? >?? >??????? create_table(threads->length()); >> ???? >?? >??????? _is_initialized = true; >> ???? >?? >????? } >> ???? > >> ???? > It was an intension to not block? while populating the table >> with the threads from the current thread list. >> ???? > There is no needs to have other threads that call >> find_JavaThread_from_java_tid()? be blocked and waiting for >> ???? >?? it to complete since the requested thread could be not >> present in the thread list that triggers the thread table >> ???? >?? initialization. Plus in case of racing initialization it >> allows threads from not original? thread lists be added to the table >> ???? > and thus avoid the linear scan when these thread are looked up >> for the first time. >> ???? I've replied to David in another email. >> ???? Let's talk once more about it tomorrow. >> ???? > 4. >> ???? >>> The case you have described is exact the reason why we still >> have a code inside >> ???? >>> ThreadsList::find_JavaThread_from_java_tid() method that does >> a linear scan and adds >> ???? >>>??? the requested thread to the thread table if it is not >> there ( lines 614-613 below). >> ???? >> I disagree because it is easy to avoid concurrent ThreadTable >> ???? >> initialization (please, see my separate email). >> ???? >> The reason for this code is to cover a case of late/lazy >> ThreadTable >> ???? >> initialization. >> ???? > David Holmes replied to this in a separate email providing a >> very detailed >> ???? > explanation of the possible cases and how the proposed >> implementation satisfies them. >> ???? Yes. Please, see above. >> ???? Thanks, >> ???? Serguei >> ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > From: mailto:serguei.spitsyn at oracle.com >> mailto:serguei.spitsyn at oracle.com >> ???? > Date: Tuesday, September 17, 2019 at 1:53 AM >> ???? > To: Daniil Titov mailto:daniil.x.titov at oracle.com, Robbin Ehn >> mailto:robbin.ehn at oracle.com, David Holmes >> mailto:david.holmes at oracle.com, mailto:daniel.daugherty at oracle.com, >> OpenJDK Serviceability mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net, Claes >> Redestad mailto:claes.redestad at oracle.com >> ???? > Subject: Re: RFR: 8185005: Improve performance of >> ThreadMXBean.getThreadInfo(long ids[], int maxDepth) >> ???? > >> ???? > Hi Daniil, >> ???? > >> ???? > Thank you for you patience in working on this issue! >> ???? > Also, I like that the current thread related optimizations in >> management.cpp were factored out. >> ???? > It was a good idea to separate them. >> ???? > >> ???? > I have a concern about the checks for thread->is_exiting(). >> ???? > The threads are added to and removed from the ThreadTable under >> protection of Threads_lock. >> ???? > However, the thread->is_exiting() checks are not protected, and >> so, they are racy. >> ???? > >> ???? > There is a couple of such checks to mention: >> ???? >?? 611 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >?? 612?? ThreadTable::lazy_initialize(this); >> ???? >?? 613?? JavaThread* thread = >> ThreadTable::find_thread_by_tid(java_tid); >> ???? >?? 614?? if (thread == NULL) { >> ???? >?? 615???? // If the thread is not found in the table find it >> ???? >?? 616???? // with a linear search and add to the table. >> ???? >?? 617???? for (uint i = 0; i < length(); i++) { >> ???? >?? 618?????? thread = thread_at(i); >> ???? >?? 619?????? oop tobj = thread->threadObj(); >> ???? >?? 620?????? // Ignore the thread if it hasn't run yet, has exited >> ???? >?? 621?????? // or is starting to exit. >> ???? >?? 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? >?? 623???????? MutexLocker ml(Threads_lock); >> ???? >?? 624???????? // Must be inside the lock to ensure that we >> don't add the thread to the table >> ???? >?? 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >?? 626???????? if (!thread->is_exiting()) { >> ???? >?? 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? >?? 628?????????? return thread; >> ???? >?? 629???????? } >> ???? >?? 630?????? } >> ???? >?? 631???? } >> ???? >?? 632?? } else if (!thread->is_exiting()) { >> ???? >?? 633?????? return thread; >> ???? >?? 634?? } >> ???? >?? 635?? return NULL; >> ???? >?? 636 } >> ???? >??? ... >> ???? >??? 93 void ThreadTable::lazy_initialize(const ThreadsList >> *threads) { >> ???? >??? 94?? if (!_is_initialized) { >> ???? >??? 95???? { >> ???? >??? 96?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? >??? 97?????? if (!_is_initialized) { >> ???? >??? 98???????? create_table(threads->length()); >> ???? >??? 99???????? _is_initialized = true; >> ???? >?? 100?????? } >> ???? >?? 101???? } >> ???? >?? 102???? for (uint i = 0; i < threads->length(); i++) { >> ???? >?? 103?????? JavaThread* thread = threads->thread_at(i); >> ???? >?? 104?????? oop tobj = thread->threadObj(); >> ???? >?? 105?????? if (tobj != NULL && !thread->is_exiting()) { >> ???? >?? 106???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???? >?? 107???????? add_thread(java_tid, thread); >> ???? >?? 108?????? } >> ???? >?? 109???? } >> ???? >?? 110?? } >> ???? >?? 111 } >> ???? > >> ???? > A thread may start exiting right after the checks at the lines >> 626 and 105. >> ???? > So that: >> ???? >?? - the lines 632-633 are useless as they do not really protect >> from returning an exiting thread >> ???? >?? - the lines 105-108 can result in adding exiting threads into >> the ThreadTable >> ???? > >> ???? > Please, note, the lines 626-629 are safe in terms of addition >> to the ThreadTable as they >> ???? > are protected with the Threads_lock. But the returned thread >> still can exit after that. >> ???? > It is interesting what might happen if an exiting thread is >> returned by the >> ???? > ThreadsList::find_JavaThread_from_java_tid (). >> ???? > >> ???? > Does it make sense to develop a test that would cover these cases? >> ???? > >> ???? > Thanks, >> ???? > Serguei >> ???? > >> ???? > >> ???? > On 9/16/19 11:18, Daniil Titov wrote: >> ???? > Hello, >> ???? > >> ???? > After investigating with Claes the impact of this change on the >> performance (thanks a lot Claes for helping with it!) the conclusion >> was that the impact on the thread startup time is not a blocker for >> this change. >> ???? > >> ???? > I also measured the memory footprint using Native Memory >> Tracking and results showed around 40 bytes per live thread. >> ???? > >> ???? > Please review a new version of the fix, webrev.06 [1].? Just to >> remind,? webrev.05 was abandoned and webrev.06 [1] is webrev.04 [3] >> minus changes in src/hotspot/share/services/management.cpp (that were >> factored out to a separate issue [4]) and plus a change in >> ThreadsList::find_JavaThread_from_java_tid() method (please, see >> below)? that addresses the problem Robbin found and puts the code that >> adds a new thread to the thread table inside Threads_lock. >> ???? > >> ???? > src/hotspot/share/runtime/threadSMR.cpp >> ???? > >> ???? > 622?????? if (tobj != NULL && java_tid == >> java_lang_Thread::thread_id(tobj)) { >> ???? > 623???????? MutexLocker ml(Threads_lock); >> ???? > 624???????? // Must be inside the lock to ensure that we don't >> add the thread to the table >> ???? > 625???????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > 626???????? if (!thread->is_exiting()) { >> ???? > 627?????????? ThreadTable::add_thread(java_tid, thread); >> ???? > 628?????????? return thread; >> ???? > 629???????? } >> ???? > 630?????? } >> ???? > >> ???? > [1] Webrev:? https://cr.openjdk.java.net/~dtitov/8185005/webrev.06 >> ???? > [2] Bug: https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > [3] https://cr.openjdk.java.net/~dtitov/8185005/webrev.04 >> ???? > [4] https://bugs.openjdk.java.net/browse/JDK-8229391 >> ???? > >> ???? > ?Thank you, >> ???? > Daniil >> ???? > >> ???? > >> ???? > >> ???? >????????? > >> ???? >????????? > ?On 8/4/19, 7:54 PM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? > >> ???? >????????? >????? Hi Daniil, >> ???? >????????? > >> ???? >????????? >????? On 3/08/2019 8:16 am, Daniil Titov wrote: >> ???? >????????? >????? > Hi David, >> ???? >????????? >????? > >> ???? >????????? >????? > Thank you for your detailed review. Please >> review a new version of the fix that includes >> ???? >????????? >????? > the changes you suggested: >> ???? >????????? >????? > - ThreadTableCreate_lock scope is reduced to >> cover the creation of the table only; >> ???? >????????? >????? > - ThreadTableCreate_lock is made >> _safepoint_check_always; >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? > - ServiceThread is no longer responsible for >> the resizing of the thread table, instead, >> ???? >????????? >????? >??? the thread table is changed to grow on >> demand by the thread that is doing the addition; >> ???? >????????? > >> ???? >????????? >????? Okay - I'm happy to get the serviceThread out >> of the picture here. >> ???? >????????? > >> ???? >????????? >????? > - fixed nits and formatting issues. >> ???? >????????? > >> ???? >????????? >????? Okay. >> ???? >????????? > >> ???? >????????? >????? >>> The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >>>?? as Daniel suggested. >> ???? >????????? >????? >> Not sure it's best to combine these, but if >> they are limited to the >> ???? >????????? >????? >> changes in management.cpp only then that may >> be okay. >> ???? >????????? >????? > >> ???? >????????? >????? > The additional optimization for some callers >> of find_JavaThread_from_java_tid() is >> ???? >????????? >????? > limited to management.cpp (plus a new test) >> so I left them in the webrev? but >> ???? >????????? >????? > I also could move it in the separate issue if >> required. >> ???? >????????? > >> ???? >????????? >????? I'd prefer this part of be separated out, but >> won't insist. Let's see if >> ???? >????????? >????? Dan or Serguei have a strong opinion. >> ???? >????????? > >> ???? >????????? >????? >??? > src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? >??? >755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??? > 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >?? >? I think it cleaner/better to just use >> ???? >????????? >????? >?? > jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? >?? > as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >?? > non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? > I had to leave this code unchanged since it >> turned out the threadObj is null >> ???? >????????? >????? > when VM is destroyed: >> ???? >????????? >????? > >> ???? >????????? >????? > V? [libjvm.so+0xe165d7] >> oopDesc::long_field(int) const+0x67 >> ???? >????????? >????? > V? [libjvm.so+0x16e06c6] >> ThreadsSMRSupport::add_thread(JavaThread*)+0x116 >> ???? >????????? >????? > V? [libjvm.so+0x16d1302] >> Threads::add(JavaThread*, bool)+0x82 >> ???? >????????? >????? > V? [libjvm.so+0xef8369] >> attach_current_thread.part.197+0xc9 >> ???? >????????? >????? > V? [libjvm.so+0xec136c]? jni_DestroyJavaVM+0x6c >> ???? >????????? >????? > C? [libjli.so+0x4333]? JavaMain+0x2c3 >> ???? >????????? >????? > C? [libjli.so+0x8159]? ThreadJavaMain+0x9 >> ???? >????????? > >> ???? >????????? >????? This is actually nothing to do with the VM >> being destroyed, but is an >> ???? >????????? >????? issue with JNI_AttachCurrentThread and its >> interaction with the >> ???? >????????? >????? ThreadSMR iterators. The attach process is: >> ???? >????????? >????? - create JavaThread >> ???? >????????? >????? - mark as "is attaching via jni" >> ???? >????????? >????? - add to ThreadsList >> ???? >????????? >????? - create java.lang.Thread object (you can only >> execute Java code after >> ???? >????????? >????? you are attached) >> ???? >????????? >????? - mark as "attach completed" >> ???? >????????? > >> ???? >????????? >????? So while a thread "is attaching" it will be >> seen by the ThreadSMR thread >> ???? >????????? >????? iterator but will have a NULL java.lang.Thread >> object. >> ???? >????????? > >> ???? >????????? >????? We special-case attaching threads in a number >> of places in the VM and I >> ???? >????????? >????? think we should be explicitly doing something >> here to filter out >> ???? >????????? >????? attaching threads, rather than just being >> tolerant of a NULL j.l.Thread >> ???? >????????? >????? object. Specifically in >> ThreadsSMRSupport::add_thread: >> ???? >????????? > >> ???? >????????? >????? if (ThreadTable::is_initialized() && >> !thread->is_attaching_via_jni()) { >> ???? >????????? >???????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >???????? ThreadTable::add_thread(tid, thread); >> ???? >????????? >????? } >> ???? >????????? > >> ???? >????????? >????? Note that in ThreadsSMRSupport::remove_thread >> we can use the same guard, >> ???? >????????? >????? which covers the case the JNI attach >> encountered an error trying to >> ???? >????????? >????? create the j.l.Thread object. >> ???? >????????? > >> ???? >????????? >????? >> src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? >> 71???? static uintx get_hash(Value const& >> value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >> The is_dead parameter still bothers me here. >> I can't make enough sense >> ???? >????????? >????? >> out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >> have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >> should not be trying to extend CHT in this >> context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? > My understanding is that is_dead parameter >> provides a mechanism for >> ???? >????????? >????? > ConcurrentHashtable to remove stale entries >> that were not explicitly >> ???? >????????? >????? > removed by calling >> ConcurrentHashTable::remove() method. >> ???? >????????? >????? > I think that just because in our case we >> don't use this mechanism doesn't >> ???? >????????? >????? > mean we should not use ConcurrentHashTable. >> ???? >????????? > >> ???? >????????? >????? Can you confirm that this usage is okay with >> Robbin Ehn please. He's >> ???? >????????? >????? back from vacation this week. >> ???? >????????? > >> ???? >????????? >????? >> I would still want to see what impact this >> has on thread >> ???? >????????? >????? >> startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? > I run a test that initializes the table by >> calling ThreadMXBean.get getThreadInfo(), >> ???? >????????? >????? > starts some threads as a worm-up, and then >> creates and starts 100,000 threads >> ???? >????????? >????? > (each thread just sleeps for 100 ms). In case >> when the thread table is enabled >> ???? >????????? >????? > 100,000 threads are created and started? for >> about 15200 ms. If the thread table >> ???? >????????? >????? > is off the test takes about 14800 ms. Based >> on this information the enabled >> ???? >????????? >????? > thread table makes the thread startup about >> 2.7% slower. >> ???? >????????? > >> ???? >????????? >????? That doesn't sound very good. I think we may >> need to Claes involved to >> ???? >????????? >????? help investigate overall performance impact here. >> ???? >????????? > >> ???? >????????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.04/ >> ???? >????????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? > >> ???? >????????? >????? No further code comments. >> ???? >????????? > >> ???? >????????? >????? I didn't look at the test in detail. >> ???? >????????? > >> ???? >????????? >????? Thanks, >> ???? >????????? >????? David >> ???? >????????? > >> ???? >????????? >????? > Thanks! >> ???? >????????? >????? > --Daniil >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > ?On 7/29/19, 12:53 AM, "David Holmes" >> mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? > >> ???? >????????? >????? >????? Hi Daniil, >> ???? >????????? >????? > >> ???? >????????? >????? >????? Overall I think this is a reasonable >> approach but I would still like to >> ???? >????????? >????? >????? see some performance and footprint >> numbers, both to verify it fixes the >> ???? >????????? >????? >????? problem reported, and that we are not >> getting penalized elsewhere. >> ???? >????????? >????? > >> ???? >????????? >????? >????? On 25/07/2019 3:21 am, Daniil Titov wrote: >> ???? >????????? >????? >????? > Hi David, Daniel, and Serguei, >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Please review the new version of the >> fix, that makes the thread table initialization on demand and >> ???? >????????? >????? >????? > moves it inside >> ThreadsList::find_JavaThread_from_java_tid(). At the creation time the >> thread table >> ???? >????????? >????? >????? >?? is initialized with the threads from >> the current thread list. We don't want to hold Threads_lock >> ???? >????????? >????? >????? > inside >> find_JavaThread_from_java_tid(),? thus new threads still could be >> created? while the thread >> ???? >????????? >????? >????? > table is being initialized . Such >> threads will be found by the linear search and added to the thread table >> ???? >????????? >????? >????? > later, in >> ThreadsList::find_JavaThread_from_java_tid(). >> ???? >????????? >????? > >> ???? >????????? >????? >????? The initialization allows the created >> but unpopulated, or partially >> ???? >????????? >????? >????? populated, table to be seen by other >> threads - is that your intention? >> ???? >????????? >????? >????? It seems it should be okay as the other >> threads will then race with the >> ???? >????????? >????? >????? initializing thread to add specific >> entries, and this is a concurrent >> ???? >????????? >????? >????? map so that should be functionally >> correct. But if so then I think you >> ???? >????????? >????? >????? can also reduce the scope of the >> ThreadTableCreate_lock so that it >> ???? >????????? >????? >????? covers creation of the table only, not >> the initial population of the table. >> ???? >????????? >????? > >> ???? >????????? >????? >????? I like the approach of only initializing >> the table when needed and using >> ???? >????????? >????? >????? that to control when the >> add/remove-thread code needs to update the >> ???? >????????? >????? >????? table. But I would still want to see >> what impact this has on thread >> ???? >????????? >????? >????? startup cost, both with and without the >> table being initialized. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > The change also includes additional >> optimization for some callers of find_JavaThread_from_java_tid() >> ???? >????????? >????? >????? > as Daniel suggested. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure it's best to combine these, but >> if they are limited to the >> ???? >????????? >????? >????? changes in management.cpp only then that >> may be okay. It helps to be >> ???? >????????? >????? >????? able to focus on the table related >> changes without being distracted by >> ???? >????????? >????? >????? other optimizations. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > That is correct that >> ResolvedMethodTable was used as a blueprint for the thread table, >> however, I tried >> ???? >????????? >????? >????? > to strip it of the all functionality >> that is not required in the thread table case. >> ???? >????????? >????? > >> ???? >????????? >????? >????? The revised version seems better in that >> regard. But I still have a >> ???? >????????? >????? >????? concern, see below. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > We need to have the thread table >> resizable and allow it to grow as the number of threads increases to >> avoid >> ???? >????????? >????? >????? > reserving excessive memory a-priori or >> deteriorating lookup times. The ServiceThread is responsible for >> ???? >????????? >????? >????? > growing the thread table when required. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Yes but why? Why can't this table be >> grown on demand by the thread that >> ???? >????????? >????? >????? is doing the addition? For other tables >> we may have to delegate to the >> ???? >????????? >????? >????? service thread because the current >> thread cannot perform the action, or >> ???? >????????? >????? >????? it doesn't want to perform it at the >> time the need for the resize is >> ???? >????????? >????? >????? detected (e.g. its detected at a >> safepoint and you want the resize to >> ???? >????????? >????? >????? happen later outside the safepoint). >> It's not apparent to me that such >> ???? >????????? >????? >????? restrictions apply here. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > There is no ConcurrentHashTable >> available in Java 8 and for backporting this fix to Java 8 another >> implementation >> ???? >????????? >????? >????? > of the hash table, probably originally >> suggested in the patch attached to the JBS issue, should be used.? It >> will make >> ???? >????????? >????? >????? > the backporting more complicated, >> however, adding a new Implementation of the hash table in Java 14 >> while it >> ???? >????????? >????? >????? > already has ConcurrentHashTable >> doesn't seem? reasonable for me. >> ???? >????????? >????? > >> ???? >????????? >????? >????? Ok. >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Webrev: >> http://cr.openjdk.java.net/~dtitov/8185005/webrev.03 >> ???? >????????? >????? > >> ???? >????????? >????? >????? Some specific code comments: >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? +?? def(ThreadTableCreate_lock?????? , >> PaddedMutex? , special, >> ???? >????????? >????? >????? false, Monitor::_safepoint_check_never); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think this needs to be a >> _safepoint_check_always lock. The table will >> ???? >????????? >????? >????? be created by regular JavaThreads and >> they should (nearly) always be >> ???? >????????? >????? >????? checking for safepoints if they are >> going to block acquiring the lock. >> ???? >????????? >????? >????? And it isn't at all obvious that the >> thread doing the creation can't go >> ???? >????????? >????? >????? to a safepoint whilst this lock is held. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 618?????? JavaThread* thread = >> thread_at(i); >> ???? >????????? >????? > >> ???? >????????? >????? >????? you could reuse the new java_thread >> local you introduced at line 613 and >> ???? >????????? >????? >????? just rename that "new" variable to >> "thread" so you don't have to change >> ???? >????????? >????? >????? all other uses. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 628?? } else if (java_thread != NULL && ... >> ???? >????????? >????? > >> ???? >????????? >????? >????? You don't need to check != NULL here as >> you only get here when >> ???? >????????? >????? >????? java_thread is not NULL. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 755???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? >??????? 926???? jlong tid = >> SharedRuntime::get_java_tid(thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? I think it cleaner/better to just use >> ???? >????????? >????? > >> ???? >????????? >????? >????? jlong tid = >> java_lang_Thread::thread_id(thread->threadObj()); >> ???? >????????? >????? > >> ???? >????????? >????? >????? as we know thread is not NULL, it is a >> JavaThread and it has to have a >> ???? >????????? >????? >????? non-null threadObj. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/management.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >????? 1323???????? if >> (THREAD->is_Java_thread()) { >> ???? >????????? >????? >????? 1324?????????? JavaThread* >> current_thread = (JavaThread*)THREAD; >> ???? >????????? >????? > >> ???? >????????? >????? >????? These calls can only be made on a >> JavaThread so this be simplified to >> ???? >????????? >????? >????? remove the is_Java_thread() call. >> Similarly in other places. >> ???? >????????? >????? > >> ???? >????????? >????? >????? --- >> ???? >????????? >????? > >> ???? >????????? >????? >????? src/hotspot/share/services/threadTable.cpp >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56?? private: >> ???? >????????? >????? >???????? 57???? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? I believe hotspot style is to not indent >> the access modifiers in C++ >> ???? >????????? >????? >????? class declarations, so the above would >> just be: >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 55 class ThreadTableEntry : public >> CHeapObj { >> ???? >????????? >????? >???????? 56 private: >> ???? >????????? >????? >???????? 57?? jlong _tid; >> ???? >????????? >????? > >> ???? >????????? >????? >????? etc. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 60???? ThreadTableEntry(jlong tid, >> JavaThread* java_thread) : >> ???? >????????? >????? >??????? 61 >> _tid(tid),_java_thread(java_thread) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 61 should be indented as it >> continues line 60. >> ???? >????????? >????? > >> ???? >????????? >????? >???????? 67 class ThreadTableConfig : public >> AllStatic { >> ???? >????????? >????? >???????? ... >> ???? >????????? >????? >???????? 71???? static uintx get_hash(Value >> const& value, bool* is_dead) { >> ???? >????????? >????? > >> ???? >????????? >????? >????? The is_dead parameter still bothers me >> here. I can't make enough sense >> ???? >????????? >????? >????? out of the template code in >> ConcurrentHashtable to see why we have to >> ???? >????????? >????? >????? have it, but I'm concerned that its very >> existence means we perhaps >> ???? >????????? >????? >????? should not be trying to extend CHT in >> this context. ?? >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = size_log >> > DefaultThreadTableSizeLog >> ???? >????????? >????? >??????? 116?? ? size_log : >> DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 116 should be indented, though in >> this case I think a better layout >> ???? >????????? >????? >????? would be: >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 115?? size_t start_size_log = >> ???? >????????? >????? >??????? 116?????? size_log > >> DefaultThreadTableSizeLog ? size_log : >> ???? >????????? >????? >????? DefaultThreadTableSizeLog; >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 131 double >> ThreadTable::get_load_factor() { >> ???? >????????? >????? >??????? 132?? return >> (double)_items_count/_current_size; >> ???? >????????? >????? >??????? 133 } >> ???? >????????? >????? > >> ???? >????????? >????? >????? Not sure that is doing what you >> want/expect. It will perform integer >> ???? >????????? >????? >????? division and then cast that whole >> integer to a double. If you want >> ???? >????????? >????? >????? double arithmetic you need: >> ???? >????????? >????? > >> ???? >????????? >????? >????? return >> ((double)_items_count)/_current_size; >> ???? >????????? >????? > >> ???? >????????? >????? >????? 180???? jlong????????? _tid; >> ???? >????????? >????? >????? 181???? uintx???????? _hash; >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: no need for all those spaces before >> the variable name. >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 183???? ThreadTableLookup(jlong tid) >> ???? >????????? >????? >??????? 184???? : _tid(tid), >> _hash(primitive_hash(tid)) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 184 should be indented. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 201???? ThreadGet():_return(NULL) {} >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after : >> ???? >????????? >????? > >> ???? >????????? >????? >??????? 211??? assert(_is_initialized, "Thread >> table is not initialized"); >> ???? >????????? >????? >??????? 212?? _has_work = false; >> ???? >????????? >????? > >> ???? >????????? >????? >????? line 211 is indented one space too far. >> ???? >????????? >????? > >> ???? >????????? >????? >????? 229???? ThreadTableEntry* entry = new >> ThreadTableEntry(tid,java_thread); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? 252?? return >> _local_table->remove(thread,lookup); >> ???? >????????? >????? > >> ???? >????????? >????? >????? Nit: need space after , >> ???? >????????? >????? > >> ???? >????????? >????? >????? Thanks, >> ???? >????????? >????? >????? David >> ???? >????????? >????? >????? ------ >> ???? >????????? >????? > >> ???? >????????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > Thanks! >> ???? >????????? >????? >????? > --Daniil >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > ?On 7/8/19, 3:24 PM, "Daniel D. >> Daugherty" mailto:daniel.daugherty at oracle.com wrote: >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? On 6/29/19 12:06 PM, Daniil Titov >> wrote: >> ???? >????????? >????? >????? >????? > Hi Serguei and David, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Serguei is right, >> ThreadTable::find_thread(java_tid) cannot? return a JavaThread with an >> unmatched java_tid. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Please find a new version of >> the fix that includes the changes Serguei suggested. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Regarding the concern about the >> maintaining the thread table when it may never even be queried, one of >> ???? >????????? >????? >????? >????? > the options could be to add >> ThreadTable ::isEnabled flag, set it to "false" by default, and wrap >> the calls to the thread table >> ???? >????????? >????? >????? >????? > in ThreadsSMRSupport >> add_thread() and remove_thread() methods to check this flag. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > When >> ThreadsList::find_JavaThread_from_java_tid() is called for the first >> time it could check if ThreadTable ::isEnabled >> ???? >????????? >????? >????? >????? > Is on and if not then set it on >> and populate the thread table with all existing threads from the >> thread list. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I have the same concerns as David >> H. about this new ThreadTable. >> ???? >????????? >????? >????? > >> ThreadsList::find_JavaThread_from_java_tid() is only called from code >> ???? >????????? >????? >????? >????? in >> src/hotspot/share/services/management.cpp so I think that table >> ???? >????????? >????? >????? >????? needs to enabled and populated >> only if it is going to be used. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? I've taken a look at the webrev >> below and I see that David has >> ???? >????????? >????? >????? >????? followed up with additional >> comments. Before I do a crawl through >> ???? >????????? >????? >????? >????? code review for this, I would >> like to see the ThreadTable stuff >> ???? >????????? >????? >????? >????? made optional and David's other >> comments addressed. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Another possible optimization is >> for callers of >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to save the calling thread's >> ???? >????????? >????? >????? >????? tid value before they loop and if >> the current tid == saved_tid >> ???? >????????? >????? >????? >????? then use the current JavaThread* >> instead of calling >> ???? >????????? >????? >????? >????? find_JavaThread_from_java_tid() >> to get the JavaThread*. >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? Dan >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.02/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > --Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > From: >> mailto:serguei.spitsyn at oracle.com >> ???? >????????? >????? >????? >????? > Organization: Oracle Corporation >> ???? >????????? >????? >????? >????? > Date: Friday, June 28, 2019 at >> 7:56 PM >> ???? >????????? >????? >????? >????? > To: Daniil Titov >> mailto:daniil.x.titov at oracle.com, OpenJDK Serviceability >> mailto:serviceability-dev at openjdk.java.net, >> mailto:hotspot-runtime-dev at openjdk.java.net >> mailto:hotspot-runtime-dev at openjdk.java.net, >> mailto:jmx-dev at openjdk.java.net mailto:jmx-dev at openjdk.java.net >> ???? >????????? >????? >????? >????? > Subject: Re: RFR: 8185005: >> Improve performance of ThreadMXBean.getThreadInfo(long ids[], int >> maxDepth) >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > I have several quick comments. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The indent in the hotspot c/c++ >> files has to be 2, not 4. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/src/hotspot/share/runtime/threadSMR.cpp.frames.html >> >> ???? >????????? >????? >????? >????? > 614 JavaThread* >> ThreadsList::find_JavaThread_from_java_tid(jlong java_tid) const { >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? >?? 616???? if (java_thread == >> NULL && java_tid == PMIMORDIAL_JAVA_TID) { >> ???? >????????? >????? >????? >????? >?? 617???????? // >> ThreadsSMRSupport::add_thread() is not called for the primordial >> ???? >????????? >????? >????? >????? >?? 618???????? // thread. Thus, >> we find this thread with a linear search and add it >> ???? >????????? >????? >????? >????? >?? 619???????? // to the thread >> table. >> ???? >????????? >????? >????? >????? >?? 620???????? for (uint i = 0; >> i < length(); i++) { >> ???? >????????? >????? >????? >????? >?? 621???????????? JavaThread* >> thread = thread_at(i); >> ???? >????????? >????? >????? >????? >?? 622???????????? if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? >?? 623 >> ThreadTable::add_thread(java_tid, thread); >> ???? >????????? >????? >????? >????? >?? 624???????????????? return >> thread; >> ???? >????????? >????? >????? >????? >?? 625???????????? } >> ???? >????????? >????? >????? >????? >?? 626???????? } >> ???? >????????? >????? >????? >????? >?? 627???? } else if >> (java_thread != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? >?? 628???????? return java_thread; >> ???? >????????? >????? >????? >????? >?? 629???? } >> ???? >????????? >????? >????? >????? >?? 630???? return NULL; >> ???? >????????? >????? >????? >????? >?? 631 } >> ???? >????????? >????? >????? >????? >?? 632 bool >> ThreadsList::is_valid_java_thread(jlong java_tid, JavaThread* >> java_thread) { >> ???? >????????? >????? >????? >????? >?? 633???? oop tobj = >> java_thread->threadObj(); >> ???? >????????? >????? >????? >????? >?? 634???? // Ignore the thread >> if it hasn't run yet, has exited >> ???? >????????? >????? >????? >????? >?? 635???? // or is starting to >> exit. >> ???? >????????? >????? >????? >????? >?? 636???? return (tobj != NULL >> && !java_thread->is_exiting() && >> ???? >????????? >????? >????? >????? >?? 637???????????? java_tid == >> java_lang_Thread::thread_id(tobj)); >> ???? >????????? >????? >????? >????? >?? 638 } >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >?? 615???? JavaThread* >> java_thread = ThreadTable::find_thread(java_tid); >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >??? I'd suggest to rename >> find_thread() to find_thread_by_tid(). >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > A space is missed after the comma: >> ???? >????????? >????? >????? >????? >??? 622 if >> (is_valid_java_thread(java_tid,thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > An empty line is needed before >> L632. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The name 'is_valid_java_thread' >> looks wrong (or confusing) to me. >> ???? >????????? >????? >????? >????? > Something like >> 'is_alive_java_thread_with_tid()' would be better. >> ???? >????????? >????? >????? >????? > It'd better to list parameters >> in the opposite order. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > The call to >> is_valid_java_thread() is confusing: >> ???? >????????? >????? >????? >????? >???? 627 } else if (java_thread >> != NULL && is_valid_java_thread(java_tid, java_thread)) { >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Why would the call >> ThreadTable::find_thread(java_tid) return a JavaThread with an >> unmatched java_tid? >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks, >> ???? >????????? >????? >????? >????? > Serguei >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19, 9:40 PM, "David >> Holmes" mailto:david.holmes at oracle.com wrote: >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Hi Daniil, >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? The definition and use of >> this hashtable (yet another hashtable >> ???? >????????? >????? >????? >????? >????? implementation!) will need >> careful examination. We have to be concerned >> ???? >????????? >????? >????? >????? >????? about the cost of >> maintaining it when it may never even be queried. You >> ???? >????????? >????? >????? >????? >????? would need to look at >> footprint cost and performance impact. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Unfortunately I'm just >> about to board a plane and will be out for the >> ???? >????????? >????? >????? >????? >????? next few days. I will try >> to look at this asap next week, but we will >> ???? >????????? >????? >????? >????? >????? need a lot more data on it. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? >????? Thanks, >> ???? >????????? >????? >????? >????? >????? David >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > On 6/28/19 3:31 PM, Daniil >> Titov wrote: >> ???? >????????? >????? >????? >????? > Please review the change that >> improves performance of ThreadMXBean MXBean methods returning the >> ???? >????????? >????? >????? >????? > information for specific >> threads. The change introduces the thread table that uses >> ConcurrentHashTable >> ???? >????????? >????? >????? >????? > to store one-to-one the mapping >> between the thread ids and JavaThread objects and replaces the linear >> ???? >????????? >????? >????? >????? > search over the thread list in >> ThreadsList::find_JavaThread_from_java_tid(jlong tid) method with the >> lookup >> ???? >????????? >????? >????? >????? > in the thread table. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Testing: Mach5 tier1,tier2 and >> tier3 tests successfully passed. >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.01/ >> ???? >????????? >????? >????? >????? > Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Thanks! >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > Best regards, >> ???? >????????? >????? >????? >????? > Daniil >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? >????? > >> ???? >????????? > >> ???? >????????? > >> ???? >????????? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> ???? > >> >> >> >> >> >> > From serguei.spitsyn at oracle.com Sat Sep 21 06:11:34 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:11:34 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: Hi Dan, On 9/20/19 18:11, Daniel D. Daugherty wrote: > On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: >> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Dan, >>> >>> Please, find a minor correction below. >>> >>> >>> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>>> Daniil, >>>> >>>> Thanks for sticking with this project through the many versions. >>>> Sorry this review is late... >>>> >>>> >>>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>>> Hi David and Serguei, >>>>> >>>>> Please review new version of the fix that includes the changes >>>>> Serguei suggested: >>>>> ? 1. If racing threads initialize the thread table only one of >>>>> these threads will populate the table with the threads from the >>>>> thread list >>>>> ? 2. The code that adds the thread to the tread table is put >>>>> inside Threads_lock to ensure that we cannot accidentally add the >>>>> thread >>>>> ????? that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> >>>>> The changes are in ThreadTable::lazy_initialize() method only. >>>>> >>>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>>> successfully passed. >>>>> >>>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> . . . >>>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ??????? Re: discussion about lazy_initialize() racing with >>>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>>> >>>> ??????? There's a couple of aspects to these two pieces of code racing >>>> ??????? with each other and racing with new thread creation. Racing >>>> with >>>> ??????? new thread creation is the easy one: >>>> >>>> ????????? If a new thread isn't added to the ThreadTable by >>>> ????????? ThreadsSMRSupport::add_thread() calling >>>> ThreadTable::add_thread(), >>>> ????????? then the point in the future where someone calls >>>> ????????? find_JavaThread_from_java_tid() will add it to the table >>>> due to >>>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>>> ????????? returns NULL. >>>> >>>> ?????? As for multi-threads calling >>>> ThreadsList::find_JavaThread_from_java_tid() >>>> ?????? at the same time which results in multi-threads in >>>> lazy_initialize() >>>> ?????? at the same time... >>>> >>>> ?????? - ThreadTable creation will be linear due to >>>> ThreadTableCreate_lock. >>>> ???????? After _is_initialized is set to true, then no more callers to >>>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>>> block. >>>> ?????? - Once the ThreadTable is created, then multi-threads can be >>>> ???????? executing the for-loop to add their ThreadsList entries to >>>> ???????? the ThreadTable. >>> >>> I guess, there is a confusion here. >>> The lines 97-101 were recently added into the latest webrev: >>> >>> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ? 94?? if (!_is_initialized) { >>> ? 95???? { >>> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >>> ? 97?????? if (_is_initialized) { >>> ? 98???????? // There is no obvious benefits in allowing the thread >>> table >>> ? 99???????? // being concurently populated during the initalization. >>> ?100???????? return; >>> ?101?????? } >>> >>> It prevents multi-threads executing the for-loop to add their >>> ThreadsList entries to the ThreadTable. >>> Instead, these threads may not find the requested threads in the >>> ThreadTable and so, will start linear search in their ThreadsList >>> to add them into the ThreadTable. > > I should have read your comment more carefully... > > Yes, you are right about the early cut out on L97-100 causing > the for-loop to be skipped by any thread that saw (!_is_initialized) > and was blocked on the ThreadTableCreate_lock. > > So my response below is wrong... > > Sorry about the noise. It is hard to follow all moves on this email thread. :) Thanks, Serguei From serguei.spitsyn at oracle.com Sat Sep 21 06:21:16 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:21:16 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: Hi David, I do not want to insist in the area (Threads_lock use) where you have much more expertise. Also, I agreed that Daniil's webrev.07 version is correct and less risky. The only my concern was about possible performance impact of linear searches from the ThreadsList::find_JavaThread_from_java_tid() calls that can be executed concurrently with the ThreadTable::lazy_initialize() if the ThreadsList is big. But it should not be that big issue. So, I'm okay with the webrev.07 modulo ThreadTable renaming. Thanks, Serguei On 9/20/19 19:20, David Holmes wrote: > Hi Serguei, > > On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Yes, the Threads_lock is still needed around thread->is_exiting() >> check and add_thread(). >> >>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>> and then the nested Threads_lock around thread->is_exiting() and >>> add_thread(java_tid, thread) lines then it will not work since the rank >>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >> >> The ThreadTableCreate_lock is only used in the lazy_initialize() >> function. >> It looks safe to adjust the ThreadTableCreate_lock to fix the above >> problem. >> Then the approach below will work as needed. >> >> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? MutexLocker ml(Threads_lock); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } > > I don't like holding a completely unrelated lock, whilst holding the > Threads_lock, particularly as it seems unnecessary for correctness. We > have to be certain that no M&M operation that holds the Threads_lock > can try to initialize the thread table. Testing can only prove the > presence of that problem, not the absence. > > I think Daniil's webrev.07 version is correct and less risky than what > you propose. Sorry. > > David > ----- From serguei.spitsyn at oracle.com Sat Sep 21 06:24:42 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:24:42 -0700 (PDT) Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <7d689838-4e2e-e5b2-b4b3-713e6c82a187@oracle.com> <8ae95ef3-27f0-7cbc-1416-66ce66187120@oracle.com> Message-ID: <7e07188c-4190-cc59-57b6-e297f4197f80@oracle.com> Hi Dan, On 9/20/19 18:11, Daniel D. Daugherty wrote: > On 9/20/19 9:07 PM, Daniel D. Daugherty wrote: >> On 9/20/19 7:15 PM, serguei.spitsyn at oracle.com wrote: >>> Hi Dan, >>> >>> Please, find a minor correction below. >>> >>> >>> On 9/20/19 2:59 PM, Daniel D. Daugherty wrote: >>>> Daniil, >>>> >>>> Thanks for sticking with this project through the many versions. >>>> Sorry this review is late... >>>> >>>> >>>> On 9/19/19 8:30 PM, Daniil Titov wrote: >>>>> Hi David and Serguei, >>>>> >>>>> Please review new version of the fix that includes the changes >>>>> Serguei suggested: >>>>> ? 1. If racing threads initialize the thread table only one of >>>>> these threads will populate the table with the threads from the >>>>> thread list >>>>> ? 2. The code that adds the thread to the tread table is put >>>>> inside Threads_lock to ensure that we cannot accidentally add the >>>>> thread >>>>> ????? that has just passed the removal point in >>>>> ThreadsSMRSupport::remove_thread() >>>>> >>>>> The changes are in ThreadTable::lazy_initialize() method only. >>>>> >>>>> Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>>>> successfully passed. >>>>> >>>>> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> . . . >>>> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ??????? Re: discussion about lazy_initialize() racing with >>>> ??????????? ThreadsList::find_JavaThread_from_java_tid() >>>> >>>> ??????? There's a couple of aspects to these two pieces of code racing >>>> ??????? with each other and racing with new thread creation. Racing >>>> with >>>> ??????? new thread creation is the easy one: >>>> >>>> ????????? If a new thread isn't added to the ThreadTable by >>>> ????????? ThreadsSMRSupport::add_thread() calling >>>> ThreadTable::add_thread(), >>>> ????????? then the point in the future where someone calls >>>> ????????? find_JavaThread_from_java_tid() will add it to the table >>>> due to >>>> ????????? the linear search when ThreadTable::find_thread_by_tid() >>>> ????????? returns NULL. >>>> >>>> ?????? As for multi-threads calling >>>> ThreadsList::find_JavaThread_from_java_tid() >>>> ?????? at the same time which results in multi-threads in >>>> lazy_initialize() >>>> ?????? at the same time... >>>> >>>> ?????? - ThreadTable creation will be linear due to >>>> ThreadTableCreate_lock. >>>> ???????? After _is_initialized is set to true, then no more callers to >>>> ???????? lazy_initialize() will be in the "if (!_is_initialized)" >>>> block. >>>> ?????? - Once the ThreadTable is created, then multi-threads can be >>>> ???????? executing the for-loop to add their ThreadsList entries to >>>> ???????? the ThreadTable. >>> >>> I guess, there is a confusion here. >>> The lines 97-101 were recently added into the latest webrev: >>> >>> ? 93 void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ? 94?? if (!_is_initialized) { >>> ? 95???? { >>> ? 96?????? MutexLocker ml(ThreadTableCreate_lock); >>> ? 97?????? if (_is_initialized) { >>> ? 98???????? // There is no obvious benefits in allowing the thread >>> table >>> ? 99???????? // being concurently populated during the initalization. >>> ?100???????? return; >>> ?101?????? } >>> >>> It prevents multi-threads executing the for-loop to add their >>> ThreadsList entries to the ThreadTable. >>> Instead, these threads may not find the requested threads in the >>> ThreadTable and so, will start linear search in their ThreadsList >>> to add them into the ThreadTable. > > I should have read your comment more carefully... > > Yes, you are right about the early cut out on L97-100 causing > the for-loop to be skipped by any thread that saw (!_is_initialized) > and was blocked on the ThreadTableCreate_lock. > > So my response below is wrong... > > Sorry about the noise. It is hard to follow all moves on this email thread. :) Thanks, Serguei > > Dan From serguei.spitsyn at oracle.com Sat Sep 21 06:26:39 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:26:39 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: Hi David, I do not want to insist in the area (Threads_lock use) where you have much more expertise. Also, I agreed that Daniil's webrev.07 version is correct and less risky. The only my concern was about possible performance impact of linear searches from the ThreadsList::find_JavaThread_from_java_tid() calls that can be executed concurrently with the ThreadTable::lazy_initialize() when the ThreadsList is big. But it should not be that big issue. So, I'm okay with the webrev.07 modulo ThreadTable renaming. Thanks, Serguei On 9/20/19 19:20, David Holmes wrote: > Hi Serguei, > > On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Yes, the Threads_lock is still needed around thread->is_exiting() >> check and add_thread(). >> >>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>> and then the nested Threads_lock around thread->is_exiting() and >>> add_thread(java_tid, thread) lines then it will not work since the rank >>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >> >> The ThreadTableCreate_lock is only used in the lazy_initialize() >> function. >> It looks safe to adjust the ThreadTableCreate_lock to fix the above >> problem. >> Then the approach below will work as needed. >> >> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ???? if (_is_initialized) { >> ?????? return; >> ???? } >> ???? MutexLocker ml(ThreadTableCreate_lock); >> ???? if (_is_initialized) { >> ?????? // There is no obvious benefits in allowing the thread table >> ?????? // being concurrently populated during the initalization. >> ?????? return; >> ???? } >> ???? create_table(threads->length()); >> ???? _is_initialized = true; >> >> ???? for (uint i = 0; i < threads->length(); i++) { >> ?????? JavaThread* thread = threads->thread_at(i); >> ?????? oop tobj = thread->threadObj(); >> ?????? if (tobj != NULL) { >> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >> ???????? MutexLocker ml(Threads_lock); >> ???????? if (!thread->is_exiting()) { >> ?????????? // Must be inside the lock to ensure that we don't add the >> thread to the table >> ?????????? // that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ?????????? add_thread(java_tid, thread); >> ???????? } >> ?????? } >> ???? } >> ?? } > > I don't like holding a completely unrelated lock, whilst holding the > Threads_lock, particularly as it seems unnecessary for correctness. We > have to be certain that no M&M operation that holds the Threads_lock > can try to initialize the thread table. Testing can only prove the > presence of that problem, not the absence. > > I think Daniil's webrev.07 version is correct and less risky than what > you propose. Sorry. > > David > ----- From david.holmes at oracle.com Sat Sep 21 06:53:09 2019 From: david.holmes at oracle.com (David Holmes) Date: Sat, 21 Sep 2019 16:53:09 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> Message-ID: <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> On 21/09/2019 4:21 pm, serguei.spitsyn at oracle.com wrote: > Hi David, > > I do not want to insist in the area (Threads_lock use) where you have > much more expertise. It's more conservatism than expertise :) I had a look through and there are not too many usages of the Threads_lock in the M&M code and they do appear safe. But I still prefer on principle to not hold nested locks if it is not necessary. > Also, I agreed that Daniil's webrev.07 version is correct and less risky. > The only my concern was about possible performance impact of linear > searches > from the ThreadsList::find_JavaThread_from_java_tid() calls that can be > executed > concurrently with the ThreadTable::lazy_initialize() if the ThreadsList > is big. > But it should not be that big issue. I think we can waste a lot of time wondering about what kinds of scenarios perform better or worse with which variant of the code. You will always be able to create a microbenchmark for which a particular code pattern works best. So I'd say stick to the simplest and least risky code and worry about performance issues like this if, or when, they turn up. > So, I'm okay with the webrev.07 modulo ThreadTable renaming. Okay. Thanks, David > Thanks, > Serguei > > > On 9/20/19 19:20, David Holmes wrote: >> Hi Serguei, >> >> On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >>> Hi Daniil, >>> >>> Yes, the Threads_lock is still needed around thread->is_exiting() >>> check and add_thread(). >>> >>>> And if we try to use 2 locks, ThreadTableCreate_lock as in your snippet >>>> and then the nested Threads_lock around thread->is_exiting() and >>>> add_thread(java_tid, thread) lines then it will not work since the rank >>>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >>> >>> The ThreadTableCreate_lock is only used in the lazy_initialize() >>> function. >>> It looks safe to adjust the ThreadTableCreate_lock to fix the above >>> problem. >>> Then the approach below will work as needed. >>> >>> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>> ???? if (_is_initialized) { >>> ?????? return; >>> ???? } >>> ???? MutexLocker ml(ThreadTableCreate_lock); >>> ???? if (_is_initialized) { >>> ?????? // There is no obvious benefits in allowing the thread table >>> ?????? // being concurrently populated during the initalization. >>> ?????? return; >>> ???? } >>> ???? create_table(threads->length()); >>> ???? _is_initialized = true; >>> >>> ???? for (uint i = 0; i < threads->length(); i++) { >>> ?????? JavaThread* thread = threads->thread_at(i); >>> ?????? oop tobj = thread->threadObj(); >>> ?????? if (tobj != NULL) { >>> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >>> ???????? MutexLocker ml(Threads_lock); >>> ???????? if (!thread->is_exiting()) { >>> ?????????? // Must be inside the lock to ensure that we don't add the >>> thread to the table >>> ?????????? // that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> ?????????? add_thread(java_tid, thread); >>> ???????? } >>> ?????? } >>> ???? } >>> ?? } >> >> I don't like holding a completely unrelated lock, whilst holding the >> Threads_lock, particularly as it seems unnecessary for correctness. We >> have to be certain that no M&M operation that holds the Threads_lock >> can try to initialize the thread table. Testing can only prove the >> presence of that problem, not the absence. >> >> I think Daniil's webrev.07 version is correct and less risky than what >> you propose. Sorry. >> >> David >> ----- From serguei.spitsyn at oracle.com Sat Sep 21 06:59:45 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 20 Sep 2019 23:59:45 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <5249166C-BB3F-4D21-962F-CF627EDF774E@oracle.com> <0612644f-40a2-d2a7-b3d3-3b81aa3c6917@oracle.com> <94203f7e-a8f5-2c8f-b05e-7dfab8f723a5@oracle.com> <48d04abd-8761-1ea4-8202-3dcd5c246b5d@oracle.com> Message-ID: On 9/20/19 23:53, David Holmes wrote: > On 21/09/2019 4:21 pm, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> I do not want to insist in the area (Threads_lock use) where you have >> much more expertise. > > It's more conservatism than expertise :) I had a look through and > there are not too many usages of the Threads_lock in the M&M code and > they do appear safe. I've figured the same. > But I still prefer on principle to not hold nested locks if it is not > necessary. Simplicity and conservatism rule! :) >> Also, I agreed that Daniil's webrev.07 version is correct and less >> risky. >> The only my concern was about possible performance impact of linear >> searches >> from the ThreadsList::find_JavaThread_from_java_tid() calls that can >> be executed >> concurrently with the ThreadTable::lazy_initialize() if the >> ThreadsList is big. >> But it should not be that big issue. > > I think we can waste a lot of time wondering about what kinds of > scenarios perform better or worse with which variant of the code. You > will always be able to create a microbenchmark for which a particular > code pattern works best. So I'd say stick to the simplest and least > risky code and worry about performance issues like this if, or when, > they turn up. Agreed. It is my usual approach too (which I've just tried to break). :) Thanks, Serguei >> So, I'm okay with the webrev.07 modulo ThreadTable renaming. > > Okay. > > Thanks, > David > >> Thanks, >> Serguei >> >> >> On 9/20/19 19:20, David Holmes wrote: >>> Hi Serguei, >>> >>> On 21/09/2019 9:50 am, serguei.spitsyn at oracle.com wrote: >>>> Hi Daniil, >>>> >>>> Yes, the Threads_lock is still needed around thread->is_exiting() >>>> check and add_thread(). >>>> >>>>> And if we try to use 2 locks, ThreadTableCreate_lock as in your >>>>> snippet >>>>> and then the nested Threads_lock around thread->is_exiting() and >>>>> add_thread(java_tid, thread) lines then it will not work since the >>>>> rank >>>>> of Threads_lock? is higher than the rank of ThreadTableCreate_lock. >>>> >>>> The ThreadTableCreate_lock is only used in the lazy_initialize() >>>> function. >>>> It looks safe to adjust the ThreadTableCreate_lock to fix the above >>>> problem. >>>> Then the approach below will work as needed. >>>> >>>> ??void ThreadTable::lazy_initialize(const ThreadsList *threads) { >>>> ???? if (_is_initialized) { >>>> ?????? return; >>>> ???? } >>>> ???? MutexLocker ml(ThreadTableCreate_lock); >>>> ???? if (_is_initialized) { >>>> ?????? // There is no obvious benefits in allowing the thread table >>>> ?????? // being concurrently populated during the initalization. >>>> ?????? return; >>>> ???? } >>>> ???? create_table(threads->length()); >>>> ???? _is_initialized = true; >>>> >>>> ???? for (uint i = 0; i < threads->length(); i++) { >>>> ?????? JavaThread* thread = threads->thread_at(i); >>>> ?????? oop tobj = thread->threadObj(); >>>> ?????? if (tobj != NULL) { >>>> ???????? jlong java_tid = java_lang_Thread::thread_id(tobj); >>>> ???????? MutexLocker ml(Threads_lock); >>>> ???????? if (!thread->is_exiting()) { >>>> ?????????? // Must be inside the lock to ensure that we don't add >>>> the thread to the table >>>> ?????????? // that has just passed the removal point in >>>> ThreadsSMRSupport::remove_thread() >>>> ?????????? add_thread(java_tid, thread); >>>> ???????? } >>>> ?????? } >>>> ???? } >>>> ?? } >>> >>> I don't like holding a completely unrelated lock, whilst holding the >>> Threads_lock, particularly as it seems unnecessary for correctness. >>> We have to be certain that no M&M operation that holds the >>> Threads_lock can try to initialize the thread table. Testing can >>> only prove the presence of that problem, not the absence. >>> >>> I think Daniil's webrev.07 version is correct and less risky than >>> what you propose. Sorry. >>> >>> David >>> ----- From daniil.x.titov at oracle.com Tue Sep 24 16:36:49 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 24 Sep 2019 09:36:49 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> Message-ID: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Hi Daniel, David and Serguei, Please review a new version of the fix (webrev.08) that as Daniel suggested renames ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and corrects formatting issues. There are no other changes in this webrev.08 comparing to the previous version webrev.07. Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 Thank you! Best regards, Daniil ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: Daniil, Thanks for sticking with this project through the many versions. Sorry this review is late... On 9/19/19 8:30 PM, Daniil Titov wrote: > Hi David and Serguei, > > Please review new version of the fix that includes the changes Serguei suggested: > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > The changes are in ThreadTable::lazy_initialize() method only. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ src/hotspot/share/runtime/mutexLocker.hpp No comments. src/hotspot/share/runtime/mutexLocker.cpp No comments. src/hotspot/share/runtime/threadSMR.cpp L623: MutexLocker ml(Threads_lock); L626: if (!thread->is_exiting()) { Re: discussion about is_exiting() The header comment is pretty clear: src/hotspot/share/runtime/thread.hpp: // thread has called JavaThread::exit() or is terminated bool is_exiting() const; is_exiting() might become true right after you have called it, but its purpose is to ask the question and not prevent the condition from becoming true. As David said, you should consider it an optimization. If you happen to see the condition is true, then you know that the JavaThread isn't going to be around much longer and should act accordingly. The is_exiting() implementation is: inline bool JavaThread::is_exiting() const { // Use load-acquire so that setting of _terminated by // JavaThread::exit() is seen more quickly. TerminatedTypes l_terminated = (TerminatedTypes) OrderAccess::load_acquire((volatile jint *) &_terminated); return l_terminated == _thread_exiting || check_is_terminated(l_terminated); } and it depends on the JavaThread's _terminated field value. // JavaThread termination support enum TerminatedTypes { _not_terminated = 0xDEAD - 2, _thread_exiting, // JavaThread::exit() has been called for this thread _thread_terminated, // JavaThread is removed from thread list _vm_exited // JavaThread is still executing native code, but VM is terminated // only VM_Exit can set _vm_exited }; so the JavaThread's _terminated field can get set to _thread_exiting independent of the Threads_lock, but it can't get set to _thread_terminated without the Threads_lock. So by grabbing the Threads_lock on L623, you make sure that ThreadTable::add_thread(java_tid, thread) does not add a JavaThread that's not on the ThreadsList. It might still become is_exiting() == true right after your L626 if (!thread->is_exiting()) { but it will still be on the main ThreadsList. And that means that when the JavaThread is removed from the main ThreadsList, you'll still call: L931: ThreadTable::remove_thread(tid); L624: // Must be inside the lock to ensure that we don't add the thread to the table typo: s/the thread/a thread/ L633: return thread; nit - L633 - indented too far (should be 2 spaces) src/hotspot/share/services/threadTable.hpp L42: static void lazy_initialize(const ThreadsList *threads); nit - put space between '*' the variable: static void lazy_initialize(const ThreadsList* threads); like you do in your other decls. L45: // Lookup and inserts Perhaps: // Lookup and list management L60-61 - nit - please delete these blank lines. src/hotspot/share/services/threadTable.cpp L28: #include "runtime/timerTrace.hpp" nit - This should be after threadSMR.hpp... (alpha sorted order) L39: static const size_t DefaultThreadTableSizeLog = 8; nit - your other 'static const' are not CamelCase. Why is this one? L45: static ThreadTableHash* volatile _local_table = NULL; L50: static volatile size_t _current_size = 0; L51: static volatile size_t _items_count = 0; nit - can you group the file statics together? (up with L41). L60: _tid(tid),_java_thread(java_thread) {} nit - space after ',' L62 jlong tid() const { return _tid;} L63 JavaThread* thread() const {return _java_thread;} nit - space before '}' nit - space after '{' on L63. L70: static uintx get_hash(Value const& value, bool* is_dead) { Parameter 'is_dead' is not used. L74: static void* allocate_node(size_t size, Value const& value) { Parameter 'value' is not used. L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { Re: discussion about lazy_initialize() racing with ThreadsList::find_JavaThread_from_java_tid() There's a couple of aspects to these two pieces of code racing with each other and racing with new thread creation. Racing with new thread creation is the easy one: If a new thread isn't added to the ThreadTable by ThreadsSMRSupport::add_thread() calling ThreadTable::add_thread(), then the point in the future where someone calls find_JavaThread_from_java_tid() will add it to the table due to the linear search when ThreadTable::find_thread_by_tid() returns NULL. As for multi-threads calling ThreadsList::find_JavaThread_from_java_tid() at the same time which results in multi-threads in lazy_initialize() at the same time... - ThreadTable creation will be linear due to ThreadTableCreate_lock. After _is_initialized is set to true, then no more callers to lazy_initialize() will be in the "if (!_is_initialized)" block. - Once the ThreadTable is created, then multi-threads can be executing the for-loop to add their ThreadsList entries to the ThreadTable. There will be a bit of Threads_lock contention as each of the multi-threads tries to add their entries and there will be some wasted work since the multi-threads will likely have similar ThreadLists. Of course, once _is_initialized is set to true, then any caller to lazy_initialize() will return quickly and ThreadsList::find_JavaThread_from_java_tid() will call ThreadTable::find_thread_by_tid(). If the target java_tid isn't found, then we do the linear search thing here and add the the entry if we find a match in our current ThreadsList. Since we're only adding the one here, we only contend for the Threads_lock here if we find it. If ThreadsList::find_JavaThread_from_java_tid() is called with a target java_tid for a JavaThread that was created after the ThreadsList object that the caller has in hand for the find_JavaThread_from_java_tid() call, then, of course, that target 'java_tid' won't be found because the JavaThread was added the main ThreadsList _after_ the ThreadsList object was created by the caller. Of course, you have to ask where the target java_tid value came from since the JavaThread wasn't around when the ThreadsList::find_JavaThread_from_java_tid() call was made with that target java_tid value... L99: // being concurently populated during the initalization. Typos? Perhaps: // to be concurrently populated during initialization. But I think those two comment lines are more appropriate above this line: L96: MutexLocker ml(ThreadTableCreate_lock); L112: // Must be inside the lock to ensure that we don't add the thread to the table typo: s/the thread/a thread/ L141: return ((double)_items_count)/_current_size; nit - need spaces around '/'. L177: bool equals(ThreadTableEntry **value, bool* is_dead) { nit - put space between '**' the variable: bool equals(ThreadTableEntry** value, Parameter 'is_dead' is not used. L214: while(true) { nit - space before '('. Short version: Thumbs up. Longer version: I don't think I've spotted anything other than nits here. Mostly I've just looked for multi-threaded races, proper usage of the Thread-SMR stuff, and minimal impact in the case where the new ThreadsTable is never needed. Dan P.S. ThreadTable is a bit of misnomer. What you really have here is a ThreadIdTable, but I'm really late to the code review flow with that comment... > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > --Daniil From serguei.spitsyn at oracle.com Tue Sep 24 18:35:35 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 24 Sep 2019 11:35:35 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Hi Daniil, This version looks good to me. Thank you for the update! Just one question about ThreadIdTable::remove_thread(jlong tid). What happens if there is no thread with the specified tid in ThreadIdTable? Is it possible? Thanks, Serguei On 9/24/19 9:36 AM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From daniel.daugherty at oracle.com Tue Sep 24 19:43:16 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 24 Sep 2019 15:43:16 -0400 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <246ece16-045f-3651-bf8c-6f5122ef017a@oracle.com> On 9/24/19 12:36 PM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ Did a "jfilemerge -r" compare between the v07 and v07 patches. Thumbs up! Dan > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From daniil.x.titov at oracle.com Tue Sep 24 19:46:07 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Tue, 24 Sep 2019 12:46:07 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: Hi Serguei, Thank you for reviewing this version of the fix. > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? It could be possible when the thread that was started while the thread table was initializing exits. At this point the thread table is initialized and the thread tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable is a correct operation that just leaves the table unchanged. src/hotspot/share/services/threadIdTable.cpp 233 bool ThreadIdTable::remove_thread(jlong tid) { 234 assert(_is_initialized, "Thread table is not initialized"); 235 Thread* thread = Thread::current(); 236 ThreadIdTableLookup lookup(tid); 237 return _local_table->remove(thread, lookup); 238 } src/hotspot/share/utilities/concurrentHashTable.hpp 422 // Returns true if items was deleted matching LOOKUP_FUNC and 423 // prior to destruction DELETE_FUNC is called. 424 template 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { 426 return internal_remove(thread, lookup_f, del_f); 427 } 428 429 // Same without DELETE_FUNC. 430 template 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { 432 return internal_remove(thread, lookup_f, noOp); 433 } src/hotspot/share/utilities/concurrentHashTable.inline.hpp 446 inline bool ConcurrentHashTable:: 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) 448 { 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); 450 assert(bucket->is_locked(), "Must be locked."); 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); 452 Node* rem_n = bucket->first(); 453 bool have_dead = false; 454 while (rem_n != NULL) { 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); 457 break; 458 } else { 459 rem_n_prev = rem_n->next_ptr(); 460 rem_n = rem_n->next(); 461 } 462 } 463 464 bucket->unlock(); 465 466 if (rem_n == NULL) { 467 return false; 468 } Best regards, Daniil ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, This version looks good to me. Thank you for the update! Just one question about ThreadIdTable::remove_thread(jlong tid). What happens if there is no thread with the specified tid in ThreadIdTable? Is it possible? Thanks, Serguei On 9/24/19 9:36 AM, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From david.holmes at oracle.com Tue Sep 24 22:45:10 2019 From: david.holmes at oracle.com (David Holmes) Date: Wed, 25 Sep 2019 08:45:10 +1000 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: Looks good to me. Thanks, David On 25/09/2019 2:36 am, Daniil Titov wrote: > Hi Daniel, David and Serguei, > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > corrects formatting issues. There are no other changes in this webrev.08 comparing > to the previous version webrev.07. > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > Thank you! > > Best regards, > Daniil > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > Daniil, > > Thanks for sticking with this project through the many versions. > Sorry this review is late... > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > Hi David and Serguei, > > > > Please review new version of the fix that includes the changes Serguei suggested: > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > src/hotspot/share/runtime/mutexLocker.hpp > No comments. > > src/hotspot/share/runtime/mutexLocker.cpp > No comments. > > src/hotspot/share/runtime/threadSMR.cpp > L623: MutexLocker ml(Threads_lock); > L626: if (!thread->is_exiting()) { > Re: discussion about is_exiting() > > The header comment is pretty clear: > > src/hotspot/share/runtime/thread.hpp: > > // thread has called JavaThread::exit() or is terminated > bool is_exiting() const; > > is_exiting() might become true right after you have called it, > but its purpose is to ask the question and not prevent the > condition from becoming true. As David said, you should consider > it an optimization. If you happen to see the condition is true, > then you know that the JavaThread isn't going to be around much > longer and should act accordingly. > > The is_exiting() implementation is: > > inline bool JavaThread::is_exiting() const { > // Use load-acquire so that setting of _terminated by > // JavaThread::exit() is seen more quickly. > TerminatedTypes l_terminated = (TerminatedTypes) > OrderAccess::load_acquire((volatile jint *) &_terminated); > return l_terminated == _thread_exiting || > check_is_terminated(l_terminated); > } > > and it depends on the JavaThread's _terminated field value. > > // JavaThread termination support > enum TerminatedTypes { > _not_terminated = 0xDEAD - 2, > _thread_exiting, // > JavaThread::exit() has been called for this thread > _thread_terminated, // JavaThread > is removed from thread list > _vm_exited // JavaThread > is still executing native code, but VM is terminated > // only VM_Exit > can set _vm_exited > }; > > so the JavaThread's _terminated field can get set to > _thread_exiting independent of the Threads_lock, but > it can't get set to _thread_terminated without the > Threads_lock. > > So by grabbing the Threads_lock on L623, you make sure > that ThreadTable::add_thread(java_tid, thread) does not > add a JavaThread that's not on the ThreadsList. It might > still become is_exiting() == true right after your > > L626 if (!thread->is_exiting()) { > > but it will still be on the main ThreadsList. And that > means that when the JavaThread is removed from the main > ThreadsList, you'll still call: > > L931: ThreadTable::remove_thread(tid); > > L624: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L633: return thread; > nit - L633 - indented too far (should be 2 spaces) > > src/hotspot/share/services/threadTable.hpp > L42: static void lazy_initialize(const ThreadsList *threads); > nit - put space between '*' the variable: > > static void lazy_initialize(const ThreadsList* threads); > > like you do in your other decls. > > L45: // Lookup and inserts > Perhaps: // Lookup and list management > > L60-61 - nit - please delete these blank lines. > > src/hotspot/share/services/threadTable.cpp > L28: #include "runtime/timerTrace.hpp" > nit - This should be after threadSMR.hpp... (alpha sorted order) > > L39: static const size_t DefaultThreadTableSizeLog = 8; > nit - your other 'static const' are not CamelCase. Why is this one? > > L45: static ThreadTableHash* volatile _local_table = NULL; > L50: static volatile size_t _current_size = 0; > L51: static volatile size_t _items_count = 0; > nit - can you group the file statics together? (up with L41). > > L60: _tid(tid),_java_thread(java_thread) {} > nit - space after ',' > > L62 jlong tid() const { return _tid;} > L63 JavaThread* thread() const {return _java_thread;} > nit - space before '}' > nit - space after '{' on L63. > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > Parameter 'is_dead' is not used. > > L74: static void* allocate_node(size_t size, Value const& value) { > Parameter 'value' is not used. > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > Re: discussion about lazy_initialize() racing with > ThreadsList::find_JavaThread_from_java_tid() > > There's a couple of aspects to these two pieces of code racing > with each other and racing with new thread creation. Racing with > new thread creation is the easy one: > > If a new thread isn't added to the ThreadTable by > ThreadsSMRSupport::add_thread() calling > ThreadTable::add_thread(), > then the point in the future where someone calls > find_JavaThread_from_java_tid() will add it to the table due to > the linear search when ThreadTable::find_thread_by_tid() > returns NULL. > > As for multi-threads calling > ThreadsList::find_JavaThread_from_java_tid() > at the same time which results in multi-threads in lazy_initialize() > at the same time... > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > After _is_initialized is set to true, then no more callers to > lazy_initialize() will be in the "if (!_is_initialized)" block. > - Once the ThreadTable is created, then multi-threads can be > executing the for-loop to add their ThreadsList entries to > the ThreadTable. There will be a bit of Threads_lock contention > as each of the multi-threads tries to add their entries and > there will be some wasted work since the multi-threads will > likely have similar ThreadLists. > > Of course, once _is_initialized is set to true, then any caller > to lazy_initialize() will return quickly and > ThreadsList::find_JavaThread_from_java_tid() will call > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > found, then we do the linear search thing here and add the > the entry if we find a match in our current ThreadsList. Since > we're only adding the one here, we only contend for the Threads_lock > here if we find it. > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > target java_tid for a JavaThread that was created after the > ThreadsList object that the caller has in hand for the > find_JavaThread_from_java_tid() call, then, of course, that > target 'java_tid' won't be found because the JavaThread was > added the main ThreadsList _after_ the ThreadsList object was > created by the caller. Of course, you have to ask where the > target java_tid value came from since the JavaThread wasn't > around when the ThreadsList::find_JavaThread_from_java_tid() > call was made with that target java_tid value... > > L99: // being concurently populated during the initalization. > Typos? Perhaps: > // to be concurrently populated during initialization. > > But I think those two comment lines are more appropriate above > this line: > > L96: MutexLocker ml(ThreadTableCreate_lock); > > L112: // Must be inside the lock to ensure that we don't > add the thread to the table > typo: s/the thread/a thread/ > > L141: return ((double)_items_count)/_current_size; > nit - need spaces around '/'. > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > nit - put space between '**' the variable: > bool equals(ThreadTableEntry** value, > > Parameter 'is_dead' is not used. > > L214: while(true) { > nit - space before '('. > > > Short version: Thumbs up. > > Longer version: I don't think I've spotted anything other than nits here. > Mostly I've just looked for multi-threaded races, proper usage of the > Thread-SMR stuff, and minimal impact in the case where the new > ThreadsTable is never needed. > > Dan > > P.S. > ThreadTable is a bit of misnomer. What you really have here is > a ThreadIdTable, but I'm really late to the code review flow > with that comment... > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > --Daniil > > > > From robbin.ehn at oracle.com Wed Sep 25 06:45:37 2019 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 25 Sep 2019 08:45:37 +0200 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: Hi Daniil, Looks good, thanks! /Robbin On 9/25/19 12:45 AM, David Holmes wrote: > Looks good to me. > > Thanks, > David > > On 25/09/2019 2:36 am, Daniil Titov wrote: >> Hi Daniel, David and Serguei, >> >> Please review a new version of the fix (webrev.08) that as Daniel suggested >> renames >> ThreadTable to ThreadIdTable (related classes and variables are renamed as >> well) and >> corrects formatting issues. There are no other changes in this webrev.08 >> comparing >> to the previous version webrev.07. >> >> Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> >> Best regards, >> Daniil >> >> ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: >> >> ???? Daniil, >> ???? Thanks for sticking with this project through the many versions. >> ???? Sorry this review is late... >> ???? On 9/19/19 8:30 PM, Daniil Titov wrote: >> ???? > Hi David and Serguei, >> ???? > >> ???? > Please review new version of the fix that includes the changes Serguei >> suggested: >> ???? >?? 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> ???? >?? 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> ???? >?????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? > >> ???? > The changes are in ThreadTable::lazy_initialize() method only. >> ???? > >> ???? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> ???? > >> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> ???? src/hotspot/share/runtime/mutexLocker.hpp >> ????????? No comments. >> ???? src/hotspot/share/runtime/mutexLocker.cpp >> ????????? No comments. >> ???? src/hotspot/share/runtime/threadSMR.cpp >> ????????? L623:???????? MutexLocker ml(Threads_lock); >> ????????? L626:???????? if (!thread->is_exiting()) { >> ????????????? Re: discussion about is_exiting() >> ????????????? The header comment is pretty clear: >> ??????????????? src/hotspot/share/runtime/thread.hpp: >> ????????????????? // thread has called JavaThread::exit() or is terminated >> ????????????????? bool is_exiting() const; >> ????????????? is_exiting() might become true right after you have called it, >> ????????????? but its purpose is to ask the question and not prevent the >> ????????????? condition from becoming true. As David said, you should consider >> ????????????? it an optimization. If you happen to see the condition is true, >> ????????????? then you know that the JavaThread isn't going to be around much >> ????????????? longer and should act accordingly. >> ????????????? The is_exiting() implementation is: >> ??????????????? inline bool JavaThread::is_exiting() const { >> ????????????????? // Use load-acquire so that setting of _terminated by >> ????????????????? // JavaThread::exit() is seen more quickly. >> ????????????????? TerminatedTypes l_terminated = (TerminatedTypes) >> ????????????????????? OrderAccess::load_acquire((volatile jint *) &_terminated); >> ????????????????? return l_terminated == _thread_exiting || >> ???? check_is_terminated(l_terminated); >> ??????????????? } >> ????????????? and it depends on the JavaThread's _terminated field value. >> ??????????????? // JavaThread termination support >> ??????????????? enum TerminatedTypes { >> ???????????????? _not_terminated = 0xDEAD - 2, >> ???????????????? _thread_exiting,???????????????????????????? // >> ???? JavaThread::exit() has been called for this thread >> ???????????????? _thread_terminated,????????????????????????? // JavaThread >> ???? is removed from thread list >> ???????????????? _vm_exited?????????????????????????????????? // JavaThread >> ???? is still executing native code, but VM is terminated >> ????????????????????????????????????????????????????????????? // only VM_Exit >> ???? can set _vm_exited >> ??????????????? }; >> ????????????? so the JavaThread's _terminated field can get set to >> ????????????? _thread_exiting independent of the Threads_lock, but >> ????????????? it can't get set to _thread_terminated without the >> ????????????? Threads_lock. >> ????????????? So by grabbing the Threads_lock on L623, you make sure >> ????????????? that ThreadTable::add_thread(java_tid, thread) does not >> ????????????? add a JavaThread that's not on the ThreadsList. It might >> ????????????? still become is_exiting() == true right after your >> ??????????????? L626???????? if (!thread->is_exiting()) { >> ????????????? but it will still be on the main ThreadsList. And that >> ????????????? means that when the JavaThread is removed from the main >> ????????????? ThreadsList, you'll still call: >> ??????????????? L931:???? ThreadTable::remove_thread(tid); >> ????????? L624:???????? // Must be inside the lock to ensure that we don't >> ???? add the thread to the table >> ????????????? typo: s/the thread/a thread/ >> ????????? L633:?????? return thread; >> ????????????? nit - L633 - indented too far (should be 2 spaces) >> ???? src/hotspot/share/services/threadTable.hpp >> ????????? L42:?? static void lazy_initialize(const ThreadsList *threads); >> ????????????? nit - put space between '*' the variable: >> ??????????????? static void lazy_initialize(const ThreadsList* threads); >> ????????????? like you do in your other decls. >> ????????? L45:?? // Lookup and inserts >> ????????????? Perhaps:? // Lookup and list management >> ????????? L60-61 - nit - please delete these blank lines. >> ???? src/hotspot/share/services/threadTable.cpp >> ????????? L28: #include "runtime/timerTrace.hpp" >> ????????????? nit - This should be after threadSMR.hpp... (alpha sorted order) >> ????????? L39: static const size_t DefaultThreadTableSizeLog = 8; >> ????????????? nit - your other 'static const' are not CamelCase. Why is this one? >> ????????? L45: static ThreadTableHash* volatile _local_table = NULL; >> ????????? L50: static volatile size_t _current_size = 0; >> ????????? L51: static volatile size_t _items_count = 0; >> ????????????? nit - can you group the file statics together? (up with L41). >> ????????? L60:???? _tid(tid),_java_thread(java_thread) {} >> ????????????? nit - space after ',' >> ????????? L62?? jlong tid() const { return _tid;} >> ????????? L63?? JavaThread* thread() const {return _java_thread;} >> ????????????? nit - space before '}' >> ????????????? nit - space after '{' on L63. >> ????????? L70:???? static uintx get_hash(Value const& value, bool* is_dead) { >> ????????????? Parameter 'is_dead' is not used. >> ????????? L74:???? static void* allocate_node(size_t size, Value const& value) { >> ????????????? Parameter 'value' is not used. >> ????????? L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> ????????????? Re: discussion about lazy_initialize() racing with >> ????????????????? ThreadsList::find_JavaThread_from_java_tid() >> ????????????? There's a couple of aspects to these two pieces of code racing >> ????????????? with each other and racing with new thread creation. Racing with >> ????????????? new thread creation is the easy one: >> ??????????????? If a new thread isn't added to the ThreadTable by >> ??????????????? ThreadsSMRSupport::add_thread() calling >> ???? ThreadTable::add_thread(), >> ??????????????? then the point in the future where someone calls >> ??????????????? find_JavaThread_from_java_tid() will add it to the table due to >> ??????????????? the linear search when ThreadTable::find_thread_by_tid() >> ??????????????? returns NULL. >> ???????????? As for multi-threads calling >> ???? ThreadsList::find_JavaThread_from_java_tid() >> ???????????? at the same time which results in multi-threads in lazy_initialize() >> ???????????? at the same time... >> ???????????? - ThreadTable creation will be linear due to ThreadTableCreate_lock. >> ?????????????? After _is_initialized is set to true, then no more callers to >> ?????????????? lazy_initialize() will be in the "if (!_is_initialized)" block. >> ???????????? - Once the ThreadTable is created, then multi-threads can be >> ?????????????? executing the for-loop to add their ThreadsList entries to >> ?????????????? the ThreadTable. There will be a bit of Threads_lock contention >> ?????????????? as each of the multi-threads tries to add their entries and >> ?????????????? there will be some wasted work since the multi-threads will >> ?????????????? likely have similar ThreadLists. >> ???????????? Of course, once _is_initialized is set to true, then any caller >> ???????????? to lazy_initialize() will return quickly and >> ???????????? ThreadsList::find_JavaThread_from_java_tid() will call >> ???????????? ThreadTable::find_thread_by_tid(). If the target java_tid isn't >> ???????????? found, then we do the linear search thing here and add the >> ???????????? the entry if we find a match in our current ThreadsList. Since >> ???????????? we're only adding the one here, we only contend for the Threads_lock >> ???????????? here if we find it. >> ???????????? If ThreadsList::find_JavaThread_from_java_tid() is called with a >> ???????????? target java_tid for a JavaThread that was created after the >> ???????????? ThreadsList object that the caller has in hand for the >> ???????????? find_JavaThread_from_java_tid() call, then, of course, that >> ???????????? target 'java_tid' won't be found because the JavaThread was >> ???????????? added the main ThreadsList _after_ the ThreadsList object was >> ???????????? created by the caller. Of course, you have to ask where the >> ???????????? target java_tid value came from since the JavaThread wasn't >> ???????????? around when the ThreadsList::find_JavaThread_from_java_tid() >> ???????????? call was made with that target java_tid value... >> ????????? L99:???????? // being concurently populated during the initalization. >> ????????????? Typos? Perhaps: >> ?????????????????????? // to be concurrently populated during initialization. >> ????????????? But I think those two comment lines are more appropriate above >> ????????????? this line: >> ????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >> ????????? L112:?????????? // Must be inside the lock to ensure that we don't >> ???? add the thread to the table >> ????????????? typo: s/the thread/a thread/ >> ????????? L141:?? return ((double)_items_count)/_current_size; >> ????????????? nit - need spaces around '/'. >> ????????? L177:?? bool equals(ThreadTableEntry **value, bool* is_dead) { >> ????????????? nit - put space between '**' the variable: >> ????????????????? bool equals(ThreadTableEntry** value, >> ????????????? Parameter 'is_dead' is not used. >> ????????? L214:?? while(true) { >> ????????????? nit - space before '('. >> ???? Short version: Thumbs up. >> ???? Longer version: I don't think I've spotted anything other than nits here. >> ???? Mostly I've just looked for multi-threaded races, proper usage of the >> ???? Thread-SMR stuff, and minimal impact in the case where the new >> ???? ThreadsTable is never needed. >> ???? Dan >> ???? P.S. >> ???? ThreadTable is a bit of misnomer. What you really have here is >> ???? a ThreadIdTable, but I'm really late to the code review flow >> ???? with that comment... >> ???? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > >> ???? > Thank you! >> ???? > --Daniil >> >> From daniil.x.titov at oracle.com Wed Sep 25 18:22:46 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Wed, 25 Sep 2019 11:22:46 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <2d6dede1-aa79-99ce-a823-773fa2e19827@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> Message-ID: <745A2116-EDB8-4F27-974D-F52F15A89477@oracle.com> Thank you, David, Daniel, Serguei, and Robbin, for reviewing this change! Best regards, Daniil ?On 9/24/19, 11:45 PM, "Robbin Ehn" wrote: Hi Daniil, Looks good, thanks! /Robbin On 9/25/19 12:45 AM, David Holmes wrote: > Looks good to me. > > Thanks, > David > > On 25/09/2019 2:36 am, Daniil Titov wrote: >> Hi Daniel, David and Serguei, >> >> Please review a new version of the fix (webrev.08) that as Daniel suggested >> renames >> ThreadTable to ThreadIdTable (related classes and variables are renamed as >> well) and >> corrects formatting issues. There are no other changes in this webrev.08 >> comparing >> to the previous version webrev.07. >> >> Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. >> >> Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> >> Thank you! >> >> Best regards, >> Daniil >> >> ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: >> >> Daniil, >> Thanks for sticking with this project through the many versions. >> Sorry this review is late... >> On 9/19/19 8:30 PM, Daniil Titov wrote: >> > Hi David and Serguei, >> > >> > Please review new version of the fix that includes the changes Serguei >> suggested: >> > 1. If racing threads initialize the thread table only one of these >> threads will populate the table with the threads from the thread list >> > 2. The code that adds the thread to the tread table is put inside >> Threads_lock to ensure that we cannot accidentally add the thread >> > that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> > >> > The changes are in ThreadTable::lazy_initialize() method only. >> > >> > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> > >> > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> src/hotspot/share/runtime/mutexLocker.hpp >> No comments. >> src/hotspot/share/runtime/mutexLocker.cpp >> No comments. >> src/hotspot/share/runtime/threadSMR.cpp >> L623: MutexLocker ml(Threads_lock); >> L626: if (!thread->is_exiting()) { >> Re: discussion about is_exiting() >> The header comment is pretty clear: >> src/hotspot/share/runtime/thread.hpp: >> // thread has called JavaThread::exit() or is terminated >> bool is_exiting() const; >> is_exiting() might become true right after you have called it, >> but its purpose is to ask the question and not prevent the >> condition from becoming true. As David said, you should consider >> it an optimization. If you happen to see the condition is true, >> then you know that the JavaThread isn't going to be around much >> longer and should act accordingly. >> The is_exiting() implementation is: >> inline bool JavaThread::is_exiting() const { >> // Use load-acquire so that setting of _terminated by >> // JavaThread::exit() is seen more quickly. >> TerminatedTypes l_terminated = (TerminatedTypes) >> OrderAccess::load_acquire((volatile jint *) &_terminated); >> return l_terminated == _thread_exiting || >> check_is_terminated(l_terminated); >> } >> and it depends on the JavaThread's _terminated field value. >> // JavaThread termination support >> enum TerminatedTypes { >> _not_terminated = 0xDEAD - 2, >> _thread_exiting, // >> JavaThread::exit() has been called for this thread >> _thread_terminated, // JavaThread >> is removed from thread list >> _vm_exited // JavaThread >> is still executing native code, but VM is terminated >> // only VM_Exit >> can set _vm_exited >> }; >> so the JavaThread's _terminated field can get set to >> _thread_exiting independent of the Threads_lock, but >> it can't get set to _thread_terminated without the >> Threads_lock. >> So by grabbing the Threads_lock on L623, you make sure >> that ThreadTable::add_thread(java_tid, thread) does not >> add a JavaThread that's not on the ThreadsList. It might >> still become is_exiting() == true right after your >> L626 if (!thread->is_exiting()) { >> but it will still be on the main ThreadsList. And that >> means that when the JavaThread is removed from the main >> ThreadsList, you'll still call: >> L931: ThreadTable::remove_thread(tid); >> L624: // Must be inside the lock to ensure that we don't >> add the thread to the table >> typo: s/the thread/a thread/ >> L633: return thread; >> nit - L633 - indented too far (should be 2 spaces) >> src/hotspot/share/services/threadTable.hpp >> L42: static void lazy_initialize(const ThreadsList *threads); >> nit - put space between '*' the variable: >> static void lazy_initialize(const ThreadsList* threads); >> like you do in your other decls. >> L45: // Lookup and inserts >> Perhaps: // Lookup and list management >> L60-61 - nit - please delete these blank lines. >> src/hotspot/share/services/threadTable.cpp >> L28: #include "runtime/timerTrace.hpp" >> nit - This should be after threadSMR.hpp... (alpha sorted order) >> L39: static const size_t DefaultThreadTableSizeLog = 8; >> nit - your other 'static const' are not CamelCase. Why is this one? >> L45: static ThreadTableHash* volatile _local_table = NULL; >> L50: static volatile size_t _current_size = 0; >> L51: static volatile size_t _items_count = 0; >> nit - can you group the file statics together? (up with L41). >> L60: _tid(tid),_java_thread(java_thread) {} >> nit - space after ',' >> L62 jlong tid() const { return _tid;} >> L63 JavaThread* thread() const {return _java_thread;} >> nit - space before '}' >> nit - space after '{' on L63. >> L70: static uintx get_hash(Value const& value, bool* is_dead) { >> Parameter 'is_dead' is not used. >> L74: static void* allocate_node(size_t size, Value const& value) { >> Parameter 'value' is not used. >> L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { >> Re: discussion about lazy_initialize() racing with >> ThreadsList::find_JavaThread_from_java_tid() >> There's a couple of aspects to these two pieces of code racing >> with each other and racing with new thread creation. Racing with >> new thread creation is the easy one: >> If a new thread isn't added to the ThreadTable by >> ThreadsSMRSupport::add_thread() calling >> ThreadTable::add_thread(), >> then the point in the future where someone calls >> find_JavaThread_from_java_tid() will add it to the table due to >> the linear search when ThreadTable::find_thread_by_tid() >> returns NULL. >> As for multi-threads calling >> ThreadsList::find_JavaThread_from_java_tid() >> at the same time which results in multi-threads in lazy_initialize() >> at the same time... >> - ThreadTable creation will be linear due to ThreadTableCreate_lock. >> After _is_initialized is set to true, then no more callers to >> lazy_initialize() will be in the "if (!_is_initialized)" block. >> - Once the ThreadTable is created, then multi-threads can be >> executing the for-loop to add their ThreadsList entries to >> the ThreadTable. There will be a bit of Threads_lock contention >> as each of the multi-threads tries to add their entries and >> there will be some wasted work since the multi-threads will >> likely have similar ThreadLists. >> Of course, once _is_initialized is set to true, then any caller >> to lazy_initialize() will return quickly and >> ThreadsList::find_JavaThread_from_java_tid() will call >> ThreadTable::find_thread_by_tid(). If the target java_tid isn't >> found, then we do the linear search thing here and add the >> the entry if we find a match in our current ThreadsList. Since >> we're only adding the one here, we only contend for the Threads_lock >> here if we find it. >> If ThreadsList::find_JavaThread_from_java_tid() is called with a >> target java_tid for a JavaThread that was created after the >> ThreadsList object that the caller has in hand for the >> find_JavaThread_from_java_tid() call, then, of course, that >> target 'java_tid' won't be found because the JavaThread was >> added the main ThreadsList _after_ the ThreadsList object was >> created by the caller. Of course, you have to ask where the >> target java_tid value came from since the JavaThread wasn't >> around when the ThreadsList::find_JavaThread_from_java_tid() >> call was made with that target java_tid value... >> L99: // being concurently populated during the initalization. >> Typos? Perhaps: >> // to be concurrently populated during initialization. >> But I think those two comment lines are more appropriate above >> this line: >> L96: MutexLocker ml(ThreadTableCreate_lock); >> L112: // Must be inside the lock to ensure that we don't >> add the thread to the table >> typo: s/the thread/a thread/ >> L141: return ((double)_items_count)/_current_size; >> nit - need spaces around '/'. >> L177: bool equals(ThreadTableEntry **value, bool* is_dead) { >> nit - put space between '**' the variable: >> bool equals(ThreadTableEntry** value, >> Parameter 'is_dead' is not used. >> L214: while(true) { >> nit - space before '('. >> Short version: Thumbs up. >> Longer version: I don't think I've spotted anything other than nits here. >> Mostly I've just looked for multi-threaded races, proper usage of the >> Thread-SMR stuff, and minimal impact in the case where the new >> ThreadsTable is never needed. >> Dan >> P.S. >> ThreadTable is a bit of misnomer. What you really have here is >> a ThreadIdTable, but I'm really late to the code review flow >> with that comment... >> > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> > >> > Thank you! >> > --Daniil >> >> From serguei.spitsyn at oracle.com Fri Sep 27 17:58:23 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 10:58:23 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: Hi Daniil, Just notice I did not reply to you. Thank you for the explanation! Have you already pushed this one? Thanks, Serguei On 9/24/19 12:46, Daniil Titov wrote: > Hi Serguei, > > Thank you for reviewing this version of the fix. > >> Just one question about ThreadIdTable::remove_thread(jlong tid). >> What happens if there is no thread with the specified tid in ThreadIdTable? >> Is it possible? > It could be possible when the thread that was started while the thread table > was initializing exits. At this point the thread table is initialized and the thread > tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable > is a correct operation that just leaves the table unchanged. > > src/hotspot/share/services/threadIdTable.cpp > > 233 bool ThreadIdTable::remove_thread(jlong tid) { > 234 assert(_is_initialized, "Thread table is not initialized"); > 235 Thread* thread = Thread::current(); > 236 ThreadIdTableLookup lookup(tid); > 237 return _local_table->remove(thread, lookup); > 238 } > > src/hotspot/share/utilities/concurrentHashTable.hpp > > 422 // Returns true if items was deleted matching LOOKUP_FUNC and > 423 // prior to destruction DELETE_FUNC is called. > 424 template > 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { > 426 return internal_remove(thread, lookup_f, del_f); > 427 } > 428 > 429 // Same without DELETE_FUNC. > 430 template > 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { > 432 return internal_remove(thread, lookup_f, noOp); > 433 } > > src/hotspot/share/utilities/concurrentHashTable.inline.hpp > > 446 inline bool ConcurrentHashTable:: > 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) > 448 { > 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); > 450 assert(bucket->is_locked(), "Must be locked."); > 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); > 452 Node* rem_n = bucket->first(); > 453 bool have_dead = false; > 454 while (rem_n != NULL) { > 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { > 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); > 457 break; > 458 } else { > 459 rem_n_prev = rem_n->next_ptr(); > 460 rem_n = rem_n->next(); > 461 } > 462 } > 463 > 464 bucket->unlock(); > 465 > 466 if (rem_n == NULL) { > 467 return false; > 468 } > > Best regards, > Daniil > > > ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > This version looks good to me. > Thank you for the update! > > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? > > Thanks, > Serguei > > On 9/24/19 9:36 AM, Daniil Titov wrote: > > Hi Daniel, David and Serguei, > > > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > > corrects formatting issues. There are no other changes in this webrev.08 comparing > > to the previous version webrev.07. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > > > Best regards, > > Daniil > > > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > > > Daniil, > > > > Thanks for sticking with this project through the many versions. > > Sorry this review is late... > > > > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > > Hi David and Serguei, > > > > > > Please review new version of the fix that includes the changes Serguei suggested: > > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > > > src/hotspot/share/runtime/mutexLocker.hpp > > No comments. > > > > src/hotspot/share/runtime/mutexLocker.cpp > > No comments. > > > > src/hotspot/share/runtime/threadSMR.cpp > > L623: MutexLocker ml(Threads_lock); > > L626: if (!thread->is_exiting()) { > > Re: discussion about is_exiting() > > > > The header comment is pretty clear: > > > > src/hotspot/share/runtime/thread.hpp: > > > > // thread has called JavaThread::exit() or is terminated > > bool is_exiting() const; > > > > is_exiting() might become true right after you have called it, > > but its purpose is to ask the question and not prevent the > > condition from becoming true. As David said, you should consider > > it an optimization. If you happen to see the condition is true, > > then you know that the JavaThread isn't going to be around much > > longer and should act accordingly. > > > > The is_exiting() implementation is: > > > > inline bool JavaThread::is_exiting() const { > > // Use load-acquire so that setting of _terminated by > > // JavaThread::exit() is seen more quickly. > > TerminatedTypes l_terminated = (TerminatedTypes) > > OrderAccess::load_acquire((volatile jint *) &_terminated); > > return l_terminated == _thread_exiting || > > check_is_terminated(l_terminated); > > } > > > > and it depends on the JavaThread's _terminated field value. > > > > // JavaThread termination support > > enum TerminatedTypes { > > _not_terminated = 0xDEAD - 2, > > _thread_exiting, // > > JavaThread::exit() has been called for this thread > > _thread_terminated, // JavaThread > > is removed from thread list > > _vm_exited // JavaThread > > is still executing native code, but VM is terminated > > // only VM_Exit > > can set _vm_exited > > }; > > > > so the JavaThread's _terminated field can get set to > > _thread_exiting independent of the Threads_lock, but > > it can't get set to _thread_terminated without the > > Threads_lock. > > > > So by grabbing the Threads_lock on L623, you make sure > > that ThreadTable::add_thread(java_tid, thread) does not > > add a JavaThread that's not on the ThreadsList. It might > > still become is_exiting() == true right after your > > > > L626 if (!thread->is_exiting()) { > > > > but it will still be on the main ThreadsList. And that > > means that when the JavaThread is removed from the main > > ThreadsList, you'll still call: > > > > L931: ThreadTable::remove_thread(tid); > > > > L624: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L633: return thread; > > nit - L633 - indented too far (should be 2 spaces) > > > > src/hotspot/share/services/threadTable.hpp > > L42: static void lazy_initialize(const ThreadsList *threads); > > nit - put space between '*' the variable: > > > > static void lazy_initialize(const ThreadsList* threads); > > > > like you do in your other decls. > > > > L45: // Lookup and inserts > > Perhaps: // Lookup and list management > > > > L60-61 - nit - please delete these blank lines. > > > > src/hotspot/share/services/threadTable.cpp > > L28: #include "runtime/timerTrace.hpp" > > nit - This should be after threadSMR.hpp... (alpha sorted order) > > > > L39: static const size_t DefaultThreadTableSizeLog = 8; > > nit - your other 'static const' are not CamelCase. Why is this one? > > > > L45: static ThreadTableHash* volatile _local_table = NULL; > > L50: static volatile size_t _current_size = 0; > > L51: static volatile size_t _items_count = 0; > > nit - can you group the file statics together? (up with L41). > > > > L60: _tid(tid),_java_thread(java_thread) {} > > nit - space after ',' > > > > L62 jlong tid() const { return _tid;} > > L63 JavaThread* thread() const {return _java_thread;} > > nit - space before '}' > > nit - space after '{' on L63. > > > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > > Parameter 'is_dead' is not used. > > > > L74: static void* allocate_node(size_t size, Value const& value) { > > Parameter 'value' is not used. > > > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > Re: discussion about lazy_initialize() racing with > > ThreadsList::find_JavaThread_from_java_tid() > > > > There's a couple of aspects to these two pieces of code racing > > with each other and racing with new thread creation. Racing with > > new thread creation is the easy one: > > > > If a new thread isn't added to the ThreadTable by > > ThreadsSMRSupport::add_thread() calling > > ThreadTable::add_thread(), > > then the point in the future where someone calls > > find_JavaThread_from_java_tid() will add it to the table due to > > the linear search when ThreadTable::find_thread_by_tid() > > returns NULL. > > > > As for multi-threads calling > > ThreadsList::find_JavaThread_from_java_tid() > > at the same time which results in multi-threads in lazy_initialize() > > at the same time... > > > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > > After _is_initialized is set to true, then no more callers to > > lazy_initialize() will be in the "if (!_is_initialized)" block. > > - Once the ThreadTable is created, then multi-threads can be > > executing the for-loop to add their ThreadsList entries to > > the ThreadTable. There will be a bit of Threads_lock contention > > as each of the multi-threads tries to add their entries and > > there will be some wasted work since the multi-threads will > > likely have similar ThreadLists. > > > > Of course, once _is_initialized is set to true, then any caller > > to lazy_initialize() will return quickly and > > ThreadsList::find_JavaThread_from_java_tid() will call > > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > > found, then we do the linear search thing here and add the > > the entry if we find a match in our current ThreadsList. Since > > we're only adding the one here, we only contend for the Threads_lock > > here if we find it. > > > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > > target java_tid for a JavaThread that was created after the > > ThreadsList object that the caller has in hand for the > > find_JavaThread_from_java_tid() call, then, of course, that > > target 'java_tid' won't be found because the JavaThread was > > added the main ThreadsList _after_ the ThreadsList object was > > created by the caller. Of course, you have to ask where the > > target java_tid value came from since the JavaThread wasn't > > around when the ThreadsList::find_JavaThread_from_java_tid() > > call was made with that target java_tid value... > > > > L99: // being concurently populated during the initalization. > > Typos? Perhaps: > > // to be concurrently populated during initialization. > > > > But I think those two comment lines are more appropriate above > > this line: > > > > L96: MutexLocker ml(ThreadTableCreate_lock); > > > > L112: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L141: return ((double)_items_count)/_current_size; > > nit - need spaces around '/'. > > > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > > nit - put space between '**' the variable: > > bool equals(ThreadTableEntry** value, > > > > Parameter 'is_dead' is not used. > > > > L214: while(true) { > > nit - space before '('. > > > > > > Short version: Thumbs up. > > > > Longer version: I don't think I've spotted anything other than nits here. > > Mostly I've just looked for multi-threaded races, proper usage of the > > Thread-SMR stuff, and minimal impact in the case where the new > > ThreadsTable is never needed. > > > > Dan > > > > P.S. > > ThreadTable is a bit of misnomer. What you really have here is > > a ThreadIdTable, but I'm really late to the code review flow > > with that comment... > > > > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thank you! > > > --Daniil > > > > > > > > > > > > From daniil.x.titov at oracle.com Fri Sep 27 18:07:51 2019 From: daniil.x.titov at oracle.com (Daniil Titov) Date: Fri, 27 Sep 2019 11:07:51 -0700 Subject: jmx-dev 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <6E7B043A-4647-4931-977C-1854CA7EBEC1@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: <7DB8CB3A-33CF-4C1D-8C7F-0324FB269A3E@oracle.com> Hi Serguei, Yes, it is already pushed in the repository [1]. [1] https://hg.openjdk.java.net/jdk/jdk/rev/f4abe950c3b0 Best regards, Daniil ?On 9/27/19, 10:58 AM, "serguei.spitsyn at oracle.com" wrote: Hi Daniil, Just notice I did not reply to you. Thank you for the explanation! Have you already pushed this one? Thanks, Serguei On 9/24/19 12:46, Daniil Titov wrote: > Hi Serguei, > > Thank you for reviewing this version of the fix. > >> Just one question about ThreadIdTable::remove_thread(jlong tid). >> What happens if there is no thread with the specified tid in ThreadIdTable? >> Is it possible? > It could be possible when the thread that was started while the thread table > was initializing exits. At this point the thread table is initialized and the thread > tries to remove itself from it. Removing non-existing entry from ConcurrentHashTable > is a correct operation that just leaves the table unchanged. > > src/hotspot/share/services/threadIdTable.cpp > > 233 bool ThreadIdTable::remove_thread(jlong tid) { > 234 assert(_is_initialized, "Thread table is not initialized"); > 235 Thread* thread = Thread::current(); > 236 ThreadIdTableLookup lookup(tid); > 237 return _local_table->remove(thread, lookup); > 238 } > > src/hotspot/share/utilities/concurrentHashTable.hpp > > 422 // Returns true if items was deleted matching LOOKUP_FUNC and > 423 // prior to destruction DELETE_FUNC is called. > 424 template > 425 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& del_f) { > 426 return internal_remove(thread, lookup_f, del_f); > 427 } > 428 > 429 // Same without DELETE_FUNC. > 430 template > 431 bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { > 432 return internal_remove(thread, lookup_f, noOp); > 433 } > > src/hotspot/share/utilities/concurrentHashTable.inline.hpp > > 446 inline bool ConcurrentHashTable:: > 447 internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, DELETE_FUNC& delete_f) > 448 { > 449 Bucket* bucket = get_bucket_locked(thread, lookup_f.get_hash()); > 450 assert(bucket->is_locked(), "Must be locked."); > 451 Node* const volatile * rem_n_prev = bucket->first_ptr(); > 452 Node* rem_n = bucket->first(); > 453 bool have_dead = false; > 454 while (rem_n != NULL) { > 455 if (lookup_f.equals(rem_n->value(), &have_dead)) { > 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); > 457 break; > 458 } else { > 459 rem_n_prev = rem_n->next_ptr(); > 460 rem_n = rem_n->next(); > 461 } > 462 } > 463 > 464 bucket->unlock(); > 465 > 466 if (rem_n == NULL) { > 467 return false; > 468 } > > Best regards, > Daniil > > > ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" wrote: > > Hi Daniil, > > This version looks good to me. > Thank you for the update! > > Just one question about ThreadIdTable::remove_thread(jlong tid). > What happens if there is no thread with the specified tid in ThreadIdTable? > Is it possible? > > Thanks, > Serguei > > On 9/24/19 9:36 AM, Daniil Titov wrote: > > Hi Daniel, David and Serguei, > > > > Please review a new version of the fix (webrev.08) that as Daniel suggested renames > > ThreadTable to ThreadIdTable (related classes and variables are renamed as well) and > > corrects formatting issues. There are no other changes in this webrev.08 comparing > > to the previous version webrev.07. > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ > > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > Thank you! > > > > Best regards, > > Daniil > > > > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" wrote: > > > > Daniil, > > > > Thanks for sticking with this project through the many versions. > > Sorry this review is late... > > > > > > On 9/19/19 8:30 PM, Daniil Titov wrote: > > > Hi David and Serguei, > > > > > > Please review new version of the fix that includes the changes Serguei suggested: > > > 1. If racing threads initialize the thread table only one of these threads will populate the table with the threads from the thread list > > > 2. The code that adds the thread to the tread table is put inside Threads_lock to ensure that we cannot accidentally add the thread > > > that has just passed the removal point in ThreadsSMRSupport::remove_thread() > > > > > > The changes are in ThreadTable::lazy_initialize() method only. > > > > > > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests successfully passed. > > > > > > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ > > > > src/hotspot/share/runtime/mutexLocker.hpp > > No comments. > > > > src/hotspot/share/runtime/mutexLocker.cpp > > No comments. > > > > src/hotspot/share/runtime/threadSMR.cpp > > L623: MutexLocker ml(Threads_lock); > > L626: if (!thread->is_exiting()) { > > Re: discussion about is_exiting() > > > > The header comment is pretty clear: > > > > src/hotspot/share/runtime/thread.hpp: > > > > // thread has called JavaThread::exit() or is terminated > > bool is_exiting() const; > > > > is_exiting() might become true right after you have called it, > > but its purpose is to ask the question and not prevent the > > condition from becoming true. As David said, you should consider > > it an optimization. If you happen to see the condition is true, > > then you know that the JavaThread isn't going to be around much > > longer and should act accordingly. > > > > The is_exiting() implementation is: > > > > inline bool JavaThread::is_exiting() const { > > // Use load-acquire so that setting of _terminated by > > // JavaThread::exit() is seen more quickly. > > TerminatedTypes l_terminated = (TerminatedTypes) > > OrderAccess::load_acquire((volatile jint *) &_terminated); > > return l_terminated == _thread_exiting || > > check_is_terminated(l_terminated); > > } > > > > and it depends on the JavaThread's _terminated field value. > > > > // JavaThread termination support > > enum TerminatedTypes { > > _not_terminated = 0xDEAD - 2, > > _thread_exiting, // > > JavaThread::exit() has been called for this thread > > _thread_terminated, // JavaThread > > is removed from thread list > > _vm_exited // JavaThread > > is still executing native code, but VM is terminated > > // only VM_Exit > > can set _vm_exited > > }; > > > > so the JavaThread's _terminated field can get set to > > _thread_exiting independent of the Threads_lock, but > > it can't get set to _thread_terminated without the > > Threads_lock. > > > > So by grabbing the Threads_lock on L623, you make sure > > that ThreadTable::add_thread(java_tid, thread) does not > > add a JavaThread that's not on the ThreadsList. It might > > still become is_exiting() == true right after your > > > > L626 if (!thread->is_exiting()) { > > > > but it will still be on the main ThreadsList. And that > > means that when the JavaThread is removed from the main > > ThreadsList, you'll still call: > > > > L931: ThreadTable::remove_thread(tid); > > > > L624: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L633: return thread; > > nit - L633 - indented too far (should be 2 spaces) > > > > src/hotspot/share/services/threadTable.hpp > > L42: static void lazy_initialize(const ThreadsList *threads); > > nit - put space between '*' the variable: > > > > static void lazy_initialize(const ThreadsList* threads); > > > > like you do in your other decls. > > > > L45: // Lookup and inserts > > Perhaps: // Lookup and list management > > > > L60-61 - nit - please delete these blank lines. > > > > src/hotspot/share/services/threadTable.cpp > > L28: #include "runtime/timerTrace.hpp" > > nit - This should be after threadSMR.hpp... (alpha sorted order) > > > > L39: static const size_t DefaultThreadTableSizeLog = 8; > > nit - your other 'static const' are not CamelCase. Why is this one? > > > > L45: static ThreadTableHash* volatile _local_table = NULL; > > L50: static volatile size_t _current_size = 0; > > L51: static volatile size_t _items_count = 0; > > nit - can you group the file statics together? (up with L41). > > > > L60: _tid(tid),_java_thread(java_thread) {} > > nit - space after ',' > > > > L62 jlong tid() const { return _tid;} > > L63 JavaThread* thread() const {return _java_thread;} > > nit - space before '}' > > nit - space after '{' on L63. > > > > L70: static uintx get_hash(Value const& value, bool* is_dead) { > > Parameter 'is_dead' is not used. > > > > L74: static void* allocate_node(size_t size, Value const& value) { > > Parameter 'value' is not used. > > > > L93: void ThreadTable::lazy_initialize(const ThreadsList *threads) { > > Re: discussion about lazy_initialize() racing with > > ThreadsList::find_JavaThread_from_java_tid() > > > > There's a couple of aspects to these two pieces of code racing > > with each other and racing with new thread creation. Racing with > > new thread creation is the easy one: > > > > If a new thread isn't added to the ThreadTable by > > ThreadsSMRSupport::add_thread() calling > > ThreadTable::add_thread(), > > then the point in the future where someone calls > > find_JavaThread_from_java_tid() will add it to the table due to > > the linear search when ThreadTable::find_thread_by_tid() > > returns NULL. > > > > As for multi-threads calling > > ThreadsList::find_JavaThread_from_java_tid() > > at the same time which results in multi-threads in lazy_initialize() > > at the same time... > > > > - ThreadTable creation will be linear due to ThreadTableCreate_lock. > > After _is_initialized is set to true, then no more callers to > > lazy_initialize() will be in the "if (!_is_initialized)" block. > > - Once the ThreadTable is created, then multi-threads can be > > executing the for-loop to add their ThreadsList entries to > > the ThreadTable. There will be a bit of Threads_lock contention > > as each of the multi-threads tries to add their entries and > > there will be some wasted work since the multi-threads will > > likely have similar ThreadLists. > > > > Of course, once _is_initialized is set to true, then any caller > > to lazy_initialize() will return quickly and > > ThreadsList::find_JavaThread_from_java_tid() will call > > ThreadTable::find_thread_by_tid(). If the target java_tid isn't > > found, then we do the linear search thing here and add the > > the entry if we find a match in our current ThreadsList. Since > > we're only adding the one here, we only contend for the Threads_lock > > here if we find it. > > > > If ThreadsList::find_JavaThread_from_java_tid() is called with a > > target java_tid for a JavaThread that was created after the > > ThreadsList object that the caller has in hand for the > > find_JavaThread_from_java_tid() call, then, of course, that > > target 'java_tid' won't be found because the JavaThread was > > added the main ThreadsList _after_ the ThreadsList object was > > created by the caller. Of course, you have to ask where the > > target java_tid value came from since the JavaThread wasn't > > around when the ThreadsList::find_JavaThread_from_java_tid() > > call was made with that target java_tid value... > > > > L99: // being concurently populated during the initalization. > > Typos? Perhaps: > > // to be concurrently populated during initialization. > > > > But I think those two comment lines are more appropriate above > > this line: > > > > L96: MutexLocker ml(ThreadTableCreate_lock); > > > > L112: // Must be inside the lock to ensure that we don't > > add the thread to the table > > typo: s/the thread/a thread/ > > > > L141: return ((double)_items_count)/_current_size; > > nit - need spaces around '/'. > > > > L177: bool equals(ThreadTableEntry **value, bool* is_dead) { > > nit - put space between '**' the variable: > > bool equals(ThreadTableEntry** value, > > > > Parameter 'is_dead' is not used. > > > > L214: while(true) { > > nit - space before '('. > > > > > > Short version: Thumbs up. > > > > Longer version: I don't think I've spotted anything other than nits here. > > Mostly I've just looked for multi-threaded races, proper usage of the > > Thread-SMR stuff, and minimal impact in the case where the new > > ThreadsTable is never needed. > > > > Dan > > > > P.S. > > ThreadTable is a bit of misnomer. What you really have here is > > a ThreadIdTable, but I'm really late to the code review flow > > with that comment... > > > > > > > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 > > > > > > Thank you! > > > --Daniil > > > > > > > > > > > > From serguei.spitsyn at oracle.com Fri Sep 27 18:08:17 2019 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Sep 2019 11:08:17 -0700 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> Message-ID: <35780c4b-1399-f585-cc41-b4c06a93c45d@oracle.com> On 9/27/19 11:06, Daniel D. Daugherty wrote: > On 9/27/19 1:58 PM, serguei.spitsyn at oracle.com wrote: >> Hi Daniil, >> >> Just notice I did not reply to you. >> Thank you for the explanation! >> >> Have you already pushed this one? > > Pushed on 2019.09.25 at 1416 ET. It has made it thru Tier7 testing > as of yesterday... Nice. Thanks, Dan! Serguei > > Dan > > >> >> Thanks, >> Serguei >> >> >> On 9/24/19 12:46, Daniil Titov wrote: >>> Hi Serguei, >>> >>> Thank you for reviewing this version of the fix. >>> >>>> ??? Just one question about ThreadIdTable::remove_thread(jlong tid). >>>> ??? What happens if there is no thread with the specified tid in >>>> ThreadIdTable? >>>> ??? Is it possible? >>> It could be possible when the thread that was started while the >>> thread table >>> was initializing exits.? At this point the thread table is >>> initialized and the thread >>> tries to remove itself from it. Removing non-existing? entry from >>> ConcurrentHashTable >>> is a correct operation that just leaves the table unchanged. >>> >>> src/hotspot/share/services/threadIdTable.cpp >>> >>> ??? 233??? bool ThreadIdTable::remove_thread(jlong tid) { >>> ??? 234????? assert(_is_initialized, "Thread table is not >>> initialized"); >>> ??? 235????? Thread* thread = Thread::current(); >>> ??? 236????? ThreadIdTableLookup lookup(tid); >>> ??? 237????? return _local_table->remove(thread, lookup); >>> ??? 238??? } >>> >>> src/hotspot/share/utilities/concurrentHashTable.hpp >>> >>> ?? 422????? // Returns true if items was deleted matching >>> LOOKUP_FUNC and >>> ??? 423????? // prior to destruction DELETE_FUNC is called. >>> ??? 424????? template >>> ??? 425????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, >>> DELETE_FUNC& del_f) { >>> ??? 426??????? return internal_remove(thread, lookup_f, del_f); >>> ??? 427????? } >>> ??? 428 >>> ??? 429????? // Same without DELETE_FUNC. >>> ??? 430????? template >>> ??? 431????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { >>> ??? 432??????? return internal_remove(thread, lookup_f, noOp); >>> ??? 433????? } >>> >>> src/hotspot/share/utilities/concurrentHashTable.inline.hpp >>> >>> ??? 446??? inline bool ConcurrentHashTable:: >>> ??? 447????? internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, >>> DELETE_FUNC& delete_f) >>> ??? 448??? { >>> ??? 449????? Bucket* bucket = get_bucket_locked(thread, >>> lookup_f.get_hash()); >>> ??? 450????? assert(bucket->is_locked(), "Must be locked."); >>> ??? 451????? Node* const volatile * rem_n_prev = bucket->first_ptr(); >>> ??? 452????? Node* rem_n = bucket->first(); >>> ??? 453????? bool have_dead = false; >>> ??? 454????? while (rem_n != NULL) { >>> ??? 455??????? if (lookup_f.equals(rem_n->value(), &have_dead)) { >>> ??? 456 bucket->release_assign_node_ptr(rem_n_prev, rem_n->next()); >>> ??? 457????????? break; >>> ??? 458??????? } else { >>> ??? 459????????? rem_n_prev = rem_n->next_ptr(); >>> ??? 460????????? rem_n = rem_n->next(); >>> ??? 461??????? } >>> ??? 462????? } >>> ??? 463 >>> ??? 464????? bucket->unlock(); >>> ??? 465 >>> ??? 466????? if (rem_n == NULL) { >>> ??? 467??????? return false; >>> ??? 468????? } >>> >>> Best regards, >>> Daniil >>> >>> >>> ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" >>> wrote: >>> >>> ???? Hi Daniil, >>> ???? ???? This version looks good to me. >>> ???? Thank you for the update! >>> ???? ???? Just one question about ThreadIdTable::remove_thread(jlong >>> tid). >>> ???? What happens if there is no thread with the specified tid in >>> ThreadIdTable? >>> ???? Is it possible? >>> ???? ???? Thanks, >>> ???? Serguei >>> ???? ???? On 9/24/19 9:36 AM, Daniil Titov wrote: >>> ???? > Hi Daniel, David and Serguei, >>> ???? > >>> ???? > Please review a new version of the fix (webrev.08) that as >>> Daniel suggested renames >>> ???? > ThreadTable to ThreadIdTable (related classes and variables >>> are renamed as well) and >>> ???? > corrects formatting issues. There are no other changes in >>> this webrev.08 comparing >>> ???? > to the previous version webrev.07. >>> ???? > >>> ???? > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >>> successfully passed. >>> ???? > >>> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >>> ???? > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> ???? > >>> ???? > Thank you! >>> ???? > >>> ???? > Best regards, >>> ???? > Daniil >>> ???? > >>> ???? > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" >>> wrote: >>> ???? > >>> ???? >????? Daniil, >>> ???? > >>> ???? >????? Thanks for sticking with this project through the many >>> versions. >>> ???? >????? Sorry this review is late... >>> ???? > >>> ???? > >>> ???? >????? On 9/19/19 8:30 PM, Daniil Titov wrote: >>> ???? >????? > Hi David and Serguei, >>> ???? >????? > >>> ???? >????? > Please review new version of the fix that includes the >>> changes Serguei suggested: >>> ???? >????? >?? 1. If racing threads initialize the thread table >>> only one of these threads will populate the table with the threads >>> from the thread list >>> ???? >????? >?? 2. The code that adds the thread to the tread table >>> is put inside Threads_lock to ensure that we cannot accidentally add >>> the thread >>> ???? >????? >?????? that has just passed the removal point in >>> ThreadsSMRSupport::remove_thread() >>> ???? >????? > >>> ???? >????? > The changes are in ThreadTable::lazy_initialize() >>> method only. >>> ???? >????? > >>> ???? >????? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 >>> tests successfully passed. >>> ???? >????? > >>> ???? >????? > Webrev: >>> https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >>> ???? > >>> ???? >????? src/hotspot/share/runtime/mutexLocker.hpp >>> ???? >?????????? No comments. >>> ???? > >>> ???? >????? src/hotspot/share/runtime/mutexLocker.cpp >>> ???? >?????????? No comments. >>> ???? > >>> ???? >????? src/hotspot/share/runtime/threadSMR.cpp >>> ???? >?????????? L623:???????? MutexLocker ml(Threads_lock); >>> ???? >?????????? L626:???????? if (!thread->is_exiting()) { >>> ???? >?????????????? Re: discussion about is_exiting() >>> ???? > >>> ???? >?????????????? The header comment is pretty clear: >>> ???? > >>> ???? > src/hotspot/share/runtime/thread.hpp: >>> ???? > >>> ???? >?????????????????? // thread has called JavaThread::exit() or >>> is terminated >>> ???? >?????????????????? bool is_exiting() const; >>> ???? > >>> ???? >?????????????? is_exiting() might become true right after you >>> have called it, >>> ???? >?????????????? but its purpose is to ask the question and not >>> prevent the >>> ???? >?????????????? condition from becoming true. As David said, >>> you should consider >>> ???? >?????????????? it an optimization. If you happen to see the >>> condition is true, >>> ???? >?????????????? then you know that the JavaThread isn't going >>> to be around much >>> ???? >?????????????? longer and should act accordingly. >>> ???? > >>> ???? >?????????????? The is_exiting() implementation is: >>> ???? > >>> ???? >???????????????? inline bool JavaThread::is_exiting() const { >>> ???? >?????????????????? // Use load-acquire so that setting of >>> _terminated by >>> ???? >?????????????????? // JavaThread::exit() is seen more quickly. >>> ???? >?????????????????? TerminatedTypes l_terminated = >>> (TerminatedTypes) >>> ???? > OrderAccess::load_acquire((volatile jint *) &_terminated); >>> ???? >?????????????????? return l_terminated == _thread_exiting || >>> ???? >????? check_is_terminated(l_terminated); >>> ???? >???????????????? } >>> ???? > >>> ???? >?????????????? and it depends on the JavaThread's _terminated >>> field value. >>> ???? > >>> ???? >???????????????? // JavaThread termination support >>> ???? >???????????????? enum TerminatedTypes { >>> ???? >????????????????? _not_terminated = 0xDEAD - 2, >>> ???? > _thread_exiting,???????????????????????????? // >>> ???? >????? JavaThread::exit() has been called for this thread >>> ???? > _thread_terminated,????????????????????????? // JavaThread >>> ???? >????? is removed from thread list >>> ???? > _vm_exited?????????????????????????????????? // JavaThread >>> ???? >????? is still executing native code, but VM is terminated >>> > // only VM_Exit >>> ???? >????? can set _vm_exited >>> ???? >???????????????? }; >>> ???? > >>> ???? >?????????????? so the JavaThread's _terminated field can get >>> set to >>> ???? >?????????????? _thread_exiting independent of the >>> Threads_lock, but >>> ???? >?????????????? it can't get set to _thread_terminated without the >>> ???? >?????????????? Threads_lock. >>> ???? > >>> ???? >?????????????? So by grabbing the Threads_lock on L623, you >>> make sure >>> ???? >?????????????? that ThreadTable::add_thread(java_tid, thread) >>> does not >>> ???? >?????????????? add a JavaThread that's not on the ThreadsList. >>> It might >>> ???? >?????????????? still become is_exiting() == true right after your >>> ???? > >>> ???? >???????????????? L626???????? if (!thread->is_exiting()) { >>> ???? > >>> ???? >?????????????? but it will still be on the main ThreadsList. >>> And that >>> ???? >?????????????? means that when the JavaThread is removed from >>> the main >>> ???? >?????????????? ThreadsList, you'll still call: >>> ???? > >>> ???? >???????????????? L931: ThreadTable::remove_thread(tid); >>> ???? > >>> ???? >?????????? L624:???????? // Must be inside the lock to ensure >>> that we don't >>> ???? >????? add the thread to the table >>> ???? >?????????????? typo: s/the thread/a thread/ >>> ???? > >>> ???? >?????????? L633:?????? return thread; >>> ???? >?????????????? nit - L633 - indented too far (should be 2 spaces) >>> ???? > >>> ???? >????? src/hotspot/share/services/threadTable.hpp >>> ???? >?????????? L42:?? static void lazy_initialize(const >>> ThreadsList *threads); >>> ???? >?????????????? nit - put space between '*' the variable: >>> ???? > >>> ???? >???????????????? static void lazy_initialize(const >>> ThreadsList* threads); >>> ???? > >>> ???? >?????????????? like you do in your other decls. >>> ???? > >>> ???? >?????????? L45:?? // Lookup and inserts >>> ???? >?????????????? Perhaps:? // Lookup and list management >>> ???? > >>> ???? >?????????? L60-61 - nit - please delete these blank lines. >>> ???? > >>> ???? >????? src/hotspot/share/services/threadTable.cpp >>> ???? >?????????? L28: #include "runtime/timerTrace.hpp" >>> ???? >?????????????? nit - This should be after threadSMR.hpp... >>> (alpha sorted order) >>> ???? > >>> ???? >?????????? L39: static const size_t DefaultThreadTableSizeLog >>> = 8; >>> ???? >?????????????? nit - your other 'static const' are not >>> CamelCase. Why is this one? >>> ???? > >>> ???? >?????????? L45: static ThreadTableHash* volatile _local_table >>> = NULL; >>> ???? >?????????? L50: static volatile size_t _current_size = 0; >>> ???? >?????????? L51: static volatile size_t _items_count = 0; >>> ???? >?????????????? nit - can you group the file statics together? >>> (up with L41). >>> ???? > >>> ???? >?????????? L60: _tid(tid),_java_thread(java_thread) {} >>> ???? >?????????????? nit - space after ',' >>> ???? > >>> ???? >?????????? L62?? jlong tid() const { return _tid;} >>> ???? >?????????? L63?? JavaThread* thread() const {return >>> _java_thread;} >>> ???? >?????????????? nit - space before '}' >>> ???? >?????????????? nit - space after '{' on L63. >>> ???? > >>> ???? >?????????? L70:???? static uintx get_hash(Value const& value, >>> bool* is_dead) { >>> ???? >?????????????? Parameter 'is_dead' is not used. >>> ???? > >>> ???? >?????????? L74:???? static void* allocate_node(size_t size, >>> Value const& value) { >>> ???? >?????????????? Parameter 'value' is not used. >>> ???? > >>> ???? >?????????? L93: void ThreadTable::lazy_initialize(const >>> ThreadsList *threads) { >>> ???? >?????????????? Re: discussion about lazy_initialize() racing with >>> ???? > ThreadsList::find_JavaThread_from_java_tid() >>> ???? > >>> ???? >?????????????? There's a couple of aspects to these two pieces >>> of code racing >>> ???? >?????????????? with each other and racing with new thread >>> creation. Racing with >>> ???? >?????????????? new thread creation is the easy one: >>> ???? > >>> ???? >???????????????? If a new thread isn't added to the >>> ThreadTable by >>> ???? >???????????????? ThreadsSMRSupport::add_thread() calling >>> ???? >????? ThreadTable::add_thread(), >>> ???? >???????????????? then the point in the future where someone calls >>> ???? >???????????????? find_JavaThread_from_java_tid() will add it >>> to the table due to >>> ???? >???????????????? the linear search when >>> ThreadTable::find_thread_by_tid() >>> ???? >???????????????? returns NULL. >>> ???? > >>> ???? >????????????? As for multi-threads calling >>> ???? >????? ThreadsList::find_JavaThread_from_java_tid() >>> ???? >????????????? at the same time which results in multi-threads >>> in lazy_initialize() >>> ???? >????????????? at the same time... >>> ???? > >>> ???? >????????????? - ThreadTable creation will be linear due to >>> ThreadTableCreate_lock. >>> ???? >??????????????? After _is_initialized is set to true, then no >>> more callers to >>> ???? >??????????????? lazy_initialize() will be in the "if >>> (!_is_initialized)" block. >>> ???? >????????????? - Once the ThreadTable is created, then >>> multi-threads can be >>> ???? >??????????????? executing the for-loop to add their >>> ThreadsList entries to >>> ???? >??????????????? the ThreadTable. There will be a bit of >>> Threads_lock contention >>> ???? >??????????????? as each of the multi-threads tries to add >>> their entries and >>> ???? >??????????????? there will be some wasted work since the >>> multi-threads will >>> ???? >??????????????? likely have similar ThreadLists. >>> ???? > >>> ???? >????????????? Of course, once _is_initialized is set to true, >>> then any caller >>> ???? >????????????? to lazy_initialize() will return quickly and >>> ???? > ThreadsList::find_JavaThread_from_java_tid() will call >>> ???? >????????????? ThreadTable::find_thread_by_tid(). If the target >>> java_tid isn't >>> ???? >????????????? found, then we do the linear search thing here >>> and add the >>> ???? >????????????? the entry if we find a match in our current >>> ThreadsList. Since >>> ???? >????????????? we're only adding the one here, we only contend >>> for the Threads_lock >>> ???? >????????????? here if we find it. >>> ???? > >>> ???? >????????????? If ThreadsList::find_JavaThread_from_java_tid() >>> is called with a >>> ???? >????????????? target java_tid for a JavaThread that was >>> created after the >>> ???? >????????????? ThreadsList object that the caller has in hand >>> for the >>> ???? >????????????? find_JavaThread_from_java_tid() call, then, of >>> course, that >>> ???? >????????????? target 'java_tid' won't be found because the >>> JavaThread was >>> ???? >????????????? added the main ThreadsList _after_ the >>> ThreadsList object was >>> ???? >????????????? created by the caller. Of course, you have to >>> ask where the >>> ???? >????????????? target java_tid value came from since the >>> JavaThread wasn't >>> ???? >????????????? around when the >>> ThreadsList::find_JavaThread_from_java_tid() >>> ???? >????????????? call was made with that target java_tid value... >>> ???? > >>> ???? >?????????? L99:???????? // being concurently populated during >>> the initalization. >>> ???? >?????????????? Typos? Perhaps: >>> ???? >??????????????????????? // to be concurrently populated during >>> initialization. >>> ???? > >>> ???? >?????????????? But I think those two comment lines are more >>> appropriate above >>> ???? >?????????????? this line: >>> ???? > >>> ???? >?????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >>> ???? > >>> ???? >?????????? L112:?????????? // Must be inside the lock to >>> ensure that we don't >>> ???? >????? add the thread to the table >>> ???? >?????????????? typo: s/the thread/a thread/ >>> ???? > >>> ???? >?????????? L141:?? return ((double)_items_count)/_current_size; >>> ???? >?????????????? nit - need spaces around '/'. >>> ???? > >>> ???? >?????????? L177:?? bool equals(ThreadTableEntry **value, bool* >>> is_dead) { >>> ???? >?????????????? nit - put space between '**' the variable: >>> ???? >?????????????????? bool equals(ThreadTableEntry** value, >>> ???? > >>> ???? >?????????????? Parameter 'is_dead' is not used. >>> ???? > >>> ???? >?????????? L214:?? while(true) { >>> ???? >?????????????? nit - space before '('. >>> ???? > >>> ???? > >>> ???? >????? Short version: Thumbs up. >>> ???? > >>> ???? >????? Longer version: I don't think I've spotted anything >>> other than nits here. >>> ???? >????? Mostly I've just looked for multi-threaded races, proper >>> usage of the >>> ???? >????? Thread-SMR stuff, and minimal impact in the case where >>> the new >>> ???? >????? ThreadsTable is never needed. >>> ???? > >>> ???? >????? Dan >>> ???? > >>> ???? >????? P.S. >>> ???? >????? ThreadTable is a bit of misnomer. What you really have >>> here is >>> ???? >????? a ThreadIdTable, but I'm really late to the code review >>> flow >>> ???? >????? with that comment... >>> ???? > >>> ???? > >>> ???? >????? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >>> ???? >????? > >>> ???? >????? > Thank you! >>> ???? >????? > --Daniil >>> ???? > >>> ???? > >>> ???? > >>> ???? > >>> >>> >> >> > From daniel.daugherty at oracle.com Fri Sep 27 18:06:12 2019 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 27 Sep 2019 14:06:12 -0400 Subject: jmx-dev RFR: 8185005: Improve performance of ThreadMXBean.getThreadInfo(long ids[], int maxDepth) In-Reply-To: References: <4C4212D0-BFFF-4C85-ACC6-05200F220C3F@oracle.com> <76BCC96D-DB5D-409A-95D5-3A64B893832D@oracle.com> <7e0ba39e-e5b7-f56b-66ea-820a0a35ec2c@oracle.com> <87748188-3BD4-4A8B-938A-89DBC8F3C57A@oracle.com> <1D2CC008-A509-4B0B-A8C7-75C1F94545AD@oracle.com> <0105ea55-9d9c-ca09-53af-3e9863e78e95@oracle.com> <5560D680-CD20-442F-8902-7F7034B0736A@oracle.com> <20b82205-c700-7a1a-d9bb-20f8b8873de2@oracle.com> <20D22E3B-1F3B-40CC-8544-482DC92909E6@oracle.com> <7c34198f-e423-cb75-3f75-500b8786ae27@oracle.com> Message-ID: <88db584d-09c5-d369-ccfc-4966cf36c461@oracle.com> On 9/27/19 1:58 PM, serguei.spitsyn at oracle.com wrote: > Hi Daniil, > > Just notice I did not reply to you. > Thank you for the explanation! > > Have you already pushed this one? Pushed on 2019.09.25 at 1416 ET. It has made it thru Tier7 testing as of yesterday... Dan > > Thanks, > Serguei > > > On 9/24/19 12:46, Daniil Titov wrote: >> Hi Serguei, >> >> Thank you for reviewing this version of the fix. >> >>> ??? Just one question about ThreadIdTable::remove_thread(jlong tid). >>> ??? What happens if there is no thread with the specified tid in >>> ThreadIdTable? >>> ??? Is it possible? >> It could be possible when the thread that was started while the >> thread table >> was initializing exits.? At this point the thread table is >> initialized and the thread >> tries to remove itself from it. Removing non-existing? entry from >> ConcurrentHashTable >> is a correct operation that just leaves the table unchanged. >> >> src/hotspot/share/services/threadIdTable.cpp >> >> ??? 233??? bool ThreadIdTable::remove_thread(jlong tid) { >> ??? 234????? assert(_is_initialized, "Thread table is not initialized"); >> ??? 235????? Thread* thread = Thread::current(); >> ??? 236????? ThreadIdTableLookup lookup(tid); >> ??? 237????? return _local_table->remove(thread, lookup); >> ??? 238??? } >> >> src/hotspot/share/utilities/concurrentHashTable.hpp >> >> ?? 422????? // Returns true if items was deleted matching LOOKUP_FUNC >> and >> ??? 423????? // prior to destruction DELETE_FUNC is called. >> ??? 424????? template >> ??? 425????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f, >> DELETE_FUNC& del_f) { >> ??? 426??????? return internal_remove(thread, lookup_f, del_f); >> ??? 427????? } >> ??? 428 >> ??? 429????? // Same without DELETE_FUNC. >> ??? 430????? template >> ??? 431????? bool remove(Thread* thread, LOOKUP_FUNC& lookup_f) { >> ??? 432??????? return internal_remove(thread, lookup_f, noOp); >> ??? 433????? } >> >> src/hotspot/share/utilities/concurrentHashTable.inline.hpp >> >> ??? 446??? inline bool ConcurrentHashTable:: >> ??? 447????? internal_remove(Thread* thread, LOOKUP_FUNC& lookup_f, >> DELETE_FUNC& delete_f) >> ??? 448??? { >> ??? 449????? Bucket* bucket = get_bucket_locked(thread, >> lookup_f.get_hash()); >> ??? 450????? assert(bucket->is_locked(), "Must be locked."); >> ??? 451????? Node* const volatile * rem_n_prev = bucket->first_ptr(); >> ??? 452????? Node* rem_n = bucket->first(); >> ??? 453????? bool have_dead = false; >> ??? 454????? while (rem_n != NULL) { >> ??? 455??????? if (lookup_f.equals(rem_n->value(), &have_dead)) { >> ??? 456????????? bucket->release_assign_node_ptr(rem_n_prev, >> rem_n->next()); >> ??? 457????????? break; >> ??? 458??????? } else { >> ??? 459????????? rem_n_prev = rem_n->next_ptr(); >> ??? 460????????? rem_n = rem_n->next(); >> ??? 461??????? } >> ??? 462????? } >> ??? 463 >> ??? 464????? bucket->unlock(); >> ??? 465 >> ??? 466????? if (rem_n == NULL) { >> ??? 467??????? return false; >> ??? 468????? } >> >> Best regards, >> Daniil >> >> >> ?On 9/24/19, 11:35 AM, "serguei.spitsyn at oracle.com" >> wrote: >> >> ???? Hi Daniil, >> ???? ???? This version looks good to me. >> ???? Thank you for the update! >> ???? ???? Just one question about ThreadIdTable::remove_thread(jlong >> tid). >> ???? What happens if there is no thread with the specified tid in >> ThreadIdTable? >> ???? Is it possible? >> ???? ???? Thanks, >> ???? Serguei >> ???? ???? On 9/24/19 9:36 AM, Daniil Titov wrote: >> ???? > Hi Daniel, David and Serguei, >> ???? > >> ???? > Please review a new version of the fix (webrev.08) that as >> Daniel suggested renames >> ???? > ThreadTable to ThreadIdTable (related classes and variables >> are renamed as well) and >> ???? > corrects formatting issues. There are no other changes in this >> webrev.08 comparing >> ???? > to the previous version webrev.07. >> ???? > >> ???? > Testing: Mach5 tier1, tier2, tier3, tier4, and tier5 tests >> successfully passed. >> ???? > >> ???? > Webrev: https://cr.openjdk.java.net/~dtitov/8185005/webrev.08/ >> ???? > Bug: : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? > >> ???? > Thank you! >> ???? > >> ???? > Best regards, >> ???? > Daniil >> ???? > >> ???? > ?On 9/20/19, 2:59 PM, "Daniel D. Daugherty" >> wrote: >> ???? > >> ???? >????? Daniil, >> ???? > >> ???? >????? Thanks for sticking with this project through the many >> versions. >> ???? >????? Sorry this review is late... >> ???? > >> ???? > >> ???? >????? On 9/19/19 8:30 PM, Daniil Titov wrote: >> ???? >????? > Hi David and Serguei, >> ???? >????? > >> ???? >????? > Please review new version of the fix that includes the >> changes Serguei suggested: >> ???? >????? >?? 1. If racing threads initialize the thread table only >> one of these threads will populate the table with the threads from >> the thread list >> ???? >????? >?? 2. The code that adds the thread to the tread table >> is put inside Threads_lock to ensure that we cannot accidentally add >> the thread >> ???? >????? >?????? that has just passed the removal point in >> ThreadsSMRSupport::remove_thread() >> ???? >????? > >> ???? >????? > The changes are in ThreadTable::lazy_initialize() >> method only. >> ???? >????? > >> ???? >????? > Testing:? Mach5 tier1, tier2, tier3, tier4, and tier5 >> tests successfully passed. >> ???? >????? > >> ???? >????? > Webrev: >> https://cr.openjdk.java.net/~dtitov/8185005/webrev.07/ >> ???? > >> ???? >????? src/hotspot/share/runtime/mutexLocker.hpp >> ???? >?????????? No comments. >> ???? > >> ???? >????? src/hotspot/share/runtime/mutexLocker.cpp >> ???? >?????????? No comments. >> ???? > >> ???? >????? src/hotspot/share/runtime/threadSMR.cpp >> ???? >?????????? L623:???????? MutexLocker ml(Threads_lock); >> ???? >?????????? L626:???????? if (!thread->is_exiting()) { >> ???? >?????????????? Re: discussion about is_exiting() >> ???? > >> ???? >?????????????? The header comment is pretty clear: >> ???? > >> ???? >???????????????? src/hotspot/share/runtime/thread.hpp: >> ???? > >> ???? >?????????????????? // thread has called JavaThread::exit() or >> is terminated >> ???? >?????????????????? bool is_exiting() const; >> ???? > >> ???? >?????????????? is_exiting() might become true right after you >> have called it, >> ???? >?????????????? but its purpose is to ask the question and not >> prevent the >> ???? >?????????????? condition from becoming true. As David said, you >> should consider >> ???? >?????????????? it an optimization. If you happen to see the >> condition is true, >> ???? >?????????????? then you know that the JavaThread isn't going to >> be around much >> ???? >?????????????? longer and should act accordingly. >> ???? > >> ???? >?????????????? The is_exiting() implementation is: >> ???? > >> ???? >???????????????? inline bool JavaThread::is_exiting() const { >> ???? >?????????????????? // Use load-acquire so that setting of >> _terminated by >> ???? >?????????????????? // JavaThread::exit() is seen more quickly. >> ???? >?????????????????? TerminatedTypes l_terminated = >> (TerminatedTypes) >> ???? > OrderAccess::load_acquire((volatile jint *) &_terminated); >> ???? >?????????????????? return l_terminated == _thread_exiting || >> ???? >????? check_is_terminated(l_terminated); >> ???? >???????????????? } >> ???? > >> ???? >?????????????? and it depends on the JavaThread's _terminated >> field value. >> ???? > >> ???? >???????????????? // JavaThread termination support >> ???? >???????????????? enum TerminatedTypes { >> ???? >????????????????? _not_terminated = 0xDEAD - 2, >> ???? > _thread_exiting,???????????????????????????? // >> ???? >????? JavaThread::exit() has been called for this thread >> ???? > _thread_terminated,????????????????????????? // JavaThread >> ???? >????? is removed from thread list >> ???? > _vm_exited?????????????????????????????????? // JavaThread >> ???? >????? is still executing native code, but VM is terminated >> > // only VM_Exit >> ???? >????? can set _vm_exited >> ???? >???????????????? }; >> ???? > >> ???? >?????????????? so the JavaThread's _terminated field can get >> set to >> ???? >?????????????? _thread_exiting independent of the Threads_lock, >> but >> ???? >?????????????? it can't get set to _thread_terminated without the >> ???? >?????????????? Threads_lock. >> ???? > >> ???? >?????????????? So by grabbing the Threads_lock on L623, you >> make sure >> ???? >?????????????? that ThreadTable::add_thread(java_tid, thread) >> does not >> ???? >?????????????? add a JavaThread that's not on the ThreadsList. >> It might >> ???? >?????????????? still become is_exiting() == true right after your >> ???? > >> ???? >???????????????? L626???????? if (!thread->is_exiting()) { >> ???? > >> ???? >?????????????? but it will still be on the main ThreadsList. >> And that >> ???? >?????????????? means that when the JavaThread is removed from >> the main >> ???? >?????????????? ThreadsList, you'll still call: >> ???? > >> ???? >???????????????? L931: ThreadTable::remove_thread(tid); >> ???? > >> ???? >?????????? L624:???????? // Must be inside the lock to ensure >> that we don't >> ???? >????? add the thread to the table >> ???? >?????????????? typo: s/the thread/a thread/ >> ???? > >> ???? >?????????? L633:?????? return thread; >> ???? >?????????????? nit - L633 - indented too far (should be 2 spaces) >> ???? > >> ???? >????? src/hotspot/share/services/threadTable.hpp >> ???? >?????????? L42:?? static void lazy_initialize(const ThreadsList >> *threads); >> ???? >?????????????? nit - put space between '*' the variable: >> ???? > >> ???? >???????????????? static void lazy_initialize(const ThreadsList* >> threads); >> ???? > >> ???? >?????????????? like you do in your other decls. >> ???? > >> ???? >?????????? L45:?? // Lookup and inserts >> ???? >?????????????? Perhaps:? // Lookup and list management >> ???? > >> ???? >?????????? L60-61 - nit - please delete these blank lines. >> ???? > >> ???? >????? src/hotspot/share/services/threadTable.cpp >> ???? >?????????? L28: #include "runtime/timerTrace.hpp" >> ???? >?????????????? nit - This should be after threadSMR.hpp... >> (alpha sorted order) >> ???? > >> ???? >?????????? L39: static const size_t DefaultThreadTableSizeLog = 8; >> ???? >?????????????? nit - your other 'static const' are not >> CamelCase. Why is this one? >> ???? > >> ???? >?????????? L45: static ThreadTableHash* volatile _local_table = >> NULL; >> ???? >?????????? L50: static volatile size_t _current_size = 0; >> ???? >?????????? L51: static volatile size_t _items_count = 0; >> ???? >?????????????? nit - can you group the file statics together? >> (up with L41). >> ???? > >> ???? >?????????? L60:???? _tid(tid),_java_thread(java_thread) {} >> ???? >?????????????? nit - space after ',' >> ???? > >> ???? >?????????? L62?? jlong tid() const { return _tid;} >> ???? >?????????? L63?? JavaThread* thread() const {return _java_thread;} >> ???? >?????????????? nit - space before '}' >> ???? >?????????????? nit - space after '{' on L63. >> ???? > >> ???? >?????????? L70:???? static uintx get_hash(Value const& value, >> bool* is_dead) { >> ???? >?????????????? Parameter 'is_dead' is not used. >> ???? > >> ???? >?????????? L74:???? static void* allocate_node(size_t size, >> Value const& value) { >> ???? >?????????????? Parameter 'value' is not used. >> ???? > >> ???? >?????????? L93: void ThreadTable::lazy_initialize(const >> ThreadsList *threads) { >> ???? >?????????????? Re: discussion about lazy_initialize() racing with >> ???? > ThreadsList::find_JavaThread_from_java_tid() >> ???? > >> ???? >?????????????? There's a couple of aspects to these two pieces >> of code racing >> ???? >?????????????? with each other and racing with new thread >> creation. Racing with >> ???? >?????????????? new thread creation is the easy one: >> ???? > >> ???? >???????????????? If a new thread isn't added to the ThreadTable by >> ???? >???????????????? ThreadsSMRSupport::add_thread() calling >> ???? >????? ThreadTable::add_thread(), >> ???? >???????????????? then the point in the future where someone calls >> ???? >???????????????? find_JavaThread_from_java_tid() will add it to >> the table due to >> ???? >???????????????? the linear search when >> ThreadTable::find_thread_by_tid() >> ???? >???????????????? returns NULL. >> ???? > >> ???? >????????????? As for multi-threads calling >> ???? >????? ThreadsList::find_JavaThread_from_java_tid() >> ???? >????????????? at the same time which results in multi-threads >> in lazy_initialize() >> ???? >????????????? at the same time... >> ???? > >> ???? >????????????? - ThreadTable creation will be linear due to >> ThreadTableCreate_lock. >> ???? >??????????????? After _is_initialized is set to true, then no >> more callers to >> ???? >??????????????? lazy_initialize() will be in the "if >> (!_is_initialized)" block. >> ???? >????????????? - Once the ThreadTable is created, then >> multi-threads can be >> ???? >??????????????? executing the for-loop to add their ThreadsList >> entries to >> ???? >??????????????? the ThreadTable. There will be a bit of >> Threads_lock contention >> ???? >??????????????? as each of the multi-threads tries to add their >> entries and >> ???? >??????????????? there will be some wasted work since the >> multi-threads will >> ???? >??????????????? likely have similar ThreadLists. >> ???? > >> ???? >????????????? Of course, once _is_initialized is set to true, >> then any caller >> ???? >????????????? to lazy_initialize() will return quickly and >> ???? > ThreadsList::find_JavaThread_from_java_tid() will call >> ???? >????????????? ThreadTable::find_thread_by_tid(). If the target >> java_tid isn't >> ???? >????????????? found, then we do the linear search thing here >> and add the >> ???? >????????????? the entry if we find a match in our current >> ThreadsList. Since >> ???? >????????????? we're only adding the one here, we only contend >> for the Threads_lock >> ???? >????????????? here if we find it. >> ???? > >> ???? >????????????? If ThreadsList::find_JavaThread_from_java_tid() >> is called with a >> ???? >????????????? target java_tid for a JavaThread that was created >> after the >> ???? >????????????? ThreadsList object that the caller has in hand >> for the >> ???? >????????????? find_JavaThread_from_java_tid() call, then, of >> course, that >> ???? >????????????? target 'java_tid' won't be found because the >> JavaThread was >> ???? >????????????? added the main ThreadsList _after_ the >> ThreadsList object was >> ???? >????????????? created by the caller. Of course, you have to ask >> where the >> ???? >????????????? target java_tid value came from since the >> JavaThread wasn't >> ???? >????????????? around when the >> ThreadsList::find_JavaThread_from_java_tid() >> ???? >????????????? call was made with that target java_tid value... >> ???? > >> ???? >?????????? L99:???????? // being concurently populated during >> the initalization. >> ???? >?????????????? Typos? Perhaps: >> ???? >??????????????????????? // to be concurrently populated during >> initialization. >> ???? > >> ???? >?????????????? But I think those two comment lines are more >> appropriate above >> ???? >?????????????? this line: >> ???? > >> ???? >?????????????? L96:?????? MutexLocker ml(ThreadTableCreate_lock); >> ???? > >> ???? >?????????? L112:?????????? // Must be inside the lock to ensure >> that we don't >> ???? >????? add the thread to the table >> ???? >?????????????? typo: s/the thread/a thread/ >> ???? > >> ???? >?????????? L141:?? return ((double)_items_count)/_current_size; >> ???? >?????????????? nit - need spaces around '/'. >> ???? > >> ???? >?????????? L177:?? bool equals(ThreadTableEntry **value, bool* >> is_dead) { >> ???? >?????????????? nit - put space between '**' the variable: >> ???? >?????????????????? bool equals(ThreadTableEntry** value, >> ???? > >> ???? >?????????????? Parameter 'is_dead' is not used. >> ???? > >> ???? >?????????? L214:?? while(true) { >> ???? >?????????????? nit - space before '('. >> ???? > >> ???? > >> ???? >????? Short version: Thumbs up. >> ???? > >> ???? >????? Longer version: I don't think I've spotted anything other >> than nits here. >> ???? >????? Mostly I've just looked for multi-threaded races, proper >> usage of the >> ???? >????? Thread-SMR stuff, and minimal impact in the case where >> the new >> ???? >????? ThreadsTable is never needed. >> ???? > >> ???? >????? Dan >> ???? > >> ???? >????? P.S. >> ???? >????? ThreadTable is a bit of misnomer. What you really have >> here is >> ???? >????? a ThreadIdTable, but I'm really late to the code review flow >> ???? >????? with that comment... >> ???? > >> ???? > >> ???? >????? > Bug : https://bugs.openjdk.java.net/browse/JDK-8185005 >> ???? >????? > >> ???? >????? > Thank you! >> ???? >????? > --Daniil >> ???? > >> ???? > >> ???? > >> ???? > >> >> > >