jmx-dev RFR: 8020875 java/lang/management/ThreadMXBean/ResetPeakThreadCount.java fails intermittently

[Chris Hegarty]

On 24/07/2013 15:08, Jaroslav Bachorik wrote:
> On 07/24/2013 03:17 PM, Chris Hegarty wrote:
>> On 24/07/2013 13:49, Jaroslav Bachorik wrote:
>>> On 07/24/2013 02:32 PM, Chris Hegarty wrote:
>>>> On 24/07/2013 12:21, David Holmes wrote:
>>>>> On 24/07/2013 7:31 PM, Mandy Chung wrote:
>>>>>>
>>>>>> On 7/24/2013 4:50 PM, shanliang wrote:
>>>>>>> So we have 2 kinds of issues here:
>>>>>>> 1) the test related, like Thread state checking, we can fix them in
>>>>>>> the test
>>>>>>> 2) MBean.getThreadCount() issue, we can create a bug to trace it (add
>>>>>>> your test case to the bug), and add a workaround (sleep or call 2
>>>>>>> times) in the test to make the test pass. Mandy is the expert and
>>>>>>> better to get her opinion.
>>>>>>
>>>>>> It's probably a race in the VM implementation in determining the
>>>>>> thread
>>>>>> count. You will need to diagnose the VM implementation and compare the
>>>>>> thread list and the implementation of getting the thread count (check
>>>>>> hotspot/src/share/vm/services/threadService.cpp)
>>>>>
>>>>> There is a considerable code path between the point where a terminating
>>>>> thread causes Thread.join() to be allowed to return, and the point
>>>>> where
>>>>> the live thread count gets decremented. So using join() does not help
>>>>> here. Arguably JVMTI should have based its counts around the lifecycle
>>>>> of the Java thread not the underlying native thread.
>>>>
>>>> It appears, from my reading of the code, that this situation ( a thread
>>>> exiting ) should be handled. Or maybe I'm looking at the wrong
>>>> interface.
>>>>
>>>> JavaThread::exit(...) {
>>>>     ...
>>>>     ThreadService::current_thread_exiting(this);
>>>>     ...
>>>>     ensure_join(..)
>>>>     ...
>>>> }
>>>>
>>>> So the exiting thread should be removed from the live thread count
>>>> before Thread.join returns.
>>>
>>> Unfortunately, ensure_join(...) is called on line 1860 but
>>> Threads::remove(this), which does the actual cleanup of the live threads
>>> counter, is called only on line 1919, leaving at least a few ns window
>>> when the thread is reported as terminated in java but the counters
>>> haven't been updated yet.
>>
>> Again, maybe I'm missing something but,
>>
>> static jlong get_live_thread_count()        { return
>> _live_threads_count->get_value() - _exiting_threads_count; }
>>
>>   ... and current_thread_exiting(..) increments _exiting_threads_count, no?
>
> Well, apparently it does.
>
> I am a complete stranger to the concurrency issues in the hotspot -
> would it be possible that in ThreadService::remove_thread(..) the
> _exiting_threads_count is decremented but _live_threads_count hasn't
> been updated yet when someone calls the get_live_thread_count() function?

I am not familiar with the intricate workings of this code, but as a 
casual observer I would say that this must be a bug in the VM. It 
appears that the original authors did take into account exiting threads, 
and went to some lengths to provide accurate diagnostic information. If 
this is not producing the correct results, then I can only imagine there 
is a bug here.

To your specific question, then yes this would appear possible. I am not 
sure what synchronization, if any, protects this code.

-Chris.


>
> -JB-
>
>>
>> -Chris.
>>
>>>
>>> -JB-
>>>
>>>>
>>>> -Chris.
>>>>
>>>>>
>>>>> David
>>>>> -----
>>>>>
>>>>>> Mandy
>>>
>