RFR: 8020875 java/lang/management/ThreadMXBean/ResetPeakThreadCount.java fails intermittently

[David Holmes]

On 23/07/2013 7:24 PM, Jaroslav Bachorik wrote:
 > The result is that the offender is j.u.l.LogManager$Cleaner thread. I
 > am attaching the profiler snapshot (can be opened in eg. jvisualvm)

That doesn't quite make sense. The Cleaner thread is a shutdownhook, it 
should not be starting unless the VM is shutting down!

David
-----

> On Tue 23 Jul 2013 11:19:13 AM CEST, David Holmes wrote:
>> On 23/07/2013 6:29 PM, Jaroslav Bachorik wrote:
>>> On 07/23/2013 10:19 AM, David Holmes wrote:
>>>> Hi Jaroslav,
>>>>
>>>> On 22/07/2013 9:55 PM, Jaroslav Bachorik wrote:
>>>>> The java/lang/management/ThreadMXBean/ResetPeakThreadCount.java test
>>>>> seems to be failing intermittently.
>>>>>
>>>>> The test checks the functionality of the
>>>>> j.l.m.ThreadMXBean.resetPeakThreadCount() method. It does so by
>>>>> capturing the current value of "getPeakThreadCount()", starting a
>>>>> predefined number of the user threads, stopping them and resetting the
>>>>> stored peak value and making sure the new peak equals to the number of
>>>>> the actually running threads.
>>>>>
>>>>> The main problem is that it is not possible to prevent JVM to
>>>>> start/stop
>>>>> arbitrary system threads while executing the test. This might lead to
>>>>> small variations of the reported peak (a short-lived system thread is
>>>>> started while the batch of the user threads is running) or the
>>>>> expected
>>>>> number of running threads (again, a short-lived system thread is
>>>>> started
>>>>> at the moment the test asks for the number of running threads).
>>>>
>>>> Do you know what "system threads" these are? I would not expect VM
>>>> internal threads to be counted in getPeakThreadCount(), but even if
>>>> they
>>>> are I can't think of any short-lived threads that get created other
>>>> than
>>>> the Signal handling thread.
>>>
>>> Unfortunatelly I don't. Capturing the thread dump at the moment of
>>> discovering the discrepancy seems to to be too late. I tried monitoring
>>> the JVM under the test from external tools but it just brings more
>>> entropy to the result.
>>
>> We'd need to instrument the thread creation logic to keep a separate
>> record. Dtrace probes could probably do it - but the problem is
>> getting the test to fail.
>
> Well, while responding to the previous email I thought about yet
> another way to try to pinpoint the mysterious thread - I've tried NB
> profiler. It filters out it's own threads and can do thread monitoring
> at the same time as tracking the call tree.
>
> The result is that the offender is j.u.l.LogManager$Cleaner thread. I
> am attaching the profiler snapshot (can be opened in eg. jvisualvm)
>
>>
>>> I am completely relying on the JVM native thread accounting to be
>>> correct and accurate - that it reports the thread count peak based on
>>> the real data.
>>
>> The spec isn't clear but I would only expect these counters to apply
>> to Java threads not VM internal threads (compiler, gc etc). So I'd
>> really like to know what thread is messing up this count.
>
> I hope my previous finding makes this clearer.
>
> -JB-
>
>>
>> David
>>
>>> -JB-
>>>
>>>>
>>>>> The patch does not fix those shortcomings as it is not really possible
>>>>> to do given the nature of the JVM threading system. It rather tries to
>>>>> relax the conditions while still maintaining the ability to detect
>>>>> functional problems - eg. decreasing peak without explicitly resetting
>>>>> it and reporting false number of threads.
>>>>>
>>>>> The webrev is at:
>>>>> http://cr.openjdk.java.net/~jbachorik/8020875/webrev.00
>>>>
>>>> Seems reasonable.
>>>>
>>>> David
>>>> -----
>>>>
>>>>> Thanks,
>>>>>
>>>>> -JB-
>>>>>
>>>
>