RFR(S) 8130448: thread dump improvements, comment, additions, new diagnostics inspired by 8077392
Daniel D. Daugherty
daniel.daugherty at oracle.com
Thu Jul 9 15:17:48 UTC 2015
Here are a couple of examples of the new options at work in
the hunt for 8077392:
In this example, ObjectMonitor::Knob_ExitRelease == 1 and
ObjectMonitor::Knob_VerifyMatch == 1 so we catch a JavaThread
exiting while a Java monitor is locked:
config java.util.stream.LoggingTestCase.before(): success
INFO: unexpected locked object: - locked <0xfffffd7be05e9478> (a
java.util.stream.Nodes$CollectorTask$OfInt)
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (synchronizer.cpp:1658), pid=10768, tid=0x0000000000000049
# guarantee(ObjectMonitor::Knob_VerifyMatch == 0) failed: exiting
JavaThread=0x0000000001069800 unexpectedly owns
ObjectMonitor=0x000000000116de00
#
# JRE version: Java(TM) SE Runtime Environment (9.0) (build
1.9.0-internal-20150708151346.ddaugher.8130448_for_jdk9-b00)
# Java VM: Java HotSpot(TM) 64-Bit Server VM
(1.9.0-internal-20150708151346.ddaugher.8130448_for_jdk9-b00 mixed mode
solaris-amd64 compressed oops)
The "INFO" line shows details about the Object and the
guarantee() line shows info about the offending thread
and the ObjectMonitor itself. With the latest change, the
guarantee() will change to a fatal()...
In this example, ObjectMonitor::Knob_ExitRelease == 0,
ObjectMonitor::Knob_VerifyMatch == 0 and
ObjectMonitor::Knob_Verbose == 1 so when we hit the hang
in the hunt for 8077392, we'll see a more detailed thread
dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM
(1.9.0-internal-2015070815134
6.ddaugher.8130448_for_jdk9-b00 mixed mode):
"MainThread" #11 prio=5 os_prio=64 tid=0x0000000000a1e800 nid=0x43 in
Object.wai
t() [0xfffffd7bba9fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <no object reference available>
at
java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java
:334)
- waiting to re-lock in wait() <0xfffffd7be69350c8> (a
java.util.stream.
Nodes$CollectorTask$OfInt)
- lockbits=
monitor(0x0000000001975502)={count=0x0000000000000000,waiter
s=0x0000000000000001,recursions=0x0000000000000000,owner=0x0000000001311000}
at
java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
With the changes for this bug (8130448):
- This line is new:
- waiting on <no object reference available>
and it tells us that this compiled code instead of interpreted code.
Prior to 8130448, the "waiting on ..." line would not be present.
- This line is new:
- waiting to re-lock in wait() <0xfffffd7be69350c8> (a java.util.stream.
Nodes$CollectorTask$OfInt)
and it tells us that the Java monitor has been notified and the thread
is waiting to re-lock the Java monitor. Prior to 8130448, the "waiting
to re-lock ..." line would not be present.
- This line was new with David C's changes:
- lockbits= monitor(0x0000000001975502)={count=0x0000000000000000,waiter
s=0x0000000000000001,recursions=0x0000000000000000,owner=0x0000000001311000}
and it tells us which JavaThread owns the Java monitor:
0x0000000001311000. A search of the rest of the thread dump
would show no such thread exists. Adding instrumentation to
the thread-start and thread-exit code paths shows that such
a thread did exist and exited just before this hang happened.
I think that's it for a demo of the new options...
Dan
On 7/8/15 3:33 PM, Daniel D. Daugherty wrote:
> Greetings,
>
> I've updated this patch based on David H's comments along with
> some additional cleanup.
>
> This work is being tracked by the following bug ID:
>
> JDK-8130448 8077392 inspired thread dump improvements, comment
> additions, new diagnostics
> https://bugs.openjdk.java.net/browse/JDK-8130448
>
> Here is the webrev URL:
>
> http://cr.openjdk.java.net/~dcubed/8130448-webrev/1-jdk9-hs-rt/
>
> The easiest way to review the delta is to download the two patch
> files and compare them in something like jfilemerge:
>
> http://cr.openjdk.java.net/~dcubed/8130448-webrev/0-jdk9-hs-rt/hotspot.patch
>
> http://cr.openjdk.java.net/~dcubed/8130448-webrev/1-jdk9-hs-rt/hotspot.patch
>
>
> Testing:
>
> - Aurora Adhoc RT/SVC nightly batches (in process)
> - JPRT test jobs
>
> Thanks, in advance, for any comments, questions or suggestions.
>
> Gory details about the changes are below...
>
> Dan
>
> Changes between CR0 and CR1 are:
>
> src/cpu/x86/vm/macroAssembler_x86.cpp
>
> - fix comment typos, clarify comment wording
>
> src/share/vm/oops/markOop.cpp
>
> - clarify output messages
> - get rid of confusing local variable
>
> src/share/vm/runtime/globals.hpp
>
> - drop VerboseStackTrace, GuaranteeOnMonitorMismatch
> and JavaThreadExitReleasesMonitors options
> - there are no longer changes to this file
>
> src/share/vm/runtime/objectMonitor.cpp
>
> - add ObjectMonitor::Knob_ExitRelease suboption to replace
> JavaThreadExitReleasesMonitors
> - add ObjectMonitor::Knob_VerifyMatch suboption to replace
> GuaranteeOnMonitorMismatch
> - cleanup messy logging:
> - switch from '::printf()' -> 'tty->print_cr()'
> - switch from '::fflush()' -> 'tty->flush()'
>
> src/share/vm/runtime/objectMonitor.hpp
>
> - add ObjectMonitor::Knob_ExitRelease suboption
> - add ObjectMonitor::Knob_VerifyMatch suboption
> - cleanup messy logging
>
> src/share/vm/runtime/synchronizer.cpp
>
> - cleanup messy logging
> - switch from GuaranteeOnMonitorMismatch to
> ObjectMonitor::Knob_VerifyMatch
> - add diagnostic information about the locked object
> when ReleaseJavaMonitorsClosure::do_monitor() finds
> a problem
>
> src/share/vm/runtime/thread.cpp
>
> - clarify comments
> - switch from JavaThreadExitReleasesMonitors to
> ObjectMonitor::Knob_ExitRelease
>
> src/share/vm/runtime/vframe.cpp
>
> - print_locked_object_class_name() is now public
> - clarify comments
> - clarify output messages
> - switch from VerboseStackTrace to the already existing
> ObjectMonitor::Knob_Verbose
>
> src/share/vm/runtime/vframe.hpp
>
> - print_locked_object_class_name() is now public
>
>
> On 7/3/15 6:14 PM, Daniel D. Daugherty wrote:
>> Greetings,
>>
>> The hunt for the following bug:
>>
>> JDK-8077392 Stream fork/join tasks occasionally fail to complete
>>
>> and David C's work on the following bug:
>>
>> JDK-8069412 Locks need better debug-printing support
>>
>> have inspired additional thread dump improvements, comment additions
>> to some Java monitor code and some new diagnostic options.
>>
>> This work is being tracked by the following bug ID:
>>
>> JDK-8130448 8077392 inspired thread dump improvements, comment
>> additions, new diagnostics
>> https://bugs.openjdk.java.net/browse/JDK-8130448
>>
>> Here is the webrev URL:
>>
>> http://cr.openjdk.java.net/~dcubed/8130448-webrev/0-jdk9-hs-rt/
>>
>> Testing:
>>
>> - RBT vm.quick batches (in process)
>> - JPRT test jobs
>>
>> Thanks, in advance, for any comments, questions or suggestions.
>>
>> Gory details about the changes are below...
>>
>> Dan
>>
>> 8130448 summary of changes:
>>
>> src/cpu/x86/vm/macroAssembler_x86.cpp
>> - comment additions for the assembly code
>>
>> src/share/vm/oops/markOop.cpp
>> - has_monitor() has to be checked before is_locked() since
>> the has_monitor() bits are a superset of the is_locked() bits
>> - code style fixes
>>
>> src/share/vm/runtime/globals.hpp
>> - add VerboseStackTrace diagnostic option
>> - add GuaranteeOnMonitorMismatch diagnostic option
>> - add JavaThreadExitReleasesMonitors product option;
>> this is the Java equivalent of JNIDetachReleasesMonitors
>>
>> src/share/vm/runtime/objectMonitor.cpp
>> - delete unused ObjectMonitor::try_enter()
>> - fix assert wording
>>
>> src/share/vm/runtime/objectMonitor.hpp
>> - delete unused ObjectMonitor::try_enter()
>>
>> src/share/vm/runtime/synchronizer.cpp
>> - add GuaranteeOnMonitorMismatch support with some
>> diagnostic output
>>
>> src/share/vm/runtime/thread.cpp
>> - add JavaThreadExitReleasesMonitors support
>>
>> src/share/vm/runtime/vframe.cpp
>> - clarify existing comments
>> - add comments to clarify what "waiting on" means
>> - add line to show when we are "waiting on", but
>> there are no locals available (previously we were
>> "waiting on" silently)
>> - add support for "waiting to relock" which can happen
>> for any of the monitors and not just the top monitor
>> - switch mark->print_on() verbose output to new
>> VerboseStackTrace switch; thread dumps are not enabled
>> with a specific switch so the '-XX:+Verbose' model of
>> being a modifier for another option does not fit
>>
>>
>
>
>
More information about the hotspot-runtime-dev
mailing list