jcmd - executing multiple commands at the same safepoint?
Frederic Parain
frederic.parain at oracle.com
Mon Jun 18 20:40:41 UTC 2018
Hi Thomas,
Thank you for experimenting with the patch and providing feedback.
I’ve looked to VM.info, and the only issue seems to be this section
of vmError.cpp:
1064 if (Universe::is_fully_initialized()) {
1065 MutexLocker hl(Heap_lock);
1066 Universe::heap()->print_on_error(st);
1067 st->cr();
1068 st->print_cr("Polling page: " INTPTR_FORMAT, p2i(os::get_polling_page()));
1069 st->cr();
1070 }
It would be interesting to have more information from the GC team about
while taking the Heap_lock is required. If it is only to prevent mutator threads
to update the counters dumped by Universe::heap()->print_on_error(st); then
it would be safe to not take the lock when running at a safepoint:
1064 if (Universe::is_fully_initialized()) {
1065 if (SafepointSynchronize::is_at_safepoint()) {
1066 Universe::heap()->print_on_error(st);
1067 st->cr();
1068 st->print_cr("Polling page: " INTPTR_FORMAT, p2i(os::get_polling_page()));
1069 st->cr();
1070 } else {
1071 MutexLocker hl(Heap_lock);
1072 Universe::heap()->print_on_error(st);
1073 st->cr();
1074 st->print_cr("Polling page: " INTPTR_FORMAT, p2i(os::get_polling_page()));
1075 st->cr();
1076 }
1077 }
If the problem is different, we would need to know what concurrency issue is
addressed here, in order to determine what can be done when run at a safepoint.
Fred
> On Jun 15, 2018, at 14:09, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>
> Hi Frederic,
>
> this is nice and now I am surprised how small the change is :)
>
> I played around with it a bit, combining VM.metaspace, VM.classloaders
> and VM.systemdictionary, including options. All worked well.
> VM.stringtable did not produce output and VM.info, as you predicted,
> crashes the VM because it needs the heap lock.
>
> (Side note: I think VM.info is too useful to be held back by not
> getting the heap lock. It would be nice if it would try to lock and if
> failing, would just omit the steps needing a heap lock.)
>
> So, yes, on first glance I find this useful. I'll play around with
> this a bit more. Lets wait for others to chime in.
>
> Best Regards, Thomas
>
>
> On Fri, Jun 15, 2018 at 5:32 PM, Frederic Parain
> <frederic.parain at oracle.com> wrote:
>> Hi,
>>
>> Here’s a first prototype to explore running multiple commands in a single safepoint:
>>
>> http://cr.openjdk.java.net/~fparain/DCmdRunAtSafepoint/webrev.00/
>>
>> The syntax is very simple, with the character ‘+’ used as the delimiter between commands.
>>
>> $ ./build/macosx-x64-debug/images/jdk/bin/jcmd 21164 VM.run_at_safepoint VM.version + VM.uptime
>> 21164:
>> Java HotSpot(TM) 64-Bit Server VM version 11-internal+0-2018-05-25-2202478.fparain.DCmd
>> JDK 11.0.0
>> 44.143 s
>>
>> Playing a bit with it, it quickly shows what are the issues when trying to run
>> diagnostic commands at a safepoint.
>>
>> Issue 1: One important point to remember is that only the VMThread can run at a safepoint,
>> all JavaThreads are strictly forbidden to run during the safepoint. So executing commands at a
>> safepoint means they will be executed by the VMThread. Unfortunately, the VMThread is
>> subject to restrictions and cannot execute all code that a JavaThread can execute. This
>> issue is visible with the VM.info command, which takes the Heap_lock. The VMThread
>> is not allowed to grab this lock because it could cause the VMThread to be blocked by
>> a JavaThread. So far, VM.info cannot be run at a safepoint.
>>
>> Issue 2: Executing a code at a safepoint requires a VMOperation. VMOperations are
>> made of a prologue, a doit() method witch contains the code executed at the safepoint,
>> and an epilogue. The prologue and the epilogue are executed by the requester of the
>> VMOperation, usually a JavaThread, and they are not executed at a safepoint. Prologues
>> and epilogues are designed to execute code the VMThread cannot execute, like taking
>> locks or loading classes. When the VMThread itself wants to execute a VMOperation,
>> what is called a nested VMOperation, it only executes the doit() method, skipping the
>> prologue and the epilogue. This is an issue for diagnostic commands using a VMOp
>> in their implementation: when these commands are called while the JVM is already
>> at a safepoint, their prologue and epilogue cannot be executed. This issue impacts
>> Thread.print and GC.run.
>>
>> There are probably other issues still to be found, so I encourage people to test
>> this patch to see if the combinations of commands they would like to run at a
>> safepoint can indeed be run in this mode.
>>
>> Note that the support for invoking the new command from the DiagnosticCommand
>> MBean has not be implemented. This is a not negligible work in order to correctly
>> check Java Permissions before executing the commands.
>>
>> Let me know if you consider this code useful or if the limitations described above
>> make this feature not as pretty as we wanted it to be.
>>
>> Regards,
>>
>> Fred
>>
>>
>>> On May 29, 2018, at 15:54, Frederic Parain <frederic.parain at oracle.com> wrote:
>>>
>>> Hi Thomas,
>>>
>>>
>>>> On May 26, 2018, at 04:07, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>>>>
>>> <snip>
>>>>>
>>>>> I think that it is a fundamental piece of the design. Adding a new diagnostic
>>>>> command to group several commands under a unique safepoint:
>>>>> 1 - requires explicit action from the user (and acknowledgment of the risks)
>>>>> 2 - doesn’t touch the current syntax/semantic at all
>>>>>
>>>>> Another reason I’m pushing for a specific diagnostic command rather than
>>>>> changing the syntax is remote execution. jcmd is not the only way to invoke
>>>>> a diagnostic command, they can also be invoke remotely using JMX and the
>>>>> com.sun.management.DiagnosticCommand dynamic MBean. Through the
>>>>> platform MBean, users do not have access to the jcmd syntax, they see
>>>>> each diagnostic command as a MBean operation (providing that they have
>>>>> been registered with the DCmd_Source_MBean flag). The initial support
>>>>> for remote execution was taking a single String and processed it just like
>>>>> a string passed by jcmd, but people developing consoles for remote
>>>>> management complained that this was not convenient for them, and asked
>>>>> for something they could use more programmatically.
>>>>> If the safepoint bundling is implemented in the syntax, JMX clients won’t
>>>>> be able to use the feature. If we implement it with a new diagnostic command,
>>>>> it will be visible from the MBean and they will be able to use it.
>>>>
>>>> That is interesting. Do you have examples for such programs? I was
>>>> looking for ways of how to use jcmd (or equivalent) remotely.
>>>
>>> If you’re looking for a console application to invoke diagnostic command
>>> on a remote JVM, Mission Control has a dedicated support for that in the
>>> “Diagnostic Commands” tab.
>>> If you’re looking for exemples of source code doing remote invocations,
>>> there’re several unit tests for the DiagnosticCommand MBean in the directory
>>> test/jdk/com/sun/management/DiagnosticCommandMBean.
>>>
>>>>
>>>>>
>>>>> At last, creating a specific diagnostic command for safepoint bundling would
>>>>> show the way on how to group commands for consistency. Safepoints is not
>>>>> the only way to get consistency. For some JVM internals reasons, you might
>>>>> want to group commands under a single lock, like the Threads_lock. Adding
>>>>> a diagnostic command for each type of bundling seems more extensible.
>>>>>
>>>>
>>>> You make good points.
>>>>
>>>> Adding a new command has also the advantage that it does not affect
>>>> downward compatibility and hence would not require a CSR. CSR seems to
>>>> slow down things a bit (I have a very simple CSR for jcmd open
>>>> undecided since 12 days:
>>>> https://bugs.openjdk.java.net/browse/JDK-8203043). So, while I think
>>>> CSR are a necessary process, I am happy to avoid them if possible.
>>>>
>>>> Lets spin this out a bit more:
>>>>
>>>> -> However the user specifies "please execute these commands at a
>>>> single safepoint", if one of the enclosed commands cannot be run
>>>> inside a safepoint the whole bundle should fail, agreed? So,
>>>> specifying e.g. JFR.start as part of such a command bundle would cause
>>>> the whole bundle to not being executed.
>>>
>>> Yes. The new command N would ask for the execution of a list of
>>> commands { A, B, C, … }, and would fail if:
>>> - any of the specified commands { A, B, C, … } is not a registered
>>> command
>>> - any of the specified commands { A, B, C, … } doesn’t support
>>> execution at safepoint
>>> - any of the specified commands { A, B, C, … } is not exported to
>>> the DCmdSource used to invoke N (*)
>>>
>>> (*) Not all commands are exported to all interfaces (jcmd,JMX), for
>>> instance, some commands related to the JMX agent are cannot
>>> be invoked through the JMX. Even if the new command can be
>>> invoked from all interfaces, it must not allow to circumvent this
>>> restriction. This should not be an issue today, because all
>>> JMX agent commands are calling back Java code, and are not
>>> suitable for execution at a safepoint. But this would ensure the
>>> consistency of the model in the long term (and the test is cheap).
>>>
>>>>
>>>> -> If we introduce a command like you propose (lets call it "VM.run"
>>>> for the sake of the argument) we still need to decide how to delimit
>>>> the commands at the command line. That brings us back to the syntax
>>>> question: separation by a explicit delimiter, if possible one that
>>>> does not clash with common unix shells?
>>>>
>>>> Lets try colon ':' :
>>>>
>>>> jcmd VM.run VM.uptime : VM.metaspace show-loaders show-classes
>>>> by-chunktype : VM.classloaders show-classes : VM.info
>>>>
>>>> Of course, we can wrap all arguments to jcmd in quotes, then the
>>>> delimiter sign does not affect things as much, unless we use quotes :)
>>>> We could e.g. use semicolon:
>>>>
>>>> jcmd ' VM.run VM.uptime ; VM.metaspace show-loaders show-classes
>>>> by-chunktype ; VM.classloaders show-classes ; VM.info ‘
>>>>
>>>
>>> I thought about colon and semi-colon too, because they are the most
>>> obvious delimiters. However, they are already as delimiters in classpaths
>>> (semi-colon on Unix, colon on Windows), so parsing to find the right
>>> delimiters would be tricky.
>>>
>>> I’d start with ‘!’ or ‘+’ in a first time, to make some progress on the other
>>> aspects of the command, and buy some time to discuss the delimiter
>>> issue in parallel.
>>>
>>> The last resort would be to force each command in the bundle to
>>> be specified with quotes (arguments included). But it would make the
>>> syntax very heavy, I’d prefer a simpler delimiter.
>>>
>>>
>>>> Or, we go back to my original idea of letting the command keyword be
>>>> the indicator for a new command start. Honestly, at this point I have
>>>> no preference. Both approaches seem reasonable.
>>>>
>>>> The only niggle I have with an explicit command is the syntax, which I
>>>> find a bit awkward. My thinking before reading your response was that
>>>> I'd prefer a jcmd flag to specify the same behavior: that all commands
>>>> should be executed at a single safepoint. E.g. [-s/--stacked]
>>>>
>>>> jcmd -s ' VM.uptime ; VM.metaspace show-loaders show-classes
>>>> by-chunktype ; VM.classloaders show-classes ; VM.info '
>>>>
>>>> But I see the disadvantage: depending on how it is implemented this
>>>> would require consistent changes in both jcmd and hotspot, and not
>>>> work with other clients using MBeans. Whereas the hypothetical
>>>> "VM.run" would work out of the box for any old jcmd version and any
>>>> other client.
>>>>
>>>> Of course one could keep the "VM.run" idea and still give jcmd a
>>>> "-s/--stacked" option, as a shallow convenience feature, which just
>>>> translates to "VM.run" and runs that.
>>>>
>>>> -> A variation of your "have special commands to run a command bundle
>>>> in one particular context" theme could be having only one command,
>>>> e.g. "VM.run", specifying the conditions under which to run the bundle
>>>> as sub options. E.g. "VM.run -s <commands>" for "run commands at
>>>> safepoint", "VM.run -t <commands>" for "run commands under
>>>> Threads_lock". That would even allow to combine these contexts. I do
>>>> not know if that makes sense.
>>>>
>>>> - Another advantage of having an explicit command just occurring to me
>>>> is that it enables more evolved command files, where one can
>>>> intersperse command bundles and single commands. For instance, I could
>>>> give this to a customer:
>>>>
>>>> <command file>
>>>> 1 VM.uptime
>>>> 2 VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
>>>> VM.classloaders details VM.classloader_stats
>>>> 3 GC.run
>>>> 4 VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
>>>> VM.classloaders details VM.classloader_stats
>>>>
>>>> to execute a command bundle before and after triggering a GC.
>>>
>>> This would work fine . The parser of the jcmd string will split the
>>> string based on ‘\n’ characters, then each line of command will be executed
>>> sequentially, and lines containing the VM.run_command will then create the
>>> safepoint and trigger the execution of the different commands specified in
>>> its argument list.
>>>
>>>>
>>>>> We have a three days week-end in the US, but I should be able to code
>>>>> something next week.
>>>>>
>>>>
>>>> Oh, you want to take over the implementation, sure! Fine by me.
>>>
>>> I’m not promising an implementation, I haven’t worked on this code
>>> since I left the serviceability team. I need to code at least a skeleton
>>> of the command to refresh my memory about all aspects of the
>>> diagnostic command framework.
>>>
>>>>
>>>>> Thank you for your ideas and your interest in diagnostic commands.
>>>>
>>>> Sure. Why I am interested in this: We strongly rely on tools similar
>>>> to jcmd for our support. We have quite capable tools in our (licensed,
>>>> non-open) VM port, which are similar to jcmd in many aspects. So we
>>>> try to bring some of the capabilities upstream, in order to ease the
>>>> merge costs for us and to help the OpenJDK community.
>>>
>>> The diagnostic command framework was designed to support an
>>> extensible set of commands. Feel free to extend, and thank you
>>> for contributing to the OpenJDK community.
>>>
>>> Fred
>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> A new separator, more convenient than newline, would be require to have
>>>>>>> a single line syntax.
>>>>>>>
>>>>>>> My 2 cents,
>>>>>>>
>>>>>>
>>>>>> Certainly worth more than 2 cents :) Thanks for your thoughts!
>>>>>>
>>>>>> ..Thomas
>>>>>>
>>>>>>> Fred
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Erik
>>>>>>>>>
>>>>>>>>> Hi Thomas,
>>>>>>>>>
>>>>>>>>> very positive suggestion.
>>>>>>>>>
>>>>>>>>> One observation:
>>>>>>>>> There already seems to be some different interpretation of the command
>>>>>>>>> line
>>>>>>>>> in different Java versions:
>>>>>>>>> For instance when I try to dump a flight recording in 1.8.0_152-ea-b05 :
>>>>>>>>> I
>>>>>>>>> can use:
>>>>>>>>> jcmd 33014 JFR.dump filename="recording1.jfr" recording=1
>>>>>>>>> but in build 9+181 , the same command must be:
>>>>>>>>> jcmd 33014 JFR.dump filename="recording1.jfr",recording=1
>>>>>>>>> (the comma to separate sub-options you mentioned earlier)
>>>>>>>>>
>>>>>>>>> Therefore the suggestion without curly brackets, giving several commands
>>>>>>>>> after one another seems good with regard to backwards compatibility.
>>>>>>>>>
>>>>>>>>> Motivation: hawt.io uses the MBean
>>>>>>>>> com.sun.management:type=DiagnosticCommand
>>>>>>>>> to access the same functionality as jcmd. Variations in option syntax
>>>>>>>>> across
>>>>>>>>> Java versions is already an issue (only affecting sub-options though, as
>>>>>>>>> each command is a separate JMX operation). So syntax compatibility is
>>>>>>>>> highly
>>>>>>>>> appreciated :-)
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> lør. 12. mai 2018 kl. 20:11 skrev Kirk Pepperdine
>>>>>>>>> <kirk.pepperdine at gmail.com>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 10, 2018, at 11:26 AM, Thomas Stüfe <thomas.stuefe at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 10, 2018 at 9:13 AM, Kirk Pepperdine
>>>>>>>>>>> <kirk.pepperdine at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> The stacking at the safe point would be a huge win. Right now thread
>>>>>>>>>>>> dump consistency can really confuse people as the tooling will show
>>>>>>>>>>>> two
>>>>>>>>>>>> threads owning the same lock at seemingly the same time. Reality, it’s
>>>>>>>>>>>> just
>>>>>>>>>>>> that people didn’t realize you get a safe point for each thread
>>>>>>>>>>>> therefor
>>>>>>>>>>>> there is motion in the system as you’re collecting the data.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am a bit confused. What tooling are you talking about?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> jstack. I’ve not seen it with jcmd. But I often see 2 threads holding
>>>>>>>>>> the
>>>>>>>>>> same lock at the “same time” which is of course non-sense. I can dig one
>>>>>>>>>> up
>>>>>>>>>> for you if you’re still confused.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> As an aside, it’s amazing how many dev’s aren’t aware of jcmd. Just
>>>>>>>>>>>> yesterday after my session at Devoxx I had someone ask me about using
>>>>>>>>>>>> jfr
>>>>>>>>>>>> from the command line, many in that session had not seen jcmd before.
>>>>>>>>>>>> The
>>>>>>>>>>>> feedback was, wow, this is very useful and I wished I had of known
>>>>>>>>>>>> about it
>>>>>>>>>>>> earlier.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes, jcmd is quite useful. I also really like the simple design, which
>>>>>>>>>>> is by default up- and downward compatible (so you can talk to any VM,
>>>>>>>>>>> new, old, it should not matter). That is just good design. We - Sap -
>>>>>>>>>>> work to extend jcmd, to help our support folks. E.g. the whole new
>>>>>>>>>>> VM.metaspace command chain.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And simple it is….well done!!!
>>>>>>>>>>
>>>>>>>>>> — Kirk
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>
More information about the serviceability-dev
mailing list