Request for Review (XXL): 7104647: Adding a diagnostic command framework

[Paul Hohensee]

For the hotspot part:

In attachListener.cpp, you might want to delete the first "return JNI_OK;"
line, since the code under HAS_PENDING_EXCEPTION just falls through.

In jmm.h, minor formatting nit: be nice to indent "(JNIEnv" on lines 318,
319 and 325 the same as the existing declarations.  Add a newline
just before it on line 322.



On 12/2/11 8:57 AM, Frederic Parain wrote:
> Hi All,
>
> [Posting to serviceability-dev, runtime-dev and core-libs-dev
> because changes are pretty big and touch all these areas]
>
> Here's a framework for issuing diagnostics commands to the JVM.
> Diagnostic commands are actions executed inside the JVM mainly
> for monitoring or management purpose. This work has two parts.
> The first part is in the hotspot repository and contains the
> framework itself with two diagnostic commands. The second
> part is in the JDK repository and contains the command line
> utility to invoke diagnostic commands as well as the
> HotSpotDiagnosticMXBean extensions. The HotSpotDiagnosticMXBean
> extensions allow a remote client to discover and invoke
> diagnostic commands using a JMX connection.
>
> This changeset only contains two diagnostic commands, more
> commands will be added in the future, as a standalone effort
> to improve the monitoring and management services of the
> JVM, or as part of other projects.
>
> Webrevs are here:
> http://cr.openjdk.java.net/~fparain/7104647/webrev.hotspot.00/
> http://cr.openjdk.java.net/~fparain/7104647/webrev.jdk.00/
>
> Here's a technical description of this work:
>
> 1 - The Diagnostic Command Framework
> 1-1 Overview
>
> The diagnostic command framework is fully implemented in native code 
> and relies on the HotSpot's internal exception mechanism.
> The rational of a pure native implementation is to be able to execute
> diagnostic commands even in critical situations like an OutOfMemory
> error. All diagnostic commands are registered in a single list, and two
> flags control the way a user can interact with them. The "hidden" flag
> prevents a diagnostic command from appearing in the list of available
> commands returned by the "help" command. However, it's still possible to
> get the detailed help message for a hidden command with the "help
> <command name>" syntax (but it requires to know the name of the hidden
> command). The second flag is "enabled" and it controls if a command can
> be invoked or not. When listed with the "help" commands, disabled
> commands appear with a "[disabled]" label in their description. If the
> user tries to invoke a disabled command, an error message is returned
> and the command is not run. This error message can be customized on a
> per command base. The framework just provides these two flags with their
> semantic, it doesn't provide any policy or mechanism to set or modify
> these flags. These actions will be delegated to the JVM or special
> diagnostic commands.
>
> 1-2 Implementation
>
> All diagnostic commands are implemented as subclasses of the DCmd class
> defined in services/diagnosticFramework.hpp. Here's the layout of the
> DCmd class and the list of methods that a new command has to define or
> overwrite:
>
> class DCmd {
> DCmd(outputStream *output);
>
> static const char *get_name();
>
> static const char *get_description();
>
> static const char *get_disabled_message();
>
> static const char *get_impact();
>
> static int get_num_arguments();
>
> virtual void print_help(outputStream* out);
>
> virtual void parse(CmdLine* line, char delim, TRAPS);
>
> virtual void execute(TRAPS);
>
> virtual void reset(TRAPS);
>
> virtual void cleanup();
>
> virtual GrowableArray<const char *>* get_argument_name_array();
>
> virtual GrowableArray<DCmdArgumentInfo*>* get_argument_info_array();
> }
>
> A diagnostic command is always instantiated with an outputStream in
> parameter. This outputStream can point either to a file, a buffer or a
> socket (see the ostream.hpp file).
>
> The get_name() method returns the string that will identify the command
> (i.e. the string to put on the command line to invoke it).
>
> The get_description() methods returns the global description of the 
> command.
>
> The get_disabled_message() returns the customized message to return when
> the command is disabled, without having to instantiate the command.
>
> The get_impact() returns a description of the intrusiveness of the
> diagnostic command on the Java Virtual Machine behavior. The rational
> for this method is that some diagnostic commands can seriously disrupt
> the behavior of the Java Virtual Machine (for instance a Thread Dump for
> an application with several tens of thousands of threads, or a Head Dump
> with a 40GB+ heap size) and other diagnostic commands have no serious
> impact on the JVM (for instance, getting the command line arguments or
> the JVM version). The recommended format for the description is <impact
> level>: [longer description], where the impact level is selected among
> this list: {low, medium, high}. The optional longer description can
> provide more specific details like the fact that Thread Dump impact
> depends on the heap size.
>
> The get_num_arguments() returns the number of options/arguments
> recognized by the diagnostic command. This method is only used by the
> JMX interface support (see below).
>
> The print_help() methods prints a detailed help on the outputStream
> passed in argument. The detailed help contains the list of all supported
> options with their type and description.
>
> The parse() method is in charge of parsing the command arguments. Each
> command is free to implement its own argument parser. However, an
> argument parser framework is provided (see section 1-3) to ease the
> implementation, but its use is optional. The parse method takes a
> delimiter character in argument, which is used to mark the limit between
> two arguments (typically invocation from jcmd will use a space character
> as a delimiter while invocation from the JVM command line parsing code
> will use a comma as a delimiter).
>
> The execute() method is naturally the one to invoke to get the
> diagnostic command executed. The parse() and the execute() methods are
> dissociated, so it's a possible perform the argument parsing in one
> thread, and delegate the execution to another thread, as long as the
> diagnostic command doesn't reference thread local variables. The
> framework allows several instances of the same diagnostic command to be
> executed in parallel. If for some reasons concurrent executions should
> not be allowed for a given diagnostic command, this is the
> responsibility of the diagnostic command implementor to enforce this
> rule, for instance by protecting the body of the execute() method with a
> global lock.
>
> The reset() method is used to initialize the internal fields of the
> diagnostic command or to reset the internal fields to their initial
> value to be able to re-use an already allocated diagnostic command 
> instance.
>
> The cleanup() method is used to perform perform cleanup (like freeing of
> all memory allocated to store internal data). The DCmd extends the
> ResourceObj class, so when allocated in a ResourceArea, destructors
> cannot be used to perform cleanup. To ensure that cleanup is performed
> in all cases, it is recommended to create a DCmdMark instance for each
> DCmd instance. DCmdMark is a stack allocated object with a pointer to a
> DCmd instance. When the DCmdMark is destroyed, its destructor calls the
> cleanup() method of the DCmd instance it points to. If the DCmd instance
> has been allocated on the C-Heap, the DCmdMark will also free the memory
> allocated to store the DCmd instance.
>
> The get_argument_name_array() and get_argument_info_array() are related
> to the JMX interface of the diagnostic command framework, so they are
> described in section 3.
>
> 1-3 DCmdParser framework
>
> The DCmdParser class is an optional framework to help development of
> argument parsers. It provides many features required by the diagnostic
> command framework (generation of the help message or the argument
> descriptions for the JMX interface) but all these features can easily be
> re-implemented if a developer decides not to use the DCmdParser 
> framework.
>
> The DCmdParser class is relying on the DCmdArgument template. This
> template must be used to define the different types of argument the
> parser will have to handle. When a new specialization of the template is
> done, three methods have to be provided:
>
> void parse_value(const char *str,size_t len,TRAPS);
> void init_value(TRAPS);
> void destroy_value();
>
> The parse_value() method is used to convert a string into an argument
> value. The print_value() method is used to display the default value
> (support for the detailed help message). The init_value() method is used
> to initialize or reset the argument value. The destroy_value() method is
> a clean-up method (useful when the argument has allocated some C-Heap
> memory to store its value and this memory has to be freed before
> destroying the DCmdArgument instance).
>
> The DCmdParser makes a distinction between options and arguments.
> Options are identified by a key name that must appear on the command
> line, while argument are identified just by the position of the argument
> on the command line. Options use the <key>=<value> syntax. In case of
> boolean options, the '=<value>' part of the syntax can be omitted to set
> the option to true. Arguments are just sequences characters delimited by
> a separator character. This separator can be specified at runtime when
> invoking the diagnostic command framework. If an argument contain a
> character that could be used as a delimiter, it's possible to enclose
> the argument between single or double quotes. Options are arguments are
> instantiated using the same DCmdArgument class but they're registered
> differently to the DCmdParser.
>
> The way to use the DCmdParser is to declare the parser and the
> option/arguments as fields of the diagnostic command class (which is
> itself a sub-class of the DCmd class), like this:
>
>
>  class EchoDCmd : public DCmd {
>  protected:
>      DCmdParser _dcmdparser;
>
>      DCmdArgument<jlong> _required;
>
>      DCmdArgument<jlong> _intval;
>
>      DCmdArgument<bool> _boolval;
>
>      DCmdArgument<char *> _stringval;
>
>      DCmdArgument<char *> _first_arg;
>
>      DCmdArgument<jlong> _second_arg;
>
>      DCmdArgument<char *> _optional_arg;
>
> }
>
> The parser and the options/arguments must be initialized before the
> diagnostic command class, and the options/arguments have to be
> registered to the parser like this:
>
>  EchoDCmd(outputStream *output) : DCmd(output),
>      _stringval("-strval","a string argument","STRING",false),
>
>      _boolval("-boolval","a boolean argument","BOOLEAN",false),
>
>      _intval("-intval","an integer argument","INTEGER",false),
>
>      _required("-req","a mandatory integer argument","INTEGER",true),
>
>      _fist_arg("first argument","a string argument","STRING",true),
>
>      _second_arg("second argument,"an integer argument,"INTEGER",true),
>
>      _optional_arg("optional argument","an optional string
> argument","STRING","false")
>
>  {
>
>  _dcmdparser.add_dcmd_option(&_stringval)
>
>  _dcmdparser.add_dcmd_option(&_boolval);
>
>  _dcmdparser.add_dcmd_option(&_intval);
>
>  _dcmdparser.add_dcmd_option(&_required);
>
>  _dcmdparser.add_argument(&_first_arg);
>
>  _dcmdparser.add_argument(&_second_arg);
>
>  _dcmdparser.add_argument(&_optional_arg);
>  };
>
> The add_dcmd_argument()/ add_dcmd_option() method is used to add an
> argument/option to the parser. The option/argument constructor takes the
> name of the option/argument, its description, a string describing its
> type and a boolean to specify if the option/argument is mandatory or
> not. The parser doesn't support option/argument duplicates (having the
> same name) but the code currently doesn't check for duplicates.The order
> used to register options has no impact on the parser. However, the order
> used to register arguments is critical because the parser will use the
> same order to parse the command line. In the example above, the parser
> expects to have a first argument of type STRING (parsed using
> _first_arg), then a second argument of type INTEGER (parsed using
> _second_arg) and optionally a third parameter of type STRING (parsed
> using _optional_arg). A mandatory option or argument has to be specify
> every time the command is invoked. If it is missing, an exception is
> thrown at the end of the parsing. Optional arguments have to be
> registered after mandatory arguments. An optional argument will be
> considered for parsing only if all arguments before it (mandatory or
> not) have already been used to parse the command line.
>
> The DCmdParser and its DCmdArgument instances are embedded in the DCmd
> instance. The rational for this design is to limit the number of C-heap
> allocations but also to be able to pre-allocate diagnostic command
> instances for critical situations. If the process is running out of
> C-heap space, it's not possible to instantiate new diagnostic commands
> to troubleshoot the situation. By pre-allocating some diagnostic
> commands, it will be possible to run them even in this critical
> situation. Of course, the diagnostic command itself should not try to
> allocate memory during its execution, this prevents the diagnostic
> command to use variable length arguments like strings. By nature,
> pre-allocated diagnostic commands aim to be re-usable, this is the
> purpose of the reset() method which restores the default status of all
> arguments.
>
> 1-4 Internal invocation
>
> Using a diagnostic command from the JVM itself is pretty easy:
> instantiate the class and invoke the parse() method then the execute()
> method. A diagnostic command can be instantiated from inside the JVM
> even it is not registered. This is a difference with the external
> invocations (from jcmd or JMX) that require the command to be registered.
>
> 2 - The JCmd interface
>
> Diagnostic commands can also be invoked from outside the JVM process,
> using the new 'jcmd' utility. The jcmd program uses the attach API
> to connect to the JVM, send requests and receive results. The
> jcmd utility must be launched on the same machine than the one running
> the JVM (its a local tool). Launched without arguments, jcmd displays a
> list of all JVMs running on the machine. The jcmd source code is in
> the jdk repository like other existing j* tools.
>
> To execute a diagnostic command in a particular JVM, the generic 
> syntax is:
>
> jcmd <pid_of_the_jvm> <command_name> [arguments]
>
> The attachListener has been modified to recognized the jcmd requests.
> When a jcmd request is identified, it is parsed to extract the command
> name. The JVM performs a look up of this command in a list of registered
> commands. To be executable by an external request, a diagnostic command
> has to be registered. The registration is performed with the DCmdFactory
> class (see services/management.cpp).
>
> 3 - The JMX interface
>
> The framework provides a JMX based interface to the diagnostic commands.
> This interface allows remote invocation of diagnostic commands through a
> JMX connection.
>
> 3-1 The interface
>
> The information related to the diagnostic commands are accessible
> through new methods added to the 
> com.sun.management.HotspotDiagnosticMXBean:
>
> public List<String> getDiagnosticCommands();
>
> public DiagnosticCommandInfo getDiagnosticCommandInfo(String command);
>
> public List<DiagnosticCommandInfo> getDiagnosticCommandInfo(List<String>
> command);
>
> public List<DiagnosticCommandInfo> getDiagnosticCommandInfo();
>
> public String execute(String commandLine) throws 
> IllegalArgumentException ;
>
> public String execute(String cmd, String... arguments)
> throws IllegalArgumentException;
>
>
> The getDiagnosticCommands() method returns an array containing the names
> of the not-hidden registered diagnostic commands.
>
> The three getDiagnosticCommandInfo() methods return one or several
> diagnostic command descriptions using the DiagnosticCommandInfo class.
>
> The two execute() methods allow the user the invoke a diagnostic command
> in different ways.
>
> The DiagnosticCommandInfo class is describing a diagnostic command with
> the following information:
>
> public class DiagnosticCommandInfo {
>
> public String getName();
>
> public String getDescription();
>
> public String getImpact();
>
> public boolean isEnabled();
>
> public List<DiagnosticCommandArgumentInfo> getArgumentsInfo();
> }
>
> The getName() method returns the name of the diagnostic command. This
> name is the one to use in execute() methods to invoke the diagnostic
> command.
>
> The getDescription() method returns a general description of the
> diagnostic command.
>
> The getImpact() method returns a description of the intrusiveness of
> diagnostic command.
>
> The isEnabled() method returns true if the method is enabled, false if
> it is disabled. A disabled method cannot be executed.
>
> The getArgumentsInfo() returns a list of descriptions for the options or
> arguments recognized by the diagnostic command. Each option/argument is
> described by a DiagnosticCommandArgumentInfo instance:
>
> public class DiagnosticCommandArgumentInfo {
>
> public String getName();
>
> public String getDescription();
>
> public String getType();
>
> public String getDefault();
>
> public boolean isMandatory();
>
> public boolean isOption();
>
> public int getPosition();
> }
>
> If the DiagnosticCommandArgumentInfo instance describes an option,
> isOption() returns true and getPosition() returns -1. Otherwise, when
> the DiagnosticCommandArgumentInfo instance describes an argument,
> isOption() returns false and getPosition() returns the expected position
> for this argument. The position of an argument is defined relatively to
> all arguments passed on the command line, options are not considered
> when defining an argument position. The getDefault() method returns the
> default value of the argument if a default has been defined, otherwise
> it returns null.
>
> 3-2 The implementation
>
> The framework has been designed in a way that prevents diagnostic
> command developers to worry about the JMX interface. In addition to
> the methods described in section 1-2, a diagnostic command developer has
> to provide three methods:
>
> int get_num_arguments()
>
> which returns the number of option and arguments supported by the 
> command.
>
> GrowableArray<const char *>* get_argument_name_array()
>
> which provides the name of the arguments supported by the command.
>
> GrowableyArray<DCmdArgumentInfo*>* get_argument_info_array()
>
> which provides the description of each argument with a DCmdArgumentInfo
> instance. DCmdArgumentInfo is a C++ class used by the framework to
> generate the sun.com.management.DcmdArgumentInfo instances. This is done
> automatically and the diagnostic command developer doesn't need to know
> how to create Java objects from the runtime.
>
> 4 - The Diagnostic Commands
>
> To avoid name collisions between diagnostic commands coming from
> different projects, use of a flat name space should be avoided and
> a more structured organization is recommended. The framework itself
> doesn't depend on this organization, so it will be a set of rules
> defining a convention in the way commands are named.
>
> Diagnostic commands can easily organized in a hierarchical way, so the
> template for a command name can be:
>
> <domain>.[sub-domain.]<command>
>
> This template can be extended with sub-sub-domains and so on.
>
> A special set of commands without domain will be reserved for the
> commands related to the diagnostic framework itself, like the "help"
> command.
>
>
> Thanks,
>
> Fred
>