From shurailine at openjdk.java.net Wed Apr 27 23:26:33 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Wed, 27 Apr 2022 23:26:33 GMT Subject: RFR: Data plugin Message-ID: * JREInstr * instrumentation plugin enhancement * build script * Implantable interface * support for all data types * other minor enhancements ------------- Commit messages: - requireNonNull - cleanup - Merge branch 'master' into data-plugin - in- and out- files - support for loading environment from an implanted property file - More implantable classes. - workaround for CODETOOLS-7903035 - JREInstr - Removed empty test method - Removed unused imports, old version of a test. - ... and 1 more: https://git.openjdk.java.net/jcov/compare/f3ee5df6...4b45ff92 Changes: https://git.openjdk.java.net/jcov/pull/19/files Webrev: https://webrevs.openjdk.java.net/?repo=jcov&pr=19&range=00 Stats: 1461 lines in 41 files changed: 1141 ins; 167 del; 153 mod Patch: https://git.openjdk.java.net/jcov/pull/19.diff Fetch: git fetch https://git.openjdk.java.net/jcov pull/19/head:pull/19 PR: https://git.openjdk.java.net/jcov/pull/19 From lkuskov at openjdk.java.net Wed Apr 27 23:26:34 2022 From: lkuskov at openjdk.java.net (Leonid Kuskov) Date: Wed, 27 Apr 2022 23:26:34 GMT Subject: RFR: Data plugin In-Reply-To: References: Message-ID: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> On Tue, 26 Apr 2022 00:46:44 GMT, Alexandre Iline wrote: > * JREInstr > * instrumentation plugin enhancement > * build script > * Implantable interface > * support for all data types > * other minor enhancements I need more time for review. It's a first portion. plugins/data_coverage/src/openjdk/jcov/data/Env.java line 63: > 61: Set keys = System.getProperties().stringPropertyNames(); > 62: keys.stream().filter(k -> k.startsWith(prefix)) > 63: .forEach(k -> System.clearProperty(k)); It's not good idea to clear/update system properties. Actually much better to use an "application" copy of them and interact with the copy as needed. Or after program shut down updated system properties could be used by an other application? plugins/data_coverage/src/openjdk/jcov/data/Env.java line 66: > 64: } > 65: > 66: public static String getStringEnv(String property, String defaultValue) { defaultValue isn't used and could be omited. line 70 of Class Collect if (!Env.getStringEnv(COVERAGE_IN, "").isEmpty()) { ``` could be chaged to if( Env.getStringEnv(COVERAGE_IN) != null Also isn't clear why you wraps IOException with UncheckedIOException here. plugins/data_coverage/src/openjdk/jcov/data/Env.java line 112: > 110: String[] params = propValue.substring(ob + 1, cb).split(","); > 111: Class[] paramTypes = new Class[params.length]; > 112: Arrays.fill(paramTypes, String.class); Here the method MethodTypeDesc.ofDescriptor(String descriptor)could be used to parse method's descriptor if you use JVM notation for method descriptors like: (IDLjava/lang/Thread;)Ljava/lang/Object; for Object m(int i, double d, Thread t). Just in case if universal parser is needed here and params are not just Strings. plugins/data_coverage/src/openjdk/jcov/data/JREInstr.java line 73: > 71: "-template", jcovTemplate, > 72: "-im", "java.base", > 73: jre}; Is Objects.requireNonNull() needed? It could be that jcovRuntime, jcovTemplate or pluginClass is null. plugins/data_coverage/src/openjdk/jcov/data/arguments/instrument/MethodFilter.java line 50: > 48: res.add(new Plugin.TypeDescriptor("[", "java/lang/Object", ALOAD)); > 49: if(desc.charAt(pos + 1) == 'L') pos = desc.indexOf(";", pos) + 1; > 50: else pos = pos + 2; Is the case [[[Ljava/lang/Object; covered? plugins/data_coverage/src/openjdk/jcov/data/arguments/runtime/Collect.java line 105: > 103: > 104: static int countParams(String desc) { > 105: if(!desc.startsWith("(")) throw new IllegalArgumentException("Not a method descriptor: " + desc); I'm not sure. But this parser should be updated with MethodTypeDesc.ofDescriptor && parameterCount() That will support all changes at method descriptors like Q references. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From shurailine at openjdk.java.net Wed Apr 27 23:26:34 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Wed, 27 Apr 2022 23:26:34 GMT Subject: RFR: Data plugin In-Reply-To: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> References: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> Message-ID: On Wed, 27 Apr 2022 01:54:06 GMT, Leonid Kuskov wrote: >> * JREInstr >> * instrumentation plugin enhancement >> * build script >> * Implantable interface >> * support for all data types >> * other minor enhancements > > plugins/data_coverage/src/openjdk/jcov/data/Env.java line 63: > >> 61: Set keys = System.getProperties().stringPropertyNames(); >> 62: keys.stream().filter(k -> k.startsWith(prefix)) >> 63: .forEach(k -> System.clearProperty(k)); > > It's not good idea to clear/update system properties. Actually much better to use an "application" copy of them and interact with the copy as needed. Or after program shut down updated system properties could be used by an other application? The way the plugin is designed, its behavior is controlled through the system properties. This is done this way to allow usage of jcov command line, such as Instr, JREInstr, etc, to specify how the code is instrumented, why code, where results are saved, etc. As such, it is _unavoidable_ to have the plugin code to _read_ the system properties. Here are a few examples of system properties which specifies the plugin behavior: Instrument.PLUGIN_CLASS, Instrument.JCOV_TEMPLATE, Collect.COVERAGE_OUT, Collect.COVERAGE_IN, Plugin.METHOD_FILTER, Collect.SERIALIZER An additional level is added to the instrumentation to allow, during instrumentation time, to specify properties which will be needed during the execution time. This is just a convenience which shortens command line needed to run the tests later. Instead of listing the properties on the command line when running the tests, the properties are stored within the instrumented code. This is, coincidently, a convenience which I would be handy for normal, non-plugin, Jcov functionality: to be able, during the instrumentation, specify a custom grabber port, to use an example. I thought, therefore, that the solution which I was suggesting was an elegant one, since the system properties which are set during the execution are exactly the same properties which the plugin is expected to be set as system properties to work. I suppose it is possible to change the code of the plugin to try to load properties from a local storage rather than from the system properties. I will have to take a close and a hard look to see if it can be done nicely. > plugins/data_coverage/src/openjdk/jcov/data/Env.java line 66: > >> 64: } >> 65: >> 66: public static String getStringEnv(String property, String defaultValue) { > > defaultValue isn't used and could be omited. > line 70 of Class Collect > > if (!Env.getStringEnv(COVERAGE_IN, "").isEmpty()) { > ``` > could be chaged to > > if( Env.getStringEnv(COVERAGE_IN) != null > > Also isn't clear why you wraps IOException with UncheckedIOException here. on line 115: } else return defaultValue; > plugins/data_coverage/src/openjdk/jcov/data/Env.java line 112: > >> 110: String[] params = propValue.substring(ob + 1, cb).split(","); >> 111: Class[] paramTypes = new Class[params.length]; >> 112: Arrays.fill(paramTypes, String.class); > > Here the method MethodTypeDesc.ofDescriptor(String descriptor)could be used to parse method's descriptor if you use JVM notation for method descriptors like: (IDLjava/lang/Thread;)Ljava/lang/Object; for Object m(int i, double d, Thread t). Just in case if universal parser is needed here and params are not just Strings. Indeed that would make things a lot easier! I would implement it as a separate enhancement, if you do not mind. > plugins/data_coverage/src/openjdk/jcov/data/JREInstr.java line 73: > >> 71: "-template", jcovTemplate, >> 72: "-im", "java.base", >> 73: jre}; > > Is Objects.requireNonNull() needed? It could be that jcovRuntime, jcovTemplate or pluginClass is null. Perhaps. And also perhaps the JCov code needs to be enhanced to handle the nulls. If an option value is null, from the code it looks like an NPE can be thrown out of EnvHandler.processOption(...) (line 299): if (value.startsWith(PROP_FILE_SPECIFIER)) { in another case nulls are handled: EnvHandler.addValuedOption(...) (line 383): throw new CLParsingException("Option '" + option + "' is invalid: value expected."); > plugins/data_coverage/src/openjdk/jcov/data/arguments/instrument/MethodFilter.java line 50: > >> 48: res.add(new Plugin.TypeDescriptor("[", "java/lang/Object", ALOAD)); >> 49: if(desc.charAt(pos + 1) == 'L') pos = desc.indexOf(";", pos) + 1; >> 50: else pos = pos + 2; > > Is the case [[[Ljava/lang/Object; covered? not currently, there is more work to do. Hence the //TODO :) > plugins/data_coverage/src/openjdk/jcov/data/arguments/runtime/Collect.java line 105: > >> 103: >> 104: static int countParams(String desc) { >> 105: if(!desc.startsWith("(")) throw new IllegalArgumentException("Not a method descriptor: " + desc); > > I'm not sure. But this parser should be updated with MethodTypeDesc.ofDescriptor && parameterCount() That will support all changes at method descriptors like Q references. Yes, as a separate enhancement. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From shurailine at openjdk.java.net Wed Apr 27 23:26:35 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Wed, 27 Apr 2022 23:26:35 GMT Subject: RFR: Data plugin In-Reply-To: References: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> Message-ID: On Wed, 27 Apr 2022 15:30:19 GMT, Alexandre Iline wrote: >> plugins/data_coverage/src/openjdk/jcov/data/Env.java line 63: >> >>> 61: Set keys = System.getProperties().stringPropertyNames(); >>> 62: keys.stream().filter(k -> k.startsWith(prefix)) >>> 63: .forEach(k -> System.clearProperty(k)); >> >> It's not good idea to clear/update system properties. Actually much better to use an "application" copy of them and interact with the copy as needed. Or after program shut down updated system properties could be used by an other application? > > The way the plugin is designed, its behavior is controlled through the system properties. This is done this way to allow usage of jcov command line, such as Instr, JREInstr, etc, to specify how the code is instrumented, why code, where results are saved, etc. As such, it is _unavoidable_ to have the plugin code to _read_ the system properties. Here are a few examples of system properties which specifies the plugin behavior: Instrument.PLUGIN_CLASS, Instrument.JCOV_TEMPLATE, Collect.COVERAGE_OUT, Collect.COVERAGE_IN, Plugin.METHOD_FILTER, Collect.SERIALIZER > > An additional level is added to the instrumentation to allow, during instrumentation time, to specify properties which will be needed during the execution time. This is just a convenience which shortens command line needed to run the tests later. Instead of listing the properties on the command line when running the tests, the properties are stored within the instrumented code. This is, coincidently, a convenience which I would be handy for normal, non-plugin, Jcov functionality: to be able, during the instrumentation, specify a custom grabber port, to use an example. I thought, therefore, that the solution which I was suggesting was an elegant one, since the system properties which are set during the execution are exactly the same properties which the plugin is expected to be set as system properties to work. > > I suppose it is possible to change the code of the plugin to try to load properties from a local storage rather than from the system properties. I will have to take a close and a hard look to see if it can be done nicely. As for reusing the system property by another application after the program shut down, I do not really know what you mean. The properties which are set are the ones which may be set anyway to control the plugin behavior. >> plugins/data_coverage/src/openjdk/jcov/data/Env.java line 66: >> >>> 64: } >>> 65: >>> 66: public static String getStringEnv(String property, String defaultValue) { >> >> defaultValue isn't used and could be omited. >> line 70 of Class Collect >> >> if (!Env.getStringEnv(COVERAGE_IN, "").isEmpty()) { >> ``` >> could be chaged to >> >> if( Env.getStringEnv(COVERAGE_IN) != null >> >> Also isn't clear why you wraps IOException with UncheckedIOException here. > > on line 115: > } else return defaultValue; on line 70 of Collect, I am defending against COVERAGE_IN property been set to an empty string as well as not been set at all. The code which you suggested is not equivalent to what I am having. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From lkuskov at openjdk.java.net Wed Apr 27 23:26:35 2022 From: lkuskov at openjdk.java.net (Leonid Kuskov) Date: Wed, 27 Apr 2022 23:26:35 GMT Subject: RFR: Data plugin In-Reply-To: References: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> Message-ID: On Wed, 27 Apr 2022 15:39:32 GMT, Alexandre Iline wrote: >> The way the plugin is designed, its behavior is controlled through the system properties. This is done this way to allow usage of jcov command line, such as Instr, JREInstr, etc, to specify how the code is instrumented, why code, where results are saved, etc. As such, it is _unavoidable_ to have the plugin code to _read_ the system properties. Here are a few examples of system properties which specifies the plugin behavior: Instrument.PLUGIN_CLASS, Instrument.JCOV_TEMPLATE, Collect.COVERAGE_OUT, Collect.COVERAGE_IN, Plugin.METHOD_FILTER, Collect.SERIALIZER >> >> An additional level is added to the instrumentation to allow, during instrumentation time, to specify properties which will be needed during the execution time. This is just a convenience which shortens command line needed to run the tests later. Instead of listing the properties on the command line when running the tests, the properties are stored within the instrumented code. This is, coincidently, a convenience which I would be handy for normal, non-plugin, Jcov functionality: to be able, during the instrumentation, specify a custom grabber port, to use an example. I thought, therefore, that the solution which I was suggesting was an elegant one, since the system properties which are set during the execution are exactly the same properties which the plugin is expected to be set as system properties to work. >> >> I suppose it is possible to change the code of the plugin to try to load properties from a local storage rather than from the system properties. I will have to take a close and a hard look to see if it can be done nicely. > > As for reusing the system property by another application after the program shut down, I do not really know what you mean. The properties which are set are the ones which may be set anyway to control the plugin behavior. It's a minor comment. Just to say that here, I mean there are 2 kinds of properties - Environment > export JCOV_OPTION=path && System.getenv("JCOV_OPTION") and java -DJCOV_OPTION=path && System.getProperty("JCOV_OPTION") First and second ways to get external properties don't have in mind their update/clean. If System properties are used as global mutable variables of the program then it would be much better to use a special singleton container like a hash table for this need. If these options are read-only then all updating/cleaning methods are superfluous. >> on line 115: >> } else return defaultValue; > > on line 70 of Collect, I am defending against COVERAGE_IN property been set to an empty string as well as not been set at all. The code which you suggested is not equivalent to what I am having. Well, I just appeal for the single way for processing all properties If option is missing then it is always NULL otherwise it has a value that should be processed in the place of using. In the program we have many options that are null by default and just one that is empty string. For this option we have a special branch in the method From my point of view the program should be more restricted If we have m:1 branches of processing something the we should covert it to m+1:0 and remove program logic for this :1 branch. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From shurailine at openjdk.java.net Wed Apr 27 23:26:35 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Wed, 27 Apr 2022 23:26:35 GMT Subject: RFR: Data plugin In-Reply-To: References: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> Message-ID: On Wed, 27 Apr 2022 17:43:15 GMT, Leonid Kuskov wrote: >> As for reusing the system property by another application after the program shut down, I do not really know what you mean. The properties which are set are the ones which may be set anyway to control the plugin behavior. > > It's a minor comment. Just to say that here, I mean there are 2 kinds of properties - Environment > export JCOV_OPTION=path && System.getenv("JCOV_OPTION") and java -DJCOV_OPTION=path && System.getProperty("JCOV_OPTION") First and second ways to get external properties don't have in mind their update/clean. If System properties are used as global mutable variables of the program then it would be much better to use a special singleton container like a hash table for this need. If these options are read-only then all updating/cleaning methods are superfluous. In fact, in the current state of the code, system properties are changed only from tests. Tests simply do what a user is suggested to do: set some system properties, run a JCov empty point class, either from command line or from some other java code. I think this is acceptable. There can be an additional interface which does not require setting system properties, but system properties will always work anyway. There is one exception when a system property is set not from a test: "jcov.data-saver" is set to be the same as "jcov.data.saver. I am looking if this is justified. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From shurailine at openjdk.java.net Wed Apr 27 23:26:35 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Wed, 27 Apr 2022 23:26:35 GMT Subject: RFR: Data plugin In-Reply-To: References: <2eD4xIiqkCeZaKC0DvyKa8b_5GbTbKO64NIAtFq8YJ4=.d4dd2243-9be0-4cb2-a65d-7e0c717ca5aa@github.com> Message-ID: On Wed, 27 Apr 2022 19:18:04 GMT, Alexandre Iline wrote: >> It's a minor comment. Just to say that here, I mean there are 2 kinds of properties - Environment > export JCOV_OPTION=path && System.getenv("JCOV_OPTION") and java -DJCOV_OPTION=path && System.getProperty("JCOV_OPTION") First and second ways to get external properties don't have in mind their update/clean. If System properties are used as global mutable variables of the program then it would be much better to use a special singleton container like a hash table for this need. If these options are read-only then all updating/cleaning methods are superfluous. > > In fact, in the current state of the code, system properties are changed only from tests. Tests simply do what a user is suggested to do: set some system properties, run a JCov empty point class, either from command line or from some other java code. I think this is acceptable. There can be an additional interface which does not require setting system properties, but system properties will always work anyway. > There is one exception when a system property is set not from a test: "jcov.data-saver" is set to be the same as "jcov.data.saver. I am looking if this is justified. Upon further review, no, this is necessary. If I want to be able to implant a name of the saver SPI class into the instrumented code (into a property file), I will later have to retrieve it from the storage (the property file) and set system property, because this is where JCov is obtaining it from - from the system property. The only way to go around that would be to add some SPI to JCov itself, which would allow setting JCov up. This may be a valuable enhancement, but it will have to be done separately. ------------- PR: https://git.openjdk.java.net/jcov/pull/19 From shurailine at openjdk.java.net Thu Apr 28 00:00:58 2022 From: shurailine at openjdk.java.net (Alexandre Iline) Date: Thu, 28 Apr 2022 00:00:58 GMT Subject: Integrated: Data plugin In-Reply-To: References: Message-ID: On Tue, 26 Apr 2022 00:46:44 GMT, Alexandre Iline wrote: > * JREInstr > * instrumentation plugin enhancement > * build script > * Implantable interface > * support for all data types > * other minor enhancements This pull request has now been integrated. Changeset: 24c8e4d7 Author: Alexandre Iline URL: https://git.openjdk.java.net/jcov/commit/24c8e4d7ba83dcde7b2dc7cc71871ba693fcd51c Stats: 1461 lines in 41 files changed: 1141 ins; 167 del; 153 mod Data plugin ------------- PR: https://git.openjdk.java.net/jcov/pull/19