From tj.fontaine at oracle.com Mon May 1 00:22:02 2017 From: tj.fontaine at oracle.com (TJ Fontaine) Date: Sun, 30 Apr 2017 17:22:02 -0700 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> References: <36e2184f-6527-0aad-b156-b12719cfbd8a@oracle.com> <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> Message-ID: <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> Hey, I?ve attached a version rebased on jdk10, it also (currently) applies cleanly to jdk9. While there is no supplied test or harness for this patch, how I built and tested is available at https://github.com/tjfontaine/jdkbuild (there?s also a preview of my follow on patch for pathmap_open as well). Thanks! TJ On 4/28/17, 3:47 PM, "serviceability-dev on behalf of TJ Fontaine" wrote: I had no doubt we?d end up on the conversation of 10 -> 9 -> 8u, I started with 8u merely because it was representative of today?s customer pain. I?ll be sure to work on retargeting it as well. Thanks! TJ On 4/28/17, 3:42 PM, "David Holmes" wrote: Hi TJ, Thanks for the patch (I haven't looked at it yet). FYI at the moment, unless this is considered a high priority bug for JDK 9 it has to be targeted to JDK 10, and then possibly backported to 9 and 8u. Cheers, David On 29/04/2017 8:23 AM, TJ Fontaine wrote: > I have attached a patch that allows jcmd to work against a java process running > inside a Docker container. Apologies if this is not in the correct format. It was > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > that is inside a container (e.g. Docker). > > A Linux container often isolates a process in a PID and Mount namespace that is > separate from the "root container" (analogous to the hypervisor/dom0 in > hardware virtualization environments, or the global zone on Solaris). A target > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > the attach sequence. > > When the target JVM is in its own PID namespace the pid of the process is > distinct from what the real pid of the process as it relates to the root > container. For example, in the root container you can observe a JVM with a pid > of 17734, however if that JVM is running inside a Docker container the pid > inside its PID namespace is likely 1. So when the target JVM receives the > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > discrepancy the target JVM will output to stderr thread status, since > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > sequence. > > The solution is to parse /proc/pid/status for the field NSpid (available since > Linux 4.1) which contains a list of pids, where the last entry is the "inner > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > which have a virtualization depth of 1) > > The rest of the Linux attach sequence assumes a shared mount namespace by > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > separate namespace because the target JVM is in a mount namepsace (or in a > chroot as well) the unix domain socket for attaching won't appear. > > Instead the attach sequence should resolve file names relative to > /proc/17734/root which has a materialized view of the rootfs for the target. > > Thanks! > > TJ > -------------- next part -------------- A non-text attachment was scrubbed... Name: jdk10-attach-namespace-aware.patch Type: application/octet-stream Size: 6527 bytes Desc: not available URL: From daniel.daugherty at oracle.com Mon May 1 17:09:07 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 1 May 2017 11:09:07 -0600 Subject: Result: New Serviceability Group Lead: Serguei Spitsyn In-Reply-To: <8bf2ab7d-79a7-567a-65c1-8f5b1ed6aaa3@oracle.com> References: <8bf2ab7d-79a7-567a-65c1-8f5b1ed6aaa3@oracle.com> Message-ID: Resending... this time I'm including registrar at openjdk.java.net. Dan On 4/19/17 10:13 AM, Daniel D. Daugherty wrote: > The vote for Serguei Spitsyn [1] is now closed. > > Yes: 5 > No: 0 > Abstain: 0 > > According to the Bylaws definition of Simple Majority[2], this is > sufficient to approve the new Group Lead. The OpenJDK Lead will > ask the Governing Board to ratify this nomination. > > Daniel Daugherty > > [1] > http://mail.openjdk.java.net/pipermail/serviceability-dev/2017-April/021177.html > [2] http://openjdk.java.net/bylaws#simple-majority > Quoted here for convenience: > > Simple Majority ? There are more Yes votes than No votes. > > From david.holmes at oracle.com Tue May 2 04:16:09 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 2 May 2017 14:16:09 +1000 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> References: <36e2184f-6527-0aad-b156-b12719cfbd8a@oracle.com> <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> Message-ID: <05b65382-b2e2-bcfa-6b2d-c8755dd530aa@oracle.com> Hi TJ, On 1/05/2017 10:22 AM, TJ Fontaine wrote: > Hey, > > I?ve attached a version rebased on jdk10, it also (currently) applies cleanly to jdk9. I have filed: https://bugs.openjdk.java.net/browse/JDK-8179498 and have hosted your patch at: http://cr.openjdk.java.net/~dholmes/8179498/webrev/ I find it a little confusing understanding when we need to use pid and when we need to use ns_pid. IIUC the client VM doing the attach should use the "global" pid in the /proc path but name the file with the ns_pid as seen by the target VM - correct? And the query "what is my pid" will actually return the ns_pid - correct? 345 // TODO XXX friggin old kernels may not have NSpid field (i.e. 3.10) 346 // fallback to original pid in the event we cannot deduce :) Suggest: // If no NSpid field found assume no namespace and fallback to original pid Overall the approach seems reasonable, but I can't validate the operational details. Lets see what others have to say. > While there is no supplied test or harness for this patch, how I built and tested is > available at https://github.com/tjfontaine/jdkbuild (there?s also a preview of my > follow on patch for pathmap_open as well). The lack of testability is somewhat of a concern. We can of course test that it works okay when the namespace is not present, but can't verify it works as you intend. And of course this doesn't help with Docker containers running under earlier versions of the Linux kernel. Thanks, David > Thanks! > > TJ > > On 4/28/17, 3:47 PM, "serviceability-dev on behalf of TJ Fontaine" wrote: > > I had no doubt we?d end up on the conversation of 10 -> 9 -> 8u, I started with 8u merely because it was representative of today?s customer pain. I?ll be sure to work on retargeting it as well. > > Thanks! > > TJ > > On 4/28/17, 3:42 PM, "David Holmes" wrote: > > Hi TJ, > > Thanks for the patch (I haven't looked at it yet). FYI at the moment, > unless this is considered a high priority bug for JDK 9 it has to be > targeted to JDK 10, and then possibly backported to 9 and 8u. > > Cheers, > David > > On 29/04/2017 8:23 AM, TJ Fontaine wrote: > > I have attached a patch that allows jcmd to work against a java process running > > inside a Docker container. Apologies if this is not in the correct format. It was > > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > > that is inside a container (e.g. Docker). > > > > A Linux container often isolates a process in a PID and Mount namespace that is > > separate from the "root container" (analogous to the hypervisor/dom0 in > > hardware virtualization environments, or the global zone on Solaris). A target > > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > > the attach sequence. > > > > When the target JVM is in its own PID namespace the pid of the process is > > distinct from what the real pid of the process as it relates to the root > > container. For example, in the root container you can observe a JVM with a pid > > of 17734, however if that JVM is running inside a Docker container the pid > > inside its PID namespace is likely 1. So when the target JVM receives the > > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > > discrepancy the target JVM will output to stderr thread status, since > > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > > sequence. > > > > The solution is to parse /proc/pid/status for the field NSpid (available since > > Linux 4.1) which contains a list of pids, where the last entry is the "inner > > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > > which have a virtualization depth of 1) > > > > The rest of the Linux attach sequence assumes a shared mount namespace by > > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > > separate namespace because the target JVM is in a mount namepsace (or in a > > chroot as well) the unix domain socket for attaching won't appear. > > > > Instead the attach sequence should resolve file names relative to > > /proc/17734/root which has a materialized view of the rootfs for the target. > > > > Thanks! > > > > TJ > > > > > > > From serguei.spitsyn at oracle.com Tue May 2 09:02:58 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 2 May 2017 02:02:58 -0700 Subject: RFR(M): 8172970: TESTBUG: need test coverage for the JVMTI functions allowed in the start phase In-Reply-To: References: <88938aee-0a83-99e9-1b95-6875173807a5@oracle.com> <5ea66d4b-f1bb-1729-ba51-5bdcd5b3c470@oracle.com> <1e443485-63d6-4473-8809-76f47a380436@oracle.com> Message-ID: <96edf450-1d0d-c21d-436d-bdbd378b4137@oracle.com> PING: I've got a thumbs up from David Holmes. One more review is needed for this jdk 10 test enhancement. Thanks! Serguei On 4/28/17 17:13, serguei.spitsyn at oracle.com wrote: > Hi David, > > > On 4/28/17 10:34, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> >> On 4/28/17 04:42, David Holmes wrote: >>> Hi Serguei, >>> >>> On 28/04/2017 6:07 PM, serguei.spitsyn at oracle.com wrote: >>>> The updated webrev is: >>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.2/ >>>> >>> >>> Thanks for the updates (the issue with long is that on win64 it is >>> only 32-bit while void* is 64-bit). >> >> Ok, thanks. >> Than you are right, using long on win64 is not compatible. >> >>> >>> I prefer to see fast-fail rather than potentially triggering >>> cascading failures (check_jvmti_error could even call exit() I >>> think). But let's see what others think - it's only a preference not >>> a necessity. >> >> Ok, I'll consider call exit() as it would keep it simple. >> > > New webrev version is: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ > > > > Thanks, > Serguei > > >> Thanks, >> Serguei >> >>> >>> Thanks, >>> David >>> >>>> >>>> I've re-arranged a little bit code in the ClassPrepare callback and >>>> the >>>> function test_class_functions(). >>>> >>>> Thanks, >>>> Serguei >>>> >>>> >>>> On 4/28/17 00:47, serguei.spitsyn at oracle.com wrote: >>>>> Hi David, >>>>> >>>>> Thank you for looking at the test! >>>>> >>>>> >>>>> On 4/27/17 23:11, David Holmes wrote: >>>>>> Hi Serguei, >>>>>> >>>>>> On 28/04/2017 3:14 PM, serguei.spitsyn at oracle.com wrote: >>>>>>> Please, review the jdk 10 fix for the test enhancement: >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8172970 >>>>>>> >>>>>>> >>>>>>> Webrev: >>>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.1/ >>>>>>> >>>>>>> >>>>>> >>>>>> Sorry but I can't quite figure out exactly what this test is doing. >>>>>> What is the overall call structure here? >>>>> >>>>> This is to make sure the functions allowed in the start and live >>>>> phases work Ok. >>>>> As the list of functions is pretty big the test does sanity checks >>>>> that the functions do not crash nor return errors. >>>>> >>>>> >>>>>> I was expecting to see a difference between things that can be >>>>>> called >>>>>> at early-start and those that can not - or are these all expected to >>>>>> work okay in either case? >>>>> >>>>> All these functions are expected to work okay in both cases. >>>>> Of course, the main concern is the early start. >>>>> But we have never had such coverage in the past so that the normal >>>>> start phase needs to be covered too. >>>>> >>>>> >>>>>> >>>>>> A few comments: >>>>>> >>>>>> 44 #define TranslateError(err) "JVMTI error" >>>>>> >>>>>> I don't see the point of the above. >>>>> >>>>> Good catch - removed. >>>>> It is a left over from another test that I used as initial template. >>>>> >>>>> >>>>>> --- >>>>>> >>>>>> 99 static long get_thread_local(jvmtiEnv *jvmti, jthread thread) { >>>>>> >>>>>> The thread local functions use "long" as the datatype but that will >>>>>> only be 32-bit on 64-bit Windows. I think you need to use intptr_t >>>>>> for complete portability. >>>>> >>>>> The type long has the same format as the type void* which has to be >>>>> portable even on win-32. >>>>> But maybe I'm missing something. >>>>> Anyway, I've replaced it with the intptr_t. >>>>> >>>>> >>>>>> >>>>>> --- >>>>>> >>>>>> 277 printf(" Filed declaring"); >>>>>> >>>>>> typo: filed -> field >>>>> >>>>> >>>>> Good catch - fixed. >>>>> >>>>>> >>>>>> --- >>>>>> >>>>>> All your little wrapper functions make the JVMTI call and then call >>>>>> check_jvmti_error - but all that does is record if it passed or >>>>>> failed. If it failed you still continue with the next operation even >>>>>> if it relies on the success of the first one eg: >>>>>> >>>>>> 378 set_thread_local(jvmti, thread, exp_val); >>>>>> 379 act_val = get_thread_local(jvmti, cur_thread); >>>>>> >>>>>> and the sequences in print_method_info: >>>>>> >>>>>> 228 err = (*jvmti)->IsMethodNative(jvmti, method, &is_native); >>>>>> 229 check_jvmti_error(jvmti, "IsMethodNative", err); >>>>>> 230 printf(" Method is native: %d\n", is_native); >>>>>> 231 >>>>>> 232 if (is_native == JNI_FALSE) { >>>>>> 233 err = (*jvmti)->GetMaxLocals(jvmti, method, >>>>>> &locals_max); >>>>>> >>>>>> The call at #233 may not be valid because the method actually is >>>>>> native but the IsMethodNative call failed for some reason. >>>>>> >>>>> >>>>> It is intentional. I have done it as a last cleanup. >>>>> The point is to simplify code by skipping all the extra checks if it >>>>> does not lead to any fatal errors. >>>>> The most important in such a case is that the static variable result >>>>> is set to FAILED. >>>>> It will cause the test to fail. >>>>> Then there is no point to analyze the printed results if a JVMTI >>>>> error >>>>> reported before. >>>>> >>>>> If you insist, I will add back all the extra check to make sure all >>>>> printed output is valid. >>>>> >>>>> >>>>> Thanks, >>>>> Serguei >>>>> >>>>>> Thanks, >>>>>> David >>>>>> ----- >>>>>> >>>>>>> >>>>>>> >>>>>>> Summary: >>>>>>> The task was to provide a test coverage for the JVMTI functions >>>>>>> allowed during the start phase. >>>>>>> It includes both enabling and disabling the >>>>>>> can_generate_early_vmstart >>>>>>> capability. >>>>>>> Testing the JVMTI functions allowed in any case has not been >>>>>>> targeted >>>>>>> by this fix. >>>>>>> >>>>>>> Testing: >>>>>>> New test is passed. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Serguei >>>>> >>>> >> > From daniel.daugherty at oracle.com Tue May 2 18:09:10 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 2 May 2017 12:09:10 -0600 Subject: RFR(M): 8172970: TESTBUG: need test coverage for the JVMTI functions allowed in the start phase In-Reply-To: <96edf450-1d0d-c21d-436d-bdbd378b4137@oracle.com> References: <88938aee-0a83-99e9-1b95-6875173807a5@oracle.com> <5ea66d4b-f1bb-1729-ba51-5bdcd5b3c470@oracle.com> <1e443485-63d6-4473-8809-76f47a380436@oracle.com> <96edf450-1d0d-c21d-436d-bdbd378b4137@oracle.com> Message-ID: <3f9716a4-ad29-4e71-21b5-802f6019b796@oracle.com> > New webrev version is: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ make/test/JtregNative.gmk No comments. test/serviceability/jvmti/StartPhase/AllowedFunctions/AllowedFunctions.java L27: * @summary Verify the functions that allowed to operate in the start phase Typo: 'that allowed' -> 'that are allowed' L28: * with and without can_generate_early_vmstart capability Please add '.' to the end. test/serviceability/jvmti/StartPhase/AllowedFunctions/libAllowedFunctions.c L27: #include Should this include be in "alpha" order? L115: printf(" ## Error: unexpected class status: 0x%02x\n", status); L117: printf(" Class status: 0x%08x\n", status); Why the different format specifications? "02x" versus "08x"? L126: printf(" class: %s\n", name); L137: printf(" Class source file name: %s\n", name); Please consider adding single-quotes around the %s. L175: check_jvmti_error(jvmti, "GetClassMethods", err); Typo: "GetClassMethods" -> "GetClassFields" L229: err = (*jvmti)->IsMethodObsolete(jvmti, method, & is_obsolete); Please delete space after '&'. L265: check_jvmti_error(jvmti, "GetMethodModifiers", err); Typo: "GetMethodModifiers" -> "GetFieldModifiers" L301: if (methods != NULL) { Typo: 'methods' -> 'fields' This one can result in a memory leak. L308: jvmtiError err; L322: jvmtiError err; 'err' is unused. Please delete it. L396: check_jvmti_error(jvmti, "AddCapabilites", err); Other errors in here include "## Agent_Initialize: "; why not this one? L398: size = (jint)sizeof(callbacks); L399: memset(&callbacks, 0, sizeof(callbacks)); Perhaps use 'size' instead of 'sizeof(callbacks)' since you have it. You have a nice list of functions in the bug report. You might want to include the list of functions that are NOT covered by this test along with a brief comment about why that is okay. Dan On 5/2/17 3:02 AM, serguei.spitsyn at oracle.com wrote: > PING: > I've got a thumbs up from David Holmes. > One more review is needed for this jdk 10 test enhancement. > > Thanks! > Serguei > > > > On 4/28/17 17:13, serguei.spitsyn at oracle.com wrote: >> Hi David, >> >> >> On 4/28/17 10:34, serguei.spitsyn at oracle.com wrote: >>> Hi David, >>> >>> >>> On 4/28/17 04:42, David Holmes wrote: >>>> Hi Serguei, >>>> >>>> On 28/04/2017 6:07 PM, serguei.spitsyn at oracle.com wrote: >>>>> The updated webrev is: >>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.2/ >>>>> >>>> >>>> Thanks for the updates (the issue with long is that on win64 it is >>>> only 32-bit while void* is 64-bit). >>> >>> Ok, thanks. >>> Than you are right, using long on win64 is not compatible. >>> >>>> >>>> I prefer to see fast-fail rather than potentially triggering >>>> cascading failures (check_jvmti_error could even call exit() I >>>> think). But let's see what others think - it's only a preference >>>> not a necessity. >>> >>> Ok, I'll consider call exit() as it would keep it simple. >>> >> >> New webrev version is: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ >> >> >> >> Thanks, >> Serguei >> >> >>> Thanks, >>> Serguei >>> >>>> >>>> Thanks, >>>> David >>>> >>>>> >>>>> I've re-arranged a little bit code in the ClassPrepare callback >>>>> and the >>>>> function test_class_functions(). >>>>> >>>>> Thanks, >>>>> Serguei >>>>> >>>>> >>>>> On 4/28/17 00:47, serguei.spitsyn at oracle.com wrote: >>>>>> Hi David, >>>>>> >>>>>> Thank you for looking at the test! >>>>>> >>>>>> >>>>>> On 4/27/17 23:11, David Holmes wrote: >>>>>>> Hi Serguei, >>>>>>> >>>>>>> On 28/04/2017 3:14 PM, serguei.spitsyn at oracle.com wrote: >>>>>>>> Please, review the jdk 10 fix for the test enhancement: >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8172970 >>>>>>>> >>>>>>>> >>>>>>>> Webrev: >>>>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.1/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> Sorry but I can't quite figure out exactly what this test is doing. >>>>>>> What is the overall call structure here? >>>>>> >>>>>> This is to make sure the functions allowed in the start and live >>>>>> phases work Ok. >>>>>> As the list of functions is pretty big the test does sanity checks >>>>>> that the functions do not crash nor return errors. >>>>>> >>>>>> >>>>>>> I was expecting to see a difference between things that can be >>>>>>> called >>>>>>> at early-start and those that can not - or are these all >>>>>>> expected to >>>>>>> work okay in either case? >>>>>> >>>>>> All these functions are expected to work okay in both cases. >>>>>> Of course, the main concern is the early start. >>>>>> But we have never had such coverage in the past so that the normal >>>>>> start phase needs to be covered too. >>>>>> >>>>>> >>>>>>> >>>>>>> A few comments: >>>>>>> >>>>>>> 44 #define TranslateError(err) "JVMTI error" >>>>>>> >>>>>>> I don't see the point of the above. >>>>>> >>>>>> Good catch - removed. >>>>>> It is a left over from another test that I used as initial template. >>>>>> >>>>>> >>>>>>> --- >>>>>>> >>>>>>> 99 static long get_thread_local(jvmtiEnv *jvmti, jthread thread) { >>>>>>> >>>>>>> The thread local functions use "long" as the datatype but that will >>>>>>> only be 32-bit on 64-bit Windows. I think you need to use intptr_t >>>>>>> for complete portability. >>>>>> >>>>>> The type long has the same format as the type void* which has to be >>>>>> portable even on win-32. >>>>>> But maybe I'm missing something. >>>>>> Anyway, I've replaced it with the intptr_t. >>>>>> >>>>>> >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> 277 printf(" Filed declaring"); >>>>>>> >>>>>>> typo: filed -> field >>>>>> >>>>>> >>>>>> Good catch - fixed. >>>>>> >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> All your little wrapper functions make the JVMTI call and then call >>>>>>> check_jvmti_error - but all that does is record if it passed or >>>>>>> failed. If it failed you still continue with the next operation >>>>>>> even >>>>>>> if it relies on the success of the first one eg: >>>>>>> >>>>>>> 378 set_thread_local(jvmti, thread, exp_val); >>>>>>> 379 act_val = get_thread_local(jvmti, cur_thread); >>>>>>> >>>>>>> and the sequences in print_method_info: >>>>>>> >>>>>>> 228 err = (*jvmti)->IsMethodNative(jvmti, method, &is_native); >>>>>>> 229 check_jvmti_error(jvmti, "IsMethodNative", err); >>>>>>> 230 printf(" Method is native: %d\n", is_native); >>>>>>> 231 >>>>>>> 232 if (is_native == JNI_FALSE) { >>>>>>> 233 err = (*jvmti)->GetMaxLocals(jvmti, method, >>>>>>> &locals_max); >>>>>>> >>>>>>> The call at #233 may not be valid because the method actually is >>>>>>> native but the IsMethodNative call failed for some reason. >>>>>>> >>>>>> >>>>>> It is intentional. I have done it as a last cleanup. >>>>>> The point is to simplify code by skipping all the extra checks if it >>>>>> does not lead to any fatal errors. >>>>>> The most important in such a case is that the static variable result >>>>>> is set to FAILED. >>>>>> It will cause the test to fail. >>>>>> Then there is no point to analyze the printed results if a JVMTI >>>>>> error >>>>>> reported before. >>>>>> >>>>>> If you insist, I will add back all the extra check to make sure all >>>>>> printed output is valid. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Serguei >>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> ----- >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Summary: >>>>>>>> The task was to provide a test coverage for the JVMTI functions >>>>>>>> allowed during the start phase. >>>>>>>> It includes both enabling and disabling the >>>>>>>> can_generate_early_vmstart >>>>>>>> capability. >>>>>>>> Testing the JVMTI functions allowed in any case has not been >>>>>>>> targeted >>>>>>>> by this fix. >>>>>>>> >>>>>>>> Testing: >>>>>>>> New test is passed. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Serguei >>>>>> >>>>> >>> >> > From serguei.spitsyn at oracle.com Tue May 2 18:40:48 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 2 May 2017 11:40:48 -0700 Subject: RFR(M): 8172970: TESTBUG: need test coverage for the JVMTI functions allowed in the start phase In-Reply-To: <3f9716a4-ad29-4e71-21b5-802f6019b796@oracle.com> References: <88938aee-0a83-99e9-1b95-6875173807a5@oracle.com> <5ea66d4b-f1bb-1729-ba51-5bdcd5b3c470@oracle.com> <1e443485-63d6-4473-8809-76f47a380436@oracle.com> <96edf450-1d0d-c21d-436d-bdbd378b4137@oracle.com> <3f9716a4-ad29-4e71-21b5-802f6019b796@oracle.com> Message-ID: Hi Dan, Thank you a lot for the comments! All are nice catches. I have to admit I've done too many typos in new test. Some of them is a result of the 'last minute' changes. The updated webrev is: http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.4/ Thanks, Serguei On 5/2/17 11:09, Daniel D. Daugherty wrote: > > New webrev version is: > > > http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ > > make/test/JtregNative.gmk > No comments. > > test/serviceability/jvmti/StartPhase/AllowedFunctions/AllowedFunctions.java > > L27: * @summary Verify the functions that allowed to operate in > the start phase > Typo: 'that allowed' -> 'that are allowed' > > L28: * with and without can_generate_early_vmstart capability > Please add '.' to the end. > > test/serviceability/jvmti/StartPhase/AllowedFunctions/libAllowedFunctions.c > > L27: #include > Should this include be in "alpha" order? > > L115: printf(" ## Error: unexpected class status: > 0x%02x\n", status); > L117: printf(" Class status: 0x%08x\n", status); > Why the different format specifications? "02x" versus "08x"? > > L126: printf(" class: %s\n", name); > L137: printf(" Class source file name: %s\n", name); > Please consider adding single-quotes around the %s. > > L175: check_jvmti_error(jvmti, "GetClassMethods", err); > Typo: "GetClassMethods" -> "GetClassFields" > > L229: err = (*jvmti)->IsMethodObsolete(jvmti, method, & > is_obsolete); > Please delete space after '&'. > > L265: check_jvmti_error(jvmti, "GetMethodModifiers", err); > Typo: "GetMethodModifiers" -> "GetFieldModifiers" > > L301: if (methods != NULL) { > Typo: 'methods' -> 'fields' > > This one can result in a memory leak. > > L308: jvmtiError err; > L322: jvmtiError err; > 'err' is unused. Please delete it. > > L396: check_jvmti_error(jvmti, "AddCapabilites", err); > Other errors in here include "## Agent_Initialize: "; why not > this one? > > L398: size = (jint)sizeof(callbacks); > L399: memset(&callbacks, 0, sizeof(callbacks)); > Perhaps use 'size' instead of 'sizeof(callbacks)' since you > have it. > > > You have a nice list of functions in the bug report. > You might want to include the list of functions that > are NOT covered by this test along with a brief comment > about why that is okay. > > Dan > > > On 5/2/17 3:02 AM, serguei.spitsyn at oracle.com wrote: >> PING: >> I've got a thumbs up from David Holmes. >> One more review is needed for this jdk 10 test enhancement. >> >> Thanks! >> Serguei >> >> >> >> On 4/28/17 17:13, serguei.spitsyn at oracle.com wrote: >>> Hi David, >>> >>> >>> On 4/28/17 10:34, serguei.spitsyn at oracle.com wrote: >>>> Hi David, >>>> >>>> >>>> On 4/28/17 04:42, David Holmes wrote: >>>>> Hi Serguei, >>>>> >>>>> On 28/04/2017 6:07 PM, serguei.spitsyn at oracle.com wrote: >>>>>> The updated webrev is: >>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.2/ >>>>>> >>>>> >>>>> Thanks for the updates (the issue with long is that on win64 it is >>>>> only 32-bit while void* is 64-bit). >>>> >>>> Ok, thanks. >>>> Than you are right, using long on win64 is not compatible. >>>> >>>>> >>>>> I prefer to see fast-fail rather than potentially triggering >>>>> cascading failures (check_jvmti_error could even call exit() I >>>>> think). But let's see what others think - it's only a preference >>>>> not a necessity. >>>> >>>> Ok, I'll consider call exit() as it would keep it simple. >>>> >>> >>> New webrev version is: >>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ >>> >>> >>> >>> Thanks, >>> Serguei >>> >>> >>>> Thanks, >>>> Serguei >>>> >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> >>>>>> I've re-arranged a little bit code in the ClassPrepare callback >>>>>> and the >>>>>> function test_class_functions(). >>>>>> >>>>>> Thanks, >>>>>> Serguei >>>>>> >>>>>> >>>>>> On 4/28/17 00:47, serguei.spitsyn at oracle.com wrote: >>>>>>> Hi David, >>>>>>> >>>>>>> Thank you for looking at the test! >>>>>>> >>>>>>> >>>>>>> On 4/27/17 23:11, David Holmes wrote: >>>>>>>> Hi Serguei, >>>>>>>> >>>>>>>> On 28/04/2017 3:14 PM, serguei.spitsyn at oracle.com wrote: >>>>>>>>> Please, review the jdk 10 fix for the test enhancement: >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8172970 >>>>>>>>> >>>>>>>>> >>>>>>>>> Webrev: >>>>>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.1/ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Sorry but I can't quite figure out exactly what this test is >>>>>>>> doing. >>>>>>>> What is the overall call structure here? >>>>>>> >>>>>>> This is to make sure the functions allowed in the start and live >>>>>>> phases work Ok. >>>>>>> As the list of functions is pretty big the test does sanity checks >>>>>>> that the functions do not crash nor return errors. >>>>>>> >>>>>>> >>>>>>>> I was expecting to see a difference between things that can be >>>>>>>> called >>>>>>>> at early-start and those that can not - or are these all >>>>>>>> expected to >>>>>>>> work okay in either case? >>>>>>> >>>>>>> All these functions are expected to work okay in both cases. >>>>>>> Of course, the main concern is the early start. >>>>>>> But we have never had such coverage in the past so that the normal >>>>>>> start phase needs to be covered too. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> A few comments: >>>>>>>> >>>>>>>> 44 #define TranslateError(err) "JVMTI error" >>>>>>>> >>>>>>>> I don't see the point of the above. >>>>>>> >>>>>>> Good catch - removed. >>>>>>> It is a left over from another test that I used as initial >>>>>>> template. >>>>>>> >>>>>>> >>>>>>>> --- >>>>>>>> >>>>>>>> 99 static long get_thread_local(jvmtiEnv *jvmti, jthread >>>>>>>> thread) { >>>>>>>> >>>>>>>> The thread local functions use "long" as the datatype but that >>>>>>>> will >>>>>>>> only be 32-bit on 64-bit Windows. I think you need to use intptr_t >>>>>>>> for complete portability. >>>>>>> >>>>>>> The type long has the same format as the type void* which has to be >>>>>>> portable even on win-32. >>>>>>> But maybe I'm missing something. >>>>>>> Anyway, I've replaced it with the intptr_t. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> --- >>>>>>>> >>>>>>>> 277 printf(" Filed declaring"); >>>>>>>> >>>>>>>> typo: filed -> field >>>>>>> >>>>>>> >>>>>>> Good catch - fixed. >>>>>>> >>>>>>>> >>>>>>>> --- >>>>>>>> >>>>>>>> All your little wrapper functions make the JVMTI call and then >>>>>>>> call >>>>>>>> check_jvmti_error - but all that does is record if it passed or >>>>>>>> failed. If it failed you still continue with the next operation >>>>>>>> even >>>>>>>> if it relies on the success of the first one eg: >>>>>>>> >>>>>>>> 378 set_thread_local(jvmti, thread, exp_val); >>>>>>>> 379 act_val = get_thread_local(jvmti, cur_thread); >>>>>>>> >>>>>>>> and the sequences in print_method_info: >>>>>>>> >>>>>>>> 228 err = (*jvmti)->IsMethodNative(jvmti, method, >>>>>>>> &is_native); >>>>>>>> 229 check_jvmti_error(jvmti, "IsMethodNative", err); >>>>>>>> 230 printf(" Method is native: %d\n", is_native); >>>>>>>> 231 >>>>>>>> 232 if (is_native == JNI_FALSE) { >>>>>>>> 233 err = (*jvmti)->GetMaxLocals(jvmti, method, >>>>>>>> &locals_max); >>>>>>>> >>>>>>>> The call at #233 may not be valid because the method actually is >>>>>>>> native but the IsMethodNative call failed for some reason. >>>>>>>> >>>>>>> >>>>>>> It is intentional. I have done it as a last cleanup. >>>>>>> The point is to simplify code by skipping all the extra checks >>>>>>> if it >>>>>>> does not lead to any fatal errors. >>>>>>> The most important in such a case is that the static variable >>>>>>> result >>>>>>> is set to FAILED. >>>>>>> It will cause the test to fail. >>>>>>> Then there is no point to analyze the printed results if a JVMTI >>>>>>> error >>>>>>> reported before. >>>>>>> >>>>>>> If you insist, I will add back all the extra check to make sure all >>>>>>> printed output is valid. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Serguei >>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> ----- >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Summary: >>>>>>>>> The task was to provide a test coverage for the JVMTI functions >>>>>>>>> allowed during the start phase. >>>>>>>>> It includes both enabling and disabling the >>>>>>>>> can_generate_early_vmstart >>>>>>>>> capability. >>>>>>>>> Testing the JVMTI functions allowed in any case has not been >>>>>>>>> targeted >>>>>>>>> by this fix. >>>>>>>>> >>>>>>>>> Testing: >>>>>>>>> New test is passed. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Serguei >>>>>>> >>>>>> >>>> >>> >> > From daniel.daugherty at oracle.com Tue May 2 19:23:10 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 2 May 2017 13:23:10 -0600 Subject: RFR(M): 8172970: TESTBUG: need test coverage for the JVMTI functions allowed in the start phase In-Reply-To: References: <88938aee-0a83-99e9-1b95-6875173807a5@oracle.com> <5ea66d4b-f1bb-1729-ba51-5bdcd5b3c470@oracle.com> <1e443485-63d6-4473-8809-76f47a380436@oracle.com> <96edf450-1d0d-c21d-436d-bdbd378b4137@oracle.com> <3f9716a4-ad29-4e71-21b5-802f6019b796@oracle.com> Message-ID: <6f4f5fec-6733-85c4-a741-20acb469f0e7@oracle.com> On 5/2/17 12:40 PM, serguei.spitsyn at oracle.com wrote: > Hi Dan, > > Thank you a lot for the comments! > All are nice catches. > I have to admit I've done too many typos in new test. > Some of them is a result of the 'last minute' changes. > > The updated webrev is: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.4/ > make/test/JtregNative.gmk No comments. test/serviceability/jvmti/StartPhase/AllowedFunctions/AllowedFunctions.java No comments. test/serviceability/jvmti/StartPhase/AllowedFunctions/libAllowedFunctions.c L126: printf(" class: \'%s\'\n", name); L137: printf(" Class source file name: \'%s\'\n", name); You don't need to escape the single-quotes with backslash here. Thumbs up! Dan > > Thanks, > Serguei > > > On 5/2/17 11:09, Daniel D. Daugherty wrote: >> > New webrev version is: >> > >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ >> >> make/test/JtregNative.gmk >> No comments. >> >> test/serviceability/jvmti/StartPhase/AllowedFunctions/AllowedFunctions.java >> >> L27: * @summary Verify the functions that allowed to operate in >> the start phase >> Typo: 'that allowed' -> 'that are allowed' >> >> L28: * with and without can_generate_early_vmstart capability >> Please add '.' to the end. >> >> test/serviceability/jvmti/StartPhase/AllowedFunctions/libAllowedFunctions.c >> >> L27: #include >> Should this include be in "alpha" order? >> >> L115: printf(" ## Error: unexpected class status: >> 0x%02x\n", status); >> L117: printf(" Class status: 0x%08x\n", status); >> Why the different format specifications? "02x" versus "08x"? >> >> L126: printf(" class: %s\n", name); >> L137: printf(" Class source file name: %s\n", name); >> Please consider adding single-quotes around the %s. >> >> L175: check_jvmti_error(jvmti, "GetClassMethods", err); >> Typo: "GetClassMethods" -> "GetClassFields" >> >> L229: err = (*jvmti)->IsMethodObsolete(jvmti, method, & >> is_obsolete); >> Please delete space after '&'. >> >> L265: check_jvmti_error(jvmti, "GetMethodModifiers", err); >> Typo: "GetMethodModifiers" -> "GetFieldModifiers" >> >> L301: if (methods != NULL) { >> Typo: 'methods' -> 'fields' >> >> This one can result in a memory leak. >> >> L308: jvmtiError err; >> L322: jvmtiError err; >> 'err' is unused. Please delete it. >> >> L396: check_jvmti_error(jvmti, "AddCapabilites", err); >> Other errors in here include "## Agent_Initialize: "; why not >> this one? >> >> L398: size = (jint)sizeof(callbacks); >> L399: memset(&callbacks, 0, sizeof(callbacks)); >> Perhaps use 'size' instead of 'sizeof(callbacks)' since you >> have it. >> >> >> You have a nice list of functions in the bug report. >> You might want to include the list of functions that >> are NOT covered by this test along with a brief comment >> about why that is okay. >> >> Dan >> >> >> On 5/2/17 3:02 AM, serguei.spitsyn at oracle.com wrote: >>> PING: >>> I've got a thumbs up from David Holmes. >>> One more review is needed for this jdk 10 test enhancement. >>> >>> Thanks! >>> Serguei >>> >>> >>> >>> On 4/28/17 17:13, serguei.spitsyn at oracle.com wrote: >>>> Hi David, >>>> >>>> >>>> On 4/28/17 10:34, serguei.spitsyn at oracle.com wrote: >>>>> Hi David, >>>>> >>>>> >>>>> On 4/28/17 04:42, David Holmes wrote: >>>>>> Hi Serguei, >>>>>> >>>>>> On 28/04/2017 6:07 PM, serguei.spitsyn at oracle.com wrote: >>>>>>> The updated webrev is: >>>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.2/ >>>>>>> >>>>>> >>>>>> Thanks for the updates (the issue with long is that on win64 it >>>>>> is only 32-bit while void* is 64-bit). >>>>> >>>>> Ok, thanks. >>>>> Than you are right, using long on win64 is not compatible. >>>>> >>>>>> >>>>>> I prefer to see fast-fail rather than potentially triggering >>>>>> cascading failures (check_jvmti_error could even call exit() I >>>>>> think). But let's see what others think - it's only a preference >>>>>> not a necessity. >>>>> >>>>> Ok, I'll consider call exit() as it would keep it simple. >>>>> >>>> >>>> New webrev version is: >>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.3/ >>>> >>>> >>>> >>>> Thanks, >>>> Serguei >>>> >>>> >>>>> Thanks, >>>>> Serguei >>>>> >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> >>>>>>> I've re-arranged a little bit code in the ClassPrepare callback >>>>>>> and the >>>>>>> function test_class_functions(). >>>>>>> >>>>>>> Thanks, >>>>>>> Serguei >>>>>>> >>>>>>> >>>>>>> On 4/28/17 00:47, serguei.spitsyn at oracle.com wrote: >>>>>>>> Hi David, >>>>>>>> >>>>>>>> Thank you for looking at the test! >>>>>>>> >>>>>>>> >>>>>>>> On 4/27/17 23:11, David Holmes wrote: >>>>>>>>> Hi Serguei, >>>>>>>>> >>>>>>>>> On 28/04/2017 3:14 PM, serguei.spitsyn at oracle.com wrote: >>>>>>>>>> Please, review the jdk 10 fix for the test enhancement: >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8172970 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Webrev: >>>>>>>>>> http://cr.openjdk.java.net/~sspitsyn/webrevs/2017/hotspot/8172970-start-phase.1/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> Sorry but I can't quite figure out exactly what this test is >>>>>>>>> doing. >>>>>>>>> What is the overall call structure here? >>>>>>>> >>>>>>>> This is to make sure the functions allowed in the start and live >>>>>>>> phases work Ok. >>>>>>>> As the list of functions is pretty big the test does sanity checks >>>>>>>> that the functions do not crash nor return errors. >>>>>>>> >>>>>>>> >>>>>>>>> I was expecting to see a difference between things that can be >>>>>>>>> called >>>>>>>>> at early-start and those that can not - or are these all >>>>>>>>> expected to >>>>>>>>> work okay in either case? >>>>>>>> >>>>>>>> All these functions are expected to work okay in both cases. >>>>>>>> Of course, the main concern is the early start. >>>>>>>> But we have never had such coverage in the past so that the normal >>>>>>>> start phase needs to be covered too. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> A few comments: >>>>>>>>> >>>>>>>>> 44 #define TranslateError(err) "JVMTI error" >>>>>>>>> >>>>>>>>> I don't see the point of the above. >>>>>>>> >>>>>>>> Good catch - removed. >>>>>>>> It is a left over from another test that I used as initial >>>>>>>> template. >>>>>>>> >>>>>>>> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> 99 static long get_thread_local(jvmtiEnv *jvmti, jthread >>>>>>>>> thread) { >>>>>>>>> >>>>>>>>> The thread local functions use "long" as the datatype but that >>>>>>>>> will >>>>>>>>> only be 32-bit on 64-bit Windows. I think you need to use >>>>>>>>> intptr_t >>>>>>>>> for complete portability. >>>>>>>> >>>>>>>> The type long has the same format as the type void* which has >>>>>>>> to be >>>>>>>> portable even on win-32. >>>>>>>> But maybe I'm missing something. >>>>>>>> Anyway, I've replaced it with the intptr_t. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> 277 printf(" Filed declaring"); >>>>>>>>> >>>>>>>>> typo: filed -> field >>>>>>>> >>>>>>>> >>>>>>>> Good catch - fixed. >>>>>>>> >>>>>>>>> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> All your little wrapper functions make the JVMTI call and then >>>>>>>>> call >>>>>>>>> check_jvmti_error - but all that does is record if it passed or >>>>>>>>> failed. If it failed you still continue with the next >>>>>>>>> operation even >>>>>>>>> if it relies on the success of the first one eg: >>>>>>>>> >>>>>>>>> 378 set_thread_local(jvmti, thread, exp_val); >>>>>>>>> 379 act_val = get_thread_local(jvmti, cur_thread); >>>>>>>>> >>>>>>>>> and the sequences in print_method_info: >>>>>>>>> >>>>>>>>> 228 err = (*jvmti)->IsMethodNative(jvmti, method, >>>>>>>>> &is_native); >>>>>>>>> 229 check_jvmti_error(jvmti, "IsMethodNative", err); >>>>>>>>> 230 printf(" Method is native: %d\n", is_native); >>>>>>>>> 231 >>>>>>>>> 232 if (is_native == JNI_FALSE) { >>>>>>>>> 233 err = (*jvmti)->GetMaxLocals(jvmti, method, >>>>>>>>> &locals_max); >>>>>>>>> >>>>>>>>> The call at #233 may not be valid because the method actually is >>>>>>>>> native but the IsMethodNative call failed for some reason. >>>>>>>>> >>>>>>>> >>>>>>>> It is intentional. I have done it as a last cleanup. >>>>>>>> The point is to simplify code by skipping all the extra checks >>>>>>>> if it >>>>>>>> does not lead to any fatal errors. >>>>>>>> The most important in such a case is that the static variable >>>>>>>> result >>>>>>>> is set to FAILED. >>>>>>>> It will cause the test to fail. >>>>>>>> Then there is no point to analyze the printed results if a >>>>>>>> JVMTI error >>>>>>>> reported before. >>>>>>>> >>>>>>>> If you insist, I will add back all the extra check to make sure >>>>>>>> all >>>>>>>> printed output is valid. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Serguei >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> ----- >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Summary: >>>>>>>>>> The task was to provide a test coverage for the JVMTI >>>>>>>>>> functions >>>>>>>>>> allowed during the start phase. >>>>>>>>>> It includes both enabling and disabling the >>>>>>>>>> can_generate_early_vmstart >>>>>>>>>> capability. >>>>>>>>>> Testing the JVMTI functions allowed in any case has not >>>>>>>>>> been targeted >>>>>>>>>> by this fix. >>>>>>>>>> >>>>>>>>>> Testing: >>>>>>>>>> New test is passed. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Serguei >>>>>>>> >>>>>>> >>>>> >>>> >>> >> > From kumar.x.srinivasan at oracle.com Tue May 2 19:40:48 2017 From: kumar.x.srinivasan at oracle.com (Kumar Srinivasan) Date: Tue, 02 May 2017 12:40:48 -0700 Subject: RFR: JDK-8179538: Update jdk.jdi to be HTML-5 friendly Message-ID: <5908E0C0.1050100@oracle.com> Hello, Please review changes to make jdk.jdi HTML5 friendly, table cell padding has not been addressed and will be fixed separately, this takes care of others. Note: Some of the errors were due to incorrect at throws but in all cases there is the correct one further down, please use sdiffs in these cases. @throws {@link InvalidTypeException} if any.... before:jdk.jdi {error=42, warning=1} after: jdk.jdi {error=1} http://cr.openjdk.java.net/~ksrini/8179538/webrev.00/index.html https://bugs.openjdk.java.net/browse/JDK-8179538 Thanks Kumar From mandy.chung at oracle.com Tue May 2 23:23:06 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 2 May 2017 16:23:06 -0700 Subject: RFR: JDK-8179538: Update jdk.jdi to be HTML-5 friendly In-Reply-To: <5908E0C0.1050100@oracle.com> References: <5908E0C0.1050100@oracle.com> Message-ID: <19AEFC8E-F440-4FAA-9A39-F06164A3C575@oracle.com> Looks fine to me. Mandy > On May 2, 2017, at 12:40 PM, Kumar Srinivasan wrote: > > Hello, > > Please review changes to make jdk.jdi HTML5 friendly, > table cell padding has not been addressed and will be fixed > separately, this takes care of others. > > Note: Some of the errors were due to incorrect at throws but > in all cases there is the correct one further down, please use > sdiffs in these cases. > > @throws {@link InvalidTypeException} if any.... > > before:jdk.jdi {error=42, warning=1} > after: jdk.jdi {error=1} > > http://cr.openjdk.java.net/~ksrini/8179538/webrev.00/index.html > https://bugs.openjdk.java.net/browse/JDK-8179538 > > Thanks > Kumar > From erik.gahlin at oracle.com Tue May 2 23:51:06 2017 From: erik.gahlin at oracle.com (Erik Gahlin) Date: Wed, 3 May 2017 01:51:06 +0200 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: References: Message-ID: <59091B6A.5070908@oracle.com> I am not a (R)eviewer, so I can't give it my blessings, but I can sponsor it. I will create a webrev so it easier to review. Erik > I have attached a patch that allows jcmd to work against a java process running > inside a Docker container. Apologies if this is not in the correct format. It was > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > that is inside a container (e.g. Docker). > > A Linux container often isolates a process in a PID and Mount namespace that is > separate from the "root container" (analogous to the hypervisor/dom0 in > hardware virtualization environments, or the global zone on Solaris). A target > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > the attach sequence. > > When the target JVM is in its own PID namespace the pid of the process is > distinct from what the real pid of the process as it relates to the root > container. For example, in the root container you can observe a JVM with a pid > of 17734, however if that JVM is running inside a Docker container the pid > inside its PID namespace is likely 1. So when the target JVM receives the > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > discrepancy the target JVM will output to stderr thread status, since > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > sequence. > > The solution is to parse /proc/pid/status for the field NSpid (available since > Linux 4.1) which contains a list of pids, where the last entry is the "inner > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > which have a virtualization depth of 1) > > The rest of the Linux attach sequence assumes a shared mount namespace by > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > separate namespace because the target JVM is in a mount namepsace (or in a > chroot as well) the unix domain socket for attaching won't appear. > > Instead the attach sequence should resolve file names relative to > /proc/17734/root which has a materialized view of the rootfs for the target. > > Thanks! > > TJ > From serguei.spitsyn at oracle.com Tue May 2 23:56:05 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 2 May 2017 16:56:05 -0700 Subject: RFR: JDK-8179538: Update jdk.jdi to be HTML-5 friendly In-Reply-To: <19AEFC8E-F440-4FAA-9A39-F06164A3C575@oracle.com> References: <5908E0C0.1050100@oracle.com> <19AEFC8E-F440-4FAA-9A39-F06164A3C575@oracle.com> Message-ID: Hi Kumar, It looks fine to me too. Thanks, Serguei On 5/2/17 16:23, Mandy Chung wrote: > Looks fine to me. > > Mandy > >> On May 2, 2017, at 12:40 PM, Kumar Srinivasan wrote: >> >> Hello, >> >> Please review changes to make jdk.jdi HTML5 friendly, >> table cell padding has not been addressed and will be fixed >> separately, this takes care of others. >> >> Note: Some of the errors were due to incorrect at throws but >> in all cases there is the correct one further down, please use >> sdiffs in these cases. >> >> @throws {@link InvalidTypeException} if any.... >> >> before:jdk.jdi {error=42, warning=1} >> after: jdk.jdi {error=1} >> >> http://cr.openjdk.java.net/~ksrini/8179538/webrev.00/index.html >> https://bugs.openjdk.java.net/browse/JDK-8179538 >> >> Thanks >> Kumar >> From mandy.chung at oracle.com Wed May 3 00:39:08 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 2 May 2017 17:39:08 -0700 Subject: VisualVM CPU sampling not working In-Reply-To: <97CDC8E9-A588-4DA7-B5AF-289D8402505D@cbfiddle.com> References: <97CDC8E9-A588-4DA7-B5AF-289D8402505D@cbfiddle.com> Message-ID: serviceability-dev (cc?ed) is the proper mailing list for this issue and so bcc jdk9-dev. https://bugs.openjdk.java.net/browse/JDK-8167121 has been pushed to jdk8u-dev. We will need to get this fix in a 8u release. Mandy > On May 2, 2017, at 2:15 PM, Alan Snyder wrote: > > Is this a known problem? It resembles JDK-8165005. Makes it hard to investigate performance problems if the tools don't work. Are there other tools that work? > > This is VisualVM 1.39 on an application running under jdk9-ea+166. > > It reports: > > Cannot access threads in target application. > > The log file shows: > > java.lang.IllegalArgumentException: Unexpected composite type for ThreadInfo > at sun.management.ThreadInfoCompositeData.validateCompositeData(ThreadInfoCompositeData.java:372) > at sun.management.ThreadInfoCompositeData.getInstance(ThreadInfoCompositeData.java:68) > at java.lang.management.ThreadInfo.(ThreadInfo.java:263) > at java.lang.management.ThreadInfo.from(ThreadInfo.java:794) > > Caused: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1018) > > Caused: java.io.InvalidObjectException: Failed to invoke from(CompositeData) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory.invalidObjectException(DefaultMXBeanMappingFactory.java:1457) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1021) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeMapping.fromNonNullOpenValue(DefaultMXBeanMappingFactory.java:919) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.fromOpenValue(DefaultMXBeanMappingFactory.java:133) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$ArrayMapping.fromNonNullOpenValue(DefaultMXBeanMappingFactory.java:584) > at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.fromOpenValue(DefaultMXBeanMappingFactory.java:133) > at com.sun.jmx.mbeanserver.ConvertingMethod.fromOpenReturnValue(ConvertingMethod.java:131) > at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:168) > at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258) > > Caused: java.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy16.dumpAllThreads(Unknown Source) > at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.dumpAllThreads(ThreadInfoProvider.java:103) > > [catch] at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.initialize(ThreadInfoProvider.java:88) > at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.(ThreadInfoProvider.java:54) > at com.sun.tools.visualvm.sampler.SamplerImpl$11.run(SamplerImpl.java:485) > at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423) > at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033) > From thomas.stuefe at gmail.com Wed May 3 06:43:35 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 3 May 2017 08:43:35 +0200 Subject: Question about location of libdt_socket vs libdt_shmem Message-ID: Hi all, I wonder why the sources of the transport libraries are located in different places: jdk/src/jdk.jdwp.agent/share/native/libdt_socket jdk/src/jdk.jdi/share/native/libdt_shmem Is there a particular reason to this? Seeing as they both are implementations of the jdwp sp interface, I would have expected them to be located side by side? Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Wed May 3 06:49:31 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 2 May 2017 23:49:31 -0700 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <59091B6A.5070908@oracle.com> References: <59091B6A.5070908@oracle.com> Message-ID: <914df43a-f136-2368-8df1-0a715945bdea@oracle.com> Hi TJ, The fix looks good to me, modulo the suggestion from David about the comment at L345. However, I'm not an expert in the namespaces area. It'd be nice to hear from other people. This has to be tested at least when the namespaces are not present. Testing should include the J*tools (jcmd, jstack, jmap, etc.) and the attach API tests. Not sure, if you can run them. But your sponsor, Erik, can probably run them for you. Also, it would be nice to have one new Jtreg test verifying this patch with a namespace. Not sure, it is a strong requirement though. Erik, Thank you for sponsoring this patch! David has already created the webrev: http://cr.openjdk.java.net/~dholmes/8179498/webrev/ Thanks, Serguei On 5/2/17 16:51, Erik Gahlin wrote: > I am not a (R)eviewer, so I can't give it my blessings, but I can > sponsor it. > > I will create a webrev so it easier to review. > > Erik > >> I have attached a patch that allows jcmd to work against a java >> process running >> inside a Docker container. Apologies if this is not in the correct >> format. It was >> built against jdk8u. I couldn?t seem to find an existing JIRA for it. >> >> Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a >> target JVM >> that is inside a container (e.g. Docker). >> >> A Linux container often isolates a process in a PID and Mount >> namespace that is >> separate from the "root container" (analogous to the hypervisor/dom0 in >> hardware virtualization environments, or the global zone on Solaris). >> A target >> JVM that is isolated in either a PID namespace, or a Mount namespace >> will fail >> the attach sequence. >> >> When the target JVM is in its own PID namespace the pid of the >> process is >> distinct from what the real pid of the process as it relates to the root >> container. For example, in the root container you can observe a JVM >> with a pid >> of 17734, however if that JVM is running inside a Docker container >> the pid >> inside its PID namespace is likely 1. So when the target JVM receives >> the >> SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the >> external >> attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. >> Given this >> discrepancy the target JVM will output to stderr thread status, since >> /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the >> attach >> sequence. >> >> The solution is to parse /proc/pid/status for the field NSpid >> (available since >> Linux 4.1) which contains a list of pids, where the last entry is the >> "inner >> most" PID namespace value. (Namespaces can be stacked, unlike Solaris >> Zones >> which have a virtualization depth of 1) >> >> The rest of the Linux attach sequence assumes a shared mount >> namespace by >> waiting for /tmp/.java_pid17734 to appear. But if the attaching >> process is in a >> separate namespace because the target JVM is in a mount namepsace (or >> in a >> chroot as well) the unix domain socket for attaching won't appear. >> >> Instead the attach sequence should resolve file names relative to >> /proc/17734/root which has a materialized view of the rootfs for the >> target. >> >> Thanks! >> >> TJ >> > From Alan.Bateman at oracle.com Wed May 3 07:03:04 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 3 May 2017 08:03:04 +0100 Subject: Question about location of libdt_socket vs libdt_shmem In-Reply-To: References: Message-ID: On 03/05/2017 07:43, Thomas St?fe wrote: > Hi all, > > I wonder why the sources of the transport libraries are located in > different places: > > jdk/src/jdk.jdwp.agent/share/native/libdt_socket > jdk/src/jdk.jdi/share/native/libdt_shmem > > Is there a particular reason to this? > > Seeing as they both are implementations of the jdwp sp interface, I > would have expected them to be located side by side? > jdk.jdwp.agent is the backend/debuggee side so it has the JDWP agent and should contain the transport libraries used by the JDWP agent. jdk.jdi is the frontend/debugger side so it has the JDI API and the connector and transport services that JDI uses. For the shared memory transport then it should ideally be split so that a reduced libdt_shmem is in jdk.jdwp.agent and the JDI side is is jdk.jdi. Refactoring is needed to do this right and just hasn't been a priority. BTW: Having it in jdk.jdi isn't actually too bad because the shared memory transport is local (debugger and debuggee on the same machine). Also it tends to be used by the launching connector which is in jdk.jdi. -Alan From thomas.stuefe at gmail.com Wed May 3 08:47:02 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 3 May 2017 10:47:02 +0200 Subject: Question about location of libdt_socket vs libdt_shmem In-Reply-To: References: Message-ID: Hi Alan, On Wed, May 3, 2017 at 9:03 AM, Alan Bateman wrote: > On 03/05/2017 07:43, Thomas St?fe wrote: > > Hi all, >> >> I wonder why the sources of the transport libraries are located in >> different places: >> >> jdk/src/jdk.jdwp.agent/share/native/libdt_socket >> jdk/src/jdk.jdi/share/native/libdt_shmem >> >> Is there a particular reason to this? >> >> Seeing as they both are implementations of the jdwp sp interface, I would >> have expected them to be located side by side? >> >> jdk.jdwp.agent is the backend/debuggee side so it has the JDWP agent and > should contain the transport libraries used by the JDWP agent. > > jdk.jdi is the frontend/debugger side so it has the JDI API and the > connector and transport services that JDI uses. > > For the shared memory transport then it should ideally be split so that a > reduced libdt_shmem is in jdk.jdwp.agent and the JDI side is is jdk.jdi. > Refactoring is needed to do this right and just hasn't been a priority. > > And for the socket transport, it is not needed at the jdi side because there the socket transport is reimplemented in pure java? > BTW: Having it in jdk.jdi isn't actually too bad because the shared memory > transport is local (debugger and debuggee on the same machine). Also it > tends to be used by the launching connector which is in jdk.jdi. thank you for clarifying! Thanks, Thomas > > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed May 3 09:03:31 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 3 May 2017 10:03:31 +0100 Subject: Question about location of libdt_socket vs libdt_shmem In-Reply-To: References: Message-ID: <9e262171-1818-a4d5-b651-b1ad6eb7117f@oracle.com> On 03/05/2017 09:47, Thomas St?fe wrote: > : > > And for the socket transport, it is not needed at the jdi side because > there the socket transport is reimplemented in pure java? > That's right. If there was an issue with the socket transport then it would have been fixed a long time ago - the reason is that the socket transport supports remote debugging and important to be able to have a tool on one machine using JDI to debug a target on another machine. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed May 3 09:31:36 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 3 May 2017 11:31:36 +0200 Subject: Question about location of libdt_socket vs libdt_shmem In-Reply-To: <9e262171-1818-a4d5-b651-b1ad6eb7117f@oracle.com> References: <9e262171-1818-a4d5-b651-b1ad6eb7117f@oracle.com> Message-ID: Hi Alan, thanks! One more question: I find no platform dependent coding on for the shared memory transport layer for Unix, just for Windows. I would have expected some mmap() calls somewhere, and a LoadLibrary(dt_shmem). Looking closer, I also cannot find a libdt_shmem.so on a jdk10 linux build. How does shared memory transport work on Unix? ..Thomas On Wed, May 3, 2017 at 11:03 AM, Alan Bateman wrote: > > > On 03/05/2017 09:47, Thomas St?fe wrote: > > : > > And for the socket transport, it is not needed at the jdi side because > there the socket transport is reimplemented in pure java? > > That's right. If there was an issue with the socket transport then it > would have been fixed a long time ago - the reason is that the socket > transport supports remote debugging and important to be able to have a tool > on one machine using JDI to debug a target on another machine. > > -Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed May 3 09:36:51 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 3 May 2017 11:36:51 +0200 Subject: Question about location of libdt_socket vs libdt_shmem In-Reply-To: References: <9e262171-1818-a4d5-b651-b1ad6eb7117f@oracle.com> Message-ID: On Wed, May 3, 2017 at 11:31 AM, Thomas St?fe wrote: > Hi Alan, > > thanks! One more question: I find no platform dependent coding on for the > shared memory transport layer for Unix, just for Windows. I would have > expected some mmap() calls somewhere, and a LoadLibrary(dt_shmem). Looking > closer, I also cannot find a libdt_shmem.so on a jdk10 linux build. > > How does shared memory transport work on Unix? > > Ah, just found out dt_shmem was only ever supported on Windows according to https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/conninv.html. Please disregard my last question then. Thanks! Thomas > ..Thomas > > > On Wed, May 3, 2017 at 11:03 AM, Alan Bateman > wrote: > >> >> >> On 03/05/2017 09:47, Thomas St?fe wrote: >> >> : >> >> And for the socket transport, it is not needed at the jdi side because >> there the socket transport is reimplemented in pure java? >> >> That's right. If there was an issue with the socket transport then it >> would have been fixed a long time ago - the reason is that the socket >> transport supports remote debugging and important to be able to have a tool >> on one machine using JDI to debug a target on another machine. >> >> -Alan >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.gahlin at oracle.com Wed May 3 13:14:29 2017 From: erik.gahlin at oracle.com (Erik Gahlin) Date: Wed, 3 May 2017 15:14:29 +0200 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> References: <36e2184f-6527-0aad-b156-b12719cfbd8a@oracle.com> <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> Message-ID: <5909D7B5.2030400@oracle.com> I noticed thatgetNamespacePid throws IOException and InvalidPathException. Perhaps we want to catch those, so we can default to the original pid if there is a I/O related problem. Erik > Hey, > > I?ve attached a version rebased on jdk10, it also (currently) applies cleanly to jdk9. > > While there is no supplied test or harness for this patch, how I built and tested is > available at https://github.com/tjfontaine/jdkbuild (there?s also a preview of my > follow on patch for pathmap_open as well). > > Thanks! > > TJ > > On 4/28/17, 3:47 PM, "serviceability-dev on behalf of TJ Fontaine" wrote: > > I had no doubt we?d end up on the conversation of 10 -> 9 -> 8u, I started with 8u merely because it was representative of today?s customer pain. I?ll be sure to work on retargeting it as well. > > Thanks! > > TJ > > On 4/28/17, 3:42 PM, "David Holmes" wrote: > > Hi TJ, > > Thanks for the patch (I haven't looked at it yet). FYI at the moment, > unless this is considered a high priority bug for JDK 9 it has to be > targeted to JDK 10, and then possibly backported to 9 and 8u. > > Cheers, > David > > On 29/04/2017 8:23 AM, TJ Fontaine wrote: > > I have attached a patch that allows jcmd to work against a java process running > > inside a Docker container. Apologies if this is not in the correct format. It was > > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > > that is inside a container (e.g. Docker). > > > > A Linux container often isolates a process in a PID and Mount namespace that is > > separate from the "root container" (analogous to the hypervisor/dom0 in > > hardware virtualization environments, or the global zone on Solaris). A target > > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > > the attach sequence. > > > > When the target JVM is in its own PID namespace the pid of the process is > > distinct from what the real pid of the process as it relates to the root > > container. For example, in the root container you can observe a JVM with a pid > > of 17734, however if that JVM is running inside a Docker container the pid > > inside its PID namespace is likely 1. So when the target JVM receives the > > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > > discrepancy the target JVM will output to stderr thread status, since > > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > > sequence. > > > > The solution is to parse /proc/pid/status for the field NSpid (available since > > Linux 4.1) which contains a list of pids, where the last entry is the "inner > > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > > which have a virtualization depth of 1) > > > > The rest of the Linux attach sequence assumes a shared mount namespace by > > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > > separate namespace because the target JVM is in a mount namepsace (or in a > > chroot as well) the unix domain socket for attaching won't appear. > > > > Instead the attach sequence should resolve file names relative to > > /proc/17734/root which has a materialized view of the rootfs for the target. > > > > Thanks! > > > > TJ > > > > > > > From tj.fontaine at oracle.com Wed May 3 15:40:48 2017 From: tj.fontaine at oracle.com (TJ Fontaine) Date: Wed, 3 May 2017 08:40:48 -0700 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <5909D7B5.2030400@oracle.com> References: <36e2184f-6527-0aad-b156-b12719cfbd8a@oracle.com> <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> <5909D7B5.2030400@oracle.com> Message-ID: <20170503154048.GA61603@tjfontai-mac> On Wed, May 03, 2017 at 03:14:29PM +0200, Erik Gahlin wrote: > I noticed thatgetNamespacePid throws IOException and InvalidPathException. > Perhaps we want to catch those, so we can default to the original pid if > there is a I/O related problem. That seems reasonable, I'll add that. > > Erik > > Hey, > > > > I?ve attached a version rebased on jdk10, it also (currently) applies cleanly to jdk9. > > > > While there is no supplied test or harness for this patch, how I built and tested is > > available at https://github.com/tjfontaine/jdkbuild (there?s also a preview of my > > follow on patch for pathmap_open as well). > > > > Thanks! > > > > TJ > > > > On 4/28/17, 3:47 PM, "serviceability-dev on behalf of TJ Fontaine" wrote: > > > > I had no doubt we?d end up on the conversation of 10 -> 9 -> 8u, I started with 8u merely because it was representative of today?s customer pain. I?ll be sure to work on retargeting it as well. > > Thanks! > > TJ > > On 4/28/17, 3:42 PM, "David Holmes" wrote: > > Hi TJ, > > Thanks for the patch (I haven't looked at it yet). FYI at the moment, > > unless this is considered a high priority bug for JDK 9 it has to be > > targeted to JDK 10, and then possibly backported to 9 and 8u. > > Cheers, > > David > > On 29/04/2017 8:23 AM, TJ Fontaine wrote: > > > I have attached a patch that allows jcmd to work against a java process running > > > inside a Docker container. Apologies if this is not in the correct format. It was > > > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > > > > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > > > that is inside a container (e.g. Docker). > > > > > > A Linux container often isolates a process in a PID and Mount namespace that is > > > separate from the "root container" (analogous to the hypervisor/dom0 in > > > hardware virtualization environments, or the global zone on Solaris). A target > > > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > > > the attach sequence. > > > > > > When the target JVM is in its own PID namespace the pid of the process is > > > distinct from what the real pid of the process as it relates to the root > > > container. For example, in the root container you can observe a JVM with a pid > > > of 17734, however if that JVM is running inside a Docker container the pid > > > inside its PID namespace is likely 1. So when the target JVM receives the > > > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > > > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > > > discrepancy the target JVM will output to stderr thread status, since > > > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > > > sequence. > > > > > > The solution is to parse /proc/pid/status for the field NSpid (available since > > > Linux 4.1) which contains a list of pids, where the last entry is the "inner > > > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > > > which have a virtualization depth of 1) > > > > > > The rest of the Linux attach sequence assumes a shared mount namespace by > > > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > > > separate namespace because the target JVM is in a mount namepsace (or in a > > > chroot as well) the unix domain socket for attaching won't appear. > > > > > > Instead the attach sequence should resolve file names relative to > > > /proc/17734/root which has a materialized view of the rootfs for the target. > > > > > > Thanks! > > > > > > TJ > > > > > > From tj.fontaine at oracle.com Wed May 3 15:41:26 2017 From: tj.fontaine at oracle.com (TJ Fontaine) Date: Wed, 3 May 2017 08:41:26 -0700 Subject: [PATCH] attach in linux should be relative to /proc/pid/root and namespace aware In-Reply-To: <05b65382-b2e2-bcfa-6b2d-c8755dd530aa@oracle.com> References: <36e2184f-6527-0aad-b156-b12719cfbd8a@oracle.com> <7E81AA87-6B37-4023-8F74-C244CC906801@oracle.com> <550735F3-412A-4F29-999D-DA48A878371F@oracle.com> <05b65382-b2e2-bcfa-6b2d-c8755dd530aa@oracle.com> Message-ID: <20170503154126.GB61603@tjfontai-mac> On Tue, May 02, 2017 at 02:16:09PM +1000, David Holmes wrote: > Hi TJ, > > On 1/05/2017 10:22 AM, TJ Fontaine wrote: > > Hey, > > > > I?ve attached a version rebased on jdk10, it also (currently) applies cleanly to jdk9. > > I have filed: > > https://bugs.openjdk.java.net/browse/JDK-8179498 > > and have hosted your patch at: > > http://cr.openjdk.java.net/~dholmes/8179498/webrev/ > > I find it a little confusing understanding when we need to use pid and when > we need to use ns_pid. IIUC the client VM doing the attach should use the > "global" pid in the /proc path but name the file with the ns_pid as seen by > the target VM - correct? And the query "what is my pid" will actually return > the ns_pid - correct? Your understanding is correct, and the confusion is natural. First because there's not simply not a lot of useful vocabulary for Linux containers (yet), and second its coupled with just how flexible (i.e. complex) the environments can be. I've been calling this the "Container Identity Crisis", there are at least two distinct identities in play. What the client initiating the attach believes the identity/pid of the target process is (as it relates to what it can find in its /proc), and what the target process believes its own identity to be. For development tools to be container aware, the calling process should always address the target process relative to what is reachable through its /proc (e.g. cwd, root), however if you need to interact with the target process you may need to be aware of the inner most namespaced pid -- as is the case with the JVM attach sequence. > > 345 // TODO XXX friggin old kernels may not have NSpid field (i.e. > 3.10) > 346 // fallback to original pid in the event we cannot deduce > > :) Suggest: > > // If no NSpid field found assume no namespace and fallback to original pid ACK > > Overall the approach seems reasonable, but I can't validate the operational > details. Lets see what others have to say. > > > While there is no supplied test or harness for this patch, how I built and tested is > > available at https://github.com/tjfontaine/jdkbuild (there?s also a preview of my > > follow on patch for pathmap_open as well). > > The lack of testability is somewhat of a concern. We can of course test that > it works okay when the namespace is not present, but can't verify it works > as you intend. > > And of course this doesn't help with Docker containers running under earlier > versions of the Linux kernel. Correct, sadly the tools will continue to be broken (for now) if we cannot deduce the namespaced pid of the target process. Let me investigate the test infrastructure more to see what can be done with regards to being able to do more validation. > > Thanks, > David > > > Thanks! > > > > TJ > > > > On 4/28/17, 3:47 PM, "serviceability-dev on behalf of TJ Fontaine" wrote: > > > > I had no doubt we?d end up on the conversation of 10 -> 9 -> 8u, I started with 8u merely because it was representative of today?s customer pain. I?ll be sure to work on retargeting it as well. > > > > Thanks! > > > > TJ > > > > On 4/28/17, 3:42 PM, "David Holmes" wrote: > > > > Hi TJ, > > > > Thanks for the patch (I haven't looked at it yet). FYI at the moment, > > unless this is considered a high priority bug for JDK 9 it has to be > > targeted to JDK 10, and then possibly backported to 9 and 8u. > > > > Cheers, > > David > > > > On 29/04/2017 8:23 AM, TJ Fontaine wrote: > > > I have attached a patch that allows jcmd to work against a java process running > > > inside a Docker container. Apologies if this is not in the correct format. It was > > > built against jdk8u. I couldn?t seem to find an existing JIRA for it. > > > > > > Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM > > > that is inside a container (e.g. Docker). > > > > > > A Linux container often isolates a process in a PID and Mount namespace that is > > > separate from the "root container" (analogous to the hypervisor/dom0 in > > > hardware virtualization environments, or the global zone on Solaris). A target > > > JVM that is isolated in either a PID namespace, or a Mount namespace will fail > > > the attach sequence. > > > > > > When the target JVM is in its own PID namespace the pid of the process is > > > distinct from what the real pid of the process as it relates to the root > > > container. For example, in the root container you can observe a JVM with a pid > > > of 17734, however if that JVM is running inside a Docker container the pid > > > inside its PID namespace is likely 1. So when the target JVM receives the > > > SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external > > > attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this > > > discrepancy the target JVM will output to stderr thread status, since > > > /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach > > > sequence. > > > > > > The solution is to parse /proc/pid/status for the field NSpid (available since > > > Linux 4.1) which contains a list of pids, where the last entry is the "inner > > > most" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones > > > which have a virtualization depth of 1) > > > > > > The rest of the Linux attach sequence assumes a shared mount namespace by > > > waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a > > > separate namespace because the target JVM is in a mount namepsace (or in a > > > chroot as well) the unix domain socket for attaching won't appear. > > > > > > Instead the attach sequence should resolve file names relative to > > > /proc/17734/root which has a materialized view of the rootfs for the target. > > > > > > Thanks! > > > > > > TJ > > > > > > > > > > > > > From ujwal.vangapally at oracle.com Thu May 4 06:44:04 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Thu, 4 May 2017 12:14:04 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications Message-ID: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> Kindly review the changes made for below bug Problem description and solution are explained in comments section https://bugs.openjdk.java.net/browse/JDK-6515161 diff for*ClientNotifForwarder.java *might be a bit confusing as it shows the method name removeNotificationListener is modified to getListenerIds and new method removeNotificationListener is introduced. Actual change is new method getListenerIds is introduced and it is called in removeNotificationListener method without affecting the functionality of removeNotificationListener. webrev : cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ Thanks, Ujwal, -------------- next part -------------- An HTML attachment was scrubbed... URL: From ujwal.vangapally at oracle.com Thu May 4 06:53:06 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Thu, 4 May 2017 12:23:06 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications Message-ID: Kindly review the changes made for below bug Problem description and solution are explained in comments section https://bugs.openjdk.java.net/browse/JDK-6515161 diff for*ClientNotifForwarder.java *might be a bit confusing as it shows the method name removeNotificationListener is modified to getListenerIds and new method removeNotificationListener is introduced. Actual change is that new method getListenerIds is introduced and and it is called in removeNotificationListener method without affecting the functionality of removeNotificationListener webrev : cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ Thanks, Ujwal, -------------- next part -------------- An HTML attachment was scrubbed... URL: From ujwal.vangapally at oracle.com Thu May 4 06:59:08 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Thu, 4 May 2017 12:29:08 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> Message-ID: <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> corrected webrev link : http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: > > Kindly review the changes made for below bug > > Problem description and solution are explained in comments section > > https://bugs.openjdk.java.net/browse/JDK-6515161 > > diff for*ClientNotifForwarder.java *might be a bit confusing as it > shows the method name > > removeNotificationListener is modified to getListenerIds and new > method removeNotificationListener is introduced. > > Actual change is new method getListenerIds is introduced and it is > called in removeNotificationListener method without > > affecting the functionality of removeNotificationListener. > > webrev : cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ > > Thanks, > > Ujwal, > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbin.ehn at oracle.com Thu May 4 09:13:05 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 4 May 2017 11:13:05 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> Hi, To me the compiler changes looks what is expected. It would be good if someone from compiler could take a look at that. Added compiler to mail thread. Also adding Serguei, It would be good with his view also. My initial take on it, read through most of the code and took it for a ride. ############################## - Regarding the compiler changes: I think we need the 'TLAB end' trickery (mentioned by Tony P) instead of a separate check for sampling in fast path for the final version. ############################## - This patch I had to apply to get it compile on JDK 10: diff -r ac3ded340b35 src/share/vm/gc/shared/collectedHeap.inline.hpp --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr 28 14:31:38 2017 +0200 +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May 04 10:22:56 2017 +0200 @@ -87,3 +87,3 @@ // support for object alloc event (no-op most of the time) - if (klass() != NULL && klass()->name() != NULL) { + if (klass != NULL && klass->name() != NULL) { Thread *base_thread = Thread::current(); diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 14:31:38 2017 +0200 +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:22:56 2017 +0200 @@ -316,3 +316,3 @@ JavaThread *thread = reinterpret_cast(Thread::current()); - assert(o->size() << LogHeapWordSize == byte_size, + assert(o->size() << LogHeapWordSize == (long)byte_size, "Object size is incorrect."); ############################## - This patch I had to apply to get it not asserting during slowdebug: --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 15:15:16 2017 +0200 +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:24:25 2017 +0200 @@ -32,3 +32,3 @@ // TODO(jcbeyler): should we make this into a JVMTI structure? -struct StackTraceData { +struct StackTraceData : CHeapObj { ASGCT_CallTrace *trace; @@ -143,3 +143,2 @@ StackTraceStorage::StackTraceStorage() : - _allocated_traces(new StackTraceData*[MaxHeapTraces]), _allocated_traces_size(MaxHeapTraces), @@ -147,2 +146,3 @@ _allocated_count(0) { + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, MaxHeapTraces, mtInternal); memset(_allocated_traces, 0, sizeof(*_allocated_traces) * MaxHeapTraces); @@ -152,3 +152,3 @@ StackTraceStorage::~StackTraceStorage() { - delete[] _allocated_traces; + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); } - Classes should extend correct base class for which type of memory is used for it e.g.: CHeapObj or StackObj or AllStatic - The style in heapMonitoring.cpp is a bit different from normal vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, os::malloc and new. - In jvmtiHeapTransition.hpp you use C cast instead. ############################## - This patch I had apply to get traces without setting an ?unrelated? capability - Should this not be a new capability? diff -r c02a5d8785bf src/share/vm/prims/forte.cpp --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 @@ -530,6 +530,6 @@ - if (!JvmtiExport::should_post_class_load()) { +/* if (!JvmtiExport::should_post_class_load()) { trace->num_frames = ticks_no_class_load; // -1 return; - } + }*/ ############################## - forte.cpp: (I know this is not part of your changes but) find_jmethod_id_or_null give me NULL for my test. It looks like we actually want the regular jmethod_id() ? Since we are the thread we are talking about (and in same ucontext) and thread is in vm and have a last java frame, I think most of the checks done in AsyncGetCallTrace is irrelevant, so you should be-able to call forte_fill_call_trace_given_top directly. But since we might need jmethod_id() if possible to avoid getting method id NULL, we need some fixes in forte code, or just do the vframStream loop inside heapMonitoring.cpp and not use forte.cpp. Something like: if (jthread->has_last_Java_frame()) { // just to be safe vframeStream vfst(jthread); while (!vfst.at_end()) { Method* m = vfst.method(); m->jmethod_id(); m->line_number_from_bci(vfst.bci()); vfst.next(); } - This is a bit confusing in forte.cpp, trace->frames[count].lineno = bci. Line number should be m->line_number_from_bci(bci); Do the heapMonitoring suppose to trace with bci or line number? I would say bci, meaning we should either rename ASGCT_CallFrame?lineno or use another data structure which says bci. ############################## - // TODO(jcbeyler): remove this extra code handling the extra trace for Please fix all these TODO's :) ############################## - heapMonitoring.hpp: // TODO(jcbeyler): is this algorithm acceptable in open source? Why is this comment here? What is the implication? Have you tested any simpler algorithm? ############################## - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for building the agent) ############################## - monitoring_period vs HeapMonitorRate, pick rate or period. ############################## - globals.hpp Why is MaxHeapTraces not settable/overridable from jvmti interface? That would be handy. ############################## - jvmtiStackTraceData + ASGCT_CallFrame memory Are the agent suppose to loop through and free all ASGCT_CallFrame? Wouldn't it be better with some kinda protocol, like: (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); Also using another data structure that have num_traces inside it simplifies things. So I'm not convinced using the async structure is the best way forward. I have more questions, but I think it's better if you respond and update the code first. Thanks! /Robbin On 04/21/2017 11:34 PM, JC Beyler wrote: > Hi all, > > I've added size information to the allocation sampling system. This allows the callback to remember the size of each sampled allocation. > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ > > The new webrev.01 also adds the actual heap monitoring sampling system in files: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.cpp.patch > and > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.hpp.patch > > My next step is to add the GC part to the webrev, which will allow users to determine what objects are live and what are garbage. > > Thanks for your attention and let me know if there are any questions! > > Have a wonderful Friday! > Jc > > On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler > wrote: > > Hi all, > > I worked on getting a few numbers for overhead and accuracy for my feature. I'm unsure if here is the right place to provide the full data, so I am just summarizing > here for now. > > - Overhead of the feature > > Using the Dacapo benchmark (http://dacapobench.org/). My initial results are that sampling provides 2.4% with a 512k sampling, 512k being our default setting. > > - Note: this was without the tradesoap, tradebeans and tomcat benchmarks since they did not work with my JDK9 (issue between Dacapo and JDK9 it seems) > - I want to rerun next week to ensure number stability > > - Accuracy of the feature > > I wrote a small microbenchmark that allocates from two different stacktraces at a given ratio. For example, 10% of stacktrace S1 and 90% from stacktrace S2. The > microbenchmark was run 20 times, I averaged the results and looked for accuracy. It seems that statistically it is sound since if I allocated10% S1 and 90% S2, with a > sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. > > Let me know if there are any questions on the numbers and if you'd like to see some more data. > > Note: this was done using our internal JDK8 implementation since the webrev provided by http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > does not yet contain the whole implementation and therefore would have been misleading. > > Thanks, > Jc > > > On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler > wrote: > > Hi all, > > To move the discussion forward, with Chuck Rasbold's help to make a webrev, we pushed this: > http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg > > This is not a final change that does the whole proposition from the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 > ; what it does show is parts of the implementation that is proposed and hopefully can start the conversation going > as I work through the details. > > For example, the changes to C2 are done here for the allocations: http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/share/vm/opto/macro.cpp.patch > > > Hopefully this all makes sense and thank you for all your future comments! > Jc > > > On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler > wrote: > > Hello all, > > This is a follow-up from Jeremy's initial email from last year: > http://mail.openjdk.java.net/pipermail/serviceability-dev/2015-June/017543.html > > I've gone ahead and started working on preparing this and Jeremy and I went down the route of actually writing it up in JEP form: > https://bugs.openjdk.java.net/browse/JDK-8171119 > > I think original conversation that happened last year in that thread still holds true: > > - We have a patch at Google that we think others might be interested in > - It provides a means to understand where the allocation hotspots are at a very low overhead > - Since it is at a low overhead, we can leave it on by default > > So I come to the mailing list with Jeremy's initial question: > "I thought I would ask if there is any interest / if I should write a JEP / if I should just forget it." > > A year ago, it seemed some thought it was a good idea, is this still true? > > Thanks, > Jc > > > > > From Alan.Bateman at oracle.com Thu May 4 14:11:23 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 4 May 2017 15:11:23 +0100 Subject: RFR 8179457: Remove demo/jvmti tests In-Reply-To: <22DDB73C-22AB-42F2-B399-5C8CA9D9140B@oracle.com> References: <22DDB73C-22AB-42F2-B399-5C8CA9D9140B@oracle.com> Message-ID: <7b623b7c-f75b-5b0e-25cc-76e91fc0d470@oracle.com> On 28/04/2017 22:41, Alexandre (Shura) Iline wrote: > Hi, > > Could you please take a quick look on a source code change to remove demo/jvmti tests: > http://cr.openjdk.java.net/~shurailine/8179457/webrev.00/ > > The tests seem to only be testing the jvmti demos and not providing any useful test coverage beyond that. The demos are now removed as part of JEP 298: Remove Demos and Samples. > One other thing to change is the `needs_jdk` group in TEST.group as it includes the demo tests. -Alan From chris.plummer at oracle.com Thu May 4 22:38:45 2017 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 4 May 2017 15:38:45 -0700 Subject: RFR(10)(M): JDK-8164563: Test nsk/jvmti/CompiledMethodUnload/compmethunload001 keeps reporting: PRODUCT BUG: class was not unloaded in 5 Message-ID: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> Hello, Please review the following changes: http://cr.openjdk.java.net/~cjplummer/8164563/webrev.00/webrev.hotspot/ https://bugs.openjdk.java.net/browse/JDK-8164563 There was an issue with CompileMethodUnload events not getting delivered. Short story is the root cause was the JvmtiDeferredEventQueue::_pending_list, which upon further review was deemed unnecessary, so all code related to it has been pulled. The _pending_list is a temporary list for CompileMethodUnload events that occur while at a safepoint. Putting them directly on the JvmtiDeferredEventQueue was thought not to be allowed at a safepoint, because doing so required acquiring the Service_lock, and it was thought that you can no do that while at a safepoint. The issue with the _pending_list is that it only gets processed if there is a subsequent JvmtiDeferredEventQueue::enqueue() call. For the test in question, this was not always happening. The test sits in a loop waiting for the unload events, but unless it triggers some compilation during this time, which results in enqueue() being called for the CompileMethodLoad event, it will never see the CompileMethodUnload events. It eventually gives up and fails. Most times however there is a compilation triggered while in the loop so the test passes. The first attempted solution was to use a VM op that triggered processing of the _pending_list. However, this also ran up against the issue of not being able to grab the Service_lock while at a safepoint because even _concurrent VM ops can end up being run synchronously and at a safepoint if you are already in a VMThread. After further review of the safepoint concern with the Service_lock, it was determined that it should be ok to grab it while at a safepoint, thus removing the need for the _pending_list. So basically the fix is to remove all _pending_list code, and have nmethod::post_compiled_method_unload() always directly call JvmtiDeferredEventQueue::enqueue(). I tested by running the failing test at least 100 times on all supported platforms (it used to fail with a fairly high frequency). I also ran our other CompileMethodUnload and CompileMethodLoad tests about 100 times, and ran our full jvmti test suite a few times on each platform, along with the jck/vm/jvmti tests. thanks, Chris From david.holmes at oracle.com Thu May 4 23:33:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 5 May 2017 09:33:27 +1000 Subject: RFR(10)(M): JDK-8164563: Test nsk/jvmti/CompiledMethodUnload/compmethunload001 keeps reporting: PRODUCT BUG: class was not unloaded in 5 In-Reply-To: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> References: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> Message-ID: Looks good Chris. I think we finally ended up with the cleanest and simplest solution. Thanks, David On 5/05/2017 8:38 AM, Chris Plummer wrote: > Hello, > > Please review the following changes: > > http://cr.openjdk.java.net/~cjplummer/8164563/webrev.00/webrev.hotspot/ > https://bugs.openjdk.java.net/browse/JDK-8164563 > > There was an issue with CompileMethodUnload events not getting > delivered. Short story is the root cause was the > JvmtiDeferredEventQueue::_pending_list, which upon further review was > deemed unnecessary, so all code related to it has been pulled. > > The _pending_list is a temporary list for CompileMethodUnload events > that occur while at a safepoint. Putting them directly on the > JvmtiDeferredEventQueue was thought not to be allowed at a safepoint, > because doing so required acquiring the Service_lock, and it was thought > that you can no do that while at a safepoint. > > The issue with the _pending_list is that it only gets processed if there > is a subsequent JvmtiDeferredEventQueue::enqueue() call. For the test in > question, this was not always happening. The test sits in a loop waiting > for the unload events, but unless it triggers some compilation during > this time, which results in enqueue() being called for the > CompileMethodLoad event, it will never see the CompileMethodUnload > events. It eventually gives up and fails. Most times however there is a > compilation triggered while in the loop so the test passes. > > The first attempted solution was to use a VM op that triggered > processing of the _pending_list. However, this also ran up against the > issue of not being able to grab the Service_lock while at a safepoint > because even _concurrent VM ops can end up being run synchronously and > at a safepoint if you are already in a VMThread. > > After further review of the safepoint concern with the Service_lock, it > was determined that it should be ok to grab it while at a safepoint, > thus removing the need for the _pending_list. So basically the fix is to > remove all _pending_list code, and have > nmethod::post_compiled_method_unload() always directly call > JvmtiDeferredEventQueue::enqueue(). > > I tested by running the failing test at least 100 times on all supported > platforms (it used to fail with a fairly high frequency). I also ran our > other CompileMethodUnload and CompileMethodLoad tests about 100 times, > and ran our full jvmti test suite a few times on each platform, along > with the jck/vm/jvmti tests. > > thanks, > > Chris From jcbeyler at google.com Fri May 5 06:02:47 2017 From: jcbeyler at google.com (JC Beyler) Date: Thu, 4 May 2017 23:02:47 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> Message-ID: So first off: thank you so much for your diligent look at the patch. A lot of the comments are due to me either using the wrong baseline (I was using JDK9) it seems or either that I first worked on keeping things simple, so maybe there are things that are still in flight and why things might seem unclear. I will try to rectify most of those in the next webrev. I will say again that I really do appreciate the time you took to give all those comments. I was expecting a high level review at first and not an in-depth one :) I'll work on the assumption that my code base is wrong and figure out how to get the right one tomorrow. I've inlined my comments which mostly are questions to help me fix the code. On Thu, May 4, 2017 at 2:13 AM, Robbin Ehn wrote: > Hi, > > To me the compiler changes looks what is expected. > It would be good if someone from compiler could take a look at that. > Added compiler to mail thread. > > Also adding Serguei, It would be good with his view also. > > My initial take on it, read through most of the code and took it for a > ride. > > ############################## > - Regarding the compiler changes: I think we need the 'TLAB end' trickery > (mentioned by Tony P) > instead of a separate check for sampling in fast path for the final > version. > > Agreed, I am still working on assessing it and the final version most likely will be TLAB-based if I can prove to myself and to you the overhead is acceptable. My initial data and the patch is simple enough that it seems like it should work. I might send out a webrev with it just to show it and people can play/comment on it. - For simplicity, I've removed your actual patches but most are either because I believe my JDK that I was basing off of was not the right one compared to the one you are using. If you have a second and can provide me a link or process on how you get the right mercurial repositories set up, I'll set myself the same way you are doing it to align myself. - I'll also double check slowdebug to ensure no problems. - Classes should extend correct base class for which type of memory is used > for it e.g.: CHeapObj or StackObj or AllStatic > - The style in heapMonitoring.cpp is a bit different from normal vm-style, > e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, os::malloc and > new. > - In jvmtiHeapTransition.hpp you use C cast instead. > > Noted, I thought I had cleaned up most of them but seemingly not. I'll go and clean it up and align myself with what I see the surrounding code is doing between code from runtime and code for the jvmti. > ############################## > - This patch I had apply to get traces without setting an ?unrelated? > capability > - Should this not be a new capability? > > diff -r c02a5d8785bf src/share/vm/prims/forte.cpp > --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 > @@ -530,6 +530,6 @@ > > - if (!JvmtiExport::should_post_class_load()) { > +/* if (!JvmtiExport::should_post_class_load()) { > trace->num_frames = ticks_no_class_load; // -1 > return; > - } > + }*/ > > So the way I've been testing all of this for now was by creating a Java agent that sets everything up. I was waiting to have this conversation for when we were a bit more into the conversations. Since you have brought it up: caps.can_get_line_numbers = 1; caps.can_get_source_file_name = 1; And I've added some callbacks too to help set things up: ernum = global_jvmti->SetEventCallbacks(&callbacks, sizeof(jvmtiEventCallbacks)); ernum = global_jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_VM_INIT, NULL); ernum = global_jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_CLASS_LOAD, NULL); The class loading callback me to force the creation of the jmethod ids for the class, which most likely solves your problem here: Quick question: Does it make sense to put my agent code somewhere in the webrev for all to see how I've been calling into this? I was waiting a bit for things to settle a bit first but I could do that easily. Is there a spot where that would make sense? ############################## > - forte.cpp: (I know this is not part of your changes but) > find_jmethod_id_or_null give me NULL for my test. > It looks like we actually want the regular jmethod_id() ? > > Since we are the thread we are talking about (and in same ucontext) and > thread is in vm and have a last java frame, > I think most of the checks done in AsyncGetCallTrace is irrelevant, so you > should be-able to call forte_fill_call_trace_given_top directly. > But since we might need jmethod_id() if possible to avoid getting method > id NULL, > we need some fixes in forte code, or just do the vframStream loop inside > heapMonitoring.cpp and not use forte.cpp. > > Something like: > > if (jthread->has_last_Java_frame()) { // just to be safe > vframeStream vfst(jthread); > while (!vfst.at_end()) { > Method* m = vfst.method(); > m->jmethod_id(); > m->line_number_from_bci(vfst.bci()); > vfst.next(); > } > I have no idea about this and will have to try :). I am open to any solution that will work and this seems to go around forte.cpp. We have a few changes in forte.cpp that I have not put here that help get more frames than the current implementation allows. Let me test this and see if it works :) > > - This is a bit confusing in forte.cpp, trace->frames[count].lineno = bci. > Line number should be m->line_number_from_bci(bci); > Do the heapMonitoring suppose to trace with bci or line number? > I would say bci, meaning we should either rename ASGCT_CallFrame?lineno or > use another data structure which says bci. > Agreed, I just did not want to create another structure straight away because I thought it was going to be confusing. + it seems that it is not necessary automatically. For the bci instead of line number, it is possible that, because this is code that has been ported over and over, we never cleaned it up or knew that a method was now available. Let me test it out and see how it works. > > ############################## > - // TODO(jcbeyler): remove this extra code handling the extra trace for > Please fix all these TODO's :) > > Will do, they were my trackers of TODOs, I prefetched a bit when I sent out the webrev. I will try to clean out most of them for the next webrev :) > ############################## > - heapMonitoring.hpp: > // TODO(jcbeyler): is this algorithm acceptable in open source? > > Why is this comment here? What is the implication? > Have you tested any simpler algorithm? > Yes sorry, this is me being paranoid and trying to figure out if the licenses are compatible. This implementation exists in the open source world but I wanted to double check via this TODO that the original code and the JDK are compatible in licenses before bringing this in. >From what I've seen here, this is a known fast log implementation/algorithm and the license seems fine so I believe it is fine but still need to do the due diligence. I will double check and ensure it is fine :) > > ############################## > - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for > building the agent) > Sounds good, I will look at how to do that :) > > ############################## > - monitoring_period vs HeapMonitorRate, pick rate or period. > Agreed. > > ############################## > - globals.hpp > Why is MaxHeapTraces not settable/overridable from jvmti interface? That > would be handy. > Just to be clear and complete: MaxHeapTraces will actually disappear in the future webrev: 1) This was here only to show how the whole system will look with especially the general jvmti interface, compiler changes, and hooks 2) In practice, we keep all live sampled objects and we free up the list via the GC events of sampled objects 3) In our implementation and the final one that would be shown here: - MaxHeapTraces disappears but we would have the analoguous MaxGarbageHeapTraces - That would could use a flag to be settable/overridable > > ############################## > - jvmtiStackTraceData + ASGCT_CallFrame memory > Are the agent suppose to loop through and free all ASGCT_CallFrame? > Wouldn't it be better with some kinda protocol, like: > (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); > (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); > I will change it like you suggest. Our implementation did do the free in the agent but in the general case, your solution is more logical. > > Also using another data structure that have num_traces inside it > simplifies things. > So I'm not convinced using the async structure is the best way forward. > I'm not yet sure about leaving the async entirely, I will double check on that before commiting to it. I will let you know. > > > I have more questions, but I think it's better if you respond and update > the code first. > I will work on the code changes and adapt to all your suggestions here. Hopefully, I've helped answer partially most of them and if you have more, please feel free to ask them now :) Or, as you said, wait until I send my next version out if you think that we should first iterate on this before opening other questions and conversations! Thank you again for taking the time to do such a thorough review and I hope I have answered the initial questions well! Jc > > Thanks! > > /Robbin > > > On 04/21/2017 11:34 PM, JC Beyler wrote: > >> Hi all, >> >> I've added size information to the allocation sampling system. This >> allows the callback to remember the size of each sampled allocation. >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ >> >> The new webrev.01 also adds the actual heap monitoring sampling system in >> files: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.cpp.patch >> and >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.hpp.patch >> >> My next step is to add the GC part to the webrev, which will allow users >> to determine what objects are live and what are garbage. >> >> Thanks for your attention and let me know if there are any questions! >> >> Have a wonderful Friday! >> Jc >> >> On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler > jcbeyler at google.com>> wrote: >> >> Hi all, >> >> I worked on getting a few numbers for overhead and accuracy for my >> feature. I'm unsure if here is the right place to provide the full data, so >> I am just summarizing >> here for now. >> >> - Overhead of the feature >> >> Using the Dacapo benchmark (http://dacapobench.org/). My initial >> results are that sampling provides 2.4% with a 512k sampling, 512k being >> our default setting. >> >> - Note: this was without the tradesoap, tradebeans and tomcat >> benchmarks since they did not work with my JDK9 (issue between Dacapo and >> JDK9 it seems) >> - I want to rerun next week to ensure number stability >> >> - Accuracy of the feature >> >> I wrote a small microbenchmark that allocates from two different >> stacktraces at a given ratio. For example, 10% of stacktrace S1 and 90% >> from stacktrace S2. The >> microbenchmark was run 20 times, I averaged the results and looked >> for accuracy. It seems that statistically it is sound since if I >> allocated10% S1 and 90% S2, with a >> sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. >> >> Let me know if there are any questions on the numbers and if you'd >> like to see some more data. >> >> Note: this was done using our internal JDK8 implementation since the >> webrev provided by http://cr.openjdk.java.net/~ra >> sbold/heapz/webrev.00/index.html >> >> does not yet contain the whole implementation and therefore would have been >> misleading. >> >> Thanks, >> Jc >> >> >> On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler > > wrote: >> >> Hi all, >> >> To move the discussion forward, with Chuck Rasbold's help to make >> a webrev, we pushed this: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html < >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html> >> 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg >> >> This is not a final change that does the whole proposition from >> the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 >> ; what it does >> show is parts of the implementation that is proposed and hopefully can >> start the conversation going >> as I work through the details. >> >> For example, the changes to C2 are done here for the allocations: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/shar >> e/vm/opto/macro.cpp.patch >> > re/vm/opto/macro.cpp.patch> >> >> Hopefully this all makes sense and thank you for all your future >> comments! >> Jc >> >> >> On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler > > wrote: >> >> Hello all, >> >> This is a follow-up from Jeremy's initial email from last >> year: >> http://mail.openjdk.java.net/pipermail/serviceability-dev/20 >> 15-June/017543.html > pipermail/serviceability-dev/2015-June/017543.html> >> >> I've gone ahead and started working on preparing this and >> Jeremy and I went down the route of actually writing it up in JEP form: >> https://bugs.openjdk.java.net/browse/JDK-8171119 >> >> I think original conversation that happened last year in that >> thread still holds true: >> >> - We have a patch at Google that we think others might be >> interested in >> - It provides a means to understand where the allocation >> hotspots are at a very low overhead >> - Since it is at a low overhead, we can leave it on by >> default >> >> So I come to the mailing list with Jeremy's initial question: >> "I thought I would ask if there is any interest / if I should >> write a JEP / if I should just forget it." >> >> A year ago, it seemed some thought it was a good idea, is >> this still true? >> >> Thanks, >> Jc >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri May 5 06:21:55 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 4 May 2017 23:21:55 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> Message-ID: <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Robbin, Thank you for forwarding! I will review it. Thanks, Serguei On 5/4/17 02:13, Robbin Ehn wrote: > Hi, > > To me the compiler changes looks what is expected. > It would be good if someone from compiler could take a look at that. > Added compiler to mail thread. > > Also adding Serguei, It would be good with his view also. > > My initial take on it, read through most of the code and took it for a > ride. > > ############################## > - Regarding the compiler changes: I think we need the 'TLAB end' > trickery (mentioned by Tony P) > instead of a separate check for sampling in fast path for the final > version. > > ############################## > - This patch I had to apply to get it compile on JDK 10: > > diff -r ac3ded340b35 src/share/vm/gc/shared/collectedHeap.inline.hpp > --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr 28 > 14:31:38 2017 +0200 > +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May 04 > 10:22:56 2017 +0200 > @@ -87,3 +87,3 @@ > // support for object alloc event (no-op most of the time) > - if (klass() != NULL && klass()->name() != NULL) { > + if (klass != NULL && klass->name() != NULL) { > Thread *base_thread = Thread::current(); > diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 14:31:38 > 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:22:56 > 2017 +0200 > @@ -316,3 +316,3 @@ > JavaThread *thread = reinterpret_cast *>(Thread::current()); > - assert(o->size() << LogHeapWordSize == byte_size, > + assert(o->size() << LogHeapWordSize == (long)byte_size, > "Object size is incorrect."); > > ############################## > - This patch I had to apply to get it not asserting during slowdebug: > > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 15:15:16 > 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:24:25 > 2017 +0200 > @@ -32,3 +32,3 @@ > // TODO(jcbeyler): should we make this into a JVMTI structure? > -struct StackTraceData { > +struct StackTraceData : CHeapObj { > ASGCT_CallTrace *trace; > @@ -143,3 +143,2 @@ > StackTraceStorage::StackTraceStorage() : > - _allocated_traces(new StackTraceData*[MaxHeapTraces]), > _allocated_traces_size(MaxHeapTraces), > @@ -147,2 +146,3 @@ > _allocated_count(0) { > + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, > MaxHeapTraces, mtInternal); > memset(_allocated_traces, 0, sizeof(*_allocated_traces) * > MaxHeapTraces); > @@ -152,3 +152,3 @@ > StackTraceStorage::~StackTraceStorage() { > - delete[] _allocated_traces; > + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); > } > > - Classes should extend correct base class for which type of memory is > used for it e.g.: CHeapObj or StackObj or AllStatic > - The style in heapMonitoring.cpp is a bit different from normal > vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, > os::malloc and new. > - In jvmtiHeapTransition.hpp you use C cast instead. > > ############################## > - This patch I had apply to get traces without setting an ?unrelated? > capability > - Should this not be a new capability? > > diff -r c02a5d8785bf src/share/vm/prims/forte.cpp > --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 > @@ -530,6 +530,6 @@ > > - if (!JvmtiExport::should_post_class_load()) { > +/* if (!JvmtiExport::should_post_class_load()) { > trace->num_frames = ticks_no_class_load; // -1 > return; > - } > + }*/ > > ############################## > - forte.cpp: (I know this is not part of your changes but) > find_jmethod_id_or_null give me NULL for my test. > It looks like we actually want the regular jmethod_id() ? > > Since we are the thread we are talking about (and in same ucontext) > and thread is in vm and have a last java frame, > I think most of the checks done in AsyncGetCallTrace is irrelevant, so > you should be-able to call forte_fill_call_trace_given_top directly. > But since we might need jmethod_id() if possible to avoid getting > method id NULL, > we need some fixes in forte code, or just do the vframStream loop > inside heapMonitoring.cpp and not use forte.cpp. > > Something like: > > if (jthread->has_last_Java_frame()) { // just to be safe > vframeStream vfst(jthread); > while (!vfst.at_end()) { > Method* m = vfst.method(); > m->jmethod_id(); > m->line_number_from_bci(vfst.bci()); > vfst.next(); > } > > - This is a bit confusing in forte.cpp, trace->frames[count].lineno = > bci. > Line number should be m->line_number_from_bci(bci); > Do the heapMonitoring suppose to trace with bci or line number? > I would say bci, meaning we should either rename > ASGCT_CallFrame?lineno or use another data structure which says bci. > > ############################## > - // TODO(jcbeyler): remove this extra code handling the extra trace for > Please fix all these TODO's :) > > ############################## > - heapMonitoring.hpp: > // TODO(jcbeyler): is this algorithm acceptable in open source? > > Why is this comment here? What is the implication? > Have you tested any simpler algorithm? > > ############################## > - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for > building the agent) > > ############################## > - monitoring_period vs HeapMonitorRate, pick rate or period. > > ############################## > - globals.hpp > Why is MaxHeapTraces not settable/overridable from jvmti interface? > That would be handy. > > ############################## > - jvmtiStackTraceData + ASGCT_CallFrame memory > Are the agent suppose to loop through and free all ASGCT_CallFrame? > Wouldn't it be better with some kinda protocol, like: > (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); > (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); > > Also using another data structure that have num_traces inside it > simplifies things. > So I'm not convinced using the async structure is the best way forward. > > > I have more questions, but I think it's better if you respond and > update the code first. > > Thanks! > > /Robbin > > > On 04/21/2017 11:34 PM, JC Beyler wrote: >> Hi all, >> >> I've added size information to the allocation sampling system. This >> allows the callback to remember the size of each sampled allocation. >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ >> >> The new webrev.01 also adds the actual heap monitoring sampling >> system in files: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.cpp.patch >> >> and >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.hpp.patch >> >> >> My next step is to add the GC part to the webrev, which will allow >> users to determine what objects are live and what are garbage. >> >> Thanks for your attention and let me know if there are any questions! >> >> Have a wonderful Friday! >> Jc >> >> On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler > > wrote: >> >> Hi all, >> >> I worked on getting a few numbers for overhead and accuracy for >> my feature. I'm unsure if here is the right place to provide the full >> data, so I am just summarizing >> here for now. >> >> - Overhead of the feature >> >> Using the Dacapo benchmark (http://dacapobench.org/). My initial >> results are that sampling provides 2.4% with a 512k sampling, 512k >> being our default setting. >> >> - Note: this was without the tradesoap, tradebeans and tomcat >> benchmarks since they did not work with my JDK9 (issue between Dacapo >> and JDK9 it seems) >> - I want to rerun next week to ensure number stability >> >> - Accuracy of the feature >> >> I wrote a small microbenchmark that allocates from two different >> stacktraces at a given ratio. For example, 10% of stacktrace S1 and >> 90% from stacktrace S2. The >> microbenchmark was run 20 times, I averaged the results and >> looked for accuracy. It seems that statistically it is sound since if >> I allocated10% S1 and 90% S2, with a >> sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. >> >> Let me know if there are any questions on the numbers and if >> you'd like to see some more data. >> >> Note: this was done using our internal JDK8 implementation since >> the webrev provided by >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html >> does >> not yet contain the whole implementation and therefore would have >> been misleading. >> >> Thanks, >> Jc >> >> >> On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler > > wrote: >> >> Hi all, >> >> To move the discussion forward, with Chuck Rasbold's help to >> make a webrev, we pushed this: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html >> >> 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg >> >> This is not a final change that does the whole proposition >> from the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 >> ; what it does show >> is parts of the implementation that is proposed and hopefully can >> start the conversation going >> as I work through the details. >> >> For example, the changes to C2 are done here for the >> allocations: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/share/vm/opto/macro.cpp.patch >> >> >> Hopefully this all makes sense and thank you for all your >> future comments! >> Jc >> >> >> On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler >> > wrote: >> >> Hello all, >> >> This is a follow-up from Jeremy's initial email from last >> year: >> http://mail.openjdk.java.net/pipermail/serviceability-dev/2015-June/017543.html >> >> >> >> I've gone ahead and started working on preparing this and >> Jeremy and I went down the route of actually writing it up in JEP form: >> https://bugs.openjdk.java.net/browse/JDK-8171119 >> >> I think original conversation that happened last year in >> that thread still holds true: >> >> - We have a patch at Google that we think others might >> be interested in >> - It provides a means to understand where the >> allocation hotspots are at a very low overhead >> - Since it is at a low overhead, we can leave it on >> by default >> >> So I come to the mailing list with Jeremy's initial >> question: >> "I thought I would ask if there is any interest / if I >> should write a JEP / if I should just forget it." >> >> A year ago, it seemed some thought it was a good idea, is >> this still true? >> >> Thanks, >> Jc >> >> >> >> >> From serguei.spitsyn at oracle.com Fri May 5 06:29:37 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 4 May 2017 23:29:37 -0700 Subject: RFR(10)(M): JDK-8164563: Test nsk/jvmti/CompiledMethodUnload/compmethunload001 keeps reporting: PRODUCT BUG: class was not unloaded in 5 In-Reply-To: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> References: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> Message-ID: <3604d39f-d18c-3800-a93d-a10629059f0a@oracle.com> Hi Chris, The fix looks good. I like the simplification. Great bug hunt, Chris! Thanks, Serguei On 5/4/17 15:38, Chris Plummer wrote: > Hello, > > Please review the following changes: > > http://cr.openjdk.java.net/~cjplummer/8164563/webrev.00/webrev.hotspot/ > https://bugs.openjdk.java.net/browse/JDK-8164563 > > There was an issue with CompileMethodUnload events not getting > delivered. Short story is the root cause was the > JvmtiDeferredEventQueue::_pending_list, which upon further review was > deemed unnecessary, so all code related to it has been pulled. > > The _pending_list is a temporary list for CompileMethodUnload events > that occur while at a safepoint. Putting them directly on the > JvmtiDeferredEventQueue was thought not to be allowed at a safepoint, > because doing so required acquiring the Service_lock, and it was > thought that you can no do that while at a safepoint. > > The issue with the _pending_list is that it only gets processed if > there is a subsequent JvmtiDeferredEventQueue::enqueue() call. For the > test in question, this was not always happening. The test sits in a > loop waiting for the unload events, but unless it triggers some > compilation during this time, which results in enqueue() being called > for the CompileMethodLoad event, it will never see the > CompileMethodUnload events. It eventually gives up and fails. Most > times however there is a compilation triggered while in the loop so > the test passes. > > The first attempted solution was to use a VM op that triggered > processing of the _pending_list. However, this also ran up against the > issue of not being able to grab the Service_lock while at a safepoint > because even _concurrent VM ops can end up being run synchronously and > at a safepoint if you are already in a VMThread. > > After further review of the safepoint concern with the Service_lock, > it was determined that it should be ok to grab it while at a > safepoint, thus removing the need for the _pending_list. So basically > the fix is to remove all _pending_list code, and have > nmethod::post_compiled_method_unload() always directly call > JvmtiDeferredEventQueue::enqueue(). > > I tested by running the failing test at least 100 times on all > supported platforms (it used to fail with a fairly high frequency). I > also ran our other CompileMethodUnload and CompileMethodLoad tests > about 100 times, and ran our full jvmti test suite a few times on each > platform, along with the jck/vm/jvmti tests. > > thanks, > > Chris From daniel.daugherty at oracle.com Fri May 5 15:41:52 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 5 May 2017 09:41:52 -0600 Subject: RFR(10)(M): JDK-8164563: Test nsk/jvmti/CompiledMethodUnload/compmethunload001 keeps reporting: PRODUCT BUG: class was not unloaded in 5 In-Reply-To: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> References: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> Message-ID: On 5/4/17 4:38 PM, Chris Plummer wrote: > Hello, > > Please review the following changes: > > http://cr.openjdk.java.net/~cjplummer/8164563/webrev.00/webrev.hotspot/ > https://bugs.openjdk.java.net/browse/JDK-8164563 src/share/vm/prims/jvmtiImpl.hpp No comments. src/share/vm/prims/jvmtiImpl.cpp No comments. src/share/vm/code/nmethod.cpp No comments. Removal of pending_list support looks clean. Thumbs up! Dan > > There was an issue with CompileMethodUnload events not getting > delivered. Short story is the root cause was the > JvmtiDeferredEventQueue::_pending_list, which upon further review was > deemed unnecessary, so all code related to it has been pulled. > > The _pending_list is a temporary list for CompileMethodUnload events > that occur while at a safepoint. Putting them directly on the > JvmtiDeferredEventQueue was thought not to be allowed at a safepoint, > because doing so required acquiring the Service_lock, and it was > thought that you can no do that while at a safepoint. > > The issue with the _pending_list is that it only gets processed if > there is a subsequent JvmtiDeferredEventQueue::enqueue() call. For the > test in question, this was not always happening. The test sits in a > loop waiting for the unload events, but unless it triggers some > compilation during this time, which results in enqueue() being called > for the CompileMethodLoad event, it will never see the > CompileMethodUnload events. It eventually gives up and fails. Most > times however there is a compilation triggered while in the loop so > the test passes. > > The first attempted solution was to use a VM op that triggered > processing of the _pending_list. However, this also ran up against the > issue of not being able to grab the Service_lock while at a safepoint > because even _concurrent VM ops can end up being run synchronously and > at a safepoint if you are already in a VMThread. > > After further review of the safepoint concern with the Service_lock, > it was determined that it should be ok to grab it while at a > safepoint, thus removing the need for the _pending_list. So basically > the fix is to remove all _pending_list code, and have > nmethod::post_compiled_method_unload() always directly call > JvmtiDeferredEventQueue::enqueue(). > > I tested by running the failing test at least 100 times on all > supported platforms (it used to fail with a fairly high frequency). I > also ran our other CompileMethodUnload and CompileMethodLoad tests > about 100 times, and ran our full jvmti test suite a few times on each > platform, along with the jck/vm/jvmti tests. > > thanks, > > Chris From chris.plummer at oracle.com Fri May 5 18:25:16 2017 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 5 May 2017 11:25:16 -0700 Subject: RFR(10)(M): JDK-8164563: Test nsk/jvmti/CompiledMethodUnload/compmethunload001 keeps reporting: PRODUCT BUG: class was not unloaded in 5 In-Reply-To: References: <2ad9e3a1-0b11-5cea-4ba8-e411e3f91a6f@oracle.com> Message-ID: <104698f5-233f-e85c-5b0d-8b627c0f2f54@oracle.com> On 5/5/17 8:41 AM, Daniel D. Daugherty wrote: > On 5/4/17 4:38 PM, Chris Plummer wrote: >> Hello, >> >> Please review the following changes: >> >> http://cr.openjdk.java.net/~cjplummer/8164563/webrev.00/webrev.hotspot/ >> https://bugs.openjdk.java.net/browse/JDK-8164563 > > src/share/vm/prims/jvmtiImpl.hpp > No comments. > > src/share/vm/prims/jvmtiImpl.cpp > No comments. > > src/share/vm/code/nmethod.cpp > No comments. > > Removal of pending_list support looks clean. Thumbs up! Thanks Dan! Chris > > Dan > > > > >> >> There was an issue with CompileMethodUnload events not getting >> delivered. Short story is the root cause was the >> JvmtiDeferredEventQueue::_pending_list, which upon further review was >> deemed unnecessary, so all code related to it has been pulled. >> >> The _pending_list is a temporary list for CompileMethodUnload events >> that occur while at a safepoint. Putting them directly on the >> JvmtiDeferredEventQueue was thought not to be allowed at a safepoint, >> because doing so required acquiring the Service_lock, and it was >> thought that you can no do that while at a safepoint. >> >> The issue with the _pending_list is that it only gets processed if >> there is a subsequent JvmtiDeferredEventQueue::enqueue() call. For >> the test in question, this was not always happening. The test sits in >> a loop waiting for the unload events, but unless it triggers some >> compilation during this time, which results in enqueue() being called >> for the CompileMethodLoad event, it will never see the >> CompileMethodUnload events. It eventually gives up and fails. Most >> times however there is a compilation triggered while in the loop so >> the test passes. >> >> The first attempted solution was to use a VM op that triggered >> processing of the _pending_list. However, this also ran up against >> the issue of not being able to grab the Service_lock while at a >> safepoint because even _concurrent VM ops can end up being run >> synchronously and at a safepoint if you are already in a VMThread. >> >> After further review of the safepoint concern with the Service_lock, >> it was determined that it should be ok to grab it while at a >> safepoint, thus removing the need for the _pending_list. So basically >> the fix is to remove all _pending_list code, and have >> nmethod::post_compiled_method_unload() always directly call >> JvmtiDeferredEventQueue::enqueue(). >> >> I tested by running the failing test at least 100 times on all >> supported platforms (it used to fail with a fairly high frequency). I >> also ran our other CompileMethodUnload and CompileMethodLoad tests >> about 100 times, and ran our full jvmti test suite a few times on >> each platform, along with the jck/vm/jvmti tests. >> >> thanks, >> >> Chris > From harsha.wardhana.b at oracle.com Mon May 8 05:53:36 2017 From: harsha.wardhana.b at oracle.com (Harsha Wardhana B) Date: Mon, 8 May 2017 11:23:36 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> Message-ID: Hi Ujwal, The fix looks fine. Some nits. In ClientNotifForwarder.java, "getListenerIds" could be renamed to getClientListenrIds. -Harsha On Thursday 04 May 2017 12:29 PM, Ujwal Vangapally wrote: > > corrected webrev link : > http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ > > > On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >> >> Kindly review the changes made for below bug >> >> Problem description and solution are explained in comments section >> >> https://bugs.openjdk.java.net/browse/JDK-6515161 >> >> diff for*ClientNotifForwarder.java *might be a bit confusing as it >> shows the method name >> >> removeNotificationListener is modified to getListenerIds and new >> method removeNotificationListener is introduced. >> >> Actual change is new method getListenerIds is introduced and it is >> called in removeNotificationListener method without >> >> affecting the functionality of removeNotificationListener. >> >> webrev : cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >> >> Thanks, >> >> Ujwal, >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From harsha.wardhana.b at oracle.com Mon May 8 05:58:05 2017 From: harsha.wardhana.b at oracle.com (Harsha Wardhana B) Date: Mon, 8 May 2017 11:28:05 +0530 Subject: RFE Review : JDK-5016517 - Replace plaintext passwords by hashed passwords for out-of-the-box JMX Agent In-Reply-To: <5dded394-ea92-687f-cbb0-3347936c1eec@oracle.com> References: <6a193cdf-efea-e4e9-4faf-72d8960cd72b@Oracle.com> <5dded394-ea92-687f-cbb0-3347936c1eec@oracle.com> Message-ID: Hello, Could I get one more reviewer for this enhancement? -Harsha On Thursday 27 April 2017 12:06 PM, Harsha Wardhana B wrote: > Hi Roger, > > Thanks for the detailed review. I will wait for more review comments > before incorporating the ones below. > > -Harsha > > > On Tuesday 25 April 2017 10:56 PM, Roger Riggs wrote: >> Hi Harsha, >> >> Thanks for this important improvement. Comments: >> >> >> * jmxremote.password.template: >> "Passwords will be hashed by server if they are in clear." Perhaps >> should be more explicit: >> >> "The jmxremote.passwords file will be re-written by the server to >> replace all plain text passwords with hashed passwords when the file >> is read by the server." >> >> line 35: "Base64 encoded hash" -> drop the "Base64" in this line >> isn't needed and >> make it seems like it should appear as 1 field instead of 2 or 3. >> >> 37+: The syntax of the file may be clearer if it includes the >> complete syntax in (line 39) not >> just the password/hash fragment. >> >> Line 41: "W = spaces"; above "tabs" are allowed as a delimiter; it >> would be good to be consistent >> and include the usualy white-space characters in the set, be as >> specific as possible. >> Is this the same set of whitespace used by Regex '\\s'. >> >> 45: "java platform"" -> "MD5, SDA-1, SHA-256 are supported >> algorithms." >> >> 49: be more specific about 'hashing is requested' how? Refer to the >> management.properties >> com.sun.management.jmxremote.password.hash value. >> >> >> >> 51: "replace hashed" -> "replace *the *hashed" >> 52: "with clear text or new" -> "with the clear text or the new" >> 52: "If new password" -> "If the new password" >> 53: "when new login" -> "when a new login" >> >> 60: "User generated" -> "A User generated" >> >> 67: Will the file be ignored if it has the wrong permissions. (With a >> logged message) >> >> * management.properties >> >> 306: "(Case for true/false ignored)" - what does this mean; I think >> it can be removed. >> >> 307: missing period at the end of the sentence. >> 309: "in password file" -> "in the password file" >> >> >> * FileLoginModule.java >> >> 102: can this match better the similar name in the >> management.properties if it has the same function: >> com.sun.management.jmxremote.password.hash >> 103: "replaces clear text passwords" -> "replaces each clear text >> password" >> 104: indent to match previous
enteries. >> >> * JMXPluggableAuthenticator.java >> >> 119: There is no need to copy the password to a new local >> >> 128: add a space after "," >> >> 256 private static final String HASH_PASSWORDS = >> 257 "jmx.remote.x.password.file.hash"; >> >> The name ".hash" part does not clearly communicate that passwords are >> to be hashed. >> "hashPasswords" might be more self explanatory. >> Also, can this be NOT duplicated here and in ConnectorBootStrap.java? >> >> >> * ConnectorBootStrap.java: >> 482: Add space after ","s; no spaces before. >> >> 770: use the same name for the option/property if possible to avoid >> confusion. >> >> 770: if the HASH_PASSWORDS static is appropriate use it instead of >> literal "true". >> >> * HashedPasswordManager >> >> 80-83: The fields can be final and use the constructor to initialize >> in all cases and make the class final >> to avoid unintentional subclassing. >> >> >> 113: canWriteToFile: It should be made clear in the template that >> *both* the Security policy >> and the file access value are used to check that the file can be >> updated. >> >> 200: loadPasswords() - should this confirm the access to the file is >> allowed and it has >> the correct file access before reading? >> >> Is the re-writing of the passwords intended to be done by a >> 'priveleged' system. >> Does this need doPrivileged? >> >> * HashedPasswordFileTest: >> >> 88: should use the TestLibrary Utils.getRandomInstance so it logs the >> seed and can be replayed if necessary. >> >> >> Thanks, Roger >> >> >> On 4/23/2017 6:20 AM, Harsha Wardhana B wrote: >>> >>> Hi All, >>> >>> Please review this enhancement to replace plain-text password for >>> JMX agent with SHA-256 hash. >>> >>> Issue: https://bugs.openjdk.java.net/browse/JDK-5016517 >>> >>> >>> webrev: http://cr.openjdk.java.net/~hb/5016517/webrev.00/ >>> >>> Overview of implementation: >>> >>> Currently, the JMX agent password file used to authenticate user, >>> stores user name and password as clear text. Though system level >>> restrictions are recommended for jmx password file, passwords are >>> vulnerable since they are stored in clear. The current RFE proposes >>> to store passwords as SHA256 hash instead of clear text. >>> >>> In current implementation, if password file is writable, and if >>> passwords are in clear, they will be replaced by SHA256 hash upon >>> agent boot-up or when login attempt is made. >>> >>> The file, >>> src/jdk.management.agent/share/conf/jmxremote.password.template >>> contains more details about the implementation. >>> >>> - Harsha >>> >>> >>> >>> >> > From daniel.fuchs at oracle.com Mon May 8 06:33:55 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 8 May 2017 07:33:55 +0100 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> Message-ID: <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> Hi Ujwal, For consistency I think the new getListenerIds method should: a) either return an array of Integer, even if it contains only 1 Integer: 1. The name of the method implies that an array is returned 2. You will need the array when you call connection.removeNotificationListeners anyway. or b) the other possibility is to remove the 's' at the end of the method. Both would be acceptable. best regards, -- daniel On 04/05/17 07:59, Ujwal Vangapally wrote: > corrected webrev link : > http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ > > > On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >> >> Kindly review the changes made for below bug >> >> Problem description and solution are explained in comments section >> >> https://bugs.openjdk.java.net/browse/JDK-6515161 >> >> diff for*ClientNotifForwarder.java *might be a bit confusing as it >> shows the method name >> >> removeNotificationListener is modified to getListenerIds and new >> method removeNotificationListener is introduced. >> >> Actual change is new method getListenerIds is introduced and it is >> called in removeNotificationListener method without >> >> affecting the functionality of removeNotificationListener. >> >> webrev : cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >> >> Thanks, >> >> Ujwal, >> > From ujwal.vangapally at oracle.com Mon May 8 08:45:22 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Mon, 8 May 2017 14:15:22 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> Message-ID: <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> Thanks for the Review Daniel, Harsha Please find the new webrev incorporating the review comments webrev : http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ Thanks, Ujwal On 5/8/2017 12:03 PM, Daniel Fuchs wrote: > Hi Ujwal, > > For consistency I think the new getListenerIds method should: > > a) either return an array of Integer, even if it contains only 1 > Integer: > > 1. The name of the method implies that an array is returned > 2. You will need the array when you call > connection.removeNotificationListeners anyway. > > or b) the other possibility is to remove the 's' at the end of the > method. > > Both would be acceptable. > > best regards, > > -- daniel > > On 04/05/17 07:59, Ujwal Vangapally wrote: >> corrected webrev link : >> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >> >> >> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>> >>> Kindly review the changes made for below bug >>> >>> Problem description and solution are explained in comments section >>> >>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>> >>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>> shows the method name >>> >>> removeNotificationListener is modified to getListenerIds and new >>> method removeNotificationListener is introduced. >>> >>> Actual change is new method getListenerIds is introduced and it is >>> called in removeNotificationListener method without >>> >>> affecting the functionality of removeNotificationListener. >>> >>> webrev : >>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>> >>> Thanks, >>> >>> Ujwal, >>> >> > From daniel.fuchs at oracle.com Mon May 8 09:46:12 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 8 May 2017 10:46:12 +0100 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> Message-ID: <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> On 08/05/2017 09:45, Ujwal Vangapally wrote: > Thanks for the Review Daniel, Harsha > > Please find the new webrev incorporating the review comments > > webrev : > http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ Looks good. In retrospect I wonder if renaming getListenerIds/getListenerId into getClientListenerIds/getClientListenerId was such a good idea given that 'listenerId' is an established terminology in the specification [1] [2] (and 'listenerId' is used all over the place in ClientNotifForwarder anyway). Harsha, what do you think? best regards, -- daniel [1] See @return in http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#addNotificationListeners-javax.management.ObjectName:A-java.rmi.MarshalledObject:A-javax.security.auth.Subject:A- [2] see @param listenerIds in http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#removeNotificationListeners-javax.management.ObjectName-java.lang.Integer:A-javax.security.auth.Subject- > > Thanks, > > Ujwal > > > On 5/8/2017 12:03 PM, Daniel Fuchs wrote: >> Hi Ujwal, >> >> For consistency I think the new getListenerIds method should: >> >> a) either return an array of Integer, even if it contains only 1 >> Integer: >> >> 1. The name of the method implies that an array is returned >> 2. You will need the array when you call >> connection.removeNotificationListeners anyway. >> >> or b) the other possibility is to remove the 's' at the end of the >> method. >> >> Both would be acceptable. >> >> best regards, >> >> -- daniel >> >> On 04/05/17 07:59, Ujwal Vangapally wrote: >>> corrected webrev link : >>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>> >>> >>> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>>> >>>> Kindly review the changes made for below bug >>>> >>>> Problem description and solution are explained in comments section >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>>> >>>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>>> shows the method name >>>> >>>> removeNotificationListener is modified to getListenerIds and new >>>> method removeNotificationListener is introduced. >>>> >>>> Actual change is new method getListenerIds is introduced and it is >>>> called in removeNotificationListener method without >>>> >>>> affecting the functionality of removeNotificationListener. >>>> >>>> webrev : >>>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>> >>>> Thanks, >>>> >>>> Ujwal, >>>> >>> >> > From harsha.wardhana.b at oracle.com Mon May 8 16:13:41 2017 From: harsha.wardhana.b at oracle.com (Harsha Wardhana B) Date: Mon, 8 May 2017 21:43:41 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> Message-ID: <791eb803-eabb-eb54-839d-40bb0f8a2a70@oracle.com> Hi Daniel, The term 'listenerid' is used in conjunction with method name/object field which adds context about the term 'listenerid'. Having a standalone method name as getClientListenerId is less ambiguous compared to method name : getListenerId. I don't really have a strong opinion on this and am okay with either names. -Harsha On Monday 08 May 2017 03:16 PM, Daniel Fuchs wrote: > On 08/05/2017 09:45, Ujwal Vangapally wrote: >> Thanks for the Review Daniel, Harsha >> >> Please find the new webrev incorporating the review comments >> >> webrev : >> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ > > Looks good. In retrospect I wonder if renaming > getListenerIds/getListenerId into > getClientListenerIds/getClientListenerId > was such a good idea given that 'listenerId' is an established > terminology in the specification [1] [2] (and 'listenerId' is used all > over the place in ClientNotifForwarder anyway). > > Harsha, what do you think? > > best regards, > > -- daniel > > [1] See @return in > http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#addNotificationListeners-javax.management.ObjectName:A-java.rmi.MarshalledObject:A-javax.security.auth.Subject:A- > [2] see @param listenerIds in > http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#removeNotificationListeners-javax.management.ObjectName-java.lang.Integer:A-javax.security.auth.Subject- > > > >> >> Thanks, >> >> Ujwal >> >> >> On 5/8/2017 12:03 PM, Daniel Fuchs wrote: >>> Hi Ujwal, >>> >>> For consistency I think the new getListenerIds method should: >>> >>> a) either return an array of Integer, even if it contains only 1 >>> Integer: >>> >>> 1. The name of the method implies that an array is returned >>> 2. You will need the array when you call >>> connection.removeNotificationListeners anyway. >>> >>> or b) the other possibility is to remove the 's' at the end of the >>> method. >>> >>> Both would be acceptable. >>> >>> best regards, >>> >>> -- daniel >>> >>> On 04/05/17 07:59, Ujwal Vangapally wrote: >>>> corrected webrev link : >>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>> >>>> >>>> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>>>> >>>>> Kindly review the changes made for below bug >>>>> >>>>> Problem description and solution are explained in comments section >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>>>> >>>>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>>>> shows the method name >>>>> >>>>> removeNotificationListener is modified to getListenerIds and new >>>>> method removeNotificationListener is introduced. >>>>> >>>>> Actual change is new method getListenerIds is introduced and it is >>>>> called in removeNotificationListener method without >>>>> >>>>> affecting the functionality of removeNotificationListener. >>>>> >>>>> webrev : >>>>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>> >>>>> Thanks, >>>>> >>>>> Ujwal, >>>>> >>>> >>> >> > From daniel.fuchs at oracle.com Mon May 8 16:27:20 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 8 May 2017 17:27:20 +0100 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <791eb803-eabb-eb54-839d-40bb0f8a2a70@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> <791eb803-eabb-eb54-839d-40bb0f8a2a70@oracle.com> Message-ID: <5ff4179c-bd85-94b5-55f1-1d057cf41168@oracle.com> Hi Harsha, Well, I think both methods should have the same name and I don't see any strong reason for changing the old name - since there is no such thing as a 'ClientListener' (there are only NotificationListeners). best regards, -- daniel On 08/05/2017 17:13, Harsha Wardhana B wrote: > Hi Daniel, > > The term 'listenerid' is used in conjunction with method name/object > field which adds context about the term 'listenerid'. Having a > standalone method name as getClientListenerId is less ambiguous compared > to method name : getListenerId. > > I don't really have a strong opinion on this and am okay with either names. > > -Harsha > > > On Monday 08 May 2017 03:16 PM, Daniel Fuchs wrote: >> On 08/05/2017 09:45, Ujwal Vangapally wrote: >>> Thanks for the Review Daniel, Harsha >>> >>> Please find the new webrev incorporating the review comments >>> >>> webrev : >>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ >> >> Looks good. In retrospect I wonder if renaming >> getListenerIds/getListenerId into >> getClientListenerIds/getClientListenerId >> was such a good idea given that 'listenerId' is an established >> terminology in the specification [1] [2] (and 'listenerId' is used all >> over the place in ClientNotifForwarder anyway). >> >> Harsha, what do you think? >> >> best regards, >> >> -- daniel >> >> [1] See @return in >> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#addNotificationListeners-javax.management.ObjectName:A-java.rmi.MarshalledObject:A-javax.security.auth.Subject:A- >> >> [2] see @param listenerIds in >> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#removeNotificationListeners-javax.management.ObjectName-java.lang.Integer:A-javax.security.auth.Subject- >> >> >> >>> >>> Thanks, >>> >>> Ujwal >>> >>> >>> On 5/8/2017 12:03 PM, Daniel Fuchs wrote: >>>> Hi Ujwal, >>>> >>>> For consistency I think the new getListenerIds method should: >>>> >>>> a) either return an array of Integer, even if it contains only 1 >>>> Integer: >>>> >>>> 1. The name of the method implies that an array is returned >>>> 2. You will need the array when you call >>>> connection.removeNotificationListeners anyway. >>>> >>>> or b) the other possibility is to remove the 's' at the end of the >>>> method. >>>> >>>> Both would be acceptable. >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>> On 04/05/17 07:59, Ujwal Vangapally wrote: >>>>> corrected webrev link : >>>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>> >>>>> >>>>> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>>>>> >>>>>> Kindly review the changes made for below bug >>>>>> >>>>>> Problem description and solution are explained in comments section >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>>>>> >>>>>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>>>>> shows the method name >>>>>> >>>>>> removeNotificationListener is modified to getListenerIds and new >>>>>> method removeNotificationListener is introduced. >>>>>> >>>>>> Actual change is new method getListenerIds is introduced and it is >>>>>> called in removeNotificationListener method without >>>>>> >>>>>> affecting the functionality of removeNotificationListener. >>>>>> >>>>>> webrev : >>>>>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ujwal, >>>>>> >>>>> >>>> >>> >> > From ujwal.vangapally at oracle.com Tue May 9 05:38:00 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Tue, 9 May 2017 11:08:00 +0530 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: <5ff4179c-bd85-94b5-55f1-1d057cf41168@oracle.com> References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> <791eb803-eabb-eb54-839d-40bb0f8a2a70@oracle.com> <5ff4179c-bd85-94b5-55f1-1d057cf41168@oracle.com> Message-ID: Hi Harsha, Daniel, updated method names as getListenerIds/getListenerId webrev : http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.02/ On 5/8/2017 9:57 PM, Daniel Fuchs wrote: > Hi Harsha, > > Well, I think both methods should have the same name > and I don't see any strong reason for changing the old > name - since there is no such thing as a 'ClientListener' > (there are only NotificationListeners). > > best regards, > > -- daniel > > On 08/05/2017 17:13, Harsha Wardhana B wrote: >> Hi Daniel, >> >> The term 'listenerid' is used in conjunction with method name/object >> field which adds context about the term 'listenerid'. Having a >> standalone method name as getClientListenerId is less ambiguous compared >> to method name : getListenerId. >> >> I don't really have a strong opinion on this and am okay with either >> names. >> >> -Harsha >> >> >> On Monday 08 May 2017 03:16 PM, Daniel Fuchs wrote: >>> On 08/05/2017 09:45, Ujwal Vangapally wrote: >>>> Thanks for the Review Daniel, Harsha >>>> >>>> Please find the new webrev incorporating the review comments >>>> >>>> webrev : >>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ >>> >>> Looks good. In retrospect I wonder if renaming >>> getListenerIds/getListenerId into >>> getClientListenerIds/getClientListenerId >>> was such a good idea given that 'listenerId' is an established >>> terminology in the specification [1] [2] (and 'listenerId' is used all >>> over the place in ClientNotifForwarder anyway). >>> >>> Harsha, what do you think? >>> >>> best regards, >>> >>> -- daniel >>> >>> [1] See @return in >>> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#addNotificationListeners-javax.management.ObjectName:A-java.rmi.MarshalledObject:A-javax.security.auth.Subject:A- >>> >>> >>> [2] see @param listenerIds in >>> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#removeNotificationListeners-javax.management.ObjectName-java.lang.Integer:A-javax.security.auth.Subject- >>> >>> >>> >>> >>>> >>>> Thanks, >>>> >>>> Ujwal >>>> >>>> >>>> On 5/8/2017 12:03 PM, Daniel Fuchs wrote: >>>>> Hi Ujwal, >>>>> >>>>> For consistency I think the new getListenerIds method should: >>>>> >>>>> a) either return an array of Integer, even if it contains only 1 >>>>> Integer: >>>>> >>>>> 1. The name of the method implies that an array is returned >>>>> 2. You will need the array when you call >>>>> connection.removeNotificationListeners anyway. >>>>> >>>>> or b) the other possibility is to remove the 's' at the end of the >>>>> method. >>>>> >>>>> Both would be acceptable. >>>>> >>>>> best regards, >>>>> >>>>> -- daniel >>>>> >>>>> On 04/05/17 07:59, Ujwal Vangapally wrote: >>>>>> corrected webrev link : >>>>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>>> >>>>>> >>>>>> >>>>>> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>>>>>> >>>>>>> Kindly review the changes made for below bug >>>>>>> >>>>>>> Problem description and solution are explained in comments section >>>>>>> >>>>>>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>>>>>> >>>>>>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>>>>>> shows the method name >>>>>>> >>>>>>> removeNotificationListener is modified to getListenerIds and new >>>>>>> method removeNotificationListener is introduced. >>>>>>> >>>>>>> Actual change is new method getListenerIds is introduced and it is >>>>>>> called in removeNotificationListener method without >>>>>>> >>>>>>> affecting the functionality of removeNotificationListener. >>>>>>> >>>>>>> webrev : >>>>>>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Ujwal, >>>>>>> >>>>>> >>>>> >>>> >>> >> > From daniel.fuchs at oracle.com Tue May 9 08:09:14 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 9 May 2017 09:09:14 +0100 Subject: RFR: JDK-6515161 If remote removeNotificationListener gets SecurityException, client no longer gets notifications In-Reply-To: References: <7b2b4d66-8a2a-13d3-3149-b1ffaa4275a4@oracle.com> <4daa7c96-51a5-6a88-4274-550f46fca4e1@oracle.com> <9ec152dd-026c-5bf5-00ff-2664143c9a7b@oracle.com> <0d3b5755-428f-792b-8ece-1c831cca3095@oracle.com> <65fd5b7d-cb37-fbf4-292c-9d92660c330d@oracle.com> <791eb803-eabb-eb54-839d-40bb0f8a2a70@oracle.com> <5ff4179c-bd85-94b5-55f1-1d057cf41168@oracle.com> Message-ID: <96f866ae-ff21-dfdd-4068-06731b2feb3d@oracle.com> +1 Thanks Ujwal! -- daniel On 09/05/17 06:38, Ujwal Vangapally wrote: > Hi Harsha, Daniel, > > updated method names as getListenerIds/getListenerId > > webrev : > http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.02/ > > > On 5/8/2017 9:57 PM, Daniel Fuchs wrote: >> Hi Harsha, >> >> Well, I think both methods should have the same name >> and I don't see any strong reason for changing the old >> name - since there is no such thing as a 'ClientListener' >> (there are only NotificationListeners). >> >> best regards, >> >> -- daniel >> >> On 08/05/2017 17:13, Harsha Wardhana B wrote: >>> Hi Daniel, >>> >>> The term 'listenerid' is used in conjunction with method name/object >>> field which adds context about the term 'listenerid'. Having a >>> standalone method name as getClientListenerId is less ambiguous compared >>> to method name : getListenerId. >>> >>> I don't really have a strong opinion on this and am okay with either >>> names. >>> >>> -Harsha >>> >>> >>> On Monday 08 May 2017 03:16 PM, Daniel Fuchs wrote: >>>> On 08/05/2017 09:45, Ujwal Vangapally wrote: >>>>> Thanks for the Review Daniel, Harsha >>>>> >>>>> Please find the new webrev incorporating the review comments >>>>> >>>>> webrev : >>>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.01/ >>>> >>>> Looks good. In retrospect I wonder if renaming >>>> getListenerIds/getListenerId into >>>> getClientListenerIds/getClientListenerId >>>> was such a good idea given that 'listenerId' is an established >>>> terminology in the specification [1] [2] (and 'listenerId' is used all >>>> over the place in ClientNotifForwarder anyway). >>>> >>>> Harsha, what do you think? >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>> [1] See @return in >>>> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#addNotificationListeners-javax.management.ObjectName:A-java.rmi.MarshalledObject:A-javax.security.auth.Subject:A- >>>> >>>> >>>> [2] see @param listenerIds in >>>> http://download.java.net/java/jdk9/docs/api/javax/management/remote/rmi/RMIConnection.html#removeNotificationListeners-javax.management.ObjectName-java.lang.Integer:A-javax.security.auth.Subject- >>>> >>>> >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Ujwal >>>>> >>>>> >>>>> On 5/8/2017 12:03 PM, Daniel Fuchs wrote: >>>>>> Hi Ujwal, >>>>>> >>>>>> For consistency I think the new getListenerIds method should: >>>>>> >>>>>> a) either return an array of Integer, even if it contains only 1 >>>>>> Integer: >>>>>> >>>>>> 1. The name of the method implies that an array is returned >>>>>> 2. You will need the array when you call >>>>>> connection.removeNotificationListeners anyway. >>>>>> >>>>>> or b) the other possibility is to remove the 's' at the end of the >>>>>> method. >>>>>> >>>>>> Both would be acceptable. >>>>>> >>>>>> best regards, >>>>>> >>>>>> -- daniel >>>>>> >>>>>> On 04/05/17 07:59, Ujwal Vangapally wrote: >>>>>>> corrected webrev link : >>>>>>> http://cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 5/4/2017 12:14 PM, Ujwal Vangapally wrote: >>>>>>>> >>>>>>>> Kindly review the changes made for below bug >>>>>>>> >>>>>>>> Problem description and solution are explained in comments section >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-6515161 >>>>>>>> >>>>>>>> diff for*ClientNotifForwarder.java *might be a bit confusing as it >>>>>>>> shows the method name >>>>>>>> >>>>>>>> removeNotificationListener is modified to getListenerIds and new >>>>>>>> method removeNotificationListener is introduced. >>>>>>>> >>>>>>>> Actual change is new method getListenerIds is introduced and it is >>>>>>>> called in removeNotificationListener method without >>>>>>>> >>>>>>>> affecting the functionality of removeNotificationListener. >>>>>>>> >>>>>>>> webrev : >>>>>>>> cr.openjdk.java.net/~uvangapally/webrev/2017/6515161/webrev.00/ >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Ujwal, >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > From serguei.spitsyn at oracle.com Tue May 9 21:42:41 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 9 May 2017 14:42:41 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> Message-ID: <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> Hi Dmitry, I'd like to resolve my questions before the upcoming design discussion on Thu. http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html (0) The design description from the bug report tells: > Than we change a negotiation protocol between JDWP and transport. > We pass maximal supported version to transport initialization routine and expect transport actual version to be returned. The modified negotiation protocol adds extra complexity. What is a motivation behind this? Is it really necessary for the transport library to return an actual version instead of rejecting the unmatched version? I do not see it is really used in the webrev.15 implementation. (1) The following change in the jdwp transport library will reject theJDWPTRANSPORT_VERSION_1_0 as it is below the version JDWPTRANSPORT_VERSION_1_1, but will except any version above the JDWPTRANSPORT_VERSION_1_1 (with providing/returning the actual transport version): jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, - jint version, jdwpTransportEnv** result) + jint version, void** env) { - if (version != JDWPTRANSPORT_VERSION_1_0) { + if (version < JDWPTRANSPORT_VERSION_1_1) { return JNI_EVERSION; } Te following change will also prevent supporting the 1_0 version of the transport library: - interface.StartListening = &socketTransport_startListening; + interface.StartListening = NULL; I'd suggest the following alternate change to the transport API allowing to support both old and new versions at the same time (it would simplify the negotiation rules): - Add new function: jdwpTransportError AllowPeers(const char* peers); - Keep the original StartListening function. The function uses the allowed peers data if it was previously cached by the AllowPeers(). - It seems, the alternate approach does not require adding the extra_data with version. But if there is still a real need to get the transport API version then it'd better to introduce a function named GetTransportVersion() or JDWP_TransportVersion(). This would allow to encapsulate any extra_data that is necessary in such a case. (2) The following error messages miss the actual IP address or mask that was found to be illegal: 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip address for allow"); 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for allow"); (3) A suggestion on the following: 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask I'd suggest a more explicit approach instead of the L400 for a better clarity: 386 if (*s == '/') { 387 // netmask specified 388 s = mask_s2u(s + 1, &mask); 389 if (*(s-1) == '/') { 390 // Input is not consumed, something bad happens 391 _peers_cnt = 0; 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for allow"); 393 } 394 } else { mask = 0xFFFFFFFF; } http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html (4) Some confusion with the fragment and its comment: + + /* Pass the latest supported version, + * expect actual transport version in t->extra_data->version + */ + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); + if (ver == JNI_EVERSION) { ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, &t); + // Special handling for versionless transports + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; + } + else { + info->transportVersion = (*t)->extra_data->version; + } + The term "latest supported version" is ambiguous in this context. Do you mean, supported by the JDWP back-end or by the agent library? Also, it is not clear in what circumstances the agent library would support the version 1_0 only. The JDWP back-end is always coupled with the socket transport library, isn't it? Is it because the libdt_shmem library can be used on Windows or because a different native library path can be used? Could you explain a little bit? As we see in (1) above the actual transport version can be different from requested only if the requested version is above the latest supported by the transport library. Otherwise, the version is matched or the JNI_EVERSION is returned. The subsequent call to the OnLoad function can succeed only if the library supports just the version 1_0. (5) Memory allocation for the info->allow field is needed only for the transport version 1_1: + if (allow != NULL) { + info->allow = jvmtiAllocate((int)strlen(allow)+1); + if (info->allow == NULL) { + serror = JDWP_ERROR(OUT_OF_MEMORY); + goto handleError; + } + (void)strcpy(info->allow, allow); + } (6) There is no handling for the condition when the *allow* parameter is passed but the transport version is 1_0 (which does not support the *allow* parameter): + /* Interface version 1.0 doesn't support versioning, so we have to + * use global variable and set the version artifically. + * Use (*t)->extra_data->version directly when 1.0 is gone. + */ + switch(info->transportVersion) { + case JDWPTRANSPORT_VERSION_1_0: err = (*trans)->StartListening(trans, address, &retAddress); + break; + case JDWPTRANSPORT_VERSION_1_1: + err = (*trans)->StartListening11(trans, address, &retAddress, info->allow); + break; + default: + err = JNI_EVERSION; + } Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: > Robbin, > >> One follow-up issue is that if you start suspended >> and than connect with >> an unallowed client the JVM starts and executes the program. > Fixed. > > see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 > > -Dmitry > > On 2017-03-16 15:59, Robbin Ehn wrote: >> Hi Dmitry, thanks for the update. >> >> One follow-up issue is that if you start suspended and than connect with >> an unallowed client the JVM starts and executes the program. >> Simple program prints "Hello". >> >> [rehn at rehn-ws vanilla-hs]$ java >> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >> -cp . H >> Listening for transport dt_socket at address: 9999 >> ERROR: Peer not allowed to connect >> Listening for transport dt_socket at address: 9999 >> Hello >> >> I think it would be good if the VM stayed suspended when an unallowed >> client connects. >> >> Thanks, Robbin >> >> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>> Robbin, >>> >>> Please, see: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>> >>>> 1: >>>> So connecting with an unallowed client terminates the VM. >>> Fixed. >>> >>>> 2: >>>> Starting with an bad allow filter terminates the VM when connecting a >>>> client. >>> Moved allowed parameter (and parser call) to StartListening. >>> >>> -Dmitry >>> >>> >>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>> Hi Dmitry, >>>> >>>> I took a look at this, I have two practical issues: >>>> >>>> 1: >>>> [rehn at rehn-ws dev]$ java >>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>> >>>> -cp runs ForEver >>>> Listening for transport dt_socket at address: 9999 >>>> ERROR: transport error 202: peer not allowed to connect: Success >>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or fatal >>>> error [transport.c:358] >>>> >>>> So connecting with an unallowed client terminates the VM. >>>> >>>> 2: >>>> [rehn at rehn-ws dev]$ java >>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>> >>>> -cp runs ForEver >>>> Listening for transport dt_socket at address: 9999 >>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>> Success >>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or fatal >>>> error [transport.c:358] >>>> >>>> Starting with an bad allow filter terminates the VM when connecting a >>>> client. >>>> >>>> >>>> Connecting with an unallowed ip/port should not terminate the VM and we >>>> should verify allow filter directly at startup. >>>> >>>> Thanks >>>> >>>> /Robbin >>>> >>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>> Everybody, >>>>> >>>>> Please review: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>> >>>>> These changes introduce new parameter[1] of the socket transport - >>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>> connect to jdwp server and it's the second part of JDWP hardening[2]. >>>>> >>>>> No restrictions are applied by default now but I'll file a separate CR >>>>> to restrict list of allowed peers to localhost by default. >>>>> >>>>> Also these changes implement versioning for jdwp transport and therefor >>>>> simplify feature development of jdwp. >>>>> >>>>> >>>>> 1. Example command line: >>>>> >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>> >>>>> Possible values for allow parameter: >>>>> * - accept connections from everywhere. >>>>> N.N.N.N - accept connections from this IP address only >>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>> >>>>> >>>>> >>>>> 2. JDK-8052136 JDWP hardening >>>>> >>>>> -Dmitry >>>>> >>> > From serguei.spitsyn at oracle.com Wed May 10 05:18:15 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 9 May 2017 22:18:15 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: Message-ID: <9b06a8aa-7832-f825-ae78-f837fb53dd9a@oracle.com> (The formatting is broken in my reply. I've changed my Zhunderbird 'Send Options' to avoid this in the future. I tried to fix the formatting below.) Hi Dmitry, I'd like to resolve my questions before the upcoming design discussion on Thu. http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html (0) The design description from the bug report tells: > Than we change a negotiation protocol between JDWP and transport. > We pass maximal supported version to transport initialization > routine and expect transport actual version to be returned. The modified negotiation protocol adds extra complexity. What is a motivation behind this? Is it really necessary for the transport library to return an actual version instead of rejecting the unmatched version? I do not see it is really used in the webrev.15 implementation. (1) The following change in the jdwp transport library will reject theJDWPTRANSPORT_VERSION_1_0 as it is below the version JDWPTRANSPORT_VERSION_1_1, but will except any version above the JDWPTRANSPORT_VERSION_1_1 (with providing/returning the actual transport version): jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, - jint version, jdwpTransportEnv** result) + jint version, void** env) { - if (version != JDWPTRANSPORT_VERSION_1_0) { + if (version < JDWPTRANSPORT_VERSION_1_1) { return JNI_EVERSION; } Te following change will also prevent supporting the 1_0 version of the transport library: - interface.StartListening = &socketTransport_startListening; + interface.StartListening = NULL; I'd suggest the following alternate change to the transport API allowing to support both old and new versions at the same time (it would simplify the negotiation rules): - Add new function: jdwpTransportError AllowPeers(const char* peers); - Keep the original StartListening function. The function uses the allowed peers data if it was previously cached by the AllowPeers(). - It seems, the alternate approach does not require adding the extra_data with version. But if there is still a real need to get the transport API version then it'd better to introduce a function named GetTransportVersion() or JDWP_TransportVersion(). This would allow to encapsulate any extra_data that is necessary in such a case. (2) The following error messages miss the actual IP address or mask that was found to be illegal: 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip address for allow"); 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for allow"); (3) A suggestion on the following: 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask I'd suggest a more explicit approach instead of the L400 for a better clarity: 386 if (*s == '/') { 387 // netmask specified 388 s = mask_s2u(s + 1, &mask); 389 if (*(s-1) == '/') { 390 // Input is not consumed, something bad happens 391 _peers_cnt = 0; 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for allow"); 393 } 394 } else { mask = 0xFFFFFFFF; } http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html (4) Some confusion with the fragment and its comment: + + /* Pass the latest supported version, + * expect actual transport version in t->extra_data->version + */ + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); + if (ver == JNI_EVERSION) { ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, &t); + // Special handling for versionless transports + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; + } + else { + info->transportVersion = (*t)->extra_data->version; + } + The term "latest supported version" is ambiguous in this context. Do you mean, supported by the JDWP back-end or by the agent library? Also, it is not clear in what circumstances the agent library would support the version 1_0 only. The JDWP back-end is always coupled with the socket transport library, isn't it? Is it because the libdt_shmem library can be used on Windows or because a different native library path can be used? Could you explain a little bit? As we see in (1) above the actual transport version can be different from requested only if the requested version is above the latest supported by the transport library. Otherwise, the version is matched or the JNI_EVERSION is returned. The subsequent call to the OnLoad function can succeed only if the library supports just the version 1_0. The returned 'ver' is not checked for an error after the second *OnLoad call. (5) Memory allocation for the info->allow filed is needed only for the transport version 1_1: + if (allow != NULL) { + info->allow = jvmtiAllocate((int)strlen(allow)+1); + if (info->allow == NULL) { + serror = JDWP_ERROR(OUT_OF_MEMORY); + goto handleError; + } + (void)strcpy(info->allow, allow); + } (6) There is no handling for the condition when the *allow* parameter is passed but the transport version is 1_0 (which does not support the *allow* parameter): + /* Interface version 1.0 doesn't support versioning, so we have to + * use global variable and set the version artifically. + * Use (*t)->extra_data->version directly when 1.0 is gone. + */ + switch(info->transportVersion) { + case JDWPTRANSPORT_VERSION_1_0: err = (*trans)->StartListening(trans, address, &retAddress); + break; + case JDWPTRANSPORT_VERSION_1_1: + err = (*trans)->StartListening11(trans, address, &retAddress, info->allow); + break; + default: + err = JNI_EVERSION; + } Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: > Robbin, > >> One follow-up issue is that if you start suspended >> and than connect with >> an unallowed client the JVM starts and executes the program. > Fixed. > > see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 > > -Dmitry > > On 2017-03-16 15:59, Robbin Ehn wrote: >> Hi Dmitry, thanks for the update. >> >> One follow-up issue is that if you start suspended and than connect with >> an unallowed client the JVM starts and executes the program. >> Simple program prints "Hello". >> >> [rehn at rehn-ws vanilla-hs]$ java >> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >> -cp . H >> Listening for transport dt_socket at address: 9999 >> ERROR: Peer not allowed to connect >> Listening for transport dt_socket at address: 9999 >> Hello >> >> I think it would be good if the VM stayed suspended when an unallowed >> client connects. >> >> Thanks, Robbin >> >> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>> Robbin, >>> >>> Please, see: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>> >>>> 1: >>>> So connecting with an unallowed client terminates the VM. >>> Fixed. >>> >>>> 2: >>>> Starting with an bad allow filter terminates the VM when connecting a >>>> client. >>> Moved allowed parameter (and parser call) to StartListening. >>> >>> -Dmitry >>> >>> >>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>> Hi Dmitry, >>>> >>>> I took a look at this, I have two practical issues: >>>> >>>> 1: >>>> [rehn at rehn-ws dev]$ java >>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>> >>>> -cp runs ForEver >>>> Listening for transport dt_socket at address: 9999 >>>> ERROR: transport error 202: peer not allowed to connect: Success >>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or fatal >>>> error [transport.c:358] >>>> >>>> So connecting with an unallowed client terminates the VM. >>>> >>>> 2: >>>> [rehn at rehn-ws dev]$ java >>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>> >>>> -cp runs ForEver >>>> Listening for transport dt_socket at address: 9999 >>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>> Success >>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or fatal >>>> error [transport.c:358] >>>> >>>> Starting with an bad allow filter terminates the VM when connecting a >>>> client. >>>> >>>> >>>> Connecting with an unallowed ip/port should not terminate the VM and we >>>> should verify allow filter directly at startup. >>>> >>>> Thanks >>>> >>>> /Robbin >>>> >>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>> Everybody, >>>>> >>>>> Please review: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>> >>>>> These changes introduce new parameter[1] of the socket transport - >>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>> connect to jdwp server and it's the second part of JDWP hardening[2]. >>>>> >>>>> No restrictions are applied by default now but I'll file a separate CR >>>>> to restrict list of allowed peers to localhost by default. >>>>> >>>>> Also these changes implement versioning for jdwp transport and therefor >>>>> simplify feature development of jdwp. >>>>> >>>>> >>>>> 1. Example command line: >>>>> >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>> >>>>> Possible values for allow parameter: >>>>> * - accept connections from everywhere. >>>>> N.N.N.N - accept connections from this IP address only >>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>> >>>>> >>>>> >>>>> 2. JDK-8052136 JDWP hardening >>>>> >>>>> -Dmitry >>>>> >>> > From fairoz.matte at oracle.com Wed May 10 07:53:18 2017 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Wed, 10 May 2017 00:53:18 -0700 (PDT) Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html Message-ID: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> Hi, Kindly review the small typo fix, applicable only for JDK8 BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ Thanks, Fairoz From dmitry.samersoff at oracle.com Wed May 10 08:10:55 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 10 May 2017 11:10:55 +0300 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> Message-ID: <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Serguei, Please see my comments in-line. On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: > Hi Dmitry, > > > I'd like to resolve my questions before the upcoming design discussion > on Thu. > > > http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html > > > > (0) The design description from the bug report tells: > > > Than we change a negotiation protocol between JDWP and transport. > > We pass maximal supported version to transport initialization > routine and expect transport actual version to be returned. > > The modified negotiation protocol adds extra complexity. > What is a motivation behind this? > Is it really necessary for the transport library to return an actual > version instead of rejecting the unmatched version? > I do not see it is really used in the webrev.15 implementation. Transport have to return it's actual version in order to allow agent to perform appropriate action. see libjdwp/transport.c:526 Today it's just a selection of proper API call but in a future it might be too-old-transport-error or deprecation warning or security warning or something else. > (1) The following change in the jdwp transport library will reject > theJDWPTRANSPORT_VERSION_1_0 as it is below > the version JDWPTRANSPORT_VERSION_1_1, but will except any version > above the JDWPTRANSPORT_VERSION_1_1 > (with providing/returning the actual transport version): > > jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, > - jint version, jdwpTransportEnv** result) > + jint version, void** env) > { > - if (version != JDWPTRANSPORT_VERSION_1_0) { > + if (version < JDWPTRANSPORT_VERSION_1_1) { > return JNI_EVERSION; > } > > > Te following change will also prevent supporting the 1_0 version of the > transport library: > > - interface.StartListening = &socketTransport_startListening; > + interface.StartListening = NULL; libdt_socket/socketTransport.c is an implementation of 1.1 *transport* it's not intended to run with 1.0 *backend*. i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* require 1.1 or greater backend. see: libjdwp/transport.c:206 for version logic on backend (agent) side > I'd suggest the following alternate change to the transport API allowing > to support > both old and new versions at the same time (it would simplify the > negotiation rules): > - Add new function: > jdwpTransportError AllowPeers(const char* peers); > > - Keep the original StartListening function. > The function uses the allowed peers data if it was previously cached > by the AllowPeers(). > > - It seems, the alternate approach does not require adding the > extra_data with version. > But if there is still a real need to get the transport API version > then it'd better > to introduce a function named GetTransportVersion() or > JDWP_TransportVersion(). > This would allow to encapsulate any extra_data that is necessary in > such a case. >From pure JDWP hardening point of view we can just add extra parameter *allow* to existing StartListening(). Caller is responsible for stack consistency so old transport continues to work. But my goal was to create a versioning in JDWP agent -> transport relations that was missed in initial JDWP design. Further, more complicated, changes (IPv6 support, UDS sockets support etc) require this logic. We have a structure jdwpTransportNativeInterface with a fixed set of functions (see jdwpTransport.h:153). To add any new function to this structure we have to create a method to detect presence of this function. So we can't use GetTransportVersion(). With as separate AllowPeer() function nobody remind an agent writer that they should use it, but extra parameter makes api changes and requirements clear visible (up to compiler warning). Also I'm against of changing behavior of existing function. > (2) The following error messages miss the actual IP address or mask that > was found to be illegal: > > 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip > address for allow"); 392 > RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for > allow"); Other parameter parsing functions (e.g. "invalid port number specified" at 274) doesn't explain what parameter is bad. I think typical allow would have one or two entry, so verbose error message is not worth significant complication of parsing code. I would prefer to leave it as is. > (3) A suggestion on the following: > > 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask > > I'd suggest a more explicit approach instead of the L400 for a better > clarity: > > 386 if (*s == '/') { > 387 // netmask specified > 388 s = mask_s2u(s + 1, &mask); > 389 if (*(s-1) == '/') { > 390 // Input is not consumed, something bad happens > 391 _peers_cnt = 0; > 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask > for allow"); > 393 } > 394 } else { mask = 0xFFFFFFFF; } I'll try it. > http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html > (4) Some confusion with the fragment and its comment: > > + > + /* Pass the latest supported version, > + * expect actual transport version in t->extra_data->version > + */ > + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); > + if (ver == JNI_EVERSION) { > ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, &t); > + // Special handling for versionless transports > + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; > + } > + else { > + info->transportVersion = (*t)->extra_data->version; > + } > + > > The term "latest supported version" is ambiguous in this context. Do you > mean, supported by the JDWP back-end or by the agent library? Supported by the JDWP backend. I'll update comments. > Also, it > is not clear in what circumstances the agent library would support the > version 1_0 only. The JDWP back-end is always coupled with the socket > transport library, isn't it? Is it because the libdt_shmem library can > be used on Windows or because a different native library path can be > used? Could you explain a little bit? It's more generic question: Do we need any backward compatibility here? I assume *yes* (can't recall why - probably it was a tree-year old decision). If we state that new backend will not support old transports today and in a future - all versioning logic is not necessary. > As we see in (1) above the actual > transport version can be different from requested only if the requested > version is above the latest supported by the transport library. > Otherwise, the version is matched or the JNI_EVERSION is returned. The > subsequent call to the OnLoad function can succeed only if the library > supports just the version 1_0. See answer to (1) above. transport library has exactly one version but JDWP backend supports couple of transport library versions back. > (5) Memory allocation for the info->allow > field is needed only for the transport version 1_1: > > + if (allow != NULL) { > + info->allow = jvmtiAllocate((int)strlen(allow)+1); > + if (info->allow == NULL) { > + serror = JDWP_ERROR(OUT_OF_MEMORY); > + goto handleError; > + } > + (void)strcpy(info->allow, allow); > + } Correct. allocation needed for 1.1 and *greater*. I can change it, but it makes code less readable. > (6) There is no handling for the condition when the *allow* parameter is > passed but the transport version is 1_0 (which does not support the > *allow* parameter): Correct. Warning or ever error should be here. I plan to open a separate follow-up CR for this problem - we have to decide how we should handle this situation (warning or error) and change the code, but I can add a plain printf() here if you like. > > + /* Interface version 1.0 doesn't support versioning, so we have to > + * use global variable and set the version artifically. > + * Use (*t)->extra_data->version directly when 1.0 is gone. > + */ > + switch(info->transportVersion) { > + case JDWPTRANSPORT_VERSION_1_0: > err = (*trans)->StartListening(trans, address, &retAddress); > + break; > + case JDWPTRANSPORT_VERSION_1_1: > + err = (*trans)->StartListening11(trans, address, &retAddress, > info->allow); > + break; > + default: > + err = JNI_EVERSION; > + } -Dmitry > > Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >> Robbin, >> >>> One follow-up issue is that if you start suspended >>> and than connect with >>> an unallowed client the JVM starts and executes the program. >> Fixed. >> >> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >> >> -Dmitry >> >> On 2017-03-16 15:59, Robbin Ehn wrote: >>> Hi Dmitry, thanks for the update. >>> >>> One follow-up issue is that if you start suspended and than connect with >>> an unallowed client the JVM starts and executes the program. >>> Simple program prints "Hello". >>> >>> [rehn at rehn-ws vanilla-hs]$ java >>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>> >>> -cp . H >>> Listening for transport dt_socket at address: 9999 >>> ERROR: Peer not allowed to connect >>> Listening for transport dt_socket at address: 9999 >>> Hello >>> >>> I think it would be good if the VM stayed suspended when an unallowed >>> client connects. >>> >>> Thanks, Robbin >>> >>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>> Robbin, >>>> >>>> Please, see: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>> >>>>> 1: >>>>> So connecting with an unallowed client terminates the VM. >>>> Fixed. >>>> >>>>> 2: >>>>> Starting with an bad allow filter terminates the VM when connecting a >>>>> client. >>>> Moved allowed parameter (and parser call) to StartListening. >>>> >>>> -Dmitry >>>> >>>> >>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>> Hi Dmitry, >>>>> >>>>> I took a look at this, I have two practical issues: >>>>> >>>>> 1: >>>>> [rehn at rehn-ws dev]$ java >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>> >>>>> >>>>> -cp runs ForEver >>>>> Listening for transport dt_socket at address: 9999 >>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>> fatal >>>>> error [transport.c:358] >>>>> >>>>> So connecting with an unallowed client terminates the VM. >>>>> >>>>> 2: >>>>> [rehn at rehn-ws dev]$ java >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>> >>>>> >>>>> -cp runs ForEver >>>>> Listening for transport dt_socket at address: 9999 >>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>> Success >>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>> fatal >>>>> error [transport.c:358] >>>>> >>>>> Starting with an bad allow filter terminates the VM when connecting a >>>>> client. >>>>> >>>>> >>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>> and we >>>>> should verify allow filter directly at startup. >>>>> >>>>> Thanks >>>>> >>>>> /Robbin >>>>> >>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>> Everybody, >>>>>> >>>>>> Please review: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>> >>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>>> connect to jdwp server and it's the second part of JDWP hardening[2]. >>>>>> >>>>>> No restrictions are applied by default now but I'll file a >>>>>> separate CR >>>>>> to restrict list of allowed peers to localhost by default. >>>>>> >>>>>> Also these changes implement versioning for jdwp transport and >>>>>> therefor >>>>>> simplify feature development of jdwp. >>>>>> >>>>>> >>>>>> 1. Example command line: >>>>>> >>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>> >>>>>> Possible values for allow parameter: >>>>>> * - accept connections from everywhere. >>>>>> N.N.N.N - accept connections from this IP address only >>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>> >>>>>> >>>>>> >>>>>> 2. JDK-8052136 JDWP hardening >>>>>> >>>>>> -Dmitry >>>>>> >>>> >> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From serguei.spitsyn at oracle.com Wed May 10 09:37:49 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 10 May 2017 02:37:49 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Message-ID: Dmitry, Thank you a lot for the detailed reply! On 5/10/17 01:10, Dmitry Samersoff wrote: > Serguei, > > Please see my comments in-line. > > > On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >> Hi Dmitry, >> >> >> I'd like to resolve my questions before the upcoming design discussion >> on Thu. >> >> >> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >> >> >> >> (0) The design description from the bug report tells: >> >> > Than we change a negotiation protocol between JDWP and transport. >> > We pass maximal supported version to transport initialization >> routine and expect transport actual version to be returned. >> >> The modified negotiation protocol adds extra complexity. >> What is a motivation behind this? >> Is it really necessary for the transport library to return an actual >> version instead of rejecting the unmatched version? >> I do not see it is really used in the webrev.15 implementation. > Transport have to return it's actual version in order to allow agent > to perform appropriate action. > > see libjdwp/transport.c:526 This requirement adds extra complexity to the rules (transport negotiation protocol). It is not really necessary. The loadTransport() already does a lookup of a version that is accepted (not rejected) by the transport library and can save that version. The transport_startTransport() then should use the version found by the loadTransport(). > Today it's just a selection of proper API call but in a future it might > be too-old-transport-error or deprecation warning or security warning or > something else. I do not understand this reason for adding more complexity. It seems, there should not be any issues in the future with rejecting all unsupported versions by the transport library. However, it will be even more simple if one transport library API could support/accept all possible versions (see my alternate suggestion below). >> (1) The following change in the jdwp transport library will reject >> theJDWPTRANSPORT_VERSION_1_0 as it is below >> the version JDWPTRANSPORT_VERSION_1_1, but will except any version >> above the JDWPTRANSPORT_VERSION_1_1 >> (with providing/returning the actual transport version): >> >> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, >> - jint version, jdwpTransportEnv** result) >> + jint version, void** env) >> { >> - if (version != JDWPTRANSPORT_VERSION_1_0) { >> + if (version < JDWPTRANSPORT_VERSION_1_1) { >> return JNI_EVERSION; >> } >> >> >> Te following change will also prevent supporting the 1_0 version of the >> transport library: >> >> - interface.StartListening = &socketTransport_startListening; >> + interface.StartListening = NULL; > libdt_socket/socketTransport.c is an implementation of 1.1 *transport* > it's not intended to run with 1.0 *backend*. Why not? It would simplifies things if the transport library (and its API) is backward compatible. > i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* > require 1.1 or greater backend. This statement creates a confusion. The truce is that the transport library can support some number of versions. The latest supported version can satisfy the agent if it supports it. > see: libjdwp/transport.c:206 for version logic on backend (agent) side The logic at L206 does not require the transport library to return its version. It will work Ok if the library rejects unsupported versions. >> I'd suggest the following alternate change to the transport API allowing >> to support >> both old and new versions at the same time (it would simplify the >> negotiation rules): >> - Add new function: >> jdwpTransportError AllowPeers(const char* peers); >> >> - Keep the original StartListening function. >> The function uses the allowed peers data if it was previously cached >> by the AllowPeers(). >> >> - It seems, the alternate approach does not require adding the >> extra_data with version. >> But if there is still a real need to get the transport API version >> then it'd better >> to introduce a function named GetTransportVersion() or >> JDWP_TransportVersion(). >> This would allow to encapsulate any extra_data that is necessary in >> such a case. > From pure JDWP hardening point of view we can just add extra parameter > *allow* to existing StartListening(). Caller is responsible for stack > consistency so old transport continues to work. We are not adding extra parameter, we are introducing new function that is a clone of another StartListening function with a version suffix '11' in its name and with an extra parameter. The original StartListening function is being removed. It is much simpler to introduce new function AllowPeers(char* peers) with the same parameter. The original StartListening function works as before. The updated API can support both versions 1_0 and 1_1. > But my goal was to create a versioning in JDWP agent -> transport > relations that was missed in initial JDWP design. Further, more > complicated, changes (IPv6 support, UDS sockets support etc) require > this logic. Would introducing function JdwpTransportVersion() achieve what you wanted? > We have a structure jdwpTransportNativeInterface with a fixed set of > functions (see jdwpTransport.h:153). To add any new function to this > structure we have to create a method to detect presence of this > function. So we can't use GetTransportVersion(). It is not clear why would you need such a method for any new function? The transport version maps to the whole set of functions. > With as separate AllowPeer() function nobody remind an agent writer that > they should use it, but extra parameter makes api changes and > requirements clear visible (up to compiler warning). I do not see any difference. No compiler warning if NULL is passed in place of the 'allow' parameter. > Also I'm against of changing behavior of existing function. > > >> (2) The following error messages miss the actual IP address or mask that >> was found to be illegal: >> >> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip >> address for allow"); 392 >> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for >> allow"); > Other parameter parsing functions (e.g. "invalid port number specified" > at 274) doesn't explain what parameter is bad. It is not good either. But new functionality is added, so the more diagnostic details the better. > I think typical allow would have one or two entry, so verbose error > message is not worth significant complication of parsing code. It is still better to print what was not accepted. It should not be a problem to print it anyway. > I would prefer to leave it as is. > > >> (3) A suggestion on the following: >> >> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask >> >> I'd suggest a more explicit approach instead of the L400 for a better >> clarity: >> >> 386 if (*s == '/') { >> 387 // netmask specified >> 388 s = mask_s2u(s + 1, &mask); >> 389 if (*(s-1) == '/') { >> 390 // Input is not consumed, something bad happens >> 391 _peers_cnt = 0; >> 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask >> for allow"); >> 393 } >> 394 } else { mask = 0xFFFFFFFF; } > I'll try it. Ok, thanks. >> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >> (4) Some confusion with the fragment and its comment: >> >> + >> + /* Pass the latest supported version, >> + * expect actual transport version in t->extra_data->version >> + */ >> + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); >> + if (ver == JNI_EVERSION) { >> ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, &t); >> + // Special handling for versionless transports >> + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; >> + } >> + else { >> + info->transportVersion = (*t)->extra_data->version; >> + } >> + >> >> The term "latest supported version" is ambiguous in this context. Do you >> mean, supported by the JDWP back-end or by the agent library? > Supported by the JDWP backend. I'll update comments. Ok, thanks. >> Also, it >> is not clear in what circumstances the agent library would support the >> version 1_0 only. The JDWP back-end is always coupled with the socket >> transport library, isn't it? Is it because the libdt_shmem library can >> be used on Windows or because a different native library path can be >> used? Could you explain a little bit? > It's more generic question: Do we need any backward compatibility here? > I assume *yes* (can't recall why - probably it was a tree-year old > decision). > > If we state that new backend will not support old transports today and > in a future - all versioning logic is not necessary. > > >> As we see in (1) above the actual >> transport version can be different from requested only if the requested >> version is above the latest supported by the transport library. >> Otherwise, the version is matched or the JNI_EVERSION is returned. The >> subsequent call to the OnLoad function can succeed only if the library >> supports just the version 1_0. > See answer to (1) above. transport library has exactly one version but > JDWP backend supports couple of transport library versions back. I've replied above too. :) >> (5) Memory allocation for the info->allow >> field is needed only for the transport version 1_1: >> >> + if (allow != NULL) { >> + info->allow = jvmtiAllocate((int)strlen(allow)+1); >> + if (info->allow == NULL) { >> + serror = JDWP_ERROR(OUT_OF_MEMORY); >> + goto handleError; >> + } >> + (void)strcpy(info->allow, allow); >> + } > Correct. allocation needed for 1.1 and *greater*. I can change it, but > it makes code less readable. The same fragment in a different place should not be less readable, maybe just less elegant. >> (6) There is no handling for the condition when the *allow* parameter is >> passed but the transport version is 1_0 (which does not support the >> *allow* parameter): > Correct. Warning or ever error should be here. > > I plan to open a separate follow-up CR for this problem - we have to > decide how we should handle this situation (warning or error) and change > the code, > > but I can add a plain printf() here if you like. I'm Ok with both error or warning but what is needed from a security point of view? We can avoid filing a separate follow-up CR in this case. Should it be an error if the *allow* parameter is used with the old transport library that does not support it? Thanks, Serguei >> + /* Interface version 1.0 doesn't support versioning, so we have to >> + * use global variable and set the version artifically. >> + * Use (*t)->extra_data->version directly when 1.0 is gone. >> + */ >> + switch(info->transportVersion) { >> + case JDWPTRANSPORT_VERSION_1_0: >> err = (*trans)->StartListening(trans, address, &retAddress); >> + break; >> + case JDWPTRANSPORT_VERSION_1_1: >> + err = (*trans)->StartListening11(trans, address, &retAddress, >> info->allow); >> + break; >> + default: >> + err = JNI_EVERSION; >> + } > -Dmitry > > >> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>> Robbin, >>> >>>> One follow-up issue is that if you start suspended >>>> and than connect with >>>> an unallowed client the JVM starts and executes the program. >>> Fixed. >>> >>> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>> >>> -Dmitry >>> >>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>> Hi Dmitry, thanks for the update. >>>> >>>> One follow-up issue is that if you start suspended and than connect with >>>> an unallowed client the JVM starts and executes the program. >>>> Simple program prints "Hello". >>>> >>>> [rehn at rehn-ws vanilla-hs]$ java >>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>> >>>> -cp . H >>>> Listening for transport dt_socket at address: 9999 >>>> ERROR: Peer not allowed to connect >>>> Listening for transport dt_socket at address: 9999 >>>> Hello >>>> >>>> I think it would be good if the VM stayed suspended when an unallowed >>>> client connects. >>>> >>>> Thanks, Robbin >>>> >>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>> Robbin, >>>>> >>>>> Please, see: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>> >>>>>> 1: >>>>>> So connecting with an unallowed client terminates the VM. >>>>> Fixed. >>>>> >>>>>> 2: >>>>>> Starting with an bad allow filter terminates the VM when connecting a >>>>>> client. >>>>> Moved allowed parameter (and parser call) to StartListening. >>>>> >>>>> -Dmitry >>>>> >>>>> >>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> I took a look at this, I have two practical issues: >>>>>> >>>>>> 1: >>>>>> [rehn at rehn-ws dev]$ java >>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>> >>>>>> >>>>>> -cp runs ForEver >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>> fatal >>>>>> error [transport.c:358] >>>>>> >>>>>> So connecting with an unallowed client terminates the VM. >>>>>> >>>>>> 2: >>>>>> [rehn at rehn-ws dev]$ java >>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>> >>>>>> >>>>>> -cp runs ForEver >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>>> Success >>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>> fatal >>>>>> error [transport.c:358] >>>>>> >>>>>> Starting with an bad allow filter terminates the VM when connecting a >>>>>> client. >>>>>> >>>>>> >>>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>>> and we >>>>>> should verify allow filter directly at startup. >>>>>> >>>>>> Thanks >>>>>> >>>>>> /Robbin >>>>>> >>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>> Everybody, >>>>>>> >>>>>>> Please review: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>> >>>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>>>> connect to jdwp server and it's the second part of JDWP hardening[2]. >>>>>>> >>>>>>> No restrictions are applied by default now but I'll file a >>>>>>> separate CR >>>>>>> to restrict list of allowed peers to localhost by default. >>>>>>> >>>>>>> Also these changes implement versioning for jdwp transport and >>>>>>> therefor >>>>>>> simplify feature development of jdwp. >>>>>>> >>>>>>> >>>>>>> 1. Example command line: >>>>>>> >>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>> >>>>>>> Possible values for allow parameter: >>>>>>> * - accept connections from everywhere. >>>>>>> N.N.N.N - accept connections from this IP address only >>>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>> >>>>>>> -Dmitry >>>>>>> > From serguei.spitsyn at oracle.com Wed May 10 09:42:56 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 10 May 2017 02:42:56 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Message-ID: <2afdb44e-ced0-acef-c8de-df5d3c57e486@oracle.com> Dmitry, Please, see one correction below. On 5/10/17 02:37, serguei.spitsyn at oracle.com wrote: > Dmitry, > > Thank you a lot for the detailed reply! > > > On 5/10/17 01:10, Dmitry Samersoff wrote: >> Serguei, >> >> Please see my comments in-line. >> >> >> On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >>> Hi Dmitry, >>> >>> >>> I'd like to resolve my questions before the upcoming design discussion >>> on Thu. >>> >>> >>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >>> >>> >>> >>> >>> (0) The design description from the bug report tells: >>> >>> > Than we change a negotiation protocol between JDWP and transport. >>> > We pass maximal supported version to transport initialization >>> routine and expect transport actual version to be returned. >>> >>> The modified negotiation protocol adds extra complexity. >>> What is a motivation behind this? >>> Is it really necessary for the transport library to return an actual >>> version instead of rejecting the unmatched version? >>> I do not see it is really used in the webrev.15 implementation. >> Transport have to return it's actual version in order to allow agent >> to perform appropriate action. >> >> see libjdwp/transport.c:526 > > This requirement adds extra complexity to the rules (transport > negotiation protocol). > It is not really necessary. > The loadTransport() already does a lookup of a version that is accepted > (not rejected) by the transport library and can save that version. > The transport_startTransport() then should use the version found by > the loadTransport(). > > >> Today it's just a selection of proper API call but in a future it might >> be too-old-transport-error or deprecation warning or security warning or >> something else. > > I do not understand this reason for adding more complexity. > It seems, there should not be any issues in the future with rejecting > all unsupported versions by the transport library. > However, it will be even more simple if one transport library API could > support/accept all possible versions (see my alternate suggestion below). > > >>> (1) The following change in the jdwp transport library will reject >>> theJDWPTRANSPORT_VERSION_1_0 as it is below >>> the version JDWPTRANSPORT_VERSION_1_1, but will except any version >>> above the JDWPTRANSPORT_VERSION_1_1 >>> (with providing/returning the actual transport version): >>> >>> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, >>> - jint version, jdwpTransportEnv** result) >>> + jint version, void** env) >>> { >>> - if (version != JDWPTRANSPORT_VERSION_1_0) { >>> + if (version < JDWPTRANSPORT_VERSION_1_1) { >>> return JNI_EVERSION; >>> } >>> >>> >>> Te following change will also prevent supporting the 1_0 version of the >>> transport library: >>> >>> - interface.StartListening = &socketTransport_startListening; >>> + interface.StartListening = NULL; >> libdt_socket/socketTransport.c is an implementation of 1.1 *transport* >> it's not intended to run with 1.0 *backend*. > > Why not? > It would simplifies things if the transport library (and its API) is > backward compatible. > >> i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* >> require 1.1 or greater backend. > > This statement creates a confusion. > The truce is that the transport library can support some number of > versions. > The latest supported version can satisfy the agent if it supports it. > >> see: libjdwp/transport.c:206 for version logic on backend (agent) side > > The logic at L206 does not require the transport library to return its > version. > It will work Ok if the library rejects unsupported versions. > > >>> I'd suggest the following alternate change to the transport API >>> allowing >>> to support >>> both old and new versions at the same time (it would simplify the >>> negotiation rules): >>> - Add new function: >>> jdwpTransportError AllowPeers(const char* peers); >>> >>> - Keep the original StartListening function. >>> The function uses the allowed peers data if it was previously >>> cached >>> by the AllowPeers(). >>> >>> - It seems, the alternate approach does not require adding the >>> extra_data with version. >>> But if there is still a real need to get the transport API version >>> then it'd better >>> to introduce a function named GetTransportVersion() or >>> JDWP_TransportVersion(). >>> This would allow to encapsulate any extra_data that is >>> necessary in >>> such a case. >> From pure JDWP hardening point of view we can just add extra parameter >> *allow* to existing StartListening(). Caller is responsible for stack >> consistency so old transport continues to work. > > We are not adding extra parameter, we are introducing new function > that is a clone of another StartListening function with a version > suffix '11' in its name and with an extra parameter. > The original StartListening function is being removed. A correction: Not removed but nulled out. Do I get it right? Thanks, Serguei > It is much simpler to introduce new function AllowPeers(char* peers) > with the same parameter. > The original StartListening function works as before. > The updated API can support both versions 1_0 and 1_1. > > >> But my goal was to create a versioning in JDWP agent -> transport >> relations that was missed in initial JDWP design. Further, more >> complicated, changes (IPv6 support, UDS sockets support etc) require >> this logic. > > Would introducing function JdwpTransportVersion() achieve what you > wanted? > > >> We have a structure jdwpTransportNativeInterface with a fixed set of >> functions (see jdwpTransport.h:153). To add any new function to this >> structure we have to create a method to detect presence of this >> function. So we can't use GetTransportVersion(). > It is not clear why would you need such a method for any new function? > The transport version maps to the whole set of functions. > > >> With as separate AllowPeer() function nobody remind an agent writer that >> they should use it, but extra parameter makes api changes and >> requirements clear visible (up to compiler warning). > > I do not see any difference. > No compiler warning if NULL is passed in place of the 'allow' parameter. > > >> Also I'm against of changing behavior of existing function. >> >> >>> (2) The following error messages miss the actual IP address or mask >>> that >>> was found to be illegal: >>> >>> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip >>> address for allow"); 392 >>> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for >>> allow"); >> Other parameter parsing functions (e.g. "invalid port number specified" >> at 274) doesn't explain what parameter is bad. > > It is not good either. > But new functionality is added, so the more diagnostic details the > better. > >> I think typical allow would have one or two entry, so verbose error >> message is not worth significant complication of parsing code. > > It is still better to print what was not accepted. > It should not be a problem to print it anyway. > > >> I would prefer to leave it as is. >> >> >>> (3) A suggestion on the following: >>> >>> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask >>> >>> I'd suggest a more explicit approach instead of the L400 for a better >>> clarity: >>> >>> 386 if (*s == '/') { >>> 387 // netmask specified >>> 388 s = mask_s2u(s + 1, &mask); >>> 389 if (*(s-1) == '/') { >>> 390 // Input is not consumed, something bad happens >>> 391 _peers_cnt = 0; >>> 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask >>> for allow"); >>> 393 } >>> 394 } else { mask = 0xFFFFFFFF; } >> I'll try it. > > Ok, thanks. > > >>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >>> >>> (4) Some confusion with the fragment and its comment: >>> >>> + >>> + /* Pass the latest supported version, >>> + * expect actual transport version in t->extra_data->version >>> + */ >>> + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); >>> + if (ver == JNI_EVERSION) { >>> ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, >>> &t); >>> + // Special handling for versionless transports >>> + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; >>> + } >>> + else { >>> + info->transportVersion = (*t)->extra_data->version; >>> + } >>> + >>> >>> The term "latest supported version" is ambiguous in this context. Do >>> you >>> mean, supported by the JDWP back-end or by the agent library? >> Supported by the JDWP backend. I'll update comments. > > Ok, thanks. > > >>> Also, it >>> is not clear in what circumstances the agent library would support the >>> version 1_0 only. The JDWP back-end is always coupled with the socket >>> transport library, isn't it? Is it because the libdt_shmem library can >>> be used on Windows or because a different native library path can be >>> used? Could you explain a little bit? >> It's more generic question: Do we need any backward compatibility here? >> I assume *yes* (can't recall why - probably it was a tree-year old >> decision). >> >> If we state that new backend will not support old transports today and >> in a future - all versioning logic is not necessary. >> >> >>> As we see in (1) above the actual >>> transport version can be different from requested only if the requested >>> version is above the latest supported by the transport library. >>> Otherwise, the version is matched or the JNI_EVERSION is returned. The >>> subsequent call to the OnLoad function can succeed only if the library >>> supports just the version 1_0. >> See answer to (1) above. transport library has exactly one version but >> JDWP backend supports couple of transport library versions back. > > I've replied above too. :) > > >>> (5) Memory allocation for the info->allow >>> field is needed only for the transport version 1_1: >>> >>> + if (allow != NULL) { >>> + info->allow = jvmtiAllocate((int)strlen(allow)+1); >>> + if (info->allow == NULL) { >>> + serror = JDWP_ERROR(OUT_OF_MEMORY); >>> + goto handleError; >>> + } >>> + (void)strcpy(info->allow, allow); >>> + } >> Correct. allocation needed for 1.1 and *greater*. I can change it, but >> it makes code less readable. > > The same fragment in a different place should not be less readable, > maybe just less elegant. > > >>> (6) There is no handling for the condition when the *allow* >>> parameter is >>> passed but the transport version is 1_0 (which does not support the >>> *allow* parameter): >> Correct. Warning or ever error should be here. >> >> I plan to open a separate follow-up CR for this problem - we have to >> decide how we should handle this situation (warning or error) and change >> the code, >> >> but I can add a plain printf() here if you like. > > I'm Ok with both error or warning but what is needed from a security > point of view? > We can avoid filing a separate follow-up CR in this case. > Should it be an error if the *allow* parameter is used with the old > transport library that does not support it? > > > Thanks, > Serguei > > >>> + /* Interface version 1.0 doesn't support versioning, so we have to >>> + * use global variable and set the version artifically. >>> + * Use (*t)->extra_data->version directly when 1.0 is gone. >>> + */ >>> + switch(info->transportVersion) { >>> + case JDWPTRANSPORT_VERSION_1_0: >>> err = (*trans)->StartListening(trans, address, &retAddress); >>> + break; >>> + case JDWPTRANSPORT_VERSION_1_1: >>> + err = (*trans)->StartListening11(trans, address, &retAddress, >>> info->allow); >>> + break; >>> + default: >>> + err = JNI_EVERSION; >>> + } >> -Dmitry >> >> >>> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>>> Robbin, >>>> >>>>> One follow-up issue is that if you start suspended >>>>> and than connect with >>>>> an unallowed client the JVM starts and executes the program. >>>> Fixed. >>>> >>>> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>>> >>>> -Dmitry >>>> >>>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>>> Hi Dmitry, thanks for the update. >>>>> >>>>> One follow-up issue is that if you start suspended and than >>>>> connect with >>>>> an unallowed client the JVM starts and executes the program. >>>>> Simple program prints "Hello". >>>>> >>>>> [rehn at rehn-ws vanilla-hs]$ java >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>>> >>>>> >>>>> -cp . H >>>>> Listening for transport dt_socket at address: 9999 >>>>> ERROR: Peer not allowed to connect >>>>> Listening for transport dt_socket at address: 9999 >>>>> Hello >>>>> >>>>> I think it would be good if the VM stayed suspended when an unallowed >>>>> client connects. >>>>> >>>>> Thanks, Robbin >>>>> >>>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>>> Robbin, >>>>>> >>>>>> Please, see: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>>> >>>>>>> 1: >>>>>>> So connecting with an unallowed client terminates the VM. >>>>>> Fixed. >>>>>> >>>>>>> 2: >>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>> connecting a >>>>>>> client. >>>>>> Moved allowed parameter (and parser call) to StartListening. >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> >>>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>>> Hi Dmitry, >>>>>>> >>>>>>> I took a look at this, I have two practical issues: >>>>>>> >>>>>>> 1: >>>>>>> [rehn at rehn-ws dev]$ java >>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -cp runs ForEver >>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>> fatal >>>>>>> error [transport.c:358] >>>>>>> >>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>> >>>>>>> 2: >>>>>>> [rehn at rehn-ws dev]$ java >>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -cp runs ForEver >>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>>>> Success >>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>> fatal >>>>>>> error [transport.c:358] >>>>>>> >>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>> connecting a >>>>>>> client. >>>>>>> >>>>>>> >>>>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>>>> and we >>>>>>> should verify allow filter directly at startup. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> /Robbin >>>>>>> >>>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>>> Everybody, >>>>>>>> >>>>>>>> Please review: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>>> >>>>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>>>> allow. Users can explicitly specify a list of hosts that >>>>>>>> allowed to >>>>>>>> connect to jdwp server and it's the second part of JDWP >>>>>>>> hardening[2]. >>>>>>>> >>>>>>>> No restrictions are applied by default now but I'll file a >>>>>>>> separate CR >>>>>>>> to restrict list of allowed peers to localhost by default. >>>>>>>> >>>>>>>> Also these changes implement versioning for jdwp transport and >>>>>>>> therefor >>>>>>>> simplify feature development of jdwp. >>>>>>>> >>>>>>>> >>>>>>>> 1. Example command line: >>>>>>>> >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>>> >>>>>>>> Possible values for allow parameter: >>>>>>>> * - accept connections from everywhere. >>>>>>>> N.N.N.N - accept connections from this IP address only >>>>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >> > From david.holmes at oracle.com Wed May 10 09:55:36 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 10 May 2017 19:55:36 +1000 Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> Message-ID: <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> Hi Fairoz, On 10/05/2017 5:53 PM, Fairoz Matte wrote: > Hi, > > Kindly review the small typo fix, applicable only for JDK8 > BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 > Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ I think "for type type" was intended to be "for the type". David > Thanks, > Fairoz > From fairoz.matte at oracle.com Wed May 10 11:30:22 2017 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Wed, 10 May 2017 04:30:22 -0700 (PDT) Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> Message-ID: Hi David, > -----Original Message----- > From: David Holmes > Sent: Wednesday, May 10, 2017 3:26 PM > To: Fairoz Matte ; serviceability- > dev at openjdk.java.net > Subject: Re: RFR: JDK-8173664: Typo in https://java.net/downloads/heap- > snapshot/hprof-binary-format.html > > Hi Fairoz, > > On 10/05/2017 5:53 PM, Fairoz Matte wrote: > > Hi, > > > > Kindly review the small typo fix, applicable only for JDK8 > > BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 > > Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ > > I think "for type type" was intended to be "for the type". > Yes it does look like after reading multiple times. Thanks for the review I will close as Not an issue Thanks, Fairoz > David > > > Thanks, > > Fairoz > > From dmitry.samersoff at oracle.com Wed May 10 15:27:20 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 10 May 2017 18:27:20 +0300 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Message-ID: Serguei, Fixed minor issues (comments, netmask etc). Added an error for attempt to use allow with an old transport. http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.17/ see also below. > I do not understand this reason for adding more complexity. > It seems, there should not be any issues in the future with rejecting > all unsupported versions by the transport library. Added a diagram explaining transport version negotiation to CR. I use future versions (1.3; 1.4; 1.5) because all this versioning staff has a long term goal and allow us to develop better transport without breaking existing one. > We are not adding extra parameter, we are introducing new function > that is a clone of another StartListening function with a version > suffix '11' in its name and with an extra parameter. Correct. We changed behavior of StartListening function and 1.1 transport shouldn't care about old one. i.e. when we document 1.1 interface we describe the only function StartListening(env, address, actualAddress, allow) that have to be placed to the StartListening11 slot. Back in 2015 I proposed to separate interfaces entirely (see webrev.04), but we (Alan?) decided that it's an overkill. > The original StartListening function is being removed. > It is much simpler to introduce new function AllowPeers(char* peers) > with the same parameter. This separate function have to be called explicitly before we start listening, It is extra communication step. IMHO, not obvious one. So I would prefer to keep StartListening11 -Dmitry On 2017-05-10 12:37, serguei.spitsyn at oracle.com wrote: > Dmitry, > > Thank you a lot for the detailed reply! > > > On 5/10/17 01:10, Dmitry Samersoff wrote: >> Serguei, >> >> Please see my comments in-line. >> >> >> On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >>> Hi Dmitry, >>> >>> >>> I'd like to resolve my questions before the upcoming design discussion >>> on Thu. >>> >>> >>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >>> >>> >>> >>> >>> (0) The design description from the bug report tells: >>> >>> > Than we change a negotiation protocol between JDWP and transport. >>> > We pass maximal supported version to transport initialization >>> routine and expect transport actual version to be returned. >>> >>> The modified negotiation protocol adds extra complexity. >>> What is a motivation behind this? >>> Is it really necessary for the transport library to return an actual >>> version instead of rejecting the unmatched version? >>> I do not see it is really used in the webrev.15 implementation. >> Transport have to return it's actual version in order to allow agent >> to perform appropriate action. >> >> see libjdwp/transport.c:526 > > This requirement adds extra complexity to the rules (transport > negotiation protocol). > It is not really necessary. > The loadTransport() already does a lookup of a version that is accepted > (not rejected) by the transport library and can save that version. > The transport_startTransport() then should use the version found by the > loadTransport(). > > >> Today it's just a selection of proper API call but in a future it might >> be too-old-transport-error or deprecation warning or security warning or >> something else. > > I do not understand this reason for adding more complexity. > It seems, there should not be any issues in the future with rejecting > all unsupported versions by the transport library. > However, it will be even more simple if one transport library API could > support/accept all possible versions (see my alternate suggestion below). > > >>> (1) The following change in the jdwp transport library will reject >>> theJDWPTRANSPORT_VERSION_1_0 as it is below >>> the version JDWPTRANSPORT_VERSION_1_1, but will except any version >>> above the JDWPTRANSPORT_VERSION_1_1 >>> (with providing/returning the actual transport version): >>> >>> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, >>> - jint version, jdwpTransportEnv** result) >>> + jint version, void** env) >>> { >>> - if (version != JDWPTRANSPORT_VERSION_1_0) { >>> + if (version < JDWPTRANSPORT_VERSION_1_1) { >>> return JNI_EVERSION; >>> } >>> >>> >>> Te following change will also prevent supporting the 1_0 version of the >>> transport library: >>> >>> - interface.StartListening = &socketTransport_startListening; >>> + interface.StartListening = NULL; >> libdt_socket/socketTransport.c is an implementation of 1.1 *transport* >> it's not intended to run with 1.0 *backend*. > > Why not? > It would simplifies things if the transport library (and its API) is > backward compatible. > >> i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* >> require 1.1 or greater backend. > > This statement creates a confusion. > The truce is that the transport library can support some number of > versions. > The latest supported version can satisfy the agent if it supports it. > >> see: libjdwp/transport.c:206 for version logic on backend (agent) side > > The logic at L206 does not require the transport library to return its > version. > It will work Ok if the library rejects unsupported versions. > > >>> I'd suggest the following alternate change to the transport API allowing >>> to support >>> both old and new versions at the same time (it would simplify the >>> negotiation rules): >>> - Add new function: >>> jdwpTransportError AllowPeers(const char* peers); >>> >>> - Keep the original StartListening function. >>> The function uses the allowed peers data if it was previously >>> cached >>> by the AllowPeers(). >>> >>> - It seems, the alternate approach does not require adding the >>> extra_data with version. >>> But if there is still a real need to get the transport API version >>> then it'd better >>> to introduce a function named GetTransportVersion() or >>> JDWP_TransportVersion(). >>> This would allow to encapsulate any extra_data that is necessary in >>> such a case. >> From pure JDWP hardening point of view we can just add extra parameter >> *allow* to existing StartListening(). Caller is responsible for stack >> consistency so old transport continues to work. > > We are not adding extra parameter, we are introducing new function > that is a clone of another StartListening function with a version > suffix '11' in its name and with an extra parameter. > The original StartListening function is being removed. > It is much simpler to introduce new function AllowPeers(char* peers) > with the same parameter. > The original StartListening function works as before. > The updated API can support both versions 1_0 and 1_1. > > >> But my goal was to create a versioning in JDWP agent -> transport >> relations that was missed in initial JDWP design. Further, more >> complicated, changes (IPv6 support, UDS sockets support etc) require >> this logic. > > Would introducing function JdwpTransportVersion() achieve what you wanted? > > >> We have a structure jdwpTransportNativeInterface with a fixed set of >> functions (see jdwpTransport.h:153). To add any new function to this >> structure we have to create a method to detect presence of this >> function. So we can't use GetTransportVersion(). > It is not clear why would you need such a method for any new function? > The transport version maps to the whole set of functions. > > >> With as separate AllowPeer() function nobody remind an agent writer that >> they should use it, but extra parameter makes api changes and >> requirements clear visible (up to compiler warning). > > I do not see any difference. > No compiler warning if NULL is passed in place of the 'allow' parameter. > > >> Also I'm against of changing behavior of existing function. >> >> >>> (2) The following error messages miss the actual IP address or mask that >>> was found to be illegal: >>> >>> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip >>> address for allow"); 392 >>> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for >>> allow"); >> Other parameter parsing functions (e.g. "invalid port number specified" >> at 274) doesn't explain what parameter is bad. > > It is not good either. > But new functionality is added, so the more diagnostic details the better. > >> I think typical allow would have one or two entry, so verbose error >> message is not worth significant complication of parsing code. > > It is still better to print what was not accepted. > It should not be a problem to print it anyway. > > >> I would prefer to leave it as is. >> >> >>> (3) A suggestion on the following: >>> >>> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask >>> >>> I'd suggest a more explicit approach instead of the L400 for a better >>> clarity: >>> >>> 386 if (*s == '/') { >>> 387 // netmask specified >>> 388 s = mask_s2u(s + 1, &mask); >>> 389 if (*(s-1) == '/') { >>> 390 // Input is not consumed, something bad happens >>> 391 _peers_cnt = 0; >>> 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask >>> for allow"); >>> 393 } >>> 394 } else { mask = 0xFFFFFFFF; } >> I'll try it. > > Ok, thanks. > > >>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >>> >>> (4) Some confusion with the fragment and its comment: >>> >>> + >>> + /* Pass the latest supported version, >>> + * expect actual transport version in t->extra_data->version >>> + */ >>> + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); >>> + if (ver == JNI_EVERSION) { >>> ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, >>> &t); >>> + // Special handling for versionless transports >>> + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; >>> + } >>> + else { >>> + info->transportVersion = (*t)->extra_data->version; >>> + } >>> + >>> >>> The term "latest supported version" is ambiguous in this context. Do you >>> mean, supported by the JDWP back-end or by the agent library? >> Supported by the JDWP backend. I'll update comments. > > Ok, thanks. > > >>> Also, it >>> is not clear in what circumstances the agent library would support the >>> version 1_0 only. The JDWP back-end is always coupled with the socket >>> transport library, isn't it? Is it because the libdt_shmem library can >>> be used on Windows or because a different native library path can be >>> used? Could you explain a little bit? >> It's more generic question: Do we need any backward compatibility here? >> I assume *yes* (can't recall why - probably it was a tree-year old >> decision). >> >> If we state that new backend will not support old transports today and >> in a future - all versioning logic is not necessary. >> >> >>> As we see in (1) above the actual >>> transport version can be different from requested only if the requested >>> version is above the latest supported by the transport library. >>> Otherwise, the version is matched or the JNI_EVERSION is returned. The >>> subsequent call to the OnLoad function can succeed only if the library >>> supports just the version 1_0. >> See answer to (1) above. transport library has exactly one version but >> JDWP backend supports couple of transport library versions back. > > I've replied above too. :) > > >>> (5) Memory allocation for the info->allow >>> field is needed only for the transport version 1_1: >>> >>> + if (allow != NULL) { >>> + info->allow = jvmtiAllocate((int)strlen(allow)+1); >>> + if (info->allow == NULL) { >>> + serror = JDWP_ERROR(OUT_OF_MEMORY); >>> + goto handleError; >>> + } >>> + (void)strcpy(info->allow, allow); >>> + } >> Correct. allocation needed for 1.1 and *greater*. I can change it, but >> it makes code less readable. > > The same fragment in a different place should not be less readable, > maybe just less elegant. > > >>> (6) There is no handling for the condition when the *allow* parameter is >>> passed but the transport version is 1_0 (which does not support the >>> *allow* parameter): >> Correct. Warning or ever error should be here. >> >> I plan to open a separate follow-up CR for this problem - we have to >> decide how we should handle this situation (warning or error) and change >> the code, >> >> but I can add a plain printf() here if you like. > > I'm Ok with both error or warning but what is needed from a security > point of view? > We can avoid filing a separate follow-up CR in this case. > Should it be an error if the *allow* parameter is used with the old > transport library that does not support it? > > > Thanks, > Serguei > > >>> + /* Interface version 1.0 doesn't support versioning, so we have to >>> + * use global variable and set the version artifically. >>> + * Use (*t)->extra_data->version directly when 1.0 is gone. >>> + */ >>> + switch(info->transportVersion) { >>> + case JDWPTRANSPORT_VERSION_1_0: >>> err = (*trans)->StartListening(trans, address, &retAddress); >>> + break; >>> + case JDWPTRANSPORT_VERSION_1_1: >>> + err = (*trans)->StartListening11(trans, address, &retAddress, >>> info->allow); >>> + break; >>> + default: >>> + err = JNI_EVERSION; >>> + } >> -Dmitry >> >> >>> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>>> Robbin, >>>> >>>>> One follow-up issue is that if you start suspended >>>>> and than connect with >>>>> an unallowed client the JVM starts and executes the program. >>>> Fixed. >>>> >>>> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>>> >>>> -Dmitry >>>> >>>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>>> Hi Dmitry, thanks for the update. >>>>> >>>>> One follow-up issue is that if you start suspended and than connect >>>>> with >>>>> an unallowed client the JVM starts and executes the program. >>>>> Simple program prints "Hello". >>>>> >>>>> [rehn at rehn-ws vanilla-hs]$ java >>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>>> >>>>> >>>>> -cp . H >>>>> Listening for transport dt_socket at address: 9999 >>>>> ERROR: Peer not allowed to connect >>>>> Listening for transport dt_socket at address: 9999 >>>>> Hello >>>>> >>>>> I think it would be good if the VM stayed suspended when an unallowed >>>>> client connects. >>>>> >>>>> Thanks, Robbin >>>>> >>>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>>> Robbin, >>>>>> >>>>>> Please, see: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>>> >>>>>>> 1: >>>>>>> So connecting with an unallowed client terminates the VM. >>>>>> Fixed. >>>>>> >>>>>>> 2: >>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>> connecting a >>>>>>> client. >>>>>> Moved allowed parameter (and parser call) to StartListening. >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> >>>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>>> Hi Dmitry, >>>>>>> >>>>>>> I took a look at this, I have two practical issues: >>>>>>> >>>>>>> 1: >>>>>>> [rehn at rehn-ws dev]$ java >>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -cp runs ForEver >>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>> fatal >>>>>>> error [transport.c:358] >>>>>>> >>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>> >>>>>>> 2: >>>>>>> [rehn at rehn-ws dev]$ java >>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -cp runs ForEver >>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>>>> Success >>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>> fatal >>>>>>> error [transport.c:358] >>>>>>> >>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>> connecting a >>>>>>> client. >>>>>>> >>>>>>> >>>>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>>>> and we >>>>>>> should verify allow filter directly at startup. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> /Robbin >>>>>>> >>>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>>> Everybody, >>>>>>>> >>>>>>>> Please review: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>>> >>>>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>>>>> connect to jdwp server and it's the second part of JDWP >>>>>>>> hardening[2]. >>>>>>>> >>>>>>>> No restrictions are applied by default now but I'll file a >>>>>>>> separate CR >>>>>>>> to restrict list of allowed peers to localhost by default. >>>>>>>> >>>>>>>> Also these changes implement versioning for jdwp transport and >>>>>>>> therefor >>>>>>>> simplify feature development of jdwp. >>>>>>>> >>>>>>>> >>>>>>>> 1. Example command line: >>>>>>>> >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>>> >>>>>>>> Possible values for allow parameter: >>>>>>>> * - accept connections from everywhere. >>>>>>>> N.N.N.N - accept connections from this IP address only >>>>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>>> >>>>>>>> -Dmitry >>>>>>>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From kumar.x.srinivasan at oracle.com Wed May 10 18:35:41 2017 From: kumar.x.srinivasan at oracle.com (Kumar Srinivasan) Date: Wed, 10 May 2017 11:35:41 -0700 Subject: RFR: JDK-8179631: Fix Html5 errors in java.management, jdk.management, jdk.jdi and jdk.attach Message-ID: <59135D7D.3010704@oracle.com> Hi All, Please review fixes to make the API doc comments HTML5 clean, there are no changes to the verbiage, and mostly fixes for the table styles defined here: http://hg.openjdk.java.net/jdk9/dev/langtools/rev/ee84b7d44339 For tables with many entries I have used "striped", there are few tables used for formatting purposes, for which "borderless" is used, and "plain" for everything else. Thanks Kumar Webrev: http://cr.openjdk.java.net/~ksrini/8179631/webrev.0/ JBS: https://bugs.openjdk.java.net/browse/JDK-8179631 From david.holmes at oracle.com Thu May 11 02:22:03 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 11 May 2017 12:22:03 +1000 Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> Message-ID: <32b0c37a-e7a8-3f5e-c154-95587d2e9999@oracle.com> On 10/05/2017 9:30 PM, Fairoz Matte wrote: > Hi David, > >> -----Original Message----- >> From: David Holmes >> Sent: Wednesday, May 10, 2017 3:26 PM >> To: Fairoz Matte ; serviceability- >> dev at openjdk.java.net >> Subject: Re: RFR: JDK-8173664: Typo in https://java.net/downloads/heap- >> snapshot/hprof-binary-format.html >> >> Hi Fairoz, >> >> On 10/05/2017 5:53 PM, Fairoz Matte wrote: >>> Hi, >>> >>> Kindly review the small typo fix, applicable only for JDK8 >>> BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 >>> Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ >> >> I think "for type type" was intended to be "for the type". >> > Yes it does look like after reading multiple times. > Thanks for the review I will close as Not an issue ??? It is still a typo that can be fixed. David > Thanks, > Fairoz > >> David >> >>> Thanks, >>> Fairoz >>> From fairoz.matte at oracle.com Thu May 11 03:34:50 2017 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Wed, 10 May 2017 20:34:50 -0700 (PDT) Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: <32b0c37a-e7a8-3f5e-c154-95587d2e9999@oracle.com> References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> <32b0c37a-e7a8-3f5e-c154-95587d2e9999@oracle.com> Message-ID: <916f10b9-a247-474b-9726-366673336853@default> Hi David, After having discussion with you over IM things are clear Expected change is - "There will be a "LOAD CLASS" tag for the type of each array in the dump" and current changeset missed "the" before type. I will make the changes and send it again. Thanks, Fairoz > -----Original Message----- > From: David Holmes > Sent: Thursday, May 11, 2017 7:52 AM > To: Fairoz Matte ; serviceability- > dev at openjdk.java.net > Subject: Re: RFR: JDK-8173664: Typo in https://java.net/downloads/heap- > snapshot/hprof-binary-format.html > > On 10/05/2017 9:30 PM, Fairoz Matte wrote: > > Hi David, > > > >> -----Original Message----- > >> From: David Holmes > >> Sent: Wednesday, May 10, 2017 3:26 PM > >> To: Fairoz Matte ; serviceability- > >> dev at openjdk.java.net > >> Subject: Re: RFR: JDK-8173664: Typo in > >> https://java.net/downloads/heap- snapshot/hprof-binary-format.html > >> > >> Hi Fairoz, > >> > >> On 10/05/2017 5:53 PM, Fairoz Matte wrote: > >>> Hi, > >>> > >>> Kindly review the small typo fix, applicable only for JDK8 > >>> BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 > >>> Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ > >> > >> I think "for type type" was intended to be "for the type". > >> > > Yes it does look like after reading multiple times. > > Thanks for the review I will close as Not an issue > > ??? It is still a typo that can be fixed. > > David > > > Thanks, > > Fairoz > > > >> David > >> > >>> Thanks, > >>> Fairoz > >>> From robbin.ehn at oracle.com Thu May 11 09:01:39 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 11 May 2017 11:01:39 +0200 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Message-ID: <5fbd99af-6607-d838-2aad-a379559b1fc4@oracle.com> Hi, I find both your approaches acceptable regarding the version and StartListening11 vs AllowPeers. Personally I prefer not using version instead using sizeof as syscall does. E.g. http://man7.org/linux/man-pages/man2/bind.2.html But obviously this also require a new StartListeningXX method. /Robbin On 05/10/2017 05:27 PM, Dmitry Samersoff wrote: > Serguei, > > Fixed minor issues (comments, netmask etc). > Added an error for attempt to use allow with an old transport. > > http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.17/ > > see also below. > > >> I do not understand this reason for adding more complexity. >> It seems, there should not be any issues in the future with rejecting >> all unsupported versions by the transport library. > > Added a diagram explaining transport version negotiation to CR. I use > future versions (1.3; 1.4; 1.5) because all this versioning staff has a > long term goal and allow us to develop better transport without breaking > existing one. > >> We are not adding extra parameter, we are introducing new function >> that is a clone of another StartListening function with a version >> suffix '11' in its name and with an extra parameter. > > Correct. We changed behavior of StartListening function and 1.1 > transport shouldn't care about old one. > > i.e. when we document 1.1 interface we describe the only function > StartListening(env, address, actualAddress, allow) > that have to be placed to the StartListening11 slot. > > > Back in 2015 I proposed to separate interfaces entirely (see webrev.04), > but we (Alan?) decided that it's an overkill. > >> The original StartListening function is being removed. >> It is much simpler to introduce new function AllowPeers(char* peers) >> with the same parameter. > > This separate function have to be called explicitly before we start > listening, It is extra communication step. IMHO, not obvious one. > > So I would prefer to keep StartListening11 > > -Dmitry > > > On 2017-05-10 12:37, serguei.spitsyn at oracle.com wrote: >> Dmitry, >> >> Thank you a lot for the detailed reply! >> >> >> On 5/10/17 01:10, Dmitry Samersoff wrote: >>> Serguei, >>> >>> Please see my comments in-line. >>> >>> >>> On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >>>> Hi Dmitry, >>>> >>>> >>>> I'd like to resolve my questions before the upcoming design discussion >>>> on Thu. >>>> >>>> >>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >>>> >>>> >>>> >>>> >>>> (0) The design description from the bug report tells: >>>> >>>> > Than we change a negotiation protocol between JDWP and transport. >>>> > We pass maximal supported version to transport initialization >>>> routine and expect transport actual version to be returned. >>>> >>>> The modified negotiation protocol adds extra complexity. >>>> What is a motivation behind this? >>>> Is it really necessary for the transport library to return an actual >>>> version instead of rejecting the unmatched version? >>>> I do not see it is really used in the webrev.15 implementation. >>> Transport have to return it's actual version in order to allow agent >>> to perform appropriate action. >>> >>> see libjdwp/transport.c:526 >> >> This requirement adds extra complexity to the rules (transport >> negotiation protocol). >> It is not really necessary. >> The loadTransport() already does a lookup of a version that is accepted >> (not rejected) by the transport library and can save that version. >> The transport_startTransport() then should use the version found by the >> loadTransport(). >> >> >>> Today it's just a selection of proper API call but in a future it might >>> be too-old-transport-error or deprecation warning or security warning or >>> something else. >> >> I do not understand this reason for adding more complexity. >> It seems, there should not be any issues in the future with rejecting >> all unsupported versions by the transport library. >> However, it will be even more simple if one transport library API could >> support/accept all possible versions (see my alternate suggestion below). >> >> >>>> (1) The following change in the jdwp transport library will reject >>>> theJDWPTRANSPORT_VERSION_1_0 as it is below >>>> the version JDWPTRANSPORT_VERSION_1_1, but will except any version >>>> above the JDWPTRANSPORT_VERSION_1_1 >>>> (with providing/returning the actual transport version): >>>> >>>> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, >>>> - jint version, jdwpTransportEnv** result) >>>> + jint version, void** env) >>>> { >>>> - if (version != JDWPTRANSPORT_VERSION_1_0) { >>>> + if (version < JDWPTRANSPORT_VERSION_1_1) { >>>> return JNI_EVERSION; >>>> } >>>> >>>> >>>> Te following change will also prevent supporting the 1_0 version of the >>>> transport library: >>>> >>>> - interface.StartListening = &socketTransport_startListening; >>>> + interface.StartListening = NULL; >>> libdt_socket/socketTransport.c is an implementation of 1.1 *transport* >>> it's not intended to run with 1.0 *backend*. >> >> Why not? >> It would simplifies things if the transport library (and its API) is >> backward compatible. >> >>> i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* >>> require 1.1 or greater backend. >> >> This statement creates a confusion. >> The truce is that the transport library can support some number of >> versions. >> The latest supported version can satisfy the agent if it supports it. >> >>> see: libjdwp/transport.c:206 for version logic on backend (agent) side >> >> The logic at L206 does not require the transport library to return its >> version. >> It will work Ok if the library rejects unsupported versions. >> >> >>>> I'd suggest the following alternate change to the transport API allowing >>>> to support >>>> both old and new versions at the same time (it would simplify the >>>> negotiation rules): >>>> - Add new function: >>>> jdwpTransportError AllowPeers(const char* peers); >>>> >>>> - Keep the original StartListening function. >>>> The function uses the allowed peers data if it was previously >>>> cached >>>> by the AllowPeers(). >>>> >>>> - It seems, the alternate approach does not require adding the >>>> extra_data with version. >>>> But if there is still a real need to get the transport API version >>>> then it'd better >>>> to introduce a function named GetTransportVersion() or >>>> JDWP_TransportVersion(). >>>> This would allow to encapsulate any extra_data that is necessary in >>>> such a case. >>> From pure JDWP hardening point of view we can just add extra parameter >>> *allow* to existing StartListening(). Caller is responsible for stack >>> consistency so old transport continues to work. >> >> We are not adding extra parameter, we are introducing new function >> that is a clone of another StartListening function with a version >> suffix '11' in its name and with an extra parameter. >> The original StartListening function is being removed. >> It is much simpler to introduce new function AllowPeers(char* peers) >> with the same parameter. >> The original StartListening function works as before. >> The updated API can support both versions 1_0 and 1_1. >> >> >>> But my goal was to create a versioning in JDWP agent -> transport >>> relations that was missed in initial JDWP design. Further, more >>> complicated, changes (IPv6 support, UDS sockets support etc) require >>> this logic. >> >> Would introducing function JdwpTransportVersion() achieve what you wanted? >> >> >>> We have a structure jdwpTransportNativeInterface with a fixed set of >>> functions (see jdwpTransport.h:153). To add any new function to this >>> structure we have to create a method to detect presence of this >>> function. So we can't use GetTransportVersion(). >> It is not clear why would you need such a method for any new function? >> The transport version maps to the whole set of functions. >> >> >>> With as separate AllowPeer() function nobody remind an agent writer that >>> they should use it, but extra parameter makes api changes and >>> requirements clear visible (up to compiler warning). >> >> I do not see any difference. >> No compiler warning if NULL is passed in place of the 'allow' parameter. >> >> >>> Also I'm against of changing behavior of existing function. >>> >>> >>>> (2) The following error messages miss the actual IP address or mask that >>>> was found to be illegal: >>>> >>>> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip >>>> address for allow"); 392 >>>> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for >>>> allow"); >>> Other parameter parsing functions (e.g. "invalid port number specified" >>> at 274) doesn't explain what parameter is bad. >> >> It is not good either. >> But new functionality is added, so the more diagnostic details the better. >> >>> I think typical allow would have one or two entry, so verbose error >>> message is not worth significant complication of parsing code. >> >> It is still better to print what was not accepted. >> It should not be a problem to print it anyway. >> >> >>> I would prefer to leave it as is. >>> >>> >>>> (3) A suggestion on the following: >>>> >>>> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask >>>> >>>> I'd suggest a more explicit approach instead of the L400 for a better >>>> clarity: >>>> >>>> 386 if (*s == '/') { >>>> 387 // netmask specified >>>> 388 s = mask_s2u(s + 1, &mask); >>>> 389 if (*(s-1) == '/') { >>>> 390 // Input is not consumed, something bad happens >>>> 391 _peers_cnt = 0; >>>> 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask >>>> for allow"); >>>> 393 } >>>> 394 } else { mask = 0xFFFFFFFF; } >>> I'll try it. >> >> Ok, thanks. >> >> >>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >>>> >>>> (4) Some confusion with the fragment and its comment: >>>> >>>> + >>>> + /* Pass the latest supported version, >>>> + * expect actual transport version in t->extra_data->version >>>> + */ >>>> + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); >>>> + if (ver == JNI_EVERSION) { >>>> ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, >>>> &t); >>>> + // Special handling for versionless transports >>>> + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; >>>> + } >>>> + else { >>>> + info->transportVersion = (*t)->extra_data->version; >>>> + } >>>> + >>>> >>>> The term "latest supported version" is ambiguous in this context. Do you >>>> mean, supported by the JDWP back-end or by the agent library? >>> Supported by the JDWP backend. I'll update comments. >> >> Ok, thanks. >> >> >>>> Also, it >>>> is not clear in what circumstances the agent library would support the >>>> version 1_0 only. The JDWP back-end is always coupled with the socket >>>> transport library, isn't it? Is it because the libdt_shmem library can >>>> be used on Windows or because a different native library path can be >>>> used? Could you explain a little bit? >>> It's more generic question: Do we need any backward compatibility here? >>> I assume *yes* (can't recall why - probably it was a tree-year old >>> decision). >>> >>> If we state that new backend will not support old transports today and >>> in a future - all versioning logic is not necessary. >>> >>> >>>> As we see in (1) above the actual >>>> transport version can be different from requested only if the requested >>>> version is above the latest supported by the transport library. >>>> Otherwise, the version is matched or the JNI_EVERSION is returned. The >>>> subsequent call to the OnLoad function can succeed only if the library >>>> supports just the version 1_0. >>> See answer to (1) above. transport library has exactly one version but >>> JDWP backend supports couple of transport library versions back. >> >> I've replied above too. :) >> >> >>>> (5) Memory allocation for the info->allow >>>> field is needed only for the transport version 1_1: >>>> >>>> + if (allow != NULL) { >>>> + info->allow = jvmtiAllocate((int)strlen(allow)+1); >>>> + if (info->allow == NULL) { >>>> + serror = JDWP_ERROR(OUT_OF_MEMORY); >>>> + goto handleError; >>>> + } >>>> + (void)strcpy(info->allow, allow); >>>> + } >>> Correct. allocation needed for 1.1 and *greater*. I can change it, but >>> it makes code less readable. >> >> The same fragment in a different place should not be less readable, >> maybe just less elegant. >> >> >>>> (6) There is no handling for the condition when the *allow* parameter is >>>> passed but the transport version is 1_0 (which does not support the >>>> *allow* parameter): >>> Correct. Warning or ever error should be here. >>> >>> I plan to open a separate follow-up CR for this problem - we have to >>> decide how we should handle this situation (warning or error) and change >>> the code, >>> >>> but I can add a plain printf() here if you like. >> >> I'm Ok with both error or warning but what is needed from a security >> point of view? >> We can avoid filing a separate follow-up CR in this case. >> Should it be an error if the *allow* parameter is used with the old >> transport library that does not support it? >> >> >> Thanks, >> Serguei >> >> >>>> + /* Interface version 1.0 doesn't support versioning, so we have to >>>> + * use global variable and set the version artifically. >>>> + * Use (*t)->extra_data->version directly when 1.0 is gone. >>>> + */ >>>> + switch(info->transportVersion) { >>>> + case JDWPTRANSPORT_VERSION_1_0: >>>> err = (*trans)->StartListening(trans, address, &retAddress); >>>> + break; >>>> + case JDWPTRANSPORT_VERSION_1_1: >>>> + err = (*trans)->StartListening11(trans, address, &retAddress, >>>> info->allow); >>>> + break; >>>> + default: >>>> + err = JNI_EVERSION; >>>> + } >>> -Dmitry >>> >>> >>>> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>>>> Robbin, >>>>> >>>>>> One follow-up issue is that if you start suspended >>>>>> and than connect with >>>>>> an unallowed client the JVM starts and executes the program. >>>>> Fixed. >>>>> >>>>> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>>>> >>>>> -Dmitry >>>>> >>>>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>>>> Hi Dmitry, thanks for the update. >>>>>> >>>>>> One follow-up issue is that if you start suspended and than connect >>>>>> with >>>>>> an unallowed client the JVM starts and executes the program. >>>>>> Simple program prints "Hello". >>>>>> >>>>>> [rehn at rehn-ws vanilla-hs]$ java >>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>>>> >>>>>> >>>>>> -cp . H >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> ERROR: Peer not allowed to connect >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> Hello >>>>>> >>>>>> I think it would be good if the VM stayed suspended when an unallowed >>>>>> client connects. >>>>>> >>>>>> Thanks, Robbin >>>>>> >>>>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>>>> Robbin, >>>>>>> >>>>>>> Please, see: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>>>> >>>>>>>> 1: >>>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>> Fixed. >>>>>>> >>>>>>>> 2: >>>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>>> connecting a >>>>>>>> client. >>>>>>> Moved allowed parameter (and parser call) to StartListening. >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> >>>>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> I took a look at this, I have two practical issues: >>>>>>>> >>>>>>>> 1: >>>>>>>> [rehn at rehn-ws dev]$ java >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -cp runs ForEver >>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>> fatal >>>>>>>> error [transport.c:358] >>>>>>>> >>>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>>> >>>>>>>> 2: >>>>>>>> [rehn at rehn-ws dev]$ java >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -cp runs ForEver >>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>>>>> Success >>>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>> fatal >>>>>>>> error [transport.c:358] >>>>>>>> >>>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>>> connecting a >>>>>>>> client. >>>>>>>> >>>>>>>> >>>>>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>>>>> and we >>>>>>>> should verify allow filter directly at startup. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> /Robbin >>>>>>>> >>>>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>>>> Everybody, >>>>>>>>> >>>>>>>>> Please review: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>>>> >>>>>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>>>>>> connect to jdwp server and it's the second part of JDWP >>>>>>>>> hardening[2]. >>>>>>>>> >>>>>>>>> No restrictions are applied by default now but I'll file a >>>>>>>>> separate CR >>>>>>>>> to restrict list of allowed peers to localhost by default. >>>>>>>>> >>>>>>>>> Also these changes implement versioning for jdwp transport and >>>>>>>>> therefor >>>>>>>>> simplify feature development of jdwp. >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. Example command line: >>>>>>>>> >>>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>>>> >>>>>>>>> Possible values for allow parameter: >>>>>>>>> * - accept connections from everywhere. >>>>>>>>> N.N.N.N - accept connections from this IP address only >>>>>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>> >> > > From fairoz.matte at oracle.com Thu May 11 09:33:08 2017 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Thu, 11 May 2017 02:33:08 -0700 (PDT) Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: <916f10b9-a247-474b-9726-366673336853@default> References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> <32b0c37a-e7a8-3f5e-c154-95587d2e9999@oracle.com> <916f10b9-a247-474b-9726-366673336853@default> Message-ID: <65cc8059-55ea-4d35-9f76-5c305829be8e@default> Hi, Please find the updated webrev with suggested changes Webrev - http://cr.openjdk.java.net/~rpatil/8173664/webrev.01/ BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 Thanks, Fairoz > -----Original Message----- > From: Fairoz Matte > Sent: Thursday, May 11, 2017 9:05 AM > To: David Holmes ; serviceability- > dev at openjdk.java.net > Subject: RE: RFR: JDK-8173664: Typo in https://java.net/downloads/heap- > snapshot/hprof-binary-format.html > > Hi David, > > After having discussion with you over IM things are clear Expected change is - > "There will be a "LOAD CLASS" tag for the type of each array in the dump" > and current changeset missed "the" before type. > > I will make the changes and send it again. > > Thanks, > Fairoz > > > -----Original Message----- > > From: David Holmes > > Sent: Thursday, May 11, 2017 7:52 AM > > To: Fairoz Matte ; serviceability- > > dev at openjdk.java.net > > Subject: Re: RFR: JDK-8173664: Typo in > > https://java.net/downloads/heap- snapshot/hprof-binary-format.html > > > > On 10/05/2017 9:30 PM, Fairoz Matte wrote: > > > Hi David, > > > > > >> -----Original Message----- > > >> From: David Holmes > > >> Sent: Wednesday, May 10, 2017 3:26 PM > > >> To: Fairoz Matte ; serviceability- > > >> dev at openjdk.java.net > > >> Subject: Re: RFR: JDK-8173664: Typo in > > >> https://java.net/downloads/heap- snapshot/hprof-binary-format.html > > >> > > >> Hi Fairoz, > > >> > > >> On 10/05/2017 5:53 PM, Fairoz Matte wrote: > > >>> Hi, > > >>> > > >>> Kindly review the small typo fix, applicable only for JDK8 > > >>> BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 > > >>> Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ > > >> > > >> I think "for type type" was intended to be "for the type". > > >> > > > Yes it does look like after reading multiple times. > > > Thanks for the review I will close as Not an issue > > > > ??? It is still a typo that can be fixed. > > > > David > > > > > Thanks, > > > Fairoz > > > > > >> David > > >> > > >>> Thanks, > > >>> Fairoz > > >>> From david.holmes at oracle.com Thu May 11 10:54:44 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 11 May 2017 20:54:44 +1000 Subject: RFR: JDK-8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html In-Reply-To: <65cc8059-55ea-4d35-9f76-5c305829be8e@default> References: <288120a2-b86d-4d91-bdf4-b12058dd38ff@default> <1eb77f9f-5e3f-d65a-52bd-8227c8b4d964@oracle.com> <32b0c37a-e7a8-3f5e-c154-95587d2e9999@oracle.com> <916f10b9-a247-474b-9726-366673336853@default> <65cc8059-55ea-4d35-9f76-5c305829be8e@default> Message-ID: <0dd11746-0654-90b5-9b2a-09add607ccdc@oracle.com> Looks good. Thanks, David On 11/05/2017 7:33 PM, Fairoz Matte wrote: > Hi, > > Please find the updated webrev with suggested changes > Webrev - http://cr.openjdk.java.net/~rpatil/8173664/webrev.01/ > BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 > > Thanks, > Fairoz > >> -----Original Message----- >> From: Fairoz Matte >> Sent: Thursday, May 11, 2017 9:05 AM >> To: David Holmes ; serviceability- >> dev at openjdk.java.net >> Subject: RE: RFR: JDK-8173664: Typo in https://java.net/downloads/heap- >> snapshot/hprof-binary-format.html >> >> Hi David, >> >> After having discussion with you over IM things are clear Expected change is - >> "There will be a "LOAD CLASS" tag for the type of each array in the dump" >> and current changeset missed "the" before type. >> >> I will make the changes and send it again. >> >> Thanks, >> Fairoz >> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Thursday, May 11, 2017 7:52 AM >>> To: Fairoz Matte ; serviceability- >>> dev at openjdk.java.net >>> Subject: Re: RFR: JDK-8173664: Typo in >>> https://java.net/downloads/heap- snapshot/hprof-binary-format.html >>> >>> On 10/05/2017 9:30 PM, Fairoz Matte wrote: >>>> Hi David, >>>> >>>>> -----Original Message----- >>>>> From: David Holmes >>>>> Sent: Wednesday, May 10, 2017 3:26 PM >>>>> To: Fairoz Matte ; serviceability- >>>>> dev at openjdk.java.net >>>>> Subject: Re: RFR: JDK-8173664: Typo in >>>>> https://java.net/downloads/heap- snapshot/hprof-binary-format.html >>>>> >>>>> Hi Fairoz, >>>>> >>>>> On 10/05/2017 5:53 PM, Fairoz Matte wrote: >>>>>> Hi, >>>>>> >>>>>> Kindly review the small typo fix, applicable only for JDK8 >>>>>> BugID: https://bugs.openjdk.java.net/browse/JDK-8173664 >>>>>> Webrev: http://cr.openjdk.java.net/~rpatil/8173664/webrev/ >>>>> >>>>> I think "for type type" was intended to be "for the type". >>>>> >>>> Yes it does look like after reading multiple times. >>>> Thanks for the review I will close as Not an issue >>> >>> ??? It is still a typo that can be fixed. >>> >>> David >>> >>>> Thanks, >>>> Fairoz >>>> >>>>> David >>>>> >>>>>> Thanks, >>>>>> Fairoz >>>>>> From kumar.x.srinivasan at oracle.com Fri May 12 00:17:03 2017 From: kumar.x.srinivasan at oracle.com (Kumar Srinivasan) Date: Thu, 11 May 2017 17:17:03 -0700 Subject: RFR: JDK-8179631: Fix Html5 errors in java.management, jdk.management, jdk.jdi and jdk.attach In-Reply-To: <59135D7D.3010704@oracle.com> References: <59135D7D.3010704@oracle.com> Message-ID: <5914FEFF.1010404@oracle.com> Hello, Ping!. Lets wrap this up soon, as we have to move to the next steps in the docs related work. Thanks Kumar > Hi All, > > Please review fixes to make the API doc comments HTML5 clean, > there are no changes to the verbiage, and mostly fixes for the table > styles defined here: > http://hg.openjdk.java.net/jdk9/dev/langtools/rev/ee84b7d44339 > > For tables with many entries I have used "striped", there are few > tables used for formatting purposes, for which "borderless" is used, > and "plain" for everything else. > > Thanks > Kumar > > Webrev: http://cr.openjdk.java.net/~ksrini/8179631/webrev.0/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8179631 From mandy.chung at oracle.com Fri May 12 02:48:46 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 11 May 2017 19:48:46 -0700 Subject: RFR: JDK-8179631: Fix Html5 errors in java.management, jdk.management, jdk.jdi and jdk.attach In-Reply-To: <59135D7D.3010704@oracle.com> References: <59135D7D.3010704@oracle.com> Message-ID: > On May 10, 2017, at 11:35 AM, Kumar Srinivasan wrote: > > Hi All, > > Please review fixes to make the API doc comments HTML5 clean, > there are no changes to the verbiage, and mostly fixes for the table > styles defined here: > http://hg.openjdk.java.net/jdk9/dev/langtools/rev/ee84b7d44339 > > For tables with many entries I have used "striped", there are few > tables used for formatting purposes, for which "borderless" is used, > and "plain" for everything else. > > Thanks > Kumar > > Webrev: http://cr.openjdk.java.net/~ksrini/8179631/webrev.0/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8179631 Looks good to me. Mandy From ioi.lam at oracle.com Fri May 12 05:21:24 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 11 May 2017 22:21:24 -0700 Subject: RFR[S] 8179769 serviceability/sa/TestCpoolForInvokeDynamic.java failing after changes for JDK-8171392 Message-ID: <59154654.9070005@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8179769 http://cr.openjdk.java.net/~iklam/jdk10/8179769-sa-cp-failures.v01/ Summary: In JDK-8171392 ( Move Klass pointers outside of ConstantPool entries so ConstantPool can be read-only), I changed how classes are stored in ConstantPool. However, I missed the changes that are required in Serviceability Agent (SA). In this patch, the SA java classes are changed to match the native code in constantPool.cpp. Testing: I tested with "rbt -j hs-tier2.js,hs-tier3.js" to run all the SA JTREG tests on all supported platforms. I also tested locally and verified in the JTR files that the bug has been fixed Thanks - Ioi -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Fri May 12 09:10:18 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 12 May 2017 02:10:18 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> Message-ID: <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> Hi Dmitry, Thank you for the update with some review comments resolved! I looked at it but will not provide my comments at this point. We will need another update according to the recent design review and items where we still have to reach an agreement. Thanks, Serguei On 5/10/17 08:27, Dmitry Samersoff wrote: > Serguei, > > Fixed minor issues (comments, netmask etc). > Added an error for attempt to use allow with an old transport. > > http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.17/ > > see also below. > > >> I do not understand this reason for adding more complexity. >> It seems, there should not be any issues in the future with rejecting >> all unsupported versions by the transport library. > Added a diagram explaining transport version negotiation to CR. I use > future versions (1.3; 1.4; 1.5) because all this versioning staff has a > long term goal and allow us to develop better transport without breaking > existing one. > >> We are not adding extra parameter, we are introducing new function >> that is a clone of another StartListening function with a version >> suffix '11' in its name and with an extra parameter. > Correct. We changed behavior of StartListening function and 1.1 > transport shouldn't care about old one. > > i.e. when we document 1.1 interface we describe the only function > StartListening(env, address, actualAddress, allow) > that have to be placed to the StartListening11 slot. > > > Back in 2015 I proposed to separate interfaces entirely (see webrev.04), > but we (Alan?) decided that it's an overkill. > >> The original StartListening function is being removed. >> It is much simpler to introduce new function AllowPeers(char* peers) >> with the same parameter. > This separate function have to be called explicitly before we start > listening, It is extra communication step. IMHO, not obvious one. > > So I would prefer to keep StartListening11 > > -Dmitry > > > On 2017-05-10 12:37, serguei.spitsyn at oracle.com wrote: >> Dmitry, >> >> Thank you a lot for the detailed reply! >> >> >> On 5/10/17 01:10, Dmitry Samersoff wrote: >>> Serguei, >>> >>> Please see my comments in-line. >>> >>> >>> On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >>>> Hi Dmitry, >>>> >>>> >>>> I'd like to resolve my questions before the upcoming design discussion >>>> on Thu. >>>> >>>> >>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >>>> >>>> >>>> >>>> >>>> (0) The design description from the bug report tells: >>>> >>>> > Than we change a negotiation protocol between JDWP and transport. >>>> > We pass maximal supported version to transport initialization >>>> routine and expect transport actual version to be returned. >>>> >>>> The modified negotiation protocol adds extra complexity. >>>> What is a motivation behind this? >>>> Is it really necessary for the transport library to return an actual >>>> version instead of rejecting the unmatched version? >>>> I do not see it is really used in the webrev.15 implementation. >>> Transport have to return it's actual version in order to allow agent >>> to perform appropriate action. >>> >>> see libjdwp/transport.c:526 >> This requirement adds extra complexity to the rules (transport >> negotiation protocol). >> It is not really necessary. >> The loadTransport() already does a lookup of a version that is accepted >> (not rejected) by the transport library and can save that version. >> The transport_startTransport() then should use the version found by the >> loadTransport(). >> >> >>> Today it's just a selection of proper API call but in a future it might >>> be too-old-transport-error or deprecation warning or security warning or >>> something else. >> I do not understand this reason for adding more complexity. >> It seems, there should not be any issues in the future with rejecting >> all unsupported versions by the transport library. >> However, it will be even more simple if one transport library API could >> support/accept all possible versions (see my alternate suggestion below). >> >> >>>> (1) The following change in the jdwp transport library will reject >>>> theJDWPTRANSPORT_VERSION_1_0 as it is below >>>> the version JDWPTRANSPORT_VERSION_1_1, but will except any version >>>> above the JDWPTRANSPORT_VERSION_1_1 >>>> (with providing/returning the actual transport version): >>>> >>>> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* cbTablePtr, >>>> - jint version, jdwpTransportEnv** result) >>>> + jint version, void** env) >>>> { >>>> - if (version != JDWPTRANSPORT_VERSION_1_0) { >>>> + if (version < JDWPTRANSPORT_VERSION_1_1) { >>>> return JNI_EVERSION; >>>> } >>>> >>>> >>>> Te following change will also prevent supporting the 1_0 version of the >>>> transport library: >>>> >>>> - interface.StartListening = &socketTransport_startListening; >>>> + interface.StartListening = NULL; >>> libdt_socket/socketTransport.c is an implementation of 1.1 *transport* >>> it's not intended to run with 1.0 *backend*. >> Why not? >> It would simplifies things if the transport library (and its API) is >> backward compatible. >> >>> i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 *transport* >>> require 1.1 or greater backend. >> This statement creates a confusion. >> The truce is that the transport library can support some number of >> versions. >> The latest supported version can satisfy the agent if it supports it. >> >>> see: libjdwp/transport.c:206 for version logic on backend (agent) side >> The logic at L206 does not require the transport library to return its >> version. >> It will work Ok if the library rejects unsupported versions. >> >> >>>> I'd suggest the following alternate change to the transport API allowing >>>> to support >>>> both old and new versions at the same time (it would simplify the >>>> negotiation rules): >>>> - Add new function: >>>> jdwpTransportError AllowPeers(const char* peers); >>>> >>>> - Keep the original StartListening function. >>>> The function uses the allowed peers data if it was previously >>>> cached >>>> by the AllowPeers(). >>>> >>>> - It seems, the alternate approach does not require adding the >>>> extra_data with version. >>>> But if there is still a real need to get the transport API version >>>> then it'd better >>>> to introduce a function named GetTransportVersion() or >>>> JDWP_TransportVersion(). >>>> This would allow to encapsulate any extra_data that is necessary in >>>> such a case. >>> From pure JDWP hardening point of view we can just add extra parameter >>> *allow* to existing StartListening(). Caller is responsible for stack >>> consistency so old transport continues to work. >> We are not adding extra parameter, we are introducing new function >> that is a clone of another StartListening function with a version >> suffix '11' in its name and with an extra parameter. >> The original StartListening function is being removed. >> It is much simpler to introduce new function AllowPeers(char* peers) >> with the same parameter. >> The original StartListening function works as before. >> The updated API can support both versions 1_0 and 1_1. >> >> >>> But my goal was to create a versioning in JDWP agent -> transport >>> relations that was missed in initial JDWP design. Further, more >>> complicated, changes (IPv6 support, UDS sockets support etc) require >>> this logic. >> Would introducing function JdwpTransportVersion() achieve what you wanted? >> >> >>> We have a structure jdwpTransportNativeInterface with a fixed set of >>> functions (see jdwpTransport.h:153). To add any new function to this >>> structure we have to create a method to detect presence of this >>> function. So we can't use GetTransportVersion(). >> It is not clear why would you need such a method for any new function? >> The transport version maps to the whole set of functions. >> >> >>> With as separate AllowPeer() function nobody remind an agent writer that >>> they should use it, but extra parameter makes api changes and >>> requirements clear visible (up to compiler warning). >> I do not see any difference. >> No compiler warning if NULL is passed in place of the 'allow' parameter. >> >> >>> Also I'm against of changing behavior of existing function. >>> >>> >>>> (2) The following error messages miss the actual IP address or mask that >>>> was found to be illegal: >>>> >>>> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid ip >>>> address for allow"); 392 >>>> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask for >>>> allow"); >>> Other parameter parsing functions (e.g. "invalid port number specified" >>> at 274) doesn't explain what parameter is bad. >> It is not good either. >> But new functionality is added, so the more diagnostic details the better. >> >>> I think typical allow would have one or two entry, so verbose error >>> message is not worth significant complication of parsing code. >> It is still better to print what was not accepted. >> It should not be a problem to print it anyway. >> >> >>> I would prefer to leave it as is. >>> >>> >>>> (3) A suggestion on the following: >>>> >>>> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // reset mask >>>> >>>> I'd suggest a more explicit approach instead of the L400 for a better >>>> clarity: >>>> >>>> 386 if (*s == '/') { >>>> 387 // netmask specified >>>> 388 s = mask_s2u(s + 1, &mask); >>>> 389 if (*(s-1) == '/') { >>>> 390 // Input is not consumed, something bad happens >>>> 391 _peers_cnt = 0; >>>> 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, "invalid netmask >>>> for allow"); >>>> 393 } >>>> 394 } else { mask = 0xFFFFFFFF; } >>> I'll try it. >> Ok, thanks. >> >> >>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >>>> >>>> (4) Some confusion with the fragment and its comment: >>>> >>>> + >>>> + /* Pass the latest supported version, >>>> + * expect actual transport version in t->extra_data->version >>>> + */ >>>> + ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); >>>> + if (ver == JNI_EVERSION) { >>>> ver = (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_0, >>>> &t); >>>> + // Special handling for versionless transports >>>> + info->transportVersion = JDWPTRANSPORT_VERSION_1_0; >>>> + } >>>> + else { >>>> + info->transportVersion = (*t)->extra_data->version; >>>> + } >>>> + >>>> >>>> The term "latest supported version" is ambiguous in this context. Do you >>>> mean, supported by the JDWP back-end or by the agent library? >>> Supported by the JDWP backend. I'll update comments. >> Ok, thanks. >> >> >>>> Also, it >>>> is not clear in what circumstances the agent library would support the >>>> version 1_0 only. The JDWP back-end is always coupled with the socket >>>> transport library, isn't it? Is it because the libdt_shmem library can >>>> be used on Windows or because a different native library path can be >>>> used? Could you explain a little bit? >>> It's more generic question: Do we need any backward compatibility here? >>> I assume *yes* (can't recall why - probably it was a tree-year old >>> decision). >>> >>> If we state that new backend will not support old transports today and >>> in a future - all versioning logic is not necessary. >>> >>> >>>> As we see in (1) above the actual >>>> transport version can be different from requested only if the requested >>>> version is above the latest supported by the transport library. >>>> Otherwise, the version is matched or the JNI_EVERSION is returned. The >>>> subsequent call to the OnLoad function can succeed only if the library >>>> supports just the version 1_0. >>> See answer to (1) above. transport library has exactly one version but >>> JDWP backend supports couple of transport library versions back. >> I've replied above too. :) >> >> >>>> (5) Memory allocation for the info->allow >>>> field is needed only for the transport version 1_1: >>>> >>>> + if (allow != NULL) { >>>> + info->allow = jvmtiAllocate((int)strlen(allow)+1); >>>> + if (info->allow == NULL) { >>>> + serror = JDWP_ERROR(OUT_OF_MEMORY); >>>> + goto handleError; >>>> + } >>>> + (void)strcpy(info->allow, allow); >>>> + } >>> Correct. allocation needed for 1.1 and *greater*. I can change it, but >>> it makes code less readable. >> The same fragment in a different place should not be less readable, >> maybe just less elegant. >> >> >>>> (6) There is no handling for the condition when the *allow* parameter is >>>> passed but the transport version is 1_0 (which does not support the >>>> *allow* parameter): >>> Correct. Warning or ever error should be here. >>> >>> I plan to open a separate follow-up CR for this problem - we have to >>> decide how we should handle this situation (warning or error) and change >>> the code, >>> >>> but I can add a plain printf() here if you like. >> I'm Ok with both error or warning but what is needed from a security >> point of view? >> We can avoid filing a separate follow-up CR in this case. >> Should it be an error if the *allow* parameter is used with the old >> transport library that does not support it? >> >> >> Thanks, >> Serguei >> >> >>>> + /* Interface version 1.0 doesn't support versioning, so we have to >>>> + * use global variable and set the version artifically. >>>> + * Use (*t)->extra_data->version directly when 1.0 is gone. >>>> + */ >>>> + switch(info->transportVersion) { >>>> + case JDWPTRANSPORT_VERSION_1_0: >>>> err = (*trans)->StartListening(trans, address, &retAddress); >>>> + break; >>>> + case JDWPTRANSPORT_VERSION_1_1: >>>> + err = (*trans)->StartListening11(trans, address, &retAddress, >>>> info->allow); >>>> + break; >>>> + default: >>>> + err = JNI_EVERSION; >>>> + } >>> -Dmitry >>> >>> >>>> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>>>> Robbin, >>>>> >>>>>> One follow-up issue is that if you start suspended >>>>>> and than connect with >>>>>> an unallowed client the JVM starts and executes the program. >>>>> Fixed. >>>>> >>>>> see http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>>>> >>>>> -Dmitry >>>>> >>>>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>>>> Hi Dmitry, thanks for the update. >>>>>> >>>>>> One follow-up issue is that if you start suspended and than connect >>>>>> with >>>>>> an unallowed client the JVM starts and executes the program. >>>>>> Simple program prints "Hello". >>>>>> >>>>>> [rehn at rehn-ws vanilla-hs]$ java >>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>>>> >>>>>> >>>>>> -cp . H >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> ERROR: Peer not allowed to connect >>>>>> Listening for transport dt_socket at address: 9999 >>>>>> Hello >>>>>> >>>>>> I think it would be good if the VM stayed suspended when an unallowed >>>>>> client connects. >>>>>> >>>>>> Thanks, Robbin >>>>>> >>>>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>>>> Robbin, >>>>>>> >>>>>>> Please, see: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>>>> >>>>>>>> 1: >>>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>> Fixed. >>>>>>> >>>>>>>> 2: >>>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>>> connecting a >>>>>>>> client. >>>>>>> Moved allowed parameter (and parser call) to StartListening. >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> >>>>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> I took a look at this, I have two practical issues: >>>>>>>> >>>>>>>> 1: >>>>>>>> [rehn at rehn-ws dev]$ java >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -cp runs ForEver >>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>> ERROR: transport error 202: peer not allowed to connect: Success >>>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>> fatal >>>>>>>> error [transport.c:358] >>>>>>>> >>>>>>>> So connecting with an unallowed client terminates the VM. >>>>>>>> >>>>>>>> 2: >>>>>>>> [rehn at rehn-ws dev]$ java >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -cp runs ForEver >>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>> ERROR: transport error 202: unable to parse list of allowed peers: >>>>>>>> Success >>>>>>>> JDWP exit error JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>> fatal >>>>>>>> error [transport.c:358] >>>>>>>> >>>>>>>> Starting with an bad allow filter terminates the VM when >>>>>>>> connecting a >>>>>>>> client. >>>>>>>> >>>>>>>> >>>>>>>> Connecting with an unallowed ip/port should not terminate the VM >>>>>>>> and we >>>>>>>> should verify allow filter directly at startup. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> /Robbin >>>>>>>> >>>>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>>>> Everybody, >>>>>>>>> >>>>>>>>> Please review: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>>>> >>>>>>>>> These changes introduce new parameter[1] of the socket transport - >>>>>>>>> allow. Users can explicitly specify a list of hosts that allowed to >>>>>>>>> connect to jdwp server and it's the second part of JDWP >>>>>>>>> hardening[2]. >>>>>>>>> >>>>>>>>> No restrictions are applied by default now but I'll file a >>>>>>>>> separate CR >>>>>>>>> to restrict list of allowed peers to localhost by default. >>>>>>>>> >>>>>>>>> Also these changes implement versioning for jdwp transport and >>>>>>>>> therefor >>>>>>>>> simplify feature development of jdwp. >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. Example command line: >>>>>>>>> >>>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>>>> >>>>>>>>> Possible values for allow parameter: >>>>>>>>> * - accept connections from everywhere. >>>>>>>>> N.N.N.N - accept connections from this IP address only >>>>>>>>> N.N.N.N/nn - accept connections from particular ip subnet >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> > From serguei.spitsyn at oracle.com Fri May 12 23:26:50 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 12 May 2017 16:26:50 -0700 Subject: RFR[S] 8179769 serviceability/sa/TestCpoolForInvokeDynamic.java failing after changes for JDK-8171392 In-Reply-To: <59154654.9070005@oracle.com> References: <59154654.9070005@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From jcbeyler at google.com Mon May 15 16:48:40 2017 From: jcbeyler at google.com (JC Beyler) Date: Mon, 15 May 2017 09:48:40 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Message-ID: Dear all, I've updated the webrev to: http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/ Robbin, I believe I have addressed most of your items with webrev 02: - I added a JTreg test to show how it works: http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_ files/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c - I've modified the code to use its own data structures both internally and externally, this will make it easier to move out of AsyncGetCallTrace as we move forward, that is still on my TODOs - I cleaned up the JVMTI API by passing a structure that handles the num_traces and put in a ReleaseTraces as well - I cleaned up other issues as well. However, I have three questions, which are probably because I'm new in this community: 1) My previous webrevs were based off of JDK9 by mistake. When I took JDK10 via : hg clone http://hg.openjdk.java.net/jdk10/jdk10 jdk10 - I don't see code compatible with what you were showing (ie your patches don't make sense for that code base; ex: klass is still accessed via klass() for example in collectedHeap.inline.hpp) - Would you know what is the right hg clone command so we are working on the same code base? 2) You mentioned I was using os::malloc, new, NEW_C_HEAP_ARRAY; I cleaned out the os::malloc but which of the new vs NEW_C_HEAP_ARRAY should I use. It might be that I don't understand when one uses one or the other but I see both used around the code base? - Is it that new is to be used for anything internal and NEW_C_HEAP_ARRAY anything provided to the JVMTI users outside of the JVM? 3) Casts: same kind question: which should I use. The code was using a bit of everything, I'll refactor it entirely but I was not clear if I should go to C casts or C++ casts as I see both in the codebase. What is the convention I should use? Final notes on this webrev: - I am still missing: - Putting a TLAB implementation so that we can compare both webrevs - Have not tried to circumvent AsyncGetCallTrace - Putting in the handling of GC'd objects - Fix a stack walker issue I have seen, I think I know the problem and will test that theory out for the next webrev I will work on integrating those items for the next webrev! Thanks for your help, Jc Ps: I tested this on a new repo: hg clone http://hg.openjdk.java.net/jdk10/jdk10 jdk10 ... building it cd test jtreg -nativepath:/build/linux-x86_64-normal-server-release/support/test/hotspot/jtreg/native/lib/ -jdk /linux-x86_64-normal-server-release/images/jdk ../hotspot/test/serviceability/jvmti/HeapMonitor/ On Thu, May 4, 2017 at 11:21 PM, serguei.spitsyn at oracle.com < serguei.spitsyn at oracle.com> wrote: > Robbin, > > Thank you for forwarding! > I will review it. > > Thanks, > Serguei > > > > On 5/4/17 02:13, Robbin Ehn wrote: > >> Hi, >> >> To me the compiler changes looks what is expected. >> It would be good if someone from compiler could take a look at that. >> Added compiler to mail thread. >> >> Also adding Serguei, It would be good with his view also. >> >> My initial take on it, read through most of the code and took it for a >> ride. >> >> ############################## >> - Regarding the compiler changes: I think we need the 'TLAB end' trickery >> (mentioned by Tony P) >> instead of a separate check for sampling in fast path for the final >> version. >> >> ############################## >> - This patch I had to apply to get it compile on JDK 10: >> >> diff -r ac3ded340b35 src/share/vm/gc/shared/collectedHeap.inline.hpp >> --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr 28 >> 14:31:38 2017 +0200 >> +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May 04 >> 10:22:56 2017 +0200 >> @@ -87,3 +87,3 @@ >> // support for object alloc event (no-op most of the time) >> - if (klass() != NULL && klass()->name() != NULL) { >> + if (klass != NULL && klass->name() != NULL) { >> Thread *base_thread = Thread::current(); >> diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 14:31:38 >> 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:22:56 >> 2017 +0200 >> @@ -316,3 +316,3 @@ >> JavaThread *thread = reinterpret_cast(Thread::current()); >> - assert(o->size() << LogHeapWordSize == byte_size, >> + assert(o->size() << LogHeapWordSize == (long)byte_size, >> "Object size is incorrect."); >> >> ############################## >> - This patch I had to apply to get it not asserting during slowdebug: >> >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 15:15:16 >> 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:24:25 >> 2017 +0200 >> @@ -32,3 +32,3 @@ >> // TODO(jcbeyler): should we make this into a JVMTI structure? >> -struct StackTraceData { >> +struct StackTraceData : CHeapObj { >> ASGCT_CallTrace *trace; >> @@ -143,3 +143,2 @@ >> StackTraceStorage::StackTraceStorage() : >> - _allocated_traces(new StackTraceData*[MaxHeapTraces]), >> _allocated_traces_size(MaxHeapTraces), >> @@ -147,2 +146,3 @@ >> _allocated_count(0) { >> + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, MaxHeapTraces, >> mtInternal); >> memset(_allocated_traces, 0, sizeof(*_allocated_traces) * >> MaxHeapTraces); >> @@ -152,3 +152,3 @@ >> StackTraceStorage::~StackTraceStorage() { >> - delete[] _allocated_traces; >> + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); >> } >> >> - Classes should extend correct base class for which type of memory is >> used for it e.g.: CHeapObj or StackObj or AllStatic >> - The style in heapMonitoring.cpp is a bit different from normal >> vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, >> os::malloc and new. >> - In jvmtiHeapTransition.hpp you use C cast instead. >> >> ############################## >> - This patch I had apply to get traces without setting an ?unrelated? >> capability >> - Should this not be a new capability? >> >> diff -r c02a5d8785bf src/share/vm/prims/forte.cpp >> --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 >> +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 >> @@ -530,6 +530,6 @@ >> >> - if (!JvmtiExport::should_post_class_load()) { >> +/* if (!JvmtiExport::should_post_class_load()) { >> trace->num_frames = ticks_no_class_load; // -1 >> return; >> - } >> + }*/ >> >> ############################## >> - forte.cpp: (I know this is not part of your changes but) >> find_jmethod_id_or_null give me NULL for my test. >> It looks like we actually want the regular jmethod_id() ? >> >> Since we are the thread we are talking about (and in same ucontext) and >> thread is in vm and have a last java frame, >> I think most of the checks done in AsyncGetCallTrace is irrelevant, so >> you should be-able to call forte_fill_call_trace_given_top directly. >> But since we might need jmethod_id() if possible to avoid getting method >> id NULL, >> we need some fixes in forte code, or just do the vframStream loop inside >> heapMonitoring.cpp and not use forte.cpp. >> >> Something like: >> >> if (jthread->has_last_Java_frame()) { // just to be safe >> vframeStream vfst(jthread); >> while (!vfst.at_end()) { >> Method* m = vfst.method(); >> m->jmethod_id(); >> m->line_number_from_bci(vfst.bci()); >> vfst.next(); >> } >> >> - This is a bit confusing in forte.cpp, trace->frames[count].lineno = bci. >> Line number should be m->line_number_from_bci(bci); >> Do the heapMonitoring suppose to trace with bci or line number? >> I would say bci, meaning we should either rename ASGCT_CallFrame?lineno >> or use another data structure which says bci. >> >> ############################## >> - // TODO(jcbeyler): remove this extra code handling the extra trace for >> Please fix all these TODO's :) >> >> ############################## >> - heapMonitoring.hpp: >> // TODO(jcbeyler): is this algorithm acceptable in open source? >> >> Why is this comment here? What is the implication? >> Have you tested any simpler algorithm? >> >> ############################## >> - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for >> building the agent) >> >> ############################## >> - monitoring_period vs HeapMonitorRate, pick rate or period. >> >> ############################## >> - globals.hpp >> Why is MaxHeapTraces not settable/overridable from jvmti interface? That >> would be handy. >> >> ############################## >> - jvmtiStackTraceData + ASGCT_CallFrame memory >> Are the agent suppose to loop through and free all ASGCT_CallFrame? >> Wouldn't it be better with some kinda protocol, like: >> (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); >> (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); >> >> Also using another data structure that have num_traces inside it >> simplifies things. >> So I'm not convinced using the async structure is the best way forward. >> >> >> I have more questions, but I think it's better if you respond and update >> the code first. >> >> Thanks! >> >> /Robbin >> >> >> On 04/21/2017 11:34 PM, JC Beyler wrote: >> >>> Hi all, >>> >>> I've added size information to the allocation sampling system. This >>> allows the callback to remember the size of each sampled allocation. >>> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ >>> >>> The new webrev.01 also adds the actual heap monitoring sampling system >>> in files: >>> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >>> are/vm/runtime/heapMonitoring.cpp.patch >>> and >>> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >>> are/vm/runtime/heapMonitoring.hpp.patch >>> >>> My next step is to add the GC part to the webrev, which will allow users >>> to determine what objects are live and what are garbage. >>> >>> Thanks for your attention and let me know if there are any questions! >>> >>> Have a wonderful Friday! >>> Jc >>> >>> On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler >> > wrote: >>> >>> Hi all, >>> >>> I worked on getting a few numbers for overhead and accuracy for my >>> feature. I'm unsure if here is the right place to provide the full data, so >>> I am just summarizing >>> here for now. >>> >>> - Overhead of the feature >>> >>> Using the Dacapo benchmark (http://dacapobench.org/). My initial >>> results are that sampling provides 2.4% with a 512k sampling, 512k being >>> our default setting. >>> >>> - Note: this was without the tradesoap, tradebeans and tomcat >>> benchmarks since they did not work with my JDK9 (issue between Dacapo and >>> JDK9 it seems) >>> - I want to rerun next week to ensure number stability >>> >>> - Accuracy of the feature >>> >>> I wrote a small microbenchmark that allocates from two different >>> stacktraces at a given ratio. For example, 10% of stacktrace S1 and 90% >>> from stacktrace S2. The >>> microbenchmark was run 20 times, I averaged the results and looked >>> for accuracy. It seems that statistically it is sound since if I >>> allocated10% S1 and 90% S2, with a >>> sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. >>> >>> Let me know if there are any questions on the numbers and if you'd >>> like to see some more data. >>> >>> Note: this was done using our internal JDK8 implementation since the >>> webrev provided by http://cr.openjdk.java.net/~ra >>> sbold/heapz/webrev.00/index.html >>> does >>> not yet contain the whole implementation and therefore would have been >>> misleading. >>> >>> Thanks, >>> Jc >>> >>> >>> On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler >> > wrote: >>> >>> Hi all, >>> >>> To move the discussion forward, with Chuck Rasbold's help to >>> make a webrev, we pushed this: >>> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html < >>> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html> >>> 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg >>> >>> This is not a final change that does the whole proposition from >>> the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 >>> ; what it does show >>> is parts of the implementation that is proposed and hopefully can start the >>> conversation going >>> as I work through the details. >>> >>> For example, the changes to C2 are done here for the >>> allocations: http://cr.openjdk.java.net/~ra >>> sbold/heapz/webrev.00/src/share/vm/opto/macro.cpp.patch >>> >> re/vm/opto/macro.cpp.patch> >>> >>> Hopefully this all makes sense and thank you for all your future >>> comments! >>> Jc >>> >>> >>> On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler >> > wrote: >>> >>> Hello all, >>> >>> This is a follow-up from Jeremy's initial email from last >>> year: >>> http://mail.openjdk.java.net/pipermail/serviceability-dev/20 >>> 15-June/017543.html >> pipermail/serviceability-dev/2015-June/017543.html> >>> >>> I've gone ahead and started working on preparing this and >>> Jeremy and I went down the route of actually writing it up in JEP form: >>> https://bugs.openjdk.java.net/browse/JDK-8171119 >>> >>> I think original conversation that happened last year in >>> that thread still holds true: >>> >>> - We have a patch at Google that we think others might be >>> interested in >>> - It provides a means to understand where the >>> allocation hotspots are at a very low overhead >>> - Since it is at a low overhead, we can leave it on by >>> default >>> >>> So I come to the mailing list with Jeremy's initial question: >>> "I thought I would ask if there is any interest / if I >>> should write a JEP / if I should just forget it." >>> >>> A year ago, it seemed some thought it was a good idea, is >>> this still true? >>> >>> Thanks, >>> Jc >>> >>> >>> >>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.sapre at oracle.com Tue May 16 08:45:00 2017 From: amit.sapre at oracle.com (Amit Sapre) Date: Tue, 16 May 2017 01:45:00 -0700 (PDT) Subject: RFR : (JDK9) JDK-8179700 : Exceptions thrown in StartManagementAgent.java Message-ID: <297e0f27-fa71-44b2-9b70-412ed084e86e@default> Hello, Please review this trivial fix to StartManagementAgent test case. Webrev : http://cr.openjdk.java.net/~asapre/webrev/2017/JDK-8179700/webrev.00/ Bug ID : https://bugs.openjdk.java.net/browse/JDK-8179700 Thanks, Amit -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fuchs at oracle.com Tue May 16 09:15:09 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 16 May 2017 10:15:09 +0100 Subject: RFR : (JDK9) JDK-8179700 : Exceptions thrown in StartManagementAgent.java In-Reply-To: <297e0f27-fa71-44b2-9b70-412ed084e86e@default> References: <297e0f27-fa71-44b2-9b70-412ed084e86e@default> Message-ID: Hi Amit, Can you point to the changeset (or the reason) which caused the exception to change? I'd like to make sure we don't have some regression lurking here... best regards, -- daniel On 16/05/2017 09:45, Amit Sapre wrote: > Hello, > > > > Please review this trivial fix to StartManagementAgent test case. > > > > Webrev : > http://cr.openjdk.java.net/~asapre/webrev/2017/JDK-8179700/webrev.00/ > > Bug ID : https://bugs.openjdk.java.net/browse/JDK-8179700 > > > > Thanks, > > Amit** > From robbin.ehn at oracle.com Tue May 16 12:20:33 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 16 May 2017 14:20:33 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Message-ID: Just a few answers, On 05/15/2017 06:48 PM, JC Beyler wrote: > Dear all, > > I've updated the webrev to: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/ I'll look at this later, thanks! > > Robbin, > I believe I have addressed most of your items with webrev 02: > - I added a JTreg test to show how it works: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_files/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c > > - I've modified the code to use its own data structures both internally and externally, this will make it easier to move out of AsyncGetCallTrace as we move forward, that > is still on my TODOs > - I cleaned up the JVMTI API by passing a structure that handles the num_traces and put in a ReleaseTraces as well > - I cleaned up other issues as well. > > However, I have three questions, which are probably because I'm new in this community: > 1) My previous webrevs were based off of JDK9 by mistake. When I took JDK10 via : hg clone http://hg.openjdk.java.net/jdk10/jdk10 > jdk10 > - I don't see code compatible with what you were showing (ie your patches don't make sense for that code base; ex: klass is still accessed via klass() for example in > collectedHeap.inline.hpp) > - Would you know what is the right hg clone command so we are working on the same code base? We use jdk10-hs, e.g. hg tclone http://hg.openjdk.java.net/jdk10/hs 10-hs There is sporadic big merges going from jdk9->jdk10->jdk10-hs and jdk10-hs->jdk10, so 10 is moving... > > 2) You mentioned I was using os::malloc, new, NEW_C_HEAP_ARRAY; I cleaned out the os::malloc but which of the new vs NEW_C_HEAP_ARRAY should I use. It might be that I > don't understand when one uses one or the other but I see both used around the code base? > - Is it that new is to be used for anything internal and NEW_C_HEAP_ARRAY anything provided to the JVMTI users outside of the JVM? We overload new operator when you extend correct base class, e.g. CHeapObj so use 'new' But for arrays you will need the macro NEW_C_HEAP_ARRAY. > > 3) Casts: same kind question: which should I use. The code was using a bit of everything, I'll refactor it entirely but I was not clear if I should go to C casts or C++ > casts as I see both in the codebase. What is the convention I should use? Just be consist, use what suites you, C++ casts might be preferable, if we are moving towards C++11. And use 'right' cast, e.g. going from Thread* to JavaThread* you should use C cast or static_cast, not reinterpret_cast I would say. > > Final notes on this webrev: > - I am still missing: > - Putting a TLAB implementation so that we can compare both webrevs > - Have not tried to circumvent AsyncGetCallTrace > - Putting in the handling of GC'd objects > - Fix a stack walker issue I have seen, I think I know the problem and will test that theory out for the next webrev > > I will work on integrating those items for the next webrev! Thanks! > > Thanks for your help, > Jc > > Ps: I tested this on a new repo: > > hg clone http://hg.openjdk.java.net/jdk10/jdk10 jdk10 > ... building it > cd test > jtreg -nativepath:/build/linux-x86_64-normal-server-release/support/test/hotspot/jtreg/native/lib/ -jdk > /linux-x86_64-normal-server-release/images/jdk ../hotspot/test/serviceability/jvmti/HeapMonitor/ > I'll test it out! /Robbin > > > On Thu, May 4, 2017 at 11:21 PM, serguei.spitsyn at oracle.com > wrote: > > Robbin, > > Thank you for forwarding! > I will review it. > > Thanks, > Serguei > > > > On 5/4/17 02:13, Robbin Ehn wrote: > > Hi, > > To me the compiler changes looks what is expected. > It would be good if someone from compiler could take a look at that. > Added compiler to mail thread. > > Also adding Serguei, It would be good with his view also. > > My initial take on it, read through most of the code and took it for a ride. > > ############################## > - Regarding the compiler changes: I think we need the 'TLAB end' trickery (mentioned by Tony P) > instead of a separate check for sampling in fast path for the final version. > > ############################## > - This patch I had to apply to get it compile on JDK 10: > > diff -r ac3ded340b35 src/share/vm/gc/shared/collectedHeap.inline.hpp > --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr 28 14:31:38 2017 +0200 > +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May 04 10:22:56 2017 +0200 > @@ -87,3 +87,3 @@ > // support for object alloc event (no-op most of the time) > - if (klass() != NULL && klass()->name() != NULL) { > + if (klass != NULL && klass->name() != NULL) { > Thread *base_thread = Thread::current(); > diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 14:31:38 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:22:56 2017 +0200 > @@ -316,3 +316,3 @@ > JavaThread *thread = reinterpret_cast(Thread::current()); > - assert(o->size() << LogHeapWordSize == byte_size, > + assert(o->size() << LogHeapWordSize == (long)byte_size, > "Object size is incorrect."); > > ############################## > - This patch I had to apply to get it not asserting during slowdebug: > > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:24:25 2017 +0200 > @@ -32,3 +32,3 @@ > // TODO(jcbeyler): should we make this into a JVMTI structure? > -struct StackTraceData { > +struct StackTraceData : CHeapObj { > ASGCT_CallTrace *trace; > @@ -143,3 +143,2 @@ > StackTraceStorage::StackTraceStorage() : > - _allocated_traces(new StackTraceData*[MaxHeapTraces]), > _allocated_traces_size(MaxHeapTraces), > @@ -147,2 +146,3 @@ > _allocated_count(0) { > + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, MaxHeapTraces, mtInternal); > memset(_allocated_traces, 0, sizeof(*_allocated_traces) * MaxHeapTraces); > @@ -152,3 +152,3 @@ > StackTraceStorage::~StackTraceStorage() { > - delete[] _allocated_traces; > + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); > } > > - Classes should extend correct base class for which type of memory is used for it e.g.: CHeapObj or StackObj or AllStatic > - The style in heapMonitoring.cpp is a bit different from normal vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, os::malloc and new. > - In jvmtiHeapTransition.hpp you use C cast instead. > > ############################## > - This patch I had apply to get traces without setting an ?unrelated? capability > - Should this not be a new capability? > > diff -r c02a5d8785bf src/share/vm/prims/forte.cpp > --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 > @@ -530,6 +530,6 @@ > > - if (!JvmtiExport::should_post_class_load()) { > +/* if (!JvmtiExport::should_post_class_load()) { > trace->num_frames = ticks_no_class_load; // -1 > return; > - } > + }*/ > > ############################## > - forte.cpp: (I know this is not part of your changes but) > find_jmethod_id_or_null give me NULL for my test. > It looks like we actually want the regular jmethod_id() ? > > Since we are the thread we are talking about (and in same ucontext) and thread is in vm and have a last java frame, > I think most of the checks done in AsyncGetCallTrace is irrelevant, so you should be-able to call forte_fill_call_trace_given_top directly. > But since we might need jmethod_id() if possible to avoid getting method id NULL, > we need some fixes in forte code, or just do the vframStream loop inside heapMonitoring.cpp and not use forte.cpp. > > Something like: > > if (jthread->has_last_Java_frame()) { // just to be safe > vframeStream vfst(jthread); > while (!vfst.at_end()) { > Method* m = vfst.method(); > m->jmethod_id(); > m->line_number_from_bci(vfst.bci()); > vfst.next(); > } > > - This is a bit confusing in forte.cpp, trace->frames[count].lineno = bci. > Line number should be m->line_number_from_bci(bci); > Do the heapMonitoring suppose to trace with bci or line number? > I would say bci, meaning we should either rename ASGCT_CallFrame?lineno or use another data structure which says bci. > > ############################## > - // TODO(jcbeyler): remove this extra code handling the extra trace for > Please fix all these TODO's :) > > ############################## > - heapMonitoring.hpp: > // TODO(jcbeyler): is this algorithm acceptable in open source? > > Why is this comment here? What is the implication? > Have you tested any simpler algorithm? > > ############################## > - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for building the agent) > > ############################## > - monitoring_period vs HeapMonitorRate, pick rate or period. > > ############################## > - globals.hpp > Why is MaxHeapTraces not settable/overridable from jvmti interface? That would be handy. > > ############################## > - jvmtiStackTraceData + ASGCT_CallFrame memory > Are the agent suppose to loop through and free all ASGCT_CallFrame? > Wouldn't it be better with some kinda protocol, like: > (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); > (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); > > Also using another data structure that have num_traces inside it simplifies things. > So I'm not convinced using the async structure is the best way forward. > > > I have more questions, but I think it's better if you respond and update the code first. > > Thanks! > > /Robbin > > > On 04/21/2017 11:34 PM, JC Beyler wrote: > > Hi all, > > I've added size information to the allocation sampling system. This allows the callback to remember the size of each sampled allocation. > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ > > The new webrev.01 also adds the actual heap monitoring sampling system in files: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.cpp.patch > > and > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.hpp.patch > > > My next step is to add the GC part to the webrev, which will allow users to determine what objects are live and what are garbage. > > Thanks for your attention and let me know if there are any questions! > > Have a wonderful Friday! > Jc > > On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler >> wrote: > > Hi all, > > I worked on getting a few numbers for overhead and accuracy for my feature. I'm unsure if here is the right place to provide the full data, so I am just > summarizing > here for now. > > - Overhead of the feature > > Using the Dacapo benchmark (http://dacapobench.org/). My initial results are that sampling provides 2.4% with a 512k sampling, 512k being our default setting. > > - Note: this was without the tradesoap, tradebeans and tomcat benchmarks since they did not work with my JDK9 (issue between Dacapo and JDK9 it seems) > - I want to rerun next week to ensure number stability > > - Accuracy of the feature > > I wrote a small microbenchmark that allocates from two different stacktraces at a given ratio. For example, 10% of stacktrace S1 and 90% from stacktrace > S2. The > microbenchmark was run 20 times, I averaged the results and looked for accuracy. It seems that statistically it is sound since if I allocated10% S1 and 90% > S2, with a > sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. > > Let me know if there are any questions on the numbers and if you'd like to see some more data. > > Note: this was done using our internal JDK8 implementation since the webrev provided by http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > > > does not yet contain the whole > implementation and therefore would have been misleading. > > Thanks, > Jc > > > On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler >> wrote: > > Hi all, > > To move the discussion forward, with Chuck Rasbold's help to make a webrev, we pushed this: > http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > > > 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg > > This is not a final change that does the whole proposition from the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 > > >; what it does show is parts of the implementation that is > proposed and hopefully can start the conversation going > as I work through the details. > > For example, the changes to C2 are done here for the allocations: http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/share/vm/opto/macro.cpp.patch > > > > > Hopefully this all makes sense and thank you for all your future comments! > Jc > > > On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler >> > wrote: > > Hello all, > > This is a follow-up from Jeremy's initial email from last year: > http://mail.openjdk.java.net/pipermail/serviceability-dev/2015-June/017543.html > > > > > I've gone ahead and started working on preparing this and Jeremy and I went down the route of actually writing it up in JEP form: > https://bugs.openjdk.java.net/browse/JDK-8171119 > > I think original conversation that happened last year in that thread still holds true: > > - We have a patch at Google that we think others might be interested in > - It provides a means to understand where the allocation hotspots are at a very low overhead > - Since it is at a low overhead, we can leave it on by default > > So I come to the mailing list with Jeremy's initial question: > "I thought I would ask if there is any interest / if I should write a JEP / if I should just forget it." > > A year ago, it seemed some thought it was a good idea, is this still true? > > Thanks, > Jc > > > > > > > From jcbeyler at google.com Wed May 17 03:47:06 2017 From: jcbeyler at google.com (JC Beyler) Date: Tue, 16 May 2017 20:47:06 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Message-ID: Dear Robbin, Thank you for the answers, much appreciated :) Jc On Tue, May 16, 2017 at 5:20 AM, Robbin Ehn wrote: > Just a few answers, > > On 05/15/2017 06:48 PM, JC Beyler wrote: > >> Dear all, >> >> I've updated the webrev to: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/ < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/> >> > > I'll look at this later, thanks! > > >> Robbin, >> I believe I have addressed most of your items with webrev 02: >> - I added a JTreg test to show how it works: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_fi >> les/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_f >> iles/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c> >> - I've modified the code to use its own data structures both internally >> and externally, this will make it easier to move out of AsyncGetCallTrace >> as we move forward, that is still on my TODOs >> - I cleaned up the JVMTI API by passing a structure that handles the >> num_traces and put in a ReleaseTraces as well >> - I cleaned up other issues as well. >> >> However, I have three questions, which are probably because I'm new in >> this community: >> 1) My previous webrevs were based off of JDK9 by mistake. When I took >> JDK10 via : hg clone http://hg.openjdk.java.net/jdk10/jdk10 < >> http://hg.openjdk.java.net/jdk10/jdk10> jdk10 >> - I don't see code compatible with what you were showing (ie your >> patches don't make sense for that code base; ex: klass is still accessed >> via klass() for example in collectedHeap.inline.hpp) >> - Would you know what is the right hg clone command so we are >> working on the same code base? >> > > We use jdk10-hs, e.g. > hg tclone http://hg.openjdk.java.net/jdk10/hs 10-hs > > There is sporadic big merges going from jdk9->jdk10->jdk10-hs and > jdk10-hs->jdk10, so 10 is moving... > > >> 2) You mentioned I was using os::malloc, new, NEW_C_HEAP_ARRAY; I >> cleaned out the os::malloc but which of the new vs NEW_C_HEAP_ARRAY should >> I use. It might be that I don't understand when one uses one or the other >> but I see both used around the code base? >> - Is it that new is to be used for anything internal and >> NEW_C_HEAP_ARRAY anything provided to the JVMTI users outside of the JVM? >> > > We overload new operator when you extend correct base class, e.g. > CHeapObj so use 'new' > But for arrays you will need the macro NEW_C_HEAP_ARRAY. > > >> 3) Casts: same kind question: which should I use. The code was using a >> bit of everything, I'll refactor it entirely but I was not clear if I >> should go to C casts or C++ casts as I see both in the codebase. What is >> the convention I should use? >> > > Just be consist, use what suites you, C++ casts might be preferable, if we > are moving towards C++11. > And use 'right' cast, e.g. going from Thread* to JavaThread* you should > use C cast or static_cast, not reinterpret_cast I would say. > > >> Final notes on this webrev: >> - I am still missing: >> - Putting a TLAB implementation so that we can compare both webrevs >> - Have not tried to circumvent AsyncGetCallTrace >> - Putting in the handling of GC'd objects >> - Fix a stack walker issue I have seen, I think I know the problem >> and will test that theory out for the next webrev >> >> I will work on integrating those items for the next webrev! >> > > Thanks! > > >> Thanks for your help, >> Jc >> >> Ps: I tested this on a new repo: >> >> hg clone http://hg.openjdk.java.net/jdk10/jdk10 < >> http://hg.openjdk.java.net/jdk10/jdk10> jdk10 >> ... building it >> cd test >> jtreg -nativepath:/build/linux-x86_64-normal-server >> -release/support/test/hotspot/jtreg/native/lib/ -jdk >> /linux-x86_64-normal-server-release/images/jdk >> ../hotspot/test/serviceability/jvmti/HeapMonitor/ >> >> > I'll test it out! > > /Robbin > > >> >> On Thu, May 4, 2017 at 11:21 PM, serguei.spitsyn at oracle.com > serguei.spitsyn at oracle.com> > serguei.spitsyn at oracle.com>> wrote: >> >> Robbin, >> >> Thank you for forwarding! >> I will review it. >> >> Thanks, >> Serguei >> >> >> >> On 5/4/17 02:13, Robbin Ehn wrote: >> >> Hi, >> >> To me the compiler changes looks what is expected. >> It would be good if someone from compiler could take a look at >> that. >> Added compiler to mail thread. >> >> Also adding Serguei, It would be good with his view also. >> >> My initial take on it, read through most of the code and took it >> for a ride. >> >> ############################## >> - Regarding the compiler changes: I think we need the 'TLAB end' >> trickery (mentioned by Tony P) >> instead of a separate check for sampling in fast path for the >> final version. >> >> ############################## >> - This patch I had to apply to get it compile on JDK 10: >> >> diff -r ac3ded340b35 src/share/vm/gc/shared/collect >> edHeap.inline.hpp >> --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr >> 28 14:31:38 2017 +0200 >> +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May >> 04 10:22:56 2017 +0200 >> @@ -87,3 +87,3 @@ >> // support for object alloc event (no-op most of the time) >> - if (klass() != NULL && klass()->name() != NULL) { >> + if (klass != NULL && klass->name() != NULL) { >> Thread *base_thread = Thread::current(); >> diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 >> 14:31:38 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 >> 10:22:56 2017 +0200 >> @@ -316,3 +316,3 @@ >> JavaThread *thread = reinterpret_cast> *>(Thread::current()); >> - assert(o->size() << LogHeapWordSize == byte_size, >> + assert(o->size() << LogHeapWordSize == (long)byte_size, >> "Object size is incorrect."); >> >> ############################## >> - This patch I had to apply to get it not asserting during >> slowdebug: >> >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 >> 15:15:16 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 >> 10:24:25 2017 +0200 >> @@ -32,3 +32,3 @@ >> // TODO(jcbeyler): should we make this into a JVMTI structure? >> -struct StackTraceData { >> +struct StackTraceData : CHeapObj { >> ASGCT_CallTrace *trace; >> @@ -143,3 +143,2 @@ >> StackTraceStorage::StackTraceStorage() : >> - _allocated_traces(new StackTraceData*[MaxHeapTraces]), >> _allocated_traces_size(MaxHeapTraces), >> @@ -147,2 +146,3 @@ >> _allocated_count(0) { >> + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, >> MaxHeapTraces, mtInternal); >> memset(_allocated_traces, 0, sizeof(*_allocated_traces) * >> MaxHeapTraces); >> @@ -152,3 +152,3 @@ >> StackTraceStorage::~StackTraceStorage() { >> - delete[] _allocated_traces; >> + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); >> } >> >> - Classes should extend correct base class for which type of >> memory is used for it e.g.: CHeapObj or StackObj or AllStatic >> - The style in heapMonitoring.cpp is a bit different from normal >> vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, >> os::malloc and new. >> - In jvmtiHeapTransition.hpp you use C cast instead. >> >> ############################## >> - This patch I had apply to get traces without setting an >> ?unrelated? capability >> - Should this not be a new capability? >> >> diff -r c02a5d8785bf src/share/vm/prims/forte.cpp >> --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 >> +0200 >> +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 >> +0200 >> @@ -530,6 +530,6 @@ >> >> - if (!JvmtiExport::should_post_class_load()) { >> +/* if (!JvmtiExport::should_post_class_load()) { >> trace->num_frames = ticks_no_class_load; // -1 >> return; >> - } >> + }*/ >> >> ############################## >> - forte.cpp: (I know this is not part of your changes but) >> find_jmethod_id_or_null give me NULL for my test. >> It looks like we actually want the regular jmethod_id() ? >> >> Since we are the thread we are talking about (and in same >> ucontext) and thread is in vm and have a last java frame, >> I think most of the checks done in AsyncGetCallTrace is >> irrelevant, so you should be-able to call forte_fill_call_trace_given_top >> directly. >> But since we might need jmethod_id() if possible to avoid getting >> method id NULL, >> we need some fixes in forte code, or just do the vframStream loop >> inside heapMonitoring.cpp and not use forte.cpp. >> >> Something like: >> >> if (jthread->has_last_Java_frame()) { // just to be safe >> vframeStream vfst(jthread); >> while (!vfst.at_end()) { >> Method* m = vfst.method(); >> m->jmethod_id(); >> m->line_number_from_bci(vfst.bci()); >> vfst.next(); >> } >> >> - This is a bit confusing in forte.cpp, >> trace->frames[count].lineno = bci. >> Line number should be m->line_number_from_bci(bci); >> Do the heapMonitoring suppose to trace with bci or line number? >> I would say bci, meaning we should either rename >> ASGCT_CallFrame?lineno or use another data structure which says bci. >> >> ############################## >> - // TODO(jcbeyler): remove this extra code handling the extra >> trace for >> Please fix all these TODO's :) >> >> ############################## >> - heapMonitoring.hpp: >> // TODO(jcbeyler): is this algorithm acceptable in open source? >> >> Why is this comment here? What is the implication? >> Have you tested any simpler algorithm? >> >> ############################## >> - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk >> for building the agent) >> >> ############################## >> - monitoring_period vs HeapMonitorRate, pick rate or period. >> >> ############################## >> - globals.hpp >> Why is MaxHeapTraces not settable/overridable from jvmti >> interface? That would be handy. >> >> ############################## >> - jvmtiStackTraceData + ASGCT_CallFrame memory >> Are the agent suppose to loop through and free all >> ASGCT_CallFrame? >> Wouldn't it be better with some kinda protocol, like: >> (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); >> (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); >> >> Also using another data structure that have num_traces inside it >> simplifies things. >> So I'm not convinced using the async structure is the best way >> forward. >> >> >> I have more questions, but I think it's better if you respond and >> update the code first. >> >> Thanks! >> >> /Robbin >> >> >> On 04/21/2017 11:34 PM, JC Beyler wrote: >> >> Hi all, >> >> I've added size information to the allocation sampling >> system. This allows the callback to remember the size of each sampled >> allocation. >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/> >> >> The new webrev.01 also adds the actual heap monitoring >> sampling system in files: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.cpp.patch >> > hare/vm/runtime/heapMonitoring.cpp.patch> >> and >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.hpp.patch >> > hare/vm/runtime/heapMonitoring.hpp.patch> >> >> My next step is to add the GC part to the webrev, which will >> allow users to determine what objects are live and what are garbage. >> >> Thanks for your attention and let me know if there are any >> questions! >> >> Have a wonderful Friday! >> Jc >> >> On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> wrote: >> >> Hi all, >> >> I worked on getting a few numbers for overhead and >> accuracy for my feature. I'm unsure if here is the right place to provide >> the full data, so I am just >> summarizing >> here for now. >> >> - Overhead of the feature >> >> Using the Dacapo benchmark (http://dacapobench.org/). >> My initial results are that sampling provides 2.4% with a 512k sampling, >> 512k being our default setting. >> >> - Note: this was without the tradesoap, tradebeans and >> tomcat benchmarks since they did not work with my JDK9 (issue between >> Dacapo and JDK9 it seems) >> - I want to rerun next week to ensure number stability >> >> - Accuracy of the feature >> >> I wrote a small microbenchmark that allocates from two >> different stacktraces at a given ratio. For example, 10% of stacktrace S1 >> and 90% from stacktrace >> S2. The >> microbenchmark was run 20 times, I averaged the results >> and looked for accuracy. It seems that statistically it is sound since if I >> allocated10% S1 and 90% >> S2, with a >> sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. >> >> Let me know if there are any questions on the numbers >> and if you'd like to see some more data. >> >> Note: this was done using our internal JDK8 >> implementation since the webrev provided by >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html >> > tml> >> > tml > >> does not yet contain the whole >> implementation and therefore would have been misleading. >> >> Thanks, >> Jc >> >> >> On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> wrote: >> >> Hi all, >> >> To move the discussion forward, with Chuck Rasbold's >> help to make a webrev, we pushed this: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.ht >> ml >> > tml > >> 415 lines changed: 399 ins; 13 del; 3 mod; 51122 >> unchg >> >> This is not a final change that does the whole >> proposition from the JBS entry: https://bugs.openjdk.java.net/ >> browse/JDK-8177374 >> >> > https://bugs.openjdk.java.net/browse/JDK-8177374>>; what it does show is >> parts of the implementation that is >> proposed and hopefully can start the conversation going >> as I work through the details. >> >> For example, the changes to C2 are done here for the >> allocations: http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/shar >> e/vm/opto/macro.cpp.patch >> > re/vm/opto/macro.cpp.patch> >> > re/vm/opto/macro.cpp.patch >> > re/vm/opto/macro.cpp.patch>> >> >> Hopefully this all makes sense and thank you for all >> your future comments! >> Jc >> >> >> On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> >> wrote: >> >> Hello all, >> >> This is a follow-up from Jeremy's initial email >> from last year: >> http://mail.openjdk.java.net/pipermail/serviceability-dev/20 >> 15-June/017543.html >> > 015-June/017543.html> >> > 015-June/017543.html > pipermail/serviceability-dev/2015-June/017543.html>> >> >> I've gone ahead and started working on preparing >> this and Jeremy and I went down the route of actually writing it up in JEP >> form: >> https://bugs.openjdk.java.net/browse/JDK-8171119 < >> https://bugs.openjdk.java.net/browse/JDK-8171119> >> >> I think original conversation that happened last >> year in that thread still holds true: >> >> - We have a patch at Google that we think >> others might be interested in >> - It provides a means to understand where >> the allocation hotspots are at a very low overhead >> - Since it is at a low overhead, we can >> leave it on by default >> >> So I come to the mailing list with Jeremy's >> initial question: >> "I thought I would ask if there is any interest >> / if I should write a JEP / if I should just forget it." >> >> A year ago, it seemed some thought it was a good >> idea, is this still true? >> >> Thanks, >> Jc >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ujwal.vangapally at oracle.com Wed May 17 10:42:37 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Wed, 17 May 2017 16:12:37 +0530 Subject: RFR: JDK-8175845 Provide javadoc descriptions for jdk.hotspot.agent module Message-ID: Kindly review this small change https://bugs.openjdk.java.net/browse/JDK-8175845 webrev: http://cr.openjdk.java.net/~uvangapally/webrev/2017/8175845/webrev.00/ Thanks, Ujwal. From mandy.chung at oracle.com Wed May 17 18:29:05 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 17 May 2017 11:29:05 -0700 Subject: RFR: JDK-8175845 Provide javadoc descriptions for jdk.hotspot.agent module In-Reply-To: References: Message-ID: <86B0C49A-F408-4474-BADA-4E30E485A097@oracle.com> > On May 17, 2017, at 3:42 AM, Ujwal Vangapally wrote: > > Kindly review this small change > > https://bugs.openjdk.java.net/browse/JDK-8175845 > > webrev: http://cr.openjdk.java.net/~uvangapally/webrev/2017/8175845/webrev.00/ IIUC jhsdb is a supported tool, right? Missing period at the end of the sentence; otherwise the change looks okay. Mandy From ujwal.vangapally at oracle.com Thu May 18 07:15:30 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Thu, 18 May 2017 12:45:30 +0530 Subject: RFR: JDK-8175845 Provide javadoc descriptions for jdk.hotspot.agent module In-Reply-To: <86B0C49A-F408-4474-BADA-4E30E485A097@oracle.com> References: <86B0C49A-F408-4474-BADA-4E30E485A097@oracle.com> Message-ID: Thanks for the review Mandy. Please find the new webrev incorporating the review comments webrev : http://cr.openjdk.java.net/~uvangapally/webrev/2017/8175845/webrev.01/ On 5/17/2017 11:59 PM, Mandy Chung wrote: >> On May 17, 2017, at 3:42 AM, Ujwal Vangapally wrote: >> >> Kindly review this small change >> >> https://bugs.openjdk.java.net/browse/JDK-8175845 >> >> webrev: http://cr.openjdk.java.net/~uvangapally/webrev/2017/8175845/webrev.00/ > IIUC jhsdb is a supported tool, right? yes jhsdb is a supported tool > > Missing period at the end of the sentence; otherwise the change looks okay. added missing period > > Mandy -Ujwal From serguei.spitsyn at oracle.com Thu May 18 10:20:30 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 18 May 2017 03:20:30 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> Message-ID: <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> An HTML attachment was scrubbed... URL: From robbin.ehn at oracle.com Thu May 18 11:05:51 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Thu, 18 May 2017 13:05:51 +0200 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> Message-ID: <6320d430-768b-ba3f-c726-04127f939071@oracle.com> Hi of those: > One more alternate solution to suggest is to add new function: > jdwpTransportError SetTransportConfiguration(jdwpTransportConfiguration config); > > Where: > typedef struct { > const char* allowed_peers; > } jdwpTransportConfiguration; > > > This approach allows to extend the jdwpTransportConfiguration in the future if necessary. +1 It would nice if a compiled client against JDK 8 headers works without recompiling with a JDK 9 runtime. To make really future proof; using a client compiled against JDK 9 with JDK 1X you will need some way to know witch version of the struct was used. E.g. SetTransportConfiguration(&config, JDWP_TRANSPORT_CONFIGURATION_CURRENT_VERSION); Thanks, Robbin! From dmitry.samersoff at oracle.com Thu May 18 11:33:26 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 18 May 2017 14:33:26 +0300 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> Message-ID: Serguei, *a* > 4. The suggested *allow* feature is too destructive for the > transport API. It makes the original function StartListening unusable > and its entry slot a garbage. It will become a big mess if we add > more function clones like StartListening11. . . . 6. The suggested > API update makes the transport API VERSION_1_1 incompatible with the > initial VERSION_1_0. Transport 1.0 uses StartListening slot. Transport 1.1 uses StartListeningWithAllow slot. I don't see any difference between this approach and approach of adding extra slot and extra function. Transport 1.0 will continue to work as is (ever without recompilation). Transport 1.1 will crash if it doesn't fill proper slot. *b* I'm against separate Allow() call because: 1) Socket communication steps is well known. Extra step (call Allow before StartListen) is not natural. 2) API have to clear visible and self documenting. If transport developer doesn't implement Allow() they doesn't see any sign of mistake until they starts testing the transport with allow parameter. As opposite unused parameter is clear visible and cause compiler warning. 3) In a future, parsing of allow list might require results of socket call (e.g. to distinguish between IPv4 only and IPv4/Ipv6 cases) *c* As for configuration structures, any jdwpConfiguration structure requires versioning i.e. see webrev.15. But we decided to go in other direction. -Dmitry On 2017-05-18 13:20, serguei.spitsyn at oracle.com wrote: > Hi Dmitry, > > Let's get to a consensus for the last item on our plate: > jdwpTransportError AllowPeers(const char* peers); .VS. > jdwpTransportError StartListeningWithAllow(const char* address, > char** actual_address, char *allow); > > > I still do not like the StartListening() function clone and its > name. It looks like a wrong direction to me. > > This is a fragment of my bug report comment on this issue: > > 4. The suggested *allow* feature is too destructive for the > transport API. It makes the original function StartListening unusable > and its entry slot a garbage. It will become a big mess if we add > more function clones like StartListening11. . . . 6. The suggested > API update makes the transport API VERSION_1_1 incompatible with the > initial VERSION_1_0. > > > One more alternate solution to suggest is to add new function: > jdwpTransportError > SetTransportConfiguration(jdwpTransportConfiguration config); > > Where: typedef struct { const char* allowed_peers; } > jdwpTransportConfiguration; > > > This approach allows to extend the jdwpTransportConfiguration in the > future if necessary. However, I'm not sure we really need this > ability. > > > Both alternatives allow for the transport library to support both > API versions. It is good by itself and also, it is a way to simplify > the transport library. > > We could formulate the following requirements to the API updates: - > the transport API update must support previous API versions - the > updated API layout must be compatible with previous versions > > It seems to me, these requirements are achievable and two alternate > approaches show it. > > Please, share your opinions? > > > Thanks, Serguei > > > > On 5/12/17 02:10, serguei.spitsyn at oracle.com wrote: >> Hi Dmitry, >> >> Thank you for the update with some review comments resolved! I >> looked at it but will not provide my comments at this point. We >> will need another update according to the recent design review and >> items where we still have to reach an agreement. >> >> Thanks, Serguei >> >> >> On 5/10/17 08:27, Dmitry Samersoff wrote: >>> Serguei, >>> >>> Fixed minor issues (comments, netmask etc). Added an error for >>> attempt to use allow with an old transport. >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.17/ >>> >>> see also below. >>> >>> >>>> I do not understand this reason for adding more complexity. It >>>> seems, there should not be any issues in the future with >>>> rejecting all unsupported versions by the transport library. >>> Added a diagram explaining transport version negotiation to CR. I >>> use future versions (1.3; 1.4; 1.5) because all this versioning >>> staff has a long term goal and allow us to develop better >>> transport without breaking existing one. >>> >>>> We are not adding extra parameter, we are introducing new >>>> function that is a clone of another StartListening function >>>> with a version suffix '11' in its name and with an extra >>>> parameter. >>> Correct. We changed behavior of StartListening function and 1.1 >>> transport shouldn't care about old one. >>> >>> i.e. when we document 1.1 interface we describe the only >>> function StartListening(env, address, actualAddress, allow) that >>> have to be placed to the StartListening11 slot. >>> >>> >>> Back in 2015 I proposed to separate interfaces entirely (see >>> webrev.04), but we (Alan?) decided that it's an overkill. >>> >>>> The original StartListening function is being removed. It is >>>> much simpler to introduce new function AllowPeers(char* peers) >>>> with the same parameter. >>> This separate function have to be called explicitly before we >>> start listening, It is extra communication step. IMHO, not >>> obvious one. >>> >>> So I would prefer to keep StartListening11 >>> >>> -Dmitry >>> >>> >>> On 2017-05-10 12:37, serguei.spitsyn at oracle.com wrote: >>>> Dmitry, >>>> >>>> Thank you a lot for the detailed reply! >>>> >>>> >>>> On 5/10/17 01:10, Dmitry Samersoff wrote: >>>>> Serguei, >>>>> >>>>> Please see my comments in-line. >>>>> >>>>> >>>>> On 2017-05-10 00:42, serguei.spitsyn at oracle.com wrote: >>>>>> Hi Dmitry, >>>>>> >>>>>> >>>>>> I'd like to resolve my questions before the upcoming >>>>>> design discussion on Thu. >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c.udiff.html >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> (0) The design description from the bug report tells: >>>>>> >>>>>>> Than we change a negotiation protocol between JDWP and >>>>>> transport. >>>>>>> We pass maximal supported version to transport >>>>>>> initialization >>>>>> routine and expect transport actual version to be >>>>>> returned. >>>>>> >>>>>> The modified negotiation protocol adds extra complexity. >>>>>> What is a motivation behind this? Is it really necessary >>>>>> for the transport library to return an actual version >>>>>> instead of rejecting the unmatched version? I do not see it >>>>>> is really used in the webrev.15 implementation. >>>>> Transport have to return it's actual version in order to >>>>> allow agent to perform appropriate action. >>>>> >>>>> see libjdwp/transport.c:526 >>>> This requirement adds extra complexity to the rules (transport >>>> negotiation protocol). It is not really necessary. The >>>> loadTransport() already does a lookup of a version that is >>>> accepted (not rejected) by the transport library and can save >>>> that version. The transport_startTransport() then should use >>>> the version found by the loadTransport(). >>>> >>>> >>>>> Today it's just a selection of proper API call but in a >>>>> future it might be too-old-transport-error or deprecation >>>>> warning or security warning or something else. >>>> I do not understand this reason for adding more complexity. It >>>> seems, there should not be any issues in the future with >>>> rejecting all unsupported versions by the transport library. >>>> However, it will be even more simple if one transport library >>>> API could support/accept all possible versions (see my >>>> alternate suggestion below). >>>> >>>> >>>>>> (1) The following change in the jdwp transport library will >>>>>> reject theJDWPTRANSPORT_VERSION_1_0 as it is below the >>>>>> version JDWPTRANSPORT_VERSION_1_1, but will except any >>>>>> version above the JDWPTRANSPORT_VERSION_1_1 (with >>>>>> providing/returning the actual transport version): >>>>>> >>>>>> jdwpTransport_OnLoad(JavaVM *vm, jdwpTransportCallback* >>>>>> cbTablePtr, - jint version, jdwpTransportEnv** result) + >>>>>> jint version, void** env) { - if (version != >>>>>> JDWPTRANSPORT_VERSION_1_0) { + if (version < >>>>>> JDWPTRANSPORT_VERSION_1_1) { return JNI_EVERSION; } >>>>>> >>>>>> >>>>>> Te following change will also prevent supporting the 1_0 >>>>>> version of the transport library: >>>>>> >>>>>> - interface.StartListening = >>>>>> &socketTransport_startListening; + interface.StartListening >>>>>> = NULL; >>>>> libdt_socket/socketTransport.c is an implementation of 1.1 >>>>> *transport* it's not intended to run with 1.0 *backend*. >>>> Why not? It would simplifies things if the transport library >>>> (and its API) is backward compatible. >>>> >>>>> i.e. 1.1 *backend* can run 1.1 and 1.0 transports but 1.1 >>>>> *transport* require 1.1 or greater backend. >>>> This statement creates a confusion. The truce is that the >>>> transport library can support some number of versions. The >>>> latest supported version can satisfy the agent if it supports >>>> it. >>>> >>>>> see: libjdwp/transport.c:206 for version logic on backend >>>>> (agent) side >>>> The logic at L206 does not require the transport library to >>>> return its version. It will work Ok if the library rejects >>>> unsupported versions. >>>> >>>> >>>>>> I'd suggest the following alternate change to the transport >>>>>> API allowing to support both old and new versions at the >>>>>> same time (it would simplify the negotiation rules): - Add >>>>>> new function: jdwpTransportError AllowPeers(const char* >>>>>> peers); >>>>>> >>>>>> - Keep the original StartListening function. The function >>>>>> uses the allowed peers data if it was previously cached by >>>>>> the AllowPeers(). >>>>>> >>>>>> - It seems, the alternate approach does not require adding >>>>>> the extra_data with version. But if there is still a real >>>>>> need to get the transport API version then it'd better to >>>>>> introduce a function named GetTransportVersion() or >>>>>> JDWP_TransportVersion(). This would allow to encapsulate >>>>>> any extra_data that is necessary in such a case. >>>>> From pure JDWP hardening point of view we can just add extra >>>>> parameter *allow* to existing StartListening(). Caller is >>>>> responsible for stack consistency so old transport continues >>>>> to work. >>>> We are not adding extra parameter, we are introducing new >>>> function that is a clone of another StartListening function >>>> with a version suffix '11' in its name and with an extra >>>> parameter. The original StartListening function is being >>>> removed. It is much simpler to introduce new function >>>> AllowPeers(char* peers) with the same parameter. The original >>>> StartListening function works as before. The updated API can >>>> support both versions 1_0 and 1_1. >>>> >>>> >>>>> But my goal was to create a versioning in JDWP agent -> >>>>> transport relations that was missed in initial JDWP design. >>>>> Further, more complicated, changes (IPv6 support, UDS sockets >>>>> support etc) require this logic. >>>> Would introducing function JdwpTransportVersion() achieve what >>>> you wanted? >>>> >>>> >>>>> We have a structure jdwpTransportNativeInterface with a fixed >>>>> set of functions (see jdwpTransport.h:153). To add any new >>>>> function to this structure we have to create a method to >>>>> detect presence of this function. So we can't use >>>>> GetTransportVersion(). >>>> It is not clear why would you need such a method for any new >>>> function? The transport version maps to the whole set of >>>> functions. >>>> >>>> >>>>> With as separate AllowPeer() function nobody remind an agent >>>>> writer that they should use it, but extra parameter makes api >>>>> changes and requirements clear visible (up to compiler >>>>> warning). >>>> I do not see any difference. No compiler warning if NULL is >>>> passed in place of the 'allow' parameter. >>>> >>>> >>>>> Also I'm against of changing behavior of existing function. >>>>> >>>>> >>>>>> (2) The following error messages miss the actual IP address >>>>>> or mask that was found to be illegal: >>>>>> >>>>>> 383 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, >>>>>> "invalid ip address for allow"); 392 >>>>>> RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, >>>>>> "invalid netmask for allow"); >>>>> Other parameter parsing functions (e.g. "invalid port number >>>>> specified" at 274) doesn't explain what parameter is bad. >>>> It is not good either. But new functionality is added, so the >>>> more diagnostic details the better. >>>> >>>>> I think typical allow would have one or two entry, so >>>>> verbose error message is not worth significant complication >>>>> of parsing code. >>>> It is still better to print what was not accepted. It should >>>> not be a problem to print it anyway. >>>> >>>> >>>>> I would prefer to leave it as is. >>>>> >>>>> >>>>>> (3) A suggestion on the following: >>>>>> >>>>>> 377 uint32_t mask = 0xFFFFFFFF; 400 mask = 0xFFFFFFFF; // >>>>>> reset mask >>>>>> >>>>>> I'd suggest a more explicit approach instead of the L400 >>>>>> for a better clarity: >>>>>> >>>>>> 386 if (*s == '/') { 387 // netmask specified 388 s = >>>>>> mask_s2u(s + 1, &mask); 389 if (*(s-1) == '/') { 390 // >>>>>> Input is not consumed, something bad happens 391 _peers_cnt >>>>>> = 0; 392 RETURN_ERROR(JDWPTRANSPORT_ERROR_ILLEGAL_ARGUMENT, >>>>>> "invalid netmask for allow"); 393 } 394 } else { mask = >>>>>> 0xFFFFFFFF; } >>>>> I'll try it. >>>> Ok, thanks. >>>> >>>> >>>>>> http://cr.openjdk.java.net/%7Edsamersoff/JDK-8061228/webrev.15/src/jdk.jdwp.agent/share/native/libjdwp/transport.c.udiff.html >>>>>> >>>>>> >>>>>> >>>>>> (4) Some confusion with the fragment and its comment: >>>>>> >>>>>> + + /* Pass the latest supported version, + * expect actual >>>>>> transport version in t->extra_data->version + */ + ver = >>>>>> (*onLoad)(jvm, &callback, JDWPTRANSPORT_VERSION_1_1, &t); + >>>>>> if (ver == JNI_EVERSION) { ver = (*onLoad)(jvm, &callback, >>>>>> JDWPTRANSPORT_VERSION_1_0, &t); + // Special handling for >>>>>> versionless transports + info->transportVersion = >>>>>> JDWPTRANSPORT_VERSION_1_0; + } + else { + >>>>>> info->transportVersion = (*t)->extra_data->version; + } + >>>>>> >>>>>> The term "latest supported version" is ambiguous in this >>>>>> context. Do you mean, supported by the JDWP back-end or by >>>>>> the agent library? >>>>> Supported by the JDWP backend. I'll update comments. >>>> Ok, thanks. >>>> >>>> >>>>>> Also, it is not clear in what circumstances the agent >>>>>> library would support the version 1_0 only. The JDWP >>>>>> back-end is always coupled with the socket transport >>>>>> library, isn't it? Is it because the libdt_shmem library >>>>>> can be used on Windows or because a different native >>>>>> library path can be used? Could you explain a little bit? >>>>> It's more generic question: Do we need any backward >>>>> compatibility here? I assume *yes* (can't recall why - >>>>> probably it was a tree-year old decision). >>>>> >>>>> If we state that new backend will not support old transports >>>>> today and in a future - all versioning logic is not >>>>> necessary. >>>>> >>>>> >>>>>> As we see in (1) above the actual transport version can be >>>>>> different from requested only if the requested version is >>>>>> above the latest supported by the transport library. >>>>>> Otherwise, the version is matched or the JNI_EVERSION is >>>>>> returned. The subsequent call to the OnLoad function can >>>>>> succeed only if the library supports just the version 1_0. >>>>> See answer to (1) above. transport library has exactly one >>>>> version but JDWP backend supports couple of transport library >>>>> versions back. >>>> I've replied above too. :) >>>> >>>> >>>>>> (5) Memory allocation for the info->allow field is needed >>>>>> only for the transport version 1_1: >>>>>> >>>>>> + if (allow != NULL) { + info->allow = >>>>>> jvmtiAllocate((int)strlen(allow)+1); + if (info->allow == >>>>>> NULL) { + serror = JDWP_ERROR(OUT_OF_MEMORY); + goto >>>>>> handleError; + } + (void)strcpy(info->allow, allow); + } >>>>> Correct. allocation needed for 1.1 and *greater*. I can >>>>> change it, but it makes code less readable. >>>> The same fragment in a different place should not be less >>>> readable, maybe just less elegant. >>>> >>>> >>>>>> (6) There is no handling for the condition when the >>>>>> *allow* parameter is passed but the transport version >>>>>> is 1_0 (which does not support the *allow* parameter): >>>>> Correct. Warning or ever error should be here. >>>>> >>>>> I plan to open a separate follow-up CR for this problem - we >>>>> have to decide how we should handle this situation (warning >>>>> or error) and change the code, >>>>> >>>>> but I can add a plain printf() here if you like. >>>> I'm Ok with both error or warning but what is needed from a >>>> security point of view? We can avoid filing a separate >>>> follow-up CR in this case. Should it be an error if the *allow* >>>> parameter is used with the old transport library that does not >>>> support it? >>>> >>>> >>>> Thanks, Serguei >>>> >>>> >>>>>> + /* Interface version 1.0 doesn't support versioning, so >>>>>> we have to + * use global variable and set the version >>>>>> artifically. + * Use (*t)->extra_data->version directly >>>>>> when 1.0 is gone. + */ + switch(info->transportVersion) { + >>>>>> case JDWPTRANSPORT_VERSION_1_0: err = >>>>>> (*trans)->StartListening(trans, address, &retAddress); + >>>>>> break; + case JDWPTRANSPORT_VERSION_1_1: + err = >>>>>> (*trans)->StartListening11(trans, address, &retAddress, >>>>>> info->allow); + break; + default: + err = JNI_EVERSION; + >>>>>> } >>>>> -Dmitry >>>>> >>>>> >>>>>> Thanks, Serguei On 3/29/17 08:10, Dmitry Samersoff wrote: >>>>>>> Robbin, >>>>>>> >>>>>>>> One follow-up issue is that if you start suspended and >>>>>>>> than connect with an unallowed client the JVM starts >>>>>>>> and executes the program. >>>>>>> Fixed. >>>>>>> >>>>>>> see >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.15 >>>>>>> >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> On 2017-03-16 15:59, Robbin Ehn wrote: >>>>>>>> Hi Dmitry, thanks for the update. >>>>>>>> >>>>>>>> One follow-up issue is that if you start suspended and >>>>>>>> than connect with an unallowed client the JVM starts >>>>>>>> and executes the program. Simple program prints >>>>>>>> "Hello". >>>>>>>> >>>>>>>> [rehn at rehn-ws vanilla-hs]$ java >>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:9999,allow=1.2.3.0/32 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -cp . H >>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>> ERROR: Peer not allowed to connect Listening for >>>>>>>> transport dt_socket at address: 9999 Hello >>>>>>>> >>>>>>>> I think it would be good if the VM stayed suspended >>>>>>>> when an unallowed client connects. >>>>>>>> >>>>>>>> Thanks, Robbin >>>>>>>> >>>>>>>> On 03/13/2017 08:07 PM, Dmitry Samersoff wrote: >>>>>>>>> Robbin, >>>>>>>>> >>>>>>>>> Please, see: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.11 >>>>>>>>> >>>>>>>>>> >>>>>>>>> 1: >>>>>>>>>> So connecting with an unallowed client terminates >>>>>>>>>> the VM. >>>>>>>>> Fixed. >>>>>>>>> >>>>>>>>>> 2: Starting with an bad allow filter terminates the >>>>>>>>>> VM when connecting a client. >>>>>>>>> Moved allowed parameter (and parser call) to >>>>>>>>> StartListening. >>>>>>>>> >>>>>>>>> -Dmitry >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2017-03-10 15:56, Robbin Ehn wrote: >>>>>>>>>> Hi Dmitry, >>>>>>>>>> >>>>>>>>>> I took a look at this, I have two practical >>>>>>>>>> issues: >>>>>>>>>> >>>>>>>>>> 1: [rehn at rehn-ws dev]$ java >>>>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.6.6.6 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -cp runs ForEver >>>>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>>>> ERROR: transport error 202: peer not allowed to >>>>>>>>>> connect: Success JDWP exit error >>>>>>>>>> JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>>>> fatal error [transport.c:358] >>>>>>>>>> >>>>>>>>>> So connecting with an unallowed client terminates >>>>>>>>>> the VM. >>>>>>>>>> >>>>>>>>>> 2: [rehn at rehn-ws dev]$ java >>>>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:9999,allow=6.BAD.6.6 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -cp runs ForEver >>>>>>>>>> Listening for transport dt_socket at address: 9999 >>>>>>>>>> ERROR: transport error 202: unable to parse list of >>>>>>>>>> allowed peers: Success JDWP exit error >>>>>>>>>> JVMTI_ERROR_NONE(0): could not connect, timeout or >>>>>>>>>> fatal error [transport.c:358] >>>>>>>>>> >>>>>>>>>> Starting with an bad allow filter terminates the VM >>>>>>>>>> when connecting a client. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Connecting with an unallowed ip/port should not >>>>>>>>>> terminate the VM and we should verify allow filter >>>>>>>>>> directly at startup. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> /Robbin >>>>>>>>>> >>>>>>>>>> On 02/28/2017 10:41 AM, Dmitry Samersoff wrote: >>>>>>>>>>> Everybody, >>>>>>>>>>> >>>>>>>>>>> Please review: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8061228/webrev.10/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> These changes introduce new parameter[1] of the socket >>>>>>>>>>> transport - allow. Users can explicitly specify a >>>>>>>>>>> list of hosts that allowed to connect to jdwp >>>>>>>>>>> server and it's the second part of JDWP >>>>>>>>>>> hardening[2]. >>>>>>>>>>> >>>>>>>>>>> No restrictions are applied by default now but >>>>>>>>>>> I'll file a separate CR to restrict list of >>>>>>>>>>> allowed peers to localhost by default. >>>>>>>>>>> >>>>>>>>>>> Also these changes implement versioning for jdwp >>>>>>>>>>> transport and therefor simplify feature >>>>>>>>>>> development of jdwp. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 1. Example command line: >>>>>>>>>>> >>>>>>>>>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n, >>>>>>>>>>> >>>>>>>>>>> address=*,allow="127.0.0.0/8;192.168.0.0/24" >>>>>>>>>>> >>>>>>>>>>> Possible values for allow parameter: * >>>>>>>>>>> - accept connections from everywhere. N.N.N.N >>>>>>>>>>> - accept connections from this IP address only >>>>>>>>>>> N.N.N.N/nn - accept connections from particular >>>>>>>>>>> ip subnet >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2. JDK-8052136 JDWP hardening >>>>>>>>>>> >>>>>>>>>>> -Dmitry >>>>>>>>>>> >>> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Thu May 18 12:07:59 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 18 May 2017 15:07:59 +0300 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <6320d430-768b-ba3f-c726-04127f939071@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> <6320d430-768b-ba3f-c726-04127f939071@oracle.com> Message-ID: <671c35b8-f5f8-d642-3e40-8b7a9f9cf7fb@oracle.com> Robbin, Please, see my answer to Serguei. -Dmitry On 2017-05-18 14:05, Robbin Ehn wrote: > Hi of those: > >> One more alternate solution to suggest is to add new function: >> jdwpTransportError >> SetTransportConfiguration(jdwpTransportConfiguration config); >> >> Where: >> typedef struct { >> const char* allowed_peers; >> } jdwpTransportConfiguration; >> >> >> This approach allows to extend the jdwpTransportConfiguration in the >> future if necessary. > > +1 > > It would nice if a compiled client against JDK 8 headers works without > recompiling with a JDK 9 runtime. > To make really future proof; using a client compiled against JDK 9 with > JDK 1X you will need some way to know witch version of the struct was used. > > E.g. SetTransportConfiguration(&config, > JDWP_TRANSPORT_CONFIGURATION_CURRENT_VERSION); > > Thanks, Robbin! -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mandy.chung at oracle.com Thu May 18 16:11:21 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 18 May 2017 09:11:21 -0700 Subject: RFR: JDK-8175845 Provide javadoc descriptions for jdk.hotspot.agent module In-Reply-To: References: <86B0C49A-F408-4474-BADA-4E30E485A097@oracle.com> Message-ID: <833CFC89-408A-42D9-9C6E-259C36BD0925@oracle.com> > On May 18, 2017, at 12:15 AM, Ujwal Vangapally wrote: > > Thanks for the review Mandy. > > Please find the new webrev incorporating the review comments > > webrev : http://cr.openjdk.java.net/~uvangapally/webrev/2017/8175845/webrev.01/ Looks okay. Mandy From Alan.Bateman at oracle.com Thu May 18 20:29:07 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 18 May 2017 21:29:07 +0100 Subject: VisualVM CPU sampling not working In-Reply-To: <4F9BC98B-53C8-4B48-8380-C9BFC7A12C74@cbfiddle.com> References: <97CDC8E9-A588-4DA7-B5AF-289D8402505D@cbfiddle.com> <4F9BC98B-53C8-4B48-8380-C9BFC7A12C74@cbfiddle.com> Message-ID: On 18/05/2017 20:39, Alan Snyder wrote: > As far as I can tell, the latest VisualVM (1.3.9) is unable to do CPU sampling or profiling on the latest EA build (9-ea+169-jigsaw-nightly-h6406-20170517). > > Is this really the case, or am I missing something? > > Is someone working on this? I took a quite look. As Mandy said in the previous mail, the JMX interop issue between JDK 8 and JDK 9 is JDK-8167121. This is fixed in jdk8u-dev but hasn't got into a released JDK 8 update yet. So as a test, I ran Visual VM 1.3.9 on a JDK 9 build to attach to an application running on the same JDK 9 build. As VisualVM (or maybe the NetBeans framework that it builds on) makes use of a number of JDK internal APIs. It has encapsulation busting command line options configured in etc/visualvm.conf but the module names aren't quite right - specifically, jdk.jvmstat was renamed to jdk.internal.jvmstat a couple of builds ago. I fixed those and it started up okay. I tried the Monitor, Threads, and Sampler tabs and they seem to work okay. -Alan From mandy.chung at oracle.com Thu May 18 20:45:49 2017 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 18 May 2017 13:45:49 -0700 Subject: VisualVM CPU sampling not working In-Reply-To: References: <97CDC8E9-A588-4DA7-B5AF-289D8402505D@cbfiddle.com> <4F9BC98B-53C8-4B48-8380-C9BFC7A12C74@cbfiddle.com> Message-ID: > On May 18, 2017, at 1:29 PM, Alan Bateman wrote: > > On 18/05/2017 20:39, Alan Snyder wrote: > >> As far as I can tell, the latest VisualVM (1.3.9) is unable to do CPU sampling or profiling on the latest EA build (9-ea+169-jigsaw-nightly-h6406-20170517). >> >> Is this really the case, or am I missing something? >> >> Is someone working on this? > I took a quite look. > > As Mandy said in the previous mail, the JMX interop issue between JDK 8 and JDK 9 is JDK-8167121. This is fixed in jdk8u-dev but hasn't got into a released JDK 8 update yet. > > So as a test, I ran Visual VM 1.3.9 on a JDK 9 build to attach to an application running on the same JDK 9 build. As VisualVM (or maybe the NetBeans framework that it builds on) makes use of a number of JDK internal APIs. It has encapsulation busting command line options configured in etc/visualvm.conf but the module names aren't quite right - specifically, jdk.jvmstat was renamed to jdk.internal.jvmstat a couple of builds ago. I fixed those and it started up okay. I tried the Monitor, Threads, and Sampler tabs and they seem to work okay. VisualVM 1.3.9 was released on Oct 2016. It has to be updated to work with the latest JDK 9 release. You may reference the ?-add-opens options used by NetBeans Dev version (under etc/netbeans.conf) Mandy From matthias.baesken at sap.com Tue May 16 10:21:57 2017 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Tue, 16 May 2017 10:21:57 +0000 Subject: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent In-Reply-To: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> References: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> Message-ID: <444e21a8d38c42338fa5c38093a32e9c@sap.com> Sure, I forward it to serviceability-dev . -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Dienstag, 16. Mai 2017 11:51 To: Baesken, Matthias ; core-libs-dev at openjdk.java.net Cc: Simonis, Volker Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent On 16/05/2017 10:04, Baesken, Matthias wrote: > Hello, could you please review this small change : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8180413/ > > it fixes a number of places in jdk.jdwp.agent where in case of an error it is attempted to write to NULL . > > Bug : JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent > > > https://bugs.openjdk.java.net/browse/JDK-8180413 > > Can you bring this to serviceability-dev as that is the mailing list where the JDWP agent is maintained? -Alan From dmitry.samersoff at oracle.com Tue May 16 11:17:32 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 16 May 2017 14:17:32 +0300 Subject: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent In-Reply-To: <444e21a8d38c42338fa5c38093a32e9c@sap.com> References: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> <444e21a8d38c42338fa5c38093a32e9c@sap.com> Message-ID: <8c96fd99-cdb2-dc78-7c35-f46d5d181d91@oracle.com> Matthias, Looks good to me. -Dmitry On 2017-05-16 13:21, Baesken, Matthias wrote: > Sure, I forward it to serviceability-dev . > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Dienstag, 16. Mai 2017 11:51 > To: Baesken, Matthias ; core-libs-dev at openjdk.java.net > Cc: Simonis, Volker > Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent > > > > On 16/05/2017 10:04, Baesken, Matthias wrote: >> Hello, could you please review this small change : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8180413/ >> >> it fixes a number of places in jdk.jdwp.agent where in case of an error it is attempted to write to NULL . >> >> Bug : JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent >> >> >> https://bugs.openjdk.java.net/browse/JDK-8180413 >> >> > Can you bring this to serviceability-dev as that is the mailing list > where the JDWP agent is maintained? > > -Alan > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From javalists at cbfiddle.com Thu May 18 19:39:01 2017 From: javalists at cbfiddle.com (Alan Snyder) Date: Thu, 18 May 2017 12:39:01 -0700 Subject: VisualVM CPU sampling not working In-Reply-To: References: <97CDC8E9-A588-4DA7-B5AF-289D8402505D@cbfiddle.com> Message-ID: <4F9BC98B-53C8-4B48-8380-C9BFC7A12C74@cbfiddle.com> As far as I can tell, the latest VisualVM (1.3.9) is unable to do CPU sampling or profiling on the latest EA build (9-ea+169-jigsaw-nightly-h6406-20170517). Is this really the case, or am I missing something? Is someone working on this? I would think that performance analysis would be important during the EA period. Is there some other tool I should be using that works? Alan P.S. No response on serviceability-dev? > On May 2, 2017, at 5:39 PM, Mandy Chung wrote: > > serviceability-dev (cc?ed) is the proper mailing list for this issue > and so bcc jdk9-dev. > > https://bugs.openjdk.java.net/browse/JDK-8167121 has been pushed > to jdk8u-dev. We will need to get this fix in a 8u release. > > Mandy > >> On May 2, 2017, at 2:15 PM, Alan Snyder wrote: >> >> Is this a known problem? It resembles JDK-8165005. Makes it hard to investigate performance problems if the tools don't work. Are there other tools that work? >> >> This is VisualVM 1.39 on an application running under jdk9-ea+166. >> >> It reports: >> >> Cannot access threads in target application. >> >> The log file shows: >> >> java.lang.IllegalArgumentException: Unexpected composite type for ThreadInfo >> at sun.management.ThreadInfoCompositeData.validateCompositeData(ThreadInfoCompositeData.java:372) >> at sun.management.ThreadInfoCompositeData.getInstance(ThreadInfoCompositeData.java:68) >> at java.lang.management.ThreadInfo.(ThreadInfo.java:263) >> at java.lang.management.ThreadInfo.from(ThreadInfo.java:794) >> >> Caused: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) >> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1018) >> >> Caused: java.io.InvalidObjectException: Failed to invoke from(CompositeData) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory.invalidObjectException(DefaultMXBeanMappingFactory.java:1457) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeBuilderViaFrom.fromCompositeData(DefaultMXBeanMappingFactory.java:1021) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeMapping.fromNonNullOpenValue(DefaultMXBeanMappingFactory.java:919) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.fromOpenValue(DefaultMXBeanMappingFactory.java:133) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$ArrayMapping.fromNonNullOpenValue(DefaultMXBeanMappingFactory.java:584) >> at com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$NonNullMXBeanMapping.fromOpenValue(DefaultMXBeanMappingFactory.java:133) >> at com.sun.jmx.mbeanserver.ConvertingMethod.fromOpenReturnValue(ConvertingMethod.java:131) >> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:168) >> at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258) >> >> Caused: java.lang.reflect.UndeclaredThrowableException >> at com.sun.proxy.$Proxy16.dumpAllThreads(Unknown Source) >> at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.dumpAllThreads(ThreadInfoProvider.java:103) >> >> [catch] at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.initialize(ThreadInfoProvider.java:88) >> at com.sun.tools.visualvm.sampler.cpu.ThreadInfoProvider.(ThreadInfoProvider.java:54) >> at com.sun.tools.visualvm.sampler.SamplerImpl$11.run(SamplerImpl.java:485) >> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423) >> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033) >> > From serguei.spitsyn at oracle.com Fri May 19 00:17:40 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 18 May 2017 17:17:40 -0700 Subject: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent In-Reply-To: <444e21a8d38c42338fa5c38093a32e9c@sap.com> References: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> <444e21a8d38c42338fa5c38093a32e9c@sap.com> Message-ID: <520c56a8-4379-e4b0-42ac-acde1939cc7b@oracle.com> Hi Matthias, The fix looks good to me. It must be tested at least with the JTreg com/sun/jdi tests. Do you need a sponsor? Thanks, Serguei On 5/16/17 03:21, Baesken, Matthias wrote: > Sure, I forward it to serviceability-dev . > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Dienstag, 16. Mai 2017 11:51 > To: Baesken, Matthias ; core-libs-dev at openjdk.java.net > Cc: Simonis, Volker > Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent > > > > On 16/05/2017 10:04, Baesken, Matthias wrote: >> Hello, could you please review this small change : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8180413/ >> >> it fixes a number of places in jdk.jdwp.agent where in case of an error it is attempted to write to NULL . >> >> Bug : JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent >> >> >> https://bugs.openjdk.java.net/browse/JDK-8180413 >> >> > Can you bring this to serviceability-dev as that is the mailing list > where the JDWP agent is maintained? > > -Alan From serguei.spitsyn at oracle.com Fri May 19 06:27:20 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 18 May 2017 23:27:20 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <6320d430-768b-ba3f-c726-04127f939071@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> <6320d430-768b-ba3f-c726-04127f939071@oracle.com> Message-ID: <2644fa90-989b-5b2a-1ee3-3c2b1f306498@oracle.com> On 5/18/17 04:05, Robbin Ehn wrote: > Hi of those: > >> One more alternate solution to suggest is to add new function: >> jdwpTransportError >> SetTransportConfiguration(jdwpTransportConfiguration config); >> >> Where: >> typedef struct { >> const char* allowed_peers; >> } jdwpTransportConfiguration; >> >> >> This approach allows to extend the jdwpTransportConfiguration in the >> future if necessary. > > +1 > > It would nice if a compiled client against JDK 8 headers works without > recompiling with a JDK 9 runtime. > To make really future proof; using a client compiled against JDK 9 > with JDK 1X you will need some way to know witch version of the struct > was used. > > E.g. SetTransportConfiguration(&config, > JDWP_TRANSPORT_CONFIGURATION_CURRENT_VERSION); No need to have a separate transport configuration versioning. It is enough to have just a transport API versioning. Thanks, Serguei > > Thanks, Robbin! From serguei.spitsyn at oracle.com Fri May 19 07:04:38 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 19 May 2017 00:04:38 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> Message-ID: <1f90a1f7-bc1e-f635-ff11-bdf0cd0c6d41@oracle.com> Dmitry, On 5/18/17 04:33, Dmitry Samersoff wrote: > Serguei, > > *a* > >> 4. The suggested *allow* feature is too destructive for the >> transport API. It makes the original function StartListening unusable >> and its entry slot a garbage. It will become a big mess if we add >> more function clones like StartListening11. . . . 6. The suggested >> API update makes the transport API VERSION_1_1 incompatible with the >> initial VERSION_1_0. > Transport 1.0 uses StartListening slot. Transport 1.1 uses > StartListeningWithAllow slot. > > I don't see any difference between this approach and approach of adding > extra slot and extra function. > > Transport 1.0 will continue to work as is (ever without recompilation). > Transport 1.1 will crash if it doesn't fill proper slot. It is not good enough. The Transport 1.1 can also support and be compatible with the 1.0 API. It works well for both alternate approaches. It does not work in your case (see the webrev.18). The API will become ugly and messy if more function clones like the StartListening clones are added in the future. The old duplicates hold the API slots that become unusable. > *b* > > I'm against separate Allow() call because: > > 1) Socket communication steps is well known. Extra step (call Allow > before StartListen) is not natural. I'm rejecting this argument. The StartListeningWithAllow step is also not well known and natural. There is no problem to add the AllowPeers step. > 2) API have to clear visible and self documenting. > > If transport developer doesn't implement Allow() they doesn't > see any sign of mistake until they starts testing the > transport with allow parameter. > > As opposite unused parameter is clear visible and cause compiler warning. This argument also looks strange to me. Why would not the transport developer implement AllowPeers() if he is trying to implement the transport 1.1 API? It is the developer responsibility. > 3) In a future, parsing of allow list might require results of socket > call (e.g. to distinguish between IPv4 only and IPv4/Ipv6 cases) There is no point to discuss the IPv6 until the API for it is suggested. Otherwise, this discussion is too abstract. > *c* > > As for configuration structures, any jdwpConfiguration structure > requires versioning i.e. see webrev.15. But we decided to go in other > direction. (already replied to Robbin) No need to have a separate versioning for the jdwpConfiguration. The transport API version will work well for it. Thanks, Serguei > > -Dmitry > > > On 2017-05-18 13:20, serguei.spitsyn at oracle.com wrote: >> Hi Dmitry, >> >> Let's get to a consensus for the last item on our plate: >> jdwpTransportError AllowPeers(const char* peers); .VS. >> jdwpTransportError StartListeningWithAllow(const char* address, >> char** actual_address, char *allow); >> >> >> I still do not like the StartListening() function clone and its >> name. It looks like a wrong direction to me. >> >> This is a fragment of my bug report comment on this issue: >> >> 4. The suggested *allow* feature is too destructive for the >> transport API. It makes the original function StartListening unusable >> and its entry slot a garbage. It will become a big mess if we add >> more function clones like StartListening11. . . . 6. The suggested >> API update makes the transport API VERSION_1_1 incompatible with the >> initial VERSION_1_0. >> >> >> One more alternate solution to suggest is to add new function: >> jdwpTransportError >> SetTransportConfiguration(jdwpTransportConfiguration config); >> >> Where: typedef struct { const char* allowed_peers; } >> jdwpTransportConfiguration; >> >> >> This approach allows to extend the jdwpTransportConfiguration in the >> future if necessary. However, I'm not sure we really need this >> ability. >> >> >> Both alternatives allow for the transport library to support both >> API versions. It is good by itself and also, it is a way to simplify >> the transport library. >> >> We could formulate the following requirements to the API updates: - >> the transport API update must support previous API versions - the >> updated API layout must be compatible with previous versions >> >> It seems to me, these requirements are achievable and two alternate >> approaches show it. >> >> Please, share your opinions? >> >> >> Thanks, Serguei From robbin.ehn at oracle.com Fri May 19 13:22:38 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Fri, 19 May 2017 15:22:38 +0200 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: <2644fa90-989b-5b2a-1ee3-3c2b1f306498@oracle.com> References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> <6320d430-768b-ba3f-c726-04127f939071@oracle.com> <2644fa90-989b-5b2a-1ee3-3c2b1f306498@oracle.com> Message-ID: Hi, On 05/19/2017 08:27 AM, serguei.spitsyn at oracle.com wrote: > On 5/18/17 04:05, Robbin Ehn wrote: >> Hi of those: >> >>> One more alternate solution to suggest is to add new function: >>> jdwpTransportError SetTransportConfiguration(jdwpTransportConfiguration config); >>> >>> Where: >>> typedef struct { >>> const char* allowed_peers; >>> } jdwpTransportConfiguration; >>> >>> >>> This approach allows to extend the jdwpTransportConfiguration in the future if necessary. >> >> +1 >> >> It would nice if a compiled client against JDK 8 headers works without recompiling with a JDK 9 runtime. >> To make really future proof; using a client compiled against JDK 9 with JDK 1X you will need some way to know witch version of the struct was used. >> >> E.g. SetTransportConfiguration(&config, JDWP_TRANSPORT_CONFIGURATION_CURRENT_VERSION); > > No need to have a separate transport configuration versioning. > It is enough to have just a transport API versioning. My thought when writing that was to not add the transport API versioning. But as you say if that is there, we already know the struct version. /Robbin > > Thanks, > Serguei > > >> >> Thanks, Robbin! > From serguei.spitsyn at oracle.com Fri May 19 18:37:21 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 19 May 2017 11:37:21 -0700 Subject: RFR(M): JDK-8061228 Allow JDWP socket connector to accept connections from certain ip addresses only In-Reply-To: References: <62f06838-ca7b-4dbf-3a32-f82518d33b41@oracle.com> <320d4094-b65a-3f59-2324-b53d86161e9c@oracle.com> <41366a7c-f92d-b02f-01e3-85832c1086db@oracle.com> <9bd91ef4-de96-ae6a-1af2-f0d889f511bf@oracle.com> <48a019f4-f153-3fde-7478-e7d8d6d98871@oracle.com> <3f4f5551-3395-e537-9aa4-82ecfd55e9f8@oracle.com> <148b5121-5004-6afa-b8c0-689e0a8a3e95@oracle.com> <6320d430-768b-ba3f-c726-04127f939071@oracle.com> <2644fa90-989b-5b2a-1ee3-3c2b1f306498@oracle.com> Message-ID: <8fa7c4f7-6917-4bf3-702f-0ae444b4fe25@oracle.com> On 5/19/17 06:22, Robbin Ehn wrote: > Hi, > > On 05/19/2017 08:27 AM, serguei.spitsyn at oracle.com wrote: >> On 5/18/17 04:05, Robbin Ehn wrote: >>> Hi of those: >>> >>>> One more alternate solution to suggest is to add new function: >>>> jdwpTransportError >>>> SetTransportConfiguration(jdwpTransportConfiguration config); >>>> >>>> Where: >>>> typedef struct { >>>> const char* allowed_peers; >>>> } jdwpTransportConfiguration; >>>> >>>> >>>> This approach allows to extend the jdwpTransportConfiguration in >>>> the future if necessary. >>> >>> +1 >>> >>> It would nice if a compiled client against JDK 8 headers works >>> without recompiling with a JDK 9 runtime. >>> To make really future proof; using a client compiled against JDK 9 >>> with JDK 1X you will need some way to know witch version of the >>> struct was used. >>> >>> E.g. SetTransportConfiguration(&config, >>> JDWP_TRANSPORT_CONFIGURATION_CURRENT_VERSION); >> >> No need to have a separate transport configuration versioning. >> It is enough to have just a transport API versioning. > > My thought when writing that was to not add the transport API versioning. > But as you say if that is there, we already know the struct version. Right. Thanks, Serguei > > /Robbin > >> >> Thanks, >> Serguei >> >> >>> >>> Thanks, Robbin! >> From matthias.baesken at sap.com Mon May 22 08:49:41 2017 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Mon, 22 May 2017 08:49:41 +0000 Subject: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent In-Reply-To: <520c56a8-4379-e4b0-42ac-acde1939cc7b@oracle.com> References: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> <444e21a8d38c42338fa5c38093a32e9c@sap.com> <520c56a8-4379-e4b0-42ac-acde1939cc7b@oracle.com> Message-ID: Hello Serguei, the com/sun/jdi tests were executed , same number of passed and errors in a patched and unpatched JDK . We do not need a sponsor because the change is in JDK not HS . Best regards, Matthias -----Original Message----- From: serguei.spitsyn at oracle.com [mailto:serguei.spitsyn at oracle.com] Sent: Freitag, 19. Mai 2017 02:18 To: Baesken, Matthias ; Alan Bateman ; core-libs-dev at openjdk.java.net; serviceability-dev at openjdk.java.net Cc: Simonis, Volker Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent Hi Matthias, The fix looks good to me. It must be tested at least with the JTreg com/sun/jdi tests. Do you need a sponsor? Thanks, Serguei On 5/16/17 03:21, Baesken, Matthias wrote: > Sure, I forward it to serviceability-dev . > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Dienstag, 16. Mai 2017 11:51 > To: Baesken, Matthias ; core-libs-dev at openjdk.java.net > Cc: Simonis, Volker > Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent > > > > On 16/05/2017 10:04, Baesken, Matthias wrote: >> Hello, could you please review this small change : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8180413/ >> >> it fixes a number of places in jdk.jdwp.agent where in case of an error it is attempted to write to NULL . >> >> Bug : JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent >> >> >> https://bugs.openjdk.java.net/browse/JDK-8180413 >> >> > Can you bring this to serviceability-dev as that is the mailing list > where the JDWP agent is maintained? > > -Alan From serguei.spitsyn at oracle.com Mon May 22 09:40:09 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 22 May 2017 02:40:09 -0700 Subject: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent In-Reply-To: References: <37ac7b12-1edd-ce65-a20f-404bd3eba75e@oracle.com> <444e21a8d38c42338fa5c38093a32e9c@sap.com> <520c56a8-4379-e4b0-42ac-acde1939cc7b@oracle.com> Message-ID: Hello Matthias, It is great, you are able to push this fix! Thanks, Serguei On 5/22/17 01:49, Baesken, Matthias wrote: > Hello Serguei, the com/sun/jdi tests were executed , same number of passed and errors in a patched and unpatched JDK . > We do not need a sponsor because the change is in JDK not HS . > > Best regards, Matthias > > > -----Original Message----- > From: serguei.spitsyn at oracle.com [mailto:serguei.spitsyn at oracle.com] > Sent: Freitag, 19. Mai 2017 02:18 > To: Baesken, Matthias ; Alan Bateman ; core-libs-dev at openjdk.java.net; serviceability-dev at openjdk.java.net > Cc: Simonis, Volker > Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent > > Hi Matthias, > > The fix looks good to me. > It must be tested at least with the JTreg com/sun/jdi tests. > Do you need a sponsor? > > Thanks, > Serguei > > > > On 5/16/17 03:21, Baesken, Matthias wrote: >> Sure, I forward it to serviceability-dev . >> >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Dienstag, 16. Mai 2017 11:51 >> To: Baesken, Matthias ; core-libs-dev at openjdk.java.net >> Cc: Simonis, Volker >> Subject: Re: [XS] JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent >> >> >> >> On 16/05/2017 10:04, Baesken, Matthias wrote: >>> Hello, could you please review this small change : >>> >>> http://cr.openjdk.java.net/~mbaesken/webrevs/8180413/ >>> >>> it fixes a number of places in jdk.jdwp.agent where in case of an error it is attempted to write to NULL . >>> >>> Bug : JDK-8180413 : avoid accessing NULL in jdk.jdwp.agent >>> >>> >>> https://bugs.openjdk.java.net/browse/JDK-8180413 >>> >>> >> Can you bring this to serviceability-dev as that is the mailing list >> where the JDWP agent is maintained? >> >> -Alan From jcbeyler at google.com Mon May 22 18:47:35 2017 From: jcbeyler at google.com (JC Beyler) Date: Mon, 22 May 2017 11:47:35 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Message-ID: Dear all, I have a new webrev up: http://cr.openjdk.java.net/~rasbold/8171119/webrev.03/ This webrev has, I hope, fixed a lot of the comments from Robbin: - The casts normally are all C++ style - Moved this to jdk10-hs - I have not tested slowdebug yet, hopefully it does not break there - Added the garbage collection system: - Now live sampled allocations are tracked throughout their lifetime - When GC happens, it moves the sampled allocation information to two lists: recent and frequent GC lists - Those lists use the array system that the live objects were using before but have different re-use strategies - Added the JVMTI API for them via a GetFrequentGarbageTraces and GetGarbageTraces - Both use the same JVMTI structures - Added the calls to them for the test, though I've kept that test simple for now: http://cr.openjdk.java.net/~rasbold/8171119/webrev.03/raw_files/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c - As I write this, I notice my webrev is missing a final change I made to the test that calls the same ReleaseTraces to each live/garbage/frequent structure. This is updated in my local repo and will get in the next webrev. Next steps for this work are: - Putting the TLAB implementation (yes not forgotten ;-)) - Adding more testing and separate the current test system to check things a bit more thoroughly - Have not tried to circumvent AsyncGetCallTrace yet - Still have to double check the stack walker a bit more Happy webrev perusal! Jc On Tue, May 16, 2017 at 5:20 AM, Robbin Ehn wrote: > Just a few answers, > > On 05/15/2017 06:48 PM, JC Beyler wrote: > >> Dear all, >> >> I've updated the webrev to: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/ < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/> >> > > I'll look at this later, thanks! > > >> Robbin, >> I believe I have addressed most of your items with webrev 02: >> - I added a JTreg test to show how it works: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_fi >> les/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_f >> iles/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c> >> - I've modified the code to use its own data structures both internally >> and externally, this will make it easier to move out of AsyncGetCallTrace >> as we move forward, that is still on my TODOs >> - I cleaned up the JVMTI API by passing a structure that handles the >> num_traces and put in a ReleaseTraces as well >> - I cleaned up other issues as well. >> >> However, I have three questions, which are probably because I'm new in >> this community: >> 1) My previous webrevs were based off of JDK9 by mistake. When I took >> JDK10 via : hg clone http://hg.openjdk.java.net/jdk10/jdk10 < >> http://hg.openjdk.java.net/jdk10/jdk10> jdk10 >> - I don't see code compatible with what you were showing (ie your >> patches don't make sense for that code base; ex: klass is still accessed >> via klass() for example in collectedHeap.inline.hpp) >> - Would you know what is the right hg clone command so we are >> working on the same code base? >> > > We use jdk10-hs, e.g. > hg tclone http://hg.openjdk.java.net/jdk10/hs 10-hs > > There is sporadic big merges going from jdk9->jdk10->jdk10-hs and > jdk10-hs->jdk10, so 10 is moving... > > >> 2) You mentioned I was using os::malloc, new, NEW_C_HEAP_ARRAY; I >> cleaned out the os::malloc but which of the new vs NEW_C_HEAP_ARRAY should >> I use. It might be that I don't understand when one uses one or the other >> but I see both used around the code base? >> - Is it that new is to be used for anything internal and >> NEW_C_HEAP_ARRAY anything provided to the JVMTI users outside of the JVM? >> > > We overload new operator when you extend correct base class, e.g. > CHeapObj so use 'new' > But for arrays you will need the macro NEW_C_HEAP_ARRAY. > > >> 3) Casts: same kind question: which should I use. The code was using a >> bit of everything, I'll refactor it entirely but I was not clear if I >> should go to C casts or C++ casts as I see both in the codebase. What is >> the convention I should use? >> > > Just be consist, use what suites you, C++ casts might be preferable, if we > are moving towards C++11. > And use 'right' cast, e.g. going from Thread* to JavaThread* you should > use C cast or static_cast, not reinterpret_cast I would say. > > >> Final notes on this webrev: >> - I am still missing: >> - Putting a TLAB implementation so that we can compare both webrevs >> - Have not tried to circumvent AsyncGetCallTrace >> - Putting in the handling of GC'd objects >> - Fix a stack walker issue I have seen, I think I know the problem >> and will test that theory out for the next webrev >> >> I will work on integrating those items for the next webrev! >> > > Thanks! > > >> Thanks for your help, >> Jc >> >> Ps: I tested this on a new repo: >> >> hg clone http://hg.openjdk.java.net/jdk10/jdk10 < >> http://hg.openjdk.java.net/jdk10/jdk10> jdk10 >> ... building it >> cd test >> jtreg -nativepath:/build/linux-x86_64-normal-server >> -release/support/test/hotspot/jtreg/native/lib/ -jdk >> /linux-x86_64-normal-server-release/images/jdk >> ../hotspot/test/serviceability/jvmti/HeapMonitor/ >> >> > I'll test it out! > > /Robbin > > >> >> On Thu, May 4, 2017 at 11:21 PM, serguei.spitsyn at oracle.com > serguei.spitsyn at oracle.com> > serguei.spitsyn at oracle.com>> wrote: >> >> Robbin, >> >> Thank you for forwarding! >> I will review it. >> >> Thanks, >> Serguei >> >> >> >> On 5/4/17 02:13, Robbin Ehn wrote: >> >> Hi, >> >> To me the compiler changes looks what is expected. >> It would be good if someone from compiler could take a look at >> that. >> Added compiler to mail thread. >> >> Also adding Serguei, It would be good with his view also. >> >> My initial take on it, read through most of the code and took it >> for a ride. >> >> ############################## >> - Regarding the compiler changes: I think we need the 'TLAB end' >> trickery (mentioned by Tony P) >> instead of a separate check for sampling in fast path for the >> final version. >> >> ############################## >> - This patch I had to apply to get it compile on JDK 10: >> >> diff -r ac3ded340b35 src/share/vm/gc/shared/collect >> edHeap.inline.hpp >> --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr >> 28 14:31:38 2017 +0200 >> +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May >> 04 10:22:56 2017 +0200 >> @@ -87,3 +87,3 @@ >> // support for object alloc event (no-op most of the time) >> - if (klass() != NULL && klass()->name() != NULL) { >> + if (klass != NULL && klass->name() != NULL) { >> Thread *base_thread = Thread::current(); >> diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 >> 14:31:38 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 >> 10:22:56 2017 +0200 >> @@ -316,3 +316,3 @@ >> JavaThread *thread = reinterpret_cast> *>(Thread::current()); >> - assert(o->size() << LogHeapWordSize == byte_size, >> + assert(o->size() << LogHeapWordSize == (long)byte_size, >> "Object size is incorrect."); >> >> ############################## >> - This patch I had to apply to get it not asserting during >> slowdebug: >> >> --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 >> 15:15:16 2017 +0200 >> +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 >> 10:24:25 2017 +0200 >> @@ -32,3 +32,3 @@ >> // TODO(jcbeyler): should we make this into a JVMTI structure? >> -struct StackTraceData { >> +struct StackTraceData : CHeapObj { >> ASGCT_CallTrace *trace; >> @@ -143,3 +143,2 @@ >> StackTraceStorage::StackTraceStorage() : >> - _allocated_traces(new StackTraceData*[MaxHeapTraces]), >> _allocated_traces_size(MaxHeapTraces), >> @@ -147,2 +146,3 @@ >> _allocated_count(0) { >> + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, >> MaxHeapTraces, mtInternal); >> memset(_allocated_traces, 0, sizeof(*_allocated_traces) * >> MaxHeapTraces); >> @@ -152,3 +152,3 @@ >> StackTraceStorage::~StackTraceStorage() { >> - delete[] _allocated_traces; >> + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); >> } >> >> - Classes should extend correct base class for which type of >> memory is used for it e.g.: CHeapObj or StackObj or AllStatic >> - The style in heapMonitoring.cpp is a bit different from normal >> vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, >> os::malloc and new. >> - In jvmtiHeapTransition.hpp you use C cast instead. >> >> ############################## >> - This patch I had apply to get traces without setting an >> ?unrelated? capability >> - Should this not be a new capability? >> >> diff -r c02a5d8785bf src/share/vm/prims/forte.cpp >> --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 >> +0200 >> +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 >> +0200 >> @@ -530,6 +530,6 @@ >> >> - if (!JvmtiExport::should_post_class_load()) { >> +/* if (!JvmtiExport::should_post_class_load()) { >> trace->num_frames = ticks_no_class_load; // -1 >> return; >> - } >> + }*/ >> >> ############################## >> - forte.cpp: (I know this is not part of your changes but) >> find_jmethod_id_or_null give me NULL for my test. >> It looks like we actually want the regular jmethod_id() ? >> >> Since we are the thread we are talking about (and in same >> ucontext) and thread is in vm and have a last java frame, >> I think most of the checks done in AsyncGetCallTrace is >> irrelevant, so you should be-able to call forte_fill_call_trace_given_top >> directly. >> But since we might need jmethod_id() if possible to avoid getting >> method id NULL, >> we need some fixes in forte code, or just do the vframStream loop >> inside heapMonitoring.cpp and not use forte.cpp. >> >> Something like: >> >> if (jthread->has_last_Java_frame()) { // just to be safe >> vframeStream vfst(jthread); >> while (!vfst.at_end()) { >> Method* m = vfst.method(); >> m->jmethod_id(); >> m->line_number_from_bci(vfst.bci()); >> vfst.next(); >> } >> >> - This is a bit confusing in forte.cpp, >> trace->frames[count].lineno = bci. >> Line number should be m->line_number_from_bci(bci); >> Do the heapMonitoring suppose to trace with bci or line number? >> I would say bci, meaning we should either rename >> ASGCT_CallFrame?lineno or use another data structure which says bci. >> >> ############################## >> - // TODO(jcbeyler): remove this extra code handling the extra >> trace for >> Please fix all these TODO's :) >> >> ############################## >> - heapMonitoring.hpp: >> // TODO(jcbeyler): is this algorithm acceptable in open source? >> >> Why is this comment here? What is the implication? >> Have you tested any simpler algorithm? >> >> ############################## >> - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk >> for building the agent) >> >> ############################## >> - monitoring_period vs HeapMonitorRate, pick rate or period. >> >> ############################## >> - globals.hpp >> Why is MaxHeapTraces not settable/overridable from jvmti >> interface? That would be handy. >> >> ############################## >> - jvmtiStackTraceData + ASGCT_CallFrame memory >> Are the agent suppose to loop through and free all >> ASGCT_CallFrame? >> Wouldn't it be better with some kinda protocol, like: >> (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); >> (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); >> >> Also using another data structure that have num_traces inside it >> simplifies things. >> So I'm not convinced using the async structure is the best way >> forward. >> >> >> I have more questions, but I think it's better if you respond and >> update the code first. >> >> Thanks! >> >> /Robbin >> >> >> On 04/21/2017 11:34 PM, JC Beyler wrote: >> >> Hi all, >> >> I've added size information to the allocation sampling >> system. This allows the callback to remember the size of each sampled >> allocation. >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ < >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/> >> >> The new webrev.01 also adds the actual heap monitoring >> sampling system in files: >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.cpp.patch >> > hare/vm/runtime/heapMonitoring.cpp.patch> >> and >> http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/sh >> are/vm/runtime/heapMonitoring.hpp.patch >> > hare/vm/runtime/heapMonitoring.hpp.patch> >> >> My next step is to add the GC part to the webrev, which will >> allow users to determine what objects are live and what are garbage. >> >> Thanks for your attention and let me know if there are any >> questions! >> >> Have a wonderful Friday! >> Jc >> >> On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> wrote: >> >> Hi all, >> >> I worked on getting a few numbers for overhead and >> accuracy for my feature. I'm unsure if here is the right place to provide >> the full data, so I am just >> summarizing >> here for now. >> >> - Overhead of the feature >> >> Using the Dacapo benchmark (http://dacapobench.org/). >> My initial results are that sampling provides 2.4% with a 512k sampling, >> 512k being our default setting. >> >> - Note: this was without the tradesoap, tradebeans and >> tomcat benchmarks since they did not work with my JDK9 (issue between >> Dacapo and JDK9 it seems) >> - I want to rerun next week to ensure number stability >> >> - Accuracy of the feature >> >> I wrote a small microbenchmark that allocates from two >> different stacktraces at a given ratio. For example, 10% of stacktrace S1 >> and 90% from stacktrace >> S2. The >> microbenchmark was run 20 times, I averaged the results >> and looked for accuracy. It seems that statistically it is sound since if I >> allocated10% S1 and 90% >> S2, with a >> sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. >> >> Let me know if there are any questions on the numbers >> and if you'd like to see some more data. >> >> Note: this was done using our internal JDK8 >> implementation since the webrev provided by >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html >> > tml> >> > tml > >> does not yet contain the whole >> implementation and therefore would have been misleading. >> >> Thanks, >> Jc >> >> >> On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> wrote: >> >> Hi all, >> >> To move the discussion forward, with Chuck Rasbold's >> help to make a webrev, we pushed this: >> http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.ht >> ml >> > tml > >> 415 lines changed: 399 ins; 13 del; 3 mod; 51122 >> unchg >> >> This is not a final change that does the whole >> proposition from the JBS entry: https://bugs.openjdk.java.net/ >> browse/JDK-8177374 >> >> > https://bugs.openjdk.java.net/browse/JDK-8177374>>; what it does show is >> parts of the implementation that is >> proposed and hopefully can start the conversation going >> as I work through the details. >> >> For example, the changes to C2 are done here for the >> allocations: http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/shar >> e/vm/opto/macro.cpp.patch >> > re/vm/opto/macro.cpp.patch> >> > re/vm/opto/macro.cpp.patch >> > re/vm/opto/macro.cpp.patch>> >> >> Hopefully this all makes sense and thank you for all >> your future comments! >> Jc >> >> >> On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler < >> jcbeyler at google.com > jcbeyler at google.com >> >> wrote: >> >> Hello all, >> >> This is a follow-up from Jeremy's initial email >> from last year: >> http://mail.openjdk.java.net/pipermail/serviceability-dev/20 >> 15-June/017543.html >> > 015-June/017543.html> >> > 015-June/017543.html > pipermail/serviceability-dev/2015-June/017543.html>> >> >> I've gone ahead and started working on preparing >> this and Jeremy and I went down the route of actually writing it up in JEP >> form: >> https://bugs.openjdk.java.net/browse/JDK-8171119 < >> https://bugs.openjdk.java.net/browse/JDK-8171119> >> >> I think original conversation that happened last >> year in that thread still holds true: >> >> - We have a patch at Google that we think >> others might be interested in >> - It provides a means to understand where >> the allocation hotspots are at a very low overhead >> - Since it is at a low overhead, we can >> leave it on by default >> >> So I come to the mailing list with Jeremy's >> initial question: >> "I thought I would ask if there is any interest >> / if I should write a JEP / if I should just forget it." >> >> A year ago, it seemed some thought it was a good >> idea, is this still true? >> >> Thanks, >> Jc >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Thu May 25 14:59:22 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 25 May 2017 08:59:22 -0600 Subject: output of jstack command In-Reply-To: References: Message-ID: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> Adding serviceability-dev at ... since jstack is a Serviceability tool. I believe jstack is experimental which means the output format can change at any time... Dan On 5/25/17 8:35 AM, Ram Krishnan wrote: > Hi, > > I would like to leverage the output of jstack command for extracting > additional information about the type of threads, thread ids etc. Since I > will be parsing the output, I need the precise format. Is there any > documentation on jstack output format changes and the openjdk release(s) > where the changes happened? > > ?Tha?nks in advance. > From kirk.pepperdine at gmail.com Thu May 25 15:07:26 2017 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Thu, 25 May 2017 17:07:26 +0200 Subject: output of jstack command In-Reply-To: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> References: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> Message-ID: <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> Hi Ramki, The source for jstack is in openJDK. Feel free to create your own copy of jstack where you can output the information in any format he likes. If you are suggesting that the existing format be changed do be aware that there are many tools that expect the current format. These have been adjusted to a change in format that was introduced with Java 8. I don?t see any reason why the format shouldn?t include information that is currently missing and is relevant. However I?d want to make sure that is is relevant and important before breaking the tool chain once again. I believe thread ids are already in the header. Certainly thread names are there. Not sure what you mean by types of threads. Kind regards, Kirk > On May 25, 2017, at 4:59 PM, Daniel D. Daugherty wrote: > > Adding serviceability-dev at ... since jstack is a Serviceability tool. > > I believe jstack is experimental which means the output format can > change at any time... > > Dan > > On 5/25/17 8:35 AM, Ram Krishnan wrote: >> Hi, >> >> I would like to leverage the output of jstack command for extracting >> additional information about the type of threads, thread ids etc. Since I >> will be parsing the output, I need the precise format. Is there any >> documentation on jstack output format changes and the openjdk release(s) >> where the changes happened? >> >> ?Tha?nks in advance. >> > From ramkri123 at gmail.com Thu May 25 16:01:03 2017 From: ramkri123 at gmail.com (Ram Krishnan) Date: Thu, 25 May 2017 09:01:03 -0700 Subject: output of jstack command In-Reply-To: <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> References: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> Message-ID: Hi Kirk, Daniel, Many thanks for the immediate reply. By type of thread, I meant GC thread vs normal thread. Looks like that information is already there in the thread name. Looks like in Java 9 the output of jstack is different, so the tools need to change for Java 9. Is it fair to count on a consistent format per Java release for jstack? Thanks, Ramki On Thu, May 25, 2017 at 8:07 AM, Kirk Pepperdine wrote: > Hi Ramki, > > The source for jstack is in openJDK. Feel free to create your own copy of > jstack where you can output the information in any format he likes. If you > are suggesting that the existing format be changed do be aware that there > are many tools that expect the current format. These have been adjusted to > a change in format that was introduced with Java 8. I don?t see any reason > why the format shouldn?t include information that is currently missing and > is relevant. However I?d want to make sure that is is relevant and > important before breaking the tool chain once again. > > I believe thread ids are already in the header. Certainly thread names are > there. Not sure what you mean by types of threads. > > Kind regards, > Kirk > > On May 25, 2017, at 4:59 PM, Daniel D. Daugherty < > daniel.daugherty at oracle.com> wrote: > > > > Adding serviceability-dev at ... since jstack is a Serviceability tool. > > > > I believe jstack is experimental which means the output format can > > change at any time... > > > > Dan > > > > On 5/25/17 8:35 AM, Ram Krishnan wrote: > >> Hi, > >> > >> I would like to leverage the output of jstack command for extracting > >> additional information about the type of threads, thread ids etc. Since > I > >> will be parsing the output, I need the precise format. Is there any > >> documentation on jstack output format changes and the openjdk release(s) > >> where the changes happened? > >> > >> ?Tha?nks in advance. > >> > > > > -- Thanks, Ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Thu May 25 18:06:44 2017 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Thu, 25 May 2017 20:06:44 +0200 Subject: output of jstack command In-Reply-To: References: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> Message-ID: Hi, The format of thread dumps have been stable until 8 when it seems that we stopped caring about breaking tooling. That it changed in 9 is of no surprise. Again, pain we?re willing to accept if there is valuable information being added. Kind regards, Kirk > On May 25, 2017, at 6:01 PM, Ram Krishnan wrote: > > Hi Kirk, Daniel, > > Many thanks for the immediate reply. > > By type of thread, I meant GC thread vs normal thread. Looks like that information is already there in the thread name. > > Looks like in Java 9 the output of jstack is different, so the tools need to change for Java 9. > > Is it fair to count on a consistent format per Java release for jstack? > > Thanks, > Ramki > > On Thu, May 25, 2017 at 8:07 AM, Kirk Pepperdine > wrote: > Hi Ramki, > > The source for jstack is in openJDK. Feel free to create your own copy of jstack where you can output the information in any format he likes. If you are suggesting that the existing format be changed do be aware that there are many tools that expect the current format. These have been adjusted to a change in format that was introduced with Java 8. I don?t see any reason why the format shouldn?t include information that is currently missing and is relevant. However I?d want to make sure that is is relevant and important before breaking the tool chain once again. > > I believe thread ids are already in the header. Certainly thread names are there. Not sure what you mean by types of threads. > > Kind regards, > Kirk > > On May 25, 2017, at 4:59 PM, Daniel D. Daugherty > wrote: > > > > Adding serviceability-dev at ... since jstack is a Serviceability tool. > > > > I believe jstack is experimental which means the output format can > > change at any time... > > > > Dan > > > > On 5/25/17 8:35 AM, Ram Krishnan wrote: > >> Hi, > >> > >> I would like to leverage the output of jstack command for extracting > >> additional information about the type of threads, thread ids etc. Since I > >> will be parsing the output, I need the precise format. Is there any > >> documentation on jstack output format changes and the openjdk release(s) > >> where the changes happened? > >> > >> ?Tha?nks in advance. > >> > > > > > > > -- > Thanks, > Ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From sharath.ballal at oracle.com Mon May 29 10:44:56 2017 From: sharath.ballal at oracle.com (Sharath Ballal) Date: Mon, 29 May 2017 03:44:56 -0700 (PDT) Subject: RFR: JDK-6760477 - Update SA to include stack traces in the heap dump Message-ID: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> Hi, Pls review the fix for including stack traces in the heap dump in SA. Bug: https://bugs.openjdk.java.net/browse/JDK-6760477 Webrev: http://cr.openjdk.java.net/~sballal/6760477/webrev.00/ Thanks & Regards, Sharath Ballal -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbin.ehn at oracle.com Mon May 29 15:33:53 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Mon, 29 May 2017 17:33:53 +0200 Subject: output of jstack command In-Reply-To: <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> References: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com> Message-ID: <27af994c-aa16-f503-d527-d2b7976c85e0@oracle.com> Hi, The text stream originates from: void Threads::print_on(outputStream* st, bool print_stacks, bool internal_format, bool print_concurrent_locks) { in hotspot/src/share/vm/runtime/thread.cpp L4491 and jstack only forwards that to your terminal. /Robbin On 05/25/2017 05:07 PM, Kirk Pepperdine wrote: > Hi Ramki, > > The source for jstack is in openJDK. Feel free to create your own copy of jstack where you can output the information in any format he likes. If you are suggesting that the existing format be changed do be aware that there are many tools that expect the current format. These have been adjusted to a change in format that was introduced with Java 8. I don?t see any reason why the format shouldn?t include information that is currently missing and is relevant. However I?d want to make sure that is is relevant and important before breaking the tool chain once again. > > I believe thread ids are already in the header. Certainly thread names are there. Not sure what you mean by types of threads. > > Kind regards, > Kirk >> On May 25, 2017, at 4:59 PM, Daniel D. Daugherty wrote: >> >> Adding serviceability-dev at ... since jstack is a Serviceability tool. >> >> I believe jstack is experimental which means the output format can >> change at any time... >> >> Dan >> >> On 5/25/17 8:35 AM, Ram Krishnan wrote: >>> Hi, >>> >>> I would like to leverage the output of jstack command for extracting >>> additional information about the type of threads, thread ids etc. Since I >>> will be parsing the output, I need the precise format. Is there any >>> documentation on jstack output format changes and the openjdk release(s) >>> where the changes happened? >>> >>> ?Tha?nks in advance. >>> >> > From dmitry.samersoff at oracle.com Mon May 29 16:52:54 2017 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Mon, 29 May 2017 19:52:54 +0300 Subject: RFR: JDK-6760477 - Update SA to include stack traces in the heap dump In-Reply-To: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> References: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> Message-ID: Sharath, Looks good to me. Minor nit: Reader.java:125 you can create an instance of HprofReader and avoid extra cast. -Dmitry On 2017-05-29 13:44, Sharath Ballal wrote: > Hi, > > > > Pls review the fix for including stack traces in the heap dump in SA. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-6760477 > > Webrev: http://cr.openjdk.java.net/~sballal/6760477/webrev.00/ > > > > > > Thanks & Regards, > > Sharath Ballal > > > > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From ecki at zusammenkunft.net Mon May 29 18:25:28 2017 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Mon, 29 May 2017 18:25:28 +0000 Subject: output of jstack command In-Reply-To: <27af994c-aa16-f503-d527-d2b7976c85e0@oracle.com> References: <5c8d22b5-5c6c-1aca-c57f-2b28733efe3f@oracle.com> <14CF8360-5840-4204-9F2D-6A123A5F9858@gmail.com>, <27af994c-aa16-f503-d527-d2b7976c85e0@oracle.com> Message-ID: And just because nobody mentioned it, don't forget `jcmd Thread.print -l`. I had the impression jcmd is preferred,over the older jps/jstack/jmap gang? Gruss Bernd -- http://bernd.eckenfels.net ________________________________ From: serviceability-dev on behalf of Robbin Ehn Sent: Monday, May 29, 2017 5:33:53 PM To: Ram Krishnan Cc: serviceability-dev at openjdk.java.net serviceability-dev at openjdk.java.net; Ray Hindman; Asif Qamar Subject: Re: output of jstack command Hi, The text stream originates from: void Threads::print_on(outputStream* st, bool print_stacks, bool internal_format, bool print_concurrent_locks) { in hotspot/src/share/vm/runtime/thread.cpp L4491 and jstack only forwards that to your terminal. /Robbin On 05/25/2017 05:07 PM, Kirk Pepperdine wrote: > Hi Ramki, > > The source for jstack is in openJDK. Feel free to create your own copy of jstack where you can output the information in any format he likes. If you are suggesting that the existing format be changed do be aware that there are many tools that expect the current format. These have been adjusted to a change in format that was introduced with Java 8. I don?t see any reason why the format shouldn?t include information that is currently missing and is relevant. However I?d want to make sure that is is relevant and important before breaking the tool chain once again. > > I believe thread ids are already in the header. Certainly thread names are there. Not sure what you mean by types of threads. > > Kind regards, > Kirk >> On May 25, 2017, at 4:59 PM, Daniel D. Daugherty wrote: >> >> Adding serviceability-dev at ... since jstack is a Serviceability tool. >> >> I believe jstack is experimental which means the output format can >> change at any time... >> >> Dan >> >> On 5/25/17 8:35 AM, Ram Krishnan wrote: >>> Hi, >>> >>> I would like to leverage the output of jstack command for extracting >>> additional information about the type of threads, thread ids etc. Since I >>> will be parsing the output, I need the precise format. Is there any >>> documentation on jstack output format changes and the openjdk release(s) >>> where the changes happened? >>> >>> ?Tha?nks in advance. >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sharath.ballal at oracle.com Tue May 30 05:00:34 2017 From: sharath.ballal at oracle.com (Sharath Ballal) Date: Mon, 29 May 2017 22:00:34 -0700 (PDT) Subject: RFR: JDK-6760477 - Update SA to include stack traces in the heap dump In-Reply-To: References: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> Message-ID: <5187866c-f969-4531-824d-7cbaf874dab5@default> Dmitry, Thanks for the review. I have done the changes. http://cr.openjdk.java.net/~sballal/6760477/webrev.01/ Thanks & Regards, Sharath Ballal -----Original Message----- From: Dmitry Samersoff Sent: Monday, May 29, 2017 10:23 PM To: Sharath Ballal; serviceability-dev at openjdk.java.net Subject: Re: RFR: JDK-6760477 - Update SA to include stack traces in the heap dump Sharath, Looks good to me. Minor nit: Reader.java:125 you can create an instance of HprofReader and avoid extra cast. -Dmitry On 2017-05-29 13:44, Sharath Ballal wrote: > Hi, > > > > Pls review the fix for including stack traces in the heap dump in SA. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-6760477 > > Webrev: http://cr.openjdk.java.net/~sballal/6760477/webrev.00/ > > > > > > Thanks & Regards, > > Sharath Ballal > > > > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jini.george at oracle.com Tue May 30 09:25:38 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 30 May 2017 14:55:38 +0530 Subject: RFR: JDK-6760477 - Update SA to include stack traces in the heap dump In-Reply-To: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> References: <2091a3f7-af89-4c8e-803a-22b4848c8788@default> Message-ID: <2078e8c7-1394-e695-674e-ec072791b48b@oracle.com> Hi Sharath, Looks good. A few nits. * Looks like the stackTraces array list is unused. * HeapHprofBinWriter.java: Line 764: It might be better to replace 2 * 4 and similar instances with 2 * INT_SIZE. Thanks, Jini. On 5/29/2017 4:14 PM, Sharath Ballal wrote: > > Hi, > > Pls review the fix for including stack traces in the heap dump in SA. > > Bug: https://bugs.openjdk.java.net/browse/JDK-6760477 > > Webrev: http://cr.openjdk.java.net/~sballal/6760477/webrev.00/ > > > Thanks & Regards, > > Sharath Ballal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbin.ehn at oracle.com Tue May 30 12:33:06 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 30 May 2017 14:33:06 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <2af975e6-3827-bd57-0c3d-fadd54867a67@oracle.com> <365499b6-3f4d-a4df-9e7e-e72a739fb26b@oracle.com> Message-ID: <102c59b8-25b6-8c21-8eef-1de7d0bbf629@oracle.com> Hi Jc, On 05/22/2017 08:47 PM, JC Beyler wrote: > Dear all, > > I have a new webrev up: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.03/ I liked this! Two small things: heapMonitoring.hpp class HeapMonitoring should extend AllStatic heapMonitoring.cpp class MuxLocker should extend StackObj But I think you should skip MuxLocker or push it separate generic enhancement. Great with the jtreg test, thanks alot! > > This webrev has, I hope, fixed a lot of the comments from Robbin: > - The casts normally are all C++ style > - Moved this to jdk10-hs > - I have not tested slowdebug yet, hopefully it does not break there > - Added the garbage collection system: > - Now live sampled allocations are tracked throughout their lifetime > - When GC happens, it moves the sampled allocation information to two lists: recent and frequent GC lists > - Those lists use the array system that the live objects were using before but have different re-use strategies > - Added the JVMTI API for them via a GetFrequentGarbageTraces and GetGarbageTraces > - Both use the same JVMTI structures > - Added the calls to them for the test, though I've kept that test simple for now: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.03/raw_files/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c > - As I write this, I notice my webrev is missing a final change I made to the test that calls the same ReleaseTraces to each live/garbage/frequent structure. This > is updated in my local repo and will get in the next webrev. > > Next steps for this work are: > - Putting the TLAB implementation (yes not forgotten ;-)) > - Adding more testing and separate the current test system to check things a bit more thoroughly > - Have not tried to circumvent AsyncGetCallTrace yet > - Still have to double check the stack walker a bit more Looking forward to this. Could someone from compiler take a look please? Thanks! /Robbin > > Happy webrev perusal! > Jc > > > On Tue, May 16, 2017 at 5:20 AM, Robbin Ehn > wrote: > > Just a few answers, > > On 05/15/2017 06:48 PM, JC Beyler wrote: > > Dear all, > > I've updated the webrev to: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/ > > > > > I'll look at this later, thanks! > > > Robbin, > I believe I have addressed most of your items with webrev 02: > - I added a JTreg test to show how it works: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.02/raw_files/new/test/serviceability/jvmti/HeapMonitor/libHeapMonitor.c > > > > - I've modified the code to use its own data structures both internally and externally, this will make it easier to move out of AsyncGetCallTrace as we move > forward, that is still on my TODOs > - I cleaned up the JVMTI API by passing a structure that handles the num_traces and put in a ReleaseTraces as well > - I cleaned up other issues as well. > > However, I have three questions, which are probably because I'm new in this community: > 1) My previous webrevs were based off of JDK9 by mistake. When I took JDK10 via : hg clone http://hg.openjdk.java.net/jdk10/jdk10 > > jdk10 > - I don't see code compatible with what you were showing (ie your patches don't make sense for that code base; ex: klass is still accessed via klass() for > example in collectedHeap.inline.hpp) > - Would you know what is the right hg clone command so we are working on the same code base? > > > We use jdk10-hs, e.g. > hg tclone http://hg.openjdk.java.net/jdk10/hs 10-hs > > There is sporadic big merges going from jdk9->jdk10->jdk10-hs and jdk10-hs->jdk10, so 10 is moving... > > > 2) You mentioned I was using os::malloc, new, NEW_C_HEAP_ARRAY; I cleaned out the os::malloc but which of the new vs NEW_C_HEAP_ARRAY should I use. It might be > that I don't understand when one uses one or the other but I see both used around the code base? > - Is it that new is to be used for anything internal and NEW_C_HEAP_ARRAY anything provided to the JVMTI users outside of the JVM? > > > We overload new operator when you extend correct base class, e.g. CHeapObj so use 'new' > But for arrays you will need the macro NEW_C_HEAP_ARRAY. > > > 3) Casts: same kind question: which should I use. The code was using a bit of everything, I'll refactor it entirely but I was not clear if I should go to C casts > or C++ casts as I see both in the codebase. What is the convention I should use? > > > Just be consist, use what suites you, C++ casts might be preferable, if we are moving towards C++11. > And use 'right' cast, e.g. going from Thread* to JavaThread* you should use C cast or static_cast, not reinterpret_cast I would say. > > > Final notes on this webrev: > - I am still missing: > - Putting a TLAB implementation so that we can compare both webrevs > - Have not tried to circumvent AsyncGetCallTrace > - Putting in the handling of GC'd objects > - Fix a stack walker issue I have seen, I think I know the problem and will test that theory out for the next webrev > > I will work on integrating those items for the next webrev! > > > Thanks! > > > Thanks for your help, > Jc > > Ps: I tested this on a new repo: > > hg clone http://hg.openjdk.java.net/jdk10/jdk10 > jdk10 > ... building it > cd test > jtreg -nativepath:/build/linux-x86_64-normal-server-release/support/test/hotspot/jtreg/native/lib/ -jdk > /linux-x86_64-normal-server-release/images/jdk ../hotspot/test/serviceability/jvmti/HeapMonitor/ > > > I'll test it out! > > /Robbin > > > > On Thu, May 4, 2017 at 11:21 PM, serguei.spitsyn at oracle.com > >> wrote: > > Robbin, > > Thank you for forwarding! > I will review it. > > Thanks, > Serguei > > > > On 5/4/17 02:13, Robbin Ehn wrote: > > Hi, > > To me the compiler changes looks what is expected. > It would be good if someone from compiler could take a look at that. > Added compiler to mail thread. > > Also adding Serguei, It would be good with his view also. > > My initial take on it, read through most of the code and took it for a ride. > > ############################## > - Regarding the compiler changes: I think we need the 'TLAB end' trickery (mentioned by Tony P) > instead of a separate check for sampling in fast path for the final version. > > ############################## > - This patch I had to apply to get it compile on JDK 10: > > diff -r ac3ded340b35 src/share/vm/gc/shared/collectedHeap.inline.hpp > --- a/src/share/vm/gc/shared/collectedHeap.inline.hpp Fri Apr 28 14:31:38 2017 +0200 > +++ b/src/share/vm/gc/shared/collectedHeap.inline.hpp Thu May 04 10:22:56 2017 +0200 > @@ -87,3 +87,3 @@ > // support for object alloc event (no-op most of the time) > - if (klass() != NULL && klass()->name() != NULL) { > + if (klass != NULL && klass->name() != NULL) { > Thread *base_thread = Thread::current(); > diff -r ac3ded340b35 src/share/vm/runtime/heapMonitoring.cpp > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 14:31:38 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:22:56 2017 +0200 > @@ -316,3 +316,3 @@ > JavaThread *thread = reinterpret_cast(Thread::current()); > - assert(o->size() << LogHeapWordSize == byte_size, > + assert(o->size() << LogHeapWordSize == (long)byte_size, > "Object size is incorrect."); > > ############################## > - This patch I had to apply to get it not asserting during slowdebug: > > --- a/src/share/vm/runtime/heapMonitoring.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/runtime/heapMonitoring.cpp Thu May 04 10:24:25 2017 +0200 > @@ -32,3 +32,3 @@ > // TODO(jcbeyler): should we make this into a JVMTI structure? > -struct StackTraceData { > +struct StackTraceData : CHeapObj { > ASGCT_CallTrace *trace; > @@ -143,3 +143,2 @@ > StackTraceStorage::StackTraceStorage() : > - _allocated_traces(new StackTraceData*[MaxHeapTraces]), > _allocated_traces_size(MaxHeapTraces), > @@ -147,2 +146,3 @@ > _allocated_count(0) { > + _allocated_traces = NEW_C_HEAP_ARRAY(StackTraceData*, MaxHeapTraces, mtInternal); > memset(_allocated_traces, 0, sizeof(*_allocated_traces) * MaxHeapTraces); > @@ -152,3 +152,3 @@ > StackTraceStorage::~StackTraceStorage() { > - delete[] _allocated_traces; > + FREE_C_HEAP_ARRAY(StackTraceData*, _allocated_traces); > } > > - Classes should extend correct base class for which type of memory is used for it e.g.: CHeapObj or StackObj or AllStatic > - The style in heapMonitoring.cpp is a bit different from normal vm-style, e.g. using C++ casts instead of C. You mix NEW_C_HEAP_ARRAY, os::malloc and new. > - In jvmtiHeapTransition.hpp you use C cast instead. > > ############################## > - This patch I had apply to get traces without setting an ?unrelated? capability > - Should this not be a new capability? > > diff -r c02a5d8785bf src/share/vm/prims/forte.cpp > --- a/src/share/vm/prims/forte.cpp Fri Apr 28 15:15:16 2017 +0200 > +++ b/src/share/vm/prims/forte.cpp Thu May 04 10:24:25 2017 +0200 > @@ -530,6 +530,6 @@ > > - if (!JvmtiExport::should_post_class_load()) { > +/* if (!JvmtiExport::should_post_class_load()) { > trace->num_frames = ticks_no_class_load; // -1 > return; > - } > + }*/ > > ############################## > - forte.cpp: (I know this is not part of your changes but) > find_jmethod_id_or_null give me NULL for my test. > It looks like we actually want the regular jmethod_id() ? > > Since we are the thread we are talking about (and in same ucontext) and thread is in vm and have a last java frame, > I think most of the checks done in AsyncGetCallTrace is irrelevant, so you should be-able to call forte_fill_call_trace_given_top directly. > But since we might need jmethod_id() if possible to avoid getting method id NULL, > we need some fixes in forte code, or just do the vframStream loop inside heapMonitoring.cpp and not use forte.cpp. > > Something like: > > if (jthread->has_last_Java_frame()) { // just to be safe > vframeStream vfst(jthread); > while (!vfst.at_end()) { > Method* m = vfst.method(); > m->jmethod_id(); > m->line_number_from_bci(vfst.bci()); > vfst.next(); > } > > - This is a bit confusing in forte.cpp, trace->frames[count].lineno = bci. > Line number should be m->line_number_from_bci(bci); > Do the heapMonitoring suppose to trace with bci or line number? > I would say bci, meaning we should either rename ASGCT_CallFrame?lineno or use another data structure which says bci. > > ############################## > - // TODO(jcbeyler): remove this extra code handling the extra trace for > Please fix all these TODO's :) > > ############################## > - heapMonitoring.hpp: > // TODO(jcbeyler): is this algorithm acceptable in open source? > > Why is this comment here? What is the implication? > Have you tested any simpler algorithm? > > ############################## > - Create a sanity jtreg test. (./hotspot/make/test/JtregNative.gmk for building the agent) > > ############################## > - monitoring_period vs HeapMonitorRate, pick rate or period. > > ############################## > - globals.hpp > Why is MaxHeapTraces not settable/overridable from jvmti interface? That would be handy. > > ############################## > - jvmtiStackTraceData + ASGCT_CallFrame memory > Are the agent suppose to loop through and free all ASGCT_CallFrame? > Wouldn't it be better with some kinda protocol, like: > (*jvmti)->GetLiveTraces(jvmti, &stack_traces, &num_traces); > (*jvmti)->ReleaseTraces(jvmti, stack_traces, num_traces); > > Also using another data structure that have num_traces inside it simplifies things. > So I'm not convinced using the async structure is the best way forward. > > > I have more questions, but I think it's better if you respond and update the code first. > > Thanks! > > /Robbin > > > On 04/21/2017 11:34 PM, JC Beyler wrote: > > Hi all, > > I've added size information to the allocation sampling system. This allows the callback to remember the size of each sampled allocation. > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/ > > > > The new webrev.01 also adds the actual heap monitoring sampling system in files: > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.cpp.patch > > > > and > http://cr.openjdk.java.net/~rasbold/8171119/webrev.01/src/share/vm/runtime/heapMonitoring.hpp.patch > > > > > My next step is to add the GC part to the webrev, which will allow users to determine what objects are live and what are garbage. > > Thanks for your attention and let me know if there are any questions! > > Have a wonderful Friday! > Jc > > On Mon, Apr 17, 2017 at 12:37 PM, JC Beyler > > >>> wrote: > > Hi all, > > I worked on getting a few numbers for overhead and accuracy for my feature. I'm unsure if here is the right place to provide the full data, so I > am just > summarizing > here for now. > > - Overhead of the feature > > Using the Dacapo benchmark (http://dacapobench.org/). My initial results are that sampling provides 2.4% with a 512k sampling, 512k being our > default setting. > > - Note: this was without the tradesoap, tradebeans and tomcat benchmarks since they did not work with my JDK9 (issue between Dacapo and JDK9 it seems) > - I want to rerun next week to ensure number stability > > - Accuracy of the feature > > I wrote a small microbenchmark that allocates from two different stacktraces at a given ratio. For example, 10% of stacktrace S1 and 90% from > stacktrace > S2. The > microbenchmark was run 20 times, I averaged the results and looked for accuracy. It seems that statistically it is sound since if I allocated10% > S1 and 90% > S2, with a > sampling rate of 512k, I obtained 9.61% S1 and 90.49% S2. > > Let me know if there are any questions on the numbers and if you'd like to see some more data. > > Note: this was done using our internal JDK8 implementation since the webrev provided by > http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > > > > >> does not yet contain the whole > implementation and therefore would have been misleading. > > Thanks, > Jc > > > On Tue, Apr 4, 2017 at 3:55 PM, JC Beyler > >>> wrote: > > Hi all, > > To move the discussion forward, with Chuck Rasbold's help to make a webrev, we pushed this: > http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/index.html > > > > >> > 415 lines changed: 399 ins; 13 del; 3 mod; 51122 unchg > > This is not a final change that does the whole proposition from the JBS entry: https://bugs.openjdk.java.net/browse/JDK-8177374 > > > > >>; what it does show is parts of the implementation that is > proposed and hopefully can start the conversation going > as I work through the details. > > For example, the changes to C2 are done here for the allocations: > http://cr.openjdk.java.net/~rasbold/heapz/webrev.00/src/share/vm/opto/macro.cpp.patch > > > > > >> > > Hopefully this all makes sense and thank you for all your future comments! > Jc > > > On Tue, Dec 13, 2016 at 1:11 PM, JC Beyler > >>> > wrote: > > Hello all, > > This is a follow-up from Jeremy's initial email from last year: > http://mail.openjdk.java.net/pipermail/serviceability-dev/2015-June/017543.html > > > >> > > I've gone ahead and started working on preparing this and Jeremy and I went down the route of actually writing it up in JEP form: > https://bugs.openjdk.java.net/browse/JDK-8171119 > > > I think original conversation that happened last year in that thread still holds true: > > - We have a patch at Google that we think others might be interested in > - It provides a means to understand where the allocation hotspots are at a very low overhead > - Since it is at a low overhead, we can leave it on by default > > So I come to the mailing list with Jeremy's initial question: > "I thought I would ask if there is any interest / if I should write a JEP / if I should just forget it." > > A year ago, it seemed some thought it was a good idea, is this still true? > > Thanks, > Jc > > > > > > > > From ujwal.vangapally at oracle.com Wed May 31 05:02:12 2017 From: ujwal.vangapally at oracle.com (Ujwal Vangapally) Date: Wed, 31 May 2017 10:32:12 +0530 Subject: RFR: JDK-8178508 Co-locate remaining MM tests Message-ID: <6d1f972d-dbec-7a79-4c1d-a960d8696a18@oracle.com> Kindly review the changes made for below bug converted tonga test to JTREG test added an additional assert statement for verifying setusageThreshold() operation is successful https://bugs.openjdk.java.net/browse/JDK-8178508 tonga test case is currently at this path : http://sqe-hgi.us.oracle.com/hg/index.cgi/testbase/javase/functional/9/vm/file/tip/src/nsk/regression/b6653214/ webrev : http://cr.openjdk.java.net/~uvangapally/webrev/2017/8178508/webrev.00/ Thanks, Ujwal. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed May 31 14:42:44 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 31 May 2017 16:42:44 +0200 Subject: Possible race in jdwp invoke handling? Message-ID: Hi all, I am looking at a possible race in JDWP invoke request handling and would like your opinion. This is how I understand the handling of invoke events (please do correct me if I am wrong): 1) JDWP InvokeXXX request comes in for thread X. Handled by the "JDWP Transport Listener" thread. We call "invoker_requestInvoke()". Here, under lock protection, we take the thread-local InvokeRequest structure and fill it with the invoke data. We only do this if request->available is true. We set request->available to false, thereby preventing further attempts to add invoke requests for this thread. Any subsequent JDWP invoke commands will now return with errors, right? 2) In the context of a JVMTI callback for thread X the actual invoke will be done. Request structure will be filled with the result (exception object handle and result). request->available stays false. 3) In a third thread, the "JDWP Event Helper Thread", the return packet will be sent to the debugger. In invoker_completeInvokeRequest(), we have two guarded sections. In the first section, we reset request->available to true (A): eventHandler_lock(); /* for proper lock order */ debugMonitorEnter(invokerLock); request = threadControl_getInvokeRequest(thread); ....... request->pending = JNI_FALSE; request->started = JNI_FALSE; A) request->available = JNI_TRUE; /* For next time around */ ...... debugMonitorExit(invokerLock); eventHandler_unlock(); Then we leave the guarded section and send the jdwp answer packet back. Then we enter a second guarded section and clean up the handles for return value and exception: ... eventHandler_lock(); // for proper lock order debugMonitorEnter(invokerLock); B) deletePotentiallySavedGlobalRefs(env, request); debugMonitorExit(invokerLock); eventHandler_unlock(); --- My question is this: would it be possible for a new invoke request to be incoming in the time between the first and the second guarded section? So, could the following sequence happen: [JDWP Transport Listener] invoker_requestInvoke (request 1) [Thread X] invoker_doInvoke (request 1) [JDWP Event Helper] invoker_completeInvokeRequest (request 1) debugMonitorEnter(invokerLock); available and request->pending> debugMonitorExit(invokerLock); .... [JDWP Transport Listener] invoker_requestInvoke (request 2) [Thread X] invoker_doInvoke (request 2) -> overwrites request->exception and request->returnValue to its new values. [JDWP Event Helper] .... debugMonitorEnter(invokerLock); deletePotentiallySavedGlobalRefs(env, request); -> releases request->exception and request->returnValue, which is now interfering with request 2. debugMonitorExit(invokerLock); ? This is all theoretical. Wanted to hear your opinions first before proceeding. Kind Regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: