From staffan.larsen at oracle.com Mon Jun 1 07:18:51 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Mon, 1 Jun 2015 09:18:51 +0200 Subject: RFR(M, v10): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556AF3CD.50900@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> Message-ID: <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> Dmitry, Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. Thanks, /Staffan > On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: > > Everyone, > > Please take a look at new version of the fix. > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ > > Changes (to previous version) are in > Finalizer.java and diagnosticCommand.cpp > > This version copy data from Map.Entry<> to array of > FinalizerHistogramEntry instances then, > VM prints content of this array. > > -Dmitry > > > On 2015-05-28 21:06, Mandy Chung wrote: >> >> On 05/28/2015 07:35 AM, Peter Levart wrote: >>> Hi Mandy, >>> >>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>> Taking it further - is it simpler to return String[] of all >>>> classnames including the duplicated ones and have the VM do the >>>> count? Are you concerned with the size of the String[]? >>> >>> Yes, the histogram is much smaller than the list of all instances. >>> There can be millions of instances waiting in finalizer queue, but >>> only a few distinct classes. >> >> Do you happen to know what libraries are the major contributors to these >> millions of finalizers? >> >> It has been strongly recommended to avoid finalizers (see Effective Java >> Item 7). I'm not surprised that existing code is still using >> finalizers while we should really encourage them to migrate away from it. >> >>> What could be done in Java to simplify things in native code but still >>> not format the output is to convert the array of Map.Entry(s) into an >>> Object[] array of alternating {String, int[], String, int[], .... } >>> >>> Would this simplify native code for the price of a little extra work >>> in Java? The sizes of old and new arrays are not big (# of distinct >>> classes of finalizable objects in the queue). >> >> I also prefer writing in Java and writing less native code (much >> simplified). It's an interface that we have to agree upon and keep it >> simple. Having the returned Object[] as alternate String and int[] is a >> reasonable compromise. >> >> ReferenceQueue.java - you can move @SuppressWarning from the method to >> just the field variable "rn" >> @SuppressWarnings("unchecked") >> Reference rn = r.next; >> >> Finalizer.java >> It's better to rename printFinalizationQueue as it inspects the >> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >> this method to the end of this class and add the comment saying that >> this is invoked by VM for jcmd -finalizerinfo support and @return to >> describe the returned content. I also think you can remove >> @SuppressWarnings for this method. >> >> Mandy > > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From magnus.ihse.bursie at oracle.com Mon Jun 1 09:55:40 2015 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Mon, 01 Jun 2015 11:55:40 +0200 Subject: RFR(S): 8081037: serviceability/sa/ tests time out on Windows In-Reply-To: <556729F0.70703@oracle.com> References: <5565C048.4050801@oracle.com> <556729F0.70703@oracle.com> Message-ID: <556C2C1C.50900@oracle.com> On 2015-05-28 16:45, Yekaterina Kantserova wrote: > Hi, > > due to https://bugs.openjdk.java.net/browse/JDK-8081381 I wasn't able > to push this fix. The problem is LingeredApp.java contains JDK 9 > feature java.lang.Process.getPid() but the test library is compiled > with JDK 8 today. This issue is not trivial to solve so I suggest a > temporary fix to test/lib/Makefile. > > webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.01 The temporary fix looks good from a build perspective, I guess, as long as you're satisfied with the lib including just the files in hprof. It looks like https://bugs.openjdk.java.net/browse/JDK-8081381 is in the intersection between hotspot testing and the build infrastructure. Please keep us in the loop when discussion solutions to that. /Magnus > > Thanks, > Katja > > > > On 05/27/2015 03:02 PM, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8081037 >> webrev root: http://cr.openjdk.java.net/~ykantser/8081037/webrev.00 >> webrev jdk: http://cr.openjdk.java.net/~ykantser/8081037.jdk/webrev.00 >> webrev hotspot: >> http://cr.openjdk.java.net/~ykantser/8081037.hotspot/webrev.00 >> >> From the bug: >> "The problem is most likely that SA will pause the target process >> while it is running. In this case, the target process is the same as >> the process that launched SA. That process is also handling the >> output from SA over a pipe, but when that pipe fills up the process >> cannot empty it and the SA process is blocked because it cannot write >> any more output. Deadlock." >> >> The solutions is to start a separate target process. Dmitry Samersoff >> has already created a test application for such cases so I've decided >> to move it on the top level library instead of duplicating it. The >> test application will reside under >> test/lib/share/classes/jdk/test/lib/apps and the test under >> test/lib-test/jdk/test/lib/apps. >> >> Thanks, >> Katja > From ysr1729 at gmail.com Mon Jun 1 18:39:51 2015 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Mon, 1 Jun 2015 11:39:51 -0700 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: References: Message-ID: Thanks for the review of the patch for 8-dev (from the ticket), Staffan. Sorry for the delay in getting the official webrev out -- it took me a while to first get set up with an hs9 repo (thanks Jon!) and then get my openjdk credentials updated (thanks Mark!). Here's the webrev against hs9 for official review:- http://cr.openjdk.java.net/~ysr/JDK-8080345/webrev.00/ I built and tested the change (on both 8-dev whose patch was attached with the original bug, as well as this with hs9) and verified that the counter value for generations, in the perfdata file, was now 2 instead of the previous 3. thanks! -- ramki On Mon, May 18, 2015 at 1:22 AM, Staffan Larsen wrote: > Looks like a good patch to me. > > /Staffan > > On 14 maj 2015, at 18:12, Srinivas Ramakrishna wrote: > > https://bugs.openjdk.java.net/browse/JDK-8080345 > > > > On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna > wrote: > >> >> With perm gen going away (and being replaced by metaspace) in JDK 8, it >> makes sense that the counter >> sun.gc.policy.generations should be "2", rather than "3". However, in JDK >> 8 that counter still says 3. >> As I understand, the intention was that this counter would allow you to >> (for example) know the range of >> the sun.gc.generation.$num.* counters describing each of $num < >> sun.gc.policy.generations in the heap. >> Recall that the erstwhile perm gen in JDK 7 used to be synonymous with >> sun.gc.generation.2, but the >> JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. >> >> The fix is simple, and I can submit a patch. Is there an existing bug for >> this? >> >> thanks! >> -- ramki >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Tue Jun 2 07:36:29 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 02 Jun 2015 00:36:29 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <555B47EA.3000900@oracle.com> References: <555B47EA.3000900@oracle.com> Message-ID: <556D5CFD.1020709@oracle.com> [Adding core-libs-dev at openjdk.java.net since this update includes changes to jdk/test library code] Please review the updated webrev: Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 There were concerns about the new hotspot tests referencing jdk tests. One concern was that if the jdk tests change, they could break the hotspot tests, and this might initially go undetected. The other concern is that if the jdk and hotspot tests are placed in separate test bundles, then it would not be possible to run the hotspot tests. Because of these concerns, I moved the tests that were in hotspot/test/runtime/CDSJDITests to instead be in jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the tests in the process. Also, I had to update the jdk version of ProcessTool.java to include the createJavaProcessBuilder() variant that is in the hotspot version of ProcessTool.java. Lastly, in CDSJITTest.java I changed: OutputAnalyzer output = new OutputAnalyzer(pb.start()); to instead be: OutputAnalyzer output = ProcessTools.executeProcess(pb); I had to do this since the jdk version of the OutputAnalyzer constructor is not public. The 1st version is what is commonly used in hostspot tests, and the 2nd version is what is commonly used in jdk tests. I decided to adopt the jdk way rather than make the OutputAnalyzer constructors public, although this will probably happen eventually when the two versions are unified. thanks, Chris On 5/19/15 7:25 AM, Chris Plummer wrote: > Hi, > > Please review the following changes for allowing java debugging when > CDS is enabled. > > Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ > Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 > > The VM changes are simple. I removed the check that prevents debugging > with CDS enabled, and added logic that will map the CDS archive RW > when debugging is enabled. > > The tests are a bit more complex. There are a bunch of existing JDI > tests for testing debugging support. Rather than start from scratch or > clone them, I instead just wrote wrapper tests that put the relevant > JDI test classes in the archive, and then invoke the JDI test. I did > this for 3 of the JDI tests. If you feel there are others that would > be good candidates, I'd be happy to add them. I'm looking for ones > that would result in modification of the RO class metadata, such as > setting a breakpoint (for which I already added two tests). > > Testing done: > -Using JPRT to run the new jtreg tests on all platforms. > -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. > -Regular JPRT "-testset hotspot" run > -Putting the JCK JVMTI tests in the archive and then running them. > -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and > then running them. > -Putting a simple test class in the archive and then setting a > breakpoint on it using jdb > > Some of the above testing resulted in the discovery of bugs that still > need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. > > I also verified that without the change to map the archive RW, the > above testing resulted in a SEGV, which is what you would expect (and > actually want to see to prove that the testing is effective). > > thanks, > > Chris > From dmitry.samersoff at oracle.com Tue Jun 2 11:12:48 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 02 Jun 2015 14:12:48 +0300 Subject: RFR(M, v11): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> Message-ID: <556D8FB0.4070608@oracle.com> Staffan, > Instead of hardcoding the field offsets, you can use > InstanceKlass::find_field and fieldDescriptor::offset to find the > offset at runtime. Done. Please, see http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to oop.inline.hpp leaving a room for further cleanup because I found couple of places in hotspot that implements mostly similar functionality. -Dmitry On 2015-06-01 10:18, Staffan Larsen wrote: > Dmitry, > > Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. > > Thanks, > /Staffan > >> On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: >> >> Everyone, >> >> Please take a look at new version of the fix. >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >> >> Changes (to previous version) are in >> Finalizer.java and diagnosticCommand.cpp >> >> This version copy data from Map.Entry<> to array of >> FinalizerHistogramEntry instances then, >> VM prints content of this array. >> >> -Dmitry >> >> >> On 2015-05-28 21:06, Mandy Chung wrote: >>> >>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>> Hi Mandy, >>>> >>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>> Taking it further - is it simpler to return String[] of all >>>>> classnames including the duplicated ones and have the VM do the >>>>> count? Are you concerned with the size of the String[]? >>>> >>>> Yes, the histogram is much smaller than the list of all instances. >>>> There can be millions of instances waiting in finalizer queue, but >>>> only a few distinct classes. >>> >>> Do you happen to know what libraries are the major contributors to these >>> millions of finalizers? >>> >>> It has been strongly recommended to avoid finalizers (see Effective Java >>> Item 7). I'm not surprised that existing code is still using >>> finalizers while we should really encourage them to migrate away from it. >>> >>>> What could be done in Java to simplify things in native code but still >>>> not format the output is to convert the array of Map.Entry(s) into an >>>> Object[] array of alternating {String, int[], String, int[], .... } >>>> >>>> Would this simplify native code for the price of a little extra work >>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>> classes of finalizable objects in the queue). >>> >>> I also prefer writing in Java and writing less native code (much >>> simplified). It's an interface that we have to agree upon and keep it >>> simple. Having the returned Object[] as alternate String and int[] is a >>> reasonable compromise. >>> >>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>> just the field variable "rn" >>> @SuppressWarnings("unchecked") >>> Reference rn = r.next; >>> >>> Finalizer.java >>> It's better to rename printFinalizationQueue as it inspects the >>> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >>> this method to the end of this class and add the comment saying that >>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>> describe the returned content. I also think you can remove >>> @SuppressWarnings for this method. >>> >>> Mandy >> >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mikael.gerdin at oracle.com Tue Jun 2 12:06:36 2015 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Tue, 02 Jun 2015 14:06:36 +0200 Subject: RFR(M, v11): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556D8FB0.4070608@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> Message-ID: <556D9C4C.9080200@oracle.com> Hi Dmitry, On 2015-06-02 13:12, Dmitry Samersoff wrote: > Staffan, > >> Instead of hardcoding the field offsets, you can use >> InstanceKlass::find_field and fieldDescriptor::offset to find the >> offset at runtime. > > Done. Please, see > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 > > I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to > oop.inline.hpp leaving a room for further cleanup because I found couple > of places in hotspot that implements mostly similar functionality. Sorry for this sort of drive-by review, but I really don't think get_field_offset_by_name should be in the oop class. If anywhere it should be on InstanceKlass, and using Symbol* to identify a Klass* seems incorrect since the same symbol can refer to different classes in different class loader contexts. /Mikael > > -Dmitry > > On 2015-06-01 10:18, Staffan Larsen wrote: >> Dmitry, >> >> Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. >> >> Thanks, >> /Staffan >> >>> On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: >>> >>> Everyone, >>> >>> Please take a look at new version of the fix. >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>> >>> Changes (to previous version) are in >>> Finalizer.java and diagnosticCommand.cpp >>> >>> This version copy data from Map.Entry<> to array of >>> FinalizerHistogramEntry instances then, >>> VM prints content of this array. >>> >>> -Dmitry >>> >>> >>> On 2015-05-28 21:06, Mandy Chung wrote: >>>> >>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>> Hi Mandy, >>>>> >>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>> Taking it further - is it simpler to return String[] of all >>>>>> classnames including the duplicated ones and have the VM do the >>>>>> count? Are you concerned with the size of the String[]? >>>>> >>>>> Yes, the histogram is much smaller than the list of all instances. >>>>> There can be millions of instances waiting in finalizer queue, but >>>>> only a few distinct classes. >>>> >>>> Do you happen to know what libraries are the major contributors to these >>>> millions of finalizers? >>>> >>>> It has been strongly recommended to avoid finalizers (see Effective Java >>>> Item 7). I'm not surprised that existing code is still using >>>> finalizers while we should really encourage them to migrate away from it. >>>> >>>>> What could be done in Java to simplify things in native code but still >>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>> >>>>> Would this simplify native code for the price of a little extra work >>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>> classes of finalizable objects in the queue). >>>> >>>> I also prefer writing in Java and writing less native code (much >>>> simplified). It's an interface that we have to agree upon and keep it >>>> simple. Having the returned Object[] as alternate String and int[] is a >>>> reasonable compromise. >>>> >>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>> just the field variable "rn" >>>> @SuppressWarnings("unchecked") >>>> Reference rn = r.next; >>>> >>>> Finalizer.java >>>> It's better to rename printFinalizationQueue as it inspects the >>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >>>> this method to the end of this class and add the comment saying that >>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>> describe the returned content. I also think you can remove >>>> @SuppressWarnings for this method. >>>> >>>> Mandy >>> >>> >>> -- >>> Dmitry Samersoff >>> Oracle Java development team, Saint Petersburg, Russia >>> * I would love to change the world, but they won't give me the sources. >> > > From yasuenag at gmail.com Tue Jun 2 15:17:55 2015 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Wed, 3 Jun 2015 00:17:55 +0900 Subject: PING: RFR: JDK-8072913: [REDO] GCCause should distinguish jcmd GC.run from System.gc() In-Reply-To: <55672019.7070406@gmail.com> References: <54DB6113.4080104@gmail.com> <54DDEF20.6010501@oracle.com> <54DF4956.7030709@gmail.com> <55003F77.80606@gmail.com> <553504CF.8080904@gmail.com> <5566259B.8010202@oracle.com> <556658C1.4090300@gmail.com> <55668C13.9080508@gmail.com> <55671C86.5000109@oracle.com> <55672019.7070406@gmail.com> Message-ID: I need more reviewer. Could you review? http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.02/ Thanks, Yasumasa 2015/05/28 23:02 "Yasumasa Suenaga" : > I'll sponsor it. >> > > Thank you Jesper! > I will send a chengeset after reviewing. > > > Yasumasa > > > On 2015/05/28 22:47, Jesper Wilhelmsson wrote: > >> Looks good! >> I'll sponsor it. >> /Jesper >> >> Yasumasa Suenaga skrev den 28/5/15 05:31: >> >>> I've uploaded new webrev: >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.02/ >>> >>> I need a Sponsor and more reviewer. >>> Please review it. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2015/05/28 8:52, Yasumasa Suenaga wrote: >>> >>>> Hi Jesper, >>>> >>>> Thank you for your comment. >>>> I will fix it. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> On 2015/05/28 5:14, Jesper Wilhelmsson wrote: >>>> >>>>> Hi, >>>>> >>>>> I like that you removed _jvmti_force_gc from is_user_requested_gc() >>>>> and used >>>>> this method throughout. It is cleaner and is_user_requested_gc() makes >>>>> more >>>>> sense now. >>>>> >>>>> In vmCMSOperations.cpp I think the comment should say >>>>> GCCause::_dcmd_gc_run. >>>>> >>>>> Besides that minor comment, looks good! >>>>> >>>>> Thanks, >>>>> /Jesper >>>>> >>>>> >>>>> Yasumasa Suenaga skrev den 20/4/15 15:53: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I've uploaded webrev for this enhancement. >>>>>> Could you review it? >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> On 2015/03/11 22:13, Yasumasa Suenaga wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> So I think we can remove _jvmti_force_gc from >>>>>>>> is_user_requested_gc() and add >>>>>>>> _dcmd_gc_run >>>>>>>> to it. >>>>>>>> >>>>>>> >>>>>>> I've uploaded new webrev, and I've applied it to new patch. >>>>>>> Could you review it? >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.01/ >>>>>>> >>>>>>> I also updated jtreg testcase. >>>>>>> It works fine in my environment. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> On 2015/02/14 22:10, Yasumasa Suenaga wrote: >>>>>>> >>>>>>>> Hi Mikael, >>>>>>>> >>>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() >>>>>>>>> which >>>>>>>>> returns true for some set of GCCause enum values, such as >>>>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>>>>> >>>>>>>> >>>>>>>> Can I add _dcmd_gc_run to GCCause::is_user_requested_gc() ? >>>>>>>> This function is used with >>>>>>>> GCCause::is_serviceability_requested_gc() . >>>>>>>> CMSCollector::is_external_interruption() and >>>>>>>> AdaptiveSizePolicy::check_gc_overhead_limit() >>>>>>>> >>>>>>>> is_user_requested_gc() and is_serviceability_requested_gc() checkes >>>>>>>> _jvmti_force_gc >>>>>>>> is selected. >>>>>>>> So I think we can remove _jvmti_force_gc from >>>>>>>> is_user_requested_gc() and add >>>>>>>> _dcmd_gc_run >>>>>>>> to it. >>>>>>>> >>>>>>>> A "grep" for _java_lang_system_gc should yield more places where >>>>>>>>> updates may >>>>>>>>> be necessary. >>>>>>>>> >>>>>>>> >>>>>>>> We can use GCCause::is_user_requested_gc() if the proposal in above >>>>>>>> is >>>>>>>> accepted. >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2015/02/13 21:33, Mikael Gerdin wrote: >>>>>>>> >>>>>>>>> Hi Yasumasa, >>>>>>>>> >>>>>>>>> On 2015-02-11 15:02, Yasumasa Suenaga wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I've committed JDK-8068589 to add new GCCause - Diagnostic >>>>>>>>>> Command. >>>>>>>>>> However, it has been backouted because test is failed [1] and it >>>>>>>>>> is not >>>>>>>>>> considered >>>>>>>>>> about concurrent GC: -XX:+ExplicitGCInvokesConcurrent [2]. >>>>>>>>>> >>>>>>>>>> I've created patch for this enhancement. >>>>>>>>>> Could you review it? >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8072913/webrev.00/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'd prefer if you could add a GCCause::is_system_gc_equivalent() >>>>>>>>> which >>>>>>>>> returns true for some set of GCCause enum values, such as >>>>>>>>> _java_lang_system_gc and _dcmd_gc_run >>>>>>>>> >>>>>>>>> Given that the documentation of the GC.run command is: >>>>>>>>> "GC.run >>>>>>>>> Call java.lang.System.gc(). >>>>>>>>> >>>>>>>>> Impact: Medium: Depends on Java heap size and content. >>>>>>>>> >>>>>>>>> Syntax: GC.run" >>>>>>>>> >>>>>>>>> I interpret the documentation that the GC is supposed to be (for >>>>>>>>> all intents >>>>>>>>> and purposes) equivalent to the application invoking System.gc(). >>>>>>>>> >>>>>>>>> This would also require updates to other places where we refer to >>>>>>>>> the >>>>>>>>> _java_lang_system_gc GCCause, such as >>>>>>>>> UseAdaptiveSizePolicyWithSystemGC >>>>>>>>> >>>>>>>>> A "grep" for _java_lang_system_gc should yield more places where >>>>>>>>> updates may >>>>>>>>> be necessary. >>>>>>>>> >>>>>>>>> /Mikael >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I'm jdk9 committer, but I'm not employee at Oracle. >>>>>>>>>> So I need a Sponsor. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> >>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011957.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [2] >>>>>>>>>> >>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-February/011962.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.plummer at oracle.com Tue Jun 2 18:55:47 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 02 Jun 2015 11:55:47 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556D5CFD.1020709@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> Message-ID: <556DFC33.4060600@oracle.com> I'm going to have to separate out the ProcessTool.java changes into a separate changeset (and CR). In the meantime, feel free to review what I have below. The code won't be changing at all when I separate out the ProcessTool.java changes. thanks, Chris On 6/2/15 12:36 AM, Chris Plummer wrote: > [Adding core-libs-dev at openjdk.java.net since this update includes > changes to jdk/test library code] > > Please review the updated webrev: > > Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 > > There were concerns about the new hotspot tests referencing jdk tests. > One concern was that if the jdk tests change, they could break the > hotspot tests, and this might initially go undetected. The other > concern is that if the jdk and hotspot tests are placed in separate > test bundles, then it would not be possible to run the hotspot tests. > > Because of these concerns, I moved the tests that were in > hotspot/test/runtime/CDSJDITests to instead be in > jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the > tests in the process. Also, I had to update the jdk version of > ProcessTool.java to include the createJavaProcessBuilder() variant > that is in the hotspot version of ProcessTool.java. > > Lastly, in CDSJITTest.java I changed: > > OutputAnalyzer output = new OutputAnalyzer(pb.start()); > > to instead be: > > OutputAnalyzer output = ProcessTools.executeProcess(pb); > > I had to do this since the jdk version of the OutputAnalyzer > constructor is not public. The 1st version is what is commonly used in > hostspot tests, and the 2nd version is what is commonly used in jdk > tests. I decided to adopt the jdk way rather than make the > OutputAnalyzer constructors public, although this will probably happen > eventually when the two versions are unified. > > thanks, > > Chris > > > On 5/19/15 7:25 AM, Chris Plummer wrote: >> Hi, >> >> Please review the following changes for allowing java debugging when >> CDS is enabled. >> >> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >> >> The VM changes are simple. I removed the check that prevents >> debugging with CDS enabled, and added logic that will map the CDS >> archive RW when debugging is enabled. >> >> The tests are a bit more complex. There are a bunch of existing JDI >> tests for testing debugging support. Rather than start from scratch >> or clone them, I instead just wrote wrapper tests that put the >> relevant JDI test classes in the archive, and then invoke the JDI >> test. I did this for 3 of the JDI tests. If you feel there are others >> that would be good candidates, I'd be happy to add them. I'm looking >> for ones that would result in modification of the RO class metadata, >> such as setting a breakpoint (for which I already added two tests). >> >> Testing done: >> -Using JPRT to run the new jtreg tests on all platforms. >> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >> -Regular JPRT "-testset hotspot" run >> -Putting the JCK JVMTI tests in the archive and then running them. >> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >> then running them. >> -Putting a simple test class in the archive and then setting a >> breakpoint on it using jdb >> >> Some of the above testing resulted in the discovery of bugs that >> still need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >> >> I also verified that without the change to map the archive RW, the >> above testing resulted in a SEGV, which is what you would expect (and >> actually want to see to prove that the testing is effective). >> >> thanks, >> >> Chris >> > From coleen.phillimore at oracle.com Tue Jun 2 19:58:12 2015 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Tue, 02 Jun 2015 15:58:12 -0400 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file Message-ID: <556E0AD4.3010207@oracle.com> Summary: Use the Events::log function to save redefined classes for output to the hs_err file. This change adds lines to the hs_err_pid file which names classes redefined: Classes redefined (2 events): Event: 11.867 Thread 0x00007fd378199800 redefined class name=RedefineRunningMethods$B, count=1 Event: 26.746 Thread 0x00007fd378199800 redefined class name=RedefineRunningMethods$B, count=2 Tested with intentional crash to see hs_err_pid files, also tested without intentional crashes vm.redefine.testlist. open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ bug link https://bugs.openjdk.java.net/browse/JDK-8081219 Thanks, Coleen From dmitry.samersoff at oracle.com Tue Jun 2 21:04:53 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 03 Jun 2015 00:04:53 +0300 Subject: RFR(M, v11): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556D9C4C.9080200@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556D9C4C.9080200@oracle.com> Message-ID: <556E1A75.3070705@oracle.com> Mikael, The reason of placing get_filed_offset_by_name to the oop.inline is that hotspot widely duplicate this code. Symbol is used to identify a field within klass, not a klass them self so I think it's OK to have it tied to the oopDesc. -Dmitry On 2015-06-02 15:06, Mikael Gerdin wrote: > Hi Dmitry, > > On 2015-06-02 13:12, Dmitry Samersoff wrote: >> Staffan, >> >>> Instead of hardcoding the field offsets, you can use >>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>> offset at runtime. >> >> Done. Please, see >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 >> >> I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to >> oop.inline.hpp leaving a room for further cleanup because I found couple >> of places in hotspot that implements mostly similar functionality. > > Sorry for this sort of drive-by review, but I really don't think > get_field_offset_by_name should be in the oop class. If anywhere it > should be on InstanceKlass, and using Symbol* to identify a Klass* seems > incorrect since the same symbol can refer to different classes in > different class loader contexts. > > /Mikael > >> >> -Dmitry >> >> On 2015-06-01 10:18, Staffan Larsen wrote: >>> Dmitry, >>> >>> Instead of hardcoding the field offsets, you can use >>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>> offset at runtime. >>> >>> Thanks, >>> /Staffan >>> >>>> On 31 maj 2015, at 13:43, Dmitry Samersoff >>>> wrote: >>>> >>>> Everyone, >>>> >>>> Please take a look at new version of the fix. >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>>> >>>> Changes (to previous version) are in >>>> Finalizer.java and diagnosticCommand.cpp >>>> >>>> This version copy data from Map.Entry<> to array of >>>> FinalizerHistogramEntry instances then, >>>> VM prints content of this array. >>>> >>>> -Dmitry >>>> >>>> >>>> On 2015-05-28 21:06, Mandy Chung wrote: >>>>> >>>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>>> Hi Mandy, >>>>>> >>>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>>> Taking it further - is it simpler to return String[] of all >>>>>>> classnames including the duplicated ones and have the VM do the >>>>>>> count? Are you concerned with the size of the String[]? >>>>>> >>>>>> Yes, the histogram is much smaller than the list of all instances. >>>>>> There can be millions of instances waiting in finalizer queue, but >>>>>> only a few distinct classes. >>>>> >>>>> Do you happen to know what libraries are the major contributors to >>>>> these >>>>> millions of finalizers? >>>>> >>>>> It has been strongly recommended to avoid finalizers (see Effective >>>>> Java >>>>> Item 7). I'm not surprised that existing code is still using >>>>> finalizers while we should really encourage them to migrate away >>>>> from it. >>>>> >>>>>> What could be done in Java to simplify things in native code but >>>>>> still >>>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>>> >>>>>> Would this simplify native code for the price of a little extra work >>>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>>> classes of finalizable objects in the queue). >>>>> >>>>> I also prefer writing in Java and writing less native code (much >>>>> simplified). It's an interface that we have to agree upon and keep it >>>>> simple. Having the returned Object[] as alternate String and int[] >>>>> is a >>>>> reasonable compromise. >>>>> >>>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>>> just the field variable "rn" >>>>> @SuppressWarnings("unchecked") >>>>> Reference rn = r.next; >>>>> >>>>> Finalizer.java >>>>> It's better to rename printFinalizationQueue as it inspects the >>>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you >>>>> move >>>>> this method to the end of this class and add the comment saying that >>>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>>> describe the returned content. I also think you can remove >>>>> @SuppressWarnings for this method. >>>>> >>>>> Mandy >>>> >>>> >>>> -- >>>> Dmitry Samersoff >>>> Oracle Java development team, Saint Petersburg, Russia >>>> * I would love to change the world, but they won't give me the sources. >>> >> >> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jon.masamitsu at oracle.com Tue Jun 2 21:16:33 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 02 Jun 2015 14:16:33 -0700 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: References: Message-ID: <556E1D31.1040507@oracle.com> Ramki, Changes look good. I'm guessing you tested by generating the perfdata by hand and verifying the contents of the perfdata. Do you think a test can be written to verify the change? If you look at test/gc/metaspace/TestMetaspacePerfCounters.java in your repository I think that is an example that can be followed. It's a jtreg test. http://openjdk.java.net/jtreg/ Jon On 06/01/2015 11:39 AM, Srinivas Ramakrishna wrote: > Thanks for the review of the patch for 8-dev (from the ticket), Staffan. > > Sorry for the delay in getting the official webrev out -- it took me a > while to first get set up with an hs9 repo (thanks Jon!) and then get > my openjdk credentials updated (thanks Mark!). > > Here's the webrev against hs9 for official review:- > > http://cr.openjdk.java.net/~ysr/JDK-8080345/webrev.00/ > > > I built and tested the change (on both 8-dev whose patch was attached > with the original bug, as well as this with hs9) and verified that the > counter value for generations, in the perfdata file, was now 2 instead > of the previous 3. > > thanks! > -- ramki > > > On Mon, May 18, 2015 at 1:22 AM, Staffan Larsen > > wrote: > > Looks like a good patch to me. > > /Staffan > >> On 14 maj 2015, at 18:12, Srinivas Ramakrishna > > wrote: >> >> https://bugs.openjdk.java.net/browse/JDK-8080345 >> >> >> >> On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna >> > wrote: >> >> >> With perm gen going away (and being replaced by metaspace) in >> JDK 8, it makes sense that the counter >> sun.gc.policy.generations should be "2", rather than "3". >> However, in JDK 8 that counter still says 3. >> As I understand, the intention was that this counter would >> allow you to (for example) know the range of >> the sun.gc.generation.$num.* counters describing each of $num >> < sun.gc.policy.generations in the heap. >> Recall that the erstwhile perm gen in JDK 7 used to be >> synonymous with sun.gc.generation.2, but the >> JDK 8 avatars are now sun.gc.metaspace and >> sun.gc.compressedclassspace. >> >> The fix is simple, and I can submit a patch. Is there an >> existing bug for this? >> >> thanks! >> -- ramki >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailo.seledtsov at oracle.com Tue Jun 2 21:39:45 2015 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Tue, 02 Jun 2015 14:39:45 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556DFC33.4060600@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556DFC33.4060600@oracle.com> Message-ID: <556E22A1.8080605@oracle.com> Changes look good to me. Misha On 6/2/2015 11:55 AM, Chris Plummer wrote: > I'm going to have to separate out the ProcessTool.java changes into a > separate changeset (and CR). In the meantime, feel free to review what > I have below. The code won't be changing at all when I separate out > the ProcessTool.java changes. > > thanks, > > Chris > > On 6/2/15 12:36 AM, Chris Plummer wrote: >> [Adding core-libs-dev at openjdk.java.net since this update includes >> changes to jdk/test library code] >> >> Please review the updated webrev: >> >> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >> >> There were concerns about the new hotspot tests referencing jdk >> tests. One concern was that if the jdk tests change, they could break >> the hotspot tests, and this might initially go undetected. The other >> concern is that if the jdk and hotspot tests are placed in separate >> test bundles, then it would not be possible to run the hotspot tests. >> >> Because of these concerns, I moved the tests that were in >> hotspot/test/runtime/CDSJDITests to instead be in >> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the >> tests in the process. Also, I had to update the jdk version of >> ProcessTool.java to include the createJavaProcessBuilder() variant >> that is in the hotspot version of ProcessTool.java. >> >> Lastly, in CDSJITTest.java I changed: >> >> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >> >> to instead be: >> >> OutputAnalyzer output = ProcessTools.executeProcess(pb); >> >> I had to do this since the jdk version of the OutputAnalyzer >> constructor is not public. The 1st version is what is commonly used >> in hostspot tests, and the 2nd version is what is commonly used in >> jdk tests. I decided to adopt the jdk way rather than make the >> OutputAnalyzer constructors public, although this will probably >> happen eventually when the two versions are unified. >> >> thanks, >> >> Chris >> >> >> On 5/19/15 7:25 AM, Chris Plummer wrote: >>> Hi, >>> >>> Please review the following changes for allowing java debugging when >>> CDS is enabled. >>> >>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> The VM changes are simple. I removed the check that prevents >>> debugging with CDS enabled, and added logic that will map the CDS >>> archive RW when debugging is enabled. >>> >>> The tests are a bit more complex. There are a bunch of existing JDI >>> tests for testing debugging support. Rather than start from scratch >>> or clone them, I instead just wrote wrapper tests that put the >>> relevant JDI test classes in the archive, and then invoke the JDI >>> test. I did this for 3 of the JDI tests. If you feel there are >>> others that would be good candidates, I'd be happy to add them. I'm >>> looking for ones that would result in modification of the RO class >>> metadata, such as setting a breakpoint (for which I already added >>> two tests). >>> >>> Testing done: >>> -Using JPRT to run the new jtreg tests on all platforms. >>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>> -Regular JPRT "-testset hotspot" run >>> -Putting the JCK JVMTI tests in the archive and then running them. >>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >>> then running them. >>> -Putting a simple test class in the archive and then setting a >>> breakpoint on it using jdb >>> >>> Some of the above testing resulted in the discovery of bugs that >>> still need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>> >>> I also verified that without the change to map the archive RW, the >>> above testing resulted in a SEGV, which is what you would expect >>> (and actually want to see to prove that the testing is effective). >>> >>> thanks, >>> >>> Chris >>> >> > From serguei.spitsyn at oracle.com Tue Jun 2 22:18:37 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 02 Jun 2015 15:18:37 -0700 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556E0AD4.3010207@oracle.com> References: <556E0AD4.3010207@oracle.com> Message-ID: <556E2BBD.2000405@oracle.com> Coleen, The fix looks good. But I have a question. Would it be useful to log the beginning of a redefinition as well so that it is known what class is currently being redefined? Thanks, Serguei On 6/2/15 12:58 PM, Coleen Phillimore wrote: > Summary: Use the Events::log function to save redefined classes for > output to the hs_err file. > > This change adds lines to the hs_err_pid file which names classes > redefined: > > Classes redefined (2 events): > Event: 11.867 Thread 0x00007fd378199800 redefined class > name=RedefineRunningMethods$B, count=1 > Event: 26.746 Thread 0x00007fd378199800 redefined class > name=RedefineRunningMethods$B, count=2 > > Tested with intentional crash to see hs_err_pid files, also tested > without intentional crashes vm.redefine.testlist. > > open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ > bug link https://bugs.openjdk.java.net/browse/JDK-8081219 > > Thanks, > Coleen > > From jiangli.zhou at oracle.com Tue Jun 2 23:23:16 2015 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 2 Jun 2015 16:23:16 -0700 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556E0AD4.3010207@oracle.com> References: <556E0AD4.3010207@oracle.com> Message-ID: <4C105391-4AE2-46B5-AFBA-12ED0788B25B@oracle.com> Hi Coleen, The change looks good to me. Thanks, Jiangli On Jun 2, 2015, at 12:58 PM, Coleen Phillimore wrote: > Summary: Use the Events::log function to save redefined classes for output to the hs_err file. > > This change adds lines to the hs_err_pid file which names classes redefined: > > Classes redefined (2 events): > Event: 11.867 Thread 0x00007fd378199800 redefined class name=RedefineRunningMethods$B, count=1 > Event: 26.746 Thread 0x00007fd378199800 redefined class name=RedefineRunningMethods$B, count=2 > > Tested with intentional crash to see hs_err_pid files, also tested without intentional crashes vm.redefine.testlist. > > open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ > bug link https://bugs.openjdk.java.net/browse/JDK-8081219 > > Thanks, > Coleen > > From chris.plummer at oracle.com Wed Jun 3 03:25:35 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Tue, 02 Jun 2015 20:25:35 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556DFC33.4060600@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556DFC33.4060600@oracle.com> Message-ID: <556E73AF.3020606@oracle.com> An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Wed Jun 3 06:06:40 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 03 Jun 2015 08:06:40 +0200 Subject: RFR(M, v11): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556D8FB0.4070608@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> Message-ID: <556E9970.8010704@gmail.com> Hi Dmitry, On 06/02/2015 01:12 PM, Dmitry Samersoff wrote: > Staffan, > >> Instead of hardcoding the field offsets, you can use >> InstanceKlass::find_field and fieldDescriptor::offset to find the >> offset at runtime. > Done. Please, see > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 In the jdk part, now that you have a FinalizerHistogramEntry class, you can simplify the code even further: private static final class FinalizerHistogramEntry { int instanceCount; final String className; int getInstanceCount() { return instanceCount; } FinalizerHistogramEntry(String className) { this.className = className; } } static Object[] getFinalizerHistogram() { Map countMap = new HashMap<>(); queue.forEach(r -> { Object referent = r.get(); if (referent != null) { countMap.computeIfAbsent( referent.getClass().getName(), FinalizerHistogramEntry::new).instanceCount++; /* Clear stack slot containing this variable, to decrease the chances of false retention with a conservative GC */ referent = null; } }); FinalizerHistogramEntry fhe[] = countMap.values() .toArray(new FinalizerHistogramEntry[countMap.size()]); Arrays.sort(fhe, Comparator.comparingInt( FinalizerHistogramEntry::getInstanceCount).reversed()); return fhe; } ... see, no copying loop required. Also notice the access modifier used on the nested class and it's fields/method/constructor. This way the class is private and fields can be accessed from getFinalizerHistogram and lambda without compiler generating access bridges. Regards, Peter > > I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to > oop.inline.hpp leaving a room for further cleanup because I found couple > of places in hotspot that implements mostly similar functionality. > > -Dmitry > > On 2015-06-01 10:18, Staffan Larsen wrote: >> Dmitry, >> >> Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. >> >> Thanks, >> /Staffan >> >>> On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: >>> >>> Everyone, >>> >>> Please take a look at new version of the fix. >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>> >>> Changes (to previous version) are in >>> Finalizer.java and diagnosticCommand.cpp >>> >>> This version copy data from Map.Entry<> to array of >>> FinalizerHistogramEntry instances then, >>> VM prints content of this array. >>> >>> -Dmitry >>> >>> >>> On 2015-05-28 21:06, Mandy Chung wrote: >>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>> Hi Mandy, >>>>> >>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>> Taking it further - is it simpler to return String[] of all >>>>>> classnames including the duplicated ones and have the VM do the >>>>>> count? Are you concerned with the size of the String[]? >>>>> Yes, the histogram is much smaller than the list of all instances. >>>>> There can be millions of instances waiting in finalizer queue, but >>>>> only a few distinct classes. >>>> Do you happen to know what libraries are the major contributors to these >>>> millions of finalizers? >>>> >>>> It has been strongly recommended to avoid finalizers (see Effective Java >>>> Item 7). I'm not surprised that existing code is still using >>>> finalizers while we should really encourage them to migrate away from it. >>>> >>>>> What could be done in Java to simplify things in native code but still >>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>> >>>>> Would this simplify native code for the price of a little extra work >>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>> classes of finalizable objects in the queue). >>>> I also prefer writing in Java and writing less native code (much >>>> simplified). It's an interface that we have to agree upon and keep it >>>> simple. Having the returned Object[] as alternate String and int[] is a >>>> reasonable compromise. >>>> >>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>> just the field variable "rn" >>>> @SuppressWarnings("unchecked") >>>> Reference rn = r.next; >>>> >>>> Finalizer.java >>>> It's better to rename printFinalizationQueue as it inspects the >>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >>>> this method to the end of this class and add the comment saying that >>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>> describe the returned content. I also think you can remove >>>> @SuppressWarnings for this method. >>>> >>>> Mandy >>> >>> -- >>> Dmitry Samersoff >>> Oracle Java development team, Saint Petersburg, Russia >>> * I would love to change the world, but they won't give me the sources. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysr1729 at gmail.com Wed Jun 3 06:15:01 2015 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Tue, 2 Jun 2015 23:15:01 -0700 Subject: PerfData counter: sun.gc.policy.generations in JDK 8 In-Reply-To: <556E1D31.1040507@oracle.com> References: <556E1D31.1040507@oracle.com> Message-ID: Thanks Jon for the review and the pointer to the test. I'll get back to you later this week with a suitable test. -- Ramki ysr1729 > On Jun 2, 2015, at 14:16, Jon Masamitsu wrote: > > Ramki, > > Changes look good. > > I'm guessing you tested by generating the > perfdata by hand and verifying the contents > of the perfdata. Do you think a test can > be written to verify the change? If you look at > > test/gc/metaspace/TestMetaspacePerfCounters.java > > in your repository I think that is an example that > can be followed. > > It's a jtreg test. > > http://openjdk.java.net/jtreg/ > > Jon > >> On 06/01/2015 11:39 AM, Srinivas Ramakrishna wrote: >> Thanks for the review of the patch for 8-dev (from the ticket), Staffan. >> >> Sorry for the delay in getting the official webrev out -- it took me a while to first get set up with an hs9 repo (thanks Jon!) and then get my openjdk credentials updated (thanks Mark!). >> >> Here's the webrev against hs9 for official review:- >> >> http://cr.openjdk.java.net/~ysr/JDK-8080345/webrev.00/ >> >> I built and tested the change (on both 8-dev whose patch was attached with the original bug, as well as this with hs9) and verified that the counter value for generations, in the perfdata file, was now 2 instead of the previous 3. >> >> thanks! >> -- ramki >> >> >> On Mon, May 18, 2015 at 1:22 AM, Staffan Larsen wrote: >>> Looks like a good patch to me. >>> >>> /Staffan >>> >>>> On 14 maj 2015, at 18:12, Srinivas Ramakrishna wrote: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8080345 >>>> >>>> >>>> >>>>> On Wed, May 13, 2015 at 1:08 PM, Srinivas Ramakrishna wrote: >>>>> >>>>> With perm gen going away (and being replaced by metaspace) in JDK 8, it makes sense that the counter >>>>> sun.gc.policy.generations should be "2", rather than "3". However, in JDK 8 that counter still says 3. >>>>> As I understand, the intention was that this counter would allow you to (for example) know the range of >>>>> the sun.gc.generation.$num.* counters describing each of $num < sun.gc.policy.generations in the heap. >>>>> Recall that the erstwhile perm gen in JDK 7 used to be synonymous with sun.gc.generation.2, but the >>>>> JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. >>>>> >>>>> The fix is simple, and I can submit a patch. Is there an existing bug for this? >>>>> >>>>> thanks! >>>>> -- ramki > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Wed Jun 3 08:40:49 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 03 Jun 2015 01:40:49 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556E73AF.3020606@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556DFC33.4060600@oracle.com> <556E73AF.3020606@oracle.com> Message-ID: <556EBD91.6030204@oracle.com> Chris, It looks good in general. I'd suggest to rename the folder: || test/com/sun/jdi/CDSJDITests to: test/com/sun/jdi/cds There is no need to spell "JDI" as it is already a sub-folder of the com/sun/jdi and there is no need to spell "Tests" too as it is in the test repo. Also, all the folders are normally named in the lower case. Thanks, Serguei On 6/2/15 8:25 PM, Chris Plummer wrote: > Ok, I'm going to keep this as one webrev, but I did create JDK-8081771 > for the ProcessTool.java changes: > > https://bugs.openjdk.java.net/browse/JDK-8081771 > > I will commit ProcessTool.java under JDK-8081771, and the rest of the > changes under JDK-8054386. Both will then be pushed together. I also > started a new thread for the review of JDK-8081771: > > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2015-June/014930.html > http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-June/033892.html > > thanks, > > Chris > > On 6/2/15 11:55 AM, Chris Plummer wrote: >> I'm going to have to separate out the ProcessTool.java changes into a >> separate changeset (and CR). In the meantime, feel free to review >> what I have below. The code won't be changing at all when I separate >> out the ProcessTool.java changes. >> >> thanks, >> >> Chris >> >> On 6/2/15 12:36 AM, Chris Plummer wrote: >>> [Adding core-libs-dev at openjdk.java.net since this update includes >>> changes to jdk/test library code] >>> >>> Please review the updated webrev: >>> >>> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> There were concerns about the new hotspot tests referencing jdk >>> tests. One concern was that if the jdk tests change, they could >>> break the hotspot tests, and this might initially go undetected. The >>> other concern is that if the jdk and hotspot tests are placed in >>> separate test bundles, then it would not be possible to run the >>> hotspot tests. >>> >>> Because of these concerns, I moved the tests that were in >>> hotspot/test/runtime/CDSJDITests to instead be in >>> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the >>> tests in the process. Also, I had to update the jdk version of >>> ProcessTool.java to include the createJavaProcessBuilder() variant >>> that is in the hotspot version of ProcessTool.java. >>> >>> Lastly, in CDSJITTest.java I changed: >>> >>> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >>> >>> to instead be: >>> >>> OutputAnalyzer output = ProcessTools.executeProcess(pb); >>> >>> I had to do this since the jdk version of the OutputAnalyzer >>> constructor is not public. The 1st version is what is commonly used >>> in hostspot tests, and the 2nd version is what is commonly used in >>> jdk tests. I decided to adopt the jdk way rather than make the >>> OutputAnalyzer constructors public, although this will probably >>> happen eventually when the two versions are unified. >>> >>> thanks, >>> >>> Chris >>> >>> >>> On 5/19/15 7:25 AM, Chris Plummer wrote: >>>> Hi, >>>> >>>> Please review the following changes for allowing java debugging >>>> when CDS is enabled. >>>> >>>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>>> >>>> The VM changes are simple. I removed the check that prevents >>>> debugging with CDS enabled, and added logic that will map the CDS >>>> archive RW when debugging is enabled. >>>> >>>> The tests are a bit more complex. There are a bunch of existing JDI >>>> tests for testing debugging support. Rather than start from scratch >>>> or clone them, I instead just wrote wrapper tests that put the >>>> relevant JDI test classes in the archive, and then invoke the JDI >>>> test. I did this for 3 of the JDI tests. If you feel there are >>>> others that would be good candidates, I'd be happy to add them. I'm >>>> looking for ones that would result in modification of the RO class >>>> metadata, such as setting a breakpoint (for which I already added >>>> two tests). >>>> >>>> Testing done: >>>> -Using JPRT to run the new jtreg tests on all platforms. >>>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>>> -Regular JPRT "-testset hotspot" run >>>> -Putting the JCK JVMTI tests in the archive and then running them. >>>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >>>> then running them. >>>> -Putting a simple test class in the archive and then setting a >>>> breakpoint on it using jdb >>>> >>>> Some of the above testing resulted in the discovery of bugs that >>>> still need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>>> >>>> I also verified that without the change to map the archive RW, the >>>> above testing resulted in a SEGV, which is what you would expect >>>> (and actually want to see to prove that the testing is effective). >>>> >>>> thanks, >>>> >>>> Chris >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Wed Jun 3 08:48:11 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 03 Jun 2015 11:48:11 +0300 Subject: RFR(M, v12): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556E9970.8010704@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> Message-ID: <556EBF4B.6070400@oracle.com> Everyone, Updated webrev: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.12 getFinalizerHistogram and FinalizerHistogramEntry moved to separate package-private class. Hotspot code changed accordingly. getFinalizerHistogram updated to use Peter's code. -Dmitry On 2015-06-03 09:06, Peter Levart wrote: > Hi Dmitry, > > On 06/02/2015 01:12 PM, Dmitry Samersoff wrote: >> Staffan, >> >>> Instead of hardcoding the field offsets, you can use >>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>> offset at runtime. >> Done. Please, see >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 > > In the jdk part, now that you have a FinalizerHistogramEntry class, you > can simplify the code even further: > > > private static final class FinalizerHistogramEntry { > int instanceCount; > final String className; > > int getInstanceCount() { > return instanceCount; > } > > FinalizerHistogramEntry(String className) { > this.className = className; > } > } > > static Object[] getFinalizerHistogram() { > Map countMap = new HashMap<>(); > queue.forEach(r -> { > Object referent = r.get(); > if (referent != null) { > countMap.computeIfAbsent( > referent.getClass().getName(), > FinalizerHistogramEntry::new).instanceCount++; > /* Clear stack slot containing this variable, to decrease > the chances of false retention with a conservative GC */ > referent = null; > } > }); > > FinalizerHistogramEntry fhe[] = countMap.values() > .toArray(new FinalizerHistogramEntry[countMap.size()]); > Arrays.sort(fhe, > Comparator.comparingInt( > > FinalizerHistogramEntry::getInstanceCount).reversed()); > return fhe; > } > > > ... see, no copying loop required. Also notice the access modifier used > on the nested class and it's fields/method/constructor. This way the > class is private and fields can be accessed from getFinalizerHistogram > and lambda without compiler generating access bridges. > > > Regards, Peter > >> >> I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to >> oop.inline.hpp leaving a room for further cleanup because I found couple >> of places in hotspot that implements mostly similar functionality. >> >> -Dmitry >> >> On 2015-06-01 10:18, Staffan Larsen wrote: >>> Dmitry, >>> >>> Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. >>> >>> Thanks, >>> /Staffan >>> >>>> On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: >>>> >>>> Everyone, >>>> >>>> Please take a look at new version of the fix. >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>>> >>>> Changes (to previous version) are in >>>> Finalizer.java and diagnosticCommand.cpp >>>> >>>> This version copy data from Map.Entry<> to array of >>>> FinalizerHistogramEntry instances then, >>>> VM prints content of this array. >>>> >>>> -Dmitry >>>> >>>> >>>> On 2015-05-28 21:06, Mandy Chung wrote: >>>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>>> Hi Mandy, >>>>>> >>>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>>> Taking it further - is it simpler to return String[] of all >>>>>>> classnames including the duplicated ones and have the VM do the >>>>>>> count? Are you concerned with the size of the String[]? >>>>>> Yes, the histogram is much smaller than the list of all instances. >>>>>> There can be millions of instances waiting in finalizer queue, but >>>>>> only a few distinct classes. >>>>> Do you happen to know what libraries are the major contributors to these >>>>> millions of finalizers? >>>>> >>>>> It has been strongly recommended to avoid finalizers (see Effective Java >>>>> Item 7). I'm not surprised that existing code is still using >>>>> finalizers while we should really encourage them to migrate away from it. >>>>> >>>>>> What could be done in Java to simplify things in native code but still >>>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>>> >>>>>> Would this simplify native code for the price of a little extra work >>>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>>> classes of finalizable objects in the queue). >>>>> I also prefer writing in Java and writing less native code (much >>>>> simplified). It's an interface that we have to agree upon and keep it >>>>> simple. Having the returned Object[] as alternate String and int[] is a >>>>> reasonable compromise. >>>>> >>>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>>> just the field variable "rn" >>>>> @SuppressWarnings("unchecked") >>>>> Reference rn = r.next; >>>>> >>>>> Finalizer.java >>>>> It's better to rename printFinalizationQueue as it inspects the >>>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >>>>> this method to the end of this class and add the comment saying that >>>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>>> describe the returned content. I also think you can remove >>>>> @SuppressWarnings for this method. >>>>> >>>>> Mandy >>>> >>>> -- >>>> Dmitry Samersoff >>>> Oracle Java development team, Saint Petersburg, Russia >>>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mikael.gerdin at oracle.com Wed Jun 3 09:14:21 2015 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 03 Jun 2015 11:14:21 +0200 Subject: RFR(M, v11): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556E1A75.3070705@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556D9C4C.9080200@oracle.com> <556E1A75.3070705@oracle.com> Message-ID: <556EC56D.3080703@oracle.com> On 2015-06-02 23:04, Dmitry Samersoff wrote: > Mikael, > > The reason of placing get_filed_offset_by_name to the oop.inline is that > hotspot widely duplicate this code. > > Symbol is used to identify a field within klass, not a klass them self > so I think it's OK to have it tied to the oopDesc. Ok, I didn't relize that both the symbols were used for the field, sorry about that. I still strongly feel that the method should be on InstanceKlass though. /Mikael > > -Dmitry > > On 2015-06-02 15:06, Mikael Gerdin wrote: >> Hi Dmitry, >> >> On 2015-06-02 13:12, Dmitry Samersoff wrote: >>> Staffan, >>> >>>> Instead of hardcoding the field offsets, you can use >>>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>>> offset at runtime. >>> >>> Done. Please, see >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 >>> >>> I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to >>> oop.inline.hpp leaving a room for further cleanup because I found couple >>> of places in hotspot that implements mostly similar functionality. >> >> Sorry for this sort of drive-by review, but I really don't think >> get_field_offset_by_name should be in the oop class. If anywhere it >> should be on InstanceKlass, and using Symbol* to identify a Klass* seems >> incorrect since the same symbol can refer to different classes in >> different class loader contexts. >> >> /Mikael >> >>> >>> -Dmitry >>> >>> On 2015-06-01 10:18, Staffan Larsen wrote: >>>> Dmitry, >>>> >>>> Instead of hardcoding the field offsets, you can use >>>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>>> offset at runtime. >>>> >>>> Thanks, >>>> /Staffan >>>> >>>>> On 31 maj 2015, at 13:43, Dmitry Samersoff >>>>> wrote: >>>>> >>>>> Everyone, >>>>> >>>>> Please take a look at new version of the fix. >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>>>> >>>>> Changes (to previous version) are in >>>>> Finalizer.java and diagnosticCommand.cpp >>>>> >>>>> This version copy data from Map.Entry<> to array of >>>>> FinalizerHistogramEntry instances then, >>>>> VM prints content of this array. >>>>> >>>>> -Dmitry >>>>> >>>>> >>>>> On 2015-05-28 21:06, Mandy Chung wrote: >>>>>> >>>>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>>>> Hi Mandy, >>>>>>> >>>>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>>>> Taking it further - is it simpler to return String[] of all >>>>>>>> classnames including the duplicated ones and have the VM do the >>>>>>>> count? Are you concerned with the size of the String[]? >>>>>>> >>>>>>> Yes, the histogram is much smaller than the list of all instances. >>>>>>> There can be millions of instances waiting in finalizer queue, but >>>>>>> only a few distinct classes. >>>>>> >>>>>> Do you happen to know what libraries are the major contributors to >>>>>> these >>>>>> millions of finalizers? >>>>>> >>>>>> It has been strongly recommended to avoid finalizers (see Effective >>>>>> Java >>>>>> Item 7). I'm not surprised that existing code is still using >>>>>> finalizers while we should really encourage them to migrate away >>>>>> from it. >>>>>> >>>>>>> What could be done in Java to simplify things in native code but >>>>>>> still >>>>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>>>> >>>>>>> Would this simplify native code for the price of a little extra work >>>>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>>>> classes of finalizable objects in the queue). >>>>>> >>>>>> I also prefer writing in Java and writing less native code (much >>>>>> simplified). It's an interface that we have to agree upon and keep it >>>>>> simple. Having the returned Object[] as alternate String and int[] >>>>>> is a >>>>>> reasonable compromise. >>>>>> >>>>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>>>> just the field variable "rn" >>>>>> @SuppressWarnings("unchecked") >>>>>> Reference rn = r.next; >>>>>> >>>>>> Finalizer.java >>>>>> It's better to rename printFinalizationQueue as it inspects the >>>>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you >>>>>> move >>>>>> this method to the end of this class and add the comment saying that >>>>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>>>> describe the returned content. I also think you can remove >>>>>> @SuppressWarnings for this method. >>>>>> >>>>>> Mandy >>>>> >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. >>>> >>> >>> > > From lois.foltan at oracle.com Wed Jun 3 11:11:18 2015 From: lois.foltan at oracle.com (Lois Foltan) Date: Wed, 03 Jun 2015 07:11:18 -0400 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556E0AD4.3010207@oracle.com> References: <556E0AD4.3010207@oracle.com> Message-ID: <556EE0D6.6000105@oracle.com> Looks good! Lois On 6/2/2015 3:58 PM, Coleen Phillimore wrote: > Summary: Use the Events::log function to save redefined classes for > output to the hs_err file. > > This change adds lines to the hs_err_pid file which names classes > redefined: > > Classes redefined (2 events): > Event: 11.867 Thread 0x00007fd378199800 redefined class > name=RedefineRunningMethods$B, count=1 > Event: 26.746 Thread 0x00007fd378199800 redefined class > name=RedefineRunningMethods$B, count=2 > > Tested with intentional crash to see hs_err_pid files, also tested > without intentional crashes vm.redefine.testlist. > > open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ > bug link https://bugs.openjdk.java.net/browse/JDK-8081219 > > Thanks, > Coleen > > From coleen.phillimore at oracle.com Wed Jun 3 14:03:55 2015 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 03 Jun 2015 10:03:55 -0400 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556E2BBD.2000405@oracle.com> References: <556E0AD4.3010207@oracle.com> <556E2BBD.2000405@oracle.com> Message-ID: <556F094B.3030008@oracle.com> Thank you Serguei for the comment. I thought of putting the event log first but really wanted it to tell which classes were already redefined. So I decided to improve the error message for printing the VM_Operation to include the class being redefined. With the following change, there's a crash during redefinition, the hs_err_pid file will have: VM_Operation (0x00007f49a294a400): RedefineClasses, mode: safepoint, requested by thread 0x00007f499 800e800, redefining class RedefineRunningMethods$B open webrev at http://cr.openjdk.java.net/~coleenp/8081219.02/ Tested manually with a ShouldNotReachHere in various places. Thanks, Coleen On 6/2/15 6:18 PM, serguei.spitsyn at oracle.com wrote: > Coleen, > > The fix looks good. > But I have a question. > Would it be useful to log the beginning of a redefinition as well so > that it is known what class is currently being redefined? > > > Thanks, > Serguei > > On 6/2/15 12:58 PM, Coleen Phillimore wrote: >> Summary: Use the Events::log function to save redefined classes for >> output to the hs_err file. >> >> This change adds lines to the hs_err_pid file which names classes >> redefined: >> >> Classes redefined (2 events): >> Event: 11.867 Thread 0x00007fd378199800 redefined class >> name=RedefineRunningMethods$B, count=1 >> Event: 26.746 Thread 0x00007fd378199800 redefined class >> name=RedefineRunningMethods$B, count=2 >> >> Tested with intentional crash to see hs_err_pid files, also tested >> without intentional crashes vm.redefine.testlist. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ >> bug link https://bugs.openjdk.java.net/browse/JDK-8081219 >> >> Thanks, >> Coleen >> >> > From dmitry.samersoff at oracle.com Wed Jun 3 15:29:48 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 03 Jun 2015 18:29:48 +0300 Subject: RFR(M, v13): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556EBF4B.6070400@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> Message-ID: <556F1D6C.6010806@oracle.com> Everyone, Updated webrev: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 Changes to oop.inline.hpp is reverted and find_field used directly is diagnostic command. -Dmitry On 2015-06-03 11:48, Dmitry Samersoff wrote: > Everyone, > > Updated webrev: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.12 > > getFinalizerHistogram and FinalizerHistogramEntry moved to separate > package-private class. Hotspot code changed accordingly. > > getFinalizerHistogram updated to use Peter's code. > > -Dmitry > > > On 2015-06-03 09:06, Peter Levart wrote: >> Hi Dmitry, >> >> On 06/02/2015 01:12 PM, Dmitry Samersoff wrote: >>> Staffan, >>> >>>> Instead of hardcoding the field offsets, you can use >>>> InstanceKlass::find_field and fieldDescriptor::offset to find the >>>> offset at runtime. >>> Done. Please, see >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.11 >> >> In the jdk part, now that you have a FinalizerHistogramEntry class, you >> can simplify the code even further: >> >> >> private static final class FinalizerHistogramEntry { >> int instanceCount; >> final String className; >> >> int getInstanceCount() { >> return instanceCount; >> } >> >> FinalizerHistogramEntry(String className) { >> this.className = className; >> } >> } >> >> static Object[] getFinalizerHistogram() { >> Map countMap = new HashMap<>(); >> queue.forEach(r -> { >> Object referent = r.get(); >> if (referent != null) { >> countMap.computeIfAbsent( >> referent.getClass().getName(), >> FinalizerHistogramEntry::new).instanceCount++; >> /* Clear stack slot containing this variable, to decrease >> the chances of false retention with a conservative GC */ >> referent = null; >> } >> }); >> >> FinalizerHistogramEntry fhe[] = countMap.values() >> .toArray(new FinalizerHistogramEntry[countMap.size()]); >> Arrays.sort(fhe, >> Comparator.comparingInt( >> >> FinalizerHistogramEntry::getInstanceCount).reversed()); >> return fhe; >> } >> >> >> ... see, no copying loop required. Also notice the access modifier used >> on the nested class and it's fields/method/constructor. This way the >> class is private and fields can be accessed from getFinalizerHistogram >> and lambda without compiler generating access bridges. >> >> >> Regards, Peter >> >>> >>> I put the function int get_filed_offset_by_name(Symbol*,Symbol*) to >>> oop.inline.hpp leaving a room for further cleanup because I found couple >>> of places in hotspot that implements mostly similar functionality. >>> >>> -Dmitry >>> >>> On 2015-06-01 10:18, Staffan Larsen wrote: >>>> Dmitry, >>>> >>>> Instead of hardcoding the field offsets, you can use InstanceKlass::find_field and fieldDescriptor::offset to find the offset at runtime. >>>> >>>> Thanks, >>>> /Staffan >>>> >>>>> On 31 maj 2015, at 13:43, Dmitry Samersoff wrote: >>>>> >>>>> Everyone, >>>>> >>>>> Please take a look at new version of the fix. >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.10/ >>>>> >>>>> Changes (to previous version) are in >>>>> Finalizer.java and diagnosticCommand.cpp >>>>> >>>>> This version copy data from Map.Entry<> to array of >>>>> FinalizerHistogramEntry instances then, >>>>> VM prints content of this array. >>>>> >>>>> -Dmitry >>>>> >>>>> >>>>> On 2015-05-28 21:06, Mandy Chung wrote: >>>>>> On 05/28/2015 07:35 AM, Peter Levart wrote: >>>>>>> Hi Mandy, >>>>>>> >>>>>>> On 05/27/2015 03:32 PM, Mandy Chung wrote: >>>>>>>> Taking it further - is it simpler to return String[] of all >>>>>>>> classnames including the duplicated ones and have the VM do the >>>>>>>> count? Are you concerned with the size of the String[]? >>>>>>> Yes, the histogram is much smaller than the list of all instances. >>>>>>> There can be millions of instances waiting in finalizer queue, but >>>>>>> only a few distinct classes. >>>>>> Do you happen to know what libraries are the major contributors to these >>>>>> millions of finalizers? >>>>>> >>>>>> It has been strongly recommended to avoid finalizers (see Effective Java >>>>>> Item 7). I'm not surprised that existing code is still using >>>>>> finalizers while we should really encourage them to migrate away from it. >>>>>> >>>>>>> What could be done in Java to simplify things in native code but still >>>>>>> not format the output is to convert the array of Map.Entry(s) into an >>>>>>> Object[] array of alternating {String, int[], String, int[], .... } >>>>>>> >>>>>>> Would this simplify native code for the price of a little extra work >>>>>>> in Java? The sizes of old and new arrays are not big (# of distinct >>>>>>> classes of finalizable objects in the queue). >>>>>> I also prefer writing in Java and writing less native code (much >>>>>> simplified). It's an interface that we have to agree upon and keep it >>>>>> simple. Having the returned Object[] as alternate String and int[] is a >>>>>> reasonable compromise. >>>>>> >>>>>> ReferenceQueue.java - you can move @SuppressWarning from the method to >>>>>> just the field variable "rn" >>>>>> @SuppressWarnings("unchecked") >>>>>> Reference rn = r.next; >>>>>> >>>>>> Finalizer.java >>>>>> It's better to rename printFinalizationQueue as it inspects the >>>>>> finalizer reference queue (maybe getFinalizerHistogram?). Can you move >>>>>> this method to the end of this class and add the comment saying that >>>>>> this is invoked by VM for jcmd -finalizerinfo support and @return to >>>>>> describe the returned content. I also think you can remove >>>>>> @SuppressWarnings for this method. >>>>>> >>>>>> Mandy >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. >>> >> > > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From serguei.spitsyn at oracle.com Wed Jun 3 18:16:24 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 03 Jun 2015 11:16:24 -0700 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556F094B.3030008@oracle.com> References: <556E0AD4.3010207@oracle.com> <556E2BBD.2000405@oracle.com> <556F094B.3030008@oracle.com> Message-ID: <556F4478.3000905@oracle.com> Coleen, The fix is nice, thank you for the update. The whole feature is going to be very useful for us! Thanks, Serguei On 6/3/15 7:03 AM, Coleen Phillimore wrote: > > Thank you Serguei for the comment. I thought of putting the event log > first but really wanted it to tell which classes were already > redefined. So I decided to improve the error message for printing the > VM_Operation to include the class being redefined. > > With the following change, there's a crash during redefinition, the > hs_err_pid file will have: > > VM_Operation (0x00007f49a294a400): RedefineClasses, mode: safepoint, > requested by thread 0x00007f499 > 800e800, redefining class RedefineRunningMethods$B > > open webrev at http://cr.openjdk.java.net/~coleenp/8081219.02/ > > Tested manually with a ShouldNotReachHere in various places. > > Thanks, > Coleen > > On 6/2/15 6:18 PM, serguei.spitsyn at oracle.com wrote: >> Coleen, >> >> The fix looks good. >> But I have a question. >> Would it be useful to log the beginning of a redefinition as well so >> that it is known what class is currently being redefined? >> >> >> Thanks, >> Serguei >> >> On 6/2/15 12:58 PM, Coleen Phillimore wrote: >>> Summary: Use the Events::log function to save redefined classes for >>> output to the hs_err file. >>> >>> This change adds lines to the hs_err_pid file which names classes >>> redefined: >>> >>> Classes redefined (2 events): >>> Event: 11.867 Thread 0x00007fd378199800 redefined class >>> name=RedefineRunningMethods$B, count=1 >>> Event: 26.746 Thread 0x00007fd378199800 redefined class >>> name=RedefineRunningMethods$B, count=2 >>> >>> Tested with intentional crash to see hs_err_pid files, also tested >>> without intentional crashes vm.redefine.testlist. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ >>> bug link https://bugs.openjdk.java.net/browse/JDK-8081219 >>> >>> Thanks, >>> Coleen >>> >>> >> > From mandy.chung at oracle.com Wed Jun 3 18:36:14 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 03 Jun 2015 11:36:14 -0700 Subject: RFR(M, v13): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556F1D6C.6010806@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> Message-ID: <556F491E.3070709@oracle.com> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: > Updated webrev: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 I reviewed the jdk change. The version looks okay and some comments: ReferenceQueue.java - you should eliminate the use of rawtype: 84 Reference rn = r.next; Change Reference to Reference The forEach method - eliminate the use of raw type and move @SuppressWarning to the field variable in line 182: @SuppressWarnings("unchecked") Reference rn = r.next; FinalizerHistogram.java Copyright year needs update. I personally think the VM code would be much simpler if you stay with alternative entry of String and int than dealing with a FinalizerHistogramEntry private class. It's okay as FinalizerHistogram class is separated. The comment in line 35 is suitable to move to the class description and this is the suggestion. // This FinalizerHistogram class is for GC.finalizer_info diagnostic command support. // It is invoked by the VM. Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM knows the returned type anyway. It's an inner class of FinalizerHistogram and it can simply be named as "Entry". I suggest to have Entry::increment method to replace ".instanceCount++". The tests are in hotspot/test. Can you add a unit test in jdk/test? Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via reflection - just a sanity test. A minor naming comment - now you have a FinalizerHistogram class. Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? Mandy From peter.levart at gmail.com Wed Jun 3 19:22:22 2015 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 03 Jun 2015 21:22:22 +0200 Subject: RFR(M, v13): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556F491E.3070709@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> Message-ID: <556F53EE.3050207@gmail.com> On 06/03/2015 08:36 PM, Mandy Chung wrote: > > > On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >> Updated webrev: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 > > I reviewed the jdk change. The version looks okay and some comments: > > ReferenceQueue.java - you should eliminate the use of rawtype: > > 84 Reference rn = r.next; > > Change Reference to Reference > > The forEach method - eliminate the use of raw type and > move @SuppressWarning to the field variable in line 182: > > @SuppressWarnings("unchecked") > Reference rn = r.next; Thanks, Mandy. This was my doing. The @SuppressWarnings can be moved to the local var declaration in line 84 too. Here's how the fixed ReferenceQueue looks like after those two changes: http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.03/ > > FinalizerHistogram.java > Copyright year needs update. > > I personally think the VM code would be much simpler if you stay with > alternative entry of String and int than dealing with a > FinalizerHistogramEntry private class. It's okay as > FinalizerHistogram class is separated. > > The comment in line 35 is suitable to move to the class description > and this is the suggestion. > > // This FinalizerHistogram class is for GC.finalizer_info > diagnostic command support. > // It is invoked by the VM. > > Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The > VM knows the returned type anyway. It's an inner class of > FinalizerHistogram and it can simply be named as "Entry". I suggest > to have Entry::increment method to replace ".instanceCount++". This FinalizerHistogram.Entry naming part came to my mind too, yes. If there is a method for incrementing, then both fields can be marked private. Regards, Peter > > The tests are in hotspot/test. Can you add a unit test in > jdk/test? Perhaps you can test > FinalizerHistogram.getFinalizerHistogram() via reflection - just a > sanity test. > > A minor naming comment - now you have a FinalizerHistogram class. > Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? > > Mandy -------------- next part -------------- An HTML attachment was scrubbed... URL: From coleen.phillimore at oracle.com Wed Jun 3 20:33:35 2015 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 03 Jun 2015 16:33:35 -0400 Subject: RFR 8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file In-Reply-To: <556F4478.3000905@oracle.com> References: <556E0AD4.3010207@oracle.com> <556E2BBD.2000405@oracle.com> <556F094B.3030008@oracle.com> <556F4478.3000905@oracle.com> Message-ID: <556F649F.4020506@oracle.com> Thank you, Serguei! Coleen On 6/3/15 2:16 PM, serguei.spitsyn at oracle.com wrote: > Coleen, > > > The fix is nice, thank you for the update. > The whole feature is going to be very useful for us! > > Thanks, > Serguei > > On 6/3/15 7:03 AM, Coleen Phillimore wrote: >> >> Thank you Serguei for the comment. I thought of putting the event >> log first but really wanted it to tell which classes were already >> redefined. So I decided to improve the error message for printing >> the VM_Operation to include the class being redefined. >> >> With the following change, there's a crash during redefinition, the >> hs_err_pid file will have: >> >> VM_Operation (0x00007f49a294a400): RedefineClasses, mode: safepoint, >> requested by thread 0x00007f499 >> 800e800, redefining class RedefineRunningMethods$B >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8081219.02/ >> >> Tested manually with a ShouldNotReachHere in various places. >> >> Thanks, >> Coleen >> >> On 6/2/15 6:18 PM, serguei.spitsyn at oracle.com wrote: >>> Coleen, >>> >>> The fix looks good. >>> But I have a question. >>> Would it be useful to log the beginning of a redefinition as well so >>> that it is known what class is currently being redefined? >>> >>> >>> Thanks, >>> Serguei >>> >>> On 6/2/15 12:58 PM, Coleen Phillimore wrote: >>>> Summary: Use the Events::log function to save redefined classes for >>>> output to the hs_err file. >>>> >>>> This change adds lines to the hs_err_pid file which names classes >>>> redefined: >>>> >>>> Classes redefined (2 events): >>>> Event: 11.867 Thread 0x00007fd378199800 redefined class >>>> name=RedefineRunningMethods$B, count=1 >>>> Event: 26.746 Thread 0x00007fd378199800 redefined class >>>> name=RedefineRunningMethods$B, count=2 >>>> >>>> Tested with intentional crash to see hs_err_pid files, also tested >>>> without intentional crashes vm.redefine.testlist. >>>> >>>> open webrev at http://cr.openjdk.java.net/~coleenp/8081219/ >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8081219 >>>> >>>> Thanks, >>>> Coleen >>>> >>>> >>> >> > From mandy.chung at oracle.com Wed Jun 3 22:56:34 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 3 Jun 2015 15:56:34 -0700 Subject: RFR(M, v13): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556F53EE.3050207@gmail.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <556F53EE.3050207! @gmail.com> Message-ID: <523E93C8-B707-4CF6-A26E-B5400E04C402@oracle.com> > On Jun 3, 2015, at 12:22 PM, Peter Levart wrote: > > > On 06/03/2015 08:36 PM, Mandy Chung wrote: >> >> >> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>> Updated webrev: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >> >> I reviewed the jdk change. The version looks okay and some comments: >> >> ReferenceQueue.java - you should eliminate the use of rawtype: >> >> 84 Reference rn = r.next; >> >> Change Reference to Reference >> >> The forEach method - eliminate the use of raw type and >> move @SuppressWarning to the field variable in line 182: >> >> @SuppressWarnings("unchecked") >> Reference rn = r.next; > > Thanks, Mandy. This was my doing. The @SuppressWarnings can be moved to the local var declaration in line 84 too. Here's how the fixed ReferenceQueue looks like after those two changes: > > http://cr.openjdk.java.net/~plevart/misc/Finalizer.printFinalizationQueue/webrev.03/ This looks good. > >> >> FinalizerHistogram.java >> Copyright year needs update. >> >> I personally think the VM code would be much simpler if you stay with alternative entry of String and int than dealing with a FinalizerHistogramEntry private class. It's okay as FinalizerHistogram class is separated. >> >> The comment in line 35 is suitable to move to the class description and this is the suggestion. >> >> // This FinalizerHistogram class is for GC.finalizer_info diagnostic command support. >> // It is invoked by the VM. >> >> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM knows the returned type anyway. It's an inner class of FinalizerHistogram and it can simply be named as "Entry". I suggest to have Entry::increment method to replace ".instanceCount++". > > This FinalizerHistogram.Entry naming part came to my mind too, yes. If there is a method for incrementing, then both fields can be marked private. Yup. Mandy > > Regards, Peter > >> >> The tests are in hotspot/test. Can you add a unit test in jdk/test? Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via reflection - just a sanity test. >> >> A minor naming comment - now you have a FinalizerHistogram class. Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >> >> Mandy > From chris.plummer at oracle.com Wed Jun 3 23:23:08 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 03 Jun 2015 16:23:08 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556EBD91.6030204@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556DFC33.4060600@oracle.com> <556E73AF.3020606@oracle.com> <556EBD91.6030204@oracle.com> Message-ID: <556F8C5C.6070704@oracle.com> Hi Serguei, I'll take care of the rename. Can I also put you down for the ProcessTool.java changes, which are officially being reviewed on another thread: http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-June/033892.html thanks, Chris On 6/3/15 1:40 AM, serguei.spitsyn at oracle.com wrote: > Chris, > > It looks good in general. > I'd suggest to rename the folder: > || test/com/sun/jdi/CDSJDITests > to: > test/com/sun/jdi/cds > > There is no need to spell "JDI" as it is already a sub-folder of the > com/sun/jdi > and there is no need to spell "Tests" too as it is in the test repo. > Also, all the folders are normally named in the lower case. > > Thanks, > Serguei > > > On 6/2/15 8:25 PM, Chris Plummer wrote: >> Ok, I'm going to keep this as one webrev, but I did create >> JDK-8081771 for the ProcessTool.java changes: >> >> https://bugs.openjdk.java.net/browse/JDK-8081771 >> >> I will commit ProcessTool.java under JDK-8081771, and the rest of the >> changes under JDK-8054386. Both will then be pushed together. I also >> started a new thread for the review of JDK-8081771: >> >> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2015-June/014930.html >> http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-June/033892.html >> >> thanks, >> >> Chris >> >> On 6/2/15 11:55 AM, Chris Plummer wrote: >>> I'm going to have to separate out the ProcessTool.java changes into >>> a separate changeset (and CR). In the meantime, feel free to review >>> what I have below. The code won't be changing at all when I separate >>> out the ProcessTool.java changes. >>> >>> thanks, >>> >>> Chris >>> >>> On 6/2/15 12:36 AM, Chris Plummer wrote: >>>> [Adding core-libs-dev at openjdk.java.net since this update includes >>>> changes to jdk/test library code] >>>> >>>> Please review the updated webrev: >>>> >>>> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >>>> >>>> There were concerns about the new hotspot tests referencing jdk >>>> tests. One concern was that if the jdk tests change, they could >>>> break the hotspot tests, and this might initially go undetected. >>>> The other concern is that if the jdk and hotspot tests are placed >>>> in separate test bundles, then it would not be possible to run the >>>> hotspot tests. >>>> >>>> Because of these concerns, I moved the tests that were in >>>> hotspot/test/runtime/CDSJDITests to instead be in >>>> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of >>>> the tests in the process. Also, I had to update the jdk version of >>>> ProcessTool.java to include the createJavaProcessBuilder() variant >>>> that is in the hotspot version of ProcessTool.java. >>>> >>>> Lastly, in CDSJITTest.java I changed: >>>> >>>> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >>>> >>>> to instead be: >>>> >>>> OutputAnalyzer output = ProcessTools.executeProcess(pb); >>>> >>>> I had to do this since the jdk version of the OutputAnalyzer >>>> constructor is not public. The 1st version is what is commonly used >>>> in hostspot tests, and the 2nd version is what is commonly used in >>>> jdk tests. I decided to adopt the jdk way rather than make the >>>> OutputAnalyzer constructors public, although this will probably >>>> happen eventually when the two versions are unified. >>>> >>>> thanks, >>>> >>>> Chris >>>> >>>> >>>> On 5/19/15 7:25 AM, Chris Plummer wrote: >>>>> Hi, >>>>> >>>>> Please review the following changes for allowing java debugging >>>>> when CDS is enabled. >>>>> >>>>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>>>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>>>> >>>>> The VM changes are simple. I removed the check that prevents >>>>> debugging with CDS enabled, and added logic that will map the CDS >>>>> archive RW when debugging is enabled. >>>>> >>>>> The tests are a bit more complex. There are a bunch of existing >>>>> JDI tests for testing debugging support. Rather than start from >>>>> scratch or clone them, I instead just wrote wrapper tests that put >>>>> the relevant JDI test classes in the archive, and then invoke the >>>>> JDI test. I did this for 3 of the JDI tests. If you feel there are >>>>> others that would be good candidates, I'd be happy to add them. >>>>> I'm looking for ones that would result in modification of the RO >>>>> class metadata, such as setting a breakpoint (for which I already >>>>> added two tests). >>>>> >>>>> Testing done: >>>>> -Using JPRT to run the new jtreg tests on all platforms. >>>>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>>>> -Regular JPRT "-testset hotspot" run >>>>> -Putting the JCK JVMTI tests in the archive and then running them. >>>>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive >>>>> and then running them. >>>>> -Putting a simple test class in the archive and then setting a >>>>> breakpoint on it using jdb >>>>> >>>>> Some of the above testing resulted in the discovery of bugs that >>>>> still need to be addressed: JDK-8078644, JDK-8078730, and >>>>> JDK-8079181. >>>>> >>>>> I also verified that without the change to map the archive RW, the >>>>> above testing resulted in a SEGV, which is what you would expect >>>>> (and actually want to see to prove that the testing is effective). >>>>> >>>>> thanks, >>>>> >>>>> Chris >>>>> >>>> >>> >> > From serguei.spitsyn at oracle.com Thu Jun 4 00:02:54 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 03 Jun 2015 17:02:54 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556F8C5C.6070704@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556DFC33.4060600@oracle.com> <556E73AF.3020606@oracle.com> <556EBD91.6030204@oracle.com> <556F8C5C.6070704@oracle.com> Message-ID: <556F95AE.7020708@oracle.com> Hi Chris, I've replied with a thumbs up on another thread. Thanks, Serguei On 6/3/15 4:23 PM, Chris Plummer wrote: > Hi Serguei, > > I'll take care of the rename. Can I also put you down for the > ProcessTool.java changes, which are officially being reviewed on > another thread: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-June/033892.html > > > thanks, > > Chris > > On 6/3/15 1:40 AM, serguei.spitsyn at oracle.com wrote: >> Chris, >> >> It looks good in general. >> I'd suggest to rename the folder: >> || test/com/sun/jdi/CDSJDITests >> to: >> test/com/sun/jdi/cds >> >> There is no need to spell "JDI" as it is already a sub-folder of the >> com/sun/jdi >> and there is no need to spell "Tests" too as it is in the test repo. >> Also, all the folders are normally named in the lower case. >> >> Thanks, >> Serguei >> >> >> On 6/2/15 8:25 PM, Chris Plummer wrote: >>> Ok, I'm going to keep this as one webrev, but I did create >>> JDK-8081771 for the ProcessTool.java changes: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8081771 >>> >>> I will commit ProcessTool.java under JDK-8081771, and the rest of >>> the changes under JDK-8054386. Both will then be pushed together. I >>> also started a new thread for the review of JDK-8081771: >>> >>> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2015-June/014930.html >>> >>> http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-June/033892.html >>> >>> >>> thanks, >>> >>> Chris >>> >>> On 6/2/15 11:55 AM, Chris Plummer wrote: >>>> I'm going to have to separate out the ProcessTool.java changes into >>>> a separate changeset (and CR). In the meantime, feel free to review >>>> what I have below. The code won't be changing at all when I >>>> separate out the ProcessTool.java changes. >>>> >>>> thanks, >>>> >>>> Chris >>>> >>>> On 6/2/15 12:36 AM, Chris Plummer wrote: >>>>> [Adding core-libs-dev at openjdk.java.net since this update includes >>>>> changes to jdk/test library code] >>>>> >>>>> Please review the updated webrev: >>>>> >>>>> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >>>>> >>>>> There were concerns about the new hotspot tests referencing jdk >>>>> tests. One concern was that if the jdk tests change, they could >>>>> break the hotspot tests, and this might initially go undetected. >>>>> The other concern is that if the jdk and hotspot tests are placed >>>>> in separate test bundles, then it would not be possible to run the >>>>> hotspot tests. >>>>> >>>>> Because of these concerns, I moved the tests that were in >>>>> hotspot/test/runtime/CDSJDITests to instead be in >>>>> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of >>>>> the tests in the process. Also, I had to update the jdk version of >>>>> ProcessTool.java to include the createJavaProcessBuilder() variant >>>>> that is in the hotspot version of ProcessTool.java. >>>>> >>>>> Lastly, in CDSJITTest.java I changed: >>>>> >>>>> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >>>>> >>>>> to instead be: >>>>> >>>>> OutputAnalyzer output = ProcessTools.executeProcess(pb); >>>>> >>>>> I had to do this since the jdk version of the OutputAnalyzer >>>>> constructor is not public. The 1st version is what is commonly >>>>> used in hostspot tests, and the 2nd version is what is commonly >>>>> used in jdk tests. I decided to adopt the jdk way rather than make >>>>> the OutputAnalyzer constructors public, although this will >>>>> probably happen eventually when the two versions are unified. >>>>> >>>>> thanks, >>>>> >>>>> Chris >>>>> >>>>> >>>>> On 5/19/15 7:25 AM, Chris Plummer wrote: >>>>>> Hi, >>>>>> >>>>>> Please review the following changes for allowing java debugging >>>>>> when CDS is enabled. >>>>>> >>>>>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>>>>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>>>>> >>>>>> The VM changes are simple. I removed the check that prevents >>>>>> debugging with CDS enabled, and added logic that will map the CDS >>>>>> archive RW when debugging is enabled. >>>>>> >>>>>> The tests are a bit more complex. There are a bunch of existing >>>>>> JDI tests for testing debugging support. Rather than start from >>>>>> scratch or clone them, I instead just wrote wrapper tests that >>>>>> put the relevant JDI test classes in the archive, and then invoke >>>>>> the JDI test. I did this for 3 of the JDI tests. If you feel >>>>>> there are others that would be good candidates, I'd be happy to >>>>>> add them. I'm looking for ones that would result in modification >>>>>> of the RO class metadata, such as setting a breakpoint (for which >>>>>> I already added two tests). >>>>>> >>>>>> Testing done: >>>>>> -Using JPRT to run the new jtreg tests on all platforms. >>>>>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>>>>> -Regular JPRT "-testset hotspot" run >>>>>> -Putting the JCK JVMTI tests in the archive and then running them. >>>>>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive >>>>>> and then running them. >>>>>> -Putting a simple test class in the archive and then setting a >>>>>> breakpoint on it using jdb >>>>>> >>>>>> Some of the above testing resulted in the discovery of bugs that >>>>>> still need to be addressed: JDK-8078644, JDK-8078730, and >>>>>> JDK-8079181. >>>>>> >>>>>> I also verified that without the change to map the archive RW, >>>>>> the above testing resulted in a SEGV, which is what you would >>>>>> expect (and actually want to see to prove that the testing is >>>>>> effective). >>>>>> >>>>>> thanks, >>>>>> >>>>>> Chris >>>>>> >>>>> >>>> >>> >> > From david.holmes at oracle.com Thu Jun 4 06:29:51 2015 From: david.holmes at oracle.com (David Holmes) Date: Thu, 04 Jun 2015 16:29:51 +1000 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556D5CFD.1020709@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> Message-ID: <556FF05F.6060300@oracle.com> Hi Chris, Hotspot change is good. Only a couple of style nits with the tests (where are our Java style guidelines ???). Taking CDSJDITest.java as an example: If you are okay with this line length: 115 public static OutputAnalyzer executeAndLog(ProcessBuilder pb, String logName) throws Exception { then you can remove a number of line breaks in the headers of other functions. (I don't follow the 70-80 char line length dogma ;) ) If you do break a header of a function the { still stays on the same line as the last header component ie: private static void addToClassList(PrintStream ps, String classes[]) throws IOException { not: 139 private static void addToClassList(PrintStream ps, String classes[]) 140 throws IOException 141 { Cheers, David On 2/06/2015 5:36 PM, Chris Plummer wrote: > [Adding core-libs-dev at openjdk.java.net since this update includes > changes to jdk/test library code] > > Please review the updated webrev: > > Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 > > There were concerns about the new hotspot tests referencing jdk tests. > One concern was that if the jdk tests change, they could break the > hotspot tests, and this might initially go undetected. The other concern > is that if the jdk and hotspot tests are placed in separate test > bundles, then it would not be possible to run the hotspot tests. > > Because of these concerns, I moved the tests that were in > hotspot/test/runtime/CDSJDITests to instead be in > jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the > tests in the process. Also, I had to update the jdk version of > ProcessTool.java to include the createJavaProcessBuilder() variant that > is in the hotspot version of ProcessTool.java. > > Lastly, in CDSJITTest.java I changed: > > OutputAnalyzer output = new OutputAnalyzer(pb.start()); > > to instead be: > > OutputAnalyzer output = ProcessTools.executeProcess(pb); > > I had to do this since the jdk version of the OutputAnalyzer constructor > is not public. The 1st version is what is commonly used in hostspot > tests, and the 2nd version is what is commonly used in jdk tests. I > decided to adopt the jdk way rather than make the OutputAnalyzer > constructors public, although this will probably happen eventually when > the two versions are unified. > > thanks, > > Chris > > > On 5/19/15 7:25 AM, Chris Plummer wrote: >> Hi, >> >> Please review the following changes for allowing java debugging when >> CDS is enabled. >> >> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >> >> The VM changes are simple. I removed the check that prevents debugging >> with CDS enabled, and added logic that will map the CDS archive RW >> when debugging is enabled. >> >> The tests are a bit more complex. There are a bunch of existing JDI >> tests for testing debugging support. Rather than start from scratch or >> clone them, I instead just wrote wrapper tests that put the relevant >> JDI test classes in the archive, and then invoke the JDI test. I did >> this for 3 of the JDI tests. If you feel there are others that would >> be good candidates, I'd be happy to add them. I'm looking for ones >> that would result in modification of the RO class metadata, such as >> setting a breakpoint (for which I already added two tests). >> >> Testing done: >> -Using JPRT to run the new jtreg tests on all platforms. >> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >> -Regular JPRT "-testset hotspot" run >> -Putting the JCK JVMTI tests in the archive and then running them. >> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >> then running them. >> -Putting a simple test class in the archive and then setting a >> breakpoint on it using jdb >> >> Some of the above testing resulted in the discovery of bugs that still >> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >> >> I also verified that without the change to map the archive RW, the >> above testing resulted in a SEGV, which is what you would expect (and >> actually want to see to prove that the testing is effective). >> >> thanks, >> >> Chris >> > From dmitry.samersoff at oracle.com Thu Jun 4 13:38:15 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 04 Jun 2015 16:38:15 +0300 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <556F491E.3070709@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> Message-ID: <557054C7.4030705@oracle.com> Mandy, Thank you for the review. Updated webrev is: http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 addressed comments, added a unit test to JDK workspace. On 2015-06-03 21:36, Mandy Chung wrote: > Should getFinalizerHistogram() return FinalizerHistogramEntry[]? > The VM knows the returned type anyway. "[java/lang/ref/Entry" signature would need a separate symbol entry in swollen vmSymbols::init() so I would prefer to stay with more generic Object[] > Perhaps the getFinalizerHistogram method should be renamed > e.g. "dump"? The line in vmSymbols looks like: template( get_finalizer_histogram_name, "getFinalizerHistogram") I would prefer to keep method name specific enough to be able to find it by grep in jdk code. (other comments are addressed) -Dmitry On 2015-06-03 21:36, Mandy Chung wrote: > > > On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >> Updated webrev: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 > > I reviewed the jdk change. The version looks okay and some comments: > > ReferenceQueue.java - you should eliminate the use of rawtype: > > 84 Reference rn = r.next; > > Change Reference to Reference done. > > The forEach method - eliminate the use of raw type and > move @SuppressWarning to the field variable in line 182: > > @SuppressWarnings("unchecked") > Reference rn = r.next; done. > > FinalizerHistogram.java > Copyright year needs update. done. > > I personally think the VM code would be much simpler if you stay with > alternative entry of String and int than dealing with a > FinalizerHistogramEntry private class. It's okay as FinalizerHistogram > class is separated. > > The comment in line 35 is suitable to move to the class description and > this is the suggestion. > > // This FinalizerHistogram class is for GC.finalizer_info diagnostic > command support. > // It is invoked by the VM. done. > > Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM > knows the returned type anyway. "[java/lang/ref/Entry" signature would need a separate symbol entry in swollen vmSymbols::init() so I would prefer to stay with more generic Object[] > It's an inner class of > FinalizerHistogram and it can simply be named as "Entry". I suggest to > have Entry::increment method to replace ".instanceCount++". done. > > The tests are in hotspot/test. Can you add a unit test in jdk/test? > Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via > reflection - just a sanity test. done. The test repeats hotspot one, but uses reflection instead of VM call. > > A minor naming comment - now you have a FinalizerHistogram class. > Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? The line in vmSymbols looks like: template( get_finalizer_histogram_name, "getFinalizerHistogram") I would prefer to keep it specific enough to be able to find it by grep in jdk code. -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From mandy.chung at oracle.com Thu Jun 4 13:56:03 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 4 Jun 2015 06:56:03 -0700 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <557054C7.4030705@oracle.com> References: <554894E9.8020908@oracle.com> <1AF219C7-1055-4D97-BB46-2FA390C4135D@oracle.com> <5548D65C.8040304@gmail.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705! @oracle.com> Message-ID: <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> > On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: > > Mandy, > > Thank you for the review. > > Updated webrev is: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 > Thanks for the update and the test. > addressed comments, added a unit test to JDK workspace. > This test does not need @library and @build, right? > > On 2015-06-03 21:36, Mandy Chung wrote: > >> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >> The VM knows the returned type anyway. > > "[java/lang/ref/Entry" signature would need a separate symbol entry in > swollen vmSymbols::init() so I would prefer to stay with more generic > Object[] > You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >> Perhaps the getFinalizerHistogram method should be renamed >> e.g. "dump"? > > The line in vmSymbols looks like: > > template( > get_finalizer_histogram_name, "getFinalizerHistogram") > > I would prefer to keep method name specific enough to be able to > find it by grep in jdk code. Grep in jdk code is easy as you have a new class :) Mandy > > (other comments are addressed) > > -Dmitry > > > On 2015-06-03 21:36, Mandy Chung wrote: >> >> >> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>> Updated webrev: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >> >> I reviewed the jdk change. The version looks okay and some comments: >> >> ReferenceQueue.java - you should eliminate the use of rawtype: >> >> 84 Reference rn = r.next; >> >> Change Reference to Reference > > done. > >> >> The forEach method - eliminate the use of raw type and >> move @SuppressWarning to the field variable in line 182: >> >> @SuppressWarnings("unchecked") >> Reference rn = r.next; > > done. > >> >> FinalizerHistogram.java >> Copyright year needs update. > > done. >> >> I personally think the VM code would be much simpler if you stay with >> alternative entry of String and int than dealing with a >> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >> class is separated. >> >> The comment in line 35 is suitable to move to the class description and >> this is the suggestion. >> >> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >> command support. >> // It is invoked by the VM. > > done. > >> >> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >> knows the returned type anyway. > > "[java/lang/ref/Entry" signature would need a separate symbol entry in > swollen vmSymbols::init() so I would prefer to stay with more generic > Object[] > >> It's an inner class of >> FinalizerHistogram and it can simply be named as "Entry". I suggest to >> have Entry::increment method to replace ".instanceCount++". > > done. > >> >> The tests are in hotspot/test. Can you add a unit test in jdk/test? >> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >> reflection - just a sanity test. > > done. The test repeats hotspot one, but uses reflection instead of VM call. > >> >> A minor naming comment - now you have a FinalizerHistogram class. >> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? > > The line in vmSymbols looks like: > > template( > get_finalizer_histogram_name, "getFinalizerHistogram") > > I would prefer to keep it specific enough to be able to > find it by grep in jdk code. > > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Thu Jun 4 15:49:31 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Thu, 04 Jun 2015 18:49:31 +0300 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> References: <554894E9.8020908@oracle.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> Message-ID: <5570738B.5040603@oracle.com> Mandy, Webrev updated in-place (press shift-reload). http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 getFinalizerHistogram() now returns Entry[] @library and @build removed from the test. -Dmitry On 2015-06-04 16:56, Mandy Chung wrote: > >> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >> >> Mandy, >> >> Thank you for the review. >> >> Updated webrev is: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >> > > Thanks for the update and the test. > >> addressed comments, added a unit test to JDK workspace. >> > > This test does not need @library and @build, right? > >> >> On 2015-06-03 21:36, Mandy Chung wrote: >> >>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>> The VM knows the returned type anyway. >> >> "[java/lang/ref/Entry" signature would need a separate symbol entry in >> swollen vmSymbols::init() so I would prefer to stay with more generic >> Object[] >> > > You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). > > >>> Perhaps the getFinalizerHistogram method should be renamed >>> e.g. "dump"? >> >> The line in vmSymbols looks like: >> >> template( >> get_finalizer_histogram_name, "getFinalizerHistogram") >> >> I would prefer to keep method name specific enough to be able to >> find it by grep in jdk code. > > > Grep in jdk code is easy as you have a new class :) > > Mandy > >> >> (other comments are addressed) >> >> -Dmitry >> >> >> On 2015-06-03 21:36, Mandy Chung wrote: >>> >>> >>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>> Updated webrev: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>> >>> I reviewed the jdk change. The version looks okay and some comments: >>> >>> ReferenceQueue.java - you should eliminate the use of rawtype: >>> >>> 84 Reference rn = r.next; >>> >>> Change Reference to Reference >> >> done. >> >>> >>> The forEach method - eliminate the use of raw type and >>> move @SuppressWarning to the field variable in line 182: >>> >>> @SuppressWarnings("unchecked") >>> Reference rn = r.next; >> >> done. >> >>> >>> FinalizerHistogram.java >>> Copyright year needs update. >> >> done. >>> >>> I personally think the VM code would be much simpler if you stay with >>> alternative entry of String and int than dealing with a >>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>> class is separated. >>> >>> The comment in line 35 is suitable to move to the class description and >>> this is the suggestion. >>> >>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>> command support. >>> // It is invoked by the VM. >> >> done. >> >>> >>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>> knows the returned type anyway. >> >> "[java/lang/ref/Entry" signature would need a separate symbol entry in >> swollen vmSymbols::init() so I would prefer to stay with more generic >> Object[] >> >>> It's an inner class of >>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>> have Entry::increment method to replace ".instanceCount++". >> >> done. >> >>> >>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>> reflection - just a sanity test. >> >> done. The test repeats hotspot one, but uses reflection instead of VM call. >> >>> >>> A minor naming comment - now you have a FinalizerHistogram class. >>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >> >> The line in vmSymbols looks like: >> >> template( >> get_finalizer_histogram_name, "getFinalizerHistogram") >> >> I would prefer to keep it specific enough to be able to >> find it by grep in jdk code. >> >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From serguei.spitsyn at oracle.com Thu Jun 4 21:23:19 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 04 Jun 2015 14:23:19 -0700 Subject: 8076110: VM crash when class is redefined with Instrumentation.redefineClasses In-Reply-To: <55703CEF.50403@oracle.com> References: <55703CEF.50403@oracle.com> Message-ID: <5570C1C7.60507@oracle.com> Added the SVC list as it impacts the Serviceability as well. Thanks, Serguei On 6/4/15 4:56 AM, Andreas Eriksson wrote: > Hi, > > Please review this fix of a crash because of a class redefinition race. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8076110 > Webrev: http://cr.openjdk.java.net/~aeriksso/8076110/webrev.00/ > > The crash happens when the redefine occurs while the redefined method > is still running. > This is because VM_RedefineClasses::redefine_single_class deletes the > resolution errors at redefine time, but the running method might still > need them. > > This is fixed by moving the deletion of the cached resolution errors > to when the {add,purge}_previous_version functions have come to the > conclusion that no running methods need the old class any more (and > therefore no one needs the constant pool and its associated resolution > errors either). > > A jtreg regression test that reproduces the problem using > redefineclasses and objectweb.asm is included in the change. > > Thanks, > Andreas From mandy.chung at oracle.com Thu Jun 4 22:20:54 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 4 Jun 2015 15:20:54 -0700 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5570738B.5040603@oracle.com> References: <554894E9.8020908@oracle.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> <5570738B.504060! 3@oracle.com> Message-ID: <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> > On Jun 4, 2015, at 8:49 AM, Dmitry Samersoff wrote: > > Mandy, > > Webrev updated in-place (press shift-reload). > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 > > getFinalizerHistogram() now returns Entry[] > > @library and @build removed from the test. Looks fine. Thanks Mandy > > -Dmitry > > On 2015-06-04 16:56, Mandy Chung wrote: >> >>> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >>> >>> Mandy, >>> >>> Thank you for the review. >>> >>> Updated webrev is: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>> >> >> Thanks for the update and the test. >> >>> addressed comments, added a unit test to JDK workspace. >>> >> >> This test does not need @library and @build, right? >> >>> >>> On 2015-06-03 21:36, Mandy Chung wrote: >>> >>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>>> The VM knows the returned type anyway. >>> >>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>> swollen vmSymbols::init() so I would prefer to stay with more generic >>> Object[] >>> >> >> You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >> >> >>>> Perhaps the getFinalizerHistogram method should be renamed >>>> e.g. "dump"? >>> >>> The line in vmSymbols looks like: >>> >>> template( >>> get_finalizer_histogram_name, "getFinalizerHistogram") >>> >>> I would prefer to keep method name specific enough to be able to >>> find it by grep in jdk code. >> >> >> Grep in jdk code is easy as you have a new class :) >> >> Mandy >> >>> >>> (other comments are addressed) >>> >>> -Dmitry >>> >>> >>> On 2015-06-03 21:36, Mandy Chung wrote: >>>> >>>> >>>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>>> Updated webrev: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>>> >>>> I reviewed the jdk change. The version looks okay and some comments: >>>> >>>> ReferenceQueue.java - you should eliminate the use of rawtype: >>>> >>>> 84 Reference rn = r.next; >>>> >>>> Change Reference to Reference >>> >>> done. >>> >>>> >>>> The forEach method - eliminate the use of raw type and >>>> move @SuppressWarning to the field variable in line 182: >>>> >>>> @SuppressWarnings("unchecked") >>>> Reference rn = r.next; >>> >>> done. >>> >>>> >>>> FinalizerHistogram.java >>>> Copyright year needs update. >>> >>> done. >>>> >>>> I personally think the VM code would be much simpler if you stay with >>>> alternative entry of String and int than dealing with a >>>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>>> class is separated. >>>> >>>> The comment in line 35 is suitable to move to the class description and >>>> this is the suggestion. >>>> >>>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>>> command support. >>>> // It is invoked by the VM. >>> >>> done. >>> >>>> >>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>>> knows the returned type anyway. >>> >>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>> swollen vmSymbols::init() so I would prefer to stay with more generic >>> Object[] >>> >>>> It's an inner class of >>>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>>> have Entry::increment method to replace ".instanceCount++". >>> >>> done. >>> >>>> >>>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>>> reflection - just a sanity test. >>> >>> done. The test repeats hotspot one, but uses reflection instead of VM call. >>> >>>> >>>> A minor naming comment - now you have a FinalizerHistogram class. >>>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >>> >>> The line in vmSymbols looks like: >>> >>> template( >>> get_finalizer_histogram_name, "getFinalizerHistogram") >>> >>> I would prefer to keep it specific enough to be able to >>> find it by grep in jdk code. >>> >>> >>> -- >>> Dmitry Samersoff >>> Oracle Java development team, Saint Petersburg, Russia >>> * I would love to change the world, but they won't give me the sources. >> > > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From chris.plummer at oracle.com Thu Jun 4 23:32:26 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 04 Jun 2015 16:32:26 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <556FF05F.6060300@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556FF05F.6060300@oracle.com> Message-ID: <5570E00A.8070400@oracle.com> Hi David, Here's an updated webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.04/ thanks, Chris On 6/3/15 11:29 PM, David Holmes wrote: > Hi Chris, > > Hotspot change is good. > > Only a couple of style nits with the tests (where are our Java style > guidelines ???). Taking CDSJDITest.java as an example: > > If you are okay with this line length: > > 115 public static OutputAnalyzer executeAndLog(ProcessBuilder pb, > String logName) throws Exception { > > then you can remove a number of line breaks in the headers of other > functions. (I don't follow the 70-80 char line length dogma ;) ) > > If you do break a header of a function the { still stays on the same > line as the last header component ie: > > private static void addToClassList(PrintStream ps, String classes[]) > throws IOException { > > not: > > 139 private static void addToClassList(PrintStream ps, String > classes[]) > 140 throws IOException > 141 { > > Cheers, > David > > On 2/06/2015 5:36 PM, Chris Plummer wrote: >> [Adding core-libs-dev at openjdk.java.net since this update includes >> changes to jdk/test library code] >> >> Please review the updated webrev: >> >> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >> >> There were concerns about the new hotspot tests referencing jdk tests. >> One concern was that if the jdk tests change, they could break the >> hotspot tests, and this might initially go undetected. The other concern >> is that if the jdk and hotspot tests are placed in separate test >> bundles, then it would not be possible to run the hotspot tests. >> >> Because of these concerns, I moved the tests that were in >> hotspot/test/runtime/CDSJDITests to instead be in >> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the >> tests in the process. Also, I had to update the jdk version of >> ProcessTool.java to include the createJavaProcessBuilder() variant that >> is in the hotspot version of ProcessTool.java. >> >> Lastly, in CDSJITTest.java I changed: >> >> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >> >> to instead be: >> >> OutputAnalyzer output = ProcessTools.executeProcess(pb); >> >> I had to do this since the jdk version of the OutputAnalyzer constructor >> is not public. The 1st version is what is commonly used in hostspot >> tests, and the 2nd version is what is commonly used in jdk tests. I >> decided to adopt the jdk way rather than make the OutputAnalyzer >> constructors public, although this will probably happen eventually when >> the two versions are unified. >> >> thanks, >> >> Chris >> >> >> On 5/19/15 7:25 AM, Chris Plummer wrote: >>> Hi, >>> >>> Please review the following changes for allowing java debugging when >>> CDS is enabled. >>> >>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> The VM changes are simple. I removed the check that prevents debugging >>> with CDS enabled, and added logic that will map the CDS archive RW >>> when debugging is enabled. >>> >>> The tests are a bit more complex. There are a bunch of existing JDI >>> tests for testing debugging support. Rather than start from scratch or >>> clone them, I instead just wrote wrapper tests that put the relevant >>> JDI test classes in the archive, and then invoke the JDI test. I did >>> this for 3 of the JDI tests. If you feel there are others that would >>> be good candidates, I'd be happy to add them. I'm looking for ones >>> that would result in modification of the RO class metadata, such as >>> setting a breakpoint (for which I already added two tests). >>> >>> Testing done: >>> -Using JPRT to run the new jtreg tests on all platforms. >>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>> -Regular JPRT "-testset hotspot" run >>> -Putting the JCK JVMTI tests in the archive and then running them. >>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >>> then running them. >>> -Putting a simple test class in the archive and then setting a >>> breakpoint on it using jdb >>> >>> Some of the above testing resulted in the discovery of bugs that still >>> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>> >>> I also verified that without the change to map the archive RW, the >>> above testing resulted in a SEGV, which is what you would expect (and >>> actually want to see to prove that the testing is effective). >>> >>> thanks, >>> >>> Chris >>> >> From serguei.spitsyn at oracle.com Fri Jun 5 00:40:41 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 04 Jun 2015 17:40:41 -0700 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <5570E00A.8070400@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556FF05F.6060300@oracle.com> <5570E00A.8070400@oracle.com> Message-ID: <5570F009.7040003@oracle.com> I guess, there is no need to re-review. It looks good anyway. Thanks, Serguei On 6/4/15 4:32 PM, Chris Plummer wrote: > Hi David, > > Here's an updated webrev: > > http://cr.openjdk.java.net/~cjplummer/8054386/webrev.04/ > > thanks, > > Chris > > On 6/3/15 11:29 PM, David Holmes wrote: >> Hi Chris, >> >> Hotspot change is good. >> >> Only a couple of style nits with the tests (where are our Java style >> guidelines ???). Taking CDSJDITest.java as an example: >> >> If you are okay with this line length: >> >> 115 public static OutputAnalyzer executeAndLog(ProcessBuilder >> pb, String logName) throws Exception { >> >> then you can remove a number of line breaks in the headers of other >> functions. (I don't follow the 70-80 char line length dogma ;) ) >> >> If you do break a header of a function the { still stays on the same >> line as the last header component ie: >> >> private static void addToClassList(PrintStream ps, String >> classes[]) >> throws IOException { >> >> not: >> >> 139 private static void addToClassList(PrintStream ps, String >> classes[]) >> 140 throws IOException >> 141 { >> >> Cheers, >> David >> >> On 2/06/2015 5:36 PM, Chris Plummer wrote: >>> [Adding core-libs-dev at openjdk.java.net since this update includes >>> changes to jdk/test library code] >>> >>> Please review the updated webrev: >>> >>> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> There were concerns about the new hotspot tests referencing jdk tests. >>> One concern was that if the jdk tests change, they could break the >>> hotspot tests, and this might initially go undetected. The other >>> concern >>> is that if the jdk and hotspot tests are placed in separate test >>> bundles, then it would not be possible to run the hotspot tests. >>> >>> Because of these concerns, I moved the tests that were in >>> hotspot/test/runtime/CDSJDITests to instead be in >>> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the >>> tests in the process. Also, I had to update the jdk version of >>> ProcessTool.java to include the createJavaProcessBuilder() variant that >>> is in the hotspot version of ProcessTool.java. >>> >>> Lastly, in CDSJITTest.java I changed: >>> >>> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >>> >>> to instead be: >>> >>> OutputAnalyzer output = ProcessTools.executeProcess(pb); >>> >>> I had to do this since the jdk version of the OutputAnalyzer >>> constructor >>> is not public. The 1st version is what is commonly used in hostspot >>> tests, and the 2nd version is what is commonly used in jdk tests. I >>> decided to adopt the jdk way rather than make the OutputAnalyzer >>> constructors public, although this will probably happen eventually when >>> the two versions are unified. >>> >>> thanks, >>> >>> Chris >>> >>> >>> On 5/19/15 7:25 AM, Chris Plummer wrote: >>>> Hi, >>>> >>>> Please review the following changes for allowing java debugging when >>>> CDS is enabled. >>>> >>>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>>> >>>> The VM changes are simple. I removed the check that prevents debugging >>>> with CDS enabled, and added logic that will map the CDS archive RW >>>> when debugging is enabled. >>>> >>>> The tests are a bit more complex. There are a bunch of existing JDI >>>> tests for testing debugging support. Rather than start from scratch or >>>> clone them, I instead just wrote wrapper tests that put the relevant >>>> JDI test classes in the archive, and then invoke the JDI test. I did >>>> this for 3 of the JDI tests. If you feel there are others that would >>>> be good candidates, I'd be happy to add them. I'm looking for ones >>>> that would result in modification of the RO class metadata, such as >>>> setting a breakpoint (for which I already added two tests). >>>> >>>> Testing done: >>>> -Using JPRT to run the new jtreg tests on all platforms. >>>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>>> -Regular JPRT "-testset hotspot" run >>>> -Putting the JCK JVMTI tests in the archive and then running them. >>>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >>>> then running them. >>>> -Putting a simple test class in the archive and then setting a >>>> breakpoint on it using jdb >>>> >>>> Some of the above testing resulted in the discovery of bugs that still >>>> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>>> >>>> I also verified that without the change to map the archive RW, the >>>> above testing resulted in a SEGV, which is what you would expect (and >>>> actually want to see to prove that the testing is effective). >>>> >>>> thanks, >>>> >>>> Chris >>>> >>> > From david.holmes at oracle.com Fri Jun 5 05:50:13 2015 From: david.holmes at oracle.com (David Holmes) Date: Fri, 05 Jun 2015 15:50:13 +1000 Subject: [9] RFR (M) 8054386: Allow Java debugging when CDS is enabled In-Reply-To: <5570E00A.8070400@oracle.com> References: <555B47EA.3000900@oracle.com> <556D5CFD.1020709@oracle.com> <556FF05F.6060300@oracle.com> <5570E00A.8070400@oracle.com> Message-ID: <55713895.4040601@oracle.com> On 5/06/2015 9:32 AM, Chris Plummer wrote: > Hi David, > > Here's an updated webrev: > > http://cr.openjdk.java.net/~cjplummer/8054386/webrev.04/ Thanks - should have said updated webrev not necessary :) Looks good. David > thanks, > > Chris > > On 6/3/15 11:29 PM, David Holmes wrote: >> Hi Chris, >> >> Hotspot change is good. >> >> Only a couple of style nits with the tests (where are our Java style >> guidelines ???). Taking CDSJDITest.java as an example: >> >> If you are okay with this line length: >> >> 115 public static OutputAnalyzer executeAndLog(ProcessBuilder pb, >> String logName) throws Exception { >> >> then you can remove a number of line breaks in the headers of other >> functions. (I don't follow the 70-80 char line length dogma ;) ) >> >> If you do break a header of a function the { still stays on the same >> line as the last header component ie: >> >> private static void addToClassList(PrintStream ps, String classes[]) >> throws IOException { >> >> not: >> >> 139 private static void addToClassList(PrintStream ps, String >> classes[]) >> 140 throws IOException >> 141 { >> >> Cheers, >> David >> >> On 2/06/2015 5:36 PM, Chris Plummer wrote: >>> [Adding core-libs-dev at openjdk.java.net since this update includes >>> changes to jdk/test library code] >>> >>> Please review the updated webrev: >>> >>> Webrev: http://cr.openjdk.java.net/~cjplummer/8054386/webrev.02/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8054386 >>> >>> There were concerns about the new hotspot tests referencing jdk tests. >>> One concern was that if the jdk tests change, they could break the >>> hotspot tests, and this might initially go undetected. The other concern >>> is that if the jdk and hotspot tests are placed in separate test >>> bundles, then it would not be possible to run the hotspot tests. >>> >>> Because of these concerns, I moved the tests that were in >>> hotspot/test/runtime/CDSJDITests to instead be in >>> jdk/test/com/sun/jdi/CDSJDITests. There was a slight renaming of the >>> tests in the process. Also, I had to update the jdk version of >>> ProcessTool.java to include the createJavaProcessBuilder() variant that >>> is in the hotspot version of ProcessTool.java. >>> >>> Lastly, in CDSJITTest.java I changed: >>> >>> OutputAnalyzer output = new OutputAnalyzer(pb.start()); >>> >>> to instead be: >>> >>> OutputAnalyzer output = ProcessTools.executeProcess(pb); >>> >>> I had to do this since the jdk version of the OutputAnalyzer constructor >>> is not public. The 1st version is what is commonly used in hostspot >>> tests, and the 2nd version is what is commonly used in jdk tests. I >>> decided to adopt the jdk way rather than make the OutputAnalyzer >>> constructors public, although this will probably happen eventually when >>> the two versions are unified. >>> >>> thanks, >>> >>> Chris >>> >>> >>> On 5/19/15 7:25 AM, Chris Plummer wrote: >>>> Hi, >>>> >>>> Please review the following changes for allowing java debugging when >>>> CDS is enabled. >>>> >>>> Webrev:http://cr.openjdk.java.net/~cjplummer/8054386/webrev.01/ >>>> Bug:https://bugs.openjdk.java.net/browse/JDK-8054386 >>>> >>>> The VM changes are simple. I removed the check that prevents debugging >>>> with CDS enabled, and added logic that will map the CDS archive RW >>>> when debugging is enabled. >>>> >>>> The tests are a bit more complex. There are a bunch of existing JDI >>>> tests for testing debugging support. Rather than start from scratch or >>>> clone them, I instead just wrote wrapper tests that put the relevant >>>> JDI test classes in the archive, and then invoke the JDI test. I did >>>> this for 3 of the JDI tests. If you feel there are others that would >>>> be good candidates, I'd be happy to add them. I'm looking for ones >>>> that would result in modification of the RO class metadata, such as >>>> setting a breakpoint (for which I already added two tests). >>>> >>>> Testing done: >>>> -Using JPRT to run the new jtreg tests on all platforms. >>>> -Using JPRT to run all jtreg runtime tests on linux x86 and x_64. >>>> -Regular JPRT "-testset hotspot" run >>>> -Putting the JCK JVMTI tests in the archive and then running them. >>>> -Putting the nsk jdb, jdwp, jvmti, and jdi tests in the archive and >>>> then running them. >>>> -Putting a simple test class in the archive and then setting a >>>> breakpoint on it using jdb >>>> >>>> Some of the above testing resulted in the discovery of bugs that still >>>> need to be addressed: JDK-8078644, JDK-8078730, and JDK-8079181. >>>> >>>> I also verified that without the change to map the archive RW, the >>>> above testing resulted in a SEGV, which is what you would expect (and >>>> actually want to see to prove that the testing is effective). >>>> >>>> thanks, >>>> >>>> Chris >>>> >>> > From staffan.larsen at oracle.com Fri Jun 5 08:20:13 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 5 Jun 2015 10:20:13 +0200 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> References: <554894E9.8020908@oracle.com> <554A8CEE.2090009@oracle.com> <55523418.5010708@oracle.com> <55527935.5010408@gmail.com> <5553B552.9060900@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> <5570738B.504060! 3@oracle.com> <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> Message-ID: Dmitry, I?d like to propose the following change to get prettier output (more in line with GC.class_histogram): diff --git a/src/share/vm/services/diagnosticCommand.cpp b/src/share/vm/services/diagnosticCommand.cpp --- a/src/share/vm/services/diagnosticCommand.cpp +++ b/src/share/vm/services/diagnosticCommand.cpp @@ -341,7 +341,6 @@ void FinalizerInfoDCmd::execute(DCmdSource source, TRAPS) { ResourceMark rm; - output()->print_cr("Unreachable instances awaiting finalization:"); Klass* k = SystemDictionary::resolve_or_null( vmSymbols::finalizer_histogram_klass(), THREAD); assert(k != NULL, "FinalizerHistogram class is not accessible"); @@ -375,12 +374,15 @@ assert(count_res != NULL && name_res != NULL, "Unexpected layout of FinalizerHistogramEntry"); + output()->print_cr("Unreachable instances waiting for finalization"); + output()->print_cr("#instances class name"); + output()->print_cr("-----------------------"); for (int i = 0; i < result_oop->length(); ++i) { oop element_oop = result_oop->obj_at(i); oop str_oop = element_oop->obj_field(name_fd.offset()); char *name = java_lang_String::as_utf8_string(str_oop); int count = element_oop->int_field(count_fd.offset()); - output()->print_cr("Class %s - %d", name, count); + output()->print_cr("%10d %s", count, name); } } A couple of nits: diagnosticCommand.cpp:359 - extra space after = diagnosticCommand.cpp:361 - spelling: finlalization -> finalization FinalizerInfoTest.java:69: - spelling: intialized -> initialized FinalizerInfoTest.java:71 - I?d like to see the ; on a new line Thanks, /Staffan > On 5 jun 2015, at 00:20, Mandy Chung wrote: > > >> On Jun 4, 2015, at 8:49 AM, Dmitry Samersoff wrote: >> >> Mandy, >> >> Webrev updated in-place (press shift-reload). >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >> >> getFinalizerHistogram() now returns Entry[] >> >> @library and @build removed from the test. > > > Looks fine. > > Thanks > Mandy > >> >> -Dmitry >> >> On 2015-06-04 16:56, Mandy Chung wrote: >>> >>>> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >>>> >>>> Mandy, >>>> >>>> Thank you for the review. >>>> >>>> Updated webrev is: >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>> >>> >>> Thanks for the update and the test. >>> >>>> addressed comments, added a unit test to JDK workspace. >>>> >>> >>> This test does not need @library and @build, right? >>> >>>> >>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>> >>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>>>> The VM knows the returned type anyway. >>>> >>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>> Object[] >>>> >>> >>> You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >>> >>> >>>>> Perhaps the getFinalizerHistogram method should be renamed >>>>> e.g. "dump"? >>>> >>>> The line in vmSymbols looks like: >>>> >>>> template( >>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>> >>>> I would prefer to keep method name specific enough to be able to >>>> find it by grep in jdk code. >>> >>> >>> Grep in jdk code is easy as you have a new class :) >>> >>> Mandy >>> >>>> >>>> (other comments are addressed) >>>> >>>> -Dmitry >>>> >>>> >>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>> >>>>> >>>>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>>>> Updated webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>>>> >>>>> I reviewed the jdk change. The version looks okay and some comments: >>>>> >>>>> ReferenceQueue.java - you should eliminate the use of rawtype: >>>>> >>>>> 84 Reference rn = r.next; >>>>> >>>>> Change Reference to Reference >>>> >>>> done. >>>> >>>>> >>>>> The forEach method - eliminate the use of raw type and >>>>> move @SuppressWarning to the field variable in line 182: >>>>> >>>>> @SuppressWarnings("unchecked") >>>>> Reference rn = r.next; >>>> >>>> done. >>>> >>>>> >>>>> FinalizerHistogram.java >>>>> Copyright year needs update. >>>> >>>> done. >>>>> >>>>> I personally think the VM code would be much simpler if you stay with >>>>> alternative entry of String and int than dealing with a >>>>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>>>> class is separated. >>>>> >>>>> The comment in line 35 is suitable to move to the class description and >>>>> this is the suggestion. >>>>> >>>>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>>>> command support. >>>>> // It is invoked by the VM. >>>> >>>> done. >>>> >>>>> >>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>>>> knows the returned type anyway. >>>> >>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>> Object[] >>>> >>>>> It's an inner class of >>>>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>>>> have Entry::increment method to replace ".instanceCount++". >>>> >>>> done. >>>> >>>>> >>>>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>>>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>>>> reflection - just a sanity test. >>>> >>>> done. The test repeats hotspot one, but uses reflection instead of VM call. >>>> >>>>> >>>>> A minor naming comment - now you have a FinalizerHistogram class. >>>>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >>>> >>>> The line in vmSymbols looks like: >>>> >>>> template( >>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>> >>>> I would prefer to keep it specific enough to be able to >>>> find it by grep in jdk code. >>>> >>>> >>>> -- >>>> Dmitry Samersoff >>>> Oracle Java development team, Saint Petersburg, Russia >>>> * I would love to change the world, but they won't give me the sources. >>> >> >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. > From dmitry.samersoff at oracle.com Fri Jun 5 11:00:27 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Fri, 05 Jun 2015 14:00:27 +0300 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: References: <554894E9.8020908@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> <5570738B.5040603@oracle.com> <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> Message-ID: <5571814B.9060706@oracle.com> Staffan, Thank you for review! Done. Webrev updated in-place (press shift-reload). http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 -Dmitry On 2015-06-05 11:20, Staffan Larsen wrote: > Dmitry, > > I?d like to propose the following change to get prettier output (more in line with GC.class_histogram): > > diff --git a/src/share/vm/services/diagnosticCommand.cpp b/src/share/vm/services/diagnosticCommand.cpp > --- a/src/share/vm/services/diagnosticCommand.cpp > +++ b/src/share/vm/services/diagnosticCommand.cpp > @@ -341,7 +341,6 @@ > void FinalizerInfoDCmd::execute(DCmdSource source, TRAPS) { > ResourceMark rm; > > - output()->print_cr("Unreachable instances awaiting finalization:"); > Klass* k = SystemDictionary::resolve_or_null( > vmSymbols::finalizer_histogram_klass(), THREAD); > assert(k != NULL, "FinalizerHistogram class is not accessible"); > @@ -375,12 +374,15 @@ > > assert(count_res != NULL && name_res != NULL, "Unexpected layout of FinalizerHistogramEntry"); > > + output()->print_cr("Unreachable instances waiting for finalization"); > + output()->print_cr("#instances class name"); > + output()->print_cr("-----------------------"); > for (int i = 0; i < result_oop->length(); ++i) { > oop element_oop = result_oop->obj_at(i); > oop str_oop = element_oop->obj_field(name_fd.offset()); > char *name = java_lang_String::as_utf8_string(str_oop); > int count = element_oop->int_field(count_fd.offset()); > - output()->print_cr("Class %s - %d", name, count); > + output()->print_cr("%10d %s", count, name); > } > } > > > A couple of nits: > > diagnosticCommand.cpp:359 - extra space after = > diagnosticCommand.cpp:361 - spelling: finlalization -> finalization > FinalizerInfoTest.java:69: - spelling: intialized -> initialized > FinalizerInfoTest.java:71 - I?d like to see the ; on a new line > > > Thanks, > /Staffan > > >> On 5 jun 2015, at 00:20, Mandy Chung wrote: >> >> >>> On Jun 4, 2015, at 8:49 AM, Dmitry Samersoff wrote: >>> >>> Mandy, >>> >>> Webrev updated in-place (press shift-reload). >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>> >>> getFinalizerHistogram() now returns Entry[] >>> >>> @library and @build removed from the test. >> >> >> Looks fine. >> >> Thanks >> Mandy >> >>> >>> -Dmitry >>> >>> On 2015-06-04 16:56, Mandy Chung wrote: >>>> >>>>> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >>>>> >>>>> Mandy, >>>>> >>>>> Thank you for the review. >>>>> >>>>> Updated webrev is: >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>>> >>>> >>>> Thanks for the update and the test. >>>> >>>>> addressed comments, added a unit test to JDK workspace. >>>>> >>>> >>>> This test does not need @library and @build, right? >>>> >>>>> >>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>> >>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>>>>> The VM knows the returned type anyway. >>>>> >>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>> Object[] >>>>> >>>> >>>> You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >>>> >>>> >>>>>> Perhaps the getFinalizerHistogram method should be renamed >>>>>> e.g. "dump"? >>>>> >>>>> The line in vmSymbols looks like: >>>>> >>>>> template( >>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>> >>>>> I would prefer to keep method name specific enough to be able to >>>>> find it by grep in jdk code. >>>> >>>> >>>> Grep in jdk code is easy as you have a new class :) >>>> >>>> Mandy >>>> >>>>> >>>>> (other comments are addressed) >>>>> >>>>> -Dmitry >>>>> >>>>> >>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>>> >>>>>> >>>>>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>>>>> Updated webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>>>>> >>>>>> I reviewed the jdk change. The version looks okay and some comments: >>>>>> >>>>>> ReferenceQueue.java - you should eliminate the use of rawtype: >>>>>> >>>>>> 84 Reference rn = r.next; >>>>>> >>>>>> Change Reference to Reference >>>>> >>>>> done. >>>>> >>>>>> >>>>>> The forEach method - eliminate the use of raw type and >>>>>> move @SuppressWarning to the field variable in line 182: >>>>>> >>>>>> @SuppressWarnings("unchecked") >>>>>> Reference rn = r.next; >>>>> >>>>> done. >>>>> >>>>>> >>>>>> FinalizerHistogram.java >>>>>> Copyright year needs update. >>>>> >>>>> done. >>>>>> >>>>>> I personally think the VM code would be much simpler if you stay with >>>>>> alternative entry of String and int than dealing with a >>>>>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>>>>> class is separated. >>>>>> >>>>>> The comment in line 35 is suitable to move to the class description and >>>>>> this is the suggestion. >>>>>> >>>>>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>>>>> command support. >>>>>> // It is invoked by the VM. >>>>> >>>>> done. >>>>> >>>>>> >>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>>>>> knows the returned type anyway. >>>>> >>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>> Object[] >>>>> >>>>>> It's an inner class of >>>>>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>>>>> have Entry::increment method to replace ".instanceCount++". >>>>> >>>>> done. >>>>> >>>>>> >>>>>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>>>>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>>>>> reflection - just a sanity test. >>>>> >>>>> done. The test repeats hotspot one, but uses reflection instead of VM call. >>>>> >>>>>> >>>>>> A minor naming comment - now you have a FinalizerHistogram class. >>>>>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >>>>> >>>>> The line in vmSymbols looks like: >>>>> >>>>> template( >>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>> >>>>> I would prefer to keep it specific enough to be able to >>>>> find it by grep in jdk code. >>>>> >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. >>>> >>> >>> >>> -- >>> Dmitry Samersoff >>> Oracle Java development team, Saint Petersburg, Russia >>> * I would love to change the world, but they won't give me the sources. >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From staffan.larsen at oracle.com Fri Jun 5 12:24:33 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Fri, 5 Jun 2015 14:24:33 +0200 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <5571814B.9060706@oracle.com> References: <554894E9.8020908@oracle.com> <555453BA.2070205@gmail.com> <55567748.6020304@oracle.com> <55567A68.9020802@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> <5570738B.5040603@oracle.com> <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> <5571! 814B.9060706@oracle.com> Message-ID: <0323E5E0-CFFA-4E11-8F51-270CB7F6964C@oracle.com> Thanks - this version looks good to me. /Staffan > On 5 jun 2015, at 13:00, Dmitry Samersoff wrote: > > Staffan, > > Thank you for review! > > Done. Webrev updated in-place (press shift-reload). > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 > > -Dmitry > > On 2015-06-05 11:20, Staffan Larsen wrote: >> Dmitry, >> >> I?d like to propose the following change to get prettier output (more in line with GC.class_histogram): >> >> diff --git a/src/share/vm/services/diagnosticCommand.cpp b/src/share/vm/services/diagnosticCommand.cpp >> --- a/src/share/vm/services/diagnosticCommand.cpp >> +++ b/src/share/vm/services/diagnosticCommand.cpp >> @@ -341,7 +341,6 @@ >> void FinalizerInfoDCmd::execute(DCmdSource source, TRAPS) { >> ResourceMark rm; >> >> - output()->print_cr("Unreachable instances awaiting finalization:"); >> Klass* k = SystemDictionary::resolve_or_null( >> vmSymbols::finalizer_histogram_klass(), THREAD); >> assert(k != NULL, "FinalizerHistogram class is not accessible"); >> @@ -375,12 +374,15 @@ >> >> assert(count_res != NULL && name_res != NULL, "Unexpected layout of FinalizerHistogramEntry"); >> >> + output()->print_cr("Unreachable instances waiting for finalization"); >> + output()->print_cr("#instances class name"); >> + output()->print_cr("-----------------------"); >> for (int i = 0; i < result_oop->length(); ++i) { >> oop element_oop = result_oop->obj_at(i); >> oop str_oop = element_oop->obj_field(name_fd.offset()); >> char *name = java_lang_String::as_utf8_string(str_oop); >> int count = element_oop->int_field(count_fd.offset()); >> - output()->print_cr("Class %s - %d", name, count); >> + output()->print_cr("%10d %s", count, name); >> } >> } >> >> >> A couple of nits: >> >> diagnosticCommand.cpp:359 - extra space after = >> diagnosticCommand.cpp:361 - spelling: finlalization -> finalization >> FinalizerInfoTest.java:69: - spelling: intialized -> initialized >> FinalizerInfoTest.java:71 - I?d like to see the ; on a new line >> >> >> Thanks, >> /Staffan >> >> >>> On 5 jun 2015, at 00:20, Mandy Chung wrote: >>> >>> >>>> On Jun 4, 2015, at 8:49 AM, Dmitry Samersoff wrote: >>>> >>>> Mandy, >>>> >>>> Webrev updated in-place (press shift-reload). >>>> >>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>> >>>> getFinalizerHistogram() now returns Entry[] >>>> >>>> @library and @build removed from the test. >>> >>> >>> Looks fine. >>> >>> Thanks >>> Mandy >>> >>>> >>>> -Dmitry >>>> >>>> On 2015-06-04 16:56, Mandy Chung wrote: >>>>> >>>>>> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >>>>>> >>>>>> Mandy, >>>>>> >>>>>> Thank you for the review. >>>>>> >>>>>> Updated webrev is: >>>>>> >>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>>>> >>>>> >>>>> Thanks for the update and the test. >>>>> >>>>>> addressed comments, added a unit test to JDK workspace. >>>>>> >>>>> >>>>> This test does not need @library and @build, right? >>>>> >>>>>> >>>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>>> >>>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>>>>>> The VM knows the returned type anyway. >>>>>> >>>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>>> Object[] >>>>>> >>>>> >>>>> You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >>>>> >>>>> >>>>>>> Perhaps the getFinalizerHistogram method should be renamed >>>>>>> e.g. "dump"? >>>>>> >>>>>> The line in vmSymbols looks like: >>>>>> >>>>>> template( >>>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>>> >>>>>> I would prefer to keep method name specific enough to be able to >>>>>> find it by grep in jdk code. >>>>> >>>>> >>>>> Grep in jdk code is easy as you have a new class :) >>>>> >>>>> Mandy >>>>> >>>>>> >>>>>> (other comments are addressed) >>>>>> >>>>>> -Dmitry >>>>>> >>>>>> >>>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>>>> >>>>>>> >>>>>>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>>>>>> Updated webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>>>>>> >>>>>>> I reviewed the jdk change. The version looks okay and some comments: >>>>>>> >>>>>>> ReferenceQueue.java - you should eliminate the use of rawtype: >>>>>>> >>>>>>> 84 Reference rn = r.next; >>>>>>> >>>>>>> Change Reference to Reference >>>>>> >>>>>> done. >>>>>> >>>>>>> >>>>>>> The forEach method - eliminate the use of raw type and >>>>>>> move @SuppressWarning to the field variable in line 182: >>>>>>> >>>>>>> @SuppressWarnings("unchecked") >>>>>>> Reference rn = r.next; >>>>>> >>>>>> done. >>>>>> >>>>>>> >>>>>>> FinalizerHistogram.java >>>>>>> Copyright year needs update. >>>>>> >>>>>> done. >>>>>>> >>>>>>> I personally think the VM code would be much simpler if you stay with >>>>>>> alternative entry of String and int than dealing with a >>>>>>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>>>>>> class is separated. >>>>>>> >>>>>>> The comment in line 35 is suitable to move to the class description and >>>>>>> this is the suggestion. >>>>>>> >>>>>>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>>>>>> command support. >>>>>>> // It is invoked by the VM. >>>>>> >>>>>> done. >>>>>> >>>>>>> >>>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>>>>>> knows the returned type anyway. >>>>>> >>>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>>> Object[] >>>>>> >>>>>>> It's an inner class of >>>>>>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>>>>>> have Entry::increment method to replace ".instanceCount++". >>>>>> >>>>>> done. >>>>>> >>>>>>> >>>>>>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>>>>>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>>>>>> reflection - just a sanity test. >>>>>> >>>>>> done. The test repeats hotspot one, but uses reflection instead of VM call. >>>>>> >>>>>>> >>>>>>> A minor naming comment - now you have a FinalizerHistogram class. >>>>>>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >>>>>> >>>>>> The line in vmSymbols looks like: >>>>>> >>>>>> template( >>>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>>> >>>>>> I would prefer to keep it specific enough to be able to >>>>>> find it by grep in jdk code. >>>>>> >>>>>> >>>>>> -- >>>>>> Dmitry Samersoff >>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>> * I would love to change the world, but they won't give me the sources. >>>>> >>>> >>>> >>>> -- >>>> Dmitry Samersoff >>>> Oracle Java development team, Saint Petersburg, Russia >>>> * I would love to change the world, but they won't give me the sources. >>> >> > > > -- > Dmitry Samersoff > Oracle Java development team, Saint Petersburg, Russia > * I would love to change the world, but they won't give me the sources. From ecki at zusammenkunft.net Sat Jun 6 17:39:50 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sat, 6 Jun 2015 19:39:50 +0200 Subject: Wrapping sun.misc.Unsafe's native methods Message-ID: <20150606193950.00006e9c.ecki@zusammenkunft.net> Hello, I had some application problems and thought it would be a good idea to wrap Unsafe.allocateMemory(long) to get a histogram of call-sites and to trace the memory consumptions. When I register an Java agent then sun.misc.Unsafe will not be loaded through the transformer, since it is a rather early system component. So I tried to redefine the class, which actually works. But I cannot get the native methods wrapped. allocateMemory() does not not delegate to a allocateMemory0() native method, so in order to wrap it I need to use the setNativeMethodPrefix() and use retransformClass(). Hower this does not work as I have to actually add the wrapped prefix_allocateMemoty() method - which is not possible for redefinitions. It looks like the netbeans/jvisualvm profiler has the same problem, it just cannot see invocations of those methods. Is there a way around this planned or possible? (I am not so good in JNI, maybe the native method could be registered to a totally new class?) For Java 9, when the Unsafe becomes more supported APIs, it would be good if native methods could be easier accessible (if this does not conflict with intrinsics?) Gruss Bernd PS: on gc-dev I just asked if it would be possible to extend BufferPoolMXBean with some more stats (allocation total cound, failed allocations, alignment flag, max size): http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013733.html From dmitry.samersoff at oracle.com Mon Jun 8 11:05:27 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Mon, 08 Jun 2015 14:05:27 +0300 Subject: RFR(M, v14): JDK-8059036 : Implement Diagnostic Commands for heap and finalizerinfo In-Reply-To: <0323E5E0-CFFA-4E11-8F51-270CB7F6964C@oracle.com> References: <554894E9.8020908@oracle.com> <5556F34B.5050701@oracle.com> <55573A31.7050306@gmail.com> <55573C86.8030807@gmail.com> <5559D85B.2050500@oracle.com> <6FB8DF19-1D67-4702-9642-4C630765D2D1@oracle.com> <555C2EEB.1050505@oracle.com> <3941B430-FE6C-4AF0-A875-76D24411C654@oracle.com> <5564802A.2010403@oracle.com> <5565738A.4040109@gmail.com> <7EB0BE71-1608-4047-B71D-1A9EAA9083BB@oracle.com> <556727A2.9060209@gmail.com> <55675908.6030406@oracle.com> <556AF3CD.50900@oracle.com> <37227441-B016-4A9A-B9CA-BE100E992C9A@oracle.com> <556D8FB0.4070608@oracle.com> <556E9970.8010704@gmail.com> <556EBF4B.6070400@oracle.com> <556F1D6C.6010806@oracle.com> <556F491E.3070709@oracle.com> <557054C7.4030705@oracle.com> <3607C2B6-F247-4291-9198-279CF8ED2AF3@oracle.com> <5570738B.5040603@oracle.com> <00038522-9EC7-4491-9AE4-73BD16D95909@oracle.com> <5571814B.9060706@oracle.com> <0323E5E0-CFFA-4E11-8F51-270CB7F6964C@oracle.com> Message-ID: <557576F7.4040000@oracle.com> Staffan, Could you review a CCC request. http://ccc.us.oracle.com/8059036 -Dmitry On 2015-06-05 15:24, Staffan Larsen wrote: > Thanks - this version looks good to me. > > /Staffan > >> On 5 jun 2015, at 13:00, Dmitry Samersoff wrote: >> >> Staffan, >> >> Thank you for review! >> >> Done. Webrev updated in-place (press shift-reload). >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >> >> -Dmitry >> >> On 2015-06-05 11:20, Staffan Larsen wrote: >>> Dmitry, >>> >>> I?d like to propose the following change to get prettier output (more in line with GC.class_histogram): >>> >>> diff --git a/src/share/vm/services/diagnosticCommand.cpp b/src/share/vm/services/diagnosticCommand.cpp >>> --- a/src/share/vm/services/diagnosticCommand.cpp >>> +++ b/src/share/vm/services/diagnosticCommand.cpp >>> @@ -341,7 +341,6 @@ >>> void FinalizerInfoDCmd::execute(DCmdSource source, TRAPS) { >>> ResourceMark rm; >>> >>> - output()->print_cr("Unreachable instances awaiting finalization:"); >>> Klass* k = SystemDictionary::resolve_or_null( >>> vmSymbols::finalizer_histogram_klass(), THREAD); >>> assert(k != NULL, "FinalizerHistogram class is not accessible"); >>> @@ -375,12 +374,15 @@ >>> >>> assert(count_res != NULL && name_res != NULL, "Unexpected layout of FinalizerHistogramEntry"); >>> >>> + output()->print_cr("Unreachable instances waiting for finalization"); >>> + output()->print_cr("#instances class name"); >>> + output()->print_cr("-----------------------"); >>> for (int i = 0; i < result_oop->length(); ++i) { >>> oop element_oop = result_oop->obj_at(i); >>> oop str_oop = element_oop->obj_field(name_fd.offset()); >>> char *name = java_lang_String::as_utf8_string(str_oop); >>> int count = element_oop->int_field(count_fd.offset()); >>> - output()->print_cr("Class %s - %d", name, count); >>> + output()->print_cr("%10d %s", count, name); >>> } >>> } >>> >>> >>> A couple of nits: >>> >>> diagnosticCommand.cpp:359 - extra space after = >>> diagnosticCommand.cpp:361 - spelling: finlalization -> finalization >>> FinalizerInfoTest.java:69: - spelling: intialized -> initialized >>> FinalizerInfoTest.java:71 - I?d like to see the ; on a new line >>> >>> >>> Thanks, >>> /Staffan >>> >>> >>>> On 5 jun 2015, at 00:20, Mandy Chung wrote: >>>> >>>> >>>>> On Jun 4, 2015, at 8:49 AM, Dmitry Samersoff wrote: >>>>> >>>>> Mandy, >>>>> >>>>> Webrev updated in-place (press shift-reload). >>>>> >>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>>> >>>>> getFinalizerHistogram() now returns Entry[] >>>>> >>>>> @library and @build removed from the test. >>>> >>>> >>>> Looks fine. >>>> >>>> Thanks >>>> Mandy >>>> >>>>> >>>>> -Dmitry >>>>> >>>>> On 2015-06-04 16:56, Mandy Chung wrote: >>>>>> >>>>>>> On Jun 4, 2015, at 6:38 AM, Dmitry Samersoff wrote: >>>>>>> >>>>>>> Mandy, >>>>>>> >>>>>>> Thank you for the review. >>>>>>> >>>>>>> Updated webrev is: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.14 >>>>>>> >>>>>> >>>>>> Thanks for the update and the test. >>>>>> >>>>>>> addressed comments, added a unit test to JDK workspace. >>>>>>> >>>>>> >>>>>> This test does not need @library and @build, right? >>>>>> >>>>>>> >>>>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>>>> >>>>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? >>>>>>>> The VM knows the returned type anyway. >>>>>>> >>>>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>>>> Object[] >>>>>>> >>>>>> >>>>>> You already add several symbols - why is it an issue for another one? Or if adding vm symbols is a concern, you should revert to String and int[] that you decide not to. The decision should apply consistently (use String and int, or new Java type). >>>>>> >>>>>> >>>>>>>> Perhaps the getFinalizerHistogram method should be renamed >>>>>>>> e.g. "dump"? >>>>>>> >>>>>>> The line in vmSymbols looks like: >>>>>>> >>>>>>> template( >>>>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>>>> >>>>>>> I would prefer to keep method name specific enough to be able to >>>>>>> find it by grep in jdk code. >>>>>> >>>>>> >>>>>> Grep in jdk code is easy as you have a new class :) >>>>>> >>>>>> Mandy >>>>>> >>>>>>> >>>>>>> (other comments are addressed) >>>>>>> >>>>>>> -Dmitry >>>>>>> >>>>>>> >>>>>>> On 2015-06-03 21:36, Mandy Chung wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 06/03/2015 08:29 AM, Dmitry Samersoff wrote: >>>>>>>>> Updated webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059036/webrev.13 >>>>>>>> >>>>>>>> I reviewed the jdk change. The version looks okay and some comments: >>>>>>>> >>>>>>>> ReferenceQueue.java - you should eliminate the use of rawtype: >>>>>>>> >>>>>>>> 84 Reference rn = r.next; >>>>>>>> >>>>>>>> Change Reference to Reference >>>>>>> >>>>>>> done. >>>>>>> >>>>>>>> >>>>>>>> The forEach method - eliminate the use of raw type and >>>>>>>> move @SuppressWarning to the field variable in line 182: >>>>>>>> >>>>>>>> @SuppressWarnings("unchecked") >>>>>>>> Reference rn = r.next; >>>>>>> >>>>>>> done. >>>>>>> >>>>>>>> >>>>>>>> FinalizerHistogram.java >>>>>>>> Copyright year needs update. >>>>>>> >>>>>>> done. >>>>>>>> >>>>>>>> I personally think the VM code would be much simpler if you stay with >>>>>>>> alternative entry of String and int than dealing with a >>>>>>>> FinalizerHistogramEntry private class. It's okay as FinalizerHistogram >>>>>>>> class is separated. >>>>>>>> >>>>>>>> The comment in line 35 is suitable to move to the class description and >>>>>>>> this is the suggestion. >>>>>>>> >>>>>>>> // This FinalizerHistogram class is for GC.finalizer_info diagnostic >>>>>>>> command support. >>>>>>>> // It is invoked by the VM. >>>>>>> >>>>>>> done. >>>>>>> >>>>>>>> >>>>>>>> Should getFinalizerHistogram() return FinalizerHistogramEntry[]? The VM >>>>>>>> knows the returned type anyway. >>>>>>> >>>>>>> "[java/lang/ref/Entry" signature would need a separate symbol entry in >>>>>>> swollen vmSymbols::init() so I would prefer to stay with more generic >>>>>>> Object[] >>>>>>> >>>>>>>> It's an inner class of >>>>>>>> FinalizerHistogram and it can simply be named as "Entry". I suggest to >>>>>>>> have Entry::increment method to replace ".instanceCount++". >>>>>>> >>>>>>> done. >>>>>>> >>>>>>>> >>>>>>>> The tests are in hotspot/test. Can you add a unit test in jdk/test? >>>>>>>> Perhaps you can test FinalizerHistogram.getFinalizerHistogram() via >>>>>>>> reflection - just a sanity test. >>>>>>> >>>>>>> done. The test repeats hotspot one, but uses reflection instead of VM call. >>>>>>> >>>>>>>> >>>>>>>> A minor naming comment - now you have a FinalizerHistogram class. >>>>>>>> Perhaps the getFinalizerHistogram method should be renamed e.g. "dump"? >>>>>>> >>>>>>> The line in vmSymbols looks like: >>>>>>> >>>>>>> template( >>>>>>> get_finalizer_histogram_name, "getFinalizerHistogram") >>>>>>> >>>>>>> I would prefer to keep it specific enough to be able to >>>>>>> find it by grep in jdk code. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dmitry Samersoff >>>>>>> Oracle Java development team, Saint Petersburg, Russia >>>>>>> * I would love to change the world, but they won't give me the sources. >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dmitry Samersoff >>>>> Oracle Java development team, Saint Petersburg, Russia >>>>> * I would love to change the world, but they won't give me the sources. >>>> >>> >> >> >> -- >> Dmitry Samersoff >> Oracle Java development team, Saint Petersburg, Russia >> * I would love to change the world, but they won't give me the sources. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From yekaterina.kantserova at oracle.com Mon Jun 8 11:21:57 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Mon, 08 Jun 2015 13:21:57 +0200 Subject: RFR(S): 8085813: The targeted processes in sun/tools tests should be launched with -XX:+UsePerfData flag in order to work on embedded platforms Message-ID: <55757AD5.6010508@oracle.com> Hi, Could I please have a review of this small fix. bug: https://bugs.openjdk.java.net/browse/JDK-8085813 webrev hotspot: http://cr.openjdk.java.net/~ykantser/8085813.hotspot/webrev.00/ webrev jdk: http://cr.openjdk.java.net/~ykantser/8085813.jdk/webrev.00/ Thanks, Katja From yekaterina.kantserova at oracle.com Mon Jun 8 13:32:14 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Mon, 08 Jun 2015 15:32:14 +0200 Subject: RFR(XS): 8085973: The targeted processes in javax/management tests should be launched with -XX:+UsePerfData flag in order to work on embedded platforms Message-ID: <5575995E.9070602@oracle.com> Hi, Could I please have a review of this small fix. bug: https://bugs.openjdk.java.net/browse/JDK-8085973 webrev: http://cr.openjdk.java.net/~ykantser/8085973/webrev.00/ Thanks, Katja From serguei.spitsyn at oracle.com Mon Jun 8 20:05:20 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 08 Jun 2015 13:05:20 -0700 Subject: RFR(S): 8085813: The targeted processes in sun/tools tests should be launched with -XX:+UsePerfData flag in order to work on embedded platforms In-Reply-To: <55757AD5.6010508@oracle.com> References: <55757AD5.6010508@oracle.com> Message-ID: <5575F580.2030209@oracle.com> It looks good. Thanks, Serguei On 6/8/15 4:21 AM, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this small fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8085813 > webrev hotspot: > http://cr.openjdk.java.net/~ykantser/8085813.hotspot/webrev.00/ > webrev jdk: http://cr.openjdk.java.net/~ykantser/8085813.jdk/webrev.00/ > > Thanks, > Katja From serguei.spitsyn at oracle.com Mon Jun 8 20:11:11 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 08 Jun 2015 13:11:11 -0700 Subject: RFR(XS): 8085973: The targeted processes in javax/management tests should be launched with -XX:+UsePerfData flag in order to work on embedded platforms In-Reply-To: <5575995E.9070602@oracle.com> References: <5575995E.9070602@oracle.com> Message-ID: <5575F6DF.2090304@oracle.com> Looks good. Thanks, Serguei On 6/8/15 6:32 AM, Yekaterina Kantserova wrote: > Hi, > > Could I please have a review of this small fix. > > bug: https://bugs.openjdk.java.net/browse/JDK-8085973 > webrev: http://cr.openjdk.java.net/~ykantser/8085973/webrev.00/ > > Thanks, > Katja From yekaterina.kantserova at oracle.com Tue Jun 9 07:04:13 2015 From: yekaterina.kantserova at oracle.com (Yekaterina Kantserova) Date: Tue, 09 Jun 2015 09:04:13 +0200 Subject: RFR(S): 8085813: The targeted processes in sun/tools tests should be launched with -XX:+UsePerfData flag in order to work on embedded platforms In-Reply-To: <5575F580.2030209@oracle.com> References: <55757AD5.6010508@oracle.com> <5575F580.2030209@oracle.com> Message-ID: <55768FED.2030202@oracle.com> Serguei, Thank you for the review! // Katja On 06/08/2015 10:05 PM, serguei.spitsyn at oracle.com wrote: > It looks good. > > Thanks, > Serguei > > On 6/8/15 4:21 AM, Yekaterina Kantserova wrote: >> Hi, >> >> Could I please have a review of this small fix. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8085813 >> webrev hotspot: >> http://cr.openjdk.java.net/~ykantser/8085813.hotspot/webrev.00/ >> webrev jdk: http://cr.openjdk.java.net/~ykantser/8085813.jdk/webrev.00/ >> >> Thanks, >> Katja > From dmitry.samersoff at oracle.com Tue Jun 9 12:48:41 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 09 Jun 2015 15:48:41 +0300 Subject: RFR(S, TESTONLY): JDK-8081576 serviceability/sa tests fail due to LingeredApp process fails to start Message-ID: <5576E0A9.9040105@oracle.com> Everybody, Please review a test-only fix: http://cr.openjdk.java.net/~dsamersoff/JDK-8081576/webrev.01/ LingeredApp can throw an exception during initialization if it not able to create a lock file for some reason. This exception cause NPE later, when the test attempts to stop LingeredApp and original exception get lost. Fixing it by adding static stopApp(app) method with null pointer check inside. -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From omajid at redhat.com Tue Jun 9 14:13:27 2015 From: omajid at redhat.com (Omair Majid) Date: Tue, 9 Jun 2015 10:13:27 -0400 Subject: Additional USDT probes for gc (was: Re: JEP 167: Event-Based JVM Tracing) In-Reply-To: <20121003161313.GB2828@redhat.com> References: <20121003161313.GB2828@redhat.com> Message-ID: <20150609141327.GB2791@redhat.com> Hi folks, * Lukas Berk [2012-10-03 12:14]: > I've written a patch that adds dtrace style probe points to various > garbage collection events, namely: > > Concurrent Mark Sweep Collections > G1 Mark Sweep Collections > ParNew Generation Collections > Parallel Scavenges > Parallel Compactions and Moves > Tenured Generation Collections > DefNew Generation Collections This patch has been included with icedtea for a little while now and some people are missing it in OpenJDK9 [1]. Hotspot already includes many probes so it seems sensible to me to add additional ones. Are there any concerns with this patch? Should I try and rebase the patch to latest JDK9 so it can be formally reviewed and pushed? Thanks, Omair [1] http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018834.html -- PGP Key: 66484681 (http://pgp.mit.edu/) Fingerprint = F072 555B 0A17 3957 4E95 0056 F286 F14F 6648 4681 From serguei.spitsyn at oracle.com Tue Jun 9 21:22:09 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 09 Jun 2015 14:22:09 -0700 Subject: RFR(S, TESTONLY): JDK-8081576 serviceability/sa tests fail due to LingeredApp process fails to start In-Reply-To: <5576E0A9.9040105@oracle.com> References: <5576E0A9.9040105@oracle.com> Message-ID: <55775901.7030203@oracle.com> Hi Dmitry, It looks good, just a couple of minor comments: root_webrev/test/lib/share/classes/jdk/test/lib/apps/LingeredApp.java 324 * Delete lock file that signal app to terminate, then 325 * waits until app is actually terminated. 326 * @throws IOException 327 */ 328 public void stopApp() throws IOException { 329 deleteLock(); 330 waitAppTerminate(); 331 int exitcode = appProcess.exitValue(); 332 if (exitcode != 0) { 333 throw new IOException("LingeredApp terminated with non-zero exit code " + exitcode); 334 } 335 } The comment needs a correction: that signal app =>that signals app waits until =>wait until It is a little bit strange that an IOException is thrown when the exit code does not match the expectation. Thanks, Serguei On 6/9/15 5:48 AM, Dmitry Samersoff wrote: > Everybody, > > Please review a test-only fix: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8081576/webrev.01/ > > LingeredApp can throw an exception during initialization if it not able > to create a lock file for some reason. This exception cause NPE later, > when the test attempts to stop LingeredApp and original exception get lost. > > Fixing it by adding static stopApp(app) method with null pointer check > inside. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysr1729 at gmail.com Wed Jun 10 07:15:30 2015 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Wed, 10 Jun 2015 00:15:30 -0700 Subject: More visibility into code cache churn Message-ID: I filed https://bugs.openjdk.java.net/browse/JDK-8087107 and attached a patch to the ticket that exposes some useful code cache stats as perf data counters: $ jcmd PerfCounter.print | grep -i "sun\.ci\." sun.ci.codeCacheCapacity=6291456 sun.ci.codeCacheMaxCapacity=6291456 sun.ci.codeCacheMethodsReclaimedNum=1030 sun.ci.codeCacheSweepsTotalNum=93 sun.ci.codeCacheSweepsTotalTimeMillis=63 sun.ci.codeCacheUsed=3386880 ... At Twitter, we've found this useful for easy monitoring of code cache activity, and would like to see this integrated into OpenJDK. thanks! -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Wed Jun 10 08:53:02 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Wed, 10 Jun 2015 10:53:02 +0200 Subject: More visibility into code cache churn In-Reply-To: References: Message-ID: Hi Ramki, Anything to improve visibility of the CodeCache would be hugely appreciated! Regards, Kirk On Jun 10, 2015, at 9:15 AM, Srinivas Ramakrishna wrote: > > I filed https://bugs.openjdk.java.net/browse/JDK-8087107 and attached a patch to the ticket that exposes some useful code cache stats as perf data counters: > > $ jcmd PerfCounter.print | grep -i "sun\.ci\." > sun.ci.codeCacheCapacity=6291456 > sun.ci.codeCacheMaxCapacity=6291456 > sun.ci.codeCacheMethodsReclaimedNum=1030 > sun.ci.codeCacheSweepsTotalNum=93 > sun.ci.codeCacheSweepsTotalTimeMillis=63 > sun.ci.codeCacheUsed=3386880 > ... > > At Twitter, we've found this useful for easy monitoring of code cache activity, and would like to see this integrated into OpenJDK. > > thanks! > -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Fri Jun 12 13:12:43 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Fri, 12 Jun 2015 16:12:43 +0300 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <555B54AE.9050100@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> Message-ID: <557ADACB.4040206@oracle.com> Andrew, Found minor nit in the fix, please update the webrev: diff -r 640718253035 -r f043d189d1e1 agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java --- a/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java Tue May 19 15:37:29 2015 +0100 +++ b/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java Thu Jun 11 16:54:41 2015 +0300 @@ -391,7 +391,7 @@ map.setIncludeArgumentOops(cb.callerMustGCArguments()); if (cb.getOopMaps() != null) { - OopMapSet.updateRegisterMap(this, cb, map, true); + ImmutableOopMapSet.updateRegisterMap(this, cb, map, true); } // Since the prolog does the save and restore of FP there is no oopmap -Dmitry On 2015-05-19 18:20, Andrew Haley wrote: > On 05/16/2015 06:03 PM, Dmitry Samersoff wrote: >> Andrew, >> >> On 2015-05-12 20:05, Andrew Haley wrote: >>> http://cr.openjdk.java.net/~aph/8078521-4/ >> >> HSDB.java: >> >> 988: If you removed else part, you don't need to check CPU here at all, >> just leave a comment. > > This will now enable the code for PPC; I guess that's OK. > >> LinuxAARCH64CFrame.java: >> >> 65 - it's better to add brackets around 2 * ADDRESS_SIZE >> 69,73 - please, remove space after bracket. > > Done. > > http://cr.openjdk.java.net/~aph/8078521-5/ > > Thanks, > Andrew. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From aph at redhat.com Fri Jun 12 14:58:42 2015 From: aph at redhat.com (Andrew Haley) Date: Fri, 12 Jun 2015 15:58:42 +0100 Subject: PING Re: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <557ADACB.4040206@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> Message-ID: <557AF3A2.4010508@redhat.com> On 06/12/2015 02:12 PM, Dmitry Samersoff wrote: > Andrew, > > Found minor nit in the fix, please update the webrev: > > diff -r 640718253035 -r f043d189d1e1 > agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java > --- > a/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java > Tue May 19 15:37:29 2015 +0100 > +++ > b/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java > Thu Jun 11 16:54:41 2015 +0300 > @@ -391,7 +391,7 @@ > map.setIncludeArgumentOops(cb.callerMustGCArguments()); > > if (cb.getOopMaps() != null) { > - OopMapSet.updateRegisterMap(this, cb, map, true); > + ImmutableOopMapSet.updateRegisterMap(this, cb, map, true); > } > > // Since the prolog does the save and restore of FP there is no > oopmap http://cr.openjdk.java.net/~aph/8078521-6/ Andrew. From alexander.kulyakhtin at oracle.com Tue Jun 16 12:29:57 2015 From: alexander.kulyakhtin at oracle.com (Alexander Kulyakhtin) Date: Tue, 16 Jun 2015 05:29:57 -0700 (PDT) Subject: RFR: JDK-8062045: Update svc regression tests to extend the default security policy instead of override Message-ID: <6d50b350-6df5-4fcd-a0fa-67b4b5281971@default> Hi, Could you, please, review the small test-only changes below: CR: https://bugs.openjdk.java.net/browse/JDK-8062045 "Update svc regression tests to extend the default security policy instead of override" Webrev: http://cr.openjdk.java.net/~akulyakh/8062045/webrev/ This changes main/othervm/policy to main/othervm/java.security.policy in the tests from the jdk_svc tests group. These tests need to extend the default system policy, not to override it. Therefore they should use 'java.security.policy'. The changes are similar to what has been previously done in JDK-8043277 and JDK-8062047 Best regards, Alexander From jaroslav.bachorik at oracle.com Tue Jun 16 12:33:48 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 16 Jun 2015 14:33:48 +0200 Subject: RFR: JDK-8062045: Update svc regression tests to extend the default security policy instead of override In-Reply-To: <6d50b350-6df5-4fcd-a0fa-67b4b5281971@default> References: <6d50b350-6df5-4fcd-a0fa-67b4b5281971@default> Message-ID: <558017AC.7070104@oracle.com> Looks good! -JB- On 16.6.2015 14:29, Alexander Kulyakhtin wrote: > Hi, > > Could you, please, review the small test-only changes below: > > CR: https://bugs.openjdk.java.net/browse/JDK-8062045 "Update svc regression tests to extend the default security policy instead of override" > Webrev: http://cr.openjdk.java.net/~akulyakh/8062045/webrev/ > > This changes main/othervm/policy to main/othervm/java.security.policy in the tests from the jdk_svc tests group. > These tests need to extend the default system policy, not to override it. Therefore they should use 'java.security.policy'. > > The changes are similar to what has been previously done in JDK-8043277 and JDK-8062047 > > Best regards, > Alexander > From alexander.kulyakhtin at oracle.com Tue Jun 16 12:53:33 2015 From: alexander.kulyakhtin at oracle.com (Alexander Kulyakhtin) Date: Tue, 16 Jun 2015 05:53:33 -0700 (PDT) Subject: RFR: JDK-8062045: Update svc regression tests to extend the default security policy instead of override Message-ID: <5f3f3e46-d360-4137-8f6c-475b0f3d1352@default> Jaroslav, Thank you very much for the review. Best regards, Alexander ----- Original Message ----- From: jaroslav.bachorik at oracle.com To: serviceability-dev at openjdk.java.net Sent: Tuesday, June 16, 2015 3:34:09 PM GMT +03:00 Iraq Subject: Re: RFR: JDK-8062045: Update svc regression tests to extend the default security policy instead of override Looks good! -JB- On 16.6.2015 14:29, Alexander Kulyakhtin wrote: > Hi, > > Could you, please, review the small test-only changes below: > > CR: https://bugs.openjdk.java.net/browse/JDK-8062045 "Update svc regression tests to extend the default security policy instead of override" > Webrev: http://cr.openjdk.java.net/~akulyakh/8062045/webrev/ > > This changes main/othervm/policy to main/othervm/java.security.policy in the tests from the jdk_svc tests group. > These tests need to extend the default system policy, not to override it. Therefore they should use 'java.security.policy'. > > The changes are similar to what has been previously done in JDK-8043277 and JDK-8062047 > > Best regards, > Alexander > From mandy.chung at oracle.com Tue Jun 16 20:50:52 2015 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 16 Jun 2015 13:50:52 -0700 Subject: RFR: JDK-8062045: Update svc regression tests to extend the default security policy instead of override In-Reply-To: <6d50b350-6df5-4fcd-a0fa-67b4b5281971@default> References: <6d50b350-6df5-4fcd-a0fa-67b4b5281971@default> Message-ID: <9201222B-0430-4CC7-B06E-170C64699505@oracle.com> Looks good. Thanks for doing that and this will make the tests behaving the same way when running standalone and with jtreg. Mandy > On Jun 16, 2015, at 5:29 AM, Alexander Kulyakhtin wrote: > > Hi, > > Could you, please, review the small test-only changes below: > > CR: https://bugs.openjdk.java.net/browse/JDK-8062045 "Update svc regression tests to extend the default security policy instead of override" > Webrev: http://cr.openjdk.java.net/~akulyakh/8062045/webrev/ > > This changes main/othervm/policy to main/othervm/java.security.policy in the tests from the jdk_svc tests group. > These tests need to extend the default system policy, not to override it. Therefore they should use 'java.security.policy'. > > The changes are similar to what has been previously done in JDK-8043277 and JDK-8062047 > > Best regards, > Alexander From jaroslav.bachorik at oracle.com Wed Jun 17 18:54:17 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Wed, 17 Jun 2015 20:54:17 +0200 Subject: RFR 8081634: Concurrent usage of a StringBuilder causes test intermittent failures Message-ID: <5581C259.5000402@oracle.com> Please, review the following test change Issue : https://bugs.openjdk.java.net/browse/JDK-8081634 Webrev: http://cr.openjdk.java.net/~jbachorik/8081634/webrev.00 This is an issue in the utility class providing the functionality to run the 'jcmd' command and parse its output. The jcmd command output is being captured in a separate thread and fed to a shared StringBuilder instance. StringBuilder not being synchronized this might lead to intermittent failures due to (mostly) the write visibility. The fix is to use StringBuffer which is the appropriate class for this kind of usage. Thanks, -JB- From staffan.larsen at oracle.com Thu Jun 18 08:50:16 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 18 Jun 2015 10:50:16 +0200 Subject: RFR 8081634: Concurrent usage of a StringBuilder causes test intermittent failures In-Reply-To: <5581C259.5000402@oracle.com> References: <5581C259.5000402@oracle.com> Message-ID: <62637387-8CE6-41BE-8D62-3C9E0499777B@oracle.com> Looks good! Thanks, /Staffan > On 17 jun 2015, at 20:54, Jaroslav Bachorik wrote: > > Please, review the following test change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8081634 > Webrev: http://cr.openjdk.java.net/~jbachorik/8081634/webrev.00 > > This is an issue in the utility class providing the functionality to run the 'jcmd' command and parse its output. The jcmd command output is being captured in a separate thread and fed to a shared StringBuilder instance. StringBuilder not being synchronized this might lead to intermittent failures due to (mostly) the write visibility. The fix is to use StringBuffer which is the appropriate class for this kind of usage. > > Thanks, > > -JB- From staffan.larsen at oracle.com Thu Jun 18 08:56:24 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 18 Jun 2015 10:56:24 +0200 Subject: RFR: JDK-8122944 perfdata used is seen as too high on sparc zone with jdk1.9 and causes a test failure Message-ID: <45E7A5EF-8BFB-46A7-B214-3293C3BD001A@oracle.com> This test is failing on machines with lots of CPUs since we create lots of compiler threads and there are 4 perf counters for each compiler thread. This eats memory. A simple solution is to just bump the default for PerfDataMemorySize from 32K to 64K - "that should be enough for everyone.? Thanks, /Staffan bug: https://bugs.openjdk.java.net/browse/JDK-8122944 diff --git a/src/share/vm/runtime/globals.hpp b/src/share/vm/runtime/globals.hpp --- a/src/share/vm/runtime/globals.hpp +++ b/src/share/vm/runtime/globals.hpp @@ -3733,7 +3733,7 @@ product(bool, PerfDisableSharedMem, false, \ "Store performance data in standard memory") \ \ - product(intx, PerfDataMemorySize, 32*K, \ + product(intx, PerfDataMemorySize, 64*K, \ "Size of performance data memory region. Will be rounded " \ "up to a multiple of the native os page size.") \ From serguei.spitsyn at oracle.com Thu Jun 18 10:11:42 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 18 Jun 2015 03:11:42 -0700 Subject: RFR: JDK-8122944 perfdata used is seen as too high on sparc zone with jdk1.9 and causes a test failure In-Reply-To: <45E7A5EF-8BFB-46A7-B214-3293C3BD001A@oracle.com> References: <45E7A5EF-8BFB-46A7-B214-3293C3BD001A@oracle.com> Message-ID: <5582995E.7040901@oracle.com> It looks good to me. Thanks, Serguei On 6/18/15 1:56 AM, Staffan Larsen wrote: > This test is failing on machines with lots of CPUs since we create lots of compiler threads and there are 4 perf counters for each compiler thread. This eats memory. > > A simple solution is to just bump the default for PerfDataMemorySize from 32K to 64K - "that should be enough for everyone.? > > Thanks, > /Staffan > > bug: https://bugs.openjdk.java.net/browse/JDK-8122944 > > > > diff --git a/src/share/vm/runtime/globals.hpp b/src/share/vm/runtime/globals.hpp > --- a/src/share/vm/runtime/globals.hpp > +++ b/src/share/vm/runtime/globals.hpp > @@ -3733,7 +3733,7 @@ > product(bool, PerfDisableSharedMem, false, \ > "Store performance data in standard memory") \ > \ > - product(intx, PerfDataMemorySize, 32*K, \ > + product(intx, PerfDataMemorySize, 64*K, \ > "Size of performance data memory region. Will be rounded " \ > "up to a multiple of the native os page size.") \ > From jaroslav.bachorik at oracle.com Thu Jun 18 11:01:06 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 18 Jun 2015 13:01:06 +0200 Subject: RFR 8080138: sun/management/jmxremote/startstop/JMXStartStopTest.java failed with java.lang.Error intermittently Message-ID: <5582A4F2.30806@oracle.com> Please, review the following test change Issue : https://bugs.openjdk.java.net/browse/JDK-8080138 Webrev: http://cr.openjdk.java.net/~jbachorik/8080138/webrev.00 The test is using DCMD to turn on the management agent. It seems that under certain conditions the time needed to actually startup the agent the RMI connector may be longer than expected and the test will try to connect to the target process while the initialization is still in progress. This will lead to intermittent failures with EOFException as the cause. The patch adds a retry logic that will keep on trying to connect to the target process for a certain amount of time (adjusted by the test suite timeout factor) and fails only if the target process is not responding after this time has elapsed. Thanks, -JB- From serguei.spitsyn at oracle.com Thu Jun 18 11:10:14 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 18 Jun 2015 04:10:14 -0700 Subject: RFR 8081634: Concurrent usage of a StringBuilder causes test intermittent failures In-Reply-To: <5581C259.5000402@oracle.com> References: <5581C259.5000402@oracle.com> Message-ID: <5582A716.70003@oracle.com> Looks good. Thanks, Serguei On 6/17/15 11:54 AM, Jaroslav Bachorik wrote: > Please, review the following test change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8081634 > Webrev: http://cr.openjdk.java.net/~jbachorik/8081634/webrev.00 > > This is an issue in the utility class providing the functionality to > run the 'jcmd' command and parse its output. The jcmd command output > is being captured in a separate thread and fed to a shared > StringBuilder instance. StringBuilder not being synchronized this > might lead to intermittent failures due to (mostly) the write > visibility. The fix is to use StringBuffer which is the appropriate > class for this kind of usage. > > Thanks, > > -JB- From staffan.larsen at oracle.com Thu Jun 18 11:16:48 2015 From: staffan.larsen at oracle.com (Staffan Larsen) Date: Thu, 18 Jun 2015 13:16:48 +0200 Subject: RFR 8080138: sun/management/jmxremote/startstop/JMXStartStopTest.java failed with java.lang.Error intermittently In-Reply-To: <5582A4F2.30806@oracle.com> References: <5582A4F2.30806@oracle.com> Message-ID: Looks good! Thanks, /Staffan > On 18 jun 2015, at 13:01, Jaroslav Bachorik wrote: > > Please, review the following test change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8080138 > Webrev: http://cr.openjdk.java.net/~jbachorik/8080138/webrev.00 > > The test is using DCMD to turn on the management agent. It seems that under certain conditions the time needed to actually startup the agent the RMI connector may be longer than expected and the test will try to connect to the target process while the initialization is still in progress. This will lead to intermittent failures with EOFException as the cause. > > The patch adds a retry logic that will keep on trying to connect to the target process for a certain amount of time (adjusted by the test suite timeout factor) and fails only if the target process is not responding after this time has elapsed. > > Thanks, > > -JB- From jaroslav.bachorik at oracle.com Thu Jun 18 16:39:32 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 18 Jun 2015 18:39:32 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector Message-ID: <5582F444.1090108@oracle.com> Please, review the following change Issue : https://bugs.openjdk.java.net/browse/JDK-8129215 Webrev: http://cr.openjdk.java.net/~jbachorik/8129215/webrev.00 The JMX Introspector will try to use the JavaBeans introspector whenever possible, delegating the requests for the property getter/setter methods. However, when the JavaBeans introspector is not available (modules) the JMX Introspector falls back to its own simple, reflection based, algorithm. The simple algorithm does not enforce the rule of property names starting with a lower-case letter. This might lead to situations when the simple introspector provides the getter method for an attribute the JavaBeans introspector would not (eg. 'Attribute' attribute) This patch changes the simple introspector behaviour to conform to the one of the JavaBeans introspector. Also, it makes the simple introspector called only when the JavaBeans introspector is not available - and not when the JavaBeans introspector fails to resolve a property getter. Thanks, -JB- From daniel.fuchs at oracle.com Thu Jun 18 16:47:00 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Thu, 18 Jun 2015 18:47:00 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5582F444.1090108@oracle.com> References: <5582F444.1090108@oracle.com> Message-ID: <5582F604.60909@oracle.com> Hi Jaroslav, I haven't looked at the code, but if I understand well, that would be a spec change. Attribute names are case sensitive in JMX. getFoo() => attribute named "Foo" getfoo() => attribute named "foo" Are you proposing to change that? best regards, -- daniel On 18/06/15 18:39, Jaroslav Bachorik wrote: > Please, review the following change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8129215 > Webrev: http://cr.openjdk.java.net/~jbachorik/8129215/webrev.00 > > The JMX Introspector will try to use the JavaBeans introspector whenever > possible, delegating the requests for the property getter/setter > methods. However, when the JavaBeans introspector is not available > (modules) the JMX Introspector falls back to its own simple, reflection > based, algorithm. > > The simple algorithm does not enforce the rule of property names > starting with a lower-case letter. This might lead to situations when > the simple introspector provides the getter method for an attribute the > JavaBeans introspector would not (eg. 'Attribute' attribute) > > This patch changes the simple introspector behaviour to conform to the > one of the JavaBeans introspector. Also, it makes the simple > introspector called only when the JavaBeans introspector is not > available - and not when the JavaBeans introspector fails to resolve a > property getter. > > Thanks, > > -JB- From jaroslav.bachorik at oracle.com Thu Jun 18 17:16:31 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 18 Jun 2015 19:16:31 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5582F604.60909@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> Message-ID: <5582FCEF.3010000@oracle.com> On 18.6.2015 18:47, Daniel Fuchs wrote: > Hi Jaroslav, > > I haven't looked at the code, but if I understand well, > that would be a spec change. > > Attribute names are case sensitive in JMX. > > getFoo() => attribute named "Foo" > getfoo() => attribute named "foo" > > Are you proposing to change that? In this case it is an even bigger mess. This would work as long as you don't go through the JavaBeans introspector. The property descriptor generated for the "getFoo" method will be named "foo" and it is indistinguishable from "getfoo". Therefore, when you are trying to reach this attribute as "Foo" the JavaBeans introspector will not resolve the getter name. The SimpleIntrospector, on the other hand, will. I am proposing to change the SimpleIntrospector so it behaves exactly as the JavaBeans introspector. But it seems that even that might be considered a specification change :/ -JB- > > best regards, > > -- daniel > > On 18/06/15 18:39, Jaroslav Bachorik wrote: >> Please, review the following change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8129215 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8129215/webrev.00 >> >> The JMX Introspector will try to use the JavaBeans introspector whenever >> possible, delegating the requests for the property getter/setter >> methods. However, when the JavaBeans introspector is not available >> (modules) the JMX Introspector falls back to its own simple, reflection >> based, algorithm. >> >> The simple algorithm does not enforce the rule of property names >> starting with a lower-case letter. This might lead to situations when >> the simple introspector provides the getter method for an attribute the >> JavaBeans introspector would not (eg. 'Attribute' attribute) >> >> This patch changes the simple introspector behaviour to conform to the >> one of the JavaBeans introspector. Also, it makes the simple >> introspector called only when the JavaBeans introspector is not >> available - and not when the JavaBeans introspector fails to resolve a >> property getter. >> >> Thanks, >> >> -JB- > From daniel.fuchs at oracle.com Thu Jun 18 22:02:25 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 19 Jun 2015 00:02:25 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5582FCEF.3010000@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> Message-ID: <55833FF1.7020007@oracle.com> On 6/18/15 7:16 PM, Jaroslav Bachorik wrote: > On 18.6.2015 18:47, Daniel Fuchs wrote: >> Hi Jaroslav, >> >> I haven't looked at the code, but if I understand well, >> that would be a spec change. >> >> Attribute names are case sensitive in JMX. >> >> getFoo() => attribute named "Foo" >> getfoo() => attribute named "foo" >> >> Are you proposing to change that? > > In this case it is an even bigger mess. This would work as long as you > don't go through the JavaBeans introspector. The property descriptor > generated for the "getFoo" method will be named "foo" and it is > indistinguishable from "getfoo". Therefore, when you are trying to > reach this attribute as "Foo" the JavaBeans introspector will not > resolve the getter name. The SimpleIntrospector, on the other hand, will. > > I am proposing to change the SimpleIntrospector so it behaves exactly > as the JavaBeans introspector. But it seems that even that might be > considered a specification change :/ In the JMX 1.4 specification - section 2.2.2.3: https://docs.oracle.com/javase/8/docs/technotes/guides/jmx/JMX_1_4_specification.pdf << 2.2.2.3 Case Sensitivity All attribute and operation names derived from these design patterns are case- sensitive. For example, this means that the methods 'getstate' and 'setState' define two attributes, one called 'state' that is read-only, and one called 'State' that is write-only. While case sensitivity applies directly to component names of standard MBeans, it is also applicable to all component names of all types of MBeans, standard or dynamic. In general, all names of classes, attributes, operations, methods, and internal elements defined in the JMX specification are case sensitive, whether they appear as data or as functional code when they are manipulated by management operations. >> -- daniel > > -JB- > >> >> best regards, >> >> -- daniel >> >> On 18/06/15 18:39, Jaroslav Bachorik wrote: >>> Please, review the following change >>> >>> Issue : https://bugs.openjdk.java.net/browse/JDK-8129215 >>> Webrev: http://cr.openjdk.java.net/~jbachorik/8129215/webrev.00 >>> >>> The JMX Introspector will try to use the JavaBeans introspector >>> whenever >>> possible, delegating the requests for the property getter/setter >>> methods. However, when the JavaBeans introspector is not available >>> (modules) the JMX Introspector falls back to its own simple, reflection >>> based, algorithm. >>> >>> The simple algorithm does not enforce the rule of property names >>> starting with a lower-case letter. This might lead to situations when >>> the simple introspector provides the getter method for an attribute the >>> JavaBeans introspector would not (eg. 'Attribute' attribute) >>> >>> This patch changes the simple introspector behaviour to conform to the >>> one of the JavaBeans introspector. Also, it makes the simple >>> introspector called only when the JavaBeans introspector is not >>> available - and not when the JavaBeans introspector fails to resolve a >>> property getter. >>> >>> Thanks, >>> >>> -JB- >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Fri Jun 19 06:00:15 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 19 Jun 2015 08:00:15 +0200 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <20150606190707.000044ba.ecki@zusammenkunft.net> References: <20150606190707.000044ba.ecki@zusammenkunft.net> Message-ID: <20150619080015.000025b1.ecki@zusammenkunft.net> Hello, Am Sat, 6 Jun 2015 19:07:07 +0200 schrieb Bernd Eckenfels : > - why is BufferPoolMXBean not tracking allocation failures, allocation > count, alignment flag and maximum size (maybe even modifyable). > Would it be worth to contribute something in this area or is that > supposed to be covered by some events (and if yes, when will they be > exposed to JMX). I had a look at it, the following patch would track number of allocations for direct and mapped buffers as well as number of "failed" allocations for the direct buffers. (The name of the JMX Attributes need to be changed, what would you suggest?) diff -r bc4bbb07768e src/java.base/share/classes/java/nio/Bits.java --- a/src/java.base/share/classes/java/nio/Bits.java Thu Jun 18 17:20:42 2015 -0700 +++ b/src/java.base/share/classes/java/nio/Bits.java Fri Jun 19 05:55:35 2015 +0000 @@ -602,6 +602,8 @@ private static final AtomicLong reservedMemory = new AtomicLong(); private static final AtomicLong totalCapacity = new AtomicLong(); private static final AtomicLong count = new AtomicLong(); + private static final AtomicLong totalAllocations = new AtomicLong(); + private static final AtomicLong failedAllocations = new AtomicLong(); private static volatile boolean memoryLimitSet = false; // max. number of sleeps during try-reserving with exponentially // increasing delay before throwing OutOfMemoryError: @@ -624,6 +626,8 @@ return; } + failedAllocations.incrementAndGet(); + final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess(); // retry while helping enqueue pending Reference objects @@ -683,6 +687,7 @@ if (totalCapacity.compareAndSet(totalCap, totalCap + cap)) { reservedMemory.addAndGet(size); count.incrementAndGet(); + totalAllocations.incrementAndGet(); return true; } } @@ -723,6 +728,13 @@ public long getMemoryUsed() { return Bits.reservedMemory.get(); } + @Override + public long getTotalAllocatedBuffers() { + return Bits.totalAllocations.get(); + } + public long getFailedAllocatedBuffers() { + return Bits.failedAllocations.get(); + } }; } @Override diff -r bc4bbb07768e src/java.base/share/classes/sun/misc/JavaNioAccess.java --- a/src/java.base/share/classes/sun/misc/JavaNioAccess.java Thu Jun 18 17:20:42 2015 -0700 +++ b/src/java.base/share/classes/sun/misc/JavaNioAccess.java Fri Jun 19 05:55:35 2015 +0000 @@ -37,6 +37,8 @@ long getCount(); long getTotalCapacity(); long getMemoryUsed(); + long getTotalAllocatedBuffers(); + long getFailedAllocatedBuffers(); } BufferPool getDirectBufferPool(); diff -r bc4bbb07768e src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java Thu Jun 18 17:20:42 2015 -0700 +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java Fri Jun 19 05:55:35 2015 +0000 @@ -797,6 +797,7 @@ static volatile int count; static volatile long totalSize; static volatile long totalCapacity; + static volatile long totalAllocated; private volatile long address; private final long size; @@ -816,6 +817,7 @@ count++; totalSize += size; totalCapacity += cap; + totalAllocated++; } } @@ -993,6 +995,14 @@ public long getMemoryUsed() { return Unmapper.totalSize; } + @Override + public long getTotalAllocatedBuffers() { + return Unmapper.totalAllocated; + } + @Override + public long getFailedAllocatedBuffers() { + return -1; + } }; } diff -r bc4bbb07768e src/java.management/share/classes/java/lang/management/BufferPoolMXBean.java --- a/src/java.management/share/classes/java/lang/management/BufferPoolMXBean.java Thu Jun 18 17:20:42 2015 -0700 +++ b/src/java.management/share/classes/java/lang/management/BufferPoolMXBean.java Fri Jun 19 05:55:35 2015 +0000 @@ -90,4 +90,25 @@ * the memory usage is not available */ long getMemoryUsed(); + + /** + * Returns an estimate of the total number of buffer allocations since start of JVM. + * + * @return Number of direct buffers (estimated) allocated since JVM start. + */ + long getTotalAllocatedBuffers(); + + /** + * Returns an estimate of the number of times the buffer limit was reached. + *

+ * This counts all allocations which do not immediatelly suceed. The counter + * gets increased no matter if the retried allocation failed or caused an + * {@link java.lang.OutOfMemoryException}. + * + * @return Number of allocations which failed or have been retried or {@code -1L} + * if no limit is enforced. This especially tracks {@code -XX:MaxDirectMemory} + * limit for {@link java.nio.ByteBuffer#allocateDirect direct} buffers. + */ + long getFailedAllocatedBuffers(); + } diff -r bc4bbb07768e src/java.management/share/classes/sun/management/ManagementFactoryHelper.java --- a/src/java.management/share/classes/sun/management/ManagementFactoryHelper.java Thu Jun 18 17:20:42 2015 -0700 +++ b/src/java.management/share/classes/sun/management/ManagementFactoryHelper.java Fri Jun 19 05:55:35 2015 +0000 @@ -257,6 +257,14 @@ public long getMemoryUsed() { return pool.getMemoryUsed(); } + @Override + public long getTotalAllocatedBuffers() { + return pool.getTotalAllocatedBuffers(); + } + @Override + public long getFailedAllocatedBuffers() { + return pool.getFailedAllocatedBuffers(); + } }; } Gruss Bernd From Alan.Bateman at oracle.com Fri Jun 19 08:10:44 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jun 2015 09:10:44 +0100 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <20150619080015.000025b1.ecki@zusammenkunft.net> References: <20150606190707.000044ba.ecki@zusammenkunft.net> <20150619080015.000025b1.ecki@zusammenkunft.net> Message-ID: <5583CE84.20800@oracle.com> On 19/06/2015 07:00, Bernd Eckenfels wrote: > Hello, > > Am Sat, 6 Jun 2015 19:07:07 +0200 > schrieb Bernd Eckenfels : > >> - why is BufferPoolMXBean not tracking allocation failures, allocation >> count, alignment flag and maximum size (maybe even modifyable). >> Would it be worth to contribute something in this area or is that >> supposed to be covered by some events (and if yes, when will they be >> exposed to JMX). > When we created BufferPoolMXBean (in JSR-203) then the goal was to make it easy to monitor current usage. It wouldn't be hard to track peak usage if that is important. Direct buffers have not been page aligned for some time so that is probably not worth trying to expose. The nio-dev and serviceability-dev would be the right place to being proposals in this area. -Alan From jaroslav.bachorik at oracle.com Fri Jun 19 09:56:19 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Fri, 19 Jun 2015 11:56:19 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55833FF1.7020007@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> Message-ID: <5583E743.1090006@oracle.com> On 19.6.2015 00:02, Daniel Fuchs wrote: > On 6/18/15 7:16 PM, Jaroslav Bachorik wrote: >> On 18.6.2015 18:47, Daniel Fuchs wrote: >>> Hi Jaroslav, >>> >>> I haven't looked at the code, but if I understand well, >>> that would be a spec change. >>> >>> Attribute names are case sensitive in JMX. >>> >>> getFoo() => attribute named "Foo" >>> getfoo() => attribute named "foo" >>> >>> Are you proposing to change that? >> >> In this case it is an even bigger mess. This would work as long as you >> don't go through the JavaBeans introspector. The property descriptor >> generated for the "getFoo" method will be named "foo" and it is >> indistinguishable from "getfoo". Therefore, when you are trying to >> reach this attribute as "Foo" the JavaBeans introspector will not >> resolve the getter name. The SimpleIntrospector, on the other hand, will. >> >> I am proposing to change the SimpleIntrospector so it behaves exactly >> as the JavaBeans introspector. But it seems that even that might be >> considered a specification change :/ > In the JMX 1.4 specification - section 2.2.2.3: > > https://docs.oracle.com/javase/8/docs/technotes/guides/jmx/JMX_1_4_specification.pdf > > << > 2.2.2.3 Case Sensitivity > > All attribute and operation names derived from these design patterns are > case- > sensitive. For example, this means that the methods 'getstate' and > 'setState' > define two attributes, one called 'state' that is read-only, and one called > 'State' that is write-only. > > While case sensitivity applies directly to component names of standard > MBeans, it is > also applicable to all component names of all types of MBeans, standard > or dynamic. > In general, all names of classes, attributes, operations, methods, and > internal > elements defined in the JMX specification are case sensitive, whether > they appear as > data or as functional code when they are manipulated by management > operations. > >> I know, I checked the spec. The problem is that the JavaBeans Introspector, which may get used internally in the JMX code, doesn't follow this spec. Try this code: ``` public static class BeanClass1 { public int getAttribute() {return 0;} public int getattribute() {return 1;} } public static class BeanClass2 { public int getAttribute() {return 0;} } public static void main(String[] args) throws Exception { System.err.println("=== BeanClass1"); printInfo(Introspector.getBeanInfo(BeanClass1.class)); System.err.println("=== BeanClass2"); printInfo(Introspector.getBeanInfo(BeanClass2.class)); } private static void printInfo(BeanInfo bi) { for(PropertyDescriptor pd : bi.getPropertyDescriptors()) { System.err.println("*** " + pd.getName() + " : " + pd.getReadMethod().getName()); } } ``` According to the JMX spec there should be 2 attributes: 'attribute' and 'Attribute' However, there is only one attribute resolved by the JavaBeans Introspector and it is 'attribute'. Furthermore, the 'getattribute' method is chosen as the getter method, rather counter-intuitively. Anyway, this specification aspect doesn't seem to be enforced neither by the JTREG test suite nor the JCK tests. I successfully ran jdk_jmx JTREG testset and api/javax_management JCK suite on my patched version of JDK. -JB- > > -- daniel > >> >> -JB- >> >>> >>> best regards, >>> >>> -- daniel >>> >>> On 18/06/15 18:39, Jaroslav Bachorik wrote: >>>> Please, review the following change >>>> >>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8129215 >>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8129215/webrev.00 >>>> >>>> The JMX Introspector will try to use the JavaBeans introspector >>>> whenever >>>> possible, delegating the requests for the property getter/setter >>>> methods. However, when the JavaBeans introspector is not available >>>> (modules) the JMX Introspector falls back to its own simple, reflection >>>> based, algorithm. >>>> >>>> The simple algorithm does not enforce the rule of property names >>>> starting with a lower-case letter. This might lead to situations when >>>> the simple introspector provides the getter method for an attribute the >>>> JavaBeans introspector would not (eg. 'Attribute' attribute) >>>> >>>> This patch changes the simple introspector behaviour to conform to the >>>> one of the JavaBeans introspector. Also, it makes the simple >>>> introspector called only when the JavaBeans introspector is not >>>> available - and not when the JavaBeans introspector fails to resolve a >>>> property getter. >>>> >>>> Thanks, >>>> >>>> -JB- >>> >> > From Alan.Bateman at oracle.com Fri Jun 19 10:23:47 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jun 2015 11:23:47 +0100 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5583E743.1090006@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> Message-ID: <5583EDB3.9010105@oracle.com> On 19/06/2015 10:56, Jaroslav Bachorik wrote: > : > > According to the JMX spec there should be 2 attributes: 'attribute' > and 'Attribute' > However, there is only one attribute resolved by the JavaBeans > Introspector and it is 'attribute'. Furthermore, the 'getattribute' > method is chosen as the getter method, rather counter-intuitively. The JMX monitor API specifies that it works like Introspector.getBeanInfo when it can't extract the value by other means. Are there are other areas in JMX where Introspector is used? -Alan From jaroslav.bachorik at oracle.com Fri Jun 19 11:38:05 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Fri, 19 Jun 2015 13:38:05 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5583EDB3.9010105@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> Message-ID: <5583FF1D.80003@oracle.com> On 19.6.2015 12:23, Alan Bateman wrote: > > > On 19/06/2015 10:56, Jaroslav Bachorik wrote: >> : >> >> According to the JMX spec there should be 2 attributes: 'attribute' >> and 'Attribute' >> However, there is only one attribute resolved by the JavaBeans >> Introspector and it is 'attribute'. Furthermore, the 'getattribute' >> method is chosen as the getter method, rather counter-intuitively. > The JMX monitor API specifies that it works like > Introspector.getBeanInfo when it can't extract the value by other means. > Are there are other areas in JMX where Introspector is used? Both the j.b.Introspector.getReadMethod() and the SimpleIntrospector.getReadMethod() are used only from c.s.j.m.Introspector.elementFromComplex() method and only to resolve the getter for a complex attribute. So, any change in the SimpleIntrospector will not affect the rest of the JMX system. Given this statement in the Monitoring javadocs (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) "If the above rules do not produce a value, and if introspection, as if by calling Introspector.getBeanInfo, for the class of v (v.getClass()) identifies a property with the name e, then x is the result of reading the property value." I would strongly propose to adjust the SimpleIntrospector to closely follow the j.b.Introspector functionality. -JB- > > -Alan From Alan.Bateman at oracle.com Fri Jun 19 11:44:23 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jun 2015 12:44:23 +0100 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <5583FF1D.80003@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> Message-ID: <55840097.7040309@oracle.com> On 19/06/2015 12:38, Jaroslav Bachorik wrote: > > Both the j.b.Introspector.getReadMethod() and the > SimpleIntrospector.getReadMethod() are used only from > c.s.j.m.Introspector.elementFromComplex() method and only to resolve > the getter for a complex attribute. > > So, any change in the SimpleIntrospector will not affect the rest of > the JMX system. > > Given this statement in the Monitoring javadocs > (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) > > "If the above rules do not produce a value, and if introspection, as > if by calling Introspector.getBeanInfo, for the class of v > (v.getClass()) identifies a property with the name e, then x is the > result of reading the property value." > > I would strongly propose to adjust the SimpleIntrospector to closely > follow the j.b.Introspector functionality. > I added the SimpleIntrospector and also adjusted the above wording in the monitor package description to make it possible to have JMX in a compact Profile of Java SE that didn't have java.beans. The intention was that the SimpleIntrospector works like the beans Introspector. So I agree this is the right thing to do but we should create a bug for the issue that Daniel brings up. -Alan. From jaroslav.bachorik at oracle.com Fri Jun 19 12:05:07 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Fri, 19 Jun 2015 14:05:07 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55840097.7040309@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> <55840097.7040309@oracle.com> Message-ID: <55840573.7000103@oracle.com> On 19.6.2015 13:44, Alan Bateman wrote: > > On 19/06/2015 12:38, Jaroslav Bachorik wrote: >> >> Both the j.b.Introspector.getReadMethod() and the >> SimpleIntrospector.getReadMethod() are used only from >> c.s.j.m.Introspector.elementFromComplex() method and only to resolve >> the getter for a complex attribute. >> >> So, any change in the SimpleIntrospector will not affect the rest of >> the JMX system. >> >> Given this statement in the Monitoring javadocs >> (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) >> >> >> "If the above rules do not produce a value, and if introspection, as >> if by calling Introspector.getBeanInfo, for the class of v >> (v.getClass()) identifies a property with the name e, then x is the >> result of reading the property value." >> >> I would strongly propose to adjust the SimpleIntrospector to closely >> follow the j.b.Introspector functionality. >> > I added the SimpleIntrospector and also adjusted the above wording in > the monitor package description to make it possible to have JMX in a > compact Profile of Java SE that didn't have java.beans. > > The intention was that the SimpleIntrospector works like the beans > Introspector. So I agree this is the right thing to do but we should > create a bug for the issue that Daniel brings up. I'm not sure there is an issue here. The monitoring api attribute names seem to be governed by different rules than the MBean attribute names. This change relates to the monitoring api only and brings the implementation up to the what is specified in the javadoc. -JB- > > -Alan. From Alan.Bateman at oracle.com Fri Jun 19 13:17:41 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jun 2015 14:17:41 +0100 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55840573.7000103@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> <55840097.7040309@oracle.com> <55840573.7000103@oracle.com> Message-ID: <55841675.6040403@oracle.com> On 19/06/2015 13:05, Jaroslav Bachorik wrote: > I'm not sure there is an issue here. The monitoring api attribute > names seem to be governed by different rules than the MBean attribute > names. This change relates to the monitoring api only and brings the > implementation up to the what is specified in the javadoc. It sounds like it might at least require a javadoc clarification so creating an issue to track it (if you agree) would be good. In any case, I agree with getting the SimpleIntrospector aligned. -Alan From daniel.fuchs at oracle.com Fri Jun 19 13:50:15 2015 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 19 Jun 2015 15:50:15 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55840573.7000103@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> <55840097.7040309@oracle.com> <55840573.7000103@oracle.com> Message-ID: <55841E17.2070001@oracle.com> Hi guys, I believe there's a difference between Attribute names, and navigation into complex attributes. At the MBean level attribute names are case sensitive. At Attribute level (= within an attribute) I believe it follows the Bean patterns (like for instance the mapping of complex MXBean attribute types to CompositeData). A property of an Attribute (as in MemoryUsage.used) is not itself an Attribute - but a bean property. A careful reading of the JMX 1.4 spec should hopefully make it possible to validate that ;-) cheers, -- daniel On 19/06/15 14:05, Jaroslav Bachorik wrote: > On 19.6.2015 13:44, Alan Bateman wrote: >> >> On 19/06/2015 12:38, Jaroslav Bachorik wrote: >>> >>> Both the j.b.Introspector.getReadMethod() and the >>> SimpleIntrospector.getReadMethod() are used only from >>> c.s.j.m.Introspector.elementFromComplex() method and only to resolve >>> the getter for a complex attribute. >>> >>> So, any change in the SimpleIntrospector will not affect the rest of >>> the JMX system. >>> >>> Given this statement in the Monitoring javadocs >>> (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) >>> >>> >>> >>> "If the above rules do not produce a value, and if introspection, as >>> if by calling Introspector.getBeanInfo, for the class of v >>> (v.getClass()) identifies a property with the name e, then x is the >>> result of reading the property value." >>> >>> I would strongly propose to adjust the SimpleIntrospector to closely >>> follow the j.b.Introspector functionality. >>> >> I added the SimpleIntrospector and also adjusted the above wording in >> the monitor package description to make it possible to have JMX in a >> compact Profile of Java SE that didn't have java.beans. >> >> The intention was that the SimpleIntrospector works like the beans >> Introspector. So I agree this is the right thing to do but we should >> create a bug for the issue that Daniel brings up. > > I'm not sure there is an issue here. The monitoring api attribute names > seem to be governed by different rules than the MBean attribute names. > This change relates to the monitoring api only and brings the > implementation up to the what is specified in the javadoc. > > -JB- > >> >> -Alan. > From jaroslav.bachorik at oracle.com Fri Jun 19 16:20:50 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Fri, 19 Jun 2015 18:20:50 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55841E17.2070001@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> <55840097.7040309@oracle.com> <55840573.7000103@oracle.com> <55841E17.2070001@oracle.com> Message-ID: <55844162.1040108@oracle.com> On 19.6.2015 15:50, Daniel Fuchs wrote: > Hi guys, > > I believe there's a difference between Attribute names, and navigation > into complex attributes. > > At the MBean level attribute names are case sensitive. > At Attribute level (= within an attribute) I believe it follows > the Bean patterns (like for instance the mapping of complex > MXBean attribute types to CompositeData). > A property of an Attribute (as in MemoryUsage.used) is not itself > an Attribute - but a bean property. > > A careful reading of the JMX 1.4 spec should hopefully make it possible > to validate that ;-) Also, some JCK tests are relying on the Attribute properties being treated as JavaBean properties. These were, in fact, the ones triggering this issue when run without j.b.Introspector being available. So, if I understand it correctly, with my fix the implementation will be in agreement with the spec again even when j.b.Introspector is not available. -JB- > > cheers, > > -- daniel > > On 19/06/15 14:05, Jaroslav Bachorik wrote: >> On 19.6.2015 13:44, Alan Bateman wrote: >>> >>> On 19/06/2015 12:38, Jaroslav Bachorik wrote: >>>> >>>> Both the j.b.Introspector.getReadMethod() and the >>>> SimpleIntrospector.getReadMethod() are used only from >>>> c.s.j.m.Introspector.elementFromComplex() method and only to resolve >>>> the getter for a complex attribute. >>>> >>>> So, any change in the SimpleIntrospector will not affect the rest of >>>> the JMX system. >>>> >>>> Given this statement in the Monitoring javadocs >>>> (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) >>>> >>>> >>>> >>>> >>>> "If the above rules do not produce a value, and if introspection, as >>>> if by calling Introspector.getBeanInfo, for the class of v >>>> (v.getClass()) identifies a property with the name e, then x is the >>>> result of reading the property value." >>>> >>>> I would strongly propose to adjust the SimpleIntrospector to closely >>>> follow the j.b.Introspector functionality. >>>> >>> I added the SimpleIntrospector and also adjusted the above wording in >>> the monitor package description to make it possible to have JMX in a >>> compact Profile of Java SE that didn't have java.beans. >>> >>> The intention was that the SimpleIntrospector works like the beans >>> Introspector. So I agree this is the right thing to do but we should >>> create a bug for the issue that Daniel brings up. >> >> I'm not sure there is an issue here. The monitoring api attribute names >> seem to be governed by different rules than the MBean attribute names. >> This change relates to the monitoring api only and brings the >> implementation up to the what is specified in the javadoc. >> >> -JB- >> >>> >>> -Alan. >> > From ecki at zusammenkunft.net Fri Jun 19 19:42:48 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 19 Jun 2015 21:42:48 +0200 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <5583CE84.20800@oracle.com> References: <20150606190707.000044ba.ecki@zusammenkunft.net> <20150619080015.000025b1.ecki@zusammenkunft.net> <5583CE84.20800@oracle.com> Message-ID: <20150619214248.000078dc.ecki@zusammenkunft.net> Hello Alan, Am Fri, 19 Jun 2015 09:10:44 +0100 schrieb Alan Bateman : > On 19/06/2015 07:00, Bernd Eckenfels wrote: > >> - why is BufferPoolMXBean not tracking allocation failures, > >> allocation count, alignment flag and maximum size (maybe even > >> modifyable). Would it be worth to contribute something in this > >> area or is that supposed to be covered by some events (and if yes, > >> when will they be exposed to JMX). > > > When we created BufferPoolMXBean (in JSR-203) then the goal was to > make it easy to monitor current usage. It wouldn't be hard to track > peak usage if that is important. From my point of view tracking native allocations in "internal" pool (via Unsafe.allocate) is hard. (see my other discussion on instrumenting Unsafe). Making some of the users (especially DirectByteBuffers) more transparent would be a good thing. This includes peak usage but also allocation rate and especially the backoff/system-gc case. So from my side, I think it is a good thing to add some metrics here and I am willing to work on a patch. > Direct buffers have not been page > aligned for some time so that is probably not worth trying to expose. I just saw it in the code and the fact that there is still capacity vs. size, and even some mentions of page alignment in 8: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/nio/Bits.java/#638 But in my proposed patch I did not expose the alignment and agree its not needed. > The nio-dev and serviceability-dev would be the right place to being > proposals in this area. Very well, I will subscribe nio-dev and include it in the discussion. Before I do so, I have one question which is serviceability related: My patch proposes a (partial) solution which is based on the existing MBeans. I do however wonder if this is also interesting for perfcounters and/or JFR. Is there a alternative API which would be able to distribute those statistics into all/other subsystems. Especially the "out of direct memory need to wait x millis" would be a very interesting FR event to capture. Any recommendation what I should look at to understand them? Gruss Bernd From dmitry.samersoff at oracle.com Sat Jun 20 15:22:51 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Sat, 20 Jun 2015 18:22:51 +0300 Subject: RFR(S, TESTONLY): JDK-8081576 serviceability/sa tests fail due to LingeredApp process fails to start In-Reply-To: <55775901.7030203@oracle.com> References: <5576E0A9.9040105@oracle.com> <55775901.7030203@oracle.com> Message-ID: <5585854B.4000804@oracle.com> Serguei, Thank you for the review. > The comment needs a correction: Typeos fixed and webrev is regenerated (in-place, press shift-reload). > It is a little bit strange that an IOException is thrown > when the exit code does not match the expectation. LingeredApp terminates with non-zero exit code if IOException happens, so I "rethrow" it in a caller. -Dmitry On 2015-06-10 00:22, serguei.spitsyn at oracle.com wrote: > Hi Dmitry, > > It looks good, just a couple of minor comments: > > root_webrev/test/lib/share/classes/jdk/test/lib/apps/LingeredApp.java > > 324 * Delete lock file that signal app to terminate, then > 325 * waits until app is actually terminated. > 326 * @throws IOException > 327 */ > 328 public void stopApp() throws IOException { > 329 deleteLock(); > 330 waitAppTerminate(); > 331 int exitcode = appProcess.exitValue(); > 332 if (exitcode != 0) { > 333 throw new IOException("LingeredApp terminated with non-zero exit code " + exitcode); > 334 } > 335 } > > The comment needs a correction: > that signal app => that signals app > waits until => wait until > > It is a little bit strange that an IOException is thrown > when the exit code does not match the expectation. > > > Thanks, > Serguei > > > On 6/9/15 5:48 AM, Dmitry Samersoff wrote: >> Everybody, >> >> Please review a test-only fix: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8081576/webrev.01/ >> >> LingeredApp can throw an exception during initialization if it not able >> to create a lock file for some reason. This exception cause NPE later, >> when the test attempts to stop LingeredApp and original exception get lost. >> >> Fixing it by adding static stopApp(app) method with null pointer check >> inside. >> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From Alan.Bateman at oracle.com Sat Jun 20 16:12:34 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 20 Jun 2015 17:12:34 +0100 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <20150619214248.000078dc.ecki@zusammenkunft.net> References: <20150606190707.000044ba.ecki@zusammenkunft.net> <20150619080015.000025b1.ecki@zusammenkunft.net> <5583CE84.20800@oracle.com> <20150619214248.000078dc.ecki@zusammenkunft.net> Message-ID: <558590F2.3040407@oracle.com> On 19/06/2015 20:42, Bernd Eckenfels wrote: > : > From my point of view tracking native allocations in "internal" pool > (via Unsafe.allocate) is hard. (see my other discussion on > instrumenting Unsafe). Making some of the users (especially > DirectByteBuffers) more transparent would be a good thing. > > This includes peak usage but also allocation rate and especially the > backoff/system-gc case. So from my side, I think it is a good thing to > add some metrics here and I am willing to work on a patch. Tracking peak usage would be consistent with some of the other platform MXBeans so I think this make sense. Back-off + gc is implementation specific and I don't think appropriate to add to the BufferPoolMXBean. This type of tracking might be better served with jvmstat counters. JFR isn't in OpenJDK but others here might be able to say whether it could make use of the counters and updated instrumentation. -Alan. From kirk.pepperdine at gmail.com Sat Jun 20 18:28:41 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Sat, 20 Jun 2015 20:28:41 +0200 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <558590F2.3040407@oracle.com> References: <20150606190707.000044ba.ecki@zusammenkunft.net> <20150619080015.000025b1.ecki@zusammenkunft.net> <5583CE84.20800@oracle.com> <20150619214248.000078dc.ecki@zusammenkunft.net> <558590F2.3040407@oracle.com> Message-ID: <010B53D0-4F0A-46A8-A3C4-34CCC36596B0@gmail.com> > > Back-off + gc is implementation specific and I don't think appropriate to add to the BufferPoolMXBean. This type of tracking might be better served with jvmstat counters. JFR isn't in OpenJDK but others here might be able to say whether it could make use of the counters and updated instrumentation. Certainly more visibility in the area can?t be harmful. Regards, Kirk From serguei.spitsyn at oracle.com Sat Jun 20 21:07:19 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Sat, 20 Jun 2015 14:07:19 -0700 Subject: RFR(S, TESTONLY): JDK-8081576 serviceability/sa tests fail due to LingeredApp process fails to start In-Reply-To: <5585854B.4000804@oracle.com> References: <5576E0A9.9040105@oracle.com> <55775901.7030203@oracle.com> <5585854B.4000804@oracle.com> Message-ID: <5585D607.7060504@oracle.com> Dmitry, On 6/20/15 8:22 AM, Dmitry Samersoff wrote: > Serguei, > > Thank you for the review. > >> The comment needs a correction: > Typeos fixed and webrev is regenerated (in-place, press shift-reload). Ok, thanks! >> It is a little bit strange that an IOException is thrown >> when the exit code does not match the expectation. > LingeredApp terminates with non-zero exit code if IOException happens, > so I "rethrow" it in a caller. I guess, it can terminate with non-zero exit code for some other reasons. Replacing it with Exception would be Ok. Consider it reviewed. Thanks, Serguei > > -Dmitry > > > > On 2015-06-10 00:22, serguei.spitsyn at oracle.com wrote: >> Hi Dmitry, >> >> It looks good, just a couple of minor comments: >> >> root_webrev/test/lib/share/classes/jdk/test/lib/apps/LingeredApp.java >> >> 324 * Delete lock file that signal app to terminate, then >> 325 * waits until app is actually terminated. >> 326 * @throws IOException >> 327 */ >> 328 public void stopApp() throws IOException { >> 329 deleteLock(); >> 330 waitAppTerminate(); >> 331 int exitcode = appProcess.exitValue(); >> 332 if (exitcode != 0) { >> 333 throw new IOException("LingeredApp terminated with non-zero exit code " + exitcode); >> 334 } >> 335 } >> >> The comment needs a correction: >> that signal app => that signals app >> waits until => wait until >> >> It is a little bit strange that an IOException is thrown >> when the exit code does not match the expectation. >> >> >> Thanks, >> Serguei >> >> >> On 6/9/15 5:48 AM, Dmitry Samersoff wrote: >>> Everybody, >>> >>> Please review a test-only fix: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8081576/webrev.01/ >>> >>> LingeredApp can throw an exception during initialization if it not able >>> to create a lock file for some reason. This exception cause NPE later, >>> when the test attempts to stop LingeredApp and original exception get lost. >>> >>> Fixing it by adding static stopApp(app) method with null pointer check >>> inside. >>> >>> > From ecki at zusammenkunft.net Sat Jun 20 21:58:38 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sat, 20 Jun 2015 23:58:38 +0200 Subject: System.gc() causes for jcl (BufferPoolMXBean) In-Reply-To: <558590F2.3040407@oracle.com> References: <20150606190707.000044ba.ecki@zusammenkunft.net> <20150619080015.000025b1.ecki@zusammenkunft.net> <5583CE84.20800@oracle.com> <20150619214248.000078dc.ecki@zusammenkunft.net> <558590F2.3040407@oracle.com> Message-ID: <20150620235838.0000177b.ecki@zusammenkunft.net> Am Sat, 20 Jun 2015 17:12:34 +0100 schrieb Alan Bateman : > Tracking peak usage would be consistent with some of the other > platform MXBeans so I think this make sense. Yes, peak usage and count is also used in for example the ThreadsMBean. > Back-off + gc is implementation specific and I don't think > appropriate to add to the BufferPoolMXBean. This type of tracking > might be better served with jvmstat counters. Ok, I will check how easy it is to get it integrated. > JFR isn't in OpenJDK > but others here might be able to say whether it could make use of the > counters and updated instrumentation. Interesting, I would have expected that some instrumentation/reporting infrastructure is in the openjdk code base. Is this a mega patch to be integrated in the commercial offering or is this a smart instrumentation layer? Isnt there some event infrastructure in OpenJDK present? Gruss Bernd From jaroslav.bachorik at oracle.com Mon Jun 22 10:06:21 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 22 Jun 2015 12:06:21 +0200 Subject: RFR 8129215: com.sun.jmx.mbeanserver.Introspector may provide results inconsistent with the JavaBeans Introspector In-Reply-To: <55841E17.2070001@oracle.com> References: <5582F444.1090108@oracle.com> <5582F604.60909@oracle.com> <5582FCEF.3010000@oracle.com> <55833FF1.7020007@oracle.com> <5583E743.1090006@oracle.com> <5583EDB3.9010105@oracle.com> <5583FF1D.80003@oracle.com> <55840097.7040309@oracle.com> <55840573.7000103@oracle.com> <55841E17.2070001@oracle.com> Message-ID: <5587DE1D.4040402@oracle.com> On 19.6.2015 15:50, Daniel Fuchs wrote: > Hi guys, > > I believe there's a difference between Attribute names, and navigation > into complex attributes. > > At the MBean level attribute names are case sensitive. > At Attribute level (= within an attribute) I believe it follows > the Bean patterns (like for instance the mapping of complex > MXBean attribute types to CompositeData). > A property of an Attribute (as in MemoryUsage.used) is not itself > an Attribute - but a bean property. > > A careful reading of the JMX 1.4 spec should hopefully make it possible > to validate that ;-) I can't find anything about the complex data properties in the JMX 1.4 spec no matter how carefully I read. The only place that deals with how the complex data property getters are to be resolved is in the monitoring package javadoc. -JB- > > cheers, > > -- daniel > > On 19/06/15 14:05, Jaroslav Bachorik wrote: >> On 19.6.2015 13:44, Alan Bateman wrote: >>> >>> On 19/06/2015 12:38, Jaroslav Bachorik wrote: >>>> >>>> Both the j.b.Introspector.getReadMethod() and the >>>> SimpleIntrospector.getReadMethod() are used only from >>>> c.s.j.m.Introspector.elementFromComplex() method and only to resolve >>>> the getter for a complex attribute. >>>> >>>> So, any change in the SimpleIntrospector will not affect the rest of >>>> the JMX system. >>>> >>>> Given this statement in the Monitoring javadocs >>>> (https://docs.oracle.com/javase/8/docs/api/javax/management/monitor/package-summary.html) >>>> >>>> >>>> >>>> >>>> "If the above rules do not produce a value, and if introspection, as >>>> if by calling Introspector.getBeanInfo, for the class of v >>>> (v.getClass()) identifies a property with the name e, then x is the >>>> result of reading the property value." >>>> >>>> I would strongly propose to adjust the SimpleIntrospector to closely >>>> follow the j.b.Introspector functionality. >>>> >>> I added the SimpleIntrospector and also adjusted the above wording in >>> the monitor package description to make it possible to have JMX in a >>> compact Profile of Java SE that didn't have java.beans. >>> >>> The intention was that the SimpleIntrospector works like the beans >>> Introspector. So I agree this is the right thing to do but we should >>> create a bug for the issue that Daniel brings up. >> >> I'm not sure there is an issue here. The monitoring api attribute names >> seem to be governed by different rules than the MBean attribute names. >> This change relates to the monitoring api only and brings the >> implementation up to the what is specified in the javadoc. >> >> -JB- >> >>> >>> -Alan. >> > From chromiumlearner at gmail.com Mon Jun 22 10:37:57 2015 From: chromiumlearner at gmail.com (Chromium Learner) Date: Mon, 22 Jun 2015 16:07:57 +0530 Subject: Minimized StackTraceElement Message-ID: Hello, I am looking to print the minimized StackTraceElement similar to addr2line functionality for native daemons. Is there a way to print only the ByteCode of StackTraceElement at runtime and then convert to Full method name post runtime (using javap or other commands ) ? Regards, Chromium -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Mon Jun 22 15:36:21 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Mon, 22 Jun 2015 17:36:21 +0200 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out Message-ID: <55882B75.1070804@oracle.com> Please, review the following test change Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 GaugeMonitorDeadlockTest fails intermittently due data race - the call to monitorProxy.start() on L105 will eventually result in incrementing the GetCount attribute. If that happens before L109 had the chance to run the loop on L113-128 will become infinite. The initial value will contain the already incremented GetCount value and GetCount attribute value will not get further incremented. I took the liberty to fix the same issue in test/javax/management/monitor/StringMonitorDeadlockTest.java, not waiting for the real test failure. Thanks, -JB- From jeremymanson at google.com Mon Jun 22 18:48:37 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Mon, 22 Jun 2015 11:48:37 -0700 Subject: Low-Overhead Heap Profiling Message-ID: Hey folks, (cc'ing Aleksey and John, because I mentioned this to them at the JVMLS last year, but I never followed up.) We have a patch at Google I've always wanted to contribute to OpenJDK, but I always figured it would be unwanted. I've recently been thinking that might not be as true, though. I thought I would ask if there is any interest / if I should write a JEP / if I should just forget it. The basic problem is that there is no low-overhead supported way to figure out where allocation hotspots are. That is, sets of stack traces where lots of allocation / large allocations took place. What I had originally done (this was in ~2007) was use bytecode rewriting to instrument allocation sites. The instrumentation would call a Java method, and the method would capture a stack trace. To reduce overhead, there was a per-thread counter that only took a stack trace once every N bytes allocated, where N is a randomly chosen point in a probability distribution that centered around ~512K. This was *way* too slow, and it didn't pick up allocations through JNI, so I instrumented allocations at the VM level, and the overhead went away. The sampling is always turned on in our internal VMs, and a user can just query an interface for a list of sampled stack traces. The allocated stack traces are held with weak refs, so you only get live samples. The additional overhead for allocations amounts to a subtraction, and an occasional stack trace, which is usually a very, very small amount of our CPU (although I had to do some jiggering in JDK8 to fix some performance regressions). There really isn't another good way to do this with low overhead. I was wondering how the gruop would feel about our contributing it? Thoughts? Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From neugens.limasoftware at gmail.com Mon Jun 22 19:34:14 2015 From: neugens.limasoftware at gmail.com (Mario Torre) Date: Mon, 22 Jun 2015 21:34:14 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: I for one would love it. Cheers, Mario 2015-06-22 20:48 GMT+02:00 Jeremy Manson : > Hey folks, > > (cc'ing Aleksey and John, because I mentioned this to them at the JVMLS last > year, but I never followed up.) > > We have a patch at Google I've always wanted to contribute to OpenJDK, but I > always figured it would be unwanted. I've recently been thinking that might > not be as true, though. I thought I would ask if there is any interest / if > I should write a JEP / if I should just forget it. > > The basic problem is that there is no low-overhead supported way to figure > out where allocation hotspots are. That is, sets of stack traces where lots > of allocation / large allocations took place. > > What I had originally done (this was in ~2007) was use bytecode rewriting to > instrument allocation sites. The instrumentation would call a Java method, > and the method would capture a stack trace. To reduce overhead, there was a > per-thread counter that only took a stack trace once every N bytes > allocated, where N is a randomly chosen point in a probability distribution > that centered around ~512K. > > This was *way* too slow, and it didn't pick up allocations through JNI, so I > instrumented allocations at the VM level, and the overhead went away. The > sampling is always turned on in our internal VMs, and a user can just query > an interface for a list of sampled stack traces. The allocated stack traces > are held with weak refs, so you only get live samples. > > The additional overhead for allocations amounts to a subtraction, and an > occasional stack trace, which is usually a very, very small amount of our > CPU (although I had to do some jiggering in JDK8 to fix some performance > regressions). > > There really isn't another good way to do this with low overhead. I was > wondering how the gruop would feel about our contributing it? > > Thoughts? > > Jeremy -- pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens Proud GNU Classpath developer: http://www.classpath.org/ OpenJDK: http://openjdk.java.net/projects/caciocavallo/ Please, support open standards: http://endsoftpatents.org/ From chris.plummer at oracle.com Mon Jun 22 20:57:34 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 22 Jun 2015 13:57:34 -0700 Subject: [RFR] (XS) 8129386: [TESTBUG] - com/sun/jdi/cds/*.java missing @build tag for libraries Message-ID: <558876BE.7030201@oracle.com> Please review the following fix for 8129386: http://cr.openjdk.java.net/~cjplummer/8129386/webrev.00/webrev.jdk/ https://bugs.openjdk.java.net/browse/JDK-8129386 thanks, Chris From ioi.lam at oracle.com Mon Jun 22 21:27:58 2015 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 22 Jun 2015 14:27:58 -0700 Subject: [RFR] (XS) 8129386: [TESTBUG] - com/sun/jdi/cds/*.java missing @build tag for libraries In-Reply-To: <558876BE.7030201@oracle.com> References: <558876BE.7030201@oracle.com> Message-ID: <55887DDE.8000807@oracle.com> Looks good to me, but someone from SQE should revew this. Thanks - Ioi On 6/22/15 1:57 PM, Chris Plummer wrote: > Please review the following fix for 8129386: > > http://cr.openjdk.java.net/~cjplummer/8129386/webrev.00/webrev.jdk/ > https://bugs.openjdk.java.net/browse/JDK-8129386 > > thanks, > > Chris > From mikhailo.seledtsov at oracle.com Mon Jun 22 23:14:51 2015 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Mon, 22 Jun 2015 16:14:51 -0700 Subject: [RFR] (XS) 8129386: [TESTBUG] - com/sun/jdi/cds/*.java missing @build tag for libraries In-Reply-To: <55887DDE.8000807@oracle.com> References: <558876BE.7030201@oracle.com> <55887DDE.8000807@oracle.com> Message-ID: <558896EB.6090900@oracle.com> Hi Chris, Please see my comments in the original issue, (Mikhailo Seledtsov added a comment - 2015-06-22 19:12). Misha On 6/22/2015 2:27 PM, Ioi Lam wrote: > Looks good to me, but someone from SQE should revew this. > > Thanks > - Ioi > > On 6/22/15 1:57 PM, Chris Plummer wrote: >> Please review the following fix for 8129386: >> >> http://cr.openjdk.java.net/~cjplummer/8129386/webrev.00/webrev.jdk/ >> https://bugs.openjdk.java.net/browse/JDK-8129386 >> >> thanks, >> >> Chris >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailo.seledtsov at oracle.com Tue Jun 23 02:14:52 2015 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Mon, 22 Jun 2015 19:14:52 -0700 Subject: [RFR] (XS) 8129386: [TESTBUG] - com/sun/jdi/cds/*.java missing @build tag for libraries In-Reply-To: <558896EB.6090900@oracle.com> References: <558876BE.7030201@oracle.com> <55887DDE.8000807@oracle.com> <558896EB.6090900@oracle.com> Message-ID: <5588C11C.6070309@oracle.com> Hi Chris, After discussion and better understanding the problem, I agree with the changes. I have also added a new comment to the issue. Misha On 6/22/2015 4:14 PM, Mikhailo Seledtsov wrote: > Hi Chris, > > Please see my comments in the original issue, (Mikhailo Seledtsov > added > a comment - 2015-06-22 19:12). > > Misha > > On 6/22/2015 2:27 PM, Ioi Lam wrote: >> Looks good to me, but someone from SQE should revew this. >> >> Thanks >> - Ioi >> >> On 6/22/15 1:57 PM, Chris Plummer wrote: >>> Please review the following fix for 8129386: >>> >>> http://cr.openjdk.java.net/~cjplummer/8129386/webrev.00/webrev.jdk/ >>> https://bugs.openjdk.java.net/browse/JDK-8129386 >>> >>> thanks, >>> >>> Chris >>> >> > From chris.plummer at oracle.com Tue Jun 23 04:21:19 2015 From: chris.plummer at oracle.com (Chris Plummer) Date: Mon, 22 Jun 2015 21:21:19 -0700 Subject: [RFR] (XS) 8129386: [TESTBUG] - com/sun/jdi/cds/*.java missing @build tag for libraries In-Reply-To: <5588C11C.6070309@oracle.com> References: <558876BE.7030201@oracle.com> <55887DDE.8000807@oracle.com> <558896EB.6090900@oracle.com> <5588C11C.6070309@oracle.com> Message-ID: <5588DEBF.5090501@oracle.com> Thanks Misha! And also thank you to Ioi for the review. Chris On 6/22/15 7:14 PM, Mikhailo Seledtsov wrote: > Hi Chris, > > After discussion and better understanding the problem, I agree with > the changes. > I have also added a new comment to the issue. > > Misha > > On 6/22/2015 4:14 PM, Mikhailo Seledtsov wrote: >> Hi Chris, >> >> Please see my comments in the original issue, (Mikhailo Seledtsov >> >> added a comment - 2015-06-22 19:12). >> >> Misha >> >> On 6/22/2015 2:27 PM, Ioi Lam wrote: >>> Looks good to me, but someone from SQE should revew this. >>> >>> Thanks >>> - Ioi >>> >>> On 6/22/15 1:57 PM, Chris Plummer wrote: >>>> Please review the following fix for 8129386: >>>> >>>> http://cr.openjdk.java.net/~cjplummer/8129386/webrev.00/webrev.jdk/ >>>> https://bugs.openjdk.java.net/browse/JDK-8129386 >>>> >>>> thanks, >>>> >>>> Chris >>>> >>> >> > From jeremymanson at google.com Tue Jun 23 04:58:40 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Mon, 22 Jun 2015 21:58:40 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: While I'm glad to hear that our colleagues at RedHat would love it, it will still need a sponsor from Oracle. Any volunteers? It will also need a little polish to be available to the world. Open design questions for upstreaming are things like: - Should the interval between samples be configurable? - Should it *just* take a stack trace, or should the behavior be configurable? For example, we have a variant that allows it to invoke a callback on allocation. Do you want this? Because it is being called during allocation, the callback can't invoke JNI (because of the potential for a safepoint), so it might be somewhat confusing to the user. - If the answer to the above is yes, should it be able to invoke *multiple* callbacks with multiple intervals? That could get very expensive and hairy. - Other than stack trace, what kind of information should the sampled data contain? Right now, the structure is: struct StackTraceData { ASGCT_CallTrace *trace; jint byte_size; jlong thread_id; const jbyte *name; jint name_length; jlong uid; }; (where name is the class name, and uid is just a unique identifier.) For general consumption, we'll probably have to change the ASGCT_CallTrace to a jvmtiStackInfo, I suppose. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.voskresensky at oracle.com Tue Jun 23 05:31:54 2015 From: vladimir.voskresensky at oracle.com (Vladimir Voskresensky) Date: Mon, 22 Jun 2015 22:31:54 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <5588EF4A.9090803@oracle.com> Hello Jeremy, If this is sampling, not tracing, then how is it different from the low-overhead memory profiling provided by JFR [1]. JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2%. Btw, JFR also does not have false positives reported by instrumented approaches for the cases when JIT was able to eliminate heap allocation. Thanks, Vladimir. [1] http://hirt.se/blog/?p=381 On 22.06.2015 11:48, Jeremy Manson wrote: > Hey folks, > > (cc'ing Aleksey and John, because I mentioned this to them at the > JVMLS last year, but I never followed up.) > > We have a patch at Google I've always wanted to contribute to OpenJDK, > but I always figured it would be unwanted. I've recently been > thinking that might not be as true, though. I thought I would ask if > there is any interest / if I should write a JEP / if I should just > forget it. > > The basic problem is that there is no low-overhead supported way to > figure out where allocation hotspots are. That is, sets of stack > traces where lots of allocation / large allocations took place. > > What I had originally done (this was in ~2007) was use bytecode > rewriting to instrument allocation sites. The instrumentation would > call a Java method, and the method would capture a stack trace. To > reduce overhead, there was a per-thread counter that only took a stack > trace once every N bytes allocated, where N is a randomly chosen point > in a probability distribution that centered around ~512K. > > This was *way* too slow, and it didn't pick up allocations through > JNI, so I instrumented allocations at the VM level, and the overhead > went away. The sampling is always turned on in our internal VMs, and > a user can just query an interface for a list of sampled stack > traces. The allocated stack traces are held with weak refs, so you > only get live samples. > > The additional overhead for allocations amounts to a subtraction, and > an occasional stack trace, which is usually a very, very small amount > of our CPU (although I had to do some jiggering in JDK8 to fix some > performance regressions). > > There really isn't another good way to do this with low overhead. I > was wondering how the gruop would feel about our contributing it? > > Thoughts? > > Jeremy From david.holmes at oracle.com Tue Jun 23 06:04:50 2015 From: david.holmes at oracle.com (David Holmes) Date: Tue, 23 Jun 2015 16:04:50 +1000 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out In-Reply-To: <55882B75.1070804@oracle.com> References: <55882B75.1070804@oracle.com> Message-ID: <5588F702.20606@oracle.com> On 23/06/2015 1:36 AM, Jaroslav Bachorik wrote: > Please, review the following test change > > Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 > Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 > > GaugeMonitorDeadlockTest fails intermittently due data race - the call > to monitorProxy.start() on L105 will eventually result in incrementing > the GetCount attribute. If that happens before L109 had the chance to > run the loop on L113-128 will become infinite. The initial value will > contain the already incremented GetCount value and GetCount attribute > value will not get further incremented. The reordering of the start() and initial observedProxy.getGetCount seems reasonable. The changes to the timeout handling less so. The alarm code doesn't look right to me. Each time you call check(true) in the loop you advance the time when the alarm "goes off". ??? David > I took the liberty to fix the same issue in > test/javax/management/monitor/StringMonitorDeadlockTest.java, not > waiting for the real test failure. > > Thanks, > > -JB- From kirk.pepperdine at gmail.com Tue Jun 23 06:08:46 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Tue, 23 Jun 2015 08:08:46 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: <5588EF4A.9090803@oracle.com> References: <5588EF4A.9090803@oracle.com> Message-ID: <23F6D08A-27F2-41AB-BAF1-0608E2B0AAC1@gmail.com> Hi Vladimir, The difference would be in the licensing. Kind regards, Kirk Pepperdine On Jun 23, 2015, at 7:31 AM, Vladimir Voskresensky wrote: > Hello Jeremy, > > If this is sampling, not tracing, then how is it different from the low-overhead memory profiling provided by JFR [1]. > JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2%. > Btw, JFR also does not have false positives reported by instrumented approaches for the cases when JIT was able to eliminate heap allocation. > > Thanks, > Vladimir. > [1] http://hirt.se/blog/?p=381 > > On 22.06.2015 11:48, Jeremy Manson wrote: >> Hey folks, >> >> (cc'ing Aleksey and John, because I mentioned this to them at the JVMLS last year, but I never followed up.) >> >> We have a patch at Google I've always wanted to contribute to OpenJDK, but I always figured it would be unwanted. I've recently been thinking that might not be as true, though. I thought I would ask if there is any interest / if I should write a JEP / if I should just forget it. >> >> The basic problem is that there is no low-overhead supported way to figure out where allocation hotspots are. That is, sets of stack traces where lots of allocation / large allocations took place. >> >> What I had originally done (this was in ~2007) was use bytecode rewriting to instrument allocation sites. The instrumentation would call a Java method, and the method would capture a stack trace. To reduce overhead, there was a per-thread counter that only took a stack trace once every N bytes allocated, where N is a randomly chosen point in a probability distribution that centered around ~512K. >> >> This was *way* too slow, and it didn't pick up allocations through JNI, so I instrumented allocations at the VM level, and the overhead went away. The sampling is always turned on in our internal VMs, and a user can just query an interface for a list of sampled stack traces. The allocated stack traces are held with weak refs, so you only get live samples. >> >> The additional overhead for allocations amounts to a subtraction, and an occasional stack trace, which is usually a very, very small amount of our CPU (although I had to do some jiggering in JDK8 to fix some performance regressions). >> >> There really isn't another good way to do this with low overhead. I was wondering how the gruop would feel about our contributing it? >> >> Thoughts? >> >> Jeremy > From kirk.pepperdine at gmail.com Tue Jun 23 06:13:44 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Tue, 23 Jun 2015 08:13:44 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <1E34B80D-A753-4C48-AF2E-FD371F11D1B3@gmail.com> Hi Jeremy, > > It will also need a little polish to be available to the world. Open design questions for upstreaming are things like: > > - Should the interval between samples be configurable? This could be helpful. > > - Should it *just* take a stack trace, or should the behavior be configurable? For example, we have a variant that allows it to invoke a callback on allocation. Do you want this? Because it is being called during allocation, the callback can't invoke JNI (because of the potential for a safepoint), so it might be somewhat confusing to the user. As long of these features don?t contribute to sampling bias I think they?d (at times) all be useful. > > - If the answer to the above is yes, should it be able to invoke *multiple* callbacks with multiple intervals? That could get very expensive and hairy. > > - Other than stack trace, what kind of information should the sampled data contain? Right now, the structure is: > > struct StackTraceData { > ASGCT_CallTrace *trace; > jint byte_size; > jlong thread_id; > const jbyte *name; > jint name_length; > jlong uid; > }; If you could get age data that could be interesting as well. Kind regards, Kirk Pepperdine From neugens.limasoftware at gmail.com Tue Jun 23 06:46:52 2015 From: neugens.limasoftware at gmail.com (Mario Torre) Date: Tue, 23 Jun 2015 08:46:52 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: 2015-06-23 6:58 GMT+02:00 Jeremy Manson : > While I'm glad to hear that our colleagues at RedHat would love it, it will > still need a sponsor from Oracle. Any volunteers? > > It will also need a little polish to be available to the world. Open design > questions for upstreaming are things like: > > - Should the interval between samples be configurable? Yeah, this would be nice. > - Should it *just* take a stack trace, or should the behavior be > configurable? For example, we have a variant that allows it to invoke a > callback on allocation. Do you want this? Because it is being called > during allocation, the callback can't invoke JNI (because of the potential > for a safe point) , so it might be somewhat confusing to the user. Yes, a perhaps optional callbacks would be nice too. I don't think the "no JNI here" is a problem given the scope of the tool, although I can see how it could be a bit confusing or kind of limiting in some cases, but probably the benefit outweighs this issue. I can see, for example, how a callback here could provide a plug for things like system tap probes etc... > - If the answer to the above is yes, should it be able to invoke *multiple* > callbacks with multiple intervals? That could get very expensive and hairy. I don't think you need multiple callbacks, since the callback would be user code, users can call additional callbacks themselves, but it also depends on how this user code is installed. For instance, if the outside world interface is some kind of JVMTi interface, then we probably need to support multiple callbacks (as in multiple agents installing their custom code). I suspect that to keep the overhead low an higher level interface is needed to collect the allocation events, while the callback interface mechanisms would be most useful to implement/extend this interface, but not really meant to be used by default. > - Other than stack trace, what kind of information should the sampled data > contain? Right now, the structure is: > > struct StackTraceData { > ASGCT_CallTrace *trace; > jint byte_size; > jlong thread_id; > const jbyte *name; > jint name_length; > jlong uid; > }; > > (where name is the class name, and uid is just a unique identifier.) For > general consumption, we'll probably have to change the ASGCT_CallTrace to a > jvmtiStackInfo, I suppose. It looks to me that most of the interesting stuff is here. The questions that remain to me are how the outside interface would look and how it would work, we can probably take a more abstracted and high level look at it and then refine the specific step by step. Cheers, Mario -- pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens Proud GNU Classpath developer: http://www.classpath.org/ OpenJDK: http://openjdk.java.net/projects/caciocavallo/ Please, support open standards: http://endsoftpatents.org/ From jeremymanson at google.com Tue Jun 23 07:31:22 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Tue, 23 Jun 2015 00:31:22 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <5588EF4A.9090803@oracle.com> References: <5588EF4A.9090803@oracle.com> Message-ID: Hm... I missed that JFR did this, when that happened. I suppose I should pay attention to JFR, but a) I had done this many years earlier, and b) the quick skim I did of JFR when it came out indicated to me that it wasn't really doing anything interesting we don't already do, and that it would be painful to get it to work in our infrastructure. Having said that, there are a number of differences between our approaches. I think ours covers some use cases yours doesn't. One difference is that we don't sample at new TLAB allocation events or outside-of-TLAB events, we sample at points generated by a randomized probability function over an exponential distribution. I suspect that our methodology is somewhat better: because TLABs are a fixed size, new "new TLAB allocation event" strategy is likely to see sampling bias when allocations are periodic. For example, if TLABs are 512K, and my app allocates, say, a predictable 128K of objects, only the first object in that 128K will ever get sampled. I'm not qualified to do the math to figure out how bad it is (my statistics are basically rusted shut from disuse), but this sort of sampling issue is a well known problem in selecting statistical samples. The additional work to make the right thing happen isn't too hard. We just had to have the various compilers/interpreters implement a counter that decrements until you reach the next sample. When you take the sample, you pick the next sample by picking a random point in the distribution. A single decrement doesn't really make allocation more expensive. Another difference would be that our interface allows you to query just the live objects in the heap, which lets people pinpoint memory leaks really quickly. It doesn't look from the blog entry as if yours does that? Perhaps I'm misreading? Similarly, ours lets you see stack traces for sampled objects that have recently been made garbage, which allows you to pinpoint allocations that might not be necessary. We have plans to have a similar buffer for sampled objects that have become garbage since the beginning of time. Another difference would be that people can write their own code to query it. I believe that JFR is pretty tied to your UI? Or am I mistaken about that? We like to roll up the stats from a bunch of running JVMs, which you have to authenticate to connect to, store them offline, and slice and dice them in various ways. This means that we have a bunch of bespoke code to export this data. And, of course, as Kirk points out, the licensing. Does that put the kibosh on any attempt to get Oracle buyin for this? Maybe we can get buyin to close the feature gap? Naturally, our approach, being in-VM, also does not have false positives for eliminated allocations. Jeremy On Mon, Jun 22, 2015 at 10:31 PM, Vladimir Voskresensky < vladimir.voskresensky at oracle.com> wrote: > Hello Jeremy, > > If this is sampling, not tracing, then how is it different from the > low-overhead memory profiling provided by JFR [1]. > JFR samples per new TLAB allocation. It provides really very good picture > and I haven't seen overhead more than 2%. > Btw, JFR also does not have false positives reported by instrumented > approaches for the cases when JIT was able to eliminate heap allocation. > > Thanks, > Vladimir. > [1] http://hirt.se/blog/?p=381 > > > On 22.06.2015 11:48, Jeremy Manson wrote: > >> Hey folks, >> >> (cc'ing Aleksey and John, because I mentioned this to them at the JVMLS >> last year, but I never followed up.) >> >> We have a patch at Google I've always wanted to contribute to OpenJDK, >> but I always figured it would be unwanted. I've recently been thinking >> that might not be as true, though. I thought I would ask if there is any >> interest / if I should write a JEP / if I should just forget it. >> >> The basic problem is that there is no low-overhead supported way to >> figure out where allocation hotspots are. That is, sets of stack traces >> where lots of allocation / large allocations took place. >> >> What I had originally done (this was in ~2007) was use bytecode rewriting >> to instrument allocation sites. The instrumentation would call a Java >> method, and the method would capture a stack trace. To reduce overhead, >> there was a per-thread counter that only took a stack trace once every N >> bytes allocated, where N is a randomly chosen point in a probability >> distribution that centered around ~512K. >> >> This was *way* too slow, and it didn't pick up allocations through JNI, >> so I instrumented allocations at the VM level, and the overhead went away. >> The sampling is always turned on in our internal VMs, and a user can just >> query an interface for a list of sampled stack traces. The allocated stack >> traces are held with weak refs, so you only get live samples. >> >> The additional overhead for allocations amounts to a subtraction, and an >> occasional stack trace, which is usually a very, very small amount of our >> CPU (although I had to do some jiggering in JDK8 to fix some performance >> regressions). >> >> There really isn't another good way to do this with low overhead. I was >> wondering how the gruop would feel about our contributing it? >> >> Thoughts? >> >> Jeremy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chromiumlearner at gmail.com Tue Jun 23 07:32:10 2015 From: chromiumlearner at gmail.com (Chromium Learner) Date: Tue, 23 Jun 2015 13:02:10 +0530 Subject: Minimized StackTraceElement In-Reply-To: References: Message-ID: Any suggestions to achieve above requirement ? On Mon, Jun 22, 2015 at 4:07 PM, Chromium Learner wrote: > Hello, > > I am looking to print the minimized StackTraceElement similar to addr2line > functionality for native daemons. > > Is there a way to print only the ByteCode of StackTraceElement at runtime > and then convert to Full method name post runtime (using javap or other > commands ) ? > > Regards, > Chromium > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Tue Jun 23 07:33:46 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Tue, 23 Jun 2015 09:33:46 +0200 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out In-Reply-To: <5588F702.20606@oracle.com> References: <55882B75.1070804@oracle.com> <5588F702.20606@oracle.com> Message-ID: <55890BDA.8010104@oracle.com> Hi David, On 23.6.2015 08:04, David Holmes wrote: > On 23/06/2015 1:36 AM, Jaroslav Bachorik wrote: >> Please, review the following test change >> >> Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 >> Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 >> >> GaugeMonitorDeadlockTest fails intermittently due data race - the call >> to monitorProxy.start() on L105 will eventually result in incrementing >> the GetCount attribute. If that happens before L109 had the chance to >> run the loop on L113-128 will become infinite. The initial value will >> contain the already incremented GetCount value and GetCount attribute >> value will not get further incremented. > > The reordering of the start() and initial observedProxy.getGetCount > seems reasonable. > > The changes to the timeout handling less so. The alarm code doesn't look > right to me. Each time you call check(true) in the loop you advance the > time when the alarm "goes off". ??? Stupid me :( Thanks for catching this. http://cr.openjdk.java.net/~jbachorik/8071487/webrev.01 -JB- > > David > > > >> I took the liberty to fix the same issue in >> test/javax/management/monitor/StringMonitorDeadlockTest.java, not >> waiting for the real test failure. >> >> Thanks, >> >> -JB- From jeremymanson at google.com Tue Jun 23 07:39:54 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Tue, 23 Jun 2015 00:39:54 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Mario and Kirk - Random followup to the comments: Mario: Internally, we have both a callback mechanism (which is off by default) and a sampling mechanism, which is always on. We can certainly contribute both. The callback mechanism has a strict rule that you don't call JNI in it. The one thing you can do is register to throw an OOME after you return from the allocation - some folks wanted to use this to rate limit how much allocation a given request could do. Kirk: We have had a bug open for a long time for an extension that says how many GC cycles a given sample survived. That might be useful, and would actually be pretty easy to implement. I don't want to do to much work on this (or, more likely, tell one of the folks on my team to do it) until we have a way forward to get this interface into some JDK distribution at some point. Jeremy On Mon, Jun 22, 2015 at 11:46 PM, Mario Torre < neugens.limasoftware at gmail.com> wrote: > 2015-06-23 6:58 GMT+02:00 Jeremy Manson : > > While I'm glad to hear that our colleagues at RedHat would love it, it > will > > still need a sponsor from Oracle. Any volunteers? > > > > It will also need a little polish to be available to the world. Open > design > > questions for upstreaming are things like: > > > > - Should the interval between samples be configurable? > > Yeah, this would be nice. > > > - Should it *just* take a stack trace, or should the behavior be > > configurable? For example, we have a variant that allows it to invoke a > > callback on allocation. Do you want this? Because it is being called > > during allocation, the callback can't invoke JNI (because of the > potential > > for a safe point) , so it might be somewhat confusing to the user. > > Yes, a perhaps optional callbacks would be nice too. I don't think the > "no JNI here" is a problem given the scope of the tool, although I can > see how it could be a bit confusing or kind of limiting in some cases, > but probably the benefit outweighs this issue. I can see, for example, > how a callback here could provide a plug for things like system tap > probes etc... > > > - If the answer to the above is yes, should it be able to invoke > *multiple* > > callbacks with multiple intervals? That could get very expensive and > hairy. > > I don't think you need multiple callbacks, since the callback would be > user code, users can call additional callbacks themselves, but it also > depends on how this user code is installed. For instance, if the > outside world interface is some kind of JVMTi interface, then we > probably need to support multiple callbacks (as in multiple agents > installing their custom code). I suspect that to keep the overhead low > an higher level interface is needed to collect the allocation events, > while the callback interface mechanisms would be most useful to > implement/extend this interface, but not really meant to be used by > default. > > > - Other than stack trace, what kind of information should the sampled > data > > contain? Right now, the structure is: > > > > struct StackTraceData { > > ASGCT_CallTrace *trace; > > jint byte_size; > > jlong thread_id; > > const jbyte *name; > > jint name_length; > > jlong uid; > > }; > > > > (where name is the class name, and uid is just a unique identifier.) For > > general consumption, we'll probably have to change the ASGCT_CallTrace > to a > > jvmtiStackInfo, I suppose. > > It looks to me that most of the interesting stuff is here. The > questions that remain to me are how the outside interface would look > and how it would work, we can probably take a more abstracted and high > level look at it and then refine the specific step by step. > > Cheers, > Mario > > -- > pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF > Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF > > Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens > Proud GNU Classpath developer: http://www.classpath.org/ > OpenJDK: http://openjdk.java.net/projects/caciocavallo/ > > Please, support open standards: > http://endsoftpatents.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.daugherty at oracle.com Tue Jun 23 13:14:30 2015 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 23 Jun 2015 07:14:30 -0600 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <55895BB6.4080206@oracle.com> > ASGCT_CallTrace *trace; So it looks like this uses the AsyncGetTrace() infrastructure. Does your tool work on Windows and MacOS X? Dan On 6/22/15 10:58 PM, Jeremy Manson wrote: > While I'm glad to hear that our colleagues at RedHat would love it, it > will still need a sponsor from Oracle. Any volunteers? > > It will also need a little polish to be available to the world. Open > design questions for upstreaming are things like: > > - Should the interval between samples be configurable? > > - Should it *just* take a stack trace, or should the behavior be > configurable? For example, we have a variant that allows it to invoke > a callback on allocation. Do you want this? Because it is being > called during allocation, the callback can't invoke JNI (because of > the potential for a safepoint), so it might be somewhat confusing to > the user. > > - If the answer to the above is yes, should it be able to invoke > *multiple* callbacks with multiple intervals? That could get very > expensive and hairy. > > - Other than stack trace, what kind of information should the sampled > data contain? Right now, the structure is: > > struct StackTraceData { > ASGCT_CallTrace *trace; > jint byte_size; > jlong thread_id; > const jbyte *name; > jint name_length; > jlong uid; > }; > > (where name is the class name, and uid is just a unique identifier.) > For general consumption, we'll probably have to change the > ASGCT_CallTrace to a jvmtiStackInfo, I suppose. > > Jeremy From dmitry.samersoff at oracle.com Tue Jun 23 14:06:01 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 23 Jun 2015 17:06:01 +0300 Subject: RFR(S): JDK-8059038 Create new launcher for SA tools Message-ID: <558967C9.9070902@oracle.com> Hi Everybody, Please review the changes: http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.04/ I'm about to file CCC request for it but would like to get internal feedback before that. This fix is introducing native launcher jhsdb for serviceability agent. jhsdb will launch command line debugger clhsdb jhsdb jstack file core jhsdb jmap file core jhsdb jinfo file core will launch corresponding SA based utility. -Dmitry -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From jeremymanson at google.com Tue Jun 23 16:14:30 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Tue, 23 Jun 2015 09:14:30 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <55895BB6.4080206@oracle.com> References: <55895BB6.4080206@oracle.com> Message-ID: I haven't tried! Another problem with our submitting things is that we can't really test on anything other than Linux. The reason we use ASGCT is that our modified version of ASGCT gathers native frames *and* Java frames, getting a mixed mode stack trace that crosses JNI boundaries. I haven't checked on the details, but we could probably use a more standard stack trace gathering mechanism for general consumption. As I said, I think we'd have to change that to jvmtiStackInfo. Would folks like to see some proposed JVMTI interfaces for this? Jeremy On Tue, Jun 23, 2015 at 6:14 AM, Daniel D. Daugherty < daniel.daugherty at oracle.com> wrote: > > ASGCT_CallTrace *trace; > > So it looks like this uses the AsyncGetTrace() infrastructure. > Does your tool work on Windows and MacOS X? > > Dan > > > > On 6/22/15 10:58 PM, Jeremy Manson wrote: > >> While I'm glad to hear that our colleagues at RedHat would love it, it >> will still need a sponsor from Oracle. Any volunteers? >> >> It will also need a little polish to be available to the world. Open >> design questions for upstreaming are things like: >> >> - Should the interval between samples be configurable? >> >> - Should it *just* take a stack trace, or should the behavior be >> configurable? For example, we have a variant that allows it to invoke a >> callback on allocation. Do you want this? Because it is being called >> during allocation, the callback can't invoke JNI (because of the potential >> for a safepoint), so it might be somewhat confusing to the user. >> >> - If the answer to the above is yes, should it be able to invoke >> *multiple* callbacks with multiple intervals? That could get very >> expensive and hairy. >> >> - Other than stack trace, what kind of information should the sampled >> data contain? Right now, the structure is: >> >> struct StackTraceData { >> ASGCT_CallTrace *trace; >> jint byte_size; >> jlong thread_id; >> const jbyte *name; >> jint name_length; >> jlong uid; >> }; >> >> (where name is the class name, and uid is just a unique identifier.) For >> general consumption, we'll probably have to change the ASGCT_CallTrace to a >> jvmtiStackInfo, I suppose. >> >> Jeremy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Tue Jun 23 17:01:10 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Tue, 23 Jun 2015 10:01:10 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <55895BB6.4080206@oracle.com> Message-ID: If it is a blocker for this work, we can find a way to test on OS X and Windows. Jeremy On Tue, Jun 23, 2015 at 9:14 AM, Jeremy Manson wrote: > I haven't tried! Another problem with our submitting things is that we > can't really test on anything other than Linux. > > The reason we use ASGCT is that our modified version of ASGCT gathers > native frames *and* Java frames, getting a mixed mode stack trace that > crosses JNI boundaries. I haven't checked on the details, but we could > probably use a more standard stack trace gathering mechanism for general > consumption. > > As I said, I think we'd have to change that to jvmtiStackInfo. Would > folks like to see some proposed JVMTI interfaces for this? > > Jeremy > > On Tue, Jun 23, 2015 at 6:14 AM, Daniel D. Daugherty < > daniel.daugherty at oracle.com> wrote: > >> > ASGCT_CallTrace *trace; >> >> So it looks like this uses the AsyncGetTrace() infrastructure. >> Does your tool work on Windows and MacOS X? >> >> Dan >> >> >> >> On 6/22/15 10:58 PM, Jeremy Manson wrote: >> >>> While I'm glad to hear that our colleagues at RedHat would love it, it >>> will still need a sponsor from Oracle. Any volunteers? >>> >>> It will also need a little polish to be available to the world. Open >>> design questions for upstreaming are things like: >>> >>> - Should the interval between samples be configurable? >>> >>> - Should it *just* take a stack trace, or should the behavior be >>> configurable? For example, we have a variant that allows it to invoke a >>> callback on allocation. Do you want this? Because it is being called >>> during allocation, the callback can't invoke JNI (because of the potential >>> for a safepoint), so it might be somewhat confusing to the user. >>> >>> - If the answer to the above is yes, should it be able to invoke >>> *multiple* callbacks with multiple intervals? That could get very >>> expensive and hairy. >>> >>> - Other than stack trace, what kind of information should the sampled >>> data contain? Right now, the structure is: >>> >>> struct StackTraceData { >>> ASGCT_CallTrace *trace; >>> jint byte_size; >>> jlong thread_id; >>> const jbyte *name; >>> jint name_length; >>> jlong uid; >>> }; >>> >>> (where name is the class name, and uid is just a unique identifier.) >>> For general consumption, we'll probably have to change the ASGCT_CallTrace >>> to a jvmtiStackInfo, I suppose. >>> >>> Jeremy >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neugens.limasoftware at gmail.com Tue Jun 23 17:22:49 2015 From: neugens.limasoftware at gmail.com (Mario Torre) Date: Tue, 23 Jun 2015 19:22:49 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <55895BB6.4080206@oracle.com> Message-ID: 2015-06-23 19:01 GMT+02:00 Jeremy Manson : > If it is a blocker for this work, we can find a way to test on OS X and > Windows. Yeah, I think if this is going to be a general interface, we need to test on all the official platforms. Cheers, Mario > Jeremy > > On Tue, Jun 23, 2015 at 9:14 AM, Jeremy Manson > wrote: >> >> I haven't tried! Another problem with our submitting things is that we >> can't really test on anything other than Linux. >> >> The reason we use ASGCT is that our modified version of ASGCT gathers >> native frames *and* Java frames, getting a mixed mode stack trace that >> crosses JNI boundaries. I haven't checked on the details, but we could >> probably use a more standard stack trace gathering mechanism for general >> consumption. >> >> As I said, I think we'd have to change that to jvmtiStackInfo. Would >> folks like to see some proposed JVMTI interfaces for this? >> >> Jeremy >> >> On Tue, Jun 23, 2015 at 6:14 AM, Daniel D. Daugherty >> wrote: >>> >>> > ASGCT_CallTrace *trace; >>> >>> So it looks like this uses the AsyncGetTrace() infrastructure. >>> Does your tool work on Windows and MacOS X? >>> >>> Dan >>> >>> >>> >>> On 6/22/15 10:58 PM, Jeremy Manson wrote: >>>> >>>> While I'm glad to hear that our colleagues at RedHat would love it, it >>>> will still need a sponsor from Oracle. Any volunteers? >>>> >>>> It will also need a little polish to be available to the world. Open >>>> design questions for upstreaming are things like: >>>> >>>> - Should the interval between samples be configurable? >>>> >>>> - Should it *just* take a stack trace, or should the behavior be >>>> configurable? For example, we have a variant that allows it to invoke a >>>> callback on allocation. Do you want this? Because it is being called during >>>> allocation, the callback can't invoke JNI (because of the potential for a >>>> safepoint), so it might be somewhat confusing to the user. >>>> >>>> - If the answer to the above is yes, should it be able to invoke >>>> *multiple* callbacks with multiple intervals? That could get very expensive >>>> and hairy. >>>> >>>> - Other than stack trace, what kind of information should the sampled >>>> data contain? Right now, the structure is: >>>> >>>> struct StackTraceData { >>>> ASGCT_CallTrace *trace; >>>> jint byte_size; >>>> jlong thread_id; >>>> const jbyte *name; >>>> jint name_length; >>>> jlong uid; >>>> }; >>>> >>>> (where name is the class name, and uid is just a unique identifier.) >>>> For general consumption, we'll probably have to change the ASGCT_CallTrace >>>> to a jvmtiStackInfo, I suppose. >>>> >>>> Jeremy >>> >>> >> > -- pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens Proud GNU Classpath developer: http://www.classpath.org/ OpenJDK: http://openjdk.java.net/projects/caciocavallo/ Please, support open standards: http://endsoftpatents.org/ From tprintezis at twitter.com Tue Jun 23 20:06:57 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Tue, 23 Jun 2015 16:06:57 -0400 Subject: Low-Overhead Heap Profiling Message-ID: Jeremy (and all), I?m not on the serviceability list so I won?t include the messages so far. :-) Also CCing the hotspot GC list, in case they have some feedback on this. Could I suggest a (much) simpler but at least as powerful and flexible way to do this? (This is something we?ve been meaning to do for a while now for TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force allocations to go into the slow path periodically by artificially setting the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to (bottom+1M). When an allocation thinks the TLAB is full (in this case, the first 1MB is full) it will call the allocation slow path. There, you can intercept it, sample the allocation (and, like in your case, you?ll also have the correct stack trace), notice that the TLAB is not actually full, extend its to top to, say, (bottom+2M), and you?re done. Advantages of this approach: * This is a much smaller, simpler, and self-contained change (no compiler changes necessary to maintain...). * When it?s off, the overhead is only one extra test at the slow path TLAB allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK using a similar mechanism and, when it?s off, I?ve observed no performance overhead). * (most importantly) You can turn this on and off, and adjust the sampling rate, dynamically. If you do the sampling based on JITed code, you?ll have to recompile all methods with allocation sites to turn the sampling on or off. (You can of course have it always on and just discard the output; it?d be nice not to have to do that though. IMHO, at least.) * You can also very cheaply turn this on and off (or adjust the sampling frequncy) per thread, if that?s be helpful in some way (just add the appropriate info on the thread?s TLAB). A few extra comments on the previous discussion: * "JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2? : When TLABs get very large, I don?t think sampling one object per TLAB is enough to get a good sample (IMHO, at least). It?s probably OK for something like jbb which mostly allocates instances of a handful of classes and has very few allocation sites. But, a lot of the code we run at Twitter is a lot more elaborate than that and, in our experience, sampling one object per TLAB is not enough. You can, of course, decrease the TLAB size to increase the sampling size. But it?d be good not to have to do that given a smaller TLAB size could increase contention across threads. * "Should it *just* take a stack trace, or should the behavior be configurable?? : I think we?d have to separate the allocation sampling mechanism from the consumption of the allocation samples. Once the sampling mechanism is in, different JVMs can take advantage of it in different ways. I assume that the Oracle folks would like at least a JFR event for every such sample. But in your build you can add extra code to collect the information in the way you have now. * Talking of JFR, it?s a bit unfortunate that the AllocObjectInNewTLAB event has both the new TLAB information and the allocation information. It would have been nice if that event was split into two, say NewTLAB and AllocObjectInTLAB, and we?d be able to fire the latter for each sample. * "Should the interval between samples be configurable?? : Totally. In fact, it?d be helpful if it was configurable dynamically. Imagine if a JVM starts misbehaving after 2-3 weeks of running. You can dynamically increase the sampling rate to get a better profile if the default is not giving fine-grain enough information. * "As long of these features don?t contribute to sampling bias? : If the sampling interval is fixed, sampling bias would be a very real concern. In the above example, I?d increment top by 1M (the sampling frequency) + p% (a fudge factor).? * "Yes, a perhaps optional callbacks would be nice too.? : Oh, no. :-) But, as I said, we should definitely separate the sampling mechanism from the mechanism that consumes the samples. * "Another problem with our submitting things is that we can't really test on anything other than Linux.? : Another reason to go with a as platform independent solution as possible. :-) Regards, Tony ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Tue Jun 23 23:21:51 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Tue, 23 Jun 2015 16:21:51 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate. There is no reason to do that. I want reasonable statistical sampling of the allocations. All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0. Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free. Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7, and no one has complained. I'm not worried about the ease of implementation here, because we've already implemented it. It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. We can also turn the sampling off, if we want. We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice. In fact, we could just have the sampling code do nothing, and no one would ever notice. Honestly, no one ever notices the overhead of the sampling, anyway. JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. However, you can do pretty much anything from the VM itself. Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't. We can't do this in a callback, if the callback can't create weak refs to the object. What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live. We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). Would people be amenable to that? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB object come from?"). Jeremy On Tue, Jun 23, 2015 at 1:06 PM, Tony Printezis wrote: > Jeremy (and all), > > I?m not on the serviceability list so I won?t include the messages so far. > :-) Also CCing the hotspot GC list, in case they have some feedback on this. > > Could I suggest a (much) simpler but at least as powerful and flexible way > to do this? (This is something we?ve been meaning to do for a while now for > TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force > allocations to go into the slow path periodically by artificially setting > the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to > (bottom+1M). When an allocation thinks the TLAB is full (in this case, the > first 1MB is full) it will call the allocation slow path. There, you can > intercept it, sample the allocation (and, like in your case, you?ll also > have the correct stack trace), notice that the TLAB is not actually full, > extend its to top to, say, (bottom+2M), and you?re done. > > Advantages of this approach: > > * This is a much smaller, simpler, and self-contained change (no compiler > changes necessary to maintain...). > > * When it?s off, the overhead is only one extra test at the slow path TLAB > allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK > using a similar mechanism and, when it?s off, I?ve observed no performance > overhead). > > * (most importantly) You can turn this on and off, and adjust the sampling > rate, dynamically. If you do the sampling based on JITed code, you?ll have > to recompile all methods with allocation sites to turn the sampling on or > off. (You can of course have it always on and just discard the output; it?d > be nice not to have to do that though. IMHO, at least.) > > * You can also very cheaply turn this on and off (or adjust the sampling > frequncy) per thread, if that?s be helpful in some way (just add the > appropriate info on the thread?s TLAB). > > A few extra comments on the previous discussion: > > * "JFR samples per new TLAB allocation. It provides really very good > picture and I haven't seen overhead more than 2? : When TLABs get very > large, I don?t think sampling one object per TLAB is enough to get a good > sample (IMHO, at least). It?s probably OK for something like jbb which > mostly allocates instances of a handful of classes and has very few > allocation sites. But, a lot of the code we run at Twitter is a lot more > elaborate than that and, in our experience, sampling one object per TLAB is > not enough. You can, of course, decrease the TLAB size to increase the > sampling size. But it?d be good not to have to do that given a smaller TLAB > size could increase contention across threads. > > * "Should it *just* take a stack trace, or should the behavior be > configurable?? : I think we?d have to separate the allocation sampling > mechanism from the consumption of the allocation samples. Once the sampling > mechanism is in, different JVMs can take advantage of it in different ways. > I assume that the Oracle folks would like at least a JFR event for every > such sample. But in your build you can add extra code to collect the > information in the way you have now. > > * Talking of JFR, it?s a bit unfortunate that the AllocObjectInNewTLAB > event has both the new TLAB information and the allocation information. It > would have been nice if that event was split into two, say NewTLAB and > AllocObjectInTLAB, and we?d be able to fire the latter for each sample. > > * "Should the interval between samples be configurable?? : Totally. In > fact, it?d be helpful if it was configurable dynamically. Imagine if a JVM > starts misbehaving after 2-3 weeks of running. You can dynamically increase > the sampling rate to get a better profile if the default is not giving > fine-grain enough information. > > * "As long of these features don?t contribute to sampling bias? : If the > sampling interval is fixed, sampling bias would be a very real concern. In > the above example, I?d increment top by 1M (the sampling frequency) + p% (a > fudge factor). > > * "Yes, a perhaps optional callbacks would be nice too.? : Oh, no. :-) > But, as I said, we should definitely separate the sampling mechanism from > the mechanism that consumes the samples. > > * "Another problem with our submitting things is that we can't really test > on anything other than Linux.? : Another reason to go with a as platform > independent solution as possible. :-) > > Regards, > > Tony > > ----- > > Tony Printezis | JVM/GC Engineer / VM Team | Twitter > > @TonyPrintezis > tprintezis at twitter.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Wed Jun 24 05:32:15 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 23 Jun 2015 22:32:15 -0700 Subject: RFR(S): JDK-8059038 Create new launcher for SA tools In-Reply-To: <558967C9.9070902@oracle.com> References: <558967C9.9070902@oracle.com> Message-ID: <558A40DF.6040201@oracle.com> Hi Dmitry, Some quick minor comments. hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SAGetopt.java Formatting is wrong: 57 if (_optind >_argv.length) { 71 String[] ca = carg.split("=",2); 80 if (los.contains(ca[0]+"=")) { Need to use camel case for java method names: 55 private void extract_optarg(String opt) { 69 private String process_long_options(String carg, String[] longOptStr) { Need to use quotes for '-': 109 // End of option batch like -abc reached, expect option to start from - Example is: 133 // At this point carg[0] contains '-' Wrong indent at 87, 139, 120-121: 85 else { 86 // Mixed style options --file name 87 extract_optarg(ca[0]); 88 } 138 else { 139 ch = carg.charAt(_optopt); 140 } 119 if (longOptStr == null || longOptStr.length == 0) { 120 // No long options specified, stop options processing 121 return null; 122 } hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SALauncher.java Uninitialized local: 128 String s; Need to use camel case: 126 String exe_or_pid = null; The main method is too long, I'd suggest to split with the sub-methods for: clhsdb, hsdb, jstack, jmap, jinfo jdk_webrev/test/sun/tools/jhsdb/BasicLauncherTest.java Wrong indent at 82, 85: 80 return toolProcess.exitValue(); 81 } finally { 82 theApp.stopApp(); 83 } 84 } catch (IOException | InterruptedException ex) { 85 throw new RuntimeException("Test ERROR " + ex, ex); 86 } I do not understand what is the need for nested try statements, just one try would be enough: 54 System.out.println("Starting LingeredApp"); 55 try { 56 try { . . . 81 } finally { 82 theApp.stopApp(); 83 } 84 } catch (IOException | InterruptedException ex) { 85 throw new RuntimeException("Test ERROR " + ex, ex); 86 } 98 try { 99 try { . . . 116 } finally { 117 theApp.stopApp(); 118 } 119 } catch (Exception ex) { 120 throw new RuntimeException("Test ERROR " + ex, ex); 121 } Why do you catch exceptions and throw the RuntimeException's in the launch() methods but catch the IOException in main? Would it be better to catch any Exception? Too many empty lines: 88 89 90 jdk_webrev/test/sun/tools/jhsdb/LingeredApp.java Too many empty lines: 275 276 369 jdk_webrev/test/sun/tools/jhsdb/SAGetoptTest.java Need to use Java naming convention: 36 private static boolean a_opt; 37 private static boolean b_opt; 38 private static boolean c_opt; 39 private static boolean e_opt; 40 private static boolean mixed_opt; 41 42 private static String d_value; 43 private static String exe_value; 44 private static String core_value; Wrong indent 2 instead of 4: 70 if (s.equals("a")) { 71 a_opt = true; 72 continue; 73 } 74 75 if (s.equals("b")) { 76 b_opt = true; 77 continue; 78 } 79 80 if (s.equals("c")) { 81 c_opt = true; 82 continue; 83 } 84 85 if (s.equals("e")) { 86 e_opt = true; 87 continue; 88 } 89 90 if (s.equals("mixed")) { 91 mixed_opt = true; 92 continue; 93 } Thanks, Serguei On 6/23/15 7:06 AM, Dmitry Samersoff wrote: > Hi Everybody, > > Please review the changes: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.04/ > > I'm about to file CCC request for it but would like to get internal > feedback before that. > > This fix is introducing native launcher jhsdb for serviceability agent. > > jhsdb > > will launch command line debugger clhsdb > > jhsdb jstack file core > jhsdb jmap file core > jhsdb jinfo file core > > will launch corresponding SA based utility. > > > -Dmitry > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.samersoff at oracle.com Wed Jun 24 09:17:31 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 24 Jun 2015 12:17:31 +0300 Subject: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <557AF3A2.4010508@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> <557AF3A2.4010508@redhat.com> Message-ID: <558A75AB.904@oracle.com> Andrew, I'm ready to push your changes. Do you have an official jdk9 reviewer sign off? -Dmitry On 2015-06-12 17:58, Andrew Haley wrote: > On 06/12/2015 02:12 PM, Dmitry Samersoff wrote: >> Andrew, >> >> Found minor nit in the fix, please update the webrev: >> >> diff -r 640718253035 -r f043d189d1e1 >> agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java >> --- >> a/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java >> Tue May 19 15:37:29 2015 +0100 >> +++ >> b/agent/src/share/classes/sun/jvm/hotspot/runtime/aarch64/AARCH64Frame.java >> Thu Jun 11 16:54:41 2015 +0300 >> @@ -391,7 +391,7 @@ >> map.setIncludeArgumentOops(cb.callerMustGCArguments()); >> >> if (cb.getOopMaps() != null) { >> - OopMapSet.updateRegisterMap(this, cb, map, true); >> + ImmutableOopMapSet.updateRegisterMap(this, cb, map, true); >> } >> >> // Since the prolog does the save and restore of FP there is no >> oopmap > > > http://cr.openjdk.java.net/~aph/8078521-6/ > > Andrew. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From david.holmes at oracle.com Wed Jun 24 09:22:27 2015 From: david.holmes at oracle.com (David Holmes) Date: Wed, 24 Jun 2015 19:22:27 +1000 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out In-Reply-To: <55890BDA.8010104@oracle.com> References: <55882B75.1070804@oracle.com> <5588F702.20606@oracle.com> <55890BDA.8010104@oracle.com> Message-ID: <558A76D3.2010902@oracle.com> On 23/06/2015 5:33 PM, Jaroslav Bachorik wrote: > Hi David, > > On 23.6.2015 08:04, David Holmes wrote: >> On 23/06/2015 1:36 AM, Jaroslav Bachorik wrote: >>> Please, review the following test change >>> >>> Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 >>> Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 >>> >>> GaugeMonitorDeadlockTest fails intermittently due data race - the call >>> to monitorProxy.start() on L105 will eventually result in incrementing >>> the GetCount attribute. If that happens before L109 had the chance to >>> run the loop on L113-128 will become infinite. The initial value will >>> contain the already incremented GetCount value and GetCount attribute >>> value will not get further incremented. >> >> The reordering of the start() and initial observedProxy.getGetCount >> seems reasonable. >> >> The changes to the timeout handling less so. The alarm code doesn't look >> right to me. Each time you call check(true) in the loop you advance the >> time when the alarm "goes off". ??? > > Stupid me :( Thanks for catching this. > > http://cr.openjdk.java.net/~jbachorik/8071487/webrev.01 That looks better, but the Alarm class would benefit from some simple descriptive comments. I must admit I don't really see the need for introducing it. Thanks, David > -JB- > >> >> David >> >> >> >>> I took the liberty to fix the same issue in >>> test/javax/management/monitor/StringMonitorDeadlockTest.java, not >>> waiting for the real test failure. >>> >>> Thanks, >>> >>> -JB- > From aph at redhat.com Wed Jun 24 09:24:24 2015 From: aph at redhat.com (Andrew Haley) Date: Wed, 24 Jun 2015 10:24:24 +0100 Subject: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <558A75AB.904@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> <557AF3A2.4010508@redhat.com> <558A75AB.904@oracle.com> Message-ID: <558A7748.301@redhat.com> On 24/06/15 10:17, Dmitry Samersoff wrote: > Do you have an official jdk9 reviewer sign off? Not that I've seen. I will send this to JDK9-dev. JDK9-dev people: we've been through this on the serviceability-dev list. http://cr.openjdk.java.net/~aph/8078521-6/ Andrew. From dmitry.samersoff at oracle.com Wed Jun 24 09:26:58 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 24 Jun 2015 12:26:58 +0300 Subject: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <558A7748.301@redhat.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> <557AF3A2.4010508@redhat.com> <558A75AB.904@oracle.com> <558A7748.301@redhat.com> Message-ID: <558A77E2.2000601@oracle.com> David, Could you take a look? -Dmitry On 2015-06-24 12:24, Andrew Haley wrote: > On 24/06/15 10:17, Dmitry Samersoff wrote: >> Do you have an official jdk9 reviewer sign off? > > Not that I've seen. I will send this to JDK9-dev. > > JDK9-dev people: we've been through this on the serviceability-dev > list. > > http://cr.openjdk.java.net/~aph/8078521-6/ > > Andrew. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From david.holmes at oracle.com Wed Jun 24 09:36:58 2015 From: david.holmes at oracle.com (David Holmes) Date: Wed, 24 Jun 2015 19:36:58 +1000 Subject: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <558A77E2.2000601@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> <557AF3A2.4010508@redhat.com> <558A75AB.904@oracle.com> <558A7748.301@redhat.com> <558A77E2.2000601@oracle.com> Message-ID: <558A7A3A.2020704@oracle.com> On 24/06/2015 7:26 PM, Dmitry Samersoff wrote: > David, > > Could you take a look? Based on the serviceability review and not seeing anything blatantly more broken than it previously was (and obviously less broken for aarch64 :) ) this looks ok. Though I have one query about the new "agent/make/filelist" - is it really meant to be there ?? If not please don't commit it. If so, then okay. :) Reviewed. David > -Dmitry > > On 2015-06-24 12:24, Andrew Haley wrote: >> On 24/06/15 10:17, Dmitry Samersoff wrote: >>> Do you have an official jdk9 reviewer sign off? >> >> Not that I've seen. I will send this to JDK9-dev. >> >> JDK9-dev people: we've been through this on the serviceability-dev >> list. >> >> http://cr.openjdk.java.net/~aph/8078521-6/ >> >> Andrew. >> > > From dmitry.samersoff at oracle.com Wed Jun 24 12:37:18 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 24 Jun 2015 15:37:18 +0300 Subject: RFR(S): JDK-8059038 Create new launcher for SA tools In-Reply-To: <558A40DF.6040201@oracle.com> References: <558967C9.9070902@oracle.com> <558A40DF.6040201@oracle.com> Message-ID: <558AA47E.3090902@oracle.com> Serguei, Thank you for the review. New webrev is here: http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.05/ I didn't change naming convention in SAGetoptTest.java and keep a_opt, b_opt etc as it gives better readability. Other concerns are addressed. BasicLauncherTest changed to use LingeredApp from testlib. -Dmitry On 2015-06-24 08:32, serguei.spitsyn at oracle.com wrote: > Hi Dmitry, > > Some quick minor comments. > > hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SAGetopt.java > > Formatting is wrong: > > 57 if (_optind >_argv.length) { > > 71 String[] ca = carg.split("=",2); > > 80 if (los.contains(ca[0]+"=")) { > > > Need to use camel case for java method names: > > 55 private void extract_optarg(String opt) { > > 69 private String process_long_options(String carg, String[] longOptStr) { > > > Need to use quotes for '-': > > 109 // End of option batch like -abc reached, expect option to start from - > > Example is: > > 133 // At this point carg[0] contains '-' > > > Wrong indent at 87, 139, 120-121: > > 85 else { > 86 // Mixed style options --file name > 87 extract_optarg(ca[0]); > 88 } > > 138 else { > 139 ch = carg.charAt(_optopt); > 140 } > > 119 if (longOptStr == null || longOptStr.length == 0) { > 120 // No long options specified, stop options processing > 121 return null; > 122 } > > > > hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SALauncher.java > > Uninitialized local: > > 128 String s; > > Need to use camel case: > > 126 String exe_or_pid = null; > > > The main method is too long, I'd suggest to split with the sub-methods for: > clhsdb, hsdb, jstack, jmap, jinfo > > > jdk_webrev/test/sun/tools/jhsdb/BasicLauncherTest.java > > Wrong indent at 82, 85: > > 80 return toolProcess.exitValue(); > 81 } finally { > 82 theApp.stopApp(); > 83 } > 84 } catch (IOException | InterruptedException ex) { > 85 throw new RuntimeException("Test ERROR " + ex, ex); > 86 } > > > I do not understand what is the need for nested try statements, just one > try would be enough: > > 54 System.out.println("Starting LingeredApp"); > 55 try { > 56 try { > . . . > 81 } finally { > 82 theApp.stopApp(); > 83 } > 84 } catch (IOException | InterruptedException ex) { > 85 throw new RuntimeException("Test ERROR " + ex, ex); > 86 } > > 98 try { > 99 try { > . . . > 116 } finally { > 117 theApp.stopApp(); > 118 } > 119 } catch (Exception ex) { > 120 throw new RuntimeException("Test ERROR " + ex, ex); > 121 } > > > Why do you catch exceptions and throw the RuntimeException's in the > launch() methods > but catch the IOException in main? Would it be better to catch any > Exception? > > Too many empty lines: > > 88 > 89 > 90 > > > jdk_webrev/test/sun/tools/jhsdb/LingeredApp.java > > Too many empty lines: > > 275 > 276 > > 369 > > > jdk_webrev/test/sun/tools/jhsdb/SAGetoptTest.java > > Need to use Java naming convention: > > 36 private static boolean a_opt; > 37 private static boolean b_opt; > 38 private static boolean c_opt; > 39 private static boolean e_opt; > 40 private static boolean mixed_opt; > 41 > 42 private static String d_value; > 43 private static String exe_value; > 44 private static String core_value; > > Wrong indent 2 instead of 4: > > 70 if (s.equals("a")) { > 71 a_opt = true; > 72 continue; > 73 } > 74 > 75 if (s.equals("b")) { > 76 b_opt = true; > 77 continue; > 78 } > 79 > 80 if (s.equals("c")) { > 81 c_opt = true; > 82 continue; > 83 } > 84 > 85 if (s.equals("e")) { > 86 e_opt = true; > 87 continue; > 88 } > 89 > 90 if (s.equals("mixed")) { > 91 mixed_opt = true; > 92 continue; > 93 } > > > Thanks, > Serguei > > > On 6/23/15 7:06 AM, Dmitry Samersoff wrote: >> Hi Everybody, >> >> Please review the changes: >> >> http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.04/ >> >> I'm about to file CCC request for it but would like to get internal >> feedback before that. >> >> This fix is introducing native launcher jhsdb for serviceability agent. >> >> jhsdb >> >> will launch command line debugger clhsdb >> >> jhsdb jstack file core >> jhsdb jmap file core >> jhsdb jinfo file core >> >> will launch corresponding SA based utility. >> >> >> -Dmitry >> >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From dmitry.samersoff at oracle.com Wed Jun 24 16:37:57 2015 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 24 Jun 2015 19:37:57 +0300 Subject: RFR: 8078521: AARCH64: Add AArch64 SA support In-Reply-To: <558A7A3A.2020704@oracle.com> References: <55391C34.3070502@redhat.com> <553954D8.2070506@oracle.com> <553A012C.1070008@redhat.com> <553A8B20.9050109@oracle.com> <55408B69.8050108@redhat.com> <554A26D5.4040003@redhat.com> <554B9381.8090907@oracle.com> <554B9617.2090406@redhat.com> <555232C8.4000704@redhat.com> <55577859.5080406@oracle.com> <555B54AE.9050100@redhat.com> <557ADACB.4040206@oracle.com> <557AF3A2.4010508@redhat.com> <558A75AB.904@oracle.com> <558A7748.301@redhat.com> <558A77E2.2000601@oracle.com> <558A7A3A.2020704@oracle.com> Message-ID: <558ADCE5.6020306@oracle.com> Andrew, The fix is pushed. Please take a look at the changeset for sanity. http://hg.openjdk.java.net/jdk9/hs-rt/hotspot/rev/0032abb6e693 -Dmitry On 2015-06-24 12:36, David Holmes wrote: > On 24/06/2015 7:26 PM, Dmitry Samersoff wrote: >> David, >> >> Could you take a look? > > Based on the serviceability review and not seeing anything blatantly > more broken than it previously was (and obviously less broken for > aarch64 :) ) this looks ok. > > Though I have one query about the new "agent/make/filelist" - is it > really meant to be there ?? If not please don't commit it. If so, then > okay. :) > > Reviewed. > > David > >> -Dmitry >> >> On 2015-06-24 12:24, Andrew Haley wrote: >>> On 24/06/15 10:17, Dmitry Samersoff wrote: >>>> Do you have an official jdk9 reviewer sign off? >>> >>> Not that I've seen. I will send this to JDK9-dev. >>> >>> JDK9-dev people: we've been through this on the serviceability-dev >>> list. >>> >>> http://cr.openjdk.java.net/~aph/8078521-6/ >>> >>> Andrew. >>> >> >> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From tprintezis at twitter.com Wed Jun 24 17:57:41 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Wed, 24 Jun 2015 13:57:41 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Hi Jeremy, Please see inline. On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote: I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate.? There is no reason to do that.? I want reasonable statistical sampling of the allocations. ? As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.) All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0.? Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free.?? Maybe on intel is cheap, but maybe it?s not on other platforms that other folks care about. Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7,? Interesting. I hadn?t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it?s the former, why isn?t it possible to calculate that from the TLABs information? and no one has complained. I'm not worried about the ease of implementation here, because we've already implemented it.?? Yeah, but someone will have to maintain it moving forward. It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. We can also turn the sampling off, if we want.? We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice.?? You still have extra instructions in the allocation path, so it?s not turned off (i.e., you have the tax without any benefit). In fact, we could just have the sampling code do nothing, and no one would ever notice. Honestly, no one ever notices the overhead of the sampling, anyway.? JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code? ? However, you can do pretty much anything from the VM itself.? Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't.? We can't do this in a callback, if the callback can't create weak refs to the object. What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live.? We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). ? Would people be amenable to that?? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey!? Where did that 1 GB object come from?"). Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the ?outside-of-TLAB allocation path? (just saying?). But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). Tony Jeremy On Tue, Jun 23, 2015 at 1:06 PM, Tony Printezis??wrote: Jeremy (and all), I?m not on the serviceability list so I won?t include the messages so far. :-) Also CCing the hotspot GC list, in case they have some feedback on this. Could I suggest a (much) simpler but at least as powerful and flexible way to do this? (This is something we?ve been meaning to do for a while now for TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force allocations to go into the slow path periodically by artificially setting the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to (bottom+1M). When an allocation thinks the TLAB is full (in this case, the first 1MB is full) it will call the allocation slow path. There, you can intercept it, sample the allocation (and, like in your case, you?ll also have the correct stack trace), notice that the TLAB is not actually full, extend its to top to, say, (bottom+2M), and you?re done. Advantages of this approach: * This is a much smaller, simpler, and self-contained change (no compiler changes necessary to maintain...). * When it?s off, the overhead is only one extra test at the slow path TLAB allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK using a similar mechanism and, when it?s off, I?ve observed no performance overhead). * (most importantly) You can turn this on and off, and adjust the sampling rate, dynamically. If you do the sampling based on JITed code, you?ll have to recompile all methods with allocation sites to turn the sampling on or off. (You can of course have it always on and just discard the output; it?d be nice not to have to do that though. IMHO, at least.) * You can also very cheaply turn this on and off (or adjust the sampling frequncy) per thread, if that?s be helpful in some way (just add the appropriate info on the thread?s TLAB). A few extra comments on the previous discussion: * "JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2? : When TLABs get very large, I don?t think sampling one object per TLAB is enough to get a good sample (IMHO, at least). It?s probably OK for something like jbb which mostly allocates instances of a handful of classes and has very few allocation sites. But, a lot of the code we run at Twitter is a lot more elaborate than that and, in our experience, sampling one object per TLAB is not enough. You can, of course, decrease the TLAB size to increase the sampling size. But it?d be good not to have to do that given a smaller TLAB size could increase contention across threads. * "Should it *just* take a stack trace, or should the behavior be configurable?? : I think we?d have to separate the allocation sampling mechanism from the consumption of the allocation samples. Once the sampling mechanism is in, different JVMs can take advantage of it in different ways. I assume that the Oracle folks would like at least a JFR event for every such sample. But in your build you can add extra code to collect the information in the way you have now. * Talking of JFR, it?s a bit unfortunate that the AllocObjectInNewTLAB event has both the new TLAB information and the allocation information. It would have been nice if that event was split into two, say NewTLAB and AllocObjectInTLAB, and we?d be able to fire the latter for each sample. * "Should the interval between samples be configurable?? : Totally. In fact, it?d be helpful if it was configurable dynamically. Imagine if a JVM starts misbehaving after 2-3 weeks of running. You can dynamically increase the sampling rate to get a better profile if the default is not giving fine-grain enough information. * "As long of these features don?t contribute to sampling bias? : If the sampling interval is fixed, sampling bias would be a very real concern. In the above example, I?d increment top by 1M (the sampling frequency) + p% (a fudge factor).? * "Yes, a perhaps optional callbacks would be nice too.? : Oh, no. :-) But, as I said, we should definitely separate the sampling mechanism from the mechanism that consumes the samples. * "Another problem with our submitting things is that we can't really test on anything other than Linux.? : Another reason to go with a as platform independent solution as possible. :-) Regards, Tony ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Wed Jun 24 23:26:35 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Wed, 24 Jun 2015 16:26:35 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis wrote: > Hi Jeremy, > > Please see inline. > > On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) > wrote: > > I don't want the size of the TLAB, which is ergonomically adjusted, to be > tied to the sampling rate. There is no reason to do that. I want > reasonable statistical sampling of the allocations. > > > As I said explicitly in my e-mail, I totally agree with this. Which is why > I never suggested to resize TLABs in order to vary the sampling rate. > (Apologies if my e-mail was not clear.) > My fault - I misread it. Doesn't your proposal miss out of TLAB allocs entirely (and, in fact, not work if TLAB support is turned off)? I might be missing something obvious (and see my response below). > All this requires is a separate counter that is set to the next sampling > interval, and decremented when an allocation happens, which goes into a > slow path when the decrement hits 0. Doing a subtraction and a pointer > bump in allocation instead of just a pointer bump is basically free. > > > Maybe on intel is cheap, but maybe it?s not on other platforms that other > folks care about. > Really? A memory read and a subtraction? Which architectures care about that? Again, notice that no one has complained about the addition that was added for total bytes allocated per thread. I note that was actually added in the 6u20 timeframe. Note that it has been doing an additional addition (to keep track of per > thread allocation) as part of allocation since Java 7, > > > Interesting. I hadn?t realized that. Does that keep track of total size > allocated per thread or number of allocated objects per thread? If it?s the > former, why isn?t it possible to calculate that from the TLABs information? > Total size allocated per thread. It isn't possible to calculate that from the TLAB because of out-of-TLAB allocation (and hypothetically disabled TLABs). For some reason, they never included it in the ThreadMXBean interface, but it is in com.sun.management.ThreadMXBean, so you can cast your ThreadMXBean to a com.sun.management.ThreadMXBean and call getThreadAllocatedBytes() on it. > and no one has complained. > > I'm not worried about the ease of implementation here, because we've > already implemented it. > > > Yeah, but someone will have to maintain it moving forward. > I've been maintaining it internally to Google for 5 years. It's actually pretty self-contained. The only work involved is when they refactor something (so I've had to move it), or when a bug in the existing implementation is discovered. It is very closely parallel to the TLAB code, which doesn't change much / at all. > It hasn't even been hard for us to do the forward port, except when the > relevant Hotspot code is significantly refactored. > > We can also turn the sampling off, if we want. We can set the sampling > rate to 2^32, have the sampling code do nothing, and no one will ever > notice. > > > You still have extra instructions in the allocation path, so it?s not > turned off (i.e., you have the tax without any benefit). > > Hey, you have a counter in your allocation path you've never noticed, which none of your code uses. Pipelining is a wonderful thing. :) In fact, we could just have the sampling code do nothing, and no one would > ever notice. > > Honestly, no one ever notices the overhead of the sampling, anyway. JDK8 > made it more expensive to grab a stack trace (the cost became proportional > to the number of loaded classes), but we have a patch that mitigates that, > which we would also be happy to upstream. > > As for the other concern: my concern about *just* having the callback > mechanism is that there is quite a lot you can't do from user code during > an allocation, because of lack of access to JNI. > > > Maybe I missed something. Are the callbacks in Java? I.e., do you call > them using JNI from the slow path you call directly from the allocation > code? > > (For context: this referred to the hypothetical feature where we can provide a callback that invokes some code from allocation.) (It's not actually hypothetical, because we've already implemented it, but let's call it hypothetical for the moment.) We invoke native code. You can't invoke any Java code during allocation, including calling JNI methods, because that would make allocation potentially reentrant, which doesn't work for all sorts of reasons. The native code doesn't even get passed a JNIEnv * - there is nothing it can do with it without making the VM crash a lot. Or, rather, you might be able to do that, but it would take a lot of Hotspot rearchitecting. When I tried to do it, I realized it would be an extremely deep dive. However, you can do pretty much anything from the VM itself. Crucially > (for us), we don't just log the stack traces, we also keep track of which > are live and which aren't. We can't do this in a callback, if the callback > can't create weak refs to the object. > > What we do at Google is to have two methods: one that you pass a callback > to (the callback gets invoked with a StackTraceData object, as I've defined > above), and another that just tells you which sampled objects are still > live. We could also add a third, which allowed a callback to set the > sampling interval (basically, the VM would call it to get the integer > number of bytes to be allocated before the next sample). > > Would people be amenable to that? It makes the code more complex, but, as > I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB > object come from?"). > > > Well, that 1GB object would have most likely been allocated outside a TLAB > and you could have identified it by instrumenting the ?outside-of-TLAB > allocation path? (just saying?). > That's orthogonal to the point I was making in the quote above - the point I was making there was that we want to be able to detect what sampled objects are live. We can do that regardless of how we implement the sampling (although it did involve my making a new kind of weak oop processing mechanism inside the VM). But to the question of whether we can just instrument the outside-of-tlab allocation path... There are a few weirdnesses here. The first one that jumps to mind is that there's also a fast path for allocating in the YG outside of TLABs, if an object is too large to fit in the current TLAB. Those objects would never get sampled. So "outside of tlab" doesn't always mean "slow path". Another one that jumps to mind is that we don't know whether the outside-of-TLAB path actually passes the sampling threshold, especially if we let users configure the sampling threshold. So how would we know whether to sample it? You also have to keep track of the sampling interval in the code where we allocate new TLABs, in case the sampling threshold is larger than the TLAB size. That's not a big deal, of course. And, every time the TLAB code changes, we have to consider whether / how those changes affect this sampling mechanism. I guess my larger point is that there are so many little corner cases with TLAB allocation, including whether it even happens, that basing the sampling strategy around it seems like a cop-out. And my belief is that the arguments against our strategy don't really hold water, especially given the presence of the per-thread allocation counter that no one noticed. Heck, I've already had it reviewed internally by a Hotspot reviewer (Chuck Rasbold). All we really need is to write an acceptable JEP, to adjust the code based on the changes the community wants, and someone from Oracle willing to say "yes". For reference, to keep track of sampling, the delta to C2 is about 150 LOC (much of which is newlines-because-of-formatting for methods that take a lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 template interpreter is about 20 LOC, and the delta for the assembler is about 40 LOC. It's not completely trivial, but the code hasn't changed substantially in the 5 years since I wrote it (other than a couple of bugfixes). Obviously, assembler/template interpreter would have to be dup'd across platforms - we can do that for PPC and aarch64, on which we do active development, at least. > But, seriously, why didn?t you like my proposal? It can do anything your > scheme can with fewer and simpler code changes. The only thing that it > cannot do is to sample based on object count (i.e., every 100 objects) > instead of based on object size (i.e., every 1MB of allocations). But I > think doing sampling based on size is the right approach here (IMHO). > I agree that sampling based on size is the right approach. (And your approach is definitely simpler - I don't mean to discount it. And if that's what it takes to get this feature accepted, we'll do it, but I'll grumble about it.) Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Wed Jun 24 23:48:38 2015 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Thu, 25 Jun 2015 01:48:38 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <20150625014838.00001605.ecki@zusammenkunft.net> Am Wed, 24 Jun 2015 16:26:35 -0700 schrieb Jeremy Manson : > > As for the other concern: my concern about *just* having the > > callback mechanism is that there is quite a lot you can't do from > > user code during an allocation, because of lack of access to JNI. > > > > > > Maybe I missed something. Are the callbacks in Java? I.e., do you > > call them using JNI from the slow path you call directly from the > > allocation code? > > > > (For context: this referred to the hypothetical feature where we can > provide a callback that invokes some code from allocation.) What about a hypothetical queueing feature, so you can process the events asynchronously (perhaps with some backpressure control). This would work well for statistics processing. (Your other use case, the throwing of OOM would not work, I guess) But its an elegant solution to provide a code environment generic enoug for all kinds of instrumentation and independent of the "allocation recursion". Greetings Bernd From jeremymanson at google.com Thu Jun 25 06:00:42 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Wed, 24 Jun 2015 23:00:42 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <20150625014838.00001605.ecki@zusammenkunft.net> References: <20150625014838.00001605.ecki@zusammenkunft.net> Message-ID: Hey Bernd, In addition to the overhead of the mechanism (bad enough to implement backpressure control?), I'd also be worried about losing thread-local state in that case. It would be hard to use TLS at the native layer to persist data, or to map invocations to particular Java threads. Also, you would have no guarantees about the callbacks being executed in a timely fashion, or at all. A user could set up a mechanism to do this themselves, if they wanted. All they would have to do is implement their own, purely native queue in the callback mechanism. Another thread could pull from the queue and use JNI / Java to process the events. In fact, we could provide an extension library to do this for them, if we wanted, with the guidance "Use this particular callback, and you can process the events in a Java thread". Jeremy On Wed, Jun 24, 2015 at 4:48 PM, Bernd Eckenfels wrote: > Am Wed, 24 Jun 2015 16:26:35 -0700 > schrieb Jeremy Manson : > > > As for the other concern: my concern about *just* having the > > > callback mechanism is that there is quite a lot you can't do from > > > user code during an allocation, because of lack of access to JNI. > > > > > > > > > Maybe I missed something. Are the callbacks in Java? I.e., do you > > > call them using JNI from the slow path you call directly from the > > > allocation code? > > > > > > (For context: this referred to the hypothetical feature where we can > > provide a callback that invokes some code from allocation.) > > What about a hypothetical queueing feature, so you can process the > events asynchronously (perhaps with some backpressure control). This > would work well for statistics processing. > > (Your other use case, the throwing of OOM would not work, I guess) > > But its an elegant solution to provide a code environment generic enoug > for all kinds of instrumentation and independent of the "allocation > recursion". > > Greetings > Bernd > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Thu Jun 25 06:54:05 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Thu, 25 Jun 2015 08:54:05 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: > > But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). > > I would think that the size based sampling would create a size based bias in your sampling. Since IME, it?s allocation frequency is more damaging to performance, I?d prefer to see time boxed sampling Kind regards, Kirk Pepperdine -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaroslav.bachorik at oracle.com Thu Jun 25 09:53:01 2015 From: jaroslav.bachorik at oracle.com (Jaroslav Bachorik) Date: Thu, 25 Jun 2015 11:53:01 +0200 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out In-Reply-To: <558A76D3.2010902@oracle.com> References: <55882B75.1070804@oracle.com> <5588F702.20606@oracle.com> <55890BDA.8010104@oracle.com> <558A76D3.2010902@oracle.com> Message-ID: <558BCF7D.6030001@oracle.com> On 24.6.2015 11:22, David Holmes wrote: > On 23/06/2015 5:33 PM, Jaroslav Bachorik wrote: >> Hi David, >> >> On 23.6.2015 08:04, David Holmes wrote: >>> On 23/06/2015 1:36 AM, Jaroslav Bachorik wrote: >>>> Please, review the following test change >>>> >>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 >>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 >>>> >>>> GaugeMonitorDeadlockTest fails intermittently due data race - the call >>>> to monitorProxy.start() on L105 will eventually result in incrementing >>>> the GetCount attribute. If that happens before L109 had the chance to >>>> run the loop on L113-128 will become infinite. The initial value will >>>> contain the already incremented GetCount value and GetCount attribute >>>> value will not get further incremented. >>> >>> The reordering of the start() and initial observedProxy.getGetCount >>> seems reasonable. >>> >>> The changes to the timeout handling less so. The alarm code doesn't look >>> right to me. Each time you call check(true) in the loop you advance the >>> time when the alarm "goes off". ??? >> >> Stupid me :( Thanks for catching this. >> >> http://cr.openjdk.java.net/~jbachorik/8071487/webrev.01 > > That looks better, but the Alarm class would benefit from some simple > descriptive comments. I must admit I don't really see the need for > introducing it. I expected to reuse it in some of the other JMX monitoring tests. But it didn't turn out that way. I've removed the Alarm class from the test and just switched to using Util.adjustTimeout() instead of doing the parsing of "test.timeout.factor" system property by hand. http://cr.openjdk.java.net/~jbachorik/8071487/webrev.02 Thanks, -JB- > > Thanks, > David > >> -JB- >> >>> >>> David >>> >>> >>> >>>> I took the liberty to fix the same issue in >>>> test/javax/management/monitor/StringMonitorDeadlockTest.java, not >>>> waiting for the real test failure. >>>> >>>> Thanks, >>>> >>>> -JB- >> From jeremymanson at google.com Thu Jun 25 17:34:11 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 10:34:11 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Why would allocation frequency be more damaging to performance? Allocation is cheap, and as long as they become dead before the YG collection, it costs the same to collect one 1MB object as it does to collection 1000 1K objects. Jeremy On Wed, Jun 24, 2015 at 11:54 PM, Kirk Pepperdine wrote: > > But, seriously, why didn?t you like my proposal? It can do anything your > scheme can with fewer and simpler code changes. The only thing that it > cannot do is to sample based on object count (i.e., every 100 objects) > instead of based on object size (i.e., every 1MB of allocations). But I > think doing sampling based on size is the right approach here (IMHO). > > > I would think that the size based sampling would create a size based bias > in your sampling. Since IME, it?s allocation frequency is more damaging to > performance, I?d prefer to see time boxed sampling > > Kind regards, > Kirk Pepperdine > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tprintezis at twitter.com Thu Jun 25 20:28:50 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Thu, 25 Jun 2015 16:28:50 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Hi Jeremy, Inline. On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) wrote: On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis??wrote: Hi Jeremy, Please see inline. On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote: I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate.? There is no reason to do that.? I want reasonable statistical sampling of the allocations. ? As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.) My fault - I misread it.? Doesn't your proposal miss out of TLAB allocs entirely This is correct: We?ll also have to intercept the outside-TLAB allocs. But, IMHO, this is a feature as it?s helpful to know how many (and which) allocations happen outside TLABs. These are generally very infrequent (and slow anyway), so sampling all of those, instead of only sampling some of them, does not have much of an overhead. But, you could also do sampling for the outside-TLAB allocs too, if you want: just accumulate their size on a separate per-thread counter and sample the one that bumps that counter goes over a limit. An additional observation (orthogonal to the main point, but I thought I?d mention it anyway): For the outside-TLAB allocs it?d be helpful to also know which generation the object ended up in (e.g., young gen or direct-to-old-gen). This is very helpful in some situations when you?re trying to work out which allocation(s) grew the old gen occupancy between two young GCs. FWIW, the existing JFR events follow the approach I described above: * one event for each new TLAB + first alloc in that TLAB (my proposal basically generalizes this and removes the 1-1 relationship between object alloc sampling and new TLAB operation) * one event for all allocs outside a TLAB I think the above separation is helpful. But if you think it could confuse users, you can of course easily just combine the information (but I strongly believe it?s better to report the information separately). (and, in fact, not work if TLAB support is turned off)?? Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a genuine question.) ?I might be missing something obvious (and see my response below). ? All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0.? Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free.?? Maybe on intel is cheap, but maybe it?s not on other platforms that other folks care about. Really?? A memory read and a subtraction?? Which architectures care about that? I was not concerned with the read and subtraction, I was more concerned with the conditional that follows them (intel has great branch prediction). And a personal pet peeve (based on past experience): How many ?free? instructions do you have to add before they are not free any more? Again, notice that no one has complained about the addition that was added for total bytes allocated per thread.? I note that was actually added in the 6u20 timeframe. Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7,? Interesting. I hadn?t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it?s the former, why isn?t it possible to calculate that from the TLABs information? Total size allocated per thread.? It isn't possible to calculate that from the TLAB because of out-of-TLAB allocation? The allocating Thread is passed to the slow (outside-TLAB) alloc path so it would be trivial to update the per-thread allocation stats from there too (in fact, it does; see below). (and hypothetically disabled TLABs). Anyone cares? :-) For some reason, they never included it in the ThreadMXBean interface, but it is in com.sun.management.ThreadMXBean, so you can cast your ThreadMXBean to a com.sun.management.ThreadMXBean and call getThreadAllocatedBytes() on it. Thanks for the tip. I?ll look into this... and no one has complained. I'm not worried about the ease of implementation here, because we've already implemented it.?? Yeah, but someone will have to maintain it moving forward. I've been maintaining it internally to Google for 5 years.? It's actually pretty self-contained.? The only work involved is when they refactor something (so I've had to move it), or when a bug in the existing implementation is discovered.? It is very closely parallel to the TLAB code, which doesn't change much / at all. The TLAB code has really not changed much for a while. ;-) (but haven?t looked at the JDK 9 source very closely though?) It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. We can also turn the sampling off, if we want.? We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice.?? You still have extra instructions in the allocation path, so it?s not turned off (i.e., you have the tax without any benefit). Hey, you have a counter in your allocation path you've never noticed, which none of your code uses.? Pipelining is a wonderful thing. ?:) See above re: ?free? instructions. In fact, we could just have the sampling code do nothing, and no one would ever notice. Honestly, no one ever notices the overhead of the sampling, anyway.? JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code? (For context: this referred to the hypothetical feature where we can provide a callback that invokes some code from allocation.) (It's not actually hypothetical, because we've already implemented it, but let's call it hypothetical for the moment.) OK. We invoke native code.? You can't invoke any Java code during allocation, including calling JNI methods, because that would make allocation potentially reentrant, which doesn't work for all sorts of reasons. That?s what I was worried about?. ? The native code doesn't even get passed a JNIEnv * - there is nothing it can do with it without making the VM crash a lot. So, thanks for the clarification. Being able to attach a callback to this in, say, the JVM it?d be totally fine. I was worried that you wanted to call Java. :-) Or, rather, you might be able to do that, but it would take a lot of Hotspot rearchitecting.? When I tried to do it, I realized it would be an extremely deep dive. I believe you. :-) ? However, you can do pretty much anything from the VM itself.? Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't.? We can't do this in a callback, if the callback can't create weak refs to the object. What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live.? We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). ? Would people be amenable to that?? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey!? Where did that 1 GB object come from?"). Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the ?outside-of-TLAB allocation path? (just saying?). That's orthogonal to the point I was making in the quote above - the point I was making there was that we want to be able to detect what sampled objects are live.? We can do that regardless of how we implement the sampling (although it did involve my making a new kind of weak oop processing mechanism inside the VM). Yeah, I was thinking of doing something similar (tracking object lifetimes, and other attributes, with WeakRefs).? But to the question of whether we can just instrument the outside-of-tlab allocation path...? There are a few weirdnesses here.? The first one that jumps to mind is that there's also a fast path for allocating in the YG outside of TLABs, if an object is too large to fit in the current TLAB.? Those objects would never get sampled.? So "outside of tlab" doesn't always mean "slow path". CollectedHeap::common_mem_allocate_noinit() is the first-level of the slow path called when a TLAB allocation fails because the object doesn?t fit in the current TLAB. It checks (alocate_from_tlab() / allocate_from_tlab_slow()) whether to refill the current TLAB or keep the TLAB and delegate to the GC (mem_allocate()) to allocate the object outside a TLAB (either in the young or old gen; the GC might also decide to do a collection at this point if, say, the eden is full...). So, it depends on what you mean by slow path but, yes, any alloocations that go through the above path should be considered as ?slow path? allocations. One more piece of data:?AllocTracer::send_allocation_outside_tlab_event() (the JFR entry point for outside-TLAB allocs) is fired from common_mem_allocate_noint(). So, if there are other non-TLAB allocation paths outside that method, that entry point has been placed incorrectly (it?s possible of course; but I think that it?s actually placed correctly). (note: I only looked at the JDK 8 sources, haven?t checked the JDK 9 sources yet, the above might have been changed) BTW, when looking at the common_mem_allocate_noinit() code I noticed the following: THREAD->incr_allocated_bytes(size * HeapWordSize); (as predicted earlier) Another one that jumps to mind is that we don't know whether the outside-of-TLAB path actually passes the sampling threshold, especially if we let users configure the sampling threshold.? So how would we know whether to sample it? See above (IMHO: sample all of them). You also have to keep track of the sampling interval in the code where we allocate new TLABs, in case the sampling threshold is larger than the TLAB size.? That's not a big deal, of course. Of course, but that?s kinda trivial. BTW, one approach here would be ?given that refilling a TLAB is slow anyway, always sample the first object in each TLAB irrespective of desired sampling frequence?. Another would be ?don?t do that, I set the sampling frequency pretty low not to be flooded with data when the TLABs are very small?. I have to say I?m in the latter camp. And, every time the TLAB code changes, we have to consider whether / how those changes affect this sampling mechanism. Yes, but how often does the TLAB code change? :-) I guess my larger point is that there are so many little corner cases with TLAB allocation, including whether it even happens, that basing the sampling strategy around it seems like a cop-out.?? There are not many little corner cases. There are two cases: allocation inside a TLAB, allocation outside a TLAB. The former is by far the most common. The latter is generally very infrequent and has a well-defined code path (I described it earlier). And, as I said, it could be very helpful and informative to treat (and account for) the two cases separately. And my belief is that the arguments against our strategy don't really hold water, especially given the presence of the per-thread allocation counter that no one noticed. ? I?ve already addressed that. Heck, I've already had it reviewed internally by a Hotspot reviewer (Chuck Rasbold).? All we really need is to write an acceptable JEP, to adjust the code based on the changes the community wants, and someone from Oracle willing to say "yes". For reference, to keep track of sampling, the delta to C2 is about 150 LOC (much of which is newlines-because-of-formatting for methods that take a lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 template interpreter is about 20 LOC, and the delta for the assembler is about 40 LOC. ? ? ?It's not completely trivial, but the code hasn't changed substantially in the 5 years since I wrote it (other than a couple of bugfixes). Obviously, assembler/template interpreter would have to be dup'd across platforms - we can do that for PPC and aarch64, on which we do active development, at least. I?ll again vote for the simplicity of having a simple change in only one place (OK, two places?). But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). I agree that sampling based on size is the right approach. ? (And your approach is definitely simpler - I don't mean to discount it.? And if that's what it takes to get this feature accepted, we'll do it, but I'll grumble about it.) That?s fine. :-) Tony ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From karen.kinnear at oracle.com Thu Jun 25 20:41:35 2015 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Thu, 25 Jun 2015 16:41:35 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Jeremy, Did I follow this correctly - that your approach modifies the compilers and interpreters and Tony's modifies the common allocation code? Given that the number of compilers and interpreters and interpreter platforms keeps expanding - I'd like to add a vote to have heap allocation profiling in common allocation code. thanks, Karen On Jun 25, 2015, at 4:28 PM, Tony Printezis wrote: > Hi Jeremy, > > Inline. > > On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) wrote: > >> >> >> On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis wrote: >> Hi Jeremy, >> >> Please see inline. >> >> On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote: >> >>> I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate. There is no reason to do that. I want reasonable statistical sampling of the allocations. >> >> >> As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.) >> >> >> My fault - I misread it. Doesn't your proposal miss out of TLAB allocs entirely > > > This is correct: We?ll also have to intercept the outside-TLAB allocs. But, IMHO, this is a feature as it?s helpful to know how many (and which) allocations happen outside TLABs. These are generally very infrequent (and slow anyway), so sampling all of those, instead of only sampling some of them, does not have much of an overhead. But, you could also do sampling for the outside-TLAB allocs too, if you want: just accumulate their size on a separate per-thread counter and sample the one that bumps that counter goes over a limit. > > An additional observation (orthogonal to the main point, but I thought I?d mention it anyway): For the outside-TLAB allocs it?d be helpful to also know which generation the object ended up in (e.g., young gen or direct-to-old-gen). This is very helpful in some situations when you?re trying to work out which allocation(s) grew the old gen occupancy between two young GCs. > > FWIW, the existing JFR events follow the approach I described above: > > * one event for each new TLAB + first alloc in that TLAB (my proposal basically generalizes this and removes the 1-1 relationship between object alloc sampling and new TLAB operation) > > * one event for all allocs outside a TLAB > > I think the above separation is helpful. But if you think it could confuse users, you can of course easily just combine the information (but I strongly believe it?s better to report the information separately). > > > >> (and, in fact, not work if TLAB support is turned off)? > > > Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a genuine question.) > > > >> I might be missing something obvious (and see my response below). > > >> >>> All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0. Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free. >> >> >> Maybe on intel is cheap, but maybe it?s not on other platforms that other folks care about. >> >> Really? A memory read and a subtraction? Which architectures care about that? > > > I was not concerned with the read and subtraction, I was more concerned with the conditional that follows them (intel has great branch prediction). > > And a personal pet peeve (based on past experience): How many ?free? instructions do you have to add before they are not free any more? > > > >> >> Again, notice that no one has complained about the addition that was added for total bytes allocated per thread. I note that was actually added in the 6u20 timeframe. >> >>> Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7, >> >> >> Interesting. I hadn?t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it?s the former, why isn?t it possible to calculate that from the TLABs information? >> >> >> Total size allocated per thread. It isn't possible to calculate that from the TLAB because of out-of-TLAB allocation > > > The allocating Thread is passed to the slow (outside-TLAB) alloc path so it would be trivial to update the per-thread allocation stats from there too (in fact, it does; see below). > > > >> (and hypothetically disabled TLABs). > > > Anyone cares? :-) > > > >> >> For some reason, they never included it in the ThreadMXBean interface, but it is in com.sun.management.ThreadMXBean, so you can cast your ThreadMXBean to a com.sun.management.ThreadMXBean and call getThreadAllocatedBytes() on it. >> > > Thanks for the tip. I?ll look into this... > >>> and no one has complained. >>> >>> I'm not worried about the ease of implementation here, because we've already implemented it. >> >> >> Yeah, but someone will have to maintain it moving forward. >> >> >> I've been maintaining it internally to Google for 5 years. It's actually pretty self-contained. The only work involved is when they refactor something (so I've had to move it), or when a bug in the existing implementation is discovered. It is very closely parallel to the TLAB code, which doesn't change much / at all. >> > > The TLAB code has really not changed much for a while. ;-) (but haven?t looked at the JDK 9 source very closely though?) > >>> It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. >>> >>> We can also turn the sampling off, if we want. We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice. >> >> >> You still have extra instructions in the allocation path, so it?s not turned off (i.e., you have the tax without any benefit). >> >> >> Hey, you have a counter in your allocation path you've never noticed, which none of your code uses. Pipelining is a wonderful thing. :) > > > See above re: ?free? instructions. > > > >>> >>> In fact, we could just have the sampling code do nothing, and no one would ever notice. >>> >>> Honestly, no one ever notices the overhead of the sampling, anyway. JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. >>> >>> As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. >> >> >> Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code? >> >> (For context: this referred to the hypothetical feature where we can provide a callback that invokes some code from allocation.) >> >> (It's not actually hypothetical, because we've already implemented it, but let's call it hypothetical for the moment.) > > > OK. > > > >> >> We invoke native code. You can't invoke any Java code during allocation, including calling JNI methods, because that would make allocation potentially reentrant, which doesn't work for all sorts of reasons. > > > That?s what I was worried about?. > > > >> The native code doesn't even get passed a JNIEnv * - there is nothing it can do with it without making the VM crash a lot. > > > So, thanks for the clarification. Being able to attach a callback to this in, say, the JVM it?d be totally fine. I was worried that you wanted to call Java. :-) > > > >> >> Or, rather, you might be able to do that, but it would take a lot of Hotspot rearchitecting. When I tried to do it, I realized it would be an extremely deep dive. > > > I believe you. :-) > > > >>> >>> However, you can do pretty much anything from the VM itself. Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't. We can't do this in a callback, if the callback can't create weak refs to the object. >>> >>> What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live. We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). >>> >>> Would people be amenable to that? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB object come from?"). >> >> >> Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the ?outside-of-TLAB allocation path? (just saying?). >> >> >> That's orthogonal to the point I was making in the quote above - the point I was making there was that we want to be able to detect what sampled objects are live. We can do that regardless of how we implement the sampling (although it did involve my making a new kind of weak oop processing mechanism inside the VM). > > > Yeah, I was thinking of doing something similar (tracking object lifetimes, and other attributes, with WeakRefs). > > > >> >> But to the question of whether we can just instrument the outside-of-tlab allocation path... There are a few weirdnesses here. The first one that jumps to mind is that there's also a fast path for allocating in the YG outside of TLABs, if an object is too large to fit in the current TLAB. Those objects would never get sampled. So "outside of tlab" doesn't always mean "slow path". > > > CollectedHeap::common_mem_allocate_noinit() is the first-level of the slow path called when a TLAB allocation fails because the object doesn?t fit in the current TLAB. It checks (alocate_from_tlab() / allocate_from_tlab_slow()) whether to refill the current TLAB or keep the TLAB and delegate to the GC (mem_allocate()) to allocate the object outside a TLAB (either in the young or old gen; the GC might also decide to do a collection at this point if, say, the eden is full...). So, it depends on what you mean by slow path but, yes, any alloocations that go through the above path should be considered as ?slow path? allocations. > > One more piece of data: AllocTracer::send_allocation_outside_tlab_event() (the JFR entry point for outside-TLAB allocs) is fired from common_mem_allocate_noint(). So, if there are other non-TLAB allocation paths outside that method, that entry point has been placed incorrectly (it?s possible of course; but I think that it?s actually placed correctly). > > (note: I only looked at the JDK 8 sources, haven?t checked the JDK 9 sources yet, the above might have been changed) > > BTW, when looking at the common_mem_allocate_noinit() code I noticed the following: > > THREAD->incr_allocated_bytes(size * HeapWordSize); > (as predicted earlier) > > > >> >> Another one that jumps to mind is that we don't know whether the outside-of-TLAB path actually passes the sampling threshold, especially if we let users configure the sampling threshold. So how would we know whether to sample it? > > > See above (IMHO: sample all of them). > > > >> >> You also have to keep track of the sampling interval in the code where we allocate new TLABs, in case the sampling threshold is larger than the TLAB size. That's not a big deal, of course. > > > Of course, but that?s kinda trivial. BTW, one approach here would be ?given that refilling a TLAB is slow anyway, always sample the first object in each TLAB irrespective of desired sampling frequence?. Another would be ?don?t do that, I set the sampling frequency pretty low not to be flooded with data when the TLABs are very small?. I have to say I?m in the latter camp. > > > >> >> >> And, every time the TLAB code changes, we have to consider whether / how those changes affect this sampling mechanism. > > > Yes, but how often does the TLAB code change? :-) > > > >> >> I guess my larger point is that there are so many little corner cases with TLAB allocation, including whether it even happens, that basing the sampling strategy around it seems like a cop-out. > > > There are not many little corner cases. There are two cases: allocation inside a TLAB, allocation outside a TLAB. The former is by far the most common. The latter is generally very infrequent and has a well-defined code path (I described it earlier). And, as I said, it could be very helpful and informative to treat (and account for) the two cases separately. > > > >> And my belief is that the arguments against our strategy don't really hold water, especially given the presence of the per-thread allocation counter that no one noticed. > > > I?ve already addressed that. > > > >> >> Heck, I've already had it reviewed internally by a Hotspot reviewer (Chuck Rasbold). All we really need is to write an acceptable JEP, to adjust the code based on the changes the community wants, and someone from Oracle willing to say "yes". > > >> >> For reference, to keep track of sampling, the delta to C2 is about 150 LOC (much of which is newlines-because-of-formatting for methods that take a lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 template interpreter is about 20 LOC, and the delta for the assembler is about 40 LOC. It's not completely trivial, but the code hasn't changed substantially in the 5 years since I wrote it (other than a couple of bugfixes). >> >> Obviously, assembler/template interpreter would have to be dup'd across platforms - we can do that for PPC and aarch64, on which we do active development, at least. > > > I?ll again vote for the simplicity of having a simple change in only one place (OK, two places?). > > > >> >> But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). >> >> I agree that sampling based on size is the right approach. >> >> (And your approach is definitely simpler - I don't mean to discount it. And if that's what it takes to get this feature accepted, we'll do it, but I'll grumble about it.) > > > That?s fine. :-) > > Tony > > > > > ----- > > Tony Printezis | JVM/GC Engineer / VM Team | Twitter > > @TonyPrintezis > tprintezis at twitter.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tprintezis at twitter.com Thu Jun 25 20:55:43 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Thu, 25 Jun 2015 16:55:43 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: <20150625014838.00001605.ecki@zusammenkunft.net> References: <20150625014838.00001605.ecki@zusammenkunft.net> Message-ID: Bernd, I like the idea of buffering up the sampled objects in, some data structure. But I assume it?d have to be a per-thread data structure to avoid conention issues. So, we?ll also need a periodic task that collects all such data structures and makes them available (somehow) to whoever wants to consume them? Tony On June 24, 2015 at 7:49:06 PM, Bernd Eckenfels (ecki at zusammenkunft.net) wrote: Am Wed, 24 Jun 2015 16:26:35 -0700 schrieb Jeremy Manson : > > As for the other concern: my concern about *just* having the > > callback mechanism is that there is quite a lot you can't do from > > user code during an allocation, because of lack of access to JNI. > > > > > > Maybe I missed something. Are the callbacks in Java? I.e., do you > > call them using JNI from the slow path you call directly from the > > allocation code? > > > > (For context: this referred to the hypothetical feature where we can > provide a callback that invokes some code from allocation.) What about a hypothetical queueing feature, so you can process the events asynchronously (perhaps with some backpressure control). This would work well for statistics processing. (Your other use case, the throwing of OOM would not work, I guess) But its an elegant solution to provide a code environment generic enoug for all kinds of instrumentation and independent of the "allocation recursion". Greetings Bernd ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tprintezis at twitter.com Thu Jun 25 21:08:45 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Thu, 25 Jun 2015 17:08:45 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Hi Kirk, (long time!) See inline. On June 25, 2015 at 2:54:04 AM, Kirk Pepperdine (kirk.pepperdine at gmail.com) wrote: But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). I would think that the size based sampling would create a size based bias in your sampling.? That?s actually true. And this could be good (if you?re interested in what?s filling up your eden, the larger objects might be of more interest) or bad (if you want to get a general idea of what?s being allocated, the size bias might make you miss some types of objects / allocation sites). Since IME, it?s allocation frequency is more damaging to performance, I?d prefer to see time boxed sampling Do you mean ?sample every X ms, say?? Tony Kind regards, Kirk Pepperdine ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tprintezis at twitter.com Thu Jun 25 21:23:58 2015 From: tprintezis at twitter.com (Tony Printezis) Date: Thu, 25 Jun 2015 17:23:58 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: BTW, Could we get a reaction from the Oracle folks on this? Even though Jeremy and myself are proposing different implementation approaches, we both agree (and Jeremy please correct me on this) that having an allocation sampling mechanism that?s more flexible to what?s already in HotSpot (in particular: the sampling frequency not being tied to the TLAB size) will be a very helpful profiling feature. Is this something that we pursue to contribute? Tony On June 24, 2015 at 1:57:44 PM, Tony Printezis (tprintezis at twitter.com) wrote: Hi Jeremy, Please see inline. On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote: I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate.? There is no reason to do that.? I want reasonable statistical sampling of the allocations. ? As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.) All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0.? Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free.?? Maybe on intel is cheap, but maybe it?s not on other platforms that other folks care about. Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7,? Interesting. I hadn?t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it?s the former, why isn?t it possible to calculate that from the TLABs information? and no one has complained. I'm not worried about the ease of implementation here, because we've already implemented it.?? Yeah, but someone will have to maintain it moving forward. It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. We can also turn the sampling off, if we want.? We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice.?? You still have extra instructions in the allocation path, so it?s not turned off (i.e., you have the tax without any benefit). In fact, we could just have the sampling code do nothing, and no one would ever notice. Honestly, no one ever notices the overhead of the sampling, anyway.? JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code? ? However, you can do pretty much anything from the VM itself.? Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't.? We can't do this in a callback, if the callback can't create weak refs to the object. What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live.? We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). ? Would people be amenable to that?? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey!? Where did that 1 GB object come from?"). Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the ?outside-of-TLAB allocation path? (just saying?). But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). Tony Jeremy On Tue, Jun 23, 2015 at 1:06 PM, Tony Printezis??wrote: Jeremy (and all), I?m not on the serviceability list so I won?t include the messages so far. :-) Also CCing the hotspot GC list, in case they have some feedback on this. Could I suggest a (much) simpler but at least as powerful and flexible way to do this? (This is something we?ve been meaning to do for a while now for TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force allocations to go into the slow path periodically by artificially setting the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to (bottom+1M). When an allocation thinks the TLAB is full (in this case, the first 1MB is full) it will call the allocation slow path. There, you can intercept it, sample the allocation (and, like in your case, you?ll also have the correct stack trace), notice that the TLAB is not actually full, extend its to top to, say, (bottom+2M), and you?re done. Advantages of this approach: * This is a much smaller, simpler, and self-contained change (no compiler changes necessary to maintain...). * When it?s off, the overhead is only one extra test at the slow path TLAB allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK using a similar mechanism and, when it?s off, I?ve observed no performance overhead). * (most importantly) You can turn this on and off, and adjust the sampling rate, dynamically. If you do the sampling based on JITed code, you?ll have to recompile all methods with allocation sites to turn the sampling on or off. (You can of course have it always on and just discard the output; it?d be nice not to have to do that though. IMHO, at least.) * You can also very cheaply turn this on and off (or adjust the sampling frequncy) per thread, if that?s be helpful in some way (just add the appropriate info on the thread?s TLAB). A few extra comments on the previous discussion: * "JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2? : When TLABs get very large, I don?t think sampling one object per TLAB is enough to get a good sample (IMHO, at least). It?s probably OK for something like jbb which mostly allocates instances of a handful of classes and has very few allocation sites. But, a lot of the code we run at Twitter is a lot more elaborate than that and, in our experience, sampling one object per TLAB is not enough. You can, of course, decrease the TLAB size to increase the sampling size. But it?d be good not to have to do that given a smaller TLAB size could increase contention across threads. * "Should it *just* take a stack trace, or should the behavior be configurable?? : I think we?d have to separate the allocation sampling mechanism from the consumption of the allocation samples. Once the sampling mechanism is in, different JVMs can take advantage of it in different ways. I assume that the Oracle folks would like at least a JFR event for every such sample. But in your build you can add extra code to collect the information in the way you have now. * Talking of JFR, it?s a bit unfortunate that the AllocObjectInNewTLAB event has both the new TLAB information and the allocation information. It would have been nice if that event was split into two, say NewTLAB and AllocObjectInTLAB, and we?d be able to fire the latter for each sample. * "Should the interval between samples be configurable?? : Totally. In fact, it?d be helpful if it was configurable dynamically. Imagine if a JVM starts misbehaving after 2-3 weeks of running. You can dynamically increase the sampling rate to get a better profile if the default is not giving fine-grain enough information. * "As long of these features don?t contribute to sampling bias? : If the sampling interval is fixed, sampling bias would be a very real concern. In the above example, I?d increment top by 1M (the sampling frequency) + p% (a fudge factor).? * "Yes, a perhaps optional callbacks would be nice too.? : Oh, no. :-) But, as I said, we should definitely separate the sampling mechanism from the mechanism that consumes the samples. * "Another problem with our submitting things is that we can't really test on anything other than Linux.? : Another reason to go with a as platform independent solution as possible. :-) Regards, Tony ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 05:27:08 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 22:27:08 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: On Thu, Jun 25, 2015 at 1:28 PM, Tony Printezis wrote: > Hi Jeremy, > > Inline. > > On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) > wrote: > > > > On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis > wrote: > >> Hi Jeremy, >> >> Please see inline. >> >> On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) >> wrote: >> >> I don't want the size of the TLAB, which is ergonomically adjusted, to be >> tied to the sampling rate. There is no reason to do that. I want >> reasonable statistical sampling of the allocations. >> >> >> As I said explicitly in my e-mail, I totally agree with this. Which is >> why I never suggested to resize TLABs in order to vary the sampling rate. >> (Apologies if my e-mail was not clear.) >> > > My fault - I misread it. Doesn't your proposal miss out of TLAB allocs > entirely > > > This is correct: We?ll also have to intercept the outside-TLAB allocs. > But, IMHO, this is a feature as it?s helpful to know how many (and which) > allocations happen outside TLABs. These are generally very infrequent (and > slow anyway), so sampling all of those, instead of only sampling some of > them, does not have much of an overhead. But, you could also do sampling > for the outside-TLAB allocs too, if you want: just accumulate their size on > a separate per-thread counter and sample the one that bumps that counter > goes over a limit. > > The outside-TLAB allocations generally get caught anyway, because they tend to be large enough to jump over the sample size immediately. > An additional observation (orthogonal to the main point, but I thought I?d > mention it anyway): For the outside-TLAB allocs it?d be helpful to also > know which generation the object ended up in (e.g., young gen or > direct-to-old-gen). This is very helpful in some situations when you?re > trying to work out which allocation(s) grew the old gen occupancy between > two young GCs. > True. We don't have this implemented, but it would be reasonably straightforward to glean it from the oop. > FWIW, the existing JFR events follow the approach I described above: > > * one event for each new TLAB + first alloc in that TLAB (my proposal > basically generalizes this and removes the 1-1 relationship between object > alloc sampling and new TLAB operation) > > * one event for all allocs outside a TLAB > > I think the above separation is helpful. But if you think it could confuse > users, you can of course easily just combine the information (but I > strongly believe it?s better to report the information separately). > I do think it would make a confusing API. It might make more sense to have a reporting mechanism that had a set number of fields with very concrete information (size, class, stacktrace), but allowed for platform-specific metadata. We end up with a very long list of things we want in the sample: generation (how do you describe a generation?), object age (by number of GCs survived? What kind of GC?), was it a TLAB allocation, etc. (and, in fact, not work if TLAB support is turned off)? > > > Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a genuine > question.) > I don't think they do. I have turned them off for various reasons (usually, I'm trying to instrument allocations and I don't want to muck about with thinking about TLABs), and the code paths seem a little crufty. ISTR at some point finding something that clearly only worked by mistake, but I can't remember now what it was. [snip] > However, you can do pretty much anything from the VM itself. Crucially >> (for us), we don't just log the stack traces, we also keep track of which >> are live and which aren't. We can't do this in a callback, if the callback >> can't create weak refs to the object. >> >> What we do at Google is to have two methods: one that you pass a callback >> to (the callback gets invoked with a StackTraceData object, as I've defined >> above), and another that just tells you which sampled objects are still >> live. We could also add a third, which allowed a callback to set the >> sampling interval (basically, the VM would call it to get the integer >> number of bytes to be allocated before the next sample). >> >> Would people be amenable to that? It makes the code more complex, but, >> as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB >> object come from?"). >> >> >> Well, that 1GB object would have most likely been allocated outside a >> TLAB and you could have identified it by instrumenting the ?outside-of-TLAB >> allocation path? (just saying?). >> > > That's orthogonal to the point I was making in the quote above - the point > I was making there was that we want to be able to detect what sampled > objects are live. We can do that regardless of how we implement the > sampling (although it did involve my making a new kind of weak oop > processing mechanism inside the VM). > > > Yeah, I was thinking of doing something similar (tracking object > lifetimes, and other attributes, with WeakRefs). > We have all of that implemented, so hopefully I can save you the trouble. :) But to the question of whether we can just instrument the outside-of-tlab > allocation path... There are a few weirdnesses here. The first one that > jumps to mind is that there's also a fast path for allocating in the YG > outside of TLABs, if an object is too large to fit in the current TLAB. > Those objects would never get sampled. So "outside of tlab" doesn't always > mean "slow path". > > > CollectedHeap::common_mem_allocate_noinit() is the first-level of the slow > path called when a TLAB allocation fails because the object doesn?t fit in > the current TLAB. It checks (alocate_from_tlab() / > allocate_from_tlab_slow()) whether to refill the current TLAB or keep the > TLAB and delegate to the GC (mem_allocate()) to allocate the object outside > a TLAB (either in the young or old gen; the GC might also decide to do a > collection at this point if, say, the eden is full...). So, it depends on > what you mean by slow path but, yes, any alloocations that go through the > above path should be considered as ?slow path? allocations. > Let me be more specific. Here is a place where allocations go through a fast path that is outside of a TLAB: http://hg.openjdk.java.net/jdk9/dev/hotspot/file/972580a0eef8/src/cpu/x86/vm/templateTable_x86.cpp#l3759 If the object won't fit in the TLAB, but will fit in the Eden, it will be allocated in the Eden, with hand-generated assembly. This case will be entirely missed by sampling just the TLAB creation (or your variant) and the slow path. I may be missing something about that code, but I can't really see what it is. One more piece of data: AllocTracer::send_allocation_outside_tlab_event() > (the JFR entry point for outside-TLAB allocs) is fired from > common_mem_allocate_noint(). So, if there are other non-TLAB allocation > paths outside that method, that entry point has been placed incorrectly > (it?s possible of course; but I think that it?s actually placed correctly). > What is happening in the line to which I referred, then? To me, it kind of reads like "this is close enough to being TLAB allocation that I don't care that it isn't". And that's really what's going on here. Your strategy is to tie what I see as a platform feature to a particular implementation. If the implementation changes, or if we really don't understand it as well as we think we do, the whole thing falls on the floor. If we mention TLABs in the docs, and TLABs do change, then it won't mean anything anymore. A particular example pops to mind: I believe Metronome doesn't have TLABs at all. Is that correct? Can J9 developers implement this feature? > For reference, to keep track of sampling, the delta to C2 is about 150 LOC > (much of which is newlines-because-of-formatting for methods that take a > lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 > template interpreter is about 20 LOC, and the delta for the assembler is > about 40 LOC. It's not completely trivial, but the code hasn't changed > substantially in the 5 years since I wrote it (other than a couple of > bugfixes). > > Obviously, assembler/template interpreter would have to be dup'd across > platforms - we can do that for PPC and aarch64, on which we do active > development, at least. > > > I?ll again vote for the simplicity of having a simple change in only one > place (OK, two places?). > This isn't a simple change anyway, if we're keeping track of live references. We have to hook into reference processing - when a weak oop is detected to be dead, we have to delete the metadata. And we have to change JVMTI. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 05:34:22 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 22:34:22 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Karen, I understand your concerns. For reference, this is the additional code in the x86 assembler. There are very small stubs in C1 and the template interpreter to call out to this macro (forgive the gratuitous use of the string "google" - we sprinkle it around a bit because it makes it a little easier to distinguish our code from upstream code). #define GOOGLE_HEAP_MONITORING(ma, thread, var_size_in_bytes, con_size_in_bytes, object, t1, t2, sample_invocation) \ do { \ { \ SkipIfEqual skip_if(ma, &GoogleHeapMonitor, 0); \ Label skip_sample; \ Register thr = thread; \ if (!thr->is_valid()) { \ NOT_LP64(assert(t1 != noreg, \ "Need temporary register for constants")); \ thr = NOT_LP64(t1) LP64_ONLY(r15_thread); \ NOT_LP64(ma -> get_thread(thr);) \ } \ /* Trigger heap monitoring event */ \ Address bus(thr, \ JavaThread::google_bytes_until_sample_offset()); \ \ if (var_size_in_bytes->is_valid()) { \ ma -> NOT_LP64(subl) LP64_ONLY(subq)(bus, var_size_in_bytes); \ } else { \ int csib = (con_size_in_bytes); \ assert(t2 != noreg, \ "Need temporary register for constants"); \ ma -> NOT_LP64(movl) LP64_ONLY(mov64)(t2, csib); \ ma -> NOT_LP64(subl) LP64_ONLY(subq)(bus, t2); \ } \ \ ma -> jcc(Assembler::positive, skip_sample); \ \ { \ sample_invocation \ } \ ma -> bind(skip_sample); \ } \ } while(0) It's not all that hard to port to additional architectures, but we'll have to think about it. Jeremy On Thu, Jun 25, 2015 at 1:41 PM, Karen Kinnear wrote: > Jeremy, > > Did I follow this correctly - that your approach modifies the compilers > and interpreters and Tony's modifies the > common allocation code? > > Given that the number of compilers and interpreters and interpreter > platforms keeps expanding - I'd like to > add a vote to have heap allocation profiling in common allocation code. > > thanks, > Karen > > On Jun 25, 2015, at 4:28 PM, Tony Printezis wrote: > > Hi Jeremy, > > Inline. > > On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) > wrote: > > > > On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis > wrote: > >> Hi Jeremy, >> >> Please see inline. >> >> On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) >> wrote: >> >> I don't want the size of the TLAB, which is ergonomically adjusted, to be >> tied to the sampling rate. There is no reason to do that. I want >> reasonable statistical sampling of the allocations. >> >> >> As I said explicitly in my e-mail, I totally agree with this. Which is >> why I never suggested to resize TLABs in order to vary the sampling rate. >> (Apologies if my e-mail was not clear.) >> > > My fault - I misread it. Doesn't your proposal miss out of TLAB allocs > entirely > > > This is correct: We?ll also have to intercept the outside-TLAB allocs. > But, IMHO, this is a feature as it?s helpful to know how many (and which) > allocations happen outside TLABs. These are generally very infrequent (and > slow anyway), so sampling all of those, instead of only sampling some of > them, does not have much of an overhead. But, you could also do sampling > for the outside-TLAB allocs too, if you want: just accumulate their size on > a separate per-thread counter and sample the one that bumps that counter > goes over a limit. > > An additional observation (orthogonal to the main point, but I thought I?d > mention it anyway): For the outside-TLAB allocs it?d be helpful to also > know which generation the object ended up in (e.g., young gen or > direct-to-old-gen). This is very helpful in some situations when you?re > trying to work out which allocation(s) grew the old gen occupancy between > two young GCs. > > FWIW, the existing JFR events follow the approach I described above: > > * one event for each new TLAB + first alloc in that TLAB (my proposal > basically generalizes this and removes the 1-1 relationship between object > alloc sampling and new TLAB operation) > > * one event for all allocs outside a TLAB > > I think the above separation is helpful. But if you think it could confuse > users, you can of course easily just combine the information (but I > strongly believe it?s better to report the information separately). > > > (and, in fact, not work if TLAB support is turned off)? > > > Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a genuine > question.) > > > I might be missing something obvious (and see my response below). > > > > >> All this requires is a separate counter that is set to the next sampling >> interval, and decremented when an allocation happens, which goes into a >> slow path when the decrement hits 0. Doing a subtraction and a pointer >> bump in allocation instead of just a pointer bump is basically free. >> >> >> Maybe on intel is cheap, but maybe it?s not on other platforms that other >> folks care about. >> > Really? A memory read and a subtraction? Which architectures care about > that? > > > I was not concerned with the read and subtraction, I was more concerned > with the conditional that follows them (intel has great branch prediction). > > And a personal pet peeve (based on past experience): How many ?free? > instructions do you have to add before they are not free any more? > > > > Again, notice that no one has complained about the addition that was added > for total bytes allocated per thread. I note that was actually added in > the 6u20 timeframe. > > Note that it has been doing an additional addition (to keep track of per >> thread allocation) as part of allocation since Java 7, >> >> >> Interesting. I hadn?t realized that. Does that keep track of total size >> allocated per thread or number of allocated objects per thread? If it?s the >> former, why isn?t it possible to calculate that from the TLABs information? >> > > Total size allocated per thread. It isn't possible to calculate that from > the TLAB because of out-of-TLAB allocation > > > The allocating Thread is passed to the slow (outside-TLAB) alloc path so > it would be trivial to update the per-thread allocation stats from there > too (in fact, it does; see below). > > > (and hypothetically disabled TLABs). > > > Anyone cares? :-) > > > > For some reason, they never included it in the ThreadMXBean interface, but > it is in com.sun.management.ThreadMXBean, so you can cast your ThreadMXBean > to a com.sun.management.ThreadMXBean and call getThreadAllocatedBytes() on > it. > > > Thanks for the tip. I?ll look into this... > > and no one has complained. >> >> I'm not worried about the ease of implementation here, because we've >> already implemented it. >> >> >> Yeah, but someone will have to maintain it moving forward. >> > > I've been maintaining it internally to Google for 5 years. It's actually > pretty self-contained. The only work involved is when they refactor > something (so I've had to move it), or when a bug in the existing > implementation is discovered. It is very closely parallel to the TLAB > code, which doesn't change much / at all. > > > The TLAB code has really not changed much for a while. ;-) (but haven?t > looked at the JDK 9 source very closely though?) > > It hasn't even been hard for us to do the forward port, except when the >> relevant Hotspot code is significantly refactored. >> >> We can also turn the sampling off, if we want. We can set the sampling >> rate to 2^32, have the sampling code do nothing, and no one will ever >> notice. >> >> >> You still have extra instructions in the allocation path, so it?s not >> turned off (i.e., you have the tax without any benefit). >> > > Hey, you have a counter in your allocation path you've never noticed, > which none of your code uses. Pipelining is a wonderful thing. :) > > > See above re: ?free? instructions. > > > >> In fact, we could just have the sampling code do nothing, and no one >> would ever notice. >> >> Honestly, no one ever notices the overhead of the sampling, anyway. JDK8 >> made it more expensive to grab a stack trace (the cost became proportional >> to the number of loaded classes), but we have a patch that mitigates that, >> which we would also be happy to upstream. >> >> As for the other concern: my concern about *just* having the callback >> mechanism is that there is quite a lot you can't do from user code during >> an allocation, because of lack of access to JNI. >> >> >> Maybe I missed something. Are the callbacks in Java? I.e., do you call >> them using JNI from the slow path you call directly from the allocation >> code? >> > (For context: this referred to the hypothetical feature where we can > provide a callback that invokes some code from allocation.) > > (It's not actually hypothetical, because we've already implemented it, but > let's call it hypothetical for the moment.) > > > OK. > > > > We invoke native code. You can't invoke any Java code during allocation, > including calling JNI methods, because that would make allocation > potentially reentrant, which doesn't work for all sorts of reasons. > > > That?s what I was worried about?. > > > The native code doesn't even get passed a JNIEnv * - there is nothing it > can do with it without making the VM crash a lot. > > > So, thanks for the clarification. Being able to attach a callback to this > in, say, the JVM it?d be totally fine. I was worried that you wanted to > call Java. :-) > > > > Or, rather, you might be able to do that, but it would take a lot of > Hotspot rearchitecting. When I tried to do it, I realized it would be an > extremely deep dive. > > > I believe you. :-) > > > >> However, you can do pretty much anything from the VM itself. Crucially >> (for us), we don't just log the stack traces, we also keep track of which >> are live and which aren't. We can't do this in a callback, if the callback >> can't create weak refs to the object. >> >> What we do at Google is to have two methods: one that you pass a callback >> to (the callback gets invoked with a StackTraceData object, as I've defined >> above), and another that just tells you which sampled objects are still >> live. We could also add a third, which allowed a callback to set the >> sampling interval (basically, the VM would call it to get the integer >> number of bytes to be allocated before the next sample). >> >> Would people be amenable to that? It makes the code more complex, but, >> as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB >> object come from?"). >> >> >> Well, that 1GB object would have most likely been allocated outside a >> TLAB and you could have identified it by instrumenting the ?outside-of-TLAB >> allocation path? (just saying?). >> > > That's orthogonal to the point I was making in the quote above - the point > I was making there was that we want to be able to detect what sampled > objects are live. We can do that regardless of how we implement the > sampling (although it did involve my making a new kind of weak oop > processing mechanism inside the VM). > > > Yeah, I was thinking of doing something similar (tracking object > lifetimes, and other attributes, with WeakRefs). > > > > But to the question of whether we can just instrument the outside-of-tlab > allocation path... There are a few weirdnesses here. The first one that > jumps to mind is that there's also a fast path for allocating in the YG > outside of TLABs, if an object is too large to fit in the current TLAB. > Those objects would never get sampled. So "outside of tlab" doesn't always > mean "slow path". > > > CollectedHeap::common_mem_allocate_noinit() is the first-level of the slow > path called when a TLAB allocation fails because the object doesn?t fit in > the current TLAB. It checks (alocate_from_tlab() / > allocate_from_tlab_slow()) whether to refill the current TLAB or keep the > TLAB and delegate to the GC (mem_allocate()) to allocate the object outside > a TLAB (either in the young or old gen; the GC might also decide to do a > collection at this point if, say, the eden is full...). So, it depends on > what you mean by slow path but, yes, any alloocations that go through the > above path should be considered as ?slow path? allocations. > > One more piece of data: AllocTracer::send_allocation_outside_tlab_event() > (the JFR entry point for outside-TLAB allocs) is fired from > common_mem_allocate_noint(). So, if there are other non-TLAB allocation > paths outside that method, that entry point has been placed incorrectly > (it?s possible of course; but I think that it?s actually placed correctly). > > (note: I only looked at the JDK 8 sources, haven?t checked the JDK 9 > sources yet, the above might have been changed) > > BTW, when looking at the common_mem_allocate_noinit() code I noticed the > following: > THREAD->incr_allocated_bytes(size * HeapWordSize); > > (as predicted earlier) > > > > Another one that jumps to mind is that we don't know whether the > outside-of-TLAB path actually passes the sampling threshold, especially if > we let users configure the sampling threshold. So how would we know > whether to sample it? > > > See above (IMHO: sample all of them). > > > > You also have to keep track of the sampling interval in the code where we > allocate new TLABs, in case the sampling threshold is larger than the TLAB > size. That's not a big deal, of course. > > > Of course, but that?s kinda trivial. BTW, one approach here would be > ?given that refilling a TLAB is slow anyway, always sample the first object > in each TLAB irrespective of desired sampling frequence?. Another would be > ?don?t do that, I set the sampling frequency pretty low not to be flooded > with data when the TLABs are very small?. I have to say I?m in the latter > camp. > > > > > And, every time the TLAB code changes, we have to consider whether / how > those changes affect this sampling mechanism. > > > Yes, but how often does the TLAB code change? :-) > > > > I guess my larger point is that there are so many little corner cases with > TLAB allocation, including whether it even happens, that basing the > sampling strategy around it seems like a cop-out. > > > There are not many little corner cases. There are two cases: allocation > inside a TLAB, allocation outside a TLAB. The former is by far the most > common. The latter is generally very infrequent and has a well-defined code > path (I described it earlier). And, as I said, it could be very helpful and > informative to treat (and account for) the two cases separately. > > > And my belief is that the arguments against our strategy don't really hold > water, especially given the presence of the per-thread allocation counter > that no one noticed. > > > I?ve already addressed that. > > > > Heck, I've already had it reviewed internally by a Hotspot reviewer (Chuck > Rasbold). All we really need is to write an acceptable JEP, to adjust the > code based on the changes the community wants, and someone from Oracle > willing to say "yes". > > > > For reference, to keep track of sampling, the delta to C2 is about 150 LOC > (much of which is newlines-because-of-formatting for methods that take a > lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 > template interpreter is about 20 LOC, and the delta for the assembler is > about 40 LOC. It's not completely trivial, but the code hasn't changed > substantially in the 5 years since I wrote it (other than a couple of > bugfixes). > > Obviously, assembler/template interpreter would have to be dup'd across > platforms - we can do that for PPC and aarch64, on which we do active > development, at least. > > > I?ll again vote for the simplicity of having a simple change in only one > place (OK, two places?). > > > >> But, seriously, why didn?t you like my proposal? It can do anything your >> scheme can with fewer and simpler code changes. The only thing that it >> cannot do is to sample based on object count (i.e., every 100 objects) >> instead of based on object size (i.e., every 1MB of allocations). But I >> think doing sampling based on size is the right approach here (IMHO). >> > I agree that sampling based on size is the right approach. > > (And your approach is definitely simpler - I don't mean to discount it. > And if that's what it takes to get this feature accepted, we'll do it, but > I'll grumble about it.) > > > That?s fine. :-) > > Tony > > > > ----- > > Tony Printezis | JVM/GC Engineer / VM Team | Twitter > > @TonyPrintezis > tprintezis at twitter.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 05:37:24 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 22:37:24 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <20150625014838.00001605.ecki@zusammenkunft.net> Message-ID: On Thu, Jun 25, 2015 at 1:55 PM, Tony Printezis wrote: > Bernd, > > I like the idea of buffering up the sampled objects in, some data > structure. But I assume it?d have to be a per-thread data structure to > avoid conention issues. So, we?ll also need a periodic task that collects > all such data structures and makes them available (somehow) to whoever > wants to consume them? > This is easily done. But, per my last message, I don't think it should be the default. It can just be available as another callback, if you want it. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 05:38:43 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 22:38:43 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: > > > > On Thu, Jun 25, 2015 at 2:23 PM, Tony Printezis > wrote: > >> BTW, Could we get a reaction from the Oracle folks on this? Even though >> Jeremy and myself are proposing different implementation approaches, we >> both agree (and Jeremy please correct me on this) that having an allocation >> sampling mechanism that?s more flexible to what?s already in HotSpot (in >> particular: the sampling frequency not being tied to the TLAB size) will be >> a very helpful profiling feature. Is this something that we pursue to >> contribute? >> >> Yes. I think we are 90% in agreement. And I think being able to query for live objects is pretty helpful, too. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 05:47:30 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 22:47:30 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: On Thu, Jun 25, 2015 at 2:08 PM, Tony Printezis wrote: > Hi Kirk, > > (long time!) See inline. > > On June 25, 2015 at 2:54:04 AM, Kirk Pepperdine (kirk.pepperdine at gmail.com) > wrote: > > > But, seriously, why didn?t you like my proposal? It can do anything your > scheme can with fewer and simpler code changes. The only thing that it > cannot do is to sample based on object count (i.e., every 100 objects) > instead of based on object size (i.e., every 1MB of allocations). But I > think doing sampling based on size is the right approach here (IMHO). > > > I would think that the size based sampling would create a size based bias > in your sampling. > > > That?s actually true. And this could be good (if you?re interested in > what?s filling up your eden, the larger objects might be of more interest) > or bad (if you want to get a general idea of what?s being allocated, the > size bias might make you miss some types of objects / allocation sites). > Note that it catches both large objects and objects that are frequently allocated in the same way. Both of those are useful pieces of information. Particularly, if we find, say, 200 of the same stack trace, and we know they aren't in the live set, then we know we have a place in the code that generates a lot of garbage. That can be a useful piece of information for tuning. Since IME, it?s allocation frequency is more damaging to performance, I?d > prefer to see time boxed sampling > > > Do you mean ?sample every X ms, say?? > This is not impossible, but a little weird. The only obvious way I can think to do it without enormous overhead is having a thread that wakes up once every X ms and sets a shared location to 1. Then you check that shared location on every allocation. If it is 1, you go into a slow path where you try to CAS it to 0. If the CAS succeeds, take the sample. You could imagine some sampling problems caused by, say, thread priority issues. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Fri Jun 26 06:42:39 2015 From: david.holmes at oracle.com (David Holmes) Date: Fri, 26 Jun 2015 16:42:39 +1000 Subject: RFR 8071487: javax/management/monitor/GaugeMonitorDeadlockTest.java timed out In-Reply-To: <558BCF7D.6030001@oracle.com> References: <55882B75.1070804@oracle.com> <5588F702.20606@oracle.com> <55890BDA.8010104@oracle.com> <558A76D3.2010902@oracle.com> <558BCF7D.6030001@oracle.com> Message-ID: <558CF45F.7060200@oracle.com> Looks good! Thanks, David On 25/06/2015 7:53 PM, Jaroslav Bachorik wrote: > On 24.6.2015 11:22, David Holmes wrote: >> On 23/06/2015 5:33 PM, Jaroslav Bachorik wrote: >>> Hi David, >>> >>> On 23.6.2015 08:04, David Holmes wrote: >>>> On 23/06/2015 1:36 AM, Jaroslav Bachorik wrote: >>>>> Please, review the following test change >>>>> >>>>> Issue : https://bugs.openjdk.java.net/browse/JDK-8071487 >>>>> Webrev: http://cr.openjdk.java.net/~jbachorik/8071487/webrev.00 >>>>> >>>>> GaugeMonitorDeadlockTest fails intermittently due data race - the call >>>>> to monitorProxy.start() on L105 will eventually result in incrementing >>>>> the GetCount attribute. If that happens before L109 had the chance to >>>>> run the loop on L113-128 will become infinite. The initial value will >>>>> contain the already incremented GetCount value and GetCount attribute >>>>> value will not get further incremented. >>>> >>>> The reordering of the start() and initial observedProxy.getGetCount >>>> seems reasonable. >>>> >>>> The changes to the timeout handling less so. The alarm code doesn't >>>> look >>>> right to me. Each time you call check(true) in the loop you advance the >>>> time when the alarm "goes off". ??? >>> >>> Stupid me :( Thanks for catching this. >>> >>> http://cr.openjdk.java.net/~jbachorik/8071487/webrev.01 >> >> That looks better, but the Alarm class would benefit from some simple >> descriptive comments. I must admit I don't really see the need for >> introducing it. > > I expected to reuse it in some of the other JMX monitoring tests. But it > didn't turn out that way. > > I've removed the Alarm class from the test and just switched to using > Util.adjustTimeout() instead of doing the parsing of > "test.timeout.factor" system property by hand. > > http://cr.openjdk.java.net/~jbachorik/8071487/webrev.02 > > Thanks, > > -JB- > >> >> Thanks, >> David >> >>> -JB- >>> >>>> >>>> David >>>> >>>> >>>> >>>>> I took the liberty to fix the same issue in >>>>> test/javax/management/monitor/StringMonitorDeadlockTest.java, not >>>>> waiting for the real test failure. >>>>> >>>>> Thanks, >>>>> >>>>> -JB- >>> > From jeremymanson at google.com Fri Jun 26 06:51:56 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Thu, 25 Jun 2015 23:51:56 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: Another thought. Since: - It would be kind of surprising for Thread->allocated_bytes() to be different from the number used as the interval for tracking (e.g., if your interval is, say, 512K, you check allocated bytes, it says 0, you allocate 512K, you check allocated bytes, it says 512K, but no sample was taken), AND - We're already taking the maintenance hit to maintain the allocated bytes counter everywhere, Maybe a good compromise would be to piggyback on the allocated bytes counter? If the allocated bytes counter is at N, and we pick a next sampling interval of K, we set a per-thread variable to N+K, and everywhere we increment the allocated bytes counter, we just test to see if it is greater than N+K? This would add an additional load and another easily predicted branch, but no additional subtraction. Also, it would have very obvious and tractable modifications to make in existing places that already have logic for the counter, so there wouldn't be much of an additional maintenance burden. Finally, it would more-or-less address my concerns, because the non-TLAB fast paths I'm worried about are already instrumented for it. Jeremy On Thu, Jun 25, 2015 at 10:27 PM, Jeremy Manson wrote: > > > On Thu, Jun 25, 2015 at 1:28 PM, Tony Printezis > wrote: > >> Hi Jeremy, >> >> Inline. >> >> On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) >> wrote: >> >> >> >> On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis >> wrote: >> >>> Hi Jeremy, >>> >>> Please see inline. >>> >>> On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) >>> wrote: >>> >>> I don't want the size of the TLAB, which is ergonomically adjusted, to >>> be tied to the sampling rate. There is no reason to do that. I want >>> reasonable statistical sampling of the allocations. >>> >>> >>> As I said explicitly in my e-mail, I totally agree with this. Which is >>> why I never suggested to resize TLABs in order to vary the sampling rate. >>> (Apologies if my e-mail was not clear.) >>> >> >> My fault - I misread it. Doesn't your proposal miss out of TLAB allocs >> entirely >> >> >> This is correct: We?ll also have to intercept the outside-TLAB allocs. >> But, IMHO, this is a feature as it?s helpful to know how many (and which) >> allocations happen outside TLABs. These are generally very infrequent (and >> slow anyway), so sampling all of those, instead of only sampling some of >> them, does not have much of an overhead. But, you could also do sampling >> for the outside-TLAB allocs too, if you want: just accumulate their size on >> a separate per-thread counter and sample the one that bumps that counter >> goes over a limit. >> >> > The outside-TLAB allocations generally get caught anyway, because they > tend to be large enough to jump over the sample size immediately. > > >> An additional observation (orthogonal to the main point, but I thought >> I?d mention it anyway): For the outside-TLAB allocs it?d be helpful to also >> know which generation the object ended up in (e.g., young gen or >> direct-to-old-gen). This is very helpful in some situations when you?re >> trying to work out which allocation(s) grew the old gen occupancy between >> two young GCs. >> > > True. We don't have this implemented, but it would be reasonably > straightforward to glean it from the oop. > > >> FWIW, the existing JFR events follow the approach I described above: >> >> * one event for each new TLAB + first alloc in that TLAB (my proposal >> basically generalizes this and removes the 1-1 relationship between object >> alloc sampling and new TLAB operation) >> >> * one event for all allocs outside a TLAB >> >> I think the above separation is helpful. But if you think it could >> confuse users, you can of course easily just combine the information (but I >> strongly believe it?s better to report the information separately). >> > > I do think it would make a confusing API. It might make more sense to > have a reporting mechanism that had a set number of fields with very > concrete information (size, class, stacktrace), but allowed for > platform-specific metadata. We end up with a very long list of things we > want in the sample: generation (how do you describe a generation?), object > age (by number of GCs survived? What kind of GC?), was it a TLAB > allocation, etc. > > > (and, in fact, not work if TLAB support is turned off)? >> >> >> Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a >> genuine question.) >> > > I don't think they do. I have turned them off for various reasons > (usually, I'm trying to instrument allocations and I don't want to muck > about with thinking about TLABs), and the code paths seem a little crufty. > ISTR at some point finding something that clearly only worked by mistake, > but I can't remember now what it was. > > [snip] > > > >> However, you can do pretty much anything from the VM itself. Crucially >>> (for us), we don't just log the stack traces, we also keep track of which >>> are live and which aren't. We can't do this in a callback, if the callback >>> can't create weak refs to the object. >>> >>> What we do at Google is to have two methods: one that you pass a >>> callback to (the callback gets invoked with a StackTraceData object, as >>> I've defined above), and another that just tells you which sampled objects >>> are still live. We could also add a third, which allowed a callback to set >>> the sampling interval (basically, the VM would call it to get the integer >>> number of bytes to be allocated before the next sample). >>> >>> Would people be amenable to that? It makes the code more complex, but, >>> as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB >>> object come from?"). >>> >>> >>> Well, that 1GB object would have most likely been allocated outside a >>> TLAB and you could have identified it by instrumenting the ?outside-of-TLAB >>> allocation path? (just saying?). >>> >> >> That's orthogonal to the point I was making in the quote above - the >> point I was making there was that we want to be able to detect what sampled >> objects are live. We can do that regardless of how we implement the >> sampling (although it did involve my making a new kind of weak oop >> processing mechanism inside the VM). >> >> >> Yeah, I was thinking of doing something similar (tracking object >> lifetimes, and other attributes, with WeakRefs). >> > > We have all of that implemented, so hopefully I can save you the trouble. > :) > > But to the question of whether we can just instrument the outside-of-tlab >> allocation path... There are a few weirdnesses here. The first one that >> jumps to mind is that there's also a fast path for allocating in the YG >> outside of TLABs, if an object is too large to fit in the current TLAB. >> Those objects would never get sampled. So "outside of tlab" doesn't always >> mean "slow path". >> >> >> CollectedHeap::common_mem_allocate_noinit() is the first-level of the >> slow path called when a TLAB allocation fails because the object doesn?t >> fit in the current TLAB. It checks (alocate_from_tlab() / >> allocate_from_tlab_slow()) whether to refill the current TLAB or keep the >> TLAB and delegate to the GC (mem_allocate()) to allocate the object outside >> a TLAB (either in the young or old gen; the GC might also decide to do a >> collection at this point if, say, the eden is full...). So, it depends on >> what you mean by slow path but, yes, any alloocations that go through the >> above path should be considered as ?slow path? allocations. >> > > Let me be more specific. Here is a place where allocations go through a > fast path that is outside of a TLAB: > > > http://hg.openjdk.java.net/jdk9/dev/hotspot/file/972580a0eef8/src/cpu/x86/vm/templateTable_x86.cpp#l3759 > > If the object won't fit in the TLAB, but will fit in the Eden, it will be > allocated in the Eden, with hand-generated assembly. This case will be > entirely missed by sampling just the TLAB creation (or your variant) and > the slow path. I may be missing something about that code, but I can't > really see what it is. > > One more piece of data: AllocTracer::send_allocation_outside_tlab_event() >> (the JFR entry point for outside-TLAB allocs) is fired from >> common_mem_allocate_noint(). So, if there are other non-TLAB allocation >> paths outside that method, that entry point has been placed incorrectly >> (it?s possible of course; but I think that it?s actually placed correctly). >> > > What is happening in the line to which I referred, then? To me, it kind > of reads like "this is close enough to being TLAB allocation that I don't > care that it isn't". > > And that's really what's going on here. Your strategy is to tie what I see > as a platform feature to a particular implementation. If the > implementation changes, or if we really don't understand it as well as we > think we do, the whole thing falls on the floor. If we mention TLABs in > the docs, and TLABs do change, then it won't mean anything anymore. > > A particular example pops to mind: I believe Metronome doesn't have TLABs > at all. Is that correct? Can J9 developers implement this feature? > >> For reference, to keep track of sampling, the delta to C2 is about 150 >> LOC (much of which is newlines-because-of-formatting for methods that take >> a lot of parameters), the delta to C1 is about 60 LOC, the delta to each >> x86 template interpreter is about 20 LOC, and the delta for the assembler >> is about 40 LOC. It's not completely trivial, but the code hasn't >> changed substantially in the 5 years since I wrote it (other than a couple >> of bugfixes). >> >> Obviously, assembler/template interpreter would have to be dup'd across >> platforms - we can do that for PPC and aarch64, on which we do active >> development, at least. >> >> >> I?ll again vote for the simplicity of having a simple change in only one >> place (OK, two places?). >> > > This isn't a simple change anyway, if we're keeping track of live > references. We have to hook into reference processing - when a weak oop is > detected to be dead, we have to delete the metadata. And we have to change > JVMTI. > > Jeremy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Fri Jun 26 07:01:34 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Fri, 26 Jun 2015 09:01:34 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <97DB8E7B-887F-4D23-BB2D-9CE64D80F309@gmail.com> Hi Jeremy, Sorry I wasn?t so clear, it?s not about collection, it?s about allocation. In this regard it?s not about about size, it?s about the frequency. People tend allocate small objects frequently and they will avoid allocating large objects frequently. The assumption is, large is expensive but small isn?t. These event will show up using execution profilers but given the safe-point bias of execution profilers and other factors, it?s often clearer to view this problem using memory profilers. Kind regards, Kirk On Jun 25, 2015, at 7:34 PM, Jeremy Manson wrote: > Why would allocation frequency be more damaging to performance? Allocation is cheap, and as long as they become dead before the YG collection, it costs the same to collect one 1MB object as it does to collection 1000 1K objects. > > Jeremy > > On Wed, Jun 24, 2015 at 11:54 PM, Kirk Pepperdine wrote: >> >> But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). >> >> > > I would think that the size based sampling would create a size based bias in your sampling. Since IME, it?s allocation frequency is more damaging to performance, I?d prefer to see time boxed sampling > > Kind regards, > Kirk Pepperdine > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Fri Jun 26 07:08:36 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Fri, 26 Jun 2015 09:08:36 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <48939138-CF4A-4B43-A340-F17532C296B4@gmail.com> Hi Tony, >> >> I would think that the size based sampling would create a size based bias in your sampling. > > > That?s actually true. And this could be good (if you?re interested in what?s filling up your eden, the larger objects might be of more interest) or bad (if you want to get a general idea of what?s being allocated, the size bias might make you miss some types of objects / allocation sites). > > IME, if I see allocation rates > ~500MB/sec I know there are some significant wins I can make by tuning allocation hotspots. So, I?m interested in frequent allocations no matter what the size. > >> Since IME, it?s allocation frequency is more damaging to performance, I?d prefer to see time boxed sampling > > > Do you mean ?sample every X ms, say?? > > I don?t find that sampling rate has to be that high to find hot allocation sites. Kind regards, Kirk Pepperdine > > Tony > > > >> >> Kind regards, >> Kirk Pepperdine >> > > > ----- > > Tony Printezis | JVM/GC Engineer / VM Team | Twitter > > @TonyPrintezis > tprintezis at twitter.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Fri Jun 26 08:01:25 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Fri, 26 Jun 2015 10:01:25 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <49A86674-5291-4F4E-9B2C-7138E3BFC760@gmail.com> > > Do you mean ?sample every X ms, say?? > > > This is not impossible, but a little weird. Yes, you are right, this is weird and I don?t think is necessary for this type of profiling. Please ignore. Regards, Kirk -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Fri Jun 26 16:10:02 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Fri, 26 Jun 2015 09:10:02 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <97DB8E7B-887F-4D23-BB2D-9CE64D80F309@gmail.com> References: <97DB8E7B-887F-4D23-BB2D-9CE64D80F309@gmail.com> Message-ID: Hey Kirk, The answer to that is not to use safepoint-biased execution profilers, I think. For Hotspot, profilers can call AsyncGetCallTrace in a signal handler. That's what we do at Google, and it works just fine (although we keep having to fix AGCT to get decent output). Sample code: https://code.google.com/p/lightweight-java-profiler/ Another thing that users can do on Linux is use perf events, as long as the right agent is in place. This is somewhat underdocumented, but Stephane Eranian wrote some sample JVMTI code to enable it: https://lkml.org/lkml/2015/2/10/570 We have vague plans to do something with this (although right now the kernel support needed doesn't quite work with our development machines). Jeremy On Fri, Jun 26, 2015 at 12:01 AM, Kirk Pepperdine wrote: > Hi Jeremy, > > Sorry I wasn?t so clear, it?s not about collection, it?s about allocation. > In this regard it?s not about about size, it?s about the frequency. People > tend allocate small objects frequently and they will avoid allocating large > objects frequently. The assumption is, large is expensive but small isn?t. > These event will show up using execution profilers but given the safe-point > bias of execution profilers and other factors, it?s often clearer to view > this problem using memory profilers. > > Kind regards, > Kirk > > On Jun 25, 2015, at 7:34 PM, Jeremy Manson > wrote: > > Why would allocation frequency be more damaging to performance? > Allocation is cheap, and as long as they become dead before the YG > collection, it costs the same to collect one 1MB object as it does to > collection 1000 1K objects. > > Jeremy > > On Wed, Jun 24, 2015 at 11:54 PM, Kirk Pepperdine < > kirk.pepperdine at gmail.com> wrote: > >> >> But, seriously, why didn?t you like my proposal? It can do anything your >> scheme can with fewer and simpler code changes. The only thing that it >> cannot do is to sample based on object count (i.e., every 100 objects) >> instead of based on object size (i.e., every 1MB of allocations). But I >> think doing sampling based on size is the right approach here (IMHO). >> >> >> I would think that the size based sampling would create a size based bias >> in your sampling. Since IME, it?s allocation frequency is more damaging to >> performance, I?d prefer to see time boxed sampling >> >> Kind regards, >> Kirk Pepperdine >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karen.kinnear at oracle.com Fri Jun 26 16:15:47 2015 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Fri, 26 Jun 2015 12:15:47 -0400 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: <1E306809-EDFE-42CB-BFE5-6196FAEAF3C3@oracle.com> Thanks Jeremy - that helps - not such a big deal in terms of maintaining - so back to the bigger discussion. thanks, Karen On Jun 26, 2015, at 1:34 AM, Jeremy Manson wrote: > Karen, > > I understand your concerns. For reference, this is the additional code in the x86 assembler. There are very small stubs in C1 and the template interpreter to call out to this macro (forgive the gratuitous use of the string "google" - we sprinkle it around a bit because it makes it a little easier to distinguish our code from upstream code). > > #define GOOGLE_HEAP_MONITORING(ma, thread, var_size_in_bytes, con_size_in_bytes, object, t1, t2, sample_invocation) \ > do { \ > { \ > SkipIfEqual skip_if(ma, &GoogleHeapMonitor, 0); \ > Label skip_sample; \ > Register thr = thread; \ > if (!thr->is_valid()) { \ > NOT_LP64(assert(t1 != noreg, \ > "Need temporary register for constants")); \ > thr = NOT_LP64(t1) LP64_ONLY(r15_thread); \ > NOT_LP64(ma -> get_thread(thr);) \ > } \ > /* Trigger heap monitoring event */ \ > Address bus(thr, \ > JavaThread::google_bytes_until_sample_offset()); \ > \ > if (var_size_in_bytes->is_valid()) { \ > ma -> NOT_LP64(subl) LP64_ONLY(subq)(bus, var_size_in_bytes); \ > } else { \ > int csib = (con_size_in_bytes); \ > assert(t2 != noreg, \ > "Need temporary register for constants"); \ > ma -> NOT_LP64(movl) LP64_ONLY(mov64)(t2, csib); \ > ma -> NOT_LP64(subl) LP64_ONLY(subq)(bus, t2); \ > } \ > \ > ma -> jcc(Assembler::positive, skip_sample); \ > \ > { \ > sample_invocation \ > } \ > ma -> bind(skip_sample); \ > } \ > } while(0) > > It's not all that hard to port to additional architectures, but we'll have to think about it. > > Jeremy > > On Thu, Jun 25, 2015 at 1:41 PM, Karen Kinnear wrote: > Jeremy, > > Did I follow this correctly - that your approach modifies the compilers and interpreters and Tony's modifies the > common allocation code? > > Given that the number of compilers and interpreters and interpreter platforms keeps expanding - I'd like to > add a vote to have heap allocation profiling in common allocation code. > > thanks, > Karen > > On Jun 25, 2015, at 4:28 PM, Tony Printezis wrote: > >> Hi Jeremy, >> >> Inline. >> >> On June 24, 2015 at 7:26:55 PM, Jeremy Manson (jeremymanson at google.com) wrote: >> >>> >>> >>> On Wed, Jun 24, 2015 at 10:57 AM, Tony Printezis wrote: >>> Hi Jeremy, >>> >>> Please see inline. >>> >>> On June 23, 2015 at 7:22:13 PM, Jeremy Manson (jeremymanson at google.com) wrote: >>> >>>> I don't want the size of the TLAB, which is ergonomically adjusted, to be tied to the sampling rate. There is no reason to do that. I want reasonable statistical sampling of the allocations. >>> >>> >>> As I said explicitly in my e-mail, I totally agree with this. Which is why I never suggested to resize TLABs in order to vary the sampling rate. (Apologies if my e-mail was not clear.) >>> >>> >>> My fault - I misread it. Doesn't your proposal miss out of TLAB allocs entirely >> >> >> This is correct: We?ll also have to intercept the outside-TLAB allocs. But, IMHO, this is a feature as it?s helpful to know how many (and which) allocations happen outside TLABs. These are generally very infrequent (and slow anyway), so sampling all of those, instead of only sampling some of them, does not have much of an overhead. But, you could also do sampling for the outside-TLAB allocs too, if you want: just accumulate their size on a separate per-thread counter and sample the one that bumps that counter goes over a limit. >> >> An additional observation (orthogonal to the main point, but I thought I?d mention it anyway): For the outside-TLAB allocs it?d be helpful to also know which generation the object ended up in (e.g., young gen or direct-to-old-gen). This is very helpful in some situations when you?re trying to work out which allocation(s) grew the old gen occupancy between two young GCs. >> >> FWIW, the existing JFR events follow the approach I described above: >> >> * one event for each new TLAB + first alloc in that TLAB (my proposal basically generalizes this and removes the 1-1 relationship between object alloc sampling and new TLAB operation) >> >> * one event for all allocs outside a TLAB >> >> I think the above separation is helpful. But if you think it could confuse users, you can of course easily just combine the information (but I strongly believe it?s better to report the information separately). >> >> >> >>> (and, in fact, not work if TLAB support is turned off)? >> >> >> Who turns off TLABs? Is -UseTLAB even tested by Oracle? (This is a genuine question.) >> >> >> >>> I might be missing something obvious (and see my response below). >> >> >>> >>>> All this requires is a separate counter that is set to the next sampling interval, and decremented when an allocation happens, which goes into a slow path when the decrement hits 0. Doing a subtraction and a pointer bump in allocation instead of just a pointer bump is basically free. >>> >>> >>> Maybe on intel is cheap, but maybe it?s not on other platforms that other folks care about. >>> >>> Really? A memory read and a subtraction? Which architectures care about that? >> >> >> I was not concerned with the read and subtraction, I was more concerned with the conditional that follows them (intel has great branch prediction). >> >> And a personal pet peeve (based on past experience): How many ?free? instructions do you have to add before they are not free any more? >> >> >> >>> >>> Again, notice that no one has complained about the addition that was added for total bytes allocated per thread. I note that was actually added in the 6u20 timeframe. >>> >>>> Note that it has been doing an additional addition (to keep track of per thread allocation) as part of allocation since Java 7, >>> >>> >>> Interesting. I hadn?t realized that. Does that keep track of total size allocated per thread or number of allocated objects per thread? If it?s the former, why isn?t it possible to calculate that from the TLABs information? >>> >>> >>> Total size allocated per thread. It isn't possible to calculate that from the TLAB because of out-of-TLAB allocation >> >> >> The allocating Thread is passed to the slow (outside-TLAB) alloc path so it would be trivial to update the per-thread allocation stats from there too (in fact, it does; see below). >> >> >> >>> (and hypothetically disabled TLABs). >> >> >> Anyone cares? :-) >> >> >> >>> >>> For some reason, they never included it in the ThreadMXBean interface, but it is in com.sun.management.ThreadMXBean, so you can cast your ThreadMXBean to a com.sun.management.ThreadMXBean and call getThreadAllocatedBytes() on it. >>> >> >> Thanks for the tip. I?ll look into this... >> >>>> and no one has complained. >>>> >>>> I'm not worried about the ease of implementation here, because we've already implemented it. >>> >>> >>> Yeah, but someone will have to maintain it moving forward. >>> >>> >>> I've been maintaining it internally to Google for 5 years. It's actually pretty self-contained. The only work involved is when they refactor something (so I've had to move it), or when a bug in the existing implementation is discovered. It is very closely parallel to the TLAB code, which doesn't change much / at all. >>> >> >> The TLAB code has really not changed much for a while. ;-) (but haven?t looked at the JDK 9 source very closely though?) >> >>>> It hasn't even been hard for us to do the forward port, except when the relevant Hotspot code is significantly refactored. >>>> >>>> We can also turn the sampling off, if we want. We can set the sampling rate to 2^32, have the sampling code do nothing, and no one will ever notice. >>> >>> >>> You still have extra instructions in the allocation path, so it?s not turned off (i.e., you have the tax without any benefit). >>> >>> >>> Hey, you have a counter in your allocation path you've never noticed, which none of your code uses. Pipelining is a wonderful thing. :) >> >> >> See above re: ?free? instructions. >> >> >> >>>> >>>> In fact, we could just have the sampling code do nothing, and no one would ever notice. >>>> >>>> Honestly, no one ever notices the overhead of the sampling, anyway. JDK8 made it more expensive to grab a stack trace (the cost became proportional to the number of loaded classes), but we have a patch that mitigates that, which we would also be happy to upstream. >>>> >>>> As for the other concern: my concern about *just* having the callback mechanism is that there is quite a lot you can't do from user code during an allocation, because of lack of access to JNI. >>> >>> >>> Maybe I missed something. Are the callbacks in Java? I.e., do you call them using JNI from the slow path you call directly from the allocation code? >>> >>> (For context: this referred to the hypothetical feature where we can provide a callback that invokes some code from allocation.) >>> >>> (It's not actually hypothetical, because we've already implemented it, but let's call it hypothetical for the moment.) >> >> >> OK. >> >> >> >>> >>> We invoke native code. You can't invoke any Java code during allocation, including calling JNI methods, because that would make allocation potentially reentrant, which doesn't work for all sorts of reasons. >> >> >> That?s what I was worried about?. >> >> >> >>> The native code doesn't even get passed a JNIEnv * - there is nothing it can do with it without making the VM crash a lot. >> >> >> So, thanks for the clarification. Being able to attach a callback to this in, say, the JVM it?d be totally fine. I was worried that you wanted to call Java. :-) >> >> >> >>> >>> Or, rather, you might be able to do that, but it would take a lot of Hotspot rearchitecting. When I tried to do it, I realized it would be an extremely deep dive. >> >> >> I believe you. :-) >> >> >> >>>> >>>> However, you can do pretty much anything from the VM itself. Crucially (for us), we don't just log the stack traces, we also keep track of which are live and which aren't. We can't do this in a callback, if the callback can't create weak refs to the object. >>>> >>>> What we do at Google is to have two methods: one that you pass a callback to (the callback gets invoked with a StackTraceData object, as I've defined above), and another that just tells you which sampled objects are still live. We could also add a third, which allowed a callback to set the sampling interval (basically, the VM would call it to get the integer number of bytes to be allocated before the next sample). >>>> >>>> Would people be amenable to that? It makes the code more complex, but, as I say, it's nice for detecting memory leaks ("Hey! Where did that 1 GB object come from?"). >>> >>> >>> Well, that 1GB object would have most likely been allocated outside a TLAB and you could have identified it by instrumenting the ?outside-of-TLAB allocation path? (just saying?). >>> >>> >>> That's orthogonal to the point I was making in the quote above - the point I was making there was that we want to be able to detect what sampled objects are live. We can do that regardless of how we implement the sampling (although it did involve my making a new kind of weak oop processing mechanism inside the VM). >> >> >> Yeah, I was thinking of doing something similar (tracking object lifetimes, and other attributes, with WeakRefs). >> >> >> >>> >>> But to the question of whether we can just instrument the outside-of-tlab allocation path... There are a few weirdnesses here. The first one that jumps to mind is that there's also a fast path for allocating in the YG outside of TLABs, if an object is too large to fit in the current TLAB. Those objects would never get sampled. So "outside of tlab" doesn't always mean "slow path". >> >> >> CollectedHeap::common_mem_allocate_noinit() is the first-level of the slow path called when a TLAB allocation fails because the object doesn?t fit in the current TLAB. It checks (alocate_from_tlab() / allocate_from_tlab_slow()) whether to refill the current TLAB or keep the TLAB and delegate to the GC (mem_allocate()) to allocate the object outside a TLAB (either in the young or old gen; the GC might also decide to do a collection at this point if, say, the eden is full...). So, it depends on what you mean by slow path but, yes, any alloocations that go through the above path should be considered as ?slow path? allocations. >> >> One more piece of data: AllocTracer::send_allocation_outside_tlab_event() (the JFR entry point for outside-TLAB allocs) is fired from common_mem_allocate_noint(). So, if there are other non-TLAB allocation paths outside that method, that entry point has been placed incorrectly (it?s possible of course; but I think that it?s actually placed correctly). >> >> (note: I only looked at the JDK 8 sources, haven?t checked the JDK 9 sources yet, the above might have been changed) >> >> BTW, when looking at the common_mem_allocate_noinit() code I noticed the following: >> >> THREAD->incr_allocated_bytes(size * HeapWordSize); >> (as predicted earlier) >> >> >> >>> >>> Another one that jumps to mind is that we don't know whether the outside-of-TLAB path actually passes the sampling threshold, especially if we let users configure the sampling threshold. So how would we know whether to sample it? >> >> >> See above (IMHO: sample all of them). >> >> >> >>> >>> You also have to keep track of the sampling interval in the code where we allocate new TLABs, in case the sampling threshold is larger than the TLAB size. That's not a big deal, of course. >> >> >> Of course, but that?s kinda trivial. BTW, one approach here would be ?given that refilling a TLAB is slow anyway, always sample the first object in each TLAB irrespective of desired sampling frequence?. Another would be ?don?t do that, I set the sampling frequency pretty low not to be flooded with data when the TLABs are very small?. I have to say I?m in the latter camp. >> >> >> >>> >>> >>> And, every time the TLAB code changes, we have to consider whether / how those changes affect this sampling mechanism. >> >> >> Yes, but how often does the TLAB code change? :-) >> >> >> >>> >>> I guess my larger point is that there are so many little corner cases with TLAB allocation, including whether it even happens, that basing the sampling strategy around it seems like a cop-out. >> >> >> There are not many little corner cases. There are two cases: allocation inside a TLAB, allocation outside a TLAB. The former is by far the most common. The latter is generally very infrequent and has a well-defined code path (I described it earlier). And, as I said, it could be very helpful and informative to treat (and account for) the two cases separately. >> >> >> >>> And my belief is that the arguments against our strategy don't really hold water, especially given the presence of the per-thread allocation counter that no one noticed. >> >> >> I?ve already addressed that. >> >> >> >>> >>> Heck, I've already had it reviewed internally by a Hotspot reviewer (Chuck Rasbold). All we really need is to write an acceptable JEP, to adjust the code based on the changes the community wants, and someone from Oracle willing to say "yes". >> >> >>> >>> For reference, to keep track of sampling, the delta to C2 is about 150 LOC (much of which is newlines-because-of-formatting for methods that take a lot of parameters), the delta to C1 is about 60 LOC, the delta to each x86 template interpreter is about 20 LOC, and the delta for the assembler is about 40 LOC. It's not completely trivial, but the code hasn't changed substantially in the 5 years since I wrote it (other than a couple of bugfixes). >>> >>> Obviously, assembler/template interpreter would have to be dup'd across platforms - we can do that for PPC and aarch64, on which we do active development, at least. >> >> >> I?ll again vote for the simplicity of having a simple change in only one place (OK, two places?). >> >> >> >>> >>> But, seriously, why didn?t you like my proposal? It can do anything your scheme can with fewer and simpler code changes. The only thing that it cannot do is to sample based on object count (i.e., every 100 objects) instead of based on object size (i.e., every 1MB of allocations). But I think doing sampling based on size is the right approach here (IMHO). >>> >>> I agree that sampling based on size is the right approach. >>> >>> (And your approach is definitely simpler - I don't mean to discount it. And if that's what it takes to get this feature accepted, we'll do it, but I'll grumble about it.) >> >> >> That?s fine. :-) >> >> Tony >> >> >> >> >> ----- >> >> Tony Printezis | JVM/GC Engineer / VM Team | Twitter >> >> @TonyPrintezis >> tprintezis at twitter.com >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serguei.spitsyn at oracle.com Sat Jun 27 00:15:31 2015 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 26 Jun 2015 17:15:31 -0700 Subject: RFR(S): JDK-8059038 Create new launcher for SA tools In-Reply-To: <558AA47E.3090902@oracle.com> References: <558967C9.9070902@oracle.com> <558A40DF.6040201@oracle.com> <558AA47E.3090902@oracle.com> Message-ID: <558DEB23.8000401@oracle.com> Hi Dmitry, Thank you for the update! The SALauncher.java changes are really nice. I have just couple of small comments. agent/src/share/classes/sun/jvm/hotspot/SALauncher.java 343 // Run tmtools 344 if (args[0].equals("jstack")) { 345 runJSTACK(oldArgs); Why the comment says "Run tmtools", not jstack? BTW, other fragments have no such a comment which is Ok at it is obvious. agent/src/share/classes/sun/jvm/hotspot/SAGetopt.java There are no checks of the carg length in several places where it is needed: 61 if (_argv[_optind].charAt(0) == '-') { 112 if (carg.charAt(0) != '-' || carg.equals("--")) { 117 if (carg.charAt(0) == '-' && carg.charAt(1) == '-') { 124 carg = carg.substring(2); 136 ch = carg.charAt(_optopt); 139 ch = carg.charAt(_optopt); Otherwise, the fix looks good. Thanks, Serguei On 6/24/15 5:37 AM, Dmitry Samersoff wrote: > Serguei, > > Thank you for the review. > > New webrev is here: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.05/ > > I didn't change naming convention in SAGetoptTest.java and keep a_opt, > b_opt etc as it gives better readability. Other concerns are addressed. > > BasicLauncherTest changed to use LingeredApp from testlib. > > -Dmitry > > > On 2015-06-24 08:32, serguei.spitsyn at oracle.com wrote: >> Hi Dmitry, >> >> Some quick minor comments. >> >> hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SAGetopt.java >> >> Formatting is wrong: >> >> 57 if (_optind >_argv.length) { >> >> 71 String[] ca = carg.split("=",2); >> >> 80 if (los.contains(ca[0]+"=")) { >> >> >> Need to use camel case for java method names: >> >> 55 private void extract_optarg(String opt) { >> >> 69 private String process_long_options(String carg, String[] longOptStr) { >> >> >> Need to use quotes for '-': >> >> 109 // End of option batch like -abc reached, expect option to start from - >> >> Example is: >> >> 133 // At this point carg[0] contains '-' >> >> >> Wrong indent at 87, 139, 120-121: >> >> 85 else { >> 86 // Mixed style options --file name >> 87 extract_optarg(ca[0]); >> 88 } >> >> 138 else { >> 139 ch = carg.charAt(_optopt); >> 140 } >> >> 119 if (longOptStr == null || longOptStr.length == 0) { >> 120 // No long options specified, stop options processing >> 121 return null; >> 122 } >> >> >> >> hotspot_webrev/agent/src/share/classes/sun/jvm/hotspot/SALauncher.java >> >> Uninitialized local: >> >> 128 String s; >> >> Need to use camel case: >> >> 126 String exe_or_pid = null; >> >> >> The main method is too long, I'd suggest to split with the sub-methods for: >> clhsdb, hsdb, jstack, jmap, jinfo >> >> >> jdk_webrev/test/sun/tools/jhsdb/BasicLauncherTest.java >> >> Wrong indent at 82, 85: >> >> 80 return toolProcess.exitValue(); >> 81 } finally { >> 82 theApp.stopApp(); >> 83 } >> 84 } catch (IOException | InterruptedException ex) { >> 85 throw new RuntimeException("Test ERROR " + ex, ex); >> 86 } >> >> >> I do not understand what is the need for nested try statements, just one >> try would be enough: >> >> 54 System.out.println("Starting LingeredApp"); >> 55 try { >> 56 try { >> . . . >> 81 } finally { >> 82 theApp.stopApp(); >> 83 } >> 84 } catch (IOException | InterruptedException ex) { >> 85 throw new RuntimeException("Test ERROR " + ex, ex); >> 86 } >> >> 98 try { >> 99 try { >> . . . >> 116 } finally { >> 117 theApp.stopApp(); >> 118 } >> 119 } catch (Exception ex) { >> 120 throw new RuntimeException("Test ERROR " + ex, ex); >> 121 } >> >> >> Why do you catch exceptions and throw the RuntimeException's in the >> launch() methods >> but catch the IOException in main? Would it be better to catch any >> Exception? >> >> Too many empty lines: >> >> 88 >> 89 >> 90 >> >> >> jdk_webrev/test/sun/tools/jhsdb/LingeredApp.java >> >> Too many empty lines: >> >> 275 >> 276 >> >> 369 >> >> >> jdk_webrev/test/sun/tools/jhsdb/SAGetoptTest.java >> >> Need to use Java naming convention: >> >> 36 private static boolean a_opt; >> 37 private static boolean b_opt; >> 38 private static boolean c_opt; >> 39 private static boolean e_opt; >> 40 private static boolean mixed_opt; >> 41 >> 42 private static String d_value; >> 43 private static String exe_value; >> 44 private static String core_value; >> >> Wrong indent 2 instead of 4: >> >> 70 if (s.equals("a")) { >> 71 a_opt = true; >> 72 continue; >> 73 } >> 74 >> 75 if (s.equals("b")) { >> 76 b_opt = true; >> 77 continue; >> 78 } >> 79 >> 80 if (s.equals("c")) { >> 81 c_opt = true; >> 82 continue; >> 83 } >> 84 >> 85 if (s.equals("e")) { >> 86 e_opt = true; >> 87 continue; >> 88 } >> 89 >> 90 if (s.equals("mixed")) { >> 91 mixed_opt = true; >> 92 continue; >> 93 } >> >> >> Thanks, >> Serguei >> >> >> On 6/23/15 7:06 AM, Dmitry Samersoff wrote: >>> Hi Everybody, >>> >>> Please review the changes: >>> >>> http://cr.openjdk.java.net/~dsamersoff/JDK-8059038/webrev.04/ >>> >>> I'm about to file CCC request for it but would like to get internal >>> feedback before that. >>> >>> This fix is introducing native launcher jhsdb for serviceability agent. >>> >>> jhsdb >>> >>> will launch command line debugger clhsdb >>> >>> jhsdb jstack file core >>> jhsdb jmap file core >>> jhsdb jinfo file core >>> >>> will launch corresponding SA based utility. >>> >>> >>> -Dmitry >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Sat Jun 27 10:16:02 2015 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Sat, 27 Jun 2015 12:16:02 +0200 Subject: Low-Overhead Heap Profiling In-Reply-To: References: <97DB8E7B-887F-4D23-BB2D-9CE64D80F309@gmail.com> Message-ID: <3C0B0DD1-5963-48A3-99BF-B071E9E21756@gmail.com> Hi Jeremy, > > The answer to that is not to use safepoint-biased execution profilers, I think. Thank you for the advice. I?ve been advocating for a number of years that people understand the sampling bias that exists in all profilers. I also would consider safe point bias being the most egregious form or sampling possible. This is why I?m very happy to see Richard?s Honest profiler (using AsyncGetCallTrace) show up and also to have inspired Pierre Laporte to start work on a lock profiler project at our (un)conference (jCrete). As an aside, we hope jCrete can help inspire and support more efforts like this. My desire here is to ensure that minimize the effects of sample bias on the results. In this regards, I like the approach you are proposing. My concern with Tony?s approach what that it would introduce a size based sampling bias. As I mentioned before, GC costs is one thing to consider however I?m more concerned with frequency related allocation costs and it?s effect on application throughputs. Also, I think we have other techniques that can infer where an allocation has taken place be it in a tlab or even in tenured. I?m not sure that we have to profile for this type of information though if we can get it for almost free?.. Kind regards, Kirk From jeremymanson at google.com Sun Jun 28 16:32:16 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Sun, 28 Jun 2015 09:32:16 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: <3C0B0DD1-5963-48A3-99BF-B071E9E21756@gmail.com> References: <97DB8E7B-887F-4D23-BB2D-9CE64D80F309@gmail.com> <3C0B0DD1-5963-48A3-99BF-B071E9E21756@gmail.com> Message-ID: I'm glad to see Richard's Honest Profiler, because it looks like he cadged my sample code and made a real profiler out of it. It means I may not have to migrate my sample code from code.google.com before the turndown. :) Jeremy On Sat, Jun 27, 2015 at 3:16 AM, Kirk Pepperdine wrote: > Hi Jeremy, > > > > > The answer to that is not to use safepoint-biased execution profilers, I > think. > > Thank you for the advice. I?ve been advocating for a number of years that > people understand the sampling bias that exists in all profilers. I also > would consider safe point bias being the most egregious form or sampling > possible. This is why I?m very happy to see Richard?s Honest profiler > (using AsyncGetCallTrace) show up and also to have inspired Pierre Laporte > to start work on a lock profiler project at our (un)conference (jCrete). As > an aside, we hope jCrete can help inspire and support more efforts like > this. > > My desire here is to ensure that minimize the effects of sample bias on > the results. In this regards, I like the approach you are proposing. My > concern with Tony?s approach what that it would introduce a size based > sampling bias. As I mentioned before, GC costs is one thing to consider > however I?m more concerned with frequency related allocation costs and it?s > effect on application throughputs. > > Also, I think we have other techniques that can infer where an allocation > has taken place be it in a tlab or even in tenured. I?m not sure that we > have to profile for this type of information though if we can get it for > almost free?.. > > Kind regards, > Kirk > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremymanson at google.com Sun Jun 28 16:38:32 2015 From: jeremymanson at google.com (Jeremy Manson) Date: Sun, 28 Jun 2015 09:38:32 -0700 Subject: Low-Overhead Heap Profiling In-Reply-To: References: Message-ID: > > > > On Thu, Jun 25, 2015 at 2:23 PM, Tony Printezis > wrote: > >> BTW, Could we get a reaction from the Oracle folks on this? >> > I contacted Staffan (who is probably the right person) offline, and he is, apparently, on vacation until the beginning of August. Unless there is someone else at Oracle who might be the right person, we might have to hold off on a final decision until then. In the meantime, we can probably continue the conversation - I might draft a JEP. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: