From jesper.wilhelmsson at oracle.com Sat Jun 1 09:50:28 2013 From: jesper.wilhelmsson at oracle.com (jesper.wilhelmsson at oracle.com) Date: Sat, 01 Jun 2013 09:50:28 +0000 Subject: hg: hsx/hotspot-gc/hotspot: 28 new changesets Message-ID: <20130601095243.D033E48EB1@hg.openjdk.java.net> Changeset: ad47de214f0c Author: katleman Date: 2013-05-23 10:47 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/ad47de214f0c Added tag jdk8-b91 for changeset 7cbdf0e3725c ! .hgtags Changeset: 38da9f4f6709 Author: amurillo Date: 2013-05-24 09:25 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/38da9f4f6709 Merge - agent/src/share/classes/sun/jvm/hotspot/debugger/cdbg/basic/amd64/AMD64CFrame.java - agent/src/share/classes/sun/jvm/hotspot/debugger/cdbg/basic/x86/X86CFrame.java - test/runtime/7158804/Test7158804.sh - test/runtime/8003985/Test8003985.java Changeset: 092018493d3b Author: amurillo Date: 2013-05-24 09:25 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/092018493d3b Added tag hs25-b34 for changeset 38da9f4f6709 ! .hgtags Changeset: 194b27b865bc Author: amurillo Date: 2013-05-24 09:35 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/194b27b865bc 8015305: new hotspot build - hs25-b35 Reviewed-by: jcoomes ! make/hotspot_version Changeset: ccdecfece956 Author: bharadwaj Date: 2013-05-21 16:17 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/ccdecfece956 8014059: JSR292: Failed to reject invalid class cplmhl00201m28n Summary: Restrict reference of interface methods by invokestatic and invokespecial to classfile version 52 or later. Reviewed-by: kvn, hseigel ! src/share/vm/classfile/classFileParser.cpp Changeset: f54c85acc043 Author: mikael Date: 2013-05-21 09:43 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/f54c85acc043 8013726: runtime/memory/ReserveMemory.java fails due to 'assert(bytes % os::vm_allocation_granularity() == 0) failed: reserve block size' Summary: Fix regression test to work on all platforms Reviewed-by: ctornqvi, dholmes ! src/share/vm/prims/whitebox.cpp ! test/runtime/memory/ReserveMemory.java ! test/testlibrary/whitebox/sun/hotspot/WhiteBox.java Changeset: 1a07e086ff28 Author: dholmes Date: 2013-05-21 19:52 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/1a07e086ff28 Merge Changeset: 6bd680e9ea35 Author: coleenp Date: 2013-05-22 14:37 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/6bd680e9ea35 8003421: NPG: Move oops out of InstanceKlass into mirror Summary: Inject protection_domain, signers, init_lock into java_lang_Class Reviewed-by: stefank, dholmes, sla ! agent/src/share/classes/sun/jvm/hotspot/memory/DictionaryEntry.java ! agent/src/share/classes/sun/jvm/hotspot/oops/InstanceKlass.java ! agent/src/share/classes/sun/jvm/hotspot/utilities/HeapGXLWriter.java ! agent/src/share/classes/sun/jvm/hotspot/utilities/HeapHprofBinWriter.java ! agent/src/share/classes/sun/jvm/hotspot/utilities/soql/JSJavaInstanceKlass.java ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/classfile/javaClasses.cpp ! src/share/vm/classfile/javaClasses.hpp ! src/share/vm/classfile/vmSymbols.hpp ! src/share/vm/oops/arrayKlass.cpp ! src/share/vm/oops/instanceKlass.cpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/oops/klass.cpp ! src/share/vm/oops/klass.hpp ! src/share/vm/oops/objArrayKlass.hpp ! src/share/vm/oops/typeArrayKlass.hpp ! src/share/vm/prims/jvm.cpp ! src/share/vm/runtime/vmStructs.cpp Changeset: 699d9df07e59 Author: ctornqvi Date: 2013-05-23 17:39 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/699d9df07e59 8009576: Test returns ClassNotFoundException Summary: Small classpath fix and move tests into open Reviewed-by: mgerdin, zgu + test/runtime/Metaspace/FragmentMetaspace.java + test/runtime/Metaspace/FragmentMetaspaceSimple.java + test/runtime/Metaspace/classes/test/Empty.java + test/runtime/testlibrary/GeneratedClassLoader.java Changeset: b7fa10a3a69a Author: sspitsyn Date: 2013-05-23 23:04 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/b7fa10a3a69a 8014288: perf regression in nashorn JDK-8008448.js test after 8008511 changes Summary: The fix of perf regression is to use method_idnum() for direct indexing into NMT Reviewed-by: twisti, kvn, coleenp, dholmes Contributed-by: serguei.spitsyn at oracle.com ! src/share/vm/oops/instanceKlass.cpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/prims/methodHandles.cpp ! src/share/vm/prims/methodHandles.hpp Changeset: cd83e1d98347 Author: dcubed Date: 2013-05-24 10:21 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/cd83e1d98347 Merge Changeset: 6c138b9851fb Author: sspitsyn Date: 2013-05-24 17:36 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/6c138b9851fb 8013945: CMS fatal error: must own lock MemberNameTable_lock Summary: The "delete mnt" needs to grab MemberNameTable_lock if !SafepointSynchronize::is_at_safepoint() Reviewed-by: sla, mgerdin, dholmes, jmasa Contributed-by: serguei.spitsyn at oracle.com ! src/share/vm/oops/instanceKlass.cpp Changeset: 3970971c91e0 Author: shade Date: 2013-05-27 12:49 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/3970971c91e0 8015270: @Contended: fix multiple issues in the layout code Summary: field count handling fixed, has_nonstatic_fields invariant fixed, oop map overrun fixed; new asserts Reviewed-by: kvn, dcubed, coleenp ! src/share/vm/classfile/classFileParser.cpp + test/runtime/contended/HasNonStatic.java + test/runtime/contended/OopMaps.java Changeset: a213d425d87a Author: ctornqvi Date: 2013-05-28 15:08 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/a213d425d87a 8015329: Print reason for failed MiniDumpWriteDump() call Summary: Printing both result from GetLastError and text representation of error. Also changed so that we produce dumps by default on client versions of Windows when running with a debug build. Also reviewed by peter.allwin at oracle.com Reviewed-by: sla, dholmes ! src/os/windows/vm/os_windows.cpp Changeset: 51af5fae397d Author: ccheung Date: 2013-05-24 17:19 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/51af5fae397d 8015265: revise the fix for 8007037 Reviewed-by: sspitsyn, dholmes, dcubed ! src/share/vm/oops/constantPool.cpp Changeset: 4cc7d4d5dc92 Author: zgu Date: 2013-05-28 08:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/4cc7d4d5dc92 Merge Changeset: 01c2bdd24bb5 Author: shade Date: 2013-05-28 19:54 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/01c2bdd24bb5 8015493: runtime/contended/OopMaps.java fails with OutOfMemory Summary: limit the memory footprint to dodge OutOfMemory errors. Reviewed-by: dcubed, ctornqvi, iignatyev ! test/runtime/contended/OopMaps.java Changeset: 9ea643afcaaf Author: dcubed Date: 2013-05-28 11:35 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/9ea643afcaaf Merge Changeset: dcb062bea05b Author: jprovino Date: 2013-05-28 11:17 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/dcb062bea05b 8013461: There is a symbol AsyncGetCallTrace in libjvm.symbols that does not exist in minimal/libjvm.a when DEBUG_LEVEL == release Summary: AsyncGetCallTrace is needed in libjvm.symbols so that programs which reference it can build correctly. Reviewed-by: dholmes, bobv ! make/excludeSrc.make ! src/share/vm/prims/forte.cpp Changeset: fb14e9ed1594 Author: jprovino Date: 2013-05-28 11:32 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/fb14e9ed1594 8011064: Some tests have failed with SIGSEGV on arm-hflt on build b82 Summary: NMT_detail is only supported when frame pointers are not omitted (-fno-omit-frame-pointer). Reviewed-by: dholmes, cjplummer ! src/share/vm/services/memTracker.cpp ! src/share/vm/utilities/globalDefinitions.hpp Changeset: 9e954e8d9139 Author: jprovino Date: 2013-05-28 15:24 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/9e954e8d9139 Merge Changeset: 9e86c5544295 Author: jiangli Date: 2013-05-30 13:19 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/9e86c5544295 Merge Changeset: f41a577cffb0 Author: jwilhelm Date: 2013-05-31 09:55 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/f41a577cffb0 Merge Changeset: 573d86d412cd Author: katleman Date: 2013-05-30 10:57 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/573d86d412cd Added tag jdk8-b92 for changeset 092018493d3b ! .hgtags Changeset: b786c04b7be1 Author: amurillo Date: 2013-05-31 09:37 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/b786c04b7be1 Merge Changeset: 5a028ee56116 Author: amurillo Date: 2013-05-31 09:37 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/5a028ee56116 Added tag hs25-b35 for changeset b786c04b7be1 ! .hgtags Changeset: b7569f617285 Author: amurillo Date: 2013-05-31 10:04 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/b7569f617285 8015690: new hotspot build - hs25-b36 Reviewed-by: jcoomes ! make/hotspot_version Changeset: c20186fa611b Author: jwilhelm Date: 2013-06-01 10:00 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/c20186fa611b Merge From tao.mao at oracle.com Sun Jun 2 04:56:04 2013 From: tao.mao at oracle.com (Tao Mao) Date: Sat, 01 Jun 2013 21:56:04 -0700 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51A83B7E.9060808@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> Message-ID: <51AAD064.8030600@oracle.com> The new webrev is updated. http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ Thanks. Tao On 5/30/13 10:56 PM, Bengt Rutisson wrote: > > Hi again, > > Just realized that I did this review a bit too early in the > morning...before the morning coffee... One correction below. > > On 5/31/13 7:44 AM, Bengt Rutisson wrote: >> >> >> Hi Tao, >> >> Comments inline, >> >> On 5/31/13 3:26 AM, Tao Mao wrote: >>> Please see inline. >>> >>> Thanks. >>> Tao >>> >>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>> >>>> Hi Tao, >>>> >>>> I think the code is a little bit confused about whether >>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>> AllocPriorites or just 2. All the for loops indicate that we think >>>> we might want to change from 2 to a larger number in the future. >>>> But the naming of a method like words_remaining_in_retired() >>>> indicate that there can only be one retired region. With the >>>> current implementation I think words_remaining_in_retired() should >>>> be called something like words_remaining_in_priority0_buffer(). >>> Done. >>> changed to words_remaining_in_priority1_buffer() >> >> Hm. Isn't this a bug? I think you want the method to be called >> words_remaining_in_priority0_buffer() and return the remaining words >> in the priority0 buffer. You call the method before you do >> alloc_buf->retire_and_set_buf(), so the priority1 buffer is probably >> not the one you are interested in. > > My bad. I thought the priorities were zero indexed, but because of > your enum they are one indexed. So, > words_remaining_in_priority1_buffer() is correct here. > > Bengt > >> >>>> >>>> I think it would be good to make this code truly general with >>>> respect to the number of priorities. We can then use 2 as default, >>>> but make sure that the code works with more priorities. To do that >>>> I think we should remove the enum GCAllocPriority and instead have >>>> a field in G1MultiParGCAllocBuffer that contains the maximum number >>>> of priorities. I think that will make the code more general and >>>> easier to read. The for loops would look like: >>>> >>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>> // do stuff >>>> } >>> It's more like code style issue. In fact, it was done this way >>> according to the Jon's earlier suggestion. Second, if we want to >>> change #buffer to 3 (it wont give more benefits to define more than >>> that number), we only need to add one more enum value, i.e. >>> GCAllocPriority3. >> >> Let me clarify a bit why I don't like the GCAllocPriority enum. There >> is really no reason to use an enum here. You are just making code >> complicated without adding any semantics. You always want to use >> 0-max and the order is important. This is exactly what you get from >> an normal int. >> >> The enum GCAllocPurpose is different since there is no natural order >> between GCAllocForTenured and GCAllocForSurvived. Thus, an enum makes >> sense there. >> >> So, please remove the GCAllocPriority enum. >> >>>> >>>> I find the name G1MultiParGCAllocBuffer confusing since it is not >>>> inheriting G1ParGCAllocBuffer. Maybe G1AllocBufferContainer or >>>> something like that would make more sense? >>> Done. >>> Changed to G1ParGCAllocBufferContainer >>>> >>>> I don't understand why you added initialization values to >>>> GCAllocPurpose. You are only using the values that are default in >>>> C++ anyway: 0, 1, 2. At least if you avoid adding the >>>> GCAllocPurposeStart value. I think it was more readable before your >>>> change. (The same argument holds for GCAllocPriority, but I prefer >>>> to remove that enum all together as I described above.) >>> See above. >> >> This is not the same issue as above. What I'm saying is that your >> changes to GCAllocPurpose made it less readable without adding any >> extra semantics. Please revert to this change. >> >>>> >>>> Have you considered moving the _retired field from >>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>> retire() method virtual? (I do think your change to virtual is >>>> needed in the current code, so good catch! But I think it might >>>> make sense to have the logic of G1ParGCAllocBuffer::retire() in >>>> ParGCAllocBuffer::retire() instead.) >>> In G1ParGCAllocBuffer, we need the field _retired to handle buffer >>> allocation failure. This is handled differently for other >>> collectors. For example, ParScanThreadState::alloc_in_to_space_slow >>> in ParNew. Thus, moving the _retired field up to its super class >>> will involve additional efforts. This is supposed to be investigated >>> in another CR JDK-7127700. >> >> OK. Good. >> >> Thanks, >> Bengt >> >>>> >>>> A couple of minor things: >>>> >>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>> >>>> should be: >>>> >>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>> Done. >>>> >>>> Please add spaces before and after "=" here: >>>> 1804 size_t result=0; >>> Done. >>>> >>>> There are two spaces after "=" here: >>>> 1812 G1ParGCAllocBuffer* retired = >>>> _priority_buffer[GCAllocPriority1]; >>> Done. >>>> >>>> Also, in g1CollectedHeap.hpp you have updated the copyright year >>>> but not in parGCAllocBuffer.hpp. As you know we in the GC team have >>>> agreed not to update the copyright year. If you absolutely want to >>>> do it I think you should do it the same way in all files. >>> Done. >>>> >>>> Thanks, >>>> Bengt >>>> >>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>> Can I have a couple of reviewers please? >>>>> >>>>> Thank you. >>>>> Tao >>>>> >>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>> Hi all, >>>>>> >>>>>> a new webrev >>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>> >>>>>> diff: >>>>>> (1) John Cuthbertson and I figured out the way to handle "retire >>>>>> an old buffer, allocate and set a new one" and it can possibly >>>>>> make the usage of allocation buffer a little more efficient. >>>>>> (2) Make the assertion as John suggested and provide some harness >>>>>> (i.e. making retire() a virtual function) to cope with it. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>> Hi Tao, >>>>>>> >>>>>>> This looks excellent. One minor question: does it make sense to >>>>>>> assert that each buffer has been retired? It might save a missed >>>>>>> call to PSS::retire_alloc_buffers. I'll leave the decision to >>>>>>> you. Ship it. >>>>>>> >>>>>>> JohnC >>>>>>> >>>>>>> >>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>> To the open list: >>>>>>>> >>>>>>>> new webrev: >>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>> >>>>>>>> I took suggestion from many reviewers into consideration and >>>>>>>> came up with the current cleaner solution. >>>>>>>> >>>>>>>> Thank you. >>>>>>>> Tao >>>>>>>> >>>>>>>> >>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>> What's the status of this review? >>>>>>>>> >>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>> from Thomas. >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>> 6976350 G1: deal with fragmentation while copying objects >>>>>>>>>> during GC >>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>> >>>>>>>>>> webrev: >>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>> >>>>>>>>>> changeset: >>>>>>>>>> Basically, we want to reuse more of par-allocation buffers >>>>>>>>>> instead of retiring it immediately when it encounters an >>>>>>>>>> object larger than its remaining part. >>>>>>>>>> >>>>>>>>>> (1) instead of previously using one allocation buffer per GC >>>>>>>>>> purpose, we use N(=2) buffers per GC purpose and modify the >>>>>>>>>> corresponding code. The changeset would easily scale up to >>>>>>>>>> whatever N (though Tony Printezis suggests 2, or 3 may be >>>>>>>>>> good enough) >>>>>>>>>> >>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>> removed due to its never being called. >>>>>>>>>> access modifier >>>>>>>>>> (public) before trim_queue() is redundant. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Mon Jun 3 08:16:51 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 03 Jun 2013 10:16:51 +0200 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51AAD064.8030600@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> <51AAD064.8030600@oracle.com> Message-ID: <51AC50F3.3000308@oracle.com> Hi Tao, On 6/2/13 6:56 AM, Tao Mao wrote: > The new webrev is updated. > http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ Thanks for fixing this. I think it looks better. Still have some comments: Line 78, int const GCAllocPriorityMax = 2; I would prefer that this was a "private static const int" inside G1ParGCAllocBufferContainer. You could call it _priority_max to avoid the assignment in the constructor. I think the name select_retired_buf() is a bit confusing. The way it is used I think the code would be more readable if we just inlined 0 in the code. In G1ParScanThreadState::allocate_slow() I think we might miss retiring the alloc buffers, right? We now call retire_and_set_buf() after we have we have tried the allcoation. If the allocation fails I think we have to retired both alloc buffers since both of them may have been allocated into. I also think the name retire_and_set_buf() indicates that this method does too much. I would prefer to have two different methods. One retire_current_buf() that retires the current alloc buffer and probably also shifts the buffers (what adjust_priority_order() does now) and one that sets up a new buffer. This will probably also be useful if we need to only retire buffers in the allocation failure case I described above. Thanks, Bengt > > Thanks. > Tao > > On 5/30/13 10:56 PM, Bengt Rutisson wrote: >> >> Hi again, >> >> Just realized that I did this review a bit too early in the >> morning...before the morning coffee... One correction below. >> >> On 5/31/13 7:44 AM, Bengt Rutisson wrote: >>> >>> >>> Hi Tao, >>> >>> Comments inline, >>> >>> On 5/31/13 3:26 AM, Tao Mao wrote: >>>> Please see inline. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>>> >>>>> Hi Tao, >>>>> >>>>> I think the code is a little bit confused about whether >>>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>>> AllocPriorites or just 2. All the for loops indicate that we think >>>>> we might want to change from 2 to a larger number in the future. >>>>> But the naming of a method like words_remaining_in_retired() >>>>> indicate that there can only be one retired region. With the >>>>> current implementation I think words_remaining_in_retired() should >>>>> be called something like words_remaining_in_priority0_buffer(). >>>> Done. >>>> changed to words_remaining_in_priority1_buffer() >>> >>> Hm. Isn't this a bug? I think you want the method to be called >>> words_remaining_in_priority0_buffer() and return the remaining words >>> in the priority0 buffer. You call the method before you do >>> alloc_buf->retire_and_set_buf(), so the priority1 buffer is probably >>> not the one you are interested in. >> >> My bad. I thought the priorities were zero indexed, but because of >> your enum they are one indexed. So, >> words_remaining_in_priority1_buffer() is correct here. >> >> Bengt >> >>> >>>>> >>>>> I think it would be good to make this code truly general with >>>>> respect to the number of priorities. We can then use 2 as default, >>>>> but make sure that the code works with more priorities. To do that >>>>> I think we should remove the enum GCAllocPriority and instead have >>>>> a field in G1MultiParGCAllocBuffer that contains the maximum >>>>> number of priorities. I think that will make the code more general >>>>> and easier to read. The for loops would look like: >>>>> >>>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>>> // do stuff >>>>> } >>>> It's more like code style issue. In fact, it was done this way >>>> according to the Jon's earlier suggestion. Second, if we want to >>>> change #buffer to 3 (it wont give more benefits to define more than >>>> that number), we only need to add one more enum value, i.e. >>>> GCAllocPriority3. >>> >>> Let me clarify a bit why I don't like the GCAllocPriority enum. >>> There is really no reason to use an enum here. You are just making >>> code complicated without adding any semantics. You always want to >>> use 0-max and the order is important. This is exactly what you get >>> from an normal int. >>> >>> The enum GCAllocPurpose is different since there is no natural order >>> between GCAllocForTenured and GCAllocForSurvived. Thus, an enum >>> makes sense there. >>> >>> So, please remove the GCAllocPriority enum. >>> >>>>> >>>>> I find the name G1MultiParGCAllocBuffer confusing since it is not >>>>> inheriting G1ParGCAllocBuffer. Maybe G1AllocBufferContainer or >>>>> something like that would make more sense? >>>> Done. >>>> Changed to G1ParGCAllocBufferContainer >>>>> >>>>> I don't understand why you added initialization values to >>>>> GCAllocPurpose. You are only using the values that are default in >>>>> C++ anyway: 0, 1, 2. At least if you avoid adding the >>>>> GCAllocPurposeStart value. I think it was more readable before >>>>> your change. (The same argument holds for GCAllocPriority, but I >>>>> prefer to remove that enum all together as I described above.) >>>> See above. >>> >>> This is not the same issue as above. What I'm saying is that your >>> changes to GCAllocPurpose made it less readable without adding any >>> extra semantics. Please revert to this change. >>> >>>>> >>>>> Have you considered moving the _retired field from >>>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>>> retire() method virtual? (I do think your change to virtual is >>>>> needed in the current code, so good catch! But I think it might >>>>> make sense to have the logic of G1ParGCAllocBuffer::retire() in >>>>> ParGCAllocBuffer::retire() instead.) >>>> In G1ParGCAllocBuffer, we need the field _retired to handle buffer >>>> allocation failure. This is handled differently for other >>>> collectors. For example, ParScanThreadState::alloc_in_to_space_slow >>>> in ParNew. Thus, moving the _retired field up to its super class >>>> will involve additional efforts. This is supposed to be >>>> investigated in another CR JDK-7127700. >>> >>> OK. Good. >>> >>> Thanks, >>> Bengt >>> >>>>> >>>>> A couple of minor things: >>>>> >>>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>>> >>>>> should be: >>>>> >>>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>>> Done. >>>>> >>>>> Please add spaces before and after "=" here: >>>>> 1804 size_t result=0; >>>> Done. >>>>> >>>>> There are two spaces after "=" here: >>>>> 1812 G1ParGCAllocBuffer* retired = >>>>> _priority_buffer[GCAllocPriority1]; >>>> Done. >>>>> >>>>> Also, in g1CollectedHeap.hpp you have updated the copyright year >>>>> but not in parGCAllocBuffer.hpp. As you know we in the GC team >>>>> have agreed not to update the copyright year. If you absolutely >>>>> want to do it I think you should do it the same way in all files. >>>> Done. >>>>> >>>>> Thanks, >>>>> Bengt >>>>> >>>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>>> Can I have a couple of reviewers please? >>>>>> >>>>>> Thank you. >>>>>> Tao >>>>>> >>>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> a new webrev >>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>>> >>>>>>> diff: >>>>>>> (1) John Cuthbertson and I figured out the way to handle "retire >>>>>>> an old buffer, allocate and set a new one" and it can possibly >>>>>>> make the usage of allocation buffer a little more efficient. >>>>>>> (2) Make the assertion as John suggested and provide some >>>>>>> harness (i.e. making retire() a virtual function) to cope with it. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>>> Hi Tao, >>>>>>>> >>>>>>>> This looks excellent. One minor question: does it make sense to >>>>>>>> assert that each buffer has been retired? It might save a >>>>>>>> missed call to PSS::retire_alloc_buffers. I'll leave the >>>>>>>> decision to you. Ship it. >>>>>>>> >>>>>>>> JohnC >>>>>>>> >>>>>>>> >>>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>>> To the open list: >>>>>>>>> >>>>>>>>> new webrev: >>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>>> >>>>>>>>> I took suggestion from many reviewers into consideration and >>>>>>>>> came up with the current cleaner solution. >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>>> What's the status of this review? >>>>>>>>>> >>>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>>> from Thomas. >>>>>>>>>> >>>>>>>>>> Jon >>>>>>>>>> >>>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>>> 6976350 G1: deal with fragmentation while copying objects >>>>>>>>>>> during GC >>>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>>> >>>>>>>>>>> webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> changeset: >>>>>>>>>>> Basically, we want to reuse more of par-allocation buffers >>>>>>>>>>> instead of retiring it immediately when it encounters an >>>>>>>>>>> object larger than its remaining part. >>>>>>>>>>> >>>>>>>>>>> (1) instead of previously using one allocation buffer per GC >>>>>>>>>>> purpose, we use N(=2) buffers per GC purpose and modify the >>>>>>>>>>> corresponding code. The changeset would easily scale up to >>>>>>>>>>> whatever N (though Tony Printezis suggests 2, or 3 may be >>>>>>>>>>> good enough) >>>>>>>>>>> >>>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>>> removed due to its never being called. >>>>>>>>>>> access modifier (public) before trim_queue() is redundant. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Mon Jun 3 13:14:17 2013 From: bengt.rutisson at oracle.com (bengt.rutisson at oracle.com) Date: Mon, 03 Jun 2013 13:14:17 +0000 Subject: hg: hsx/hotspot-gc/hotspot: 8013895: G1: G1SummarizeRSetStats output on Linux needs improvemen Message-ID: <20130603131425.F321348EE3@hg.openjdk.java.net> Changeset: e72f7eecc96d Author: tschatzl Date: 2013-05-28 09:32 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/e72f7eecc96d 8013895: G1: G1SummarizeRSetStats output on Linux needs improvemen Summary: Fixed the output of G1SummarizeRSetStats: too small datatype for the number of concurrently processed cards, added concurrent remembered set thread time retrieval for Linux and Windows (BSD uses os::elapsedTime() now), and other cleanup. The information presented during VM operation is now relative to the previous output, not always cumulative if G1SummarizeRSetStatsPeriod > 0. At VM exit, the code prints a cumulative summary. Reviewed-by: johnc, jwilhelm ! make/excludeSrc.make ! src/os/bsd/vm/os_bsd.cpp ! src/os/linux/vm/os_linux.cpp ! src/os/windows/vm/os_windows.cpp ! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp ! src/share/vm/gc_implementation/g1/concurrentG1Refine.hpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.hpp + src/share/vm/gc_implementation/g1/g1RemSetSummary.cpp + src/share/vm/gc_implementation/g1/g1RemSetSummary.hpp + test/gc/g1/TestSummarizeRSetStats.java From jesper.wilhelmsson at oracle.com Mon Jun 3 17:46:05 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Mon, 03 Jun 2013 19:46:05 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <1369729575.2616.27.camel@cirrus> References: <1369729575.2616.27.camel@cirrus> Message-ID: <51ACD65D.6000708@oracle.com> Thomas, Looks good. Since you touch the code in HRRSStatsIter::print_summary_on; There are spaces around some operators but not others. It would be nice if it was consistent. I would prefer if you add spaces in the places where they are missing (rather than removing the ones that exists). Whether you clean up the spaces or not, Ship it! /Jesper Thomas Schatzl skrev 28/5/13 10:26 AM: > Hi all, > > could I get some reviews for the following change? It improves the > remembered set summary information available with -XX: > +G1SummarizeRSetStats by adding memory size and occupancy summaries > broken down by region type (young, humonguous, free, other). > > Webrev: > http://cr.openjdk.java.net/~tschatzl/8014078/webrev/ > > Note that this webrev is based on 8013895; I provided a full changeset > including 8013895 at > http://cr.openjdk.java.net/~tschatzl/8014078/webrev/8014078.8013895.diff > > JIRA: > https://jbs.oracle.com/bugs/browse/JDK-8014078 > > bugs.sun.com: > http://bugs.sun.com/view_bug.do?bug_id=8014078 > > Testing: > jprt, jtreg tests > > About the change: > > - the output of G1SummarizeRSetStats now looks as follows (for > dacapo/eclipse): > > After GC RS summary > > Concurrent RS processed 15299 cards > Of 64 completed buffers: > 64 (100.0%) by concurrent RS threads. > 0 ( 0.0%) by mutator threads. > Concurrent RS threads times (s) > 0.06 > Concurrent sampling threads times (s) > 0.00 > Total heap region rem set sizes = 512K. Max = 13K. >>>>> 17K ( 3.4%) by 7 Young regions >>>>> 7K ( 1.4%) by 3 Humonguous regions >>>>> 157K ( 30.6%) by 64 Free regions >>>>> 330K ( 64.6%) by 44 Other regions > Static structures = 1K, free_lists = 254K. > 277592 occupied cards represented. >>>>> 0 ( 0.0%) entries by 7 Young regions >>>>> 2 ( 0.0%) entries by 3 Humonguous regions >>>>> 0 ( 0.0%) entries by 64 Free regions >>>>> 277590 (100.0%) entries by 44 Other regions > Max size region = > 3:(O)[0x00000000f0900000,0x00000000f0a00000,0x00000000f0a00000], size = > 14K, occupied = 25K. > Did 0 coarsenings. > > Notice the breakdown of memory size and card occupancy by region type > (marked with >>>> here). > > "Young" regions include all types of young regions (i.e. survivor) as > survivor does not have a meaning for g1 outside of a gc. > "Humonguous" and "Free" are self-explaining. "Other" regions include > everything else. > > - added a new flag G1SummarizeRSetStatsTime that is used to control the > time, i.e. before/after gc, when the periodicaly summaries are printed. > This integer is treated as a bitset, where a set first bit indicates a > request to print a summary after gc, the second bit a request to print a > summary before gc. > > The alternative (which I'd like more) would be to change > G1SummarizeRSetStats to act like this new flag to avoid introducing a > new flag; however I'm hesitant to change this as G1SummarizeRSetStats is > a publicly available flag (although diagnostic). > > - cleanup of the test cases: a large part of the code base for this CR > reuses utility methods from 8013895, so I went ahead and factored out > common code into a separate file that both the tests for 801385 and this > CR use. > > Thanks, > Thomas > From tao.mao at oracle.com Mon Jun 3 19:16:49 2013 From: tao.mao at oracle.com (Tao Mao) Date: Mon, 03 Jun 2013 12:16:49 -0700 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51AC50F3.3000308@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> <51AAD064.8030600@oracle.com> <51AC50F3.3000308@oracle.com> Message-ID: <51ACEBA1.40302@oracle.com> new webrev: http://cr.openjdk.java.net/~tamao/6976350/webrev.07/ Please see inline. Thanks. Tao On 6/3/13 1:16 AM, Bengt Rutisson wrote: > > Hi Tao, > > On 6/2/13 6:56 AM, Tao Mao wrote: >> The new webrev is updated. >> http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ > > Thanks for fixing this. I think it looks better. Still have some comments: > > > Line 78, int const GCAllocPriorityMax = 2; > > I would prefer that this was a "private static const int" inside > G1ParGCAllocBufferContainer. You could call it _priority_max to avoid > the assignment in the constructor. Done. > > > I think the name select_retired_buf() is a bit confusing. The way it > is used I think the code would be more readable if we just inlined 0 > in the code. Inlining done. > > > In G1ParScanThreadState::allocate_slow() I think we might miss > retiring the alloc buffers, right? > > We now call retire_and_set_buf() after we have we have tried the > allcoation. If the allocation fails I think we have to retired both > alloc buffers since both of them may have been allocated into. The current implementation is right. (1) Even if the code reaches the point where it need to allocate a new buffer and fails, the old buffer is still usable. There's no reason we should retire it entirely. In addition, I designed to keep the buffer when the allocation fails so that the code doesn't need additional checkings in order to make sure the buffer is still valid, when trying to allocate a new object and to retire it per se. In fact, the current implementation simplifies the logic of object allocation in the buffer and retiring buffers if you get it. (2) Please check out the function retire_alloc_buffers(). It guards the clearing of all buffers in the end of copying phase, so we don't have to worry about that part. (3) A subtle benefit to mention: I still keep the buffer when the attempted allocation fails such way that we hope that the buffer may be allocated again to contain a new "smaller" object. It can happen to improve heap efficiency. > > I also think the name retire_and_set_buf() indicates that this method > does too much. I would prefer to have two different methods. One > retire_current_buf() that retires the current alloc buffer and > probably also shifts the buffers (what adjust_priority_order() does > now) and one that sets up a new buffer. I think it's a single operation and can't be separated. If we separated this function, we would end up with exposing unnecessary details to the caller. Also, the order of retire(), set_buf(), set_word_size() and adjust_priority_order() matters. I don't want to the caller to handle this rather than handle the ordering issue in class G1ParGCAllocBufferContainer. > > This will probably also be useful if we need to only retire buffers in > the allocation failure case I described above. As I mentioned above, retiring buffers upon the allocation failure would introduce additional complexity to handling the buffer usage and, even, retiring process itself. Please also refer to the last third comments above. > > Thanks, > Bengt > >> >> Thanks. >> Tao >> >> On 5/30/13 10:56 PM, Bengt Rutisson wrote: >>> >>> Hi again, >>> >>> Just realized that I did this review a bit too early in the >>> morning...before the morning coffee... One correction below. >>> >>> On 5/31/13 7:44 AM, Bengt Rutisson wrote: >>>> >>>> >>>> Hi Tao, >>>> >>>> Comments inline, >>>> >>>> On 5/31/13 3:26 AM, Tao Mao wrote: >>>>> Please see inline. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>>>> >>>>>> Hi Tao, >>>>>> >>>>>> I think the code is a little bit confused about whether >>>>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>>>> AllocPriorites or just 2. All the for loops indicate that we >>>>>> think we might want to change from 2 to a larger number in the >>>>>> future. But the naming of a method like >>>>>> words_remaining_in_retired() indicate that there can only be one >>>>>> retired region. With the current implementation I think >>>>>> words_remaining_in_retired() should be called something like >>>>>> words_remaining_in_priority0_buffer(). >>>>> Done. >>>>> changed to words_remaining_in_priority1_buffer() >>>> >>>> Hm. Isn't this a bug? I think you want the method to be called >>>> words_remaining_in_priority0_buffer() and return the remaining >>>> words in the priority0 buffer. You call the method before you do >>>> alloc_buf->retire_and_set_buf(), so the priority1 buffer is >>>> probably not the one you are interested in. >>> >>> My bad. I thought the priorities were zero indexed, but because of >>> your enum they are one indexed. So, >>> words_remaining_in_priority1_buffer() is correct here. >>> >>> Bengt >>> >>>> >>>>>> >>>>>> I think it would be good to make this code truly general with >>>>>> respect to the number of priorities. We can then use 2 as >>>>>> default, but make sure that the code works with more priorities. >>>>>> To do that I think we should remove the enum GCAllocPriority and >>>>>> instead have a field in G1MultiParGCAllocBuffer that contains the >>>>>> maximum number of priorities. I think that will make the code >>>>>> more general and easier to read. The for loops would look like: >>>>>> >>>>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>>>> // do stuff >>>>>> } >>>>> It's more like code style issue. In fact, it was done this way >>>>> according to the Jon's earlier suggestion. Second, if we want to >>>>> change #buffer to 3 (it wont give more benefits to define more >>>>> than that number), we only need to add one more enum value, i.e. >>>>> GCAllocPriority3. >>>> >>>> Let me clarify a bit why I don't like the GCAllocPriority enum. >>>> There is really no reason to use an enum here. You are just making >>>> code complicated without adding any semantics. You always want to >>>> use 0-max and the order is important. This is exactly what you get >>>> from an normal int. >>>> >>>> The enum GCAllocPurpose is different since there is no natural >>>> order between GCAllocForTenured and GCAllocForSurvived. Thus, an >>>> enum makes sense there. >>>> >>>> So, please remove the GCAllocPriority enum. >>>> >>>>>> >>>>>> I find the name G1MultiParGCAllocBuffer confusing since it is not >>>>>> inheriting G1ParGCAllocBuffer. Maybe G1AllocBufferContainer or >>>>>> something like that would make more sense? >>>>> Done. >>>>> Changed to G1ParGCAllocBufferContainer >>>>>> >>>>>> I don't understand why you added initialization values to >>>>>> GCAllocPurpose. You are only using the values that are default in >>>>>> C++ anyway: 0, 1, 2. At least if you avoid adding the >>>>>> GCAllocPurposeStart value. I think it was more readable before >>>>>> your change. (The same argument holds for GCAllocPriority, but I >>>>>> prefer to remove that enum all together as I described above.) >>>>> See above. >>>> >>>> This is not the same issue as above. What I'm saying is that your >>>> changes to GCAllocPurpose made it less readable without adding any >>>> extra semantics. Please revert to this change. >>>> >>>>>> >>>>>> Have you considered moving the _retired field from >>>>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>>>> retire() method virtual? (I do think your change to virtual is >>>>>> needed in the current code, so good catch! But I think it might >>>>>> make sense to have the logic of G1ParGCAllocBuffer::retire() in >>>>>> ParGCAllocBuffer::retire() instead.) >>>>> In G1ParGCAllocBuffer, we need the field _retired to handle buffer >>>>> allocation failure. This is handled differently for other >>>>> collectors. For example, >>>>> ParScanThreadState::alloc_in_to_space_slow in ParNew. Thus, moving >>>>> the _retired field up to its super class will involve additional >>>>> efforts. This is supposed to be investigated in another CR >>>>> JDK-7127700. >>>> >>>> OK. Good. >>>> >>>> Thanks, >>>> Bengt >>>> >>>>>> >>>>>> A couple of minor things: >>>>>> >>>>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>>>> >>>>>> should be: >>>>>> >>>>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>>>> Done. >>>>>> >>>>>> Please add spaces before and after "=" here: >>>>>> 1804 size_t result=0; >>>>> Done. >>>>>> >>>>>> There are two spaces after "=" here: >>>>>> 1812 G1ParGCAllocBuffer* retired = >>>>>> _priority_buffer[GCAllocPriority1]; >>>>> Done. >>>>>> >>>>>> Also, in g1CollectedHeap.hpp you have updated the copyright year >>>>>> but not in parGCAllocBuffer.hpp. As you know we in the GC team >>>>>> have agreed not to update the copyright year. If you absolutely >>>>>> want to do it I think you should do it the same way in all files. >>>>> Done. >>>>>> >>>>>> Thanks, >>>>>> Bengt >>>>>> >>>>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>>>> Can I have a couple of reviewers please? >>>>>>> >>>>>>> Thank you. >>>>>>> Tao >>>>>>> >>>>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> a new webrev >>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>>>> >>>>>>>> diff: >>>>>>>> (1) John Cuthbertson and I figured out the way to handle >>>>>>>> "retire an old buffer, allocate and set a new one" and it can >>>>>>>> possibly make the usage of allocation buffer a little more >>>>>>>> efficient. >>>>>>>> (2) Make the assertion as John suggested and provide some >>>>>>>> harness (i.e. making retire() a virtual function) to cope with it. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>>>> Hi Tao, >>>>>>>>> >>>>>>>>> This looks excellent. One minor question: does it make sense >>>>>>>>> to assert that each buffer has been retired? It might save a >>>>>>>>> missed call to PSS::retire_alloc_buffers. I'll leave the >>>>>>>>> decision to you. Ship it. >>>>>>>>> >>>>>>>>> JohnC >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>>>> To the open list: >>>>>>>>>> >>>>>>>>>> new webrev: >>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>>>> >>>>>>>>>> I took suggestion from many reviewers into consideration and >>>>>>>>>> came up with the current cleaner solution. >>>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>>>> What's the status of this review? >>>>>>>>>>> >>>>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>>>> from Thomas. >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>> >>>>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>>>> 6976350 G1: deal with fragmentation while copying objects >>>>>>>>>>>> during GC >>>>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>>>> >>>>>>>>>>>> webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>>>> >>>>>>>>>>>> changeset: >>>>>>>>>>>> Basically, we want to reuse more of par-allocation buffers >>>>>>>>>>>> instead of retiring it immediately when it encounters an >>>>>>>>>>>> object larger than its remaining part. >>>>>>>>>>>> >>>>>>>>>>>> (1) instead of previously using one allocation buffer per >>>>>>>>>>>> GC purpose, we use N(=2) buffers per GC purpose and modify >>>>>>>>>>>> the corresponding code. The changeset would easily scale up >>>>>>>>>>>> to whatever N (though Tony Printezis suggests 2, or 3 may >>>>>>>>>>>> be good enough) >>>>>>>>>>>> >>>>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>>>> removed due to its never being called. >>>>>>>>>>>> access >>>>>>>>>>>> modifier (public) before trim_queue() is redundant. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>> >>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Mon Jun 3 20:55:07 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 03 Jun 2013 22:55:07 +0200 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51ACEBA1.40302@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> <51AAD064.8030600@oracle.com> <51AC50F3.3000308@oracle.com> <51ACEBA1.40302@oracle.com> Message-ID: <51AD02AB.4090902@oracle.com> Tao, On 6/3/13 9:16 PM, Tao Mao wrote: > new webrev: > http://cr.openjdk.java.net/~tamao/6976350/webrev.07/ This looks good to me. Thanks for working through all of these iterations! There are superfluous spaces on these lines before _priority_buffer[0]: 1803 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; 1812 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; Other than that it looks good! > > Please see inline. Thanks for the detailed description about retiring alloc buffers. It was very helpful. Bengt > > Thanks. > Tao > > On 6/3/13 1:16 AM, Bengt Rutisson wrote: >> >> Hi Tao, >> >> On 6/2/13 6:56 AM, Tao Mao wrote: >>> The new webrev is updated. >>> http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ >> >> Thanks for fixing this. I think it looks better. Still have some >> comments: >> >> >> Line 78, int const GCAllocPriorityMax = 2; >> >> I would prefer that this was a "private static const int" inside >> G1ParGCAllocBufferContainer. You could call it _priority_max to avoid >> the assignment in the constructor. > Done. > >> >> >> I think the name select_retired_buf() is a bit confusing. The way it >> is used I think the code would be more readable if we just inlined 0 >> in the code. > Inlining done. > >> >> >> In G1ParScanThreadState::allocate_slow() I think we might miss >> retiring the alloc buffers, right? >> >> We now call retire_and_set_buf() after we have we have tried the >> allcoation. If the allocation fails I think we have to retired both >> alloc buffers since both of them may have been allocated into. > The current implementation is right. > > (1) Even if the code reaches the point where it need to allocate a new > buffer and fails, the old buffer is still usable. There's no reason we > should retire it entirely. > > In addition, I designed to keep the buffer when the allocation fails > so that the code doesn't need additional checkings in order to make > sure the buffer is still valid, when trying to allocate a new object > and to retire it per se. > > In fact, the current implementation simplifies the logic of object > allocation in the buffer and retiring buffers if you get it. > > (2) Please check out the function retire_alloc_buffers(). It guards > the clearing of all buffers in the end of copying phase, so we don't > have to worry about that part. > > (3) A subtle benefit to mention: I still keep the buffer when the > attempted allocation fails such way that we hope that the buffer may > be allocated again to contain a new "smaller" object. It can happen to > improve heap efficiency. > >> >> I also think the name retire_and_set_buf() indicates that this method >> does too much. I would prefer to have two different methods. One >> retire_current_buf() that retires the current alloc buffer and >> probably also shifts the buffers (what adjust_priority_order() does >> now) and one that sets up a new buffer. > I think it's a single operation and can't be separated. If we > separated this function, we would end up with exposing unnecessary > details to the caller. Also, the order of retire(), set_buf(), > set_word_size() and adjust_priority_order() matters. I don't want to > the caller to handle this rather than handle the ordering issue in > class G1ParGCAllocBufferContainer. > >> >> This will probably also be useful if we need to only retire buffers >> in the allocation failure case I described above. > As I mentioned above, retiring buffers upon the allocation failure > would introduce additional complexity to handling the buffer usage > and, even, retiring process itself. Please also refer to the last > third comments above. > >> >> Thanks, >> Bengt >> >>> >>> Thanks. >>> Tao >>> >>> On 5/30/13 10:56 PM, Bengt Rutisson wrote: >>>> >>>> Hi again, >>>> >>>> Just realized that I did this review a bit too early in the >>>> morning...before the morning coffee... One correction below. >>>> >>>> On 5/31/13 7:44 AM, Bengt Rutisson wrote: >>>>> >>>>> >>>>> Hi Tao, >>>>> >>>>> Comments inline, >>>>> >>>>> On 5/31/13 3:26 AM, Tao Mao wrote: >>>>>> Please see inline. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>>>>> >>>>>>> Hi Tao, >>>>>>> >>>>>>> I think the code is a little bit confused about whether >>>>>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>>>>> AllocPriorites or just 2. All the for loops indicate that we >>>>>>> think we might want to change from 2 to a larger number in the >>>>>>> future. But the naming of a method like >>>>>>> words_remaining_in_retired() indicate that there can only be one >>>>>>> retired region. With the current implementation I think >>>>>>> words_remaining_in_retired() should be called something like >>>>>>> words_remaining_in_priority0_buffer(). >>>>>> Done. >>>>>> changed to words_remaining_in_priority1_buffer() >>>>> >>>>> Hm. Isn't this a bug? I think you want the method to be called >>>>> words_remaining_in_priority0_buffer() and return the remaining >>>>> words in the priority0 buffer. You call the method before you do >>>>> alloc_buf->retire_and_set_buf(), so the priority1 buffer is >>>>> probably not the one you are interested in. >>>> >>>> My bad. I thought the priorities were zero indexed, but because of >>>> your enum they are one indexed. So, >>>> words_remaining_in_priority1_buffer() is correct here. >>>> >>>> Bengt >>>> >>>>> >>>>>>> >>>>>>> I think it would be good to make this code truly general with >>>>>>> respect to the number of priorities. We can then use 2 as >>>>>>> default, but make sure that the code works with more priorities. >>>>>>> To do that I think we should remove the enum GCAllocPriority and >>>>>>> instead have a field in G1MultiParGCAllocBuffer that contains >>>>>>> the maximum number of priorities. I think that will make the >>>>>>> code more general and easier to read. The for loops would look like: >>>>>>> >>>>>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>>>>> // do stuff >>>>>>> } >>>>>> It's more like code style issue. In fact, it was done this way >>>>>> according to the Jon's earlier suggestion. Second, if we want to >>>>>> change #buffer to 3 (it wont give more benefits to define more >>>>>> than that number), we only need to add one more enum value, i.e. >>>>>> GCAllocPriority3. >>>>> >>>>> Let me clarify a bit why I don't like the GCAllocPriority enum. >>>>> There is really no reason to use an enum here. You are just making >>>>> code complicated without adding any semantics. You always want to >>>>> use 0-max and the order is important. This is exactly what you get >>>>> from an normal int. >>>>> >>>>> The enum GCAllocPurpose is different since there is no natural >>>>> order between GCAllocForTenured and GCAllocForSurvived. Thus, an >>>>> enum makes sense there. >>>>> >>>>> So, please remove the GCAllocPriority enum. >>>>> >>>>>>> >>>>>>> I find the name G1MultiParGCAllocBuffer confusing since it is >>>>>>> not inheriting G1ParGCAllocBuffer. Maybe G1AllocBufferContainer >>>>>>> or something like that would make more sense? >>>>>> Done. >>>>>> Changed to G1ParGCAllocBufferContainer >>>>>>> >>>>>>> I don't understand why you added initialization values to >>>>>>> GCAllocPurpose. You are only using the values that are default >>>>>>> in C++ anyway: 0, 1, 2. At least if you avoid adding the >>>>>>> GCAllocPurposeStart value. I think it was more readable before >>>>>>> your change. (The same argument holds for GCAllocPriority, but I >>>>>>> prefer to remove that enum all together as I described above.) >>>>>> See above. >>>>> >>>>> This is not the same issue as above. What I'm saying is that your >>>>> changes to GCAllocPurpose made it less readable without adding any >>>>> extra semantics. Please revert to this change. >>>>> >>>>>>> >>>>>>> Have you considered moving the _retired field from >>>>>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>>>>> retire() method virtual? (I do think your change to virtual is >>>>>>> needed in the current code, so good catch! But I think it might >>>>>>> make sense to have the logic of G1ParGCAllocBuffer::retire() in >>>>>>> ParGCAllocBuffer::retire() instead.) >>>>>> In G1ParGCAllocBuffer, we need the field _retired to handle >>>>>> buffer allocation failure. This is handled differently for other >>>>>> collectors. For example, >>>>>> ParScanThreadState::alloc_in_to_space_slow in ParNew. Thus, >>>>>> moving the _retired field up to its super class will involve >>>>>> additional efforts. This is supposed to be investigated in >>>>>> another CR JDK-7127700. >>>>> >>>>> OK. Good. >>>>> >>>>> Thanks, >>>>> Bengt >>>>> >>>>>>> >>>>>>> A couple of minor things: >>>>>>> >>>>>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>>>>> >>>>>>> should be: >>>>>>> >>>>>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>>>>> Done. >>>>>>> >>>>>>> Please add spaces before and after "=" here: >>>>>>> 1804 size_t result=0; >>>>>> Done. >>>>>>> >>>>>>> There are two spaces after "=" here: >>>>>>> 1812 G1ParGCAllocBuffer* retired = >>>>>>> _priority_buffer[GCAllocPriority1]; >>>>>> Done. >>>>>>> >>>>>>> Also, in g1CollectedHeap.hpp you have updated the copyright year >>>>>>> but not in parGCAllocBuffer.hpp. As you know we in the GC team >>>>>>> have agreed not to update the copyright year. If you absolutely >>>>>>> want to do it I think you should do it the same way in all files. >>>>>> Done. >>>>>>> >>>>>>> Thanks, >>>>>>> Bengt >>>>>>> >>>>>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>>>>> Can I have a couple of reviewers please? >>>>>>>> >>>>>>>> Thank you. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> a new webrev >>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>>>>> >>>>>>>>> diff: >>>>>>>>> (1) John Cuthbertson and I figured out the way to handle >>>>>>>>> "retire an old buffer, allocate and set a new one" and it can >>>>>>>>> possibly make the usage of allocation buffer a little more >>>>>>>>> efficient. >>>>>>>>> (2) Make the assertion as John suggested and provide some >>>>>>>>> harness (i.e. making retire() a virtual function) to cope with >>>>>>>>> it. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>>>>> Hi Tao, >>>>>>>>>> >>>>>>>>>> This looks excellent. One minor question: does it make sense >>>>>>>>>> to assert that each buffer has been retired? It might save a >>>>>>>>>> missed call to PSS::retire_alloc_buffers. I'll leave the >>>>>>>>>> decision to you. Ship it. >>>>>>>>>> >>>>>>>>>> JohnC >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>>>>> To the open list: >>>>>>>>>>> >>>>>>>>>>> new webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>>>>> >>>>>>>>>>> I took suggestion from many reviewers into consideration and >>>>>>>>>>> came up with the current cleaner solution. >>>>>>>>>>> >>>>>>>>>>> Thank you. >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>>>>> What's the status of this review? >>>>>>>>>>>> >>>>>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>>>>> from Thomas. >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>>>>> 6976350 G1: deal with fragmentation while copying objects >>>>>>>>>>>>> during GC >>>>>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>>>>> >>>>>>>>>>>>> webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>>>>> >>>>>>>>>>>>> changeset: >>>>>>>>>>>>> Basically, we want to reuse more of par-allocation buffers >>>>>>>>>>>>> instead of retiring it immediately when it encounters an >>>>>>>>>>>>> object larger than its remaining part. >>>>>>>>>>>>> >>>>>>>>>>>>> (1) instead of previously using one allocation buffer per >>>>>>>>>>>>> GC purpose, we use N(=2) buffers per GC purpose and modify >>>>>>>>>>>>> the corresponding code. The changeset would easily scale >>>>>>>>>>>>> up to whatever N (though Tony Printezis suggests 2, or 3 >>>>>>>>>>>>> may be good enough) >>>>>>>>>>>>> >>>>>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>>>>> removed due to its never being called. >>>>>>>>>>>>> access modifier (public) before trim_queue() is redundant. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tao.mao at oracle.com Mon Jun 3 21:23:59 2013 From: tao.mao at oracle.com (Tao Mao) Date: Mon, 03 Jun 2013 14:23:59 -0700 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51AD02AB.4090902@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> <51AAD064.8030600@oracle.com> <51AC50F3.3000308@oracle.com> <51ACEBA1.40302@oracle.com> <51AD02AB.4090902@oracle.com> Message-ID: <51AD096F.4080100@oracle.com> Thank you, Bengt. I've included your last suggestion in the final webrev. http://cr.openjdk.java.net/~tamao/6976350/webrev.08/ Tao On 6/3/13 1:55 PM, Bengt Rutisson wrote: > > Tao, > > On 6/3/13 9:16 PM, Tao Mao wrote: >> new webrev: >> http://cr.openjdk.java.net/~tamao/6976350/webrev.07/ > > This looks good to me. Thanks for working through all of these iterations! > > There are superfluous spaces on these lines before _priority_buffer[0]: > > 1803 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; > 1812 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; > > Other than that it looks good! > >> >> Please see inline. > > Thanks for the detailed description about retiring alloc buffers. It > was very helpful. > > Bengt > >> >> Thanks. >> Tao >> >> On 6/3/13 1:16 AM, Bengt Rutisson wrote: >>> >>> Hi Tao, >>> >>> On 6/2/13 6:56 AM, Tao Mao wrote: >>>> The new webrev is updated. >>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ >>> >>> Thanks for fixing this. I think it looks better. Still have some >>> comments: >>> >>> >>> Line 78, int const GCAllocPriorityMax = 2; >>> >>> I would prefer that this was a "private static const int" inside >>> G1ParGCAllocBufferContainer. You could call it _priority_max to >>> avoid the assignment in the constructor. >> Done. >> >>> >>> >>> I think the name select_retired_buf() is a bit confusing. The way it >>> is used I think the code would be more readable if we just inlined 0 >>> in the code. >> Inlining done. >> >>> >>> >>> In G1ParScanThreadState::allocate_slow() I think we might miss >>> retiring the alloc buffers, right? >>> >>> We now call retire_and_set_buf() after we have we have tried the >>> allcoation. If the allocation fails I think we have to retired both >>> alloc buffers since both of them may have been allocated into. >> The current implementation is right. >> >> (1) Even if the code reaches the point where it need to allocate a >> new buffer and fails, the old buffer is still usable. There's no >> reason we should retire it entirely. >> >> In addition, I designed to keep the buffer when the allocation fails >> so that the code doesn't need additional checkings in order to make >> sure the buffer is still valid, when trying to allocate a new object >> and to retire it per se. >> >> In fact, the current implementation simplifies the logic of object >> allocation in the buffer and retiring buffers if you get it. >> >> (2) Please check out the function retire_alloc_buffers(). It guards >> the clearing of all buffers in the end of copying phase, so we don't >> have to worry about that part. >> >> (3) A subtle benefit to mention: I still keep the buffer when the >> attempted allocation fails such way that we hope that the buffer may >> be allocated again to contain a new "smaller" object. It can happen >> to improve heap efficiency. >> >>> >>> I also think the name retire_and_set_buf() indicates that this >>> method does too much. I would prefer to have two different methods. >>> One retire_current_buf() that retires the current alloc buffer and >>> probably also shifts the buffers (what adjust_priority_order() does >>> now) and one that sets up a new buffer. >> I think it's a single operation and can't be separated. If we >> separated this function, we would end up with exposing unnecessary >> details to the caller. Also, the order of retire(), set_buf(), >> set_word_size() and adjust_priority_order() matters. I don't want to >> the caller to handle this rather than handle the ordering issue in >> class G1ParGCAllocBufferContainer. >> >>> >>> This will probably also be useful if we need to only retire buffers >>> in the allocation failure case I described above. >> As I mentioned above, retiring buffers upon the allocation failure >> would introduce additional complexity to handling the buffer usage >> and, even, retiring process itself. Please also refer to the last >> third comments above. >> >>> >>> Thanks, >>> Bengt >>> >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/30/13 10:56 PM, Bengt Rutisson wrote: >>>>> >>>>> Hi again, >>>>> >>>>> Just realized that I did this review a bit too early in the >>>>> morning...before the morning coffee... One correction below. >>>>> >>>>> On 5/31/13 7:44 AM, Bengt Rutisson wrote: >>>>>> >>>>>> >>>>>> Hi Tao, >>>>>> >>>>>> Comments inline, >>>>>> >>>>>> On 5/31/13 3:26 AM, Tao Mao wrote: >>>>>>> Please see inline. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>>>>>> >>>>>>>> Hi Tao, >>>>>>>> >>>>>>>> I think the code is a little bit confused about whether >>>>>>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>>>>>> AllocPriorites or just 2. All the for loops indicate that we >>>>>>>> think we might want to change from 2 to a larger number in the >>>>>>>> future. But the naming of a method like >>>>>>>> words_remaining_in_retired() indicate that there can only be >>>>>>>> one retired region. With the current implementation I think >>>>>>>> words_remaining_in_retired() should be called something like >>>>>>>> words_remaining_in_priority0_buffer(). >>>>>>> Done. >>>>>>> changed to words_remaining_in_priority1_buffer() >>>>>> >>>>>> Hm. Isn't this a bug? I think you want the method to be called >>>>>> words_remaining_in_priority0_buffer() and return the remaining >>>>>> words in the priority0 buffer. You call the method before you do >>>>>> alloc_buf->retire_and_set_buf(), so the priority1 buffer is >>>>>> probably not the one you are interested in. >>>>> >>>>> My bad. I thought the priorities were zero indexed, but because of >>>>> your enum they are one indexed. So, >>>>> words_remaining_in_priority1_buffer() is correct here. >>>>> >>>>> Bengt >>>>> >>>>>> >>>>>>>> >>>>>>>> I think it would be good to make this code truly general with >>>>>>>> respect to the number of priorities. We can then use 2 as >>>>>>>> default, but make sure that the code works with more >>>>>>>> priorities. To do that I think we should remove the enum >>>>>>>> GCAllocPriority and instead have a field in >>>>>>>> G1MultiParGCAllocBuffer that contains the maximum number of >>>>>>>> priorities. I think that will make the code more general and >>>>>>>> easier to read. The for loops would look like: >>>>>>>> >>>>>>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>>>>>> // do stuff >>>>>>>> } >>>>>>> It's more like code style issue. In fact, it was done this way >>>>>>> according to the Jon's earlier suggestion. Second, if we want to >>>>>>> change #buffer to 3 (it wont give more benefits to define more >>>>>>> than that number), we only need to add one more enum value, i.e. >>>>>>> GCAllocPriority3. >>>>>> >>>>>> Let me clarify a bit why I don't like the GCAllocPriority enum. >>>>>> There is really no reason to use an enum here. You are just >>>>>> making code complicated without adding any semantics. You always >>>>>> want to use 0-max and the order is important. This is exactly >>>>>> what you get from an normal int. >>>>>> >>>>>> The enum GCAllocPurpose is different since there is no natural >>>>>> order between GCAllocForTenured and GCAllocForSurvived. Thus, an >>>>>> enum makes sense there. >>>>>> >>>>>> So, please remove the GCAllocPriority enum. >>>>>> >>>>>>>> >>>>>>>> I find the name G1MultiParGCAllocBuffer confusing since it is >>>>>>>> not inheriting G1ParGCAllocBuffer. Maybe G1AllocBufferContainer >>>>>>>> or something like that would make more sense? >>>>>>> Done. >>>>>>> Changed to G1ParGCAllocBufferContainer >>>>>>>> >>>>>>>> I don't understand why you added initialization values to >>>>>>>> GCAllocPurpose. You are only using the values that are default >>>>>>>> in C++ anyway: 0, 1, 2. At least if you avoid adding the >>>>>>>> GCAllocPurposeStart value. I think it was more readable before >>>>>>>> your change. (The same argument holds for GCAllocPriority, but >>>>>>>> I prefer to remove that enum all together as I described above.) >>>>>>> See above. >>>>>> >>>>>> This is not the same issue as above. What I'm saying is that your >>>>>> changes to GCAllocPurpose made it less readable without adding >>>>>> any extra semantics. Please revert to this change. >>>>>> >>>>>>>> >>>>>>>> Have you considered moving the _retired field from >>>>>>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>>>>>> retire() method virtual? (I do think your change to virtual is >>>>>>>> needed in the current code, so good catch! But I think it might >>>>>>>> make sense to have the logic of G1ParGCAllocBuffer::retire() in >>>>>>>> ParGCAllocBuffer::retire() instead.) >>>>>>> In G1ParGCAllocBuffer, we need the field _retired to handle >>>>>>> buffer allocation failure. This is handled differently for other >>>>>>> collectors. For example, >>>>>>> ParScanThreadState::alloc_in_to_space_slow in ParNew. Thus, >>>>>>> moving the _retired field up to its super class will involve >>>>>>> additional efforts. This is supposed to be investigated in >>>>>>> another CR JDK-7127700. >>>>>> >>>>>> OK. Good. >>>>>> >>>>>> Thanks, >>>>>> Bengt >>>>>> >>>>>>>> >>>>>>>> A couple of minor things: >>>>>>>> >>>>>>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>>>>>> >>>>>>>> should be: >>>>>>>> >>>>>>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>>>>>> Done. >>>>>>>> >>>>>>>> Please add spaces before and after "=" here: >>>>>>>> 1804 size_t result=0; >>>>>>> Done. >>>>>>>> >>>>>>>> There are two spaces after "=" here: >>>>>>>> 1812 G1ParGCAllocBuffer* retired = >>>>>>>> _priority_buffer[GCAllocPriority1]; >>>>>>> Done. >>>>>>>> >>>>>>>> Also, in g1CollectedHeap.hpp you have updated the copyright >>>>>>>> year but not in parGCAllocBuffer.hpp. As you know we in the GC >>>>>>>> team have agreed not to update the copyright year. If you >>>>>>>> absolutely want to do it I think you should do it the same way >>>>>>>> in all files. >>>>>>> Done. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Bengt >>>>>>>> >>>>>>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>>>>>> Can I have a couple of reviewers please? >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> a new webrev >>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>>>>>> >>>>>>>>>> diff: >>>>>>>>>> (1) John Cuthbertson and I figured out the way to handle >>>>>>>>>> "retire an old buffer, allocate and set a new one" and it can >>>>>>>>>> possibly make the usage of allocation buffer a little more >>>>>>>>>> efficient. >>>>>>>>>> (2) Make the assertion as John suggested and provide some >>>>>>>>>> harness (i.e. making retire() a virtual function) to cope >>>>>>>>>> with it. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>>>>>> Hi Tao, >>>>>>>>>>> >>>>>>>>>>> This looks excellent. One minor question: does it make sense >>>>>>>>>>> to assert that each buffer has been retired? It might save a >>>>>>>>>>> missed call to PSS::retire_alloc_buffers. I'll leave the >>>>>>>>>>> decision to you. Ship it. >>>>>>>>>>> >>>>>>>>>>> JohnC >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>>>>>> To the open list: >>>>>>>>>>>> >>>>>>>>>>>> new webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>>>>>> >>>>>>>>>>>> I took suggestion from many reviewers into consideration >>>>>>>>>>>> and came up with the current cleaner solution. >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Tao >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>>>>>> What's the status of this review? >>>>>>>>>>>>> >>>>>>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>>>>>> from Thomas. >>>>>>>>>>>>> >>>>>>>>>>>>> Jon >>>>>>>>>>>>> >>>>>>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>>>>>> 6976350 G1: deal with fragmentation while copying objects >>>>>>>>>>>>>> during GC >>>>>>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>>>>>> >>>>>>>>>>>>>> webrev: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> changeset: >>>>>>>>>>>>>> Basically, we want to reuse more of par-allocation >>>>>>>>>>>>>> buffers instead of retiring it immediately when it >>>>>>>>>>>>>> encounters an object larger than its remaining part. >>>>>>>>>>>>>> >>>>>>>>>>>>>> (1) instead of previously using one allocation buffer per >>>>>>>>>>>>>> GC purpose, we use N(=2) buffers per GC purpose and >>>>>>>>>>>>>> modify the corresponding code. The changeset would easily >>>>>>>>>>>>>> scale up to whatever N (though Tony Printezis suggests 2, >>>>>>>>>>>>>> or 3 may be good enough) >>>>>>>>>>>>>> >>>>>>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>>>>>> removed due to its never being called. >>>>>>>>>>>>>> access >>>>>>>>>>>>>> modifier (public) before trim_queue() is redundant. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Tue Jun 4 04:44:27 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Tue, 04 Jun 2013 06:44:27 +0200 Subject: Request for review: 6976350 G1: deal with fragmentation while copying objects during GC In-Reply-To: <51AD096F.4080100@oracle.com> References: <5106DDD7.6090900@oracle.com> <5192AC15.90703@oracle.com> <5192B548.2010801@oracle.com> <5194757F.4070505@oracle.com> <519ABBBA.80103@oracle.com> <519EA6EA.1080308@oracle.com> <51A74BBC.7070003@oracle.com> <51A7FC3A.4030601@oracle.com> <51A838C1.6070100@oracle.com> <51A83B7E.9060808@oracle.com> <51AAD064.8030600@oracle.com> <51AC50F3.3000308@oracle.com> <51ACEBA1.40302@oracle.com> <51AD02AB.4090902@oracle.com> <51AD096F.4080100@oracle.com> Message-ID: <51AD70AB.2040406@oracle.com> On 6/3/13 11:23 PM, Tao Mao wrote: > Thank you, Bengt. I've included your last suggestion in the final webrev. > http://cr.openjdk.java.net/~tamao/6976350/webrev.08/ > Looks good. Bengt > Tao > > > On 6/3/13 1:55 PM, Bengt Rutisson wrote: >> >> Tao, >> >> On 6/3/13 9:16 PM, Tao Mao wrote: >>> new webrev: >>> http://cr.openjdk.java.net/~tamao/6976350/webrev.07/ >> >> This looks good to me. Thanks for working through all of these >> iterations! >> >> There are superfluous spaces on these lines before _priority_buffer[0]: >> >> 1803 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; >> 1812 G1ParGCAllocBuffer* retired_and_set = _priority_buffer[0]; >> >> Other than that it looks good! >> >>> >>> Please see inline. >> >> Thanks for the detailed description about retiring alloc buffers. It >> was very helpful. >> >> Bengt >> >>> >>> Thanks. >>> Tao >>> >>> On 6/3/13 1:16 AM, Bengt Rutisson wrote: >>>> >>>> Hi Tao, >>>> >>>> On 6/2/13 6:56 AM, Tao Mao wrote: >>>>> The new webrev is updated. >>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.06/ >>>> >>>> Thanks for fixing this. I think it looks better. Still have some >>>> comments: >>>> >>>> >>>> Line 78, int const GCAllocPriorityMax = 2; >>>> >>>> I would prefer that this was a "private static const int" inside >>>> G1ParGCAllocBufferContainer. You could call it _priority_max to >>>> avoid the assignment in the constructor. >>> Done. >>> >>>> >>>> >>>> I think the name select_retired_buf() is a bit confusing. The way >>>> it is used I think the code would be more readable if we just >>>> inlined 0 in the code. >>> Inlining done. >>> >>>> >>>> >>>> In G1ParScanThreadState::allocate_slow() I think we might miss >>>> retiring the alloc buffers, right? >>>> >>>> We now call retire_and_set_buf() after we have we have tried the >>>> allcoation. If the allocation fails I think we have to retired both >>>> alloc buffers since both of them may have been allocated into. >>> The current implementation is right. >>> >>> (1) Even if the code reaches the point where it need to allocate a >>> new buffer and fails, the old buffer is still usable. There's no >>> reason we should retire it entirely. >>> >>> In addition, I designed to keep the buffer when the allocation fails >>> so that the code doesn't need additional checkings in order to make >>> sure the buffer is still valid, when trying to allocate a new object >>> and to retire it per se. >>> >>> In fact, the current implementation simplifies the logic of object >>> allocation in the buffer and retiring buffers if you get it. >>> >>> (2) Please check out the function retire_alloc_buffers(). It guards >>> the clearing of all buffers in the end of copying phase, so we don't >>> have to worry about that part. >>> >>> (3) A subtle benefit to mention: I still keep the buffer when the >>> attempted allocation fails such way that we hope that the buffer may >>> be allocated again to contain a new "smaller" object. It can happen >>> to improve heap efficiency. >>> >>>> >>>> I also think the name retire_and_set_buf() indicates that this >>>> method does too much. I would prefer to have two different methods. >>>> One retire_current_buf() that retires the current alloc buffer and >>>> probably also shifts the buffers (what adjust_priority_order() does >>>> now) and one that sets up a new buffer. >>> I think it's a single operation and can't be separated. If we >>> separated this function, we would end up with exposing unnecessary >>> details to the caller. Also, the order of retire(), set_buf(), >>> set_word_size() and adjust_priority_order() matters. I don't want to >>> the caller to handle this rather than handle the ordering issue in >>> class G1ParGCAllocBufferContainer. >>> >>>> >>>> This will probably also be useful if we need to only retire buffers >>>> in the allocation failure case I described above. >>> As I mentioned above, retiring buffers upon the allocation failure >>> would introduce additional complexity to handling the buffer usage >>> and, even, retiring process itself. Please also refer to the last >>> third comments above. >>> >>>> >>>> Thanks, >>>> Bengt >>>> >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/30/13 10:56 PM, Bengt Rutisson wrote: >>>>>> >>>>>> Hi again, >>>>>> >>>>>> Just realized that I did this review a bit too early in the >>>>>> morning...before the morning coffee... One correction below. >>>>>> >>>>>> On 5/31/13 7:44 AM, Bengt Rutisson wrote: >>>>>>> >>>>>>> >>>>>>> Hi Tao, >>>>>>> >>>>>>> Comments inline, >>>>>>> >>>>>>> On 5/31/13 3:26 AM, Tao Mao wrote: >>>>>>>> Please see inline. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/30/13 5:53 AM, Bengt Rutisson wrote: >>>>>>>>> >>>>>>>>> Hi Tao, >>>>>>>>> >>>>>>>>> I think the code is a little bit confused about whether >>>>>>>>> G1MultiParGCAllocBuffer can handle an arbitary number of >>>>>>>>> AllocPriorites or just 2. All the for loops indicate that we >>>>>>>>> think we might want to change from 2 to a larger number in the >>>>>>>>> future. But the naming of a method like >>>>>>>>> words_remaining_in_retired() indicate that there can only be >>>>>>>>> one retired region. With the current implementation I think >>>>>>>>> words_remaining_in_retired() should be called something like >>>>>>>>> words_remaining_in_priority0_buffer(). >>>>>>>> Done. >>>>>>>> changed to words_remaining_in_priority1_buffer() >>>>>>> >>>>>>> Hm. Isn't this a bug? I think you want the method to be called >>>>>>> words_remaining_in_priority0_buffer() and return the remaining >>>>>>> words in the priority0 buffer. You call the method before you do >>>>>>> alloc_buf->retire_and_set_buf(), so the priority1 buffer is >>>>>>> probably not the one you are interested in. >>>>>> >>>>>> My bad. I thought the priorities were zero indexed, but because >>>>>> of your enum they are one indexed. So, >>>>>> words_remaining_in_priority1_buffer() is correct here. >>>>>> >>>>>> Bengt >>>>>> >>>>>>> >>>>>>>>> >>>>>>>>> I think it would be good to make this code truly general with >>>>>>>>> respect to the number of priorities. We can then use 2 as >>>>>>>>> default, but make sure that the code works with more >>>>>>>>> priorities. To do that I think we should remove the enum >>>>>>>>> GCAllocPriority and instead have a field in >>>>>>>>> G1MultiParGCAllocBuffer that contains the maximum number of >>>>>>>>> priorities. I think that will make the code more general and >>>>>>>>> easier to read. The for loops would look like: >>>>>>>>> >>>>>>>>> for (int pr = 0; pr < _max_priorities; ++pr) { >>>>>>>>> // do stuff >>>>>>>>> } >>>>>>>> It's more like code style issue. In fact, it was done this way >>>>>>>> according to the Jon's earlier suggestion. Second, if we want >>>>>>>> to change #buffer to 3 (it wont give more benefits to define >>>>>>>> more than that number), we only need to add one more enum >>>>>>>> value, i.e. GCAllocPriority3. >>>>>>> >>>>>>> Let me clarify a bit why I don't like the GCAllocPriority enum. >>>>>>> There is really no reason to use an enum here. You are just >>>>>>> making code complicated without adding any semantics. You always >>>>>>> want to use 0-max and the order is important. This is exactly >>>>>>> what you get from an normal int. >>>>>>> >>>>>>> The enum GCAllocPurpose is different since there is no natural >>>>>>> order between GCAllocForTenured and GCAllocForSurvived. Thus, an >>>>>>> enum makes sense there. >>>>>>> >>>>>>> So, please remove the GCAllocPriority enum. >>>>>>> >>>>>>>>> >>>>>>>>> I find the name G1MultiParGCAllocBuffer confusing since it is >>>>>>>>> not inheriting G1ParGCAllocBuffer. Maybe >>>>>>>>> G1AllocBufferContainer or something like that would make more >>>>>>>>> sense? >>>>>>>> Done. >>>>>>>> Changed to G1ParGCAllocBufferContainer >>>>>>>>> >>>>>>>>> I don't understand why you added initialization values to >>>>>>>>> GCAllocPurpose. You are only using the values that are default >>>>>>>>> in C++ anyway: 0, 1, 2. At least if you avoid adding the >>>>>>>>> GCAllocPurposeStart value. I think it was more readable before >>>>>>>>> your change. (The same argument holds for GCAllocPriority, but >>>>>>>>> I prefer to remove that enum all together as I described above.) >>>>>>>> See above. >>>>>>> >>>>>>> This is not the same issue as above. What I'm saying is that >>>>>>> your changes to GCAllocPurpose made it less readable without >>>>>>> adding any extra semantics. Please revert to this change. >>>>>>> >>>>>>>>> >>>>>>>>> Have you considered moving the _retired field from >>>>>>>>> G1ParGCAllocBuffer to ParGCAllocBuffer instead of making the >>>>>>>>> retire() method virtual? (I do think your change to virtual is >>>>>>>>> needed in the current code, so good catch! But I think it >>>>>>>>> might make sense to have the logic of >>>>>>>>> G1ParGCAllocBuffer::retire() in ParGCAllocBuffer::retire() >>>>>>>>> instead.) >>>>>>>> In G1ParGCAllocBuffer, we need the field _retired to handle >>>>>>>> buffer allocation failure. This is handled differently for >>>>>>>> other collectors. For example, >>>>>>>> ParScanThreadState::alloc_in_to_space_slow in ParNew. Thus, >>>>>>>> moving the _retired field up to its super class will involve >>>>>>>> additional efforts. This is supposed to be investigated in >>>>>>>> another CR JDK-7127700. >>>>>>> >>>>>>> OK. Good. >>>>>>> >>>>>>> Thanks, >>>>>>> Bengt >>>>>>> >>>>>>>>> >>>>>>>>> A couple of minor things: >>>>>>>>> >>>>>>>>> 1800 if (finish_undo != true) ShouldNotReachHere(); >>>>>>>>> >>>>>>>>> should be: >>>>>>>>> >>>>>>>>> 1800 if (!finish_undo) ShouldNotReachHere(); >>>>>>>> Done. >>>>>>>>> >>>>>>>>> Please add spaces before and after "=" here: >>>>>>>>> 1804 size_t result=0; >>>>>>>> Done. >>>>>>>>> >>>>>>>>> There are two spaces after "=" here: >>>>>>>>> 1812 G1ParGCAllocBuffer* retired = >>>>>>>>> _priority_buffer[GCAllocPriority1]; >>>>>>>> Done. >>>>>>>>> >>>>>>>>> Also, in g1CollectedHeap.hpp you have updated the copyright >>>>>>>>> year but not in parGCAllocBuffer.hpp. As you know we in the GC >>>>>>>>> team have agreed not to update the copyright year. If you >>>>>>>>> absolutely want to do it I think you should do it the same way >>>>>>>>> in all files. >>>>>>>> Done. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Bengt >>>>>>>>> >>>>>>>>> On 5/24/13 1:31 AM, Tao Mao wrote: >>>>>>>>>> Can I have a couple of reviewers please? >>>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/20/13 5:11 PM, Tao Mao wrote: >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> a new webrev >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.04/ >>>>>>>>>>> >>>>>>>>>>> diff: >>>>>>>>>>> (1) John Cuthbertson and I figured out the way to handle >>>>>>>>>>> "retire an old buffer, allocate and set a new one" and it >>>>>>>>>>> can possibly make the usage of allocation buffer a little >>>>>>>>>>> more efficient. >>>>>>>>>>> (2) Make the assertion as John suggested and provide some >>>>>>>>>>> harness (i.e. making retire() a virtual function) to cope >>>>>>>>>>> with it. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/15/13 10:58 PM, John Cuthbertson wrote: >>>>>>>>>>>> Hi Tao, >>>>>>>>>>>> >>>>>>>>>>>> This looks excellent. One minor question: does it make >>>>>>>>>>>> sense to assert that each buffer has been retired? It might >>>>>>>>>>>> save a missed call to PSS::retire_alloc_buffers. I'll leave >>>>>>>>>>>> the decision to you. Ship it. >>>>>>>>>>>> >>>>>>>>>>>> JohnC >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 5/14/2013 3:06 PM, Tao Mao wrote: >>>>>>>>>>>>> To the open list: >>>>>>>>>>>>> >>>>>>>>>>>>> new webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.03/ >>>>>>>>>>>>> >>>>>>>>>>>>> I took suggestion from many reviewers into consideration >>>>>>>>>>>>> and came up with the current cleaner solution. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you. >>>>>>>>>>>>> Tao >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/14/13 2:26 PM, Jon Masamitsu wrote: >>>>>>>>>>>>>> What's the status of this review? >>>>>>>>>>>>>> >>>>>>>>>>>>>> The last mail I could find in my mail boxes was a comment >>>>>>>>>>>>>> from Thomas. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 1/28/13 12:21 PM, Tao Mao wrote: >>>>>>>>>>>>>>> 6976350 G1: deal with fragmentation while copying >>>>>>>>>>>>>>> objects during GC >>>>>>>>>>>>>>> https://jbs.oracle.com/bugs/browse/JDK-6976350 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> webrev: >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/6976350/webrev.00/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> changeset: >>>>>>>>>>>>>>> Basically, we want to reuse more of par-allocation >>>>>>>>>>>>>>> buffers instead of retiring it immediately when it >>>>>>>>>>>>>>> encounters an object larger than its remaining part. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (1) instead of previously using one allocation buffer >>>>>>>>>>>>>>> per GC purpose, we use N(=2) buffers per GC purpose and >>>>>>>>>>>>>>> modify the corresponding code. The changeset would >>>>>>>>>>>>>>> easily scale up to whatever N (though Tony Printezis >>>>>>>>>>>>>>> suggests 2, or 3 may be good enough) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *(2) Two places of cleanup: allocate_during_gc_slow() is >>>>>>>>>>>>>>> removed due to its never being called. >>>>>>>>>>>>>>> access modifier (public) before trim_queue() is redundant. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Tue Jun 4 06:10:16 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Tue, 04 Jun 2013 08:10:16 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <1369729575.2616.27.camel@cirrus> References: <1369729575.2616.27.camel@cirrus> Message-ID: <51AD84C8.7010300@oracle.com> Hi Thomas, First, a couple of questions about the CR. It is a P4 enhancement and we have passed feature complete. Should this still go in to hs25? If yes, could you update the CR a bit with why this extra remembered set information is important. Just so we understand why it needs to go in now. Also, I think it would be good if the CR mentions that the remembered set information is now printed before GC as well as after. Some code comments: g1CollectedHeap.cpp. This comment needs to be updated since we are _before_ gc: 3542 // we are at the end of the GC. Total collections has already been increased. I don't really like the flag G1SummarizeRSetStatsTime. I would prefer two separate flags, G1SummarizeRSetStatsBeforeGC and G1SummarizeRSetStatsAfterGC. Or maybe just one flag that always prints before and after GC, similar to PrintHeapAtGC. g1RemSetSummary.cpp Why is region_type_counter_t a struct and not a class? I would prefer it to be a class and be called something like RegionTypeCounter. Also I think it would be nicer to declare it outside of HRRSStatsIter. I think the getters for young(), humongous(), free() and other() are unnecessary. I would use the variables directly since I think the getters introduce a lot of line noise in this class. If you want to keep the getters I would suggest to make them one-liners similar to the other getters in the class, so: region_type_counter_t& young() { return _young; } Also, isn't the "other" category actually "old" regions? If yes, would you mind changing the name of them to "old"? The method print_summary_on() is very hard to read. Can you try to format it in a way that makes it more readable? I think just adding a few line breaks between the different section would help a lot. I have not looked too closely at the tests but some comments: In TestSummarizeRSetStats.java you could minimize the diff and make the test more readable by using static imports for TestSummarizeRSetStatsTools.runTest() and TestSummarizeRSetStatsTools.expectStatistics() I think the name TestSummarizeRSetStats2 is a bit awkward. How about TestSummarizeRSetStatsPerRegion? The method TestSummarizeRSetStats2.expectStatistics() is a bit surprising since the other tests use TestSummarizeRSetStatsTools.expectStatistics(). Could we move TestSummarizeRSetStats2.expectStatistics() into TestSummarizeRSetStatsTools too? Either with a new name or we could parametrize the existing expectStatistics() method in there. Thanks, Bengt On 5/28/13 10:26 AM, Thomas Schatzl wrote: > Hi all, > > could I get some reviews for the following change? It improves the > remembered set summary information available with -XX: > +G1SummarizeRSetStats by adding memory size and occupancy summaries > broken down by region type (young, humonguous, free, other). > > Webrev: > http://cr.openjdk.java.net/~tschatzl/8014078/webrev/ > > Note that this webrev is based on 8013895; I provided a full changeset > including 8013895 at > http://cr.openjdk.java.net/~tschatzl/8014078/webrev/8014078.8013895.diff > > JIRA: > https://jbs.oracle.com/bugs/browse/JDK-8014078 > > bugs.sun.com: > http://bugs.sun.com/view_bug.do?bug_id=8014078 > > Testing: > jprt, jtreg tests > > About the change: > > - the output of G1SummarizeRSetStats now looks as follows (for > dacapo/eclipse): > > After GC RS summary > > Concurrent RS processed 15299 cards > Of 64 completed buffers: > 64 (100.0%) by concurrent RS threads. > 0 ( 0.0%) by mutator threads. > Concurrent RS threads times (s) > 0.06 > Concurrent sampling threads times (s) > 0.00 > Total heap region rem set sizes = 512K. Max = 13K. >>>>> 17K ( 3.4%) by 7 Young regions >>>>> 7K ( 1.4%) by 3 Humonguous regions >>>>> 157K ( 30.6%) by 64 Free regions >>>>> 330K ( 64.6%) by 44 Other regions > Static structures = 1K, free_lists = 254K. > 277592 occupied cards represented. >>>>> 0 ( 0.0%) entries by 7 Young regions >>>>> 2 ( 0.0%) entries by 3 Humonguous regions >>>>> 0 ( 0.0%) entries by 64 Free regions >>>>> 277590 (100.0%) entries by 44 Other regions > Max size region = > 3:(O)[0x00000000f0900000,0x00000000f0a00000,0x00000000f0a00000], size = > 14K, occupied = 25K. > Did 0 coarsenings. > > Notice the breakdown of memory size and card occupancy by region type > (marked with >>>> here). > > "Young" regions include all types of young regions (i.e. survivor) as > survivor does not have a meaning for g1 outside of a gc. > "Humonguous" and "Free" are self-explaining. "Other" regions include > everything else. > > - added a new flag G1SummarizeRSetStatsTime that is used to control the > time, i.e. before/after gc, when the periodicaly summaries are printed. > This integer is treated as a bitset, where a set first bit indicates a > request to print a summary after gc, the second bit a request to print a > summary before gc. > > The alternative (which I'd like more) would be to change > G1SummarizeRSetStats to act like this new flag to avoid introducing a > new flag; however I'm hesitant to change this as G1SummarizeRSetStats is > a publicly available flag (although diagnostic). > > - cleanup of the test cases: a large part of the code base for this CR > reuses utility methods from 8013895, so I went ahead and factored out > common code into a separate file that both the tests for 801385 and this > CR use. > > Thanks, > Thomas > From thomas.schatzl at oracle.com Tue Jun 4 11:39:27 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Jun 2013 13:39:27 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <51AD84C8.7010300@oracle.com> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> Message-ID: <1370345967.2650.27.camel@cirrus> Hi, On Tue, 2013-06-04 at 08:10 +0200, Bengt Rutisson wrote: > Hi Thomas, > > First, a couple of questions about the CR. It is a P4 enhancement and we > have passed feature complete. Should this still go in to hs25? If yes, > could you update the CR a bit with why this extra remembered set > information is important. Just so we understand why it needs to go in > now. Also, I think it would be good if the CR mentions that the > remembered set information is now printed before GC as well as after. Updated. > > Some code comments: > > g1CollectedHeap.cpp. > > This comment needs to be updated since we are _before_ gc: > > 3542 // we are at the end of the GC. Total collections has already > been increased. Fixed. Forgot to update the comment. > I don't really like the flag G1SummarizeRSetStatsTime. I would prefer > two separate flags, G1SummarizeRSetStatsBeforeGC and > G1SummarizeRSetStatsAfterGC. Or maybe just one flag that always prints > before and after GC, similar to PrintHeapAtGC. I was thinking about that too but rejected it due to backwards compatibility. What to do about G1SummarizeRSetStats after introducing the Before/AfterGC flags? It seems awkward to need both G1SummarizeRSetStats and G1SummarizeRSetStatsBeforeGC/G1SummarizeRSetStatsAfterGC. What are the rules on removing diagnostic flags? (Note that I think that G1SummarizeRSetStats has had a lot of use - before cr 8013895 its output has been very buggy). > g1RemSetSummary.cpp > > Why is region_type_counter_t a struct and not a class? I would prefer it > to be a class and be called something like RegionTypeCounter. Also I > think it would be nicer to declare it outside of HRRSStatsIter. Fine with me. Done. > > I think the getters for young(), humongous(), free() and other() are > unnecessary. I would use the variables directly since I think the > getters introduce a lot of line noise in this class. If you want to keep > the getters I would suggest to make them one-liners similar to the other > getters in the class, so: > > region_type_counter_t& young() { return _young; } Removed the getters, in this case I agree on second thought. > > Also, isn't the "other" category actually "old" regions? If yes, would > you mind changing the name of them to "old"? G1 surprisingly does not have an explicit notion of old (and free, but there's at least a single predicate to determine that) regions, old is just what is left if you look at the if-then-else cascades in the code (not only in these changes). That's the reason for this name. I changed it though. > > The method print_summary_on() is very hard to read. Can you try to > format it in a way that makes it more readable? I think just adding a > few line breaks between the different section would help a lot. > Done. > I have not looked too closely at the tests but some comments: > > In TestSummarizeRSetStats.java you could minimize the diff and make the > test more readable by using static imports for > TestSummarizeRSetStatsTools.runTest() and > TestSummarizeRSetStatsTools.expectStatistics() I did not know that language feature... thanks! :) > > I think the name TestSummarizeRSetStats2 is a bit awkward. How about > TestSummarizeRSetStatsPerRegion? Done. > > The method TestSummarizeRSetStats2.expectStatistics() is a bit > surprising since the other tests use > TestSummarizeRSetStatsTools.expectStatistics(). Could we move > TestSummarizeRSetStats2.expectStatistics() into > TestSummarizeRSetStatsTools too? Either with a new name or we could I will clean this up - but I would like to have your comment on the flag issue first. Also, I will then send a new webrev. > parametrize the existing expectStatistics() method in there. Thanks, Thomas From bengt.rutisson at oracle.com Tue Jun 4 11:50:08 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Tue, 04 Jun 2013 13:50:08 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <1370345967.2650.27.camel@cirrus> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> <1370345967.2650.27.camel@cirrus> Message-ID: <51ADD470.6080707@oracle.com> Hi Thomas, Thanks for fixing these things. Some comments inline. On 6/4/13 1:39 PM, Thomas Schatzl wrote: > Hi, > > On Tue, 2013-06-04 at 08:10 +0200, Bengt Rutisson wrote: >> Hi Thomas, >> >> First, a couple of questions about the CR. It is a P4 enhancement and we >> have passed feature complete. Should this still go in to hs25? If yes, >> could you update the CR a bit with why this extra remembered set >> information is important. Just so we understand why it needs to go in >> now. Also, I think it would be good if the CR mentions that the >> remembered set information is now printed before GC as well as after. > Updated. > >> Some code comments: >> >> g1CollectedHeap.cpp. >> >> This comment needs to be updated since we are _before_ gc: >> >> 3542 // we are at the end of the GC. Total collections has already >> been increased. > Fixed. Forgot to update the comment. > >> I don't really like the flag G1SummarizeRSetStatsTime. I would prefer >> two separate flags, G1SummarizeRSetStatsBeforeGC and >> G1SummarizeRSetStatsAfterGC. Or maybe just one flag that always prints >> before and after GC, similar to PrintHeapAtGC. > I was thinking about that too but rejected it due to backwards > compatibility. What to do about G1SummarizeRSetStats after introducing > the Before/AfterGC flags? > > It seems awkward to need both G1SummarizeRSetStats and > G1SummarizeRSetStatsBeforeGC/G1SummarizeRSetStatsAfterGC. What are the > rules on removing diagnostic flags? I think I vote for only having G1SummarizeRSetStats and let that print the information both before and after GC. That is backwards compatible in the sense that you get at least the information you got before with this flag. It is also similar to the PrintHeapAtGC flag. > > (Note that I think that G1SummarizeRSetStats has had a lot of use - > before cr 8013895 its output has been very buggy). I think you mean that it has _not_ had a lot of use, right? > >> g1RemSetSummary.cpp >> >> Why is region_type_counter_t a struct and not a class? I would prefer it >> to be a class and be called something like RegionTypeCounter. Also I >> think it would be nicer to declare it outside of HRRSStatsIter. > Fine with me. Done. > >> I think the getters for young(), humongous(), free() and other() are >> unnecessary. I would use the variables directly since I think the >> getters introduce a lot of line noise in this class. If you want to keep >> the getters I would suggest to make them one-liners similar to the other >> getters in the class, so: >> >> region_type_counter_t& young() { return _young; } > Removed the getters, in this case I agree on second thought. > >> Also, isn't the "other" category actually "old" regions? If yes, would >> you mind changing the name of them to "old"? > G1 surprisingly does not have an explicit notion of old (and free, but > there's at least a single predicate to determine that) regions, old is > just what is left if you look at the if-then-else cascades in the code > (not only in these changes). > > That's the reason for this name. I changed it though. > >> The method print_summary_on() is very hard to read. Can you try to >> format it in a way that makes it more readable? I think just adding a >> few line breaks between the different section would help a lot. >> > Done. > >> I have not looked too closely at the tests but some comments: >> >> In TestSummarizeRSetStats.java you could minimize the diff and make the >> test more readable by using static imports for >> TestSummarizeRSetStatsTools.runTest() and >> TestSummarizeRSetStatsTools.expectStatistics() > I did not know that language feature... thanks! :) > >> I think the name TestSummarizeRSetStats2 is a bit awkward. How about >> TestSummarizeRSetStatsPerRegion? > Done. > >> The method TestSummarizeRSetStats2.expectStatistics() is a bit >> surprising since the other tests use >> TestSummarizeRSetStatsTools.expectStatistics(). Could we move >> TestSummarizeRSetStats2.expectStatistics() into >> TestSummarizeRSetStatsTools too? Either with a new name or we could > I will clean this up - but I would like to have your comment on the flag > issue first. Also, I will then send a new webrev. Sounds like a good plan. I hope what I wrote above is enough. Thanks, Bengt > >> parametrize the existing expectStatistics() method in there. > Thanks, > > Thomas > > From thomas.schatzl at oracle.com Tue Jun 4 14:58:33 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Jun 2013 16:58:33 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <51ADD470.6080707@oracle.com> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> <1370345967.2650.27.camel@cirrus> <51ADD470.6080707@oracle.com> Message-ID: <1370357913.2650.42.camel@cirrus> Hi all, On Tue, 2013-06-04 at 13:50 +0200, Bengt Rutisson wrote: > Hi Thomas, > > Thanks for fixing these things. A new webrev is up at http://cr.openjdk.java.net/~tschatzl/8014078/webrev.1/ Changes: - spacing suggested by Jesper - removed comment in g1CollectedHeap::gc_prologue() indicating that the following line subtracted one because we are at the end of collection; both statements were just wrong. - remove G1SummarizeRSetStatsTime and print summaries both at start and end of collection - renamed region_type_counter_t to RegionTypeCounter and moved it in the top level scope - removed superfluous getters - renamed "Other" regions to "Old" regions in output - added some line spacing in G1HRRSStatsIter::print_summary_on() - refactoring of test cases: - did _not_ use static imports because the java language does not allow them for the default package - removed the testing for the G1SummarizeRSetStatsTime flag - renamed of *Statistics() to *RSetSummaries() - TestSummarizeRSetStats2 renamed to TestSummarizeRSetStatsPerRegion - moved TestSummarizeRSetStatsPerRegion.expectStatistics to TestSummarizeRSetStatsTools.expectPerRegionRSetSummaries() - adapted test for 8013895 to account for printing the summaries both at start and end of gc Testing: jtreg passed for this cr and 8013895, jprt still running. > Some comments inline. > > On 6/4/13 1:39 PM, Thomas Schatzl wrote: > > Hi, > > > > On Tue, 2013-06-04 at 08:10 +0200, Bengt Rutisson wrote: > >> Hi Thomas, > > It seems awkward to need both G1SummarizeRSetStats and > > G1SummarizeRSetStatsBeforeGC/G1SummarizeRSetStatsAfterGC. What are the > > rules on removing diagnostic flags? > > I think I vote for only having G1SummarizeRSetStats and let that print > the information both before and after GC. That is backwards compatible > in the sense that you get at least the information you got before with > this flag. It is also similar to the PrintHeapAtGC flag. I do not really mind. > > (Note that I think that G1SummarizeRSetStats has had a lot of use - > > before cr 8013895 its output has been very buggy). > > I think you mean that it has _not_ had a lot of use, right? Sure :) > >> I have not looked too closely at the tests but some comments: > >> > >> In TestSummarizeRSetStats.java you could minimize the diff and make the > >> test more readable by using static imports for > >> TestSummarizeRSetStatsTools.runTest() and > >> TestSummarizeRSetStatsTools.expectStatistics() > > I did not know that language feature... thanks! :) > > > >> I think the name TestSummarizeRSetStats2 is a bit awkward. How about > >> TestSummarizeRSetStatsPerRegion? > > Done. > > > >> The method TestSummarizeRSetStats2.expectStatistics() is a bit > >> surprising since the other tests use > >> TestSummarizeRSetStatsTools.expectStatistics(). Could we move > >> TestSummarizeRSetStats2.expectStatistics() into > >> TestSummarizeRSetStatsTools too? Either with a new name or we could > > I will clean this up - but I would like to have your comment on the flag > > issue first. Also, I will then send a new webrev. > > Sounds like a good plan. I hope what I wrote above is enough. I hope too - if it isn't, just tell me. > >> parametrize the existing expectStatistics() method in there. I did not do that and kept two methods, although giving them to more distinct names. Thomas From thomas.schatzl at oracle.com Tue Jun 4 15:17:58 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 4 Jun 2013 08:17:58 -0700 (PDT) Subject: RFR (M/L): 8010722 assert: failed: heap size is too big for compressed oops (possibly CR 8009778) In-Reply-To: <1368542230.2677.0.camel@cirrus> References: <1368542230.2677.0.camel@cirrus> Message-ID: <1370359078.2650.55.camel@cirrus> Hi all, please have a look at a slightly updated version of this CR after comments from SQE and runtime. Changes: - moved the comment that explained the choice of a 100M default ClassMetaSpaceSize to the declaration of this value - changed the default size of ClassMetaSpaceSize to 128M as this is a value that results in less automatic resizing due to alignment (on request). Runtime intends to change this code in the future so that the ClassMetaSpaceSize is not dependent on java heap alignment any more (as far as I understood). - resolved a small merge conflict when updating to latest hotspot-gc: ParallelScavengeHeap::intra_heap_alignment() needs to be a static method to be used in a static context. This is not an issue here as it returns a constant value anyway. - added -XX:+UseParallelGC -XX:-UseParallelOldGC as gc combination to test in the test case. This required some changes in passing down these arguments to the tests themselves. Testing: jtreg tests, jprt There is a small introduction on the change below. On Tue, 2013-05-14 at 16:37 +0200, Thomas Schatzl wrote: > Hi all, > > can I have a review for the following change? > > In argument processing related to ergonomically determining compressed > oops use, there were a few use-before-set issues leading to crashes that > this fix tries to resolve. > > bugs.sun.com > http://bugs.sun.com/view_bug.do?bug_id=8010722 > > JIRA: > https://jbs.oracle.com/bugs/browse/JDK-8010722 > > webrev: > http://cr.openjdk.java.net/~tschatzl/8010722/webrev/ > > testing: > jprt, test cases > > Here's a walkthrough of the changes: > > As mentioned, the cause of this CR is that ergonomics for determining > the maximum java heap size usable for compressed oops uses variables > that are later changed ergonomically. > > It is best to look at the changes beginning from > Arguments::set_ergonomics_flags(): the idea of this change is to avoid > later overflow, so the change tries to conservatively estimate sizes of > the non-java heap parts. The complication is that not even the later > effective alignment of these heap parts has been determined at this > point. > > So the change first calculates the maximum possible heap alignment by > calling set_max_heap_alignment(); this size is influenced by OS page > sizes, so the change needs to initialize large pages by calling > os::large_page_init() in Arguments::parse(), before the call to > set_ergonomics_flags(). The maximum possible alignment is then > calculated by asking the active GC for its maximum alignment, as at this > point the GC has already been determined, the maximum page size, and > other requirements, like alignment for card table size etc. > > Now the code can calculate the conservative estimate for actual maximum > heap for compressed oops used in set_use_compressed_oops(), by > subtracting the conservatively aligned sizes of the other heap parts. > (In Arguments::max_heap_for_compressed_oops()) The result is the maximum > possible heap that can use compressed oops, minus the aligned metaspace > size, minus the aligned null page size. > > There is another circular dependency problem here, the metaspace size itself is later ergonomically sized; the change fixes this problem by anticipating that in Arguments::max_heap_for_compressed_oops(), using the same predicate for determining whether to ergonomically size it or not [this is CR8009778 I think]. > > The other changes are straightforward: the os_* changes result from that > large page initialization must be done earlier now; the changes in the > collectors themselves are simply about providing the collector's maximum > alignment. The change in Universe::reserve_heap() contains one assertion that checks whether the conservative estimate for alignment has been conservative enough earlier. > > The test case tests test cases from the CR that work now, and additional > border cases related to ergonomically deciding heap size for compressed > oops. > > One side effect of this change is that the ergonomically determined heap size is slighly smaller now (so that it always fits :). Thanks, Thomas From jon.masamitsu at oracle.com Tue Jun 4 17:09:12 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 04 Jun 2013 10:09:12 -0700 Subject: RFR: 8013590: NPG: Add a memory pool MXBean for Metaspace In-Reply-To: <20130528162227.GE2000@ehelin-thinkpad> References: <20130528162227.GE2000@ehelin-thinkpad> Message-ID: <51AE1F38.9030801@oracle.com> Erik, http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/src/share/vm/services/memoryPool.hpp.frames.html For _undefined_size you could use max_uintx defined in globalDefinitions.hpp. Using the name max_uintx makes it a little more clear that undefined mean a maximum value instead of an error value. http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/src/share/vm/services/memoryService.hpp.frames.html Really a nit but instead of 77 static MemoryPool* _cks_pool; _cks_pool really tells an reader very little. Would you consider _compressed_klass_pool? Or even _cklass_pool? http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/test/gc/metaspace/TestMetaspaceMemoryPool.java.html The functions assertEquals(), assertTrue(), and assertDefined() seem useful. Is there a more global place where they can be put so other can use them? Rest looks good. Jon On 5/28/13 9:22 AM, Erik Helin wrote: > Hi all, > > this change adds two memory pools for metaspace, one for Metaspace and > one for compressed klass space. The memory pool for compressed klass > space will only have valus that differ from 0 or -1 (undefined) if > compressed klass pointers are used. > > This change also adds a manager for the pools: Metaspace Manager. > > I have also added a test that checks that the metaspace manager is > present and that the two pools are present. The test also verifies that > the compressed klass space pool act correct according to the > UseCompressedKlass flag. > > The last time I added metaspace memory pools, it triggered some > unforeseen bugs: > - Two asserts in jmm_GetMemoryUsage that asserted that a memory pool was > either of heap type or had an undefined init/max size. > - The jdk/test/java/lang/management/MemoryMXBean/MemoryTest.java failed > - The service lock was taken out of order with the metaspace locks > > These bugs have all been fixed: > - The asserts have been removed since they are no longer true > - The test has been updated but is not part of this change since it is a > JDK change > - This change does not make use of functions requiring the metaspace > lock. I had to remove some verification code in free_chunks_total to > ensure this. > > Webrev: > http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/ > > Testing: > - One new jtreg test > - JPRT > - All the tests that failed in nighly testing last time now pass > > Thanks, > Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Tue Jun 4 19:00:17 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 04 Jun 2013 12:00:17 -0700 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <519FE787.70408@oracle.com> References: <519FE787.70408@oracle.com> Message-ID: <51AE3941.90609@oracle.com> Hi Everyone, Here's a new webrev for this change: http://cr.openjdk.java.net/~johnc/8015237/webrev.1 Changes from before: * Refactored the code that loops over the buckets into its own routine. * Removed the commented out instrumentation (oops). * Changed the types to int to be consistent with the rest of symbolTable and allow removal of the casts. * Increase the number of buckets per claim to 32 based upon a verbal comment from John Coomes. * Removed the additional worker ID parameter for the sake of peace and harmony. Testing: jprt. Thanks, JohnC On 5/24/2013 3:19 PM, John Cuthbertson wrote: > Hi Everyone, > > Can I have a couple of reviewers look over these changes - the webrev > is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ > > Summary: > On some workloads we are seeing that the scan of the intern string > table (among others) can sometimes take quite a while. This showed up > on some FMW workloads with G1 where the scan of the string table > dominated the pause time for some pauses. G1 was particularly affected > since it doesn't do class unloading (and hence pruning of the string > table) except at full GCs. The solution was to change the string table > from being considered a single root task and treat similarly to the > Java thread stacks: each GC worker claims a given number of buckets > and scans the entries in those buckets. > > Testing > Kitchensink; jprt; GC test suite. With all collectors. > > Overhead: > Not real performance numbers but I did some measurement of the > synchronization overhead of using 1 GC worker thread. They are > summarized here: > > > 0-threads (ms) > 1-thread-chunked (ms) > Min 0.200 > 0.300 > Max 6.900 > 8.800 > Avg 0.658 > 0.794 > > > These were from 1 hour long runs of Kitchensink with around ~2800 GCs > in each run. > > Thanks, > > JohnC -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.thalinger at oracle.com Tue Jun 4 20:08:58 2013 From: christian.thalinger at oracle.com (Christian Thalinger) Date: Tue, 4 Jun 2013 13:08:58 -0700 (PDT) Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AE3941.90609@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> Message-ID: Since I'm messing around in the same code right now I took a look. +volatile int StringTable::_par_claimed_idx = 0; Can't we just say parallel instead of par? + for (int i = start_idx; i < end_idx; i += 1) { The += 1 is perfectly sane but it looks odd :-) -- Chris On Jun 4, 2013, at 12:00 PM, John Cuthbertson wrote: > Hi Everyone, > > Here's a new webrev for this change: http://cr.openjdk.java.net/~johnc/8015237/webrev.1 > > Changes from before: > * Refactored the code that loops over the buckets into its own routine. > * Removed the commented out instrumentation (oops). > * Changed the types to int to be consistent with the rest of symbolTable and allow removal of the casts. > * Increase the number of buckets per claim to 32 based upon a verbal comment from John Coomes. > * Removed the additional worker ID parameter for the sake of peace and harmony. > > Testing: jprt. > > Thanks, > > JohnC > > On 5/24/2013 3:19 PM, John Cuthbertson wrote: >> Hi Everyone, >> >> Can I have a couple of reviewers look over these changes - the webrev is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >> >> Summary: >> On some workloads we are seeing that the scan of the intern string table (among others) can sometimes take quite a while. This showed up on some FMW workloads with G1 where the scan of the string table dominated the pause time for some pauses. G1 was particularly affected since it doesn't do class unloading (and hence pruning of the string table) except at full GCs. The solution was to change the string table from being considered a single root task and treat similarly to the Java thread stacks: each GC worker claims a given number of buckets and scans the entries in those buckets. >> >> Testing >> Kitchensink; jprt; GC test suite. With all collectors. >> >> Overhead: >> Not real performance numbers but I did some measurement of the synchronization overhead of using 1 GC worker thread. They are summarized here: >> >> >> 0-threads (ms) >> 1-thread-chunked (ms) >> Min 0.200 >> 0.300 >> Max 6.900 >> 8.800 >> Avg 0.658 >> 0.794 >> >> These were from 1 hour long runs of Kitchensink with around ~2800 GCs in each run. >> >> Thanks, >> >> JohnC > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Jun 4 20:24:15 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Jun 2013 22:24:15 +0200 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AE3941.90609@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> Message-ID: <1370377455.2688.31.camel@cirrus> Hi, On Tue, 2013-06-04 at 12:00 -0700, John Cuthbertson wrote: > Hi Everyone, > > Here's a new webrev for this change: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1 > > Changes from before: > * Refactored the code that loops over the buckets into its own > routine. > * Removed the commented out instrumentation (oops). > * Changed the types to int to be consistent with the rest of > symbolTable and allow removal of the casts. > * Increase the number of buckets per claim to 32 based upon a verbal > comment from John Coomes. > * Removed the additional worker ID parameter for the sake of peace and > harmony. Looks good, thanks a lot. Thomas From john.cuthbertson at oracle.com Tue Jun 4 20:50:14 2013 From: john.cuthbertson at oracle.com (john.cuthbertson at oracle.com) Date: Tue, 04 Jun 2013 20:50:14 +0000 Subject: hg: hsx/hotspot-gc/hotspot: 8015244: G1: Verification after a full GC is incorrectly placed. Message-ID: <20130604205016.73FDE48F4A@hg.openjdk.java.net> Changeset: 3a4805ad0005 Author: johnc Date: 2013-06-04 10:04 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/3a4805ad0005 8015244: G1: Verification after a full GC is incorrectly placed. Summary: In a full GC, move the verification after the GC to after RSet rebuilding. Verify RSet entries during a full GC under control of a flag. Reviewed-by: tschatzl, brutisso ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp ! src/share/vm/gc_implementation/g1/heapRegion.cpp From john.cuthbertson at oracle.com Tue Jun 4 20:58:05 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 04 Jun 2013 13:58:05 -0700 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> Message-ID: <51AE54DD.3030000@oracle.com> Hi Chris, Thanks for looking. Replies inline... On 6/4/2013 1:08 PM, Christian Thalinger wrote: > Since I'm messing around in the same code right now I took a look. > +volatile int StringTable::_par_claimed_idx = 0; > Can't we just say parallel instead of par? Sure. No problem. > + for (int i = start_idx; i < end_idx; i += 1) { > The += 1 is perfectly sane but it looks odd :-) I've been burned before with prefix and postfix increment operators (not in loop guards) so I tend to use the in-fix operator. JohnC -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Tue Jun 4 20:58:45 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 04 Jun 2013 13:58:45 -0700 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <1370377455.2688.31.camel@cirrus> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> <1370377455.2688.31.camel@cirrus> Message-ID: <51AE5505.7030800@oracle.com> Hi Thomas, That was fast. Thanks for looking. JohnC On 6/4/2013 1:24 PM, Thomas Schatzl wrote: > Hi, > > On Tue, 2013-06-04 at 12:00 -0700, John Cuthbertson wrote: >> Hi Everyone, >> >> Here's a new webrev for this change: >> http://cr.openjdk.java.net/~johnc/8015237/webrev.1 >> >> Changes from before: >> * Refactored the code that loops over the buckets into its own >> routine. >> * Removed the commented out instrumentation (oops). >> * Changed the types to int to be consistent with the rest of >> symbolTable and allow removal of the casts. >> * Increase the number of buckets per claim to 32 based upon a verbal >> comment from John Coomes. >> * Removed the additional worker ID parameter for the sake of peace and >> harmony. > Looks good, thanks a lot. > > Thomas > > From tao.mao at oracle.com Tue Jun 4 21:37:13 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 04 Jun 2013 14:37:13 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51A78AAD.8060102@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> Message-ID: <51AE5E09.9080808@oracle.com> Hi all, Need reviews to catch RDP2. The current webrev is a working solution to all platforms, Linux, Windows, and Solaris. http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ Thanks. Tao On 5/30/13 10:21 AM, Tao Mao wrote: > Included runtime dev to see whether they have some idea to handle the > compilation choices. > > For now, it's been verified that the fix is functionally sufficient. > > Thanks. > Tao > > On 5/29/13 5:27 PM, Tao Mao wrote: >> Thank you, Mikael. >> >> Please see inline. >> >> Reviewers, please review it based on the following new observation. >> >> Tao >> >> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>> Tao, >>> >>> On 2013-05-25 02:19, Tao Mao wrote: >>>> 7ux bug >>>> >>>> webrev: >>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>> >>>> changeset: >>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating ostream.o >>>> >>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>> applicable? >>>> >>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; however, >>>> there are at least five code conflicts if introducing the flag >>>> globally >>>> to Solaris. >>>> >>>> One was resolved as in os_solaris.inline.hpp, but the rest four files >>>> had conflicts deep in c library. Even if they are excluded from >>>> setting >>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>> >>>> (2) For now, no Windows solution. >>>> I haven't found any clean solution for solving this problem on >>>> Windows. >>> >>> This seems like an insufficient fix if you can't make it work on all >>> platforms. I tried building with "-D_LARGEFILE64_SOURCE >>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>> saying it wasn't supported so I understand your problem there. >> Yes, that's my grief :( you touched them, a bunch of them. That's why >> I chose to apply the flag only to the files (ostream.cpp and >> ostream.hpp) I want the effect. >>> >>> Instead I suggest that you use the compatibility API described in >>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>> >>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has the >>> added advantage of not changing any existing symbols and therefore >>> we can set the define for all files instead of just ostream >>> >>> This approach has the added advantage that it more closely resembles >>> the changes which will be needed for Windows anyway. Those changes >>> would consist of changing calls to ftell/fseek to 64-bit versions >>> and changing fopen to fopen64 on Solaris/Linux. >> Both ways have pros and cons. The current implementation excludes the >> usage of fopen64, providing portability (since there's no fopen64 for >> Windows). Meanwhile, I understand your suggestion provides other >> benefits. >> >> This Sun White Paper >> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >> summarizes the usage of the flags on solaris (Page 5-26). And, it >> should apply to Linux the same way as was agreed across platforms. >> >>> >>> Since there is no fopen64 on Windows it seems that the default fopen >>> already supports large files. >> I tested, and you are correct that the 32-bit VM on Windows can write >> beyond 2GB (and beyond 4GB). Thank you, it's solved "half of my >> problem" :) >>> >>> /Mikael >>> >>>> >>>> test: >>>> (1) Ability to write over 2g file for 32-bit builds were verified >>>> on the >>>> following configurations. >>>> >>>> Linux * i586 >>>> Solaris * i586 >>>> Solaris * sparc >>>> >>>> (2) Need a JPRT test for sanity check. >>> From john.cuthbertson at oracle.com Tue Jun 4 23:01:31 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 04 Jun 2013 16:01:31 -0700 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <1370357913.2650.42.camel@cirrus> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> <1370345967.2650.27.camel@cirrus> <51ADD470.6080707@oracle.com> <1370357913.2650.42.camel@cirrus> Message-ID: <51AE71CB.7010502@oracle.com> Hi Thomas, I think RegionTypeCounter should inherit from _ValueObj (by means of VALUE_OBJ_CLASS_SPEC). Also a couple of nitty questions (feel free to ignore): * Does the value parameter to RegionTypeCounter::percent_of need to be passed as a pointer? It looks like you pass the address of a field in the RegionTypeCounter instance and then dereference. * For continuous humongous would it make sense to skip them until we process the StartsHumongous region and sum up the memory size and the occupied value for the entire humongous region (HS and HC) before adding to the humongous region counter? Other than the above - it looks great! Regards, JohnC On 6/4/2013 7:58 AM, Thomas Schatzl wrote: > Hi all, > > On Tue, 2013-06-04 at 13:50 +0200, Bengt Rutisson wrote: >> Hi Thomas, >> >> Thanks for fixing these things. > A new webrev is up at > http://cr.openjdk.java.net/~tschatzl/8014078/webrev.1/ > > Changes: > - spacing suggested by Jesper > - removed comment in g1CollectedHeap::gc_prologue() indicating that the > following line subtracted one because we are at the end of collection; > both statements were just wrong. > - remove G1SummarizeRSetStatsTime and print summaries both at start and > end of collection > - renamed region_type_counter_t to RegionTypeCounter and moved it in the > top level scope > - removed superfluous getters > - renamed "Other" regions to "Old" regions in output > - added some line spacing in G1HRRSStatsIter::print_summary_on() > - refactoring of test cases: > - did _not_ use static imports because the java language does not > allow them for the default package > - removed the testing for the G1SummarizeRSetStatsTime flag > - renamed of *Statistics() to *RSetSummaries() > - TestSummarizeRSetStats2 renamed to TestSummarizeRSetStatsPerRegion > - moved TestSummarizeRSetStatsPerRegion.expectStatistics to > TestSummarizeRSetStatsTools.expectPerRegionRSetSummaries() > - adapted test for 8013895 to account for printing the summaries both > at start and end of gc > > > Testing: > jtreg passed for this cr and 8013895, jprt still running. > >> Some comments inline. >> >> On 6/4/13 1:39 PM, Thomas Schatzl wrote: >>> Hi, >>> >>> On Tue, 2013-06-04 at 08:10 +0200, Bengt Rutisson wrote: >>>> Hi Thomas, >>> It seems awkward to need both G1SummarizeRSetStats and >>> G1SummarizeRSetStatsBeforeGC/G1SummarizeRSetStatsAfterGC. What are the >>> rules on removing diagnostic flags? >> I think I vote for only having G1SummarizeRSetStats and let that print >> the information both before and after GC. That is backwards compatible >> in the sense that you get at least the information you got before with >> this flag. It is also similar to the PrintHeapAtGC flag. > I do not really mind. > >>> (Note that I think that G1SummarizeRSetStats has had a lot of use - >>> before cr 8013895 its output has been very buggy). >> I think you mean that it has _not_ had a lot of use, right? > Sure :) > >>>> I have not looked too closely at the tests but some comments: >>>> >>>> In TestSummarizeRSetStats.java you could minimize the diff and make the >>>> test more readable by using static imports for >>>> TestSummarizeRSetStatsTools.runTest() and >>>> TestSummarizeRSetStatsTools.expectStatistics() >>> I did not know that language feature... thanks! :) >>> >>>> I think the name TestSummarizeRSetStats2 is a bit awkward. How about >>>> TestSummarizeRSetStatsPerRegion? >>> Done. >>> >>>> The method TestSummarizeRSetStats2.expectStatistics() is a bit >>>> surprising since the other tests use >>>> TestSummarizeRSetStatsTools.expectStatistics(). Could we move >>>> TestSummarizeRSetStats2.expectStatistics() into >>>> TestSummarizeRSetStatsTools too? Either with a new name or we could >>> I will clean this up - but I would like to have your comment on the flag >>> issue first. Also, I will then send a new webrev. >> Sounds like a good plan. I hope what I wrote above is enough. > I hope too - if it isn't, just tell me. > >>>> parametrize the existing expectStatistics() method in there. > I did not do that and kept two methods, although giving them to more > distinct names. > > Thomas > > From tao.mao at oracle.com Tue Jun 4 23:03:50 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 4 Jun 2013 16:03:50 -0700 (PDT) Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE5E09.9080808@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> Message-ID: <51AE7256.1070609@oracle.com> Since the changeset touched makefiles, I've included build-dev at openjdk.java.net . I need to push the hsx24 bug asap. Please review it. Thanks. Tao On 6/4/13 2:37 PM, Tao Mao wrote: > Hi all, > > Need reviews to catch RDP2. > > The current webrev is a working solution to all platforms, Linux, > Windows, and Solaris. > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ > > Thanks. > Tao > > On 5/30/13 10:21 AM, Tao Mao wrote: >> Included runtime dev to see whether they have some idea to handle the >> compilation choices. >> >> For now, it's been verified that the fix is functionally sufficient. >> >> Thanks. >> Tao >> >> On 5/29/13 5:27 PM, Tao Mao wrote: >>> Thank you, Mikael. >>> >>> Please see inline. >>> >>> Reviewers, please review it based on the following new observation. >>> >>> Tao >>> >>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>> Tao, >>>> >>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>> 7ux bug >>>>> >>>>> webrev: >>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>> >>>>> changeset: >>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>> ostream.o >>>>> >>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>>> applicable? >>>>> >>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>> however, >>>>> there are at least five code conflicts if introducing the flag >>>>> globally >>>>> to Solaris. >>>>> >>>>> One was resolved as in os_solaris.inline.hpp, but the rest four files >>>>> had conflicts deep in c library. Even if they are excluded from >>>>> setting >>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>> >>>>> (2) For now, no Windows solution. >>>>> I haven't found any clean solution for solving this problem on >>>>> Windows. >>>> >>>> This seems like an insufficient fix if you can't make it work on >>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>>> saying it wasn't supported so I understand your problem there. >>> Yes, that's my grief :( you touched them, a bunch of them. That's >>> why I chose to apply the flag only to the files (ostream.cpp and >>> ostream.hpp) I want the effect. >>>> >>>> Instead I suggest that you use the compatibility API described in >>>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>> >>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has the >>>> added advantage of not changing any existing symbols and therefore >>>> we can set the define for all files instead of just ostream >>>> >>>> This approach has the added advantage that it more closely >>>> resembles the changes which will be needed for Windows anyway. >>>> Those changes would consist of changing calls to ftell/fseek to >>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>> Both ways have pros and cons. The current implementation excludes >>> the usage of fopen64, providing portability (since there's no >>> fopen64 for Windows). Meanwhile, I understand your suggestion >>> provides other benefits. >>> >>> This Sun White Paper >>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>> summarizes the usage of the flags on solaris (Page 5-26). And, it >>> should apply to Linux the same way as was agreed across platforms. >>> >>>> >>>> Since there is no fopen64 on Windows it seems that the default >>>> fopen already supports large files. >>> I tested, and you are correct that the 32-bit VM on Windows can >>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half of >>> my problem" :) >>>> >>>> /Mikael >>>> >>>>> >>>>> test: >>>>> (1) Ability to write over 2g file for 32-bit builds were verified >>>>> on the >>>>> following configurations. >>>>> >>>>> Linux * i586 >>>>> Solaris * i586 >>>>> Solaris * sparc >>>>> >>>>> (2) Need a JPRT test for sanity check. >>>> From thomas.schatzl at oracle.com Tue Jun 4 23:11:57 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 05 Jun 2013 01:11:57 +0200 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE7256.1070609@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> Message-ID: <1370387517.2688.53.camel@cirrus> On Tue, 2013-06-04 at 16:03 -0700, Tao Mao wrote: > Since the changeset touched makefiles, I've included > build-dev at openjdk.java.net . > > I need to push the hsx24 bug asap. Please review it. Looks okay to me. Thomas From yumin.qi at oracle.com Tue Jun 4 23:14:21 2013 From: yumin.qi at oracle.com (Yumin Qi) Date: Tue, 04 Jun 2013 16:14:21 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE7256.1070609@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> Message-ID: <51AE74CD.8000000@oracle.com> Hi, Tao The change works only when LP64 is not defined as "1", I would think you need to check if it is defined instead. (how about LP64=other?, source code only check if it is defined). I am not a reviewer. Thanks Yumin On 6/4/2013 4:03 PM, Tao Mao wrote: > Since the changeset touched makefiles, I've included > build-dev at openjdk.java.net . > > I need to push the hsx24 bug asap. Please review it. > > Thanks. > Tao > > On 6/4/13 2:37 PM, Tao Mao wrote: >> Hi all, >> >> Need reviews to catch RDP2. >> >> The current webrev is a working solution to all platforms, Linux, >> Windows, and Solaris. >> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >> >> Thanks. >> Tao >> >> On 5/30/13 10:21 AM, Tao Mao wrote: >>> Included runtime dev to see whether they have some idea to handle >>> the compilation choices. >>> >>> For now, it's been verified that the fix is functionally sufficient. >>> >>> Thanks. >>> Tao >>> >>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>> Thank you, Mikael. >>>> >>>> Please see inline. >>>> >>>> Reviewers, please review it based on the following new observation. >>>> >>>> Tao >>>> >>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>> Tao, >>>>> >>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>> 7ux bug >>>>>> >>>>>> webrev: >>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> changeset: >>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>> ostream.o >>>>>> >>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>>>> applicable? >>>>>> >>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>> however, >>>>>> there are at least five code conflicts if introducing the flag >>>>>> globally >>>>>> to Solaris. >>>>>> >>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>> files >>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>> setting >>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>> >>>>>> (2) For now, no Windows solution. >>>>>> I haven't found any clean solution for solving this problem on >>>>>> Windows. >>>>> >>>>> This seems like an insufficient fix if you can't make it work on >>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>>>> saying it wasn't supported so I understand your problem there. >>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>> why I chose to apply the flag only to the files (ostream.cpp and >>>> ostream.hpp) I want the effect. >>>>> >>>>> Instead I suggest that you use the compatibility API described in >>>>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>> >>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has the >>>>> added advantage of not changing any existing symbols and therefore >>>>> we can set the define for all files instead of just ostream >>>>> >>>>> This approach has the added advantage that it more closely >>>>> resembles the changes which will be needed for Windows anyway. >>>>> Those changes would consist of changing calls to ftell/fseek to >>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>> Both ways have pros and cons. The current implementation excludes >>>> the usage of fopen64, providing portability (since there's no >>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>> provides other benefits. >>>> >>>> This Sun White Paper >>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>> summarizes the usage of the flags on solaris (Page 5-26). And, it >>>> should apply to Linux the same way as was agreed across platforms. >>>> >>>>> >>>>> Since there is no fopen64 on Windows it seems that the default >>>>> fopen already supports large files. >>>> I tested, and you are correct that the 32-bit VM on Windows can >>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half of >>>> my problem" :) >>>>> >>>>> /Mikael >>>>> >>>>>> >>>>>> test: >>>>>> (1) Ability to write over 2g file for 32-bit builds were verified >>>>>> on the >>>>>> following configurations. >>>>>> >>>>>> Linux * i586 >>>>>> Solaris * i586 >>>>>> Solaris * sparc >>>>>> >>>>>> (2) Need a JPRT test for sanity check. >>>>> From tao.mao at oracle.com Tue Jun 4 23:21:32 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 04 Jun 2013 16:21:32 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <1370387517.2688.53.camel@cirrus> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <1370387517.2688.53.camel@cirrus> Message-ID: <51AE767C.7070606@oracle.com> Thank you, Thomas. Tao On 6/4/13 4:11 PM, Thomas Schatzl wrote: > On Tue, 2013-06-04 at 16:03 -0700, Tao Mao wrote: >> Since the changeset touched makefiles, I've included >> build-dev at openjdk.java.net . >> >> I need to push the hsx24 bug asap. Please review it. > Looks okay to me. > > Thomas > > From tao.mao at oracle.com Tue Jun 4 23:26:09 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 04 Jun 2013 16:26:09 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE74CD.8000000@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE74CD.8000000@oracle.com> Message-ID: <51AE7791.1000305@oracle.com> Thank you for looking at it, Yumin. Please see inline. Tao On 6/4/13 4:14 PM, Yumin Qi wrote: > Hi, Tao > > The change works only when LP64 is not defined as "1", I would think > you need to check if it is defined instead. (how about LP64=other?, > source code only check if it is defined). Is that true? I found the following exist in the current make files. make//bsd/makefiles/defs.make:34:ifeq ($(LP64), 1) make//linux/makefiles/defs.make:40:ifeq ($(LP64), 1) make//solaris/makefiles/defs.make:33:ifeq ($(LP64), 1) ifeq ($(LP64), 1) ARCH_DATA_MODEL ?= 64 else ARCH_DATA_MODEL ?= 32 endif It looks that it assumes users should set LP64=1 in order to get a correct build. Just presenting something I found. I'm not sure about this, either. The make files are a mystery to me. > I am not a reviewer. > > Thanks > Yumin > > On 6/4/2013 4:03 PM, Tao Mao wrote: >> Since the changeset touched makefiles, I've included >> build-dev at openjdk.java.net . >> >> I need to push the hsx24 bug asap. Please review it. >> >> Thanks. >> Tao >> >> On 6/4/13 2:37 PM, Tao Mao wrote: >>> Hi all, >>> >>> Need reviews to catch RDP2. >>> >>> The current webrev is a working solution to all platforms, Linux, >>> Windows, and Solaris. >>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>> >>> Thanks. >>> Tao >>> >>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>> Included runtime dev to see whether they have some idea to handle >>>> the compilation choices. >>>> >>>> For now, it's been verified that the fix is functionally sufficient. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>> Thank you, Mikael. >>>>> >>>>> Please see inline. >>>>> >>>>> Reviewers, please review it based on the following new observation. >>>>> >>>>> Tao >>>>> >>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>> Tao, >>>>>> >>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>> 7ux bug >>>>>>> >>>>>>> webrev: >>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> changeset: >>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>> ostream.o >>>>>>> >>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>>>>> applicable? >>>>>>> >>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>> however, >>>>>>> there are at least five code conflicts if introducing the flag >>>>>>> globally >>>>>>> to Solaris. >>>>>>> >>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>> files >>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>> setting >>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>> >>>>>>> (2) For now, no Windows solution. >>>>>>> I haven't found any clean solution for solving this problem on >>>>>>> Windows. >>>>>> >>>>>> This seems like an insufficient fix if you can't make it work on >>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>>>>> saying it wasn't supported so I understand your problem there. >>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>> ostream.hpp) I want the effect. >>>>>> >>>>>> Instead I suggest that you use the compatibility API described in >>>>>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>> >>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>> the added advantage of not changing any existing symbols and >>>>>> therefore we can set the define for all files instead of just >>>>>> ostream >>>>>> >>>>>> This approach has the added advantage that it more closely >>>>>> resembles the changes which will be needed for Windows anyway. >>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>> Both ways have pros and cons. The current implementation excludes >>>>> the usage of fopen64, providing portability (since there's no >>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>> provides other benefits. >>>>> >>>>> This Sun White Paper >>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>> summarizes the usage of the flags on solaris (Page 5-26). And, it >>>>> should apply to Linux the same way as was agreed across platforms. >>>>> >>>>>> >>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>> fopen already supports large files. >>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half of >>>>> my problem" :) >>>>>> >>>>>> /Mikael >>>>>> >>>>>>> >>>>>>> test: >>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>> verified on the >>>>>>> following configurations. >>>>>>> >>>>>>> Linux * i586 >>>>>>> Solaris * i586 >>>>>>> Solaris * sparc >>>>>>> >>>>>>> (2) Need a JPRT test for sanity check. >>>>>> > From daniel.daugherty at oracle.com Tue Jun 4 23:35:11 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 04 Jun 2013 17:35:11 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE7256.1070609@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> Message-ID: <51AE79AF.9070706@oracle.com> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ Tao, I think the lack of response to this review request is the absolutely strange nature of these changes. And I thought I put out some weird code reviews... :-) make/linux/makefiles/vm.make Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing obvious in this webrev about what this will mean so I took a look at src/share/vm/utilities/ostream.{c,h}pp and I see no use of _FILE_OFFSET_BITS in either of those source files. Must be in the source files somewhere, but I can't find any use of _FILE_OFFSET_BITS in the entire hotspot source base. make/solaris/makefiles/vm.make Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. OK, I looked for _FILE_OFFSET_BITS in /usr/include on my Solaris box. Lots of references, but nothing that helps me understand what you're doing here. src/os/solaris/vm/os_solaris.inline.hpp The addition of _FILE_OFFSET_BITS==64 means that the os::readdir() function will use the safer, multi-threaded version of readdir_r(). Seems fine to me. Here's what I need to know: - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? - if ostream.o has one idea about the value of _FILE_OFFSET_BITS what happens if another part of the VM has a different idea about the value of _FILE_OFFSET_BITS? I saw this in the post to the Runtime alias: > Included runtime dev to see whether they have some idea to handle > the compilation choices. And I still have no idea what you're asking here? What compilation choices? Are you asking about your Makefile changes? Are asking about defining _FILE_OFFSET_BITS for the entire build instead of just one object (ostream.o)? Are you worried that this VM is going to have mis-matched pieces and be unstable? So I reviewed it, but I definitely can't approve it without more info. I realize that you're up against the RDP2 limit, but this change has too many open questions (for now)... BTW, it is not at all clear whether Win32 will be able to write a 2GB+ GC log or not. The conversation below didn't help me at all. Dan On 6/4/13 5:03 PM, Tao Mao wrote: > Since the changeset touched makefiles, I've included > build-dev at openjdk.java.net . > > I need to push the hsx24 bug asap. Please review it. > > Thanks. > Tao > > On 6/4/13 2:37 PM, Tao Mao wrote: >> Hi all, >> >> Need reviews to catch RDP2. >> >> The current webrev is a working solution to all platforms, Linux, >> Windows, and Solaris. >> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >> >> Thanks. >> Tao >> >> On 5/30/13 10:21 AM, Tao Mao wrote: >>> Included runtime dev to see whether they have some idea to handle >>> the compilation choices. >>> >>> For now, it's been verified that the fix is functionally sufficient. >>> >>> Thanks. >>> Tao >>> >>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>> Thank you, Mikael. >>>> >>>> Please see inline. >>>> >>>> Reviewers, please review it based on the following new observation. >>>> >>>> Tao >>>> >>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>> Tao, >>>>> >>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>> 7ux bug >>>>>> >>>>>> webrev: >>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> changeset: >>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>> ostream.o >>>>>> >>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>>>> applicable? >>>>>> >>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>> however, >>>>>> there are at least five code conflicts if introducing the flag >>>>>> globally >>>>>> to Solaris. >>>>>> >>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>> files >>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>> setting >>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>> >>>>>> (2) For now, no Windows solution. >>>>>> I haven't found any clean solution for solving this problem on >>>>>> Windows. >>>>> >>>>> This seems like an insufficient fix if you can't make it work on >>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>>>> saying it wasn't supported so I understand your problem there. >>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>> why I chose to apply the flag only to the files (ostream.cpp and >>>> ostream.hpp) I want the effect. >>>>> >>>>> Instead I suggest that you use the compatibility API described in >>>>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>> >>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has the >>>>> added advantage of not changing any existing symbols and therefore >>>>> we can set the define for all files instead of just ostream >>>>> >>>>> This approach has the added advantage that it more closely >>>>> resembles the changes which will be needed for Windows anyway. >>>>> Those changes would consist of changing calls to ftell/fseek to >>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>> Both ways have pros and cons. The current implementation excludes >>>> the usage of fopen64, providing portability (since there's no >>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>> provides other benefits. >>>> >>>> This Sun White Paper >>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>> summarizes the usage of the flags on solaris (Page 5-26). And, it >>>> should apply to Linux the same way as was agreed across platforms. >>>> >>>>> >>>>> Since there is no fopen64 on Windows it seems that the default >>>>> fopen already supports large files. >>>> I tested, and you are correct that the 32-bit VM on Windows can >>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half of >>>> my problem" :) >>>>> >>>>> /Mikael >>>>> >>>>>> >>>>>> test: >>>>>> (1) Ability to write over 2g file for 32-bit builds were verified >>>>>> on the >>>>>> following configurations. >>>>>> >>>>>> Linux * i586 >>>>>> Solaris * i586 >>>>>> Solaris * sparc >>>>>> >>>>>> (2) Need a JPRT test for sanity check. >>>>> From thomas.schatzl at oracle.com Tue Jun 4 23:38:44 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 05 Jun 2013 01:38:44 +0200 Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <51AE71CB.7010502@oracle.com> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> <1370345967.2650.27.camel@cirrus> <51ADD470.6080707@oracle.com> <1370357913.2650.42.camel@cirrus> <51AE71CB.7010502@oracle.com> Message-ID: <1370389124.2688.66.camel@cirrus> Hi, On Tue, 2013-06-04 at 16:01 -0700, John Cuthbertson wrote: > Hi Thomas, > > I think RegionTypeCounter should inherit from _ValueObj (by means of > VALUE_OBJ_CLASS_SPEC). > > Also a couple of nitty questions (feel free to ignore): > > * Does the value parameter to RegionTypeCounter::percent_of need to be > passed as a pointer? It looks like you pass the address of a field in > the RegionTypeCounter instance and then dereference. Fixed both things. I am not sure why the code passes the address of the field, but this is probably a leftover of some earlier version. I also combined the calc_percentage() function with the RegionTypeCounter::percent_of() method as they are the same. > * For continuous humongous would it make sense to skip them until we > process the StartsHumongous region and sum up the memory size and the > occupied value for the entire humongous region (HS and HC) before adding > to the humongous region counter? Imho there is no need to do that - the memory sizes and occupancies of HS and HC regions are completely independent. I.e. the order of evaluation does not matter here. Doing so seems to only complicate the code. > Other than the above - it looks great! Thank you. I will provide a new webrev with these changes. Thomas From tao.mao at oracle.com Wed Jun 5 00:06:44 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 04 Jun 2013 17:06:44 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE79AF.9070706@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> Message-ID: <51AE8114.80600@oracle.com> Thank you for review, Dan. I'll try to answer as much as I can. Please see inline. Thanks. Tao On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: > > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ > > Tao, > > I think the lack of response to this review request is the absolutely > strange nature of these changes. And I thought I put out some weird > code reviews... :-) > > make/linux/makefiles/vm.make > Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing > obvious in this webrev about what this will mean so I took a > look at src/share/vm/utilities/ostream.{c,h}pp and I see no > use of _FILE_OFFSET_BITS in either of those source files. > > Must be in the source files somewhere, but I can't find any > use of _FILE_OFFSET_BITS in the entire hotspot source base. > > make/solaris/makefiles/vm.make > Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. > > OK, I looked for _FILE_OFFSET_BITS in /usr/include on my > Solaris box. Lots of references, but nothing that helps me > understand what you're doing here. > src/os/solaris/vm/os_solaris.inline.hpp > The addition of _FILE_OFFSET_BITS==64 means that the > os::readdir() function will use the safer, multi-threaded > version of readdir_r(). Seems fine to me. > > Here's what I need to know: > > - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? _FILE_OFFSET_BITS is set to be picked by c++ compiler. For why we need to set _FILE_OFFSET_BITS==64 in this case, please refer to the following document This Sun White Paper (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) summarizes the usage of the flags on solaris (page "5-26"). And, it should apply to Linux the same way as was agreed across platforms (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). > - if ostream.o has one idea about the value of _FILE_OFFSET_BITS > what happens if another part of the VM has a different idea about > the value of _FILE_OFFSET_BITS? _FILE_OFFSET_BITS is not set for other particular effects, but for extending the ability to deal with large files in ostream.{c,h}pp. So, if other files have a different idea about _FILE_OFFSET_BITS, they can't deal with large files. No more no less. > > I saw this in the post to the Runtime alias: > > > Included runtime dev to see whether they have some idea to handle > > the compilation choices. > > And I still have no idea what you're asking here? What compilation > choices? Are you asking about your Makefile changes? Are asking > about defining _FILE_OFFSET_BITS for the entire build instead of > just one object (ostream.o)? Are you worried that this VM is going > to have mis-matched pieces and be unstable? "Are asking about defining _FILE_OFFSET_BITS for the entire build instead of just one object (ostream.o)?" is my main question I originally tried to ask. > > So I reviewed it, but I definitely can't approve it without more > info. I realize that you're up against the RDP2 limit, but this > change has too many open questions (for now)... > > BTW, it is not at all clear whether Win32 will be able to write a 2GB+ > GC log or not. The conversation below didn't help me at all. I used a jdk7 (just any) to successfully generate a log file larger than 4GB. So, it shouldn't be a problem for Win32. > > Dan > > > > On 6/4/13 5:03 PM, Tao Mao wrote: >> Since the changeset touched makefiles, I've included >> build-dev at openjdk.java.net . >> >> I need to push the hsx24 bug asap. Please review it. >> >> Thanks. >> Tao >> >> On 6/4/13 2:37 PM, Tao Mao wrote: >>> Hi all, >>> >>> Need reviews to catch RDP2. >>> >>> The current webrev is a working solution to all platforms, Linux, >>> Windows, and Solaris. >>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>> >>> Thanks. >>> Tao >>> >>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>> Included runtime dev to see whether they have some idea to handle >>>> the compilation choices. >>>> >>>> For now, it's been verified that the fix is functionally sufficient. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>> Thank you, Mikael. >>>>> >>>>> Please see inline. >>>>> >>>>> Reviewers, please review it based on the following new observation. >>>>> >>>>> Tao >>>>> >>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>> Tao, >>>>>> >>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>> 7ux bug >>>>>>> >>>>>>> webrev: >>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> changeset: >>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>> ostream.o >>>>>>> >>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 globally >>>>>>> applicable? >>>>>>> >>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>> however, >>>>>>> there are at least five code conflicts if introducing the flag >>>>>>> globally >>>>>>> to Solaris. >>>>>>> >>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>> files >>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>> setting >>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>> >>>>>>> (2) For now, no Windows solution. >>>>>>> I haven't found any clean solution for solving this problem on >>>>>>> Windows. >>>>>> >>>>>> This seems like an insufficient fix if you can't make it work on >>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in libelf.h >>>>>> saying it wasn't supported so I understand your problem there. >>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>> ostream.hpp) I want the effect. >>>>>> >>>>>> Instead I suggest that you use the compatibility API described in >>>>>> lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>> >>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>> the added advantage of not changing any existing symbols and >>>>>> therefore we can set the define for all files instead of just >>>>>> ostream >>>>>> >>>>>> This approach has the added advantage that it more closely >>>>>> resembles the changes which will be needed for Windows anyway. >>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>> Both ways have pros and cons. The current implementation excludes >>>>> the usage of fopen64, providing portability (since there's no >>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>> provides other benefits. >>>>> >>>>> This Sun White Paper >>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>> summarizes the usage of the flags on solaris (Page 5-26). And, it >>>>> should apply to Linux the same way as was agreed across platforms. >>>>> >>>>>> >>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>> fopen already supports large files. >>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half of >>>>> my problem" :) >>>>>> >>>>>> /Mikael >>>>>> >>>>>>> >>>>>>> test: >>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>> verified on the >>>>>>> following configurations. >>>>>>> >>>>>>> Linux * i586 >>>>>>> Solaris * i586 >>>>>>> Solaris * sparc >>>>>>> >>>>>>> (2) Need a JPRT test for sanity check. >>>>>> > From john.cuthbertson at oracle.com Wed Jun 5 00:16:24 2013 From: john.cuthbertson at oracle.com (john.cuthbertson at oracle.com) Date: Wed, 05 Jun 2013 00:16:24 +0000 Subject: hg: hsx/hotspot-gc/hotspot: 2 new changesets Message-ID: <20130605001631.4805648F58@hg.openjdk.java.net> Changeset: 87c64c0438fb Author: tamao Date: 2013-06-03 14:37 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/87c64c0438fb 6976350: G1: deal with fragmentation while copying objects during GC Summary: Create G1ParGCAllocBufferContainer to contain two buffers instead of previously using one buffer, in order to hold the first priority buffer longer. Thus, when some large objects hits the value of free space left in the first priority buffer it has an alternative to fit in the second priority buffer while the first priority buffer is given more chances to try allocating smaller objects. Overall, it will improve heap space efficiency. Reviewed-by: johnc, jmasa, brutisso Contributed-by: tamao ! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp ! src/share/vm/gc_implementation/shared/parGCAllocBuffer.hpp Changeset: 2f7a31318b84 Author: johnc Date: 2013-06-04 14:00 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/2f7a31318b84 Merge From daniel.daugherty at oracle.com Wed Jun 5 00:21:28 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 04 Jun 2013 18:21:28 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE8114.80600@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> Message-ID: <51AE8488.5050101@oracle.com> OK, based on the largefiles.pdf write-up, your use of _FILE_OFFSET_BITS=64 is going to cause ostream.o to bind to various 64-bit versions of some functions. Based on my very hazy memory of my days in file system code, this binding is going to affect ostream.o only unless ostream.o is writing to some file with the foo64() function and some other code is also writing to the same file at the same time with foo(). However, if we have two different functions writing to the same open file at the same time, then we have bigger issues. :-) I'm good with these changes now. I agree that solving the problem of setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to solved right now. Dan On 6/4/13 6:06 PM, Tao Mao wrote: > Thank you for review, Dan. > > I'll try to answer as much as I can. Please see inline. > > Thanks. > Tao > > On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >> >> Tao, >> >> I think the lack of response to this review request is the absolutely >> strange nature of these changes. And I thought I put out some weird >> code reviews... :-) >> >> make/linux/makefiles/vm.make >> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >> obvious in this webrev about what this will mean so I took a >> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >> use of _FILE_OFFSET_BITS in either of those source files. >> >> Must be in the source files somewhere, but I can't find any >> use of _FILE_OFFSET_BITS in the entire hotspot source base. >> >> make/solaris/makefiles/vm.make >> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >> >> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >> Solaris box. Lots of references, but nothing that helps me >> understand what you're doing here. >> src/os/solaris/vm/os_solaris.inline.hpp >> The addition of _FILE_OFFSET_BITS==64 means that the >> os::readdir() function will use the safer, multi-threaded >> version of readdir_r(). Seems fine to me. >> >> Here's what I need to know: >> >> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? > _FILE_OFFSET_BITS is set to be picked by c++ compiler. > > For why we need to set _FILE_OFFSET_BITS==64 in this case, please > refer to the following document > > This Sun White Paper > (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) > summarizes the usage of the flags on solaris (page "5-26"). And, it > should apply to Linux the same way as was agreed across platforms > (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). > >> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >> what happens if another part of the VM has a different idea about >> the value of _FILE_OFFSET_BITS? > _FILE_OFFSET_BITS is not set for other particular effects, but for > extending the ability to deal with large files in ostream.{c,h}pp. So, > if other files have a different idea about _FILE_OFFSET_BITS, they > can't deal with large files. No more no less. > >> >> I saw this in the post to the Runtime alias: >> >> > Included runtime dev to see whether they have some idea to handle >> > the compilation choices. >> >> And I still have no idea what you're asking here? What compilation >> choices? Are you asking about your Makefile changes? Are asking >> about defining _FILE_OFFSET_BITS for the entire build instead of >> just one object (ostream.o)? Are you worried that this VM is going >> to have mis-matched pieces and be unstable? > "Are asking about defining _FILE_OFFSET_BITS for the entire build > instead of just one object (ostream.o)?" is my main question I > originally tried to ask. > >> >> So I reviewed it, but I definitely can't approve it without more >> info. I realize that you're up against the RDP2 limit, but this >> change has too many open questions (for now)... >> >> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >> GC log or not. The conversation below didn't help me at all. > I used a jdk7 (just any) to successfully generate a log file larger > than 4GB. So, it shouldn't be a problem for Win32. >> >> Dan >> >> >> >> On 6/4/13 5:03 PM, Tao Mao wrote: >>> Since the changeset touched makefiles, I've included >>> build-dev at openjdk.java.net . >>> >>> I need to push the hsx24 bug asap. Please review it. >>> >>> Thanks. >>> Tao >>> >>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>> Hi all, >>>> >>>> Need reviews to catch RDP2. >>>> >>>> The current webrev is a working solution to all platforms, Linux, >>>> Windows, and Solaris. >>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>> Included runtime dev to see whether they have some idea to handle >>>>> the compilation choices. >>>>> >>>>> For now, it's been verified that the fix is functionally sufficient. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>> Thank you, Mikael. >>>>>> >>>>>> Please see inline. >>>>>> >>>>>> Reviewers, please review it based on the following new observation. >>>>>> >>>>>> Tao >>>>>> >>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>> Tao, >>>>>>> >>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>> 7ux bug >>>>>>>> >>>>>>>> webrev: >>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> changeset: >>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>> ostream.o >>>>>>>> >>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>> globally >>>>>>>> applicable? >>>>>>>> >>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>> however, >>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>> globally >>>>>>>> to Solaris. >>>>>>>> >>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>>> files >>>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>>> setting >>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>> >>>>>>>> (2) For now, no Windows solution. >>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>> Windows. >>>>>>> >>>>>>> This seems like an insufficient fix if you can't make it work on >>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>> libelf.h saying it wasn't supported so I understand your problem >>>>>>> there. >>>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>> ostream.hpp) I want the effect. >>>>>>> >>>>>>> Instead I suggest that you use the compatibility API described >>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>> >>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>> the added advantage of not changing any existing symbols and >>>>>>> therefore we can set the define for all files instead of just >>>>>>> ostream >>>>>>> >>>>>>> This approach has the added advantage that it more closely >>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>> Both ways have pros and cons. The current implementation excludes >>>>>> the usage of fopen64, providing portability (since there's no >>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>> provides other benefits. >>>>>> >>>>>> This Sun White Paper >>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) summarizes >>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>> apply to Linux the same way as was agreed across platforms. >>>>>> >>>>>>> >>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>> fopen already supports large files. >>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>> of my problem" :) >>>>>>> >>>>>>> /Mikael >>>>>>> >>>>>>>> >>>>>>>> test: >>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>> verified on the >>>>>>>> following configurations. >>>>>>>> >>>>>>>> Linux * i586 >>>>>>>> Solaris * i586 >>>>>>>> Solaris * sparc >>>>>>>> >>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>> >> From tao.mao at oracle.com Wed Jun 5 00:25:30 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 04 Jun 2013 17:25:30 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE8488.5050101@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> Message-ID: <51AE857A.8090400@oracle.com> Thank you for your explanation and a "OK", Dan. Tao On 6/4/13 5:21 PM, Daniel D. Daugherty wrote: > OK, based on the largefiles.pdf write-up, your use of > _FILE_OFFSET_BITS=64 > is going to cause ostream.o to bind to various 64-bit versions of some > functions. Based on my very hazy memory of my days in file system code, > this binding is going to affect ostream.o only unless ostream.o is > writing to some file with the foo64() function and some other code is > also writing to the same file at the same time with foo(). However, if we > have two different functions writing to the same open file at the same > time, then we have bigger issues. :-) > > I'm good with these changes now. I agree that solving the problem of > setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to > solved right now. > > Dan > > > On 6/4/13 6:06 PM, Tao Mao wrote: >> Thank you for review, Dan. >> >> I'll try to answer as much as I can. Please see inline. >> >> Thanks. >> Tao >> >> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>> >>> Tao, >>> >>> I think the lack of response to this review request is the absolutely >>> strange nature of these changes. And I thought I put out some weird >>> code reviews... :-) >>> >>> make/linux/makefiles/vm.make >>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>> obvious in this webrev about what this will mean so I took a >>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>> use of _FILE_OFFSET_BITS in either of those source files. >>> >>> Must be in the source files somewhere, but I can't find any >>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>> >>> make/solaris/makefiles/vm.make >>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>> >>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>> Solaris box. Lots of references, but nothing that helps me >>> understand what you're doing here. >>> src/os/solaris/vm/os_solaris.inline.hpp >>> The addition of _FILE_OFFSET_BITS==64 means that the >>> os::readdir() function will use the safer, multi-threaded >>> version of readdir_r(). Seems fine to me. >>> >>> Here's what I need to know: >>> >>> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? >> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >> >> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >> refer to the following document >> >> This Sun White Paper >> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >> summarizes the usage of the flags on solaris (page "5-26"). And, it >> should apply to Linux the same way as was agreed across platforms >> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >> >>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>> what happens if another part of the VM has a different idea about >>> the value of _FILE_OFFSET_BITS? >> _FILE_OFFSET_BITS is not set for other particular effects, but for >> extending the ability to deal with large files in ostream.{c,h}pp. >> So, if other files have a different idea about _FILE_OFFSET_BITS, >> they can't deal with large files. No more no less. >> >>> >>> I saw this in the post to the Runtime alias: >>> >>> > Included runtime dev to see whether they have some idea to handle >>> > the compilation choices. >>> >>> And I still have no idea what you're asking here? What compilation >>> choices? Are you asking about your Makefile changes? Are asking >>> about defining _FILE_OFFSET_BITS for the entire build instead of >>> just one object (ostream.o)? Are you worried that this VM is going >>> to have mis-matched pieces and be unstable? >> "Are asking about defining _FILE_OFFSET_BITS for the entire build >> instead of just one object (ostream.o)?" is my main question I >> originally tried to ask. >> >>> >>> So I reviewed it, but I definitely can't approve it without more >>> info. I realize that you're up against the RDP2 limit, but this >>> change has too many open questions (for now)... >>> >>> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >>> GC log or not. The conversation below didn't help me at all. >> I used a jdk7 (just any) to successfully generate a log file larger >> than 4GB. So, it shouldn't be a problem for Win32. >>> >>> Dan >>> >>> >>> >>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>> Since the changeset touched makefiles, I've included >>>> build-dev at openjdk.java.net . >>>> >>>> I need to push the hsx24 bug asap. Please review it. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>> Hi all, >>>>> >>>>> Need reviews to catch RDP2. >>>>> >>>>> The current webrev is a working solution to all platforms, Linux, >>>>> Windows, and Solaris. >>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>> Included runtime dev to see whether they have some idea to handle >>>>>> the compilation choices. >>>>>> >>>>>> For now, it's been verified that the fix is functionally sufficient. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>> Thank you, Mikael. >>>>>>> >>>>>>> Please see inline. >>>>>>> >>>>>>> Reviewers, please review it based on the following new observation. >>>>>>> >>>>>>> Tao >>>>>>> >>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>> Tao, >>>>>>>> >>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>> 7ux bug >>>>>>>>> >>>>>>>>> webrev: >>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>> >>>>>>>>> changeset: >>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>> ostream.o >>>>>>>>> >>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>> globally >>>>>>>>> applicable? >>>>>>>>> >>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>> however, >>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>> globally >>>>>>>>> to Solaris. >>>>>>>>> >>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>> four files >>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>> from setting >>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>> >>>>>>>>> (2) For now, no Windows solution. >>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>> Windows. >>>>>>>> >>>>>>>> This seems like an insufficient fix if you can't make it work >>>>>>>> on all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>> problem there. >>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>> That's why I chose to apply the flag only to the files >>>>>>> (ostream.cpp and ostream.hpp) I want the effect. >>>>>>>> >>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>> and friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>> >>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>> therefore we can set the define for all files instead of just >>>>>>>> ostream >>>>>>>> >>>>>>>> This approach has the added advantage that it more closely >>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>> Both ways have pros and cons. The current implementation >>>>>>> excludes the usage of fopen64, providing portability (since >>>>>>> there's no fopen64 for Windows). Meanwhile, I understand your >>>>>>> suggestion provides other benefits. >>>>>>> >>>>>>> This Sun White Paper >>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>> summarizes the usage of the flags on solaris (Page 5-26). And, >>>>>>> it should apply to Linux the same way as was agreed across >>>>>>> platforms. >>>>>>> >>>>>>>> >>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>> fopen already supports large files. >>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>> of my problem" :) >>>>>>>> >>>>>>>> /Mikael >>>>>>>> >>>>>>>>> >>>>>>>>> test: >>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>> verified on the >>>>>>>>> following configurations. >>>>>>>>> >>>>>>>>> Linux * i586 >>>>>>>>> Solaris * i586 >>>>>>>>> Solaris * sparc >>>>>>>>> >>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>> >>> > From daniel.daugherty at oracle.com Wed Jun 5 00:29:44 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 04 Jun 2013 18:29:44 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE857A.8090400@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AE857A.8090400@oracle.com> Message-ID: <51AE8678.1010406@oracle.com> No problem. Sorry for the delay in someone from Runtime getting to this review. I _think_ the build-dev guys still trust my ability to review Makefile changes, but after imposing FDS on the world, they may want to see my Makefile changes rot in a dungeon somewhere... :-) Or at least somewhere where diskspace is available... :-( Dan On 6/4/13 6:25 PM, Tao Mao wrote: > Thank you for your explanation and a "OK", Dan. > > Tao > > On 6/4/13 5:21 PM, Daniel D. Daugherty wrote: >> OK, based on the largefiles.pdf write-up, your use of >> _FILE_OFFSET_BITS=64 >> is going to cause ostream.o to bind to various 64-bit versions of some >> functions. Based on my very hazy memory of my days in file system code, >> this binding is going to affect ostream.o only unless ostream.o is >> writing to some file with the foo64() function and some other code is >> also writing to the same file at the same time with foo(). However, >> if we >> have two different functions writing to the same open file at the same >> time, then we have bigger issues. :-) >> >> I'm good with these changes now. I agree that solving the problem of >> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >> solved right now. >> >> Dan >> >> >> On 6/4/13 6:06 PM, Tao Mao wrote: >>> Thank you for review, Dan. >>> >>> I'll try to answer as much as I can. Please see inline. >>> >>> Thanks. >>> Tao >>> >>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>> >>>> Tao, >>>> >>>> I think the lack of response to this review request is the absolutely >>>> strange nature of these changes. And I thought I put out some weird >>>> code reviews... :-) >>>> >>>> make/linux/makefiles/vm.make >>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>> obvious in this webrev about what this will mean so I took a >>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>> use of _FILE_OFFSET_BITS in either of those source files. >>>> >>>> Must be in the source files somewhere, but I can't find any >>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>> >>>> make/solaris/makefiles/vm.make >>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>> >>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>> Solaris box. Lots of references, but nothing that helps me >>>> understand what you're doing here. >>>> src/os/solaris/vm/os_solaris.inline.hpp >>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>> os::readdir() function will use the safer, multi-threaded >>>> version of readdir_r(). Seems fine to me. >>>> >>>> Here's what I need to know: >>>> >>>> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? >>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>> >>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>> refer to the following document >>> >>> This Sun White Paper >>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>> should apply to Linux the same way as was agreed across platforms >>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>> >>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>> what happens if another part of the VM has a different idea about >>>> the value of _FILE_OFFSET_BITS? >>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>> extending the ability to deal with large files in ostream.{c,h}pp. >>> So, if other files have a different idea about _FILE_OFFSET_BITS, >>> they can't deal with large files. No more no less. >>> >>>> >>>> I saw this in the post to the Runtime alias: >>>> >>>> > Included runtime dev to see whether they have some idea to handle >>>> > the compilation choices. >>>> >>>> And I still have no idea what you're asking here? What compilation >>>> choices? Are you asking about your Makefile changes? Are asking >>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>> just one object (ostream.o)? Are you worried that this VM is going >>>> to have mis-matched pieces and be unstable? >>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>> instead of just one object (ostream.o)?" is my main question I >>> originally tried to ask. >>> >>>> >>>> So I reviewed it, but I definitely can't approve it without more >>>> info. I realize that you're up against the RDP2 limit, but this >>>> change has too many open questions (for now)... >>>> >>>> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >>>> GC log or not. The conversation below didn't help me at all. >>> I used a jdk7 (just any) to successfully generate a log file larger >>> than 4GB. So, it shouldn't be a problem for Win32. >>>> >>>> Dan >>>> >>>> >>>> >>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>> Since the changeset touched makefiles, I've included >>>>> build-dev at openjdk.java.net . >>>>> >>>>> I need to push the hsx24 bug asap. Please review it. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>> Hi all, >>>>>> >>>>>> Need reviews to catch RDP2. >>>>>> >>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>> Windows, and Solaris. >>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>> Included runtime dev to see whether they have some idea to >>>>>>> handle the compilation choices. >>>>>>> >>>>>>> For now, it's been verified that the fix is functionally >>>>>>> sufficient. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>> Thank you, Mikael. >>>>>>>> >>>>>>>> Please see inline. >>>>>>>> >>>>>>>> Reviewers, please review it based on the following new >>>>>>>> observation. >>>>>>>> >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>> Tao, >>>>>>>>> >>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>> 7ux bug >>>>>>>>>> >>>>>>>>>> webrev: >>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>> >>>>>>>>>> changeset: >>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>> ostream.o >>>>>>>>>> >>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>> globally >>>>>>>>>> applicable? >>>>>>>>>> >>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>> however, >>>>>>>>>> there are at least five code conflicts if introducing the >>>>>>>>>> flag globally >>>>>>>>>> to Solaris. >>>>>>>>>> >>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>> four files >>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>> from setting >>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>> >>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>> I haven't found any clean solution for solving this problem >>>>>>>>>> on Windows. >>>>>>>>> >>>>>>>>> This seems like an insufficient fix if you can't make it work >>>>>>>>> on all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>> problem there. >>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>> That's why I chose to apply the flag only to the files >>>>>>>> (ostream.cpp and ostream.hpp) I want the effect. >>>>>>>>> >>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>> and friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>> >>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>> ostream >>>>>>>>> >>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>> Those changes would consist of changing calls to ftell/fseek >>>>>>>>> to 64-bit versions and changing fopen to fopen64 on >>>>>>>>> Solaris/Linux. >>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>> excludes the usage of fopen64, providing portability (since >>>>>>>> there's no fopen64 for Windows). Meanwhile, I understand your >>>>>>>> suggestion provides other benefits. >>>>>>>> >>>>>>>> This Sun White Paper >>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>> summarizes the usage of the flags on solaris (Page 5-26). And, >>>>>>>> it should apply to Linux the same way as was agreed across >>>>>>>> platforms. >>>>>>>> >>>>>>>>> >>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>> fopen already supports large files. >>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>> of my problem" :) >>>>>>>>> >>>>>>>>> /Mikael >>>>>>>>> >>>>>>>>>> >>>>>>>>>> test: >>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>> verified on the >>>>>>>>>> following configurations. >>>>>>>>>> >>>>>>>>>> Linux * i586 >>>>>>>>>> Solaris * i586 >>>>>>>>>> Solaris * sparc >>>>>>>>>> >>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>> >>>> >> > From tao.mao at oracle.com Wed Jun 5 00:34:44 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 4 Jun 2013 17:34:44 -0700 (PDT) Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE86BC.2010805@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AE857A.8090400@oracle.com> <51AE86BC.2010805@oracle.com> Message-ID: <51AE87A4.7090002@oracle.com> Thank you for reviewing, Tim. Tao On 6/4/13 5:30 PM, Tim Bell wrote: > I am OK with the Makefile changes. > > Dan - Thanks for looking after the deeper questions. > > Tim > > On 06/ 4/13 05:25 PM, Tao Mao wrote: >> Thank you for your explanation and a "OK", Dan. >> >> Tao >> >> On 6/4/13 5:21 PM, Daniel D. Daugherty wrote: >>> OK, based on the largefiles.pdf write-up, your use of >>> _FILE_OFFSET_BITS=64 >>> is going to cause ostream.o to bind to various 64-bit versions of some >>> functions. Based on my very hazy memory of my days in file system code, >>> this binding is going to affect ostream.o only unless ostream.o is >>> writing to some file with the foo64() function and some other code is >>> also writing to the same file at the same time with foo(). However, >>> if we >>> have two different functions writing to the same open file at the same >>> time, then we have bigger issues. :-) >>> >>> I'm good with these changes now. I agree that solving the problem of >>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>> solved right now. >>> >>> Dan >>> >>> >>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>> Thank you for review, Dan. >>>> >>>> I'll try to answer as much as I can. Please see inline. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>> >>>>> Tao, >>>>> >>>>> I think the lack of response to this review request is the absolutely >>>>> strange nature of these changes. And I thought I put out some weird >>>>> code reviews... :-) >>>>> >>>>> make/linux/makefiles/vm.make >>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>> obvious in this webrev about what this will mean so I took a >>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>> >>>>> Must be in the source files somewhere, but I can't find any >>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>> >>>>> make/solaris/makefiles/vm.make >>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>> >>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>> Solaris box. Lots of references, but nothing that helps me >>>>> understand what you're doing here. >>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>> os::readdir() function will use the safer, multi-threaded >>>>> version of readdir_r(). Seems fine to me. >>>>> >>>>> Here's what I need to know: >>>>> >>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>> ostream.{c,h}pp? >>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>> >>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>> refer to the following document >>>> >>>> This Sun White Paper >>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>> should apply to Linux the same way as was agreed across platforms >>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>> >>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>> what happens if another part of the VM has a different idea about >>>>> the value of _FILE_OFFSET_BITS? >>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>> So, if other files have a different idea about _FILE_OFFSET_BITS, >>>> they can't deal with large files. No more no less. >>>> >>>>> >>>>> I saw this in the post to the Runtime alias: >>>>> >>>>> > Included runtime dev to see whether they have some idea to handle >>>>> > the compilation choices. >>>>> >>>>> And I still have no idea what you're asking here? What compilation >>>>> choices? Are you asking about your Makefile changes? Are asking >>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>> to have mis-matched pieces and be unstable? >>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>> instead of just one object (ostream.o)?" is my main question I >>>> originally tried to ask. >>>> >>>>> >>>>> So I reviewed it, but I definitely can't approve it without more >>>>> info. I realize that you're up against the RDP2 limit, but this >>>>> change has too many open questions (for now)... >>>>> >>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>> 2GB+ >>>>> GC log or not. The conversation below didn't help me at all. >>>> I used a jdk7 (just any) to successfully generate a log file larger >>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>> >>>>> Dan >>>>> >>>>> >>>>> >>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>> Since the changeset touched makefiles, I've included >>>>>> build-dev at openjdk.java.net . >>>>>> >>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> Need reviews to catch RDP2. >>>>>>> >>>>>>> The current webrev is a working solution to all platforms, >>>>>>> Linux, Windows, and Solaris. >>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>> Included runtime dev to see whether they have some idea to >>>>>>>> handle the compilation choices. >>>>>>>> >>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>> sufficient. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>> Thank you, Mikael. >>>>>>>>> >>>>>>>>> Please see inline. >>>>>>>>> >>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>> observation. >>>>>>>>> >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>> Tao, >>>>>>>>>> >>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>> 7ux bug >>>>>>>>>>> >>>>>>>>>>> webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> changeset: >>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>> ostream.o >>>>>>>>>>> >>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>> globally >>>>>>>>>>> applicable? >>>>>>>>>>> >>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works >>>>>>>>>>> fine; however, >>>>>>>>>>> there are at least five code conflicts if introducing the >>>>>>>>>>> flag globally >>>>>>>>>>> to Solaris. >>>>>>>>>>> >>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>> four files >>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>> from setting >>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>> >>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>> I haven't found any clean solution for solving this problem >>>>>>>>>>> on Windows. >>>>>>>>>> >>>>>>>>>> This seems like an insufficient fix if you can't make it work >>>>>>>>>> on all platforms. I tried building with >>>>>>>>>> "-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" ons Solaris >>>>>>>>>> and hit an #error in libelf.h saying it wasn't supported so I >>>>>>>>>> understand your problem there. >>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>> That's why I chose to apply the flag only to the files >>>>>>>>> (ostream.cpp and ostream.hpp) I want the effect. >>>>>>>>>> >>>>>>>>>> Instead I suggest that you use the compatibility API >>>>>>>>>> described in lf64(5) on Solaris. This API consists of >>>>>>>>>> fopen64, ftell64 and friends and is exposed when >>>>>>>>>> "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>> >>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and >>>>>>>>>> has the added advantage of not changing any existing symbols >>>>>>>>>> and therefore we can set the define for all files instead of >>>>>>>>>> just ostream >>>>>>>>>> >>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>> resembles the changes which will be needed for Windows >>>>>>>>>> anyway. Those changes would consist of changing calls to >>>>>>>>>> ftell/fseek to 64-bit versions and changing fopen to fopen64 >>>>>>>>>> on Solaris/Linux. >>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>> excludes the usage of fopen64, providing portability (since >>>>>>>>> there's no fopen64 for Windows). Meanwhile, I understand your >>>>>>>>> suggestion provides other benefits. >>>>>>>>> >>>>>>>>> This Sun White Paper >>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>> summarizes the usage of the flags on solaris (Page 5-26). And, >>>>>>>>> it should apply to Linux the same way as was agreed across >>>>>>>>> platforms. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Since there is no fopen64 on Windows it seems that the >>>>>>>>>> default fopen already supports large files. >>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows >>>>>>>>> can write beyond 2GB (and beyond 4GB). Thank you, it's solved >>>>>>>>> "half of my problem" :) >>>>>>>>>> >>>>>>>>>> /Mikael >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> test: >>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>> verified on the >>>>>>>>>>> following configurations. >>>>>>>>>>> >>>>>>>>>>> Linux * i586 >>>>>>>>>>> Solaris * i586 >>>>>>>>>>> Solaris * sparc >>>>>>>>>>> >>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>> >>>>> >>> > From yumin.qi at oracle.com Wed Jun 5 00:55:13 2013 From: yumin.qi at oracle.com (Yumin Qi) Date: Tue, 04 Jun 2013 17:55:13 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE7791.1000305@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE74CD.8000000@oracle.com> <51AE7791.1000305@oracle.com> Message-ID: <51AE8C71.7050105@oracle.com> You can ignore my noise. Thanks Yumin On 6/4/2013 4:26 PM, Tao Mao wrote: > Thank you for looking at it, Yumin. > > Please see inline. > > Tao > > On 6/4/13 4:14 PM, Yumin Qi wrote: >> Hi, Tao >> >> The change works only when LP64 is not defined as "1", I would >> think you need to check if it is defined instead. (how about >> LP64=other?, source code only check if it is defined). > Is that true? > > I found the following exist in the current make files. > > make//bsd/makefiles/defs.make:34:ifeq ($(LP64), 1) > make//linux/makefiles/defs.make:40:ifeq ($(LP64), 1) > make//solaris/makefiles/defs.make:33:ifeq ($(LP64), 1) > > ifeq ($(LP64), 1) > ARCH_DATA_MODEL ?= 64 > else > ARCH_DATA_MODEL ?= 32 > endif > > > It looks that it assumes users should set LP64=1 in order to get a > correct build. > > Just presenting something I found. I'm not sure about this, either. > The make files are a mystery to me. > >> I am not a reviewer. >> >> Thanks >> Yumin >> >> On 6/4/2013 4:03 PM, Tao Mao wrote: >>> Since the changeset touched makefiles, I've included >>> build-dev at openjdk.java.net . >>> >>> I need to push the hsx24 bug asap. Please review it. >>> >>> Thanks. >>> Tao >>> >>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>> Hi all, >>>> >>>> Need reviews to catch RDP2. >>>> >>>> The current webrev is a working solution to all platforms, Linux, >>>> Windows, and Solaris. >>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>> Included runtime dev to see whether they have some idea to handle >>>>> the compilation choices. >>>>> >>>>> For now, it's been verified that the fix is functionally sufficient. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>> Thank you, Mikael. >>>>>> >>>>>> Please see inline. >>>>>> >>>>>> Reviewers, please review it based on the following new observation. >>>>>> >>>>>> Tao >>>>>> >>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>> Tao, >>>>>>> >>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>> 7ux bug >>>>>>>> >>>>>>>> webrev: >>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> changeset: >>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>> ostream.o >>>>>>>> >>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>> globally >>>>>>>> applicable? >>>>>>>> >>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>> however, >>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>> globally >>>>>>>> to Solaris. >>>>>>>> >>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>>> files >>>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>>> setting >>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>> >>>>>>>> (2) For now, no Windows solution. >>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>> Windows. >>>>>>> >>>>>>> This seems like an insufficient fix if you can't make it work on >>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>> libelf.h saying it wasn't supported so I understand your problem >>>>>>> there. >>>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>> ostream.hpp) I want the effect. >>>>>>> >>>>>>> Instead I suggest that you use the compatibility API described >>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>> >>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>> the added advantage of not changing any existing symbols and >>>>>>> therefore we can set the define for all files instead of just >>>>>>> ostream >>>>>>> >>>>>>> This approach has the added advantage that it more closely >>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>> Both ways have pros and cons. The current implementation excludes >>>>>> the usage of fopen64, providing portability (since there's no >>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>> provides other benefits. >>>>>> >>>>>> This Sun White Paper >>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) summarizes >>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>> apply to Linux the same way as was agreed across platforms. >>>>>> >>>>>>> >>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>> fopen already supports large files. >>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>> of my problem" :) >>>>>>> >>>>>>> /Mikael >>>>>>> >>>>>>>> >>>>>>>> test: >>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>> verified on the >>>>>>>> following configurations. >>>>>>>> >>>>>>>> Linux * i586 >>>>>>>> Solaris * i586 >>>>>>>> Solaris * sparc >>>>>>>> >>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>> >> From coleen.phillimore at oracle.com Wed Jun 5 01:59:53 2013 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Tue, 4 Jun 2013 18:59:53 -0700 (PDT) Subject: RFR (M/L): 8010722 assert: failed: heap size is too big for compressed oops (possibly CR 8009778) In-Reply-To: <1370359078.2650.55.camel@cirrus> References: <1368542230.2677.0.camel@cirrus> <1370359078.2650.55.camel@cirrus> Message-ID: <51AE9B99.5070704@oracle.com> I found it: http://cr.openjdk.java.net/~tschatzl/8010722/webrev.1/ I looked at arguments.cpp and universe.cpp for the runtime. Looks good. Coleen On 6/4/2013 11:17 AM, Thomas Schatzl wrote: > Hi all, > > please have a look at a slightly updated version of this CR after > comments from SQE and runtime. > > Changes: > - moved the comment that explained the choice of a 100M default > ClassMetaSpaceSize to the declaration of this value > - changed the default size of ClassMetaSpaceSize to 128M as this is a > value that results in less automatic resizing due to alignment (on > request). > Runtime intends to change this code in the future so that the > ClassMetaSpaceSize is not dependent on java heap alignment any more (as > far as I understood). > - resolved a small merge conflict when updating to latest hotspot-gc: > ParallelScavengeHeap::intra_heap_alignment() needs to be a static method > to be used in a static context. This is not an issue here as it returns > a constant value anyway. > - added -XX:+UseParallelGC -XX:-UseParallelOldGC as gc combination to > test in the test case. This required some changes in passing down these > arguments to the tests themselves. > > Testing: > jtreg tests, jprt > > There is a small introduction on the change below. > > On Tue, 2013-05-14 at 16:37 +0200, Thomas Schatzl wrote: >> Hi all, >> >> can I have a review for the following change? >> >> In argument processing related to ergonomically determining compressed >> oops use, there were a few use-before-set issues leading to crashes that >> this fix tries to resolve. >> >> bugs.sun.com >> http://bugs.sun.com/view_bug.do?bug_id=8010722 >> >> JIRA: >> https://jbs.oracle.com/bugs/browse/JDK-8010722 >> >> webrev: >> http://cr.openjdk.java.net/~tschatzl/8010722/webrev/ >> >> testing: >> jprt, test cases >> >> Here's a walkthrough of the changes: >> >> As mentioned, the cause of this CR is that ergonomics for determining >> the maximum java heap size usable for compressed oops uses variables >> that are later changed ergonomically. >> >> It is best to look at the changes beginning from >> Arguments::set_ergonomics_flags(): the idea of this change is to avoid >> later overflow, so the change tries to conservatively estimate sizes of >> the non-java heap parts. The complication is that not even the later >> effective alignment of these heap parts has been determined at this >> point. >> >> So the change first calculates the maximum possible heap alignment by >> calling set_max_heap_alignment(); this size is influenced by OS page >> sizes, so the change needs to initialize large pages by calling >> os::large_page_init() in Arguments::parse(), before the call to >> set_ergonomics_flags(). The maximum possible alignment is then >> calculated by asking the active GC for its maximum alignment, as at this >> point the GC has already been determined, the maximum page size, and >> other requirements, like alignment for card table size etc. >> >> Now the code can calculate the conservative estimate for actual maximum >> heap for compressed oops used in set_use_compressed_oops(), by >> subtracting the conservatively aligned sizes of the other heap parts. >> (In Arguments::max_heap_for_compressed_oops()) The result is the maximum >> possible heap that can use compressed oops, minus the aligned metaspace >> size, minus the aligned null page size. >> >> There is another circular dependency problem here, the metaspace size itself is later ergonomically sized; the change fixes this problem by anticipating that in Arguments::max_heap_for_compressed_oops(), using the same predicate for determining whether to ergonomically size it or not [this is CR8009778 I think]. >> >> The other changes are straightforward: the os_* changes result from that >> large page initialization must be done earlier now; the changes in the >> collectors themselves are simply about providing the collector's maximum >> alignment. The change in Universe::reserve_heap() contains one assertion that checks whether the conservative estimate for alignment has been conservative enough earlier. >> >> The test case tests test cases from the CR that work now, and additional >> border cases related to ergonomically deciding heap size for compressed >> oops. >> >> One side effect of this change is that the ergonomically determined heap size is slighly smaller now (so that it always fits :). > Thanks, > Thomas > From mikael.gerdin at oracle.com Wed Jun 5 08:19:52 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 5 Jun 2013 01:19:52 -0700 (PDT) Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AE8488.5050101@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> Message-ID: <51AEF4A8.4030500@oracle.com> On 2013-06-05 02:21, Daniel D. Daugherty wrote: > OK, based on the largefiles.pdf write-up, your use of _FILE_OFFSET_BITS=64 > is going to cause ostream.o to bind to various 64-bit versions of some > functions. Based on my very hazy memory of my days in file system code, > this binding is going to affect ostream.o only unless ostream.o is > writing to some file with the foo64() function and some other code is > also writing to the same file at the same time with foo(). However, if we > have two different functions writing to the same open file at the same > time, then we have bigger issues. :-) > > I'm good with these changes now. I agree that solving the problem of > setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to > solved right now. I think this change is really scary, setting _FILE_OFFSET_BITS=64 when compiling ostream.cpp will effectively cause the headers to swap out the implementations of the f[open,tell,seek] to 64 bit versions for all headers that are included and inlined in ostream.cpp. Other parts of the code using the same headers will see different versions of the functions with different parameters due to off_t changing sizes. I think that what we should do is to use the "Transitional compilation environment" detailed in ?2.6.2 in largefiles.pdf and change the calls in ostream.cpp to use the appropriate f[open,tell,seek]64 functions directly. I feel this is especially important at this late stage in the release to make an explicit change instead of setting a #define which has propagating side-effects. As Tao mentioned this will require us to handle the fact that there is no fopen64() call on Windows, that the CRT fopen() already seems to handle large files and that ftell64() and fseek64() have slightly different names on Windows. I don't think this is a large hurdle and I think we know how to solve this problem. /Mikael > > Dan > > > On 6/4/13 6:06 PM, Tao Mao wrote: >> Thank you for review, Dan. >> >> I'll try to answer as much as I can. Please see inline. >> >> Thanks. >> Tao >> >> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>> >>> Tao, >>> >>> I think the lack of response to this review request is the absolutely >>> strange nature of these changes. And I thought I put out some weird >>> code reviews... :-) >>> >>> make/linux/makefiles/vm.make >>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>> obvious in this webrev about what this will mean so I took a >>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>> use of _FILE_OFFSET_BITS in either of those source files. >>> >>> Must be in the source files somewhere, but I can't find any >>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>> >>> make/solaris/makefiles/vm.make >>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>> >>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>> Solaris box. Lots of references, but nothing that helps me >>> understand what you're doing here. >>> src/os/solaris/vm/os_solaris.inline.hpp >>> The addition of _FILE_OFFSET_BITS==64 means that the >>> os::readdir() function will use the safer, multi-threaded >>> version of readdir_r(). Seems fine to me. >>> >>> Here's what I need to know: >>> >>> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? >> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >> >> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >> refer to the following document >> >> This Sun White Paper >> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >> summarizes the usage of the flags on solaris (page "5-26"). And, it >> should apply to Linux the same way as was agreed across platforms >> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >> >>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>> what happens if another part of the VM has a different idea about >>> the value of _FILE_OFFSET_BITS? >> _FILE_OFFSET_BITS is not set for other particular effects, but for >> extending the ability to deal with large files in ostream.{c,h}pp. So, >> if other files have a different idea about _FILE_OFFSET_BITS, they >> can't deal with large files. No more no less. >> >>> >>> I saw this in the post to the Runtime alias: >>> >>> > Included runtime dev to see whether they have some idea to handle >>> > the compilation choices. >>> >>> And I still have no idea what you're asking here? What compilation >>> choices? Are you asking about your Makefile changes? Are asking >>> about defining _FILE_OFFSET_BITS for the entire build instead of >>> just one object (ostream.o)? Are you worried that this VM is going >>> to have mis-matched pieces and be unstable? >> "Are asking about defining _FILE_OFFSET_BITS for the entire build >> instead of just one object (ostream.o)?" is my main question I >> originally tried to ask. >> >>> >>> So I reviewed it, but I definitely can't approve it without more >>> info. I realize that you're up against the RDP2 limit, but this >>> change has too many open questions (for now)... >>> >>> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >>> GC log or not. The conversation below didn't help me at all. >> I used a jdk7 (just any) to successfully generate a log file larger >> than 4GB. So, it shouldn't be a problem for Win32. >>> >>> Dan >>> >>> >>> >>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>> Since the changeset touched makefiles, I've included >>>> build-dev at openjdk.java.net . >>>> >>>> I need to push the hsx24 bug asap. Please review it. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>> Hi all, >>>>> >>>>> Need reviews to catch RDP2. >>>>> >>>>> The current webrev is a working solution to all platforms, Linux, >>>>> Windows, and Solaris. >>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>> Included runtime dev to see whether they have some idea to handle >>>>>> the compilation choices. >>>>>> >>>>>> For now, it's been verified that the fix is functionally sufficient. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>> Thank you, Mikael. >>>>>>> >>>>>>> Please see inline. >>>>>>> >>>>>>> Reviewers, please review it based on the following new observation. >>>>>>> >>>>>>> Tao >>>>>>> >>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>> Tao, >>>>>>>> >>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>> 7ux bug >>>>>>>>> >>>>>>>>> webrev: >>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>> >>>>>>>>> changeset: >>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>> ostream.o >>>>>>>>> >>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>> globally >>>>>>>>> applicable? >>>>>>>>> >>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>> however, >>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>> globally >>>>>>>>> to Solaris. >>>>>>>>> >>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>>>> files >>>>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>>>> setting >>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>> >>>>>>>>> (2) For now, no Windows solution. >>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>> Windows. >>>>>>>> >>>>>>>> This seems like an insufficient fix if you can't make it work on >>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>> libelf.h saying it wasn't supported so I understand your problem >>>>>>>> there. >>>>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>> ostream.hpp) I want the effect. >>>>>>>> >>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>> >>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>> therefore we can set the define for all files instead of just >>>>>>>> ostream >>>>>>>> >>>>>>>> This approach has the added advantage that it more closely >>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>> Both ways have pros and cons. The current implementation excludes >>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>> provides other benefits. >>>>>>> >>>>>>> This Sun White Paper >>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) summarizes >>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>> >>>>>>>> >>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>> fopen already supports large files. >>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>> of my problem" :) >>>>>>>> >>>>>>>> /Mikael >>>>>>>> >>>>>>>>> >>>>>>>>> test: >>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>> verified on the >>>>>>>>> following configurations. >>>>>>>>> >>>>>>>>> Linux * i586 >>>>>>>>> Solaris * i586 >>>>>>>>> Solaris * sparc >>>>>>>>> >>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>> >>> > From thomas.schatzl at oracle.com Wed Jun 5 09:15:00 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 05 Jun 2013 11:15:00 +0200 Subject: RFR (M/L): 8010722 assert: failed: heap size is too big for compressed oops (possibly CR 8009778) In-Reply-To: <51AE9B99.5070704@oracle.com> References: <1368542230.2677.0.camel@cirrus> <1370359078.2650.55.camel@cirrus> <51AE9B99.5070704@oracle.com> Message-ID: <1370423700.2599.0.camel@cirrus> Hi all, On Tue, 2013-06-04 at 18:59 -0700, Coleen Phillimore wrote: > I found it: > http://cr.openjdk.java.net/~tschatzl/8010722/webrev.1/ > > > I looked at arguments.cpp and universe.cpp for the runtime. Looks good. sorry for that - and another thanks for looking at runtime changes. Thanks, Thomas > On 6/4/2013 11:17 AM, Thomas Schatzl wrote: > > Hi all, > > > > please have a look at a slightly updated version of this CR after > > comments from SQE and runtime. > > > > Changes: > > - moved the comment that explained the choice of a 100M default > > ClassMetaSpaceSize to the declaration of this value > > - changed the default size of ClassMetaSpaceSize to 128M as this is a > > value that results in less automatic resizing due to alignment (on > > request). > > Runtime intends to change this code in the future so that the > > ClassMetaSpaceSize is not dependent on java heap alignment any more (as > > far as I understood). > > - resolved a small merge conflict when updating to latest hotspot-gc: > > ParallelScavengeHeap::intra_heap_alignment() needs to be a static method > > to be used in a static context. This is not an issue here as it returns > > a constant value anyway. > > - added -XX:+UseParallelGC -XX:-UseParallelOldGC as gc combination to > > test in the test case. This required some changes in passing down these > > arguments to the tests themselves. > > > > Testing: > > jtreg tests, jprt > > > > There is a small introduction on the change below. > > > > On Tue, 2013-05-14 at 16:37 +0200, Thomas Schatzl wrote: > >> Hi all, > >> > >> can I have a review for the following change? > >> > >> In argument processing related to ergonomically determining compressed > >> oops use, there were a few use-before-set issues leading to crashes that > >> this fix tries to resolve. > >> > >> bugs.sun.com > >> http://bugs.sun.com/view_bug.do?bug_id=8010722 > >> > >> JIRA: > >> https://jbs.oracle.com/bugs/browse/JDK-8010722 > >> > >> webrev: > >> http://cr.openjdk.java.net/~tschatzl/8010722/webrev/ > >> > >> testing: > >> jprt, test cases > >> > >> Here's a walkthrough of the changes: > >> > >> As mentioned, the cause of this CR is that ergonomics for determining > >> the maximum java heap size usable for compressed oops uses variables > >> that are later changed ergonomically. > >> > >> It is best to look at the changes beginning from > >> Arguments::set_ergonomics_flags(): the idea of this change is to avoid > >> later overflow, so the change tries to conservatively estimate sizes of > >> the non-java heap parts. The complication is that not even the later > >> effective alignment of these heap parts has been determined at this > >> point. > >> > >> So the change first calculates the maximum possible heap alignment by > >> calling set_max_heap_alignment(); this size is influenced by OS page > >> sizes, so the change needs to initialize large pages by calling > >> os::large_page_init() in Arguments::parse(), before the call to > >> set_ergonomics_flags(). The maximum possible alignment is then > >> calculated by asking the active GC for its maximum alignment, as at this > >> point the GC has already been determined, the maximum page size, and > >> other requirements, like alignment for card table size etc. > >> > >> Now the code can calculate the conservative estimate for actual maximum > >> heap for compressed oops used in set_use_compressed_oops(), by > >> subtracting the conservatively aligned sizes of the other heap parts. > >> (In Arguments::max_heap_for_compressed_oops()) The result is the maximum > >> possible heap that can use compressed oops, minus the aligned metaspace > >> size, minus the aligned null page size. > >> > >> There is another circular dependency problem here, the metaspace size itself is later ergonomically sized; the change fixes this problem by anticipating that in Arguments::max_heap_for_compressed_oops(), using the same predicate for determining whether to ergonomically size it or not [this is CR8009778 I think]. > >> > >> The other changes are straightforward: the os_* changes result from that > >> large page initialization must be done earlier now; the changes in the > >> collectors themselves are simply about providing the collector's maximum > >> alignment. The change in Universe::reserve_heap() contains one assertion that checks whether the conservative estimate for alignment has been conservative enough earlier. > >> > >> The test case tests test cases from the CR that work now, and additional > >> border cases related to ergonomically deciding heap size for compressed > >> oops. > >> > >> One side effect of this change is that the ergonomically determined heap size is slighly smaller now (so that it always fits :). > > Thanks, > > Thomas > > > From per.liden at oracle.com Wed Jun 5 09:50:14 2013 From: per.liden at oracle.com (Per Liden) Date: Wed, 05 Jun 2013 11:50:14 +0200 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AE3941.90609@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> Message-ID: <51AF09D6.30803@oracle.com> Looks good! cheers, /Per On 2013-06-04 21:00, John Cuthbertson wrote: > Hi Everyone, > > Here's a new webrev for this change: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1 > > Changes from before: > * Refactored the code that loops over the buckets into its own routine. > * Removed the commented out instrumentation (oops). > * Changed the types to int to be consistent with the rest of > symbolTable and allow removal of the casts. > * Increase the number of buckets per claim to 32 based upon a verbal > comment from John Coomes. > * Removed the additional worker ID parameter for the sake of peace and > harmony. > > Testing: jprt. > > Thanks, > > JohnC > > On 5/24/2013 3:19 PM, John Cuthbertson wrote: >> Hi Everyone, >> >> Can I have a couple of reviewers look over these changes - the webrev >> is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >> >> Summary: >> On some workloads we are seeing that the scan of the intern string >> table (among others) can sometimes take quite a while. This showed up >> on some FMW workloads with G1 where the scan of the string table >> dominated the pause time for some pauses. G1 was particularly >> affected since it doesn't do class unloading (and hence pruning of >> the string table) except at full GCs. The solution was to change the >> string table from being considered a single root task and treat >> similarly to the Java thread stacks: each GC worker claims a given >> number of buckets and scans the entries in those buckets. >> >> Testing >> Kitchensink; jprt; GC test suite. With all collectors. >> >> Overhead: >> Not real performance numbers but I did some measurement of the >> synchronization overhead of using 1 GC worker thread. They are >> summarized here: >> >> >> 0-threads (ms) >> 1-thread-chunked (ms) >> Min 0.200 >> 0.300 >> Max 6.900 >> 8.800 >> Avg 0.658 >> 0.794 >> >> >> These were from 1 hour long runs of Kitchensink with around ~2800 GCs >> in each run. >> >> Thanks, >> >> JohnC > From thomas.schatzl at oracle.com Wed Jun 5 10:20:14 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 5 Jun 2013 03:20:14 -0700 (PDT) Subject: RFR (S/M): 8014078: G1: improve remembered set summary information by providing per region type information In-Reply-To: <1370389124.2688.66.camel@cirrus> References: <1369729575.2616.27.camel@cirrus> <51AD84C8.7010300@oracle.com> <1370345967.2650.27.camel@cirrus> <51ADD470.6080707@oracle.com> <1370357913.2650.42.camel@cirrus> <51AE71CB.7010502@oracle.com> <1370389124.2688.66.camel@cirrus> Message-ID: <1370427614.2599.21.camel@cirrus> Hi all, On Wed, 2013-06-05 at 01:38 +0200, Thomas Schatzl wrote: > Hi, > > On Tue, 2013-06-04 at 16:01 -0700, John Cuthbertson wrote: > > Hi Thomas, > > > > I think RegionTypeCounter should inherit from _ValueObj (by means of > > VALUE_OBJ_CLASS_SPEC). > > > > Also a couple of nitty questions (feel free to ignore): > > > > * Does the value parameter to RegionTypeCounter::percent_of need to be > > passed as a pointer? It looks like you pass the address of a field in > > the RegionTypeCounter instance and then dereference. > > Fixed both things. I am not sure why the code passes the address of the > field, but this is probably a leftover of some earlier version. > > I also combined the calc_percentage() function with the > RegionTypeCounter::percent_of() method as they are the same. > > > * For continuous humongous would it make sense to skip them until we > > process the StartsHumongous region and sum up the memory size and the > > occupied value for the entire humongous region (HS and HC) before adding > > to the humongous region counter? > > Imho there is no need to do that - the memory sizes and occupancies of > HS and HC regions are completely independent. I.e. the order of > evaluation does not matter here. Doing so seems to only complicate the > code. a current webrev is at http://cr.openjdk.java.net/~tschatzl/8014078/webrev.2/ incorporating the suggested changes. In detail: - let RegionTypeCounter inherit from _ValueObj (via VALUE_OBJ_CLASS_SPEC) - fixed the first parameter of percent_of to not pass the value by address - merged RegionTypeCounter::percent_of() and calc_percentage() - did _not_ change the iteration over the humonguous regions as suggested, for the reason above Thanks, Thomas From stefan.karlsson at oracle.com Wed Jun 5 11:10:14 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 5 Jun 2013 04:10:14 -0700 (PDT) Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AE3941.90609@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> Message-ID: <51AF1C96.4070404@oracle.com> On 06/04/2013 09:00 PM, John Cuthbertson wrote: > Hi Everyone, > > Here's a new webrev for this change: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1 Looks good. Thanks for doing all the cleanups. > > Changes from before: > * Refactored the code that loops over the buckets into its own routine. > * Removed the commented out instrumentation (oops). > * Changed the types to int to be consistent with the rest of > symbolTable and allow removal of the casts. > * Increase the number of buckets per claim to 32 based upon a verbal > comment from John Coomes. Care to describe the reasoning why 32 should be better? > * Removed the additional worker ID parameter for the sake of peace and > harmony. Thanks. StefanK > > Testing: jprt. > > Thanks, > > JohnC > > On 5/24/2013 3:19 PM, John Cuthbertson wrote: >> Hi Everyone, >> >> Can I have a couple of reviewers look over these changes - the webrev >> is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >> >> Summary: >> On some workloads we are seeing that the scan of the intern string >> table (among others) can sometimes take quite a while. This showed up >> on some FMW workloads with G1 where the scan of the string table >> dominated the pause time for some pauses. G1 was particularly >> affected since it doesn't do class unloading (and hence pruning of >> the string table) except at full GCs. The solution was to change the >> string table from being considered a single root task and treat >> similarly to the Java thread stacks: each GC worker claims a given >> number of buckets and scans the entries in those buckets. >> >> Testing >> Kitchensink; jprt; GC test suite. With all collectors. >> >> Overhead: >> Not real performance numbers but I did some measurement of the >> synchronization overhead of using 1 GC worker thread. They are >> summarized here: >> >> >> 0-threads (ms) >> 1-thread-chunked (ms) >> Min 0.200 >> 0.300 >> Max 6.900 >> 8.800 >> Avg 0.658 >> 0.794 >> >> >> These were from 1 hour long runs of Kitchensink with around ~2800 GCs >> in each run. >> >> Thanks, >> >> JohnC > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikael.gerdin at oracle.com Wed Jun 5 14:04:38 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 05 Jun 2013 16:04:38 +0200 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk Message-ID: <51AF4576.3080203@oracle.com> Hi, Can I have some reviews of this small fix to the Metaspace memory allocation path. Problem: When a Metaspace allocation request cannot be satisfied by the current chunk the chunk is retired and a new chunk is requested. This causes whatever is left in the chunk to be effectively leaked. Suggested fix: Put the remaining memory in each chunk on the Metablock freelist so it can be used to satisfy future allocations. Possible addition: When allocating from the block free list, use FreeBlockDictionary::atLeast instead of FreeBlockDictionary::exactly and split the Metablock if it's large enough. One might argue that this increases the fragmentation of the memory on the block free list but I think that we primarily want to use the block free list for small allocations and allocate from chunks for large allocations. Webrev: Only fix: http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ Incremental webrev for splitting blocks: http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ Bug links: https://jbs.oracle.com/bugs/browse/JDK-8009561 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 Thanks /Mikael From tao.mao at oracle.com Wed Jun 5 15:48:12 2013 From: tao.mao at oracle.com (Tao Mao) Date: Wed, 5 Jun 2013 08:48:12 -0700 (PDT) Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AEF4A8.4030500@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> Message-ID: <51AF5DBC.3080107@oracle.com> Thank you for comments. One thing I want to point out is that the current change has not touched on Windows code. Please see inline. Tao On 6/5/13 1:19 AM, Mikael Gerdin wrote: > > > On 2013-06-05 02:21, Daniel D. Daugherty wrote: >> OK, based on the largefiles.pdf write-up, your use of >> _FILE_OFFSET_BITS=64 >> is going to cause ostream.o to bind to various 64-bit versions of some >> functions. Based on my very hazy memory of my days in file system code, >> this binding is going to affect ostream.o only unless ostream.o is >> writing to some file with the foo64() function and some other code is >> also writing to the same file at the same time with foo(). However, >> if we >> have two different functions writing to the same open file at the same >> time, then we have bigger issues. :-) >> >> I'm good with these changes now. I agree that solving the problem of >> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >> solved right now. > > I think this change is really scary, setting _FILE_OFFSET_BITS=64 when > compiling ostream.cpp will effectively cause the headers to swap out > the implementations of the f[open,tell,seek] to 64 bit versions for > all headers that are included and inlined in ostream.cpp. > Other parts of the code using the same headers will see different > versions of the functions with different parameters due to off_t > changing sizes. The change is currently effective for Linux and Solaris if you look at the file directories. Nothing changed for Windows and BSD, as they don't need such change. > > I think that what we should do is to use the "Transitional compilation > environment" detailed in ?2.6.2 in largefiles.pdf and change the calls > in ostream.cpp to use the appropriate f[open,tell,seek]64 functions > directly. I feel this is especially important at this late stage in > the release to make an explicit change instead of setting a #define > which has propagating side-effects. How do you see "propagating side-effects" and to where? > > As Tao mentioned this will require us to handle the fact that there is > no fopen64() call on Windows, that the CRT fopen() already seems to > handle large files and that ftell64() and fseek64() have slightly > different names on Windows. I don't think this is a large hurdle and I > think we know how to solve this problem. As I said, nothing was changed for Windows code. > > /Mikael > >> >> Dan >> >> >> On 6/4/13 6:06 PM, Tao Mao wrote: >>> Thank you for review, Dan. >>> >>> I'll try to answer as much as I can. Please see inline. >>> >>> Thanks. >>> Tao >>> >>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>> >>>> Tao, >>>> >>>> I think the lack of response to this review request is the absolutely >>>> strange nature of these changes. And I thought I put out some weird >>>> code reviews... :-) >>>> >>>> make/linux/makefiles/vm.make >>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>> obvious in this webrev about what this will mean so I took a >>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>> use of _FILE_OFFSET_BITS in either of those source files. >>>> >>>> Must be in the source files somewhere, but I can't find any >>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>> >>>> make/solaris/makefiles/vm.make >>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>> >>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>> Solaris box. Lots of references, but nothing that helps me >>>> understand what you're doing here. >>>> src/os/solaris/vm/os_solaris.inline.hpp >>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>> os::readdir() function will use the safer, multi-threaded >>>> version of readdir_r(). Seems fine to me. >>>> >>>> Here's what I need to know: >>>> >>>> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? >>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>> >>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>> refer to the following document >>> >>> This Sun White Paper >>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>> should apply to Linux the same way as was agreed across platforms >>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>> >>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>> what happens if another part of the VM has a different idea about >>>> the value of _FILE_OFFSET_BITS? >>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>> extending the ability to deal with large files in ostream.{c,h}pp. So, >>> if other files have a different idea about _FILE_OFFSET_BITS, they >>> can't deal with large files. No more no less. >>> >>>> >>>> I saw this in the post to the Runtime alias: >>>> >>>> > Included runtime dev to see whether they have some idea to handle >>>> > the compilation choices. >>>> >>>> And I still have no idea what you're asking here? What compilation >>>> choices? Are you asking about your Makefile changes? Are asking >>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>> just one object (ostream.o)? Are you worried that this VM is going >>>> to have mis-matched pieces and be unstable? >>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>> instead of just one object (ostream.o)?" is my main question I >>> originally tried to ask. >>> >>>> >>>> So I reviewed it, but I definitely can't approve it without more >>>> info. I realize that you're up against the RDP2 limit, but this >>>> change has too many open questions (for now)... >>>> >>>> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >>>> GC log or not. The conversation below didn't help me at all. >>> I used a jdk7 (just any) to successfully generate a log file larger >>> than 4GB. So, it shouldn't be a problem for Win32. >>>> >>>> Dan >>>> >>>> >>>> >>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>> Since the changeset touched makefiles, I've included >>>>> build-dev at openjdk.java.net . >>>>> >>>>> I need to push the hsx24 bug asap. Please review it. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>> Hi all, >>>>>> >>>>>> Need reviews to catch RDP2. >>>>>> >>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>> Windows, and Solaris. >>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>> the compilation choices. >>>>>>> >>>>>>> For now, it's been verified that the fix is functionally >>>>>>> sufficient. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>> Thank you, Mikael. >>>>>>>> >>>>>>>> Please see inline. >>>>>>>> >>>>>>>> Reviewers, please review it based on the following new >>>>>>>> observation. >>>>>>>> >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>> Tao, >>>>>>>>> >>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>> 7ux bug >>>>>>>>>> >>>>>>>>>> webrev: >>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>> >>>>>>>>>> changeset: >>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>> ostream.o >>>>>>>>>> >>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>> globally >>>>>>>>>> applicable? >>>>>>>>>> >>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>> however, >>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>> globally >>>>>>>>>> to Solaris. >>>>>>>>>> >>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>>>>> files >>>>>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>>>>> setting >>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>> >>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>> Windows. >>>>>>>>> >>>>>>>>> This seems like an insufficient fix if you can't make it work on >>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>> libelf.h saying it wasn't supported so I understand your problem >>>>>>>>> there. >>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>> ostream.hpp) I want the effect. >>>>>>>>> >>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>> >>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>> ostream >>>>>>>>> >>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>> Both ways have pros and cons. The current implementation excludes >>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>> provides other benefits. >>>>>>>> >>>>>>>> This Sun White Paper >>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>> summarizes >>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>> >>>>>>>>> >>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>> fopen already supports large files. >>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>> of my problem" :) >>>>>>>>> >>>>>>>>> /Mikael >>>>>>>>> >>>>>>>>>> >>>>>>>>>> test: >>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>> verified on the >>>>>>>>>> following configurations. >>>>>>>>>> >>>>>>>>>> Linux * i586 >>>>>>>>>> Solaris * i586 >>>>>>>>>> Solaris * sparc >>>>>>>>>> >>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>> >>>> >> From mikael.gerdin at oracle.com Wed Jun 5 16:02:46 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 05 Jun 2013 18:02:46 +0200 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AF5DBC.3080107@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> Message-ID: <51AF6126.5050106@oracle.com> Tao, On 2013-06-05 17:48, Tao Mao wrote: > Thank you for comments. One thing I want to point out is that the > current change has not touched on Windows code. > > Please see inline. > > Tao > > On 6/5/13 1:19 AM, Mikael Gerdin wrote: >> >> >> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>> OK, based on the largefiles.pdf write-up, your use of >>> _FILE_OFFSET_BITS=64 >>> is going to cause ostream.o to bind to various 64-bit versions of some >>> functions. Based on my very hazy memory of my days in file system code, >>> this binding is going to affect ostream.o only unless ostream.o is >>> writing to some file with the foo64() function and some other code is >>> also writing to the same file at the same time with foo(). However, >>> if we >>> have two different functions writing to the same open file at the same >>> time, then we have bigger issues. :-) >>> >>> I'm good with these changes now. I agree that solving the problem of >>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>> solved right now. >> >> I think this change is really scary, setting _FILE_OFFSET_BITS=64 when >> compiling ostream.cpp will effectively cause the headers to swap out >> the implementations of the f[open,tell,seek] to 64 bit versions for >> all headers that are included and inlined in ostream.cpp. >> Other parts of the code using the same headers will see different >> versions of the functions with different parameters due to off_t >> changing sizes. > The change is currently effective for Linux and Solaris if you look at > the file directories. Nothing changed for Windows and BSD, as they don't > need such change. Right. But if you use my suggested approach you would need to change calls to fopen() in ostream.cpp to fopen_pd where if (linux || solaris) && 32bit #define fopen_pd fopen64 else #define fopen_pd fopen >> >> I think that what we should do is to use the "Transitional compilation >> environment" detailed in ?2.6.2 in largefiles.pdf and change the calls >> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >> directly. I feel this is especially important at this late stage in >> the release to make an explicit change instead of setting a #define >> which has propagating side-effects. > How do you see "propagating side-effects" and to where? _FILE_OFFSET_BITS=64 changes the definition of fopen for every file including stdio.h. It's confusing when a call to "fopen()" in one file calls the 64 bit version and in other files it doesn't. How will this work with precompiled headers? Which version of fopen will be in the precompiled header file? >> >> As Tao mentioned this will require us to handle the fact that there is >> no fopen64() call on Windows, that the CRT fopen() already seems to >> handle large files and that ftell64() and fseek64() have slightly >> different names on Windows. I don't think this is a large hurdle and I >> think we know how to solve this problem. > As I said, nothing was changed for Windows code. No, but to be consistent you'd need to update the ftell* fseek* to use 64 bit versions, right? /Mikael >> >> /Mikael >> >>> >>> Dan >>> >>> >>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>> Thank you for review, Dan. >>>> >>>> I'll try to answer as much as I can. Please see inline. >>>> >>>> Thanks. >>>> Tao >>>> >>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>> >>>>> Tao, >>>>> >>>>> I think the lack of response to this review request is the absolutely >>>>> strange nature of these changes. And I thought I put out some weird >>>>> code reviews... :-) >>>>> >>>>> make/linux/makefiles/vm.make >>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>> obvious in this webrev about what this will mean so I took a >>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>> >>>>> Must be in the source files somewhere, but I can't find any >>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>> >>>>> make/solaris/makefiles/vm.make >>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>> >>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>> Solaris box. Lots of references, but nothing that helps me >>>>> understand what you're doing here. >>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>> os::readdir() function will use the safer, multi-threaded >>>>> version of readdir_r(). Seems fine to me. >>>>> >>>>> Here's what I need to know: >>>>> >>>>> - what effect does _FILE_OFFSET_BITS have on building ostream.{c,h}pp? >>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>> >>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>> refer to the following document >>>> >>>> This Sun White Paper >>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>> should apply to Linux the same way as was agreed across platforms >>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>> >>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>> what happens if another part of the VM has a different idea about >>>>> the value of _FILE_OFFSET_BITS? >>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>> extending the ability to deal with large files in ostream.{c,h}pp. So, >>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>> can't deal with large files. No more no less. >>>> >>>>> >>>>> I saw this in the post to the Runtime alias: >>>>> >>>>> > Included runtime dev to see whether they have some idea to handle >>>>> > the compilation choices. >>>>> >>>>> And I still have no idea what you're asking here? What compilation >>>>> choices? Are you asking about your Makefile changes? Are asking >>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>> to have mis-matched pieces and be unstable? >>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>> instead of just one object (ostream.o)?" is my main question I >>>> originally tried to ask. >>>> >>>>> >>>>> So I reviewed it, but I definitely can't approve it without more >>>>> info. I realize that you're up against the RDP2 limit, but this >>>>> change has too many open questions (for now)... >>>>> >>>>> BTW, it is not at all clear whether Win32 will be able to write a 2GB+ >>>>> GC log or not. The conversation below didn't help me at all. >>>> I used a jdk7 (just any) to successfully generate a log file larger >>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>> >>>>> Dan >>>>> >>>>> >>>>> >>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>> Since the changeset touched makefiles, I've included >>>>>> build-dev at openjdk.java.net . >>>>>> >>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> Need reviews to catch RDP2. >>>>>>> >>>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>>> Windows, and Solaris. >>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>>> the compilation choices. >>>>>>>> >>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>> sufficient. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>> Thank you, Mikael. >>>>>>>>> >>>>>>>>> Please see inline. >>>>>>>>> >>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>> observation. >>>>>>>>> >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>> Tao, >>>>>>>>>> >>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>> 7ux bug >>>>>>>>>>> >>>>>>>>>>> webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> changeset: >>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>> ostream.o >>>>>>>>>>> >>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>> globally >>>>>>>>>>> applicable? >>>>>>>>>>> >>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>>> however, >>>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>>> globally >>>>>>>>>>> to Solaris. >>>>>>>>>>> >>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest four >>>>>>>>>>> files >>>>>>>>>>> had conflicts deep in c library. Even if they are excluded from >>>>>>>>>>> setting >>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>> >>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>>> Windows. >>>>>>>>>> >>>>>>>>>> This seems like an insufficient fix if you can't make it work on >>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>> libelf.h saying it wasn't supported so I understand your problem >>>>>>>>>> there. >>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. That's >>>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>> >>>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 and >>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>> >>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>> ostream >>>>>>>>>> >>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>>> Both ways have pros and cons. The current implementation excludes >>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>> provides other benefits. >>>>>>>>> >>>>>>>>> This Sun White Paper >>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>> summarizes >>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>>> fopen already supports large files. >>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>>> of my problem" :) >>>>>>>>>> >>>>>>>>>> /Mikael >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> test: >>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>> verified on the >>>>>>>>>>> following configurations. >>>>>>>>>>> >>>>>>>>>>> Linux * i586 >>>>>>>>>>> Solaris * i586 >>>>>>>>>>> Solaris * sparc >>>>>>>>>>> >>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>> >>>>> >>> From daniel.daugherty at oracle.com Wed Jun 5 16:06:41 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 05 Jun 2013 10:06:41 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AF6126.5050106@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> Message-ID: <51AF6211.8020607@oracle.com> On 6/5/13 10:02 AM, Mikael Gerdin wrote: > Tao, > > On 2013-06-05 17:48, Tao Mao wrote: >> Thank you for comments. One thing I want to point out is that the >> current change has not touched on Windows code. >> >> Please see inline. >> >> Tao >> >> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>> >>> >>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>> OK, based on the largefiles.pdf write-up, your use of >>>> _FILE_OFFSET_BITS=64 >>>> is going to cause ostream.o to bind to various 64-bit versions of some >>>> functions. Based on my very hazy memory of my days in file system >>>> code, >>>> this binding is going to affect ostream.o only unless ostream.o is >>>> writing to some file with the foo64() function and some other code is >>>> also writing to the same file at the same time with foo(). However, >>>> if we >>>> have two different functions writing to the same open file at the same >>>> time, then we have bigger issues. :-) >>>> >>>> I'm good with these changes now. I agree that solving the problem of >>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>> solved right now. >>> >>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 when >>> compiling ostream.cpp will effectively cause the headers to swap out >>> the implementations of the f[open,tell,seek] to 64 bit versions for >>> all headers that are included and inlined in ostream.cpp. >>> Other parts of the code using the same headers will see different >>> versions of the functions with different parameters due to off_t >>> changing sizes. >> The change is currently effective for Linux and Solaris if you look at >> the file directories. Nothing changed for Windows and BSD, as they don't >> need such change. > > Right. > But if you use my suggested approach you would need to change calls to > fopen() in ostream.cpp to fopen_pd where > > if (linux || solaris) && 32bit > #define fopen_pd fopen64 > else > #define fopen_pd fopen > >>> >>> I think that what we should do is to use the "Transitional compilation >>> environment" detailed in ?2.6.2 in largefiles.pdf and change the calls >>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>> directly. I feel this is especially important at this late stage in >>> the release to make an explicit change instead of setting a #define >>> which has propagating side-effects. >> How do you see "propagating side-effects" and to where? > > _FILE_OFFSET_BITS=64 changes the definition of fopen for every file > including stdio.h. > > It's confusing when a call to "fopen()" in one file calls the 64 bit > version and in other files it doesn't. > > How will this work with precompiled headers? Which version of fopen > will be in the precompiled header file? You're thinking this is like a "#define" and it is not. The ostream.o binary will bind to the foo64() version of functions and not the foo() version of functions. This is not the same as using a #define. Dan > >>> >>> As Tao mentioned this will require us to handle the fact that there is >>> no fopen64() call on Windows, that the CRT fopen() already seems to >>> handle large files and that ftell64() and fseek64() have slightly >>> different names on Windows. I don't think this is a large hurdle and I >>> think we know how to solve this problem. >> As I said, nothing was changed for Windows code. > > No, but to be consistent you'd need to update the ftell* fseek* to use > 64 bit versions, right? > > /Mikael > >>> >>> /Mikael >>> >>>> >>>> Dan >>>> >>>> >>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>> Thank you for review, Dan. >>>>> >>>>> I'll try to answer as much as I can. Please see inline. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> Tao, >>>>>> >>>>>> I think the lack of response to this review request is the >>>>>> absolutely >>>>>> strange nature of these changes. And I thought I put out some weird >>>>>> code reviews... :-) >>>>>> >>>>>> make/linux/makefiles/vm.make >>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>> obvious in this webrev about what this will mean so I took a >>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>> >>>>>> Must be in the source files somewhere, but I can't find any >>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>> >>>>>> make/solaris/makefiles/vm.make >>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>> >>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>> understand what you're doing here. >>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>> os::readdir() function will use the safer, multi-threaded >>>>>> version of readdir_r(). Seems fine to me. >>>>>> >>>>>> Here's what I need to know: >>>>>> >>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>> ostream.{c,h}pp? >>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>> >>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>> refer to the following document >>>>> >>>>> This Sun White Paper >>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>> should apply to Linux the same way as was agreed across platforms >>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>> >>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>> what happens if another part of the VM has a different idea about >>>>>> the value of _FILE_OFFSET_BITS? >>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>> So, >>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>> can't deal with large files. No more no less. >>>>> >>>>>> >>>>>> I saw this in the post to the Runtime alias: >>>>>> >>>>>> > Included runtime dev to see whether they have some idea to handle >>>>>> > the compilation choices. >>>>>> >>>>>> And I still have no idea what you're asking here? What compilation >>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>> to have mis-matched pieces and be unstable? >>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>> instead of just one object (ostream.o)?" is my main question I >>>>> originally tried to ask. >>>>> >>>>>> >>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>> change has too many open questions (for now)... >>>>>> >>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>> 2GB+ >>>>>> GC log or not. The conversation below didn't help me at all. >>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>> >>>>>> Dan >>>>>> >>>>>> >>>>>> >>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>> Since the changeset touched makefiles, I've included >>>>>>> build-dev at openjdk.java.net . >>>>>>> >>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Need reviews to catch RDP2. >>>>>>>> >>>>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>>>> Windows, and Solaris. >>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>>>> the compilation choices. >>>>>>>>> >>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>> sufficient. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>> Thank you, Mikael. >>>>>>>>>> >>>>>>>>>> Please see inline. >>>>>>>>>> >>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>> observation. >>>>>>>>>> >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>> Tao, >>>>>>>>>>> >>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>> 7ux bug >>>>>>>>>>>> >>>>>>>>>>>> webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>> >>>>>>>>>>>> changeset: >>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>> ostream.o >>>>>>>>>>>> >>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>> globally >>>>>>>>>>>> applicable? >>>>>>>>>>>> >>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>>>> however, >>>>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>>>> globally >>>>>>>>>>>> to Solaris. >>>>>>>>>>>> >>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>> four >>>>>>>>>>>> files >>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>> from >>>>>>>>>>>> setting >>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>> >>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>>>> Windows. >>>>>>>>>>> >>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>> work on >>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>> problem >>>>>>>>>>> there. >>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>> That's >>>>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>> >>>>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>> and >>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>> >>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>> ostream >>>>>>>>>>> >>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>> excludes >>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>> provides other benefits. >>>>>>>>>> >>>>>>>>>> This Sun White Paper >>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>> >>>>>>>>>> summarizes >>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>>>> fopen already supports large files. >>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>>>> of my problem" :) >>>>>>>>>>> >>>>>>>>>>> /Mikael >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> test: >>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>> verified on the >>>>>>>>>>>> following configurations. >>>>>>>>>>>> >>>>>>>>>>>> Linux * i586 >>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>> >>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>> >>>>>> >>>> From tao.mao at oracle.com Wed Jun 5 17:03:31 2013 From: tao.mao at oracle.com (Tao Mao) Date: Wed, 05 Jun 2013 10:03:31 -0700 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AF6126.5050106@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> Message-ID: <51AF6F63.3010906@oracle.com> Hi Mikael, Let me explain the changeset first before I go into your comments. (1) make/linux/makefiles/vm.make # Large File Support ifneq ($(LP64), 1) CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 endif # ifneq ($(LP64), 1) For "Linux" code, set _FILE_OFFSET_BITS=64 "only" to ostream.o (i.e. ostream.{c,h}pp) in order to bind all foo() functions to foo64() functions when linking libraries. This will enable functions in ostream.{c,h}pp to deal with large files while all other files won't have the ability. And that's what we want at this point. (2) make/solaris/makefiles/vm.make # Large File Support ifneq ($(LP64), 1) CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 endif # ifneq ($(LP64), 1) Similarly, for "Solaris" code, set _FILE_OFFSET_BITS=64 "only" to ostream.o (i.e. ostream.{c,h}pp) in order to bind all foo() functions to foo64() functions when linking libraries. This will enable functions in ostream.{c,h}pp to deal with large files while all other files won't have the ability. (1) & (2) altogether set _FILE_OFFSET_BITS=64 "only" to ostream.o in "Linux" and "Solaris" code base and have not changed Windows and BSD code, as they can handle large files currently. No need to change. Remember the current implementation has a limited affected range (i.e. ostream.o for Linux and Solaris). No more no less. (3) src/os/solaris/vm/os_solaris.inline.hpp #if defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 dirent* p; int status; if((status = ::readdir_r(dirp, dbuf,&p)) != 0) { errno = status; return NULL; } else return p; #else // defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 return ::readdir_r(dirp, dbuf); #endif // defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 This piece of code handles some exception I need to fix. It is not what I want to change but rather what I have to change. Please see inline. Thanks. Tao On 6/5/13 9:02 AM, Mikael Gerdin wrote: > Tao, > > On 2013-06-05 17:48, Tao Mao wrote: >> Thank you for comments. One thing I want to point out is that the >> current change has not touched on Windows code. >> >> Please see inline. >> >> Tao >> >> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>> >>> >>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>> OK, based on the largefiles.pdf write-up, your use of >>>> _FILE_OFFSET_BITS=64 >>>> is going to cause ostream.o to bind to various 64-bit versions of some >>>> functions. Based on my very hazy memory of my days in file system >>>> code, >>>> this binding is going to affect ostream.o only unless ostream.o is >>>> writing to some file with the foo64() function and some other code is >>>> also writing to the same file at the same time with foo(). However, >>>> if we >>>> have two different functions writing to the same open file at the same >>>> time, then we have bigger issues. :-) >>>> >>>> I'm good with these changes now. I agree that solving the problem of >>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>> solved right now. >>> >>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 when >>> compiling ostream.cpp will effectively cause the headers to swap out >>> the implementations of the f[open,tell,seek] to 64 bit versions for >>> all headers that are included and inlined in ostream.cpp. >>> Other parts of the code using the same headers will see different >>> versions of the functions with different parameters due to off_t >>> changing sizes. >> The change is currently effective for Linux and Solaris if you look at >> the file directories. Nothing changed for Windows and BSD, as they don't >> need such change. > > Right. > But if you use my suggested approach you would need to change calls to > fopen() in ostream.cpp to fopen_pd where > > if (linux || solaris) && 32bit > #define fopen_pd fopen64 > else > #define fopen_pd fopen I saw what you suggested. But, what's the benefit of doing so, or, what's the severe problem about the current changeset? I don't see that one's better than another. > >>> >>> I think that what we should do is to use the "Transitional compilation >>> environment" detailed in ?2.6.2 in largefiles.pdf and change the calls >>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>> directly. I feel this is especially important at this late stage in >>> the release to make an explicit change instead of setting a #define >>> which has propagating side-effects. >> How do you see "propagating side-effects" and to where? > > _FILE_OFFSET_BITS=64 changes the definition of fopen for every file > including stdio.h. The current implementation has a limited affected range (i.e. ostream.o for Linux and Solaris). No more no less. How do you see this flag will affect every file? > > It's confusing when a call to "fopen()" in one file calls the 64 bit > version and in other files it doesn't. > > How will this work with precompiled headers? Which version of fopen > will be in the precompiled header file? > >>> >>> As Tao mentioned this will require us to handle the fact that there is >>> no fopen64() call on Windows, that the CRT fopen() already seems to >>> handle large files and that ftell64() and fseek64() have slightly >>> different names on Windows. I don't think this is a large hurdle and I >>> think we know how to solve this problem. >> As I said, nothing was changed for Windows code. > > No, but to be consistent you'd need to update the ftell* fseek* to use > 64 bit versions, right? > > /Mikael > >>> >>> /Mikael >>> >>>> >>>> Dan >>>> >>>> >>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>> Thank you for review, Dan. >>>>> >>>>> I'll try to answer as much as I can. Please see inline. >>>>> >>>>> Thanks. >>>>> Tao >>>>> >>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>> >>>>>> Tao, >>>>>> >>>>>> I think the lack of response to this review request is the >>>>>> absolutely >>>>>> strange nature of these changes. And I thought I put out some weird >>>>>> code reviews... :-) >>>>>> >>>>>> make/linux/makefiles/vm.make >>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>> obvious in this webrev about what this will mean so I took a >>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>> >>>>>> Must be in the source files somewhere, but I can't find any >>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>> >>>>>> make/solaris/makefiles/vm.make >>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>> >>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>> understand what you're doing here. >>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>> os::readdir() function will use the safer, multi-threaded >>>>>> version of readdir_r(). Seems fine to me. >>>>>> >>>>>> Here's what I need to know: >>>>>> >>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>> ostream.{c,h}pp? >>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>> >>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>> refer to the following document >>>>> >>>>> This Sun White Paper >>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>> should apply to Linux the same way as was agreed across platforms >>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>> >>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>> what happens if another part of the VM has a different idea about >>>>>> the value of _FILE_OFFSET_BITS? >>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>> So, >>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>> can't deal with large files. No more no less. >>>>> >>>>>> >>>>>> I saw this in the post to the Runtime alias: >>>>>> >>>>>> > Included runtime dev to see whether they have some idea to handle >>>>>> > the compilation choices. >>>>>> >>>>>> And I still have no idea what you're asking here? What compilation >>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>> to have mis-matched pieces and be unstable? >>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>> instead of just one object (ostream.o)?" is my main question I >>>>> originally tried to ask. >>>>> >>>>>> >>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>> change has too many open questions (for now)... >>>>>> >>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>> 2GB+ >>>>>> GC log or not. The conversation below didn't help me at all. >>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>> >>>>>> Dan >>>>>> >>>>>> >>>>>> >>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>> Since the changeset touched makefiles, I've included >>>>>>> build-dev at openjdk.java.net . >>>>>>> >>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Need reviews to catch RDP2. >>>>>>>> >>>>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>>>> Windows, and Solaris. >>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>>>> the compilation choices. >>>>>>>>> >>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>> sufficient. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>> Thank you, Mikael. >>>>>>>>>> >>>>>>>>>> Please see inline. >>>>>>>>>> >>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>> observation. >>>>>>>>>> >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>> Tao, >>>>>>>>>>> >>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>> 7ux bug >>>>>>>>>>>> >>>>>>>>>>>> webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>> >>>>>>>>>>>> changeset: >>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>> ostream.o >>>>>>>>>>>> >>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>> globally >>>>>>>>>>>> applicable? >>>>>>>>>>>> >>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>>>> however, >>>>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>>>> globally >>>>>>>>>>>> to Solaris. >>>>>>>>>>>> >>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>> four >>>>>>>>>>>> files >>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>> from >>>>>>>>>>>> setting >>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>> >>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>>>> Windows. >>>>>>>>>>> >>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>> work on >>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>> problem >>>>>>>>>>> there. >>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>> That's >>>>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>> >>>>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>> and >>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>> >>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>> ostream >>>>>>>>>>> >>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>> excludes >>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>> provides other benefits. >>>>>>>>>> >>>>>>>>>> This Sun White Paper >>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>> >>>>>>>>>> summarizes >>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>>>> fopen already supports large files. >>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>>>> of my problem" :) >>>>>>>>>>> >>>>>>>>>>> /Mikael >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> test: >>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>> verified on the >>>>>>>>>>>> following configurations. >>>>>>>>>>>> >>>>>>>>>>>> Linux * i586 >>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>> >>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>> >>>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Wed Jun 5 17:05:12 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Wed, 05 Jun 2013 10:05:12 -0700 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AF1C96.4070404@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> <51AF1C96.4070404@oracle.com> Message-ID: <51AF6FC8.5070501@oracle.com> Hi Stefan, He wanted a power of two. JohnC On 6/5/2013 4:10 AM, Stefan Karlsson wrote: > On 06/04/2013 09:00 PM, John Cuthbertson wrote: >> Hi Everyone, >> >> Here's a new webrev for this change: >> http://cr.openjdk.java.net/~johnc/8015237/webrev.1 > > Looks good. Thanks for doing all the cleanups. > >> >> Changes from before: >> * Refactored the code that loops over the buckets into its own routine. >> * Removed the commented out instrumentation (oops). >> * Changed the types to int to be consistent with the rest of >> symbolTable and allow removal of the casts. >> * Increase the number of buckets per claim to 32 based upon a verbal >> comment from John Coomes. > > Care to describe the reasoning why 32 should be better? > >> * Removed the additional worker ID parameter for the sake of peace >> and harmony. > > Thanks. > > StefanK > >> >> Testing: jprt. >> >> Thanks, >> >> JohnC >> >> On 5/24/2013 3:19 PM, John Cuthbertson wrote: >>> Hi Everyone, >>> >>> Can I have a couple of reviewers look over these changes - the >>> webrev is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >>> >>> Summary: >>> On some workloads we are seeing that the scan of the intern string >>> table (among others) can sometimes take quite a while. This showed >>> up on some FMW workloads with G1 where the scan of the string table >>> dominated the pause time for some pauses. G1 was particularly >>> affected since it doesn't do class unloading (and hence pruning of >>> the string table) except at full GCs. The solution was to change the >>> string table from being considered a single root task and treat >>> similarly to the Java thread stacks: each GC worker claims a given >>> number of buckets and scans the entries in those buckets. >>> >>> Testing >>> Kitchensink; jprt; GC test suite. With all collectors. >>> >>> Overhead: >>> Not real performance numbers but I did some measurement of the >>> synchronization overhead of using 1 GC worker thread. They are >>> summarized here: >>> >>> >>> 0-threads (ms) >>> 1-thread-chunked (ms) >>> Min 0.200 >>> 0.300 >>> Max 6.900 >>> 8.800 >>> Avg 0.658 >>> 0.794 >>> >>> >>> These were from 1 hour long runs of Kitchensink with around ~2800 >>> GCs in each run. >>> >>> Thanks, >>> >>> JohnC >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.helin at oracle.com Wed Jun 5 17:11:08 2013 From: erik.helin at oracle.com (Erik Helin) Date: Wed, 5 Jun 2013 19:11:08 +0200 Subject: RFR (S): 8015972: Refactor the sending of the object count after GC event Message-ID: <20130605171108.GA3096@ehelin-thinkpad> Hi all, this small change refactors the way the object count after GC trace event is sent. Webrev: http://cr.openjdk.java.net/~ehelin/8015972/webrev.00/ Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8015972 Testing: JPRT Thanks, Erik From stefan.karlsson at oracle.com Wed Jun 5 18:03:24 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 5 Jun 2013 11:03:24 -0700 (PDT) Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AF6FC8.5070501@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> <51AF1C96.4070404@oracle.com> <51AF6FC8.5070501@oracle.com> Message-ID: <51AF7D6C.7010808@oracle.com> On 6/5/13 7:05 PM, John Cuthbertson wrote: > Hi Stefan, > > He wanted a power of two. And the reason being? It won't help the cache alignment "issues" that Per talked about earlier, if that's the reason. The array will still not be aligned against a cache line size. Anyways, I'm happy with the change. StefanK > > JohnC > > On 6/5/2013 4:10 AM, Stefan Karlsson wrote: >> On 06/04/2013 09:00 PM, John Cuthbertson wrote: >>> Hi Everyone, >>> >>> Here's a new webrev for this change: >>> http://cr.openjdk.java.net/~johnc/8015237/webrev.1 >> >> Looks good. Thanks for doing all the cleanups. >> >>> >>> Changes from before: >>> * Refactored the code that loops over the buckets into its own routine. >>> * Removed the commented out instrumentation (oops). >>> * Changed the types to int to be consistent with the rest of >>> symbolTable and allow removal of the casts. >>> * Increase the number of buckets per claim to 32 based upon a verbal >>> comment from John Coomes. >> >> Care to describe the reasoning why 32 should be better? >> >>> * Removed the additional worker ID parameter for the sake of peace >>> and harmony. >> >> Thanks. >> >> StefanK >> >>> >>> Testing: jprt. >>> >>> Thanks, >>> >>> JohnC >>> >>> On 5/24/2013 3:19 PM, John Cuthbertson wrote: >>>> Hi Everyone, >>>> >>>> Can I have a couple of reviewers look over these changes - the >>>> webrev is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >>>> >>>> Summary: >>>> On some workloads we are seeing that the scan of the intern string >>>> table (among others) can sometimes take quite a while. This showed >>>> up on some FMW workloads with G1 where the scan of the string table >>>> dominated the pause time for some pauses. G1 was particularly >>>> affected since it doesn't do class unloading (and hence pruning of >>>> the string table) except at full GCs. The solution was to change >>>> the string table from being considered a single root task and treat >>>> similarly to the Java thread stacks: each GC worker claims a given >>>> number of buckets and scans the entries in those buckets. >>>> >>>> Testing >>>> Kitchensink; jprt; GC test suite. With all collectors. >>>> >>>> Overhead: >>>> Not real performance numbers but I did some measurement of the >>>> synchronization overhead of using 1 GC worker thread. They are >>>> summarized here: >>>> >>>> >>>> 0-threads (ms) >>>> 1-thread-chunked (ms) >>>> Min 0.200 >>>> 0.300 >>>> Max 6.900 >>>> 8.800 >>>> Avg 0.658 >>>> 0.794 >>>> >>>> >>>> These were from 1 hour long runs of Kitchensink with around ~2800 >>>> GCs in each run. >>>> >>>> Thanks, >>>> >>>> JohnC >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Wed Jun 5 18:34:15 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Wed, 05 Jun 2013 11:34:15 -0700 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AF7D6C.7010808@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> <51AF1C96.4070404@oracle.com> <51AF6FC8.5070501@oracle.com> <51AF7D6C.7010808@oracle.com> Message-ID: <51AF84A7.1010305@oracle.com> Hi Stefan, I think it was for aesthetics. 20 looked odd (even though it's even) :) I went with 32 because a lower number would increase the number of atomic operations. 128 seemed too high for the old default table size on the haswell system I was testing on. So 32 and 64 seemed to be the Goldilocks values. I chose 32. JohnC On 6/5/2013 11:03 AM, Stefan Karlsson wrote: > On 6/5/13 7:05 PM, John Cuthbertson wrote: >> Hi Stefan, >> >> He wanted a power of two. > > And the reason being? > > It won't help the cache alignment "issues" that Per talked about > earlier, if that's the reason. The array will still not be aligned > against a cache line size. > > Anyways, I'm happy with the change. > > StefanK >> >> JohnC >> >> On 6/5/2013 4:10 AM, Stefan Karlsson wrote: >>> On 06/04/2013 09:00 PM, John Cuthbertson wrote: >>>> Hi Everyone, >>>> >>>> Here's a new webrev for this change: >>>> http://cr.openjdk.java.net/~johnc/8015237/webrev.1 >>> >>> Looks good. Thanks for doing all the cleanups. >>> >>>> >>>> Changes from before: >>>> * Refactored the code that loops over the buckets into its own routine. >>>> * Removed the commented out instrumentation (oops). >>>> * Changed the types to int to be consistent with the rest of >>>> symbolTable and allow removal of the casts. >>>> * Increase the number of buckets per claim to 32 based upon a >>>> verbal comment from John Coomes. >>> >>> Care to describe the reasoning why 32 should be better? >>> >>>> * Removed the additional worker ID parameter for the sake of peace >>>> and harmony. >>> >>> Thanks. >>> >>> StefanK >>> >>>> >>>> Testing: jprt. >>>> >>>> Thanks, >>>> >>>> JohnC >>>> >>>> On 5/24/2013 3:19 PM, John Cuthbertson wrote: >>>>> Hi Everyone, >>>>> >>>>> Can I have a couple of reviewers look over these changes - the >>>>> webrev is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >>>>> >>>>> Summary: >>>>> On some workloads we are seeing that the scan of the intern string >>>>> table (among others) can sometimes take quite a while. This showed >>>>> up on some FMW workloads with G1 where the scan of the string >>>>> table dominated the pause time for some pauses. G1 was >>>>> particularly affected since it doesn't do class unloading (and >>>>> hence pruning of the string table) except at full GCs. The >>>>> solution was to change the string table from being considered a >>>>> single root task and treat similarly to the Java thread stacks: >>>>> each GC worker claims a given number of buckets and scans the >>>>> entries in those buckets. >>>>> >>>>> Testing >>>>> Kitchensink; jprt; GC test suite. With all collectors. >>>>> >>>>> Overhead: >>>>> Not real performance numbers but I did some measurement of the >>>>> synchronization overhead of using 1 GC worker thread. They are >>>>> summarized here: >>>>> >>>>> >>>>> 0-threads (ms) >>>>> 1-thread-chunked (ms) >>>>> Min 0.200 >>>>> 0.300 >>>>> Max 6.900 >>>>> 8.800 >>>>> Avg 0.658 >>>>> 0.794 >>>>> >>>>> >>>>> These were from 1 hour long runs of Kitchensink with around ~2800 >>>>> GCs in each run. >>>>> >>>>> Thanks, >>>>> >>>>> JohnC >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.karlsson at oracle.com Wed Jun 5 18:51:36 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 05 Jun 2013 20:51:36 +0200 Subject: RFR(S): 8015237: Parallelize string table scanning during strong root processing In-Reply-To: <51AF84A7.1010305@oracle.com> References: <519FE787.70408@oracle.com> <51AE3941.90609@oracle.com> <51AF1C96.4070404@oracle.com> <51AF6FC8.5070501@oracle.com> <51AF7D6C.7010808@oracle.com> <51AF84A7.1010305@oracle.com> Message-ID: <51AF88B8.7010208@oracle.com> On 6/5/13 8:34 PM, John Cuthbertson wrote: > Hi Stefan, > > I think it was for aesthetics. 20 looked odd (even though it's even) :) > > I went with 32 because a lower number would increase the number of > atomic operations. 128 seemed too high for the old default table size > on the haswell system I was testing on. So 32 and 64 seemed to be the > Goldilocks values. I chose 32. OK. Thanks. StefanK > > JohnC > > On 6/5/2013 11:03 AM, Stefan Karlsson wrote: >> On 6/5/13 7:05 PM, John Cuthbertson wrote: >>> Hi Stefan, >>> >>> He wanted a power of two. >> >> And the reason being? >> >> It won't help the cache alignment "issues" that Per talked about >> earlier, if that's the reason. The array will still not be aligned >> against a cache line size. >> >> Anyways, I'm happy with the change. >> >> StefanK >>> >>> JohnC >>> >>> On 6/5/2013 4:10 AM, Stefan Karlsson wrote: >>>> On 06/04/2013 09:00 PM, John Cuthbertson wrote: >>>>> Hi Everyone, >>>>> >>>>> Here's a new webrev for this change: >>>>> http://cr.openjdk.java.net/~johnc/8015237/webrev.1 >>>> >>>> Looks good. Thanks for doing all the cleanups. >>>> >>>>> >>>>> Changes from before: >>>>> * Refactored the code that loops over the buckets into its own >>>>> routine. >>>>> * Removed the commented out instrumentation (oops). >>>>> * Changed the types to int to be consistent with the rest of >>>>> symbolTable and allow removal of the casts. >>>>> * Increase the number of buckets per claim to 32 based upon a >>>>> verbal comment from John Coomes. >>>> >>>> Care to describe the reasoning why 32 should be better? >>>> >>>>> * Removed the additional worker ID parameter for the sake of peace >>>>> and harmony. >>>> >>>> Thanks. >>>> >>>> StefanK >>>> >>>>> >>>>> Testing: jprt. >>>>> >>>>> Thanks, >>>>> >>>>> JohnC >>>>> >>>>> On 5/24/2013 3:19 PM, John Cuthbertson wrote: >>>>>> Hi Everyone, >>>>>> >>>>>> Can I have a couple of reviewers look over these changes - the >>>>>> webrev is: http://cr.openjdk.java.net/~johnc/8015237/webrev.0/ >>>>>> >>>>>> Summary: >>>>>> On some workloads we are seeing that the scan of the intern >>>>>> string table (among others) can sometimes take quite a while. >>>>>> This showed up on some FMW workloads with G1 where the scan of >>>>>> the string table dominated the pause time for some pauses. G1 was >>>>>> particularly affected since it doesn't do class unloading (and >>>>>> hence pruning of the string table) except at full GCs. The >>>>>> solution was to change the string table from being considered a >>>>>> single root task and treat similarly to the Java thread stacks: >>>>>> each GC worker claims a given number of buckets and scans the >>>>>> entries in those buckets. >>>>>> >>>>>> Testing >>>>>> Kitchensink; jprt; GC test suite. With all collectors. >>>>>> >>>>>> Overhead: >>>>>> Not real performance numbers but I did some measurement of the >>>>>> synchronization overhead of using 1 GC worker thread. They are >>>>>> summarized here: >>>>>> >>>>>> >>>>>> 0-threads (ms) >>>>>> 1-thread-chunked (ms) >>>>>> Min 0.200 >>>>>> 0.300 >>>>>> Max 6.900 >>>>>> 8.800 >>>>>> Avg 0.658 >>>>>> 0.794 >>>>>> >>>>>> >>>>>> These were from 1 hour long runs of Kitchensink with around ~2800 >>>>>> GCs in each run. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> JohnC >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Wed Jun 5 20:18:52 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 13:18:52 -0700 Subject: Request for review (vs) - 8014546: MetaspaceAux print_metaspace_change() should print "used" after GC not capacity Message-ID: <51AF9D2C.3030005@oracle.com> As the summary says, print the used metaspace in print_metaspace_change() instead of capacity. The output used to be previous_used -> capacity (reserved) and now is previous_used -> used_after_GC (reserved) This makes the print_metaspace_change() more similar to the GC print_heap_change(). http://cr.openjdk.java.net/~jmasa/8014546/webrev.00/ Two lines changed. --- a/src/share/vm/memory/metaspace.cpp +++ b/src/share/vm/memory/metaspace.cpp @@ -2603,14 +2603,14 @@ "->" SIZE_FORMAT "(" SIZE_FORMAT ")", prev_metadata_used, - allocated_capacity_bytes(), + allocated_used_bytes(), reserved_in_bytes()); } else { gclog_or_tty->print(" " SIZE_FORMAT "K" "->" SIZE_FORMAT "K" "(" SIZE_FORMAT "K)", prev_metadata_used / K, - allocated_capacity_bytes() / K, + allocated_used_bytes() / K, reserved_in_bytes()/ K); } Thanks. Jon From jon.masamitsu at oracle.com Wed Jun 5 20:27:44 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 13:27:44 -0700 Subject: Request for review (s) - 8014546 :UseAdaptiveGCBoundary is broken Message-ID: <51AF9F40.6040204@oracle.com> A recent changeset effectively removed the initialization of performance counters used by the option UseAdaptiveGCBoundary This changeset is built on top of the one for 8014546 and the webrev comes in two parts. The fix plus the test http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ Some cleanups of the code http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ Thanks. Jon From jesper.wilhelmsson at oracle.com Wed Jun 5 21:50:38 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Wed, 05 Jun 2013 23:50:38 +0200 Subject: RFR (XS): 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 Message-ID: <51AFB2AE.7040604@oracle.com> Hi, Can I have a couple of reviews for this really minor fix? 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 A missing line break in the adaptive size verbose logging has been added. When looking at this I also found that an existing line break caused a slightly ugly verbose output when breaking a line causing a second line without a identifier at the beginning. I removed that line break as well in this change. webrev: http://cr.openjdk.java.net/~jwilhelm/8015903/hotspot-webrev/ Output before: AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 overflow: falseAdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 target_size: 1048576 Output after: AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 overflow: false AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 target_size: 1048576 Thanks, /Jesper From john.cuthbertson at oracle.com Wed Jun 5 22:00:48 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Wed, 5 Jun 2013 15:00:48 -0700 (PDT) Subject: Request for review (vs) - 8014546: MetaspaceAux print_metaspace_change() should print "used" after GC not capacity In-Reply-To: <51AF9D2C.3030005@oracle.com> References: <51AF9D2C.3030005@oracle.com> Message-ID: <51AFB510.7030006@oracle.com> Hi Jon, Code changes look good. JohnC On 6/5/2013 1:18 PM, Jon Masamitsu wrote: > As the summary says, print the used metaspace in > print_metaspace_change() instead > of capacity. The output used to be > > previous_used -> capacity (reserved) > > and now is > > previous_used -> used_after_GC (reserved) > > This makes the print_metaspace_change() more similar to the > GC print_heap_change(). > > http://cr.openjdk.java.net/~jmasa/8014546/webrev.00/ > > Two lines changed. > > --- a/src/share/vm/memory/metaspace.cpp > +++ b/src/share/vm/memory/metaspace.cpp > @@ -2603,14 +2603,14 @@ > "->" SIZE_FORMAT > "(" SIZE_FORMAT ")", > prev_metadata_used, > - allocated_capacity_bytes(), > + allocated_used_bytes(), > reserved_in_bytes()); > } else { > gclog_or_tty->print(" " SIZE_FORMAT "K" > "->" SIZE_FORMAT "K" > "(" SIZE_FORMAT "K)", > prev_metadata_used / K, > - allocated_capacity_bytes() / K, > + allocated_used_bytes() / K, > reserved_in_bytes()/ K); > } > > Thanks. > > Jon From jon.masamitsu at oracle.com Thu Jun 6 00:12:59 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 17:12:59 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <51A6AD6D.8010309@oracle.com> Message-ID: <51AFD40B.8020300@oracle.com> Hiroshi, I've looked at the parallel initial mark changeset so far. Looks good. Would you says that the FlexibleWorkerGang // Parallel initial mark task class CMSParInitialMarkTask: public CMSParMarkTask { FlexibleWorkGang* _workers; is unused above just as it is unnecessary in the constructor for CMSParRemarkTask? I'll look at the second changeset in a bit. Jon On 5/30/2013 10:56 AM, Hiroshi Yamauchi wrote: > Thanks, Jon. Please let me know when you know more. > > On Wed, May 29, 2013 at 6:37 PM, Jon Masamitsu wrote: >> Hiroshi, >> >> I'm still reviewing the changes but so far this looks >> very promising. I've patched you changes into a repository >> and started running a few tests. I've turned on >> >> -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways >> >> Thanks. >> >> Jon >> >> >> On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: >>> Hi, >>> >>> I'd like to have the following contributed if it makes sense. >>> >>> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >>> >>> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ >>> >>> that implements a parallel version of the initial mark phase of the >>> CMS collector. It's relatively a straightforward parallelization of >>> the existing single-threaded code. With the above patch, I see about >>> ~3-6x speedup in the initial mark pause times. >>> >>> 2) Now, here's a related issue and a suggested fix/patch for it: >>> >>> I see that the initial mark and remark pause times sometimes spike >>> with a large young generation. For example, under a 1 GB young gen / 3 >>> GB heap setting, they occasionally spike up to ~500 milliseconds from >>> the normal < 100 ms range, on my machine. As far as I can tell, this >>> happens when the eden is fairly occupied (> 700 MB full) and not >>> sufficiently divided up and the parallelism decreases (at the worst >>> case it becomes almost single-threaded.) >>> >>> Here's a suggested patch in a separate patch: >>> >>> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ >>> >>> that attempts to improve on this issue by implementing an alternative >>> way of dividing up the eden into chunks for an increased parallelism >>> (or better load balancing between the GC threads) for the young gen >>> scan portion of the remark phase (and the now-parallelized initial >>> mark phase.) It uses a CAS-based mechanism that samples the object >>> boundaries in the eden space on the slow allocation code paths (eg. at >>> the TLAB refill and large object allocation times) at all times. >>> >>> This approach is in contrast to the original mechanism that samples >>> object boundaries in the eden space asynchronously during the preclean >>> phase. I think the reason that the above issue happens is that when >>> the young generation is large, a large portion of the eden space could >>> get filled/allocated outside of the preclean phase (or a concurrent >>> collection) and the object boundaries do not get sampled >>> often/regularly enough. Also, it isn't very suited for the parallel >>> initial mark because the initial mark phase isn't preceded by the >>> preclean phase unlike the remark phase. According to the Dacapo >>> benchmarks, this alternative sampling mechanism does not have >>> noticeable runtime overhead despite it is engaged at all times. >>> >>> With this patch, I see that the (parallel) initial mark and remark >>> pause times stay below 100 ms (no spikes) under the same setting. >>> >>> Both of these features/flags are disabled by default. You're welcome >>> to handle the two patches separately. >>> >>> Thanks, >>> Hiroshi >> From jon.masamitsu at oracle.com Thu Jun 6 00:23:54 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 17:23:54 -0700 Subject: Request for review (vs) - 8014546: MetaspaceAux print_metaspace_change() should print "used" after GC not capacity In-Reply-To: <51AFB510.7030006@oracle.com> References: <51AF9D2C.3030005@oracle.com> <51AFB510.7030006@oracle.com> Message-ID: <51AFD69A.6000100@oracle.com> Thanks. On 6/5/2013 3:00 PM, John Cuthbertson wrote: > Hi Jon, > > Code changes look good. > > JohnC > > On 6/5/2013 1:18 PM, Jon Masamitsu wrote: >> As the summary says, print the used metaspace in >> print_metaspace_change() instead >> of capacity. The output used to be >> >> previous_used -> capacity (reserved) >> >> and now is >> >> previous_used -> used_after_GC (reserved) >> >> This makes the print_metaspace_change() more similar to the >> GC print_heap_change(). >> >> http://cr.openjdk.java.net/~jmasa/8014546/webrev.00/ >> >> Two lines changed. >> >> --- a/src/share/vm/memory/metaspace.cpp >> +++ b/src/share/vm/memory/metaspace.cpp >> @@ -2603,14 +2603,14 @@ >> "->" SIZE_FORMAT >> "(" SIZE_FORMAT ")", >> prev_metadata_used, >> - allocated_capacity_bytes(), >> + allocated_used_bytes(), >> reserved_in_bytes()); >> } else { >> gclog_or_tty->print(" " SIZE_FORMAT "K" >> "->" SIZE_FORMAT "K" >> "(" SIZE_FORMAT "K)", >> prev_metadata_used / K, >> - allocated_capacity_bytes() / K, >> + allocated_used_bytes() / K, >> reserved_in_bytes()/ K); >> } >> >> Thanks. >> >> Jon > From jon.masamitsu at oracle.com Thu Jun 6 00:25:37 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 17:25:37 -0700 Subject: Request for review (s) - 8015851 :UseAdaptiveGCBoundary is broken In-Reply-To: <51AF9F40.6040204@oracle.com> References: <51AF9F40.6040204@oracle.com> Message-ID: <51AFD701.90400@oracle.com> Fixed the CR # in the subject. On 6/5/2013 1:27 PM, Jon Masamitsu wrote: > A recent changeset effectively removed the initialization of > performance counters used by the option UseAdaptiveGCBoundary > > This changeset is built on top of the one for 8014546 and the > webrev comes in two parts. > > The fix plus the test > > http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ > > Some cleanups of the code > > http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ > > Thanks. > > Jon From jon.masamitsu at oracle.com Thu Jun 6 02:41:01 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 19:41:01 -0700 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51AF4576.3080203@oracle.com> References: <51AF4576.3080203@oracle.com> Message-ID: <51AFF6BD.60501@oracle.com> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: > Hi, > > Can I have some reviews of this small fix to the Metaspace memory > allocation path. > > Problem: > When a Metaspace allocation request cannot be satisfied by the current > chunk the chunk is retired and a new chunk is requested. This causes > whatever is left in the chunk to be effectively leaked. > > Suggested fix: > Put the remaining memory in each chunk on the Metablock freelist so it > can be used to satisfy future allocations. > > Possible addition: > When allocating from the block free list, use > FreeBlockDictionary::atLeast instead of > FreeBlockDictionary::exactly and split the Metablock if > it's large enough. > > One might argue that this increases the fragmentation of the memory on > the block free list but I think that we primarily want to use the > block free list for small allocations and allocate from chunks for > large allocations. > > Webrev: > Only fix: > http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ The "Only fix" looks good. Did you test with metaspace_slow_verify=true? > > Incremental webrev for splitting blocks: > http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ Change looks good. Did you do any long running tests with the block splitting? Such as 24hours with kitchensink? Something that would reuse Metablocks so that we can see if we are fragmenting instead of reusing? Jon > > Bug links: > https://jbs.oracle.com/bugs/browse/JDK-8009561 > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 > > Thanks > /Mikael From jon.masamitsu at oracle.com Thu Jun 6 04:07:31 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 05 Jun 2013 21:07:31 -0700 Subject: CMS parallel initial mark In-Reply-To: References: Message-ID: <51B00B03.1020503@oracle.com> Hiroshi, For the sampling change. I appreciate that you allow for reverting to the old behavior of sampling during precleaning but am curious about whether you've seen an occasion where it was preferable. http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.hpp.frames.html 739 // This is meant to be a boolean flag, but jbyte for CAS. 740 jbyte _eden_chunk_sampling_active; Other than the card table I'm used to seeing the atomic operations on word sized variables. Would jint work as well and be simpler to think about? Maybe more later. Jon On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: > Hi, > > I'd like to have the following contributed if it makes sense. > > 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ > > that implements a parallel version of the initial mark phase of the > CMS collector. It's relatively a straightforward parallelization of > the existing single-threaded code. With the above patch, I see about > ~3-6x speedup in the initial mark pause times. > > 2) Now, here's a related issue and a suggested fix/patch for it: > > I see that the initial mark and remark pause times sometimes spike > with a large young generation. For example, under a 1 GB young gen / 3 > GB heap setting, they occasionally spike up to ~500 milliseconds from > the normal < 100 ms range, on my machine. As far as I can tell, this > happens when the eden is fairly occupied (> 700 MB full) and not > sufficiently divided up and the parallelism decreases (at the worst > case it becomes almost single-threaded.) > > Here's a suggested patch in a separate patch: > > http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ > > that attempts to improve on this issue by implementing an alternative > way of dividing up the eden into chunks for an increased parallelism > (or better load balancing between the GC threads) for the young gen > scan portion of the remark phase (and the now-parallelized initial > mark phase.) It uses a CAS-based mechanism that samples the object > boundaries in the eden space on the slow allocation code paths (eg. at > the TLAB refill and large object allocation times) at all times. > > This approach is in contrast to the original mechanism that samples > object boundaries in the eden space asynchronously during the preclean > phase. I think the reason that the above issue happens is that when > the young generation is large, a large portion of the eden space could > get filled/allocated outside of the preclean phase (or a concurrent > collection) and the object boundaries do not get sampled > often/regularly enough. Also, it isn't very suited for the parallel > initial mark because the initial mark phase isn't preceded by the > preclean phase unlike the remark phase. According to the Dacapo > benchmarks, this alternative sampling mechanism does not have > noticeable runtime overhead despite it is engaged at all times. > > With this patch, I see that the (parallel) initial mark and remark > pause times stay below 100 ms (no spikes) under the same setting. > > Both of these features/flags are disabled by default. You're welcome > to handle the two patches separately. > > Thanks, > Hiroshi From thomas.schatzl at oracle.com Thu Jun 6 08:46:43 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Jun 2013 10:46:43 +0200 Subject: Request for review (vs) - 8014546: MetaspaceAux print_metaspace_change() should print "used" after GC not capacity In-Reply-To: <51AF9D2C.3030005@oracle.com> References: <51AF9D2C.3030005@oracle.com> Message-ID: <1370508403.2590.6.camel@cirrus> Hi, On Wed, 2013-06-05 at 13:18 -0700, Jon Masamitsu wrote: > As the summary says, print the used metaspace in > print_metaspace_change() instead > of capacity. The output used to be > > previous_used -> capacity (reserved) > > and now is > > previous_used -> used_after_GC (reserved) > > This makes the print_metaspace_change() more similar to the > GC print_heap_change(). > > http://cr.openjdk.java.net/~jmasa/8014546/webrev.00/ > > Two lines changed. Looks good :) Thomas From mikael.gerdin at oracle.com Thu Jun 6 09:10:24 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Thu, 06 Jun 2013 11:10:24 +0200 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AF6211.8020607@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> <51AF6211.8020607@oracle.com> Message-ID: <51B05200.3000202@oracle.com> Dan, On 2013-06-05 18:06, Daniel D. Daugherty wrote: > On 6/5/13 10:02 AM, Mikael Gerdin wrote: >> Tao, >> >> On 2013-06-05 17:48, Tao Mao wrote: >>> Thank you for comments. One thing I want to point out is that the >>> current change has not touched on Windows code. >>> >>> Please see inline. >>> >>> Tao >>> >>> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>>> >>>> >>>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>>> OK, based on the largefiles.pdf write-up, your use of >>>>> _FILE_OFFSET_BITS=64 >>>>> is going to cause ostream.o to bind to various 64-bit versions of some >>>>> functions. Based on my very hazy memory of my days in file system >>>>> code, >>>>> this binding is going to affect ostream.o only unless ostream.o is >>>>> writing to some file with the foo64() function and some other code is >>>>> also writing to the same file at the same time with foo(). However, >>>>> if we >>>>> have two different functions writing to the same open file at the same >>>>> time, then we have bigger issues. :-) >>>>> >>>>> I'm good with these changes now. I agree that solving the problem of >>>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>>> solved right now. >>>> >>>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 when >>>> compiling ostream.cpp will effectively cause the headers to swap out >>>> the implementations of the f[open,tell,seek] to 64 bit versions for >>>> all headers that are included and inlined in ostream.cpp. >>>> Other parts of the code using the same headers will see different >>>> versions of the functions with different parameters due to off_t >>>> changing sizes. >>> The change is currently effective for Linux and Solaris if you look at >>> the file directories. Nothing changed for Windows and BSD, as they don't >>> need such change. >> >> Right. >> But if you use my suggested approach you would need to change calls to >> fopen() in ostream.cpp to fopen_pd where >> >> if (linux || solaris) && 32bit >> #define fopen_pd fopen64 >> else >> #define fopen_pd fopen >> >>>> >>>> I think that what we should do is to use the "Transitional compilation >>>> environment" detailed in ?2.6.2 in largefiles.pdf and change the calls >>>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>>> directly. I feel this is especially important at this late stage in >>>> the release to make an explicit change instead of setting a #define >>>> which has propagating side-effects. >>> How do you see "propagating side-effects" and to where? >> >> _FILE_OFFSET_BITS=64 changes the definition of fopen for every file >> including stdio.h. >> >> It's confusing when a call to "fopen()" in one file calls the 64 bit >> version and in other files it doesn't. >> >> How will this work with precompiled headers? Which version of fopen >> will be in the precompiled header file? > > You're thinking this is like a "#define" and it is not. > The ostream.o binary will bind to the foo64() version of > functions and not the foo() version of functions. This > is not the same as using a #define. I fail to see how: #ifndef __USE_FILE_OFFSET64 /* Open a file and create a new stream for it. This function is a possible cancellation point and therefore not marked with __THROW. */ extern FILE *fopen (__const char *__restrict __filename, __const char *__restrict __modes) __wur; /* Open a file, replacing an existing stream with it. This function is a possible cancellation point and therefore not marked with __THROW. */ extern FILE *freopen (__const char *__restrict __filename, __const char *__restrict __modes, FILE *__restrict __stream) __wur; #else # ifdef __REDIRECT extern FILE *__REDIRECT (fopen, (__const char *__restrict __filename, __const char *__restrict __modes), fopen64) __wur; extern FILE *__REDIRECT (freopen, (__const char *__restrict __filename, __const char *__restrict __modes, FILE *__restrict __stream), freopen64) __wur; # else # define fopen fopen64 # define freopen freopen64 # endif #endif is not (at least in some cases) #define-based. It is my understanding that if _FILE_OFFSET_BITS=64 then stdio.h will "#define fopen fopen64" * I think that it's confusing if we only do that for ostream.cpp * If we ever add a "fopen()" call to a .hpp file which is included from ostream.ccp then the caller of fopen will have different behavior in based on which file it's called from. I've written a small example to detail what kind of problems we could get: https://gist.github.com/anonymous/b690f753401504f1f096 If ostream.cpp is the first to call into Logger::log we will get one behavior, if another file does we get the default behavior. Hope this makes my reasons for arguing about this a bit clearer. /Mikael > > Dan > > >> >>>> >>>> As Tao mentioned this will require us to handle the fact that there is >>>> no fopen64() call on Windows, that the CRT fopen() already seems to >>>> handle large files and that ftell64() and fseek64() have slightly >>>> different names on Windows. I don't think this is a large hurdle and I >>>> think we know how to solve this problem. >>> As I said, nothing was changed for Windows code. >> >> No, but to be consistent you'd need to update the ftell* fseek* to use >> 64 bit versions, right? >> >> /Mikael >> >>>> >>>> /Mikael >>>> >>>>> >>>>> Dan >>>>> >>>>> >>>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>>> Thank you for review, Dan. >>>>>> >>>>>> I'll try to answer as much as I can. Please see inline. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> Tao, >>>>>>> >>>>>>> I think the lack of response to this review request is the >>>>>>> absolutely >>>>>>> strange nature of these changes. And I thought I put out some weird >>>>>>> code reviews... :-) >>>>>>> >>>>>>> make/linux/makefiles/vm.make >>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>>> obvious in this webrev about what this will mean so I took a >>>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>>> >>>>>>> Must be in the source files somewhere, but I can't find any >>>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>>> >>>>>>> make/solaris/makefiles/vm.make >>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>>> >>>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>>> understand what you're doing here. >>>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>>> os::readdir() function will use the safer, multi-threaded >>>>>>> version of readdir_r(). Seems fine to me. >>>>>>> >>>>>>> Here's what I need to know: >>>>>>> >>>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>>> ostream.{c,h}pp? >>>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>>> >>>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>>> refer to the following document >>>>>> >>>>>> This Sun White Paper >>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>>> should apply to Linux the same way as was agreed across platforms >>>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>>> >>>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>>> what happens if another part of the VM has a different idea about >>>>>>> the value of _FILE_OFFSET_BITS? >>>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>>> So, >>>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>>> can't deal with large files. No more no less. >>>>>> >>>>>>> >>>>>>> I saw this in the post to the Runtime alias: >>>>>>> >>>>>>> > Included runtime dev to see whether they have some idea to handle >>>>>>> > the compilation choices. >>>>>>> >>>>>>> And I still have no idea what you're asking here? What compilation >>>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>>> to have mis-matched pieces and be unstable? >>>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>>> instead of just one object (ostream.o)?" is my main question I >>>>>> originally tried to ask. >>>>>> >>>>>>> >>>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>>> change has too many open questions (for now)... >>>>>>> >>>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>>> 2GB+ >>>>>>> GC log or not. The conversation below didn't help me at all. >>>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>>> Since the changeset touched makefiles, I've included >>>>>>>> build-dev at openjdk.java.net . >>>>>>>> >>>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Need reviews to catch RDP2. >>>>>>>>> >>>>>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>>>>> Windows, and Solaris. >>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>>>>> the compilation choices. >>>>>>>>>> >>>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>>> sufficient. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>>> Thank you, Mikael. >>>>>>>>>>> >>>>>>>>>>> Please see inline. >>>>>>>>>>> >>>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>>> observation. >>>>>>>>>>> >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>>> Tao, >>>>>>>>>>>> >>>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>>> 7ux bug >>>>>>>>>>>>> >>>>>>>>>>>>> webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>>> >>>>>>>>>>>>> changeset: >>>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>>> ostream.o >>>>>>>>>>>>> >>>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>>> globally >>>>>>>>>>>>> applicable? >>>>>>>>>>>>> >>>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>>>>> however, >>>>>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>>>>> globally >>>>>>>>>>>>> to Solaris. >>>>>>>>>>>>> >>>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>>> four >>>>>>>>>>>>> files >>>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>>> from >>>>>>>>>>>>> setting >>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>>> >>>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>>>>> Windows. >>>>>>>>>>>> >>>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>>> work on >>>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>>> problem >>>>>>>>>>>> there. >>>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>>> That's >>>>>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>>> >>>>>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>>> and >>>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>>> >>>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>>> ostream >>>>>>>>>>>> >>>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>>> excludes >>>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>>> provides other benefits. >>>>>>>>>>> >>>>>>>>>>> This Sun White Paper >>>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>>> >>>>>>>>>>> summarizes >>>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>>>>> fopen already supports large files. >>>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>>>>> of my problem" :) >>>>>>>>>>>> >>>>>>>>>>>> /Mikael >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> test: >>>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>>> verified on the >>>>>>>>>>>>> following configurations. >>>>>>>>>>>>> >>>>>>>>>>>>> Linux * i586 >>>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>>> >>>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>>> >>>>>>> >>>>> > From mikael.gerdin at oracle.com Thu Jun 6 09:20:01 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Thu, 06 Jun 2013 11:20:01 +0200 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51AF6F63.3010906@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> <51AF6F63.3010906@oracle.com> Message-ID: <51B05441.60402@oracle.com> Tao, On 2013-06-05 19:03, Tao Mao wrote: > Hi Mikael, > > Let me explain the changeset first before I go into your comments. Ok. > > (1) make/linux/makefiles/vm.make > > # Large File Support > ifneq ($(LP64), 1) > CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 > endif # ifneq ($(LP64), 1) > > For "Linux" code, set _FILE_OFFSET_BITS=64 "only" to ostream.o (i.e. > ostream.{c,h}pp) in order to bind all foo() functions to foo64() > functions when linking libraries. This will enable functions in > ostream.{c,h}pp to deal with large files while all other files won't > have the ability. And that's what we want at this point. This is slightly incorrect. Your change has no thing to do with ostream.hpp other than the fact that the version of ostream.hpp which is preprocessed into ostream.cpp will have different preprocessor behavior. In reality, your change will affect all the header files transitively included from ostream.cpp during its compilation. I agree that in effect this will cause references to fopen() in ostream.cpp to become "call fopen64" in the .o file. > > (2) make/solaris/makefiles/vm.make > > # Large File Support > ifneq ($(LP64), 1) > CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 > endif # ifneq ($(LP64), 1) > > Similarly, for "Solaris" code, set _FILE_OFFSET_BITS=64 "only" to > ostream.o (i.e. ostream.{c,h}pp) in order to bind all foo() functions to > foo64() functions when linking libraries. This will enable functions in > ostream.{c,h}pp to deal with large files while all other files won't > have the ability. > > (1) & (2) altogether set _FILE_OFFSET_BITS=64 "only" to ostream.o in > "Linux" and "Solaris" code base and have not changed Windows and BSD > code, as they can handle large files currently. No need to change. > > Remember the current implementation has a limited affected range (i.e. > ostream.o for Linux and Solaris). No more no less. > > (3) src/os/solaris/vm/os_solaris.inline.hpp > > #if defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 > dirent* p; > int status; > > if((status = ::readdir_r(dirp, dbuf, &p)) != 0) { > errno = status; > return NULL; > } else > return p; > #else // defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 > return ::readdir_r(dirp, dbuf); > #endif // defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 > > > This piece of code handles some exception I need to fix. It is not what > I want to change but rather what I have to change. > > Please see inline. > > Thanks. > Tao > > > On 6/5/13 9:02 AM, Mikael Gerdin wrote: >> Tao, >> >> On 2013-06-05 17:48, Tao Mao wrote: >>> Thank you for comments. One thing I want to point out is that the >>> current change has not touched on Windows code. >>> >>> Please see inline. >>> >>> Tao >>> >>> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>>> >>>> >>>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>>> OK, based on the largefiles.pdf write-up, your use of >>>>> _FILE_OFFSET_BITS=64 >>>>> is going to cause ostream.o to bind to various 64-bit versions of some >>>>> functions. Based on my very hazy memory of my days in file system >>>>> code, >>>>> this binding is going to affect ostream.o only unless ostream.o is >>>>> writing to some file with the foo64() function and some other code is >>>>> also writing to the same file at the same time with foo(). However, >>>>> if we >>>>> have two different functions writing to the same open file at the same >>>>> time, then we have bigger issues. :-) >>>>> >>>>> I'm good with these changes now. I agree that solving the problem of >>>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>>> solved right now. >>>> >>>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 when >>>> compiling ostream.cpp will effectively cause the headers to swap out >>>> the implementations of the f[open,tell,seek] to 64 bit versions for >>>> all headers that are included and inlined in ostream.cpp. >>>> Other parts of the code using the same headers will see different >>>> versions of the functions with different parameters due to off_t >>>> changing sizes. >>> The change is currently effective for Linux and Solaris if you look at >>> the file directories. Nothing changed for Windows and BSD, as they don't >>> need such change. >> >> Right. >> But if you use my suggested approach you would need to change calls to >> fopen() in ostream.cpp to fopen_pd where >> >> if (linux || solaris) && 32bit >> #define fopen_pd fopen64 >> else >> #define fopen_pd fopen > I saw what you suggested. > > But, what's the benefit of doing so, or, what's the severe problem about > the current changeset? I don't see that one's better than another. Because I want to avoid the situation I described in my mail to Dan. I wrote a small code example to describe the situation I want to avoid: https://gist.github.com/anonymous/b690f753401504f1f096 I know that we currently don't have calls to fopen() from .hpp files in the VM but if anyone ever was to add one strange, hard to track down bugs will happen. > >> >>>> >>>> I think that what we should do is to use the "Transitional compilation >>>> environment" detailed in ?2.6.2 in largefiles.pdf and change the calls >>>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>>> directly. I feel this is especially important at this late stage in >>>> the release to make an explicit change instead of setting a #define >>>> which has propagating side-effects. >>> How do you see "propagating side-effects" and to where? >> >> _FILE_OFFSET_BITS=64 changes the definition of fopen for every file >> including stdio.h. > The current implementation has a limited affected range (i.e. ostream.o > for Linux and Solaris). No more no less. > > How do you see this flag will affect every file? Right, I meant that it will transitively affect all files included from ostream.cpp, as I mentioned above. /Mikael >> >> It's confusing when a call to "fopen()" in one file calls the 64 bit >> version and in other files it doesn't. >> >> How will this work with precompiled headers? Which version of fopen >> will be in the precompiled header file? >> >>>> >>>> As Tao mentioned this will require us to handle the fact that there is >>>> no fopen64() call on Windows, that the CRT fopen() already seems to >>>> handle large files and that ftell64() and fseek64() have slightly >>>> different names on Windows. I don't think this is a large hurdle and I >>>> think we know how to solve this problem. >>> As I said, nothing was changed for Windows code. >> >> No, but to be consistent you'd need to update the ftell* fseek* to use >> 64 bit versions, right? >> >> /Mikael >> >>>> >>>> /Mikael >>>> >>>>> >>>>> Dan >>>>> >>>>> >>>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>>> Thank you for review, Dan. >>>>>> >>>>>> I'll try to answer as much as I can. Please see inline. >>>>>> >>>>>> Thanks. >>>>>> Tao >>>>>> >>>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>> >>>>>>> Tao, >>>>>>> >>>>>>> I think the lack of response to this review request is the >>>>>>> absolutely >>>>>>> strange nature of these changes. And I thought I put out some weird >>>>>>> code reviews... :-) >>>>>>> >>>>>>> make/linux/makefiles/vm.make >>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>>> obvious in this webrev about what this will mean so I took a >>>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>>> >>>>>>> Must be in the source files somewhere, but I can't find any >>>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>>> >>>>>>> make/solaris/makefiles/vm.make >>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>>> >>>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>>> understand what you're doing here. >>>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>>> os::readdir() function will use the safer, multi-threaded >>>>>>> version of readdir_r(). Seems fine to me. >>>>>>> >>>>>>> Here's what I need to know: >>>>>>> >>>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>>> ostream.{c,h}pp? >>>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>>> >>>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>>> refer to the following document >>>>>> >>>>>> This Sun White Paper >>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>>> should apply to Linux the same way as was agreed across platforms >>>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>>> >>>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>>> what happens if another part of the VM has a different idea about >>>>>>> the value of _FILE_OFFSET_BITS? >>>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>>> So, >>>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>>> can't deal with large files. No more no less. >>>>>> >>>>>>> >>>>>>> I saw this in the post to the Runtime alias: >>>>>>> >>>>>>> > Included runtime dev to see whether they have some idea to handle >>>>>>> > the compilation choices. >>>>>>> >>>>>>> And I still have no idea what you're asking here? What compilation >>>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>>> to have mis-matched pieces and be unstable? >>>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>>> instead of just one object (ostream.o)?" is my main question I >>>>>> originally tried to ask. >>>>>> >>>>>>> >>>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>>> change has too many open questions (for now)... >>>>>>> >>>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>>> 2GB+ >>>>>>> GC log or not. The conversation below didn't help me at all. >>>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>>> Since the changeset touched makefiles, I've included >>>>>>>> build-dev at openjdk.java.net . >>>>>>>> >>>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Need reviews to catch RDP2. >>>>>>>>> >>>>>>>>> The current webrev is a working solution to all platforms, Linux, >>>>>>>>> Windows, and Solaris. >>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>>> Included runtime dev to see whether they have some idea to handle >>>>>>>>>> the compilation choices. >>>>>>>>>> >>>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>>> sufficient. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>>> Thank you, Mikael. >>>>>>>>>>> >>>>>>>>>>> Please see inline. >>>>>>>>>>> >>>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>>> observation. >>>>>>>>>>> >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>>> Tao, >>>>>>>>>>>> >>>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>>> 7ux bug >>>>>>>>>>>>> >>>>>>>>>>>>> webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>>> >>>>>>>>>>>>> changeset: >>>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>>> ostream.o >>>>>>>>>>>>> >>>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>>> globally >>>>>>>>>>>>> applicable? >>>>>>>>>>>>> >>>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works fine; >>>>>>>>>>>>> however, >>>>>>>>>>>>> there are at least five code conflicts if introducing the flag >>>>>>>>>>>>> globally >>>>>>>>>>>>> to Solaris. >>>>>>>>>>>>> >>>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>>> four >>>>>>>>>>>>> files >>>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>>> from >>>>>>>>>>>>> setting >>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>>> >>>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>>> I haven't found any clean solution for solving this problem on >>>>>>>>>>>>> Windows. >>>>>>>>>>>> >>>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>>> work on >>>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>>> problem >>>>>>>>>>>> there. >>>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>>> That's >>>>>>>>>>> why I chose to apply the flag only to the files (ostream.cpp and >>>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>>> >>>>>>>>>>>> Instead I suggest that you use the compatibility API described >>>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>>> and >>>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>>> >>>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and has >>>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>>> ostream >>>>>>>>>>>> >>>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>>> resembles the changes which will be needed for Windows anyway. >>>>>>>>>>>> Those changes would consist of changing calls to ftell/fseek to >>>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on Solaris/Linux. >>>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>>> excludes >>>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>>> provides other benefits. >>>>>>>>>>> >>>>>>>>>>> This Sun White Paper >>>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>>> >>>>>>>>>>> summarizes >>>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Since there is no fopen64 on Windows it seems that the default >>>>>>>>>>>> fopen already supports large files. >>>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows can >>>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved "half >>>>>>>>>>> of my problem" :) >>>>>>>>>>>> >>>>>>>>>>>> /Mikael >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> test: >>>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>>> verified on the >>>>>>>>>>>>> following configurations. >>>>>>>>>>>>> >>>>>>>>>>>>> Linux * i586 >>>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>>> >>>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>>> >>>>>>> >>>>> From mikael.gerdin at oracle.com Thu Jun 6 09:22:58 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Thu, 06 Jun 2013 11:22:58 +0200 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51AFF6BD.60501@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> Message-ID: <51B054F2.7000807@oracle.com> Jon, On 2013-06-06 04:41, Jon Masamitsu wrote: > > On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >> Hi, >> >> Can I have some reviews of this small fix to the Metaspace memory >> allocation path. >> >> Problem: >> When a Metaspace allocation request cannot be satisfied by the current >> chunk the chunk is retired and a new chunk is requested. This causes >> whatever is left in the chunk to be effectively leaked. >> >> Suggested fix: >> Put the remaining memory in each chunk on the Metablock freelist so it >> can be used to satisfy future allocations. >> >> Possible addition: >> When allocating from the block free list, use >> FreeBlockDictionary::atLeast instead of >> FreeBlockDictionary::exactly and split the Metablock if >> it's large enough. >> >> One might argue that this increases the fragmentation of the memory on >> the block free list but I think that we primarily want to use the >> block free list for small allocations and allocate from chunks for >> large allocations. >> >> Webrev: >> Only fix: >> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ > > The "Only fix" looks good. Did you test with metaspace_slow_verify=true? > >> >> Incremental webrev for splitting blocks: >> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ > > Change looks good. > > Did you do any long running tests with the block splitting? Such as > 24hours with kitchensink? Something that would reuse Metablocks > so that we can see if we are fragmenting instead of reusing? > I did some runs earlier but I don't have any data from them. I can try to get an instrumented build together and run KS over the weekend. /Mikael > Jon > > >> >> Bug links: >> https://jbs.oracle.com/bugs/browse/JDK-8009561 >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >> >> Thanks >> /Mikael > From daniel.daugherty at oracle.com Thu Jun 6 13:43:03 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 06 Jun 2013 07:43:03 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51B05200.3000202@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> <51AF6211.8020607@oracle.com> <51B05200.3000202@oracle.com> Message-ID: <51B091E7.4010300@oracle.com> On 6/6/13 3:10 AM, Mikael Gerdin wrote: > Dan, > > On 2013-06-05 18:06, Daniel D. Daugherty wrote: >> On 6/5/13 10:02 AM, Mikael Gerdin wrote: >>> Tao, >>> >>> On 2013-06-05 17:48, Tao Mao wrote: >>>> Thank you for comments. One thing I want to point out is that the >>>> current change has not touched on Windows code. >>>> >>>> Please see inline. >>>> >>>> Tao >>>> >>>> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>>>> >>>>> >>>>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>>>> OK, based on the largefiles.pdf write-up, your use of >>>>>> _FILE_OFFSET_BITS=64 >>>>>> is going to cause ostream.o to bind to various 64-bit versions of >>>>>> some >>>>>> functions. Based on my very hazy memory of my days in file system >>>>>> code, >>>>>> this binding is going to affect ostream.o only unless ostream.o is >>>>>> writing to some file with the foo64() function and some other >>>>>> code is >>>>>> also writing to the same file at the same time with foo(). However, >>>>>> if we >>>>>> have two different functions writing to the same open file at the >>>>>> same >>>>>> time, then we have bigger issues. :-) >>>>>> >>>>>> I'm good with these changes now. I agree that solving the problem of >>>>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>>>> solved right now. >>>>> >>>>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 >>>>> when >>>>> compiling ostream.cpp will effectively cause the headers to swap out >>>>> the implementations of the f[open,tell,seek] to 64 bit versions for >>>>> all headers that are included and inlined in ostream.cpp. >>>>> Other parts of the code using the same headers will see different >>>>> versions of the functions with different parameters due to off_t >>>>> changing sizes. >>>> The change is currently effective for Linux and Solaris if you look at >>>> the file directories. Nothing changed for Windows and BSD, as they >>>> don't >>>> need such change. >>> >>> Right. >>> But if you use my suggested approach you would need to change calls to >>> fopen() in ostream.cpp to fopen_pd where >>> >>> if (linux || solaris) && 32bit >>> #define fopen_pd fopen64 >>> else >>> #define fopen_pd fopen >>> >>>>> >>>>> I think that what we should do is to use the "Transitional >>>>> compilation >>>>> environment" detailed in ?2.6.2 in largefiles.pdf and change the >>>>> calls >>>>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>>>> directly. I feel this is especially important at this late stage in >>>>> the release to make an explicit change instead of setting a #define >>>>> which has propagating side-effects. >>>> How do you see "propagating side-effects" and to where? >>> >>> _FILE_OFFSET_BITS=64 changes the definition of fopen for every file >>> including stdio.h. >>> >>> It's confusing when a call to "fopen()" in one file calls the 64 bit >>> version and in other files it doesn't. >>> >>> How will this work with precompiled headers? Which version of fopen >>> will be in the precompiled header file? >> >> You're thinking this is like a "#define" and it is not. >> The ostream.o binary will bind to the foo64() version of >> functions and not the foo() version of functions. This >> is not the same as using a #define. > > I fail to see how: > > #ifndef __USE_FILE_OFFSET64 > /* Open a file and create a new stream for it. > > This function is a possible cancellation point and therefore not > marked with __THROW. */ > extern FILE *fopen (__const char *__restrict __filename, > __const char *__restrict __modes) __wur; > /* Open a file, replacing an existing stream with it. > > This function is a possible cancellation point and therefore not > marked with __THROW. */ > extern FILE *freopen (__const char *__restrict __filename, > __const char *__restrict __modes, > FILE *__restrict __stream) __wur; > #else > # ifdef __REDIRECT > extern FILE *__REDIRECT (fopen, (__const char *__restrict __filename, > __const char *__restrict __modes), fopen64) > > __wur; > extern FILE *__REDIRECT (freopen, (__const char *__restrict __filename, > __const char *__restrict __modes, FILE *__restrict __stream), > freopen64) > __wur; > # else > # define fopen fopen64 > # define freopen freopen64 > # endif > #endif > > is not (at least in some cases) #define-based. Interesting. I'm guessing that the above is from stdio.h that is not Solaris. Here's the relevant part of the Solaris version: #if defined(__PRAGMA_REDEFINE_EXTNAME) /* large file compilation environment setup */ #if !defined(_LP64) && _FILE_OFFSET_BITS == 64 #pragma redefine_extname fopen fopen64 #pragma redefine_extname freopen freopen64 #pragma redefine_extname tmpfile tmpfile64 #pragma redefine_extname fgetpos fgetpos64 #pragma redefine_extname fsetpos fsetpos64 #pragma redefine_extname fseeko fseeko64 #pragma redefine_extname ftello ftello64 #endif /* !defined(_LP64) && _FILE_OFFSET_BITS == 64 */ /* In the LP64 compilation environment, all APIs are already large file */ #if defined(_LP64) && defined(_LARGEFILE64_SOURCE) #pragma redefine_extname fopen64 fopen #pragma redefine_extname freopen64 freopen #pragma redefine_extname tmpfile64 tmpfile #pragma redefine_extname fgetpos64 fgetpos #pragma redefine_extname fsetpos64 fsetpos #pragma redefine_extname fseeko64 fseeko #pragma redefine_extname ftello64 ftello #endif /* defined(_LP64) && defined(_LARGEFILE64_SOURCE) */ #endif /* __PRAGMA_REDEFINE_EXTNAME */ So I'm not worried about this change for Solaris (that much anyway). I would say for the example that you quoted above, you need to know about the "_REDIRECT" symbol and how that plays into the compile system on whatever environment you got the stdio.h quote from. > It is my understanding that if _FILE_OFFSET_BITS=64 then stdio.h will > "#define fopen fopen64" Maybe. Maybe not. What you've quoted is not (yet) definitive. > * I think that it's confusing if we only do that for ostream.cpp I think we've both made the point that doing this just for ostream.o can result in confusion. > * If we ever add a "fopen()" call to a .hpp file which is included > from ostream.ccp then the caller of fopen will have different behavior > in based on which file it's called from. I'm not finding any reference to 'fopen' in any .hpp file in the HotSpot code base, but, of course, that's not really your point. Yes, we can screw things up in the future. I think that's a given. > I've written a small example to detail what kind of problems we could > get: https://gist.github.com/anonymous/b690f753401504f1f096 Yup. I "git" the example. I just don't quite understand why you couldn't put the example here in the e-mail thread and had to drop it on github... > If ostream.cpp is the first to call into Logger::log we will get one > behavior, if another file does we get the default behavior. > > Hope this makes my reasons for arguing about this a bit clearer. Yup, I understand your reasons and I think I alluded to other scenarios earlier in the e-mail thread. There is no doubt that this is a risk and that there could be unforeseen side effects. Dan > /Mikael > >> >> Dan >> >> >>> >>>>> >>>>> As Tao mentioned this will require us to handle the fact that >>>>> there is >>>>> no fopen64() call on Windows, that the CRT fopen() already seems to >>>>> handle large files and that ftell64() and fseek64() have slightly >>>>> different names on Windows. I don't think this is a large hurdle >>>>> and I >>>>> think we know how to solve this problem. >>>> As I said, nothing was changed for Windows code. >>> >>> No, but to be consistent you'd need to update the ftell* fseek* to use >>> 64 bit versions, right? >>> >>> /Mikael >>> >>>>> >>>>> /Mikael >>>>> >>>>>> >>>>>> Dan >>>>>> >>>>>> >>>>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>>>> Thank you for review, Dan. >>>>>>> >>>>>>> I'll try to answer as much as I can. Please see inline. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> Tao, >>>>>>>> >>>>>>>> I think the lack of response to this review request is the >>>>>>>> absolutely >>>>>>>> strange nature of these changes. And I thought I put out some >>>>>>>> weird >>>>>>>> code reviews... :-) >>>>>>>> >>>>>>>> make/linux/makefiles/vm.make >>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>>>> obvious in this webrev about what this will mean so I took a >>>>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>>>> >>>>>>>> Must be in the source files somewhere, but I can't find any >>>>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>>>> >>>>>>>> make/solaris/makefiles/vm.make >>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>>>> >>>>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>>>> understand what you're doing here. >>>>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>>>> os::readdir() function will use the safer, multi-threaded >>>>>>>> version of readdir_r(). Seems fine to me. >>>>>>>> >>>>>>>> Here's what I need to know: >>>>>>>> >>>>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>>>> ostream.{c,h}pp? >>>>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>>>> >>>>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>>>> refer to the following document >>>>>>> >>>>>>> This Sun White Paper >>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>>>> should apply to Linux the same way as was agreed across platforms >>>>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>>>> >>>>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>>>> what happens if another part of the VM has a different idea >>>>>>>> about >>>>>>>> the value of _FILE_OFFSET_BITS? >>>>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>>>> So, >>>>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>>>> can't deal with large files. No more no less. >>>>>>> >>>>>>>> >>>>>>>> I saw this in the post to the Runtime alias: >>>>>>>> >>>>>>>> > Included runtime dev to see whether they have some idea to >>>>>>>> handle >>>>>>>> > the compilation choices. >>>>>>>> >>>>>>>> And I still have no idea what you're asking here? What compilation >>>>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>>>> to have mis-matched pieces and be unstable? >>>>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>>>> instead of just one object (ostream.o)?" is my main question I >>>>>>> originally tried to ask. >>>>>>> >>>>>>>> >>>>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>>>> change has too many open questions (for now)... >>>>>>>> >>>>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>>>> 2GB+ >>>>>>>> GC log or not. The conversation below didn't help me at all. >>>>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>>>> >>>>>>>> Dan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>>>> Since the changeset touched makefiles, I've included >>>>>>>>> build-dev at openjdk.java.net . >>>>>>>>> >>>>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Need reviews to catch RDP2. >>>>>>>>>> >>>>>>>>>> The current webrev is a working solution to all platforms, >>>>>>>>>> Linux, >>>>>>>>>> Windows, and Solaris. >>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>>>> Included runtime dev to see whether they have some idea to >>>>>>>>>>> handle >>>>>>>>>>> the compilation choices. >>>>>>>>>>> >>>>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>>>> sufficient. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>>>> Thank you, Mikael. >>>>>>>>>>>> >>>>>>>>>>>> Please see inline. >>>>>>>>>>>> >>>>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>>>> observation. >>>>>>>>>>>> >>>>>>>>>>>> Tao >>>>>>>>>>>> >>>>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>>>> Tao, >>>>>>>>>>>>> >>>>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>>>> 7ux bug >>>>>>>>>>>>>> >>>>>>>>>>>>>> webrev: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> changeset: >>>>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>>>> ostream.o >>>>>>>>>>>>>> >>>>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>>>> globally >>>>>>>>>>>>>> applicable? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works >>>>>>>>>>>>>> fine; >>>>>>>>>>>>>> however, >>>>>>>>>>>>>> there are at least five code conflicts if introducing the >>>>>>>>>>>>>> flag >>>>>>>>>>>>>> globally >>>>>>>>>>>>>> to Solaris. >>>>>>>>>>>>>> >>>>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>>>> four >>>>>>>>>>>>>> files >>>>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>>>> from >>>>>>>>>>>>>> setting >>>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>>>> I haven't found any clean solution for solving this >>>>>>>>>>>>>> problem on >>>>>>>>>>>>>> Windows. >>>>>>>>>>>>> >>>>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>>>> work on >>>>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>>>> problem >>>>>>>>>>>>> there. >>>>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>>>> That's >>>>>>>>>>>> why I chose to apply the flag only to the files >>>>>>>>>>>> (ostream.cpp and >>>>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>>>> >>>>>>>>>>>>> Instead I suggest that you use the compatibility API >>>>>>>>>>>>> described >>>>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>>>> and >>>>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>>>> >>>>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and >>>>>>>>>>>>> has >>>>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>>>> ostream >>>>>>>>>>>>> >>>>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>>>> resembles the changes which will be needed for Windows >>>>>>>>>>>>> anyway. >>>>>>>>>>>>> Those changes would consist of changing calls to >>>>>>>>>>>>> ftell/fseek to >>>>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on >>>>>>>>>>>>> Solaris/Linux. >>>>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>>>> excludes >>>>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>>>> provides other benefits. >>>>>>>>>>>> >>>>>>>>>>>> This Sun White Paper >>>>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> summarizes >>>>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Since there is no fopen64 on Windows it seems that the >>>>>>>>>>>>> default >>>>>>>>>>>>> fopen already supports large files. >>>>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows >>>>>>>>>>>> can >>>>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved >>>>>>>>>>>> "half >>>>>>>>>>>> of my problem" :) >>>>>>>>>>>>> >>>>>>>>>>>>> /Mikael >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> test: >>>>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>>>> verified on the >>>>>>>>>>>>>> following configurations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Linux * i586 >>>>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>>>> >>>>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>>>> >>>>>>>> >>>>>> >> > From daniel.daugherty at oracle.com Thu Jun 6 13:57:21 2013 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 06 Jun 2013 07:57:21 -0600 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51B05441.60402@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> <51AF6F63.3010906@oracle.com> <51B05441.60402@oracle.com> Message-ID: <51B09541.8090503@oracle.com> Mikael, Sorry for top posting a reply here, but the thread below is just getting too muddled. I'm not quite clear on why you keep trying to distinguish between something being a "#define" or being a bound linkage symbol. In either case, the end result will be the same. The ostream.o module will call foo64() versions of functions instead of foo() versions. Consider this: - ostream.cpp can include a great many .hpp files or not - those .hpp files may have functions affected by _FILE_OFFSET_BITS or not - the only functions in any of the .hpp files that matter to ostream.o are the ones that code in ostream.o actually calls - if ostream.o doesn't attempt to use/bind to a function then it doesn't matter whether it has been #define'ed or rebound to a new name What really matters here is that ostream.o will call foo64() and some other object file, bar.o, will call foo(). If ostream.o and bar.o are operating on the same FILE or file, then we will have problems. How we got there (#define or binding) is not really the critical point. Dan P.S. There is a reason the "#pragma redefine_extname" mechanism is preferred to the "#define" mechanism that is discussed in the largefiles.pdf, but that point is not critical to this discussion. On 6/6/13 3:20 AM, Mikael Gerdin wrote: > Tao, > > On 2013-06-05 19:03, Tao Mao wrote: >> Hi Mikael, >> >> Let me explain the changeset first before I go into your comments. > > Ok. > >> >> (1) make/linux/makefiles/vm.make >> >> # Large File Support >> ifneq ($(LP64), 1) >> CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 >> endif # ifneq ($(LP64), 1) >> >> For "Linux" code, set _FILE_OFFSET_BITS=64 "only" to ostream.o (i.e. >> ostream.{c,h}pp) in order to bind all foo() functions to foo64() >> functions when linking libraries. This will enable functions in >> ostream.{c,h}pp to deal with large files while all other files won't >> have the ability. And that's what we want at this point. > > This is slightly incorrect. Your change has no thing to do with > ostream.hpp other than the fact that the version of ostream.hpp which > is preprocessed into ostream.cpp will have different preprocessor > behavior. > > In reality, your change will affect all the header files transitively > included from ostream.cpp during its compilation. > > I agree that in effect this will cause references to fopen() in > ostream.cpp to become "call fopen64" in the .o file. > > >> >> (2) make/solaris/makefiles/vm.make >> >> # Large File Support >> ifneq ($(LP64), 1) >> CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 >> endif # ifneq ($(LP64), 1) >> >> Similarly, for "Solaris" code, set _FILE_OFFSET_BITS=64 "only" to >> ostream.o (i.e. ostream.{c,h}pp) in order to bind all foo() functions to >> foo64() functions when linking libraries. This will enable functions in >> ostream.{c,h}pp to deal with large files while all other files won't >> have the ability. >> >> (1) & (2) altogether set _FILE_OFFSET_BITS=64 "only" to ostream.o in >> "Linux" and "Solaris" code base and have not changed Windows and BSD >> code, as they can handle large files currently. No need to change. >> >> Remember the current implementation has a limited affected range (i.e. >> ostream.o for Linux and Solaris). No more no less. >> >> (3) src/os/solaris/vm/os_solaris.inline.hpp >> >> #if defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 >> dirent* p; >> int status; >> >> if((status = ::readdir_r(dirp, dbuf, &p)) != 0) { >> errno = status; >> return NULL; >> } else >> return p; >> #else // defined(_LP64) || defined(_GNU_SOURCE) || >> _FILE_OFFSET_BITS==64 >> return ::readdir_r(dirp, dbuf); >> #endif // defined(_LP64) || defined(_GNU_SOURCE) || >> _FILE_OFFSET_BITS==64 >> >> >> This piece of code handles some exception I need to fix. It is not what >> I want to change but rather what I have to change. >> >> Please see inline. >> >> Thanks. >> Tao >> >> >> On 6/5/13 9:02 AM, Mikael Gerdin wrote: >>> Tao, >>> >>> On 2013-06-05 17:48, Tao Mao wrote: >>>> Thank you for comments. One thing I want to point out is that the >>>> current change has not touched on Windows code. >>>> >>>> Please see inline. >>>> >>>> Tao >>>> >>>> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>>>> >>>>> >>>>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>>>> OK, based on the largefiles.pdf write-up, your use of >>>>>> _FILE_OFFSET_BITS=64 >>>>>> is going to cause ostream.o to bind to various 64-bit versions of >>>>>> some >>>>>> functions. Based on my very hazy memory of my days in file system >>>>>> code, >>>>>> this binding is going to affect ostream.o only unless ostream.o is >>>>>> writing to some file with the foo64() function and some other >>>>>> code is >>>>>> also writing to the same file at the same time with foo(). However, >>>>>> if we >>>>>> have two different functions writing to the same open file at the >>>>>> same >>>>>> time, then we have bigger issues. :-) >>>>>> >>>>>> I'm good with these changes now. I agree that solving the problem of >>>>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>>>> solved right now. >>>>> >>>>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 >>>>> when >>>>> compiling ostream.cpp will effectively cause the headers to swap out >>>>> the implementations of the f[open,tell,seek] to 64 bit versions for >>>>> all headers that are included and inlined in ostream.cpp. >>>>> Other parts of the code using the same headers will see different >>>>> versions of the functions with different parameters due to off_t >>>>> changing sizes. >>>> The change is currently effective for Linux and Solaris if you look at >>>> the file directories. Nothing changed for Windows and BSD, as they >>>> don't >>>> need such change. >>> >>> Right. >>> But if you use my suggested approach you would need to change calls to >>> fopen() in ostream.cpp to fopen_pd where >>> >>> if (linux || solaris) && 32bit >>> #define fopen_pd fopen64 >>> else >>> #define fopen_pd fopen >> I saw what you suggested. >> >> But, what's the benefit of doing so, or, what's the severe problem about >> the current changeset? I don't see that one's better than another. > > Because I want to avoid the situation I described in my mail to Dan. > I wrote a small code example to describe the situation I want to avoid: > https://gist.github.com/anonymous/b690f753401504f1f096 > > I know that we currently don't have calls to fopen() from .hpp files > in the VM but if anyone ever was to add one strange, hard to track > down bugs will happen. > >> >>> >>>>> >>>>> I think that what we should do is to use the "Transitional >>>>> compilation >>>>> environment" detailed in ?2.6.2 in largefiles.pdf and change the >>>>> calls >>>>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>>>> directly. I feel this is especially important at this late stage in >>>>> the release to make an explicit change instead of setting a #define >>>>> which has propagating side-effects. >>>> How do you see "propagating side-effects" and to where? >>> >>> _FILE_OFFSET_BITS=64 changes the definition of fopen for every file >>> including stdio.h. >> The current implementation has a limited affected range (i.e. ostream.o >> for Linux and Solaris). No more no less. >> >> How do you see this flag will affect every file? > > Right, I meant that it will transitively affect all files included > from ostream.cpp, as I mentioned above. > > /Mikael > >>> >>> It's confusing when a call to "fopen()" in one file calls the 64 bit >>> version and in other files it doesn't. >>> >>> How will this work with precompiled headers? Which version of fopen >>> will be in the precompiled header file? >>> >>>>> >>>>> As Tao mentioned this will require us to handle the fact that >>>>> there is >>>>> no fopen64() call on Windows, that the CRT fopen() already seems to >>>>> handle large files and that ftell64() and fseek64() have slightly >>>>> different names on Windows. I don't think this is a large hurdle >>>>> and I >>>>> think we know how to solve this problem. >>>> As I said, nothing was changed for Windows code. >>> >>> No, but to be consistent you'd need to update the ftell* fseek* to use >>> 64 bit versions, right? >>> >>> /Mikael >>> >>>>> >>>>> /Mikael >>>>> >>>>>> >>>>>> Dan >>>>>> >>>>>> >>>>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>>>> Thank you for review, Dan. >>>>>>> >>>>>>> I'll try to answer as much as I can. Please see inline. >>>>>>> >>>>>>> Thanks. >>>>>>> Tao >>>>>>> >>>>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>> >>>>>>>> Tao, >>>>>>>> >>>>>>>> I think the lack of response to this review request is the >>>>>>>> absolutely >>>>>>>> strange nature of these changes. And I thought I put out some >>>>>>>> weird >>>>>>>> code reviews... :-) >>>>>>>> >>>>>>>> make/linux/makefiles/vm.make >>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>>>> obvious in this webrev about what this will mean so I took a >>>>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>>>> >>>>>>>> Must be in the source files somewhere, but I can't find any >>>>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>>>> >>>>>>>> make/solaris/makefiles/vm.make >>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>>>> >>>>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>>>> understand what you're doing here. >>>>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>>>> os::readdir() function will use the safer, multi-threaded >>>>>>>> version of readdir_r(). Seems fine to me. >>>>>>>> >>>>>>>> Here's what I need to know: >>>>>>>> >>>>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>>>> ostream.{c,h}pp? >>>>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>>>> >>>>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>>>> refer to the following document >>>>>>> >>>>>>> This Sun White Paper >>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>>>> should apply to Linux the same way as was agreed across platforms >>>>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>>>> >>>>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>>>> what happens if another part of the VM has a different idea >>>>>>>> about >>>>>>>> the value of _FILE_OFFSET_BITS? >>>>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>>>> So, >>>>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>>>> can't deal with large files. No more no less. >>>>>>> >>>>>>>> >>>>>>>> I saw this in the post to the Runtime alias: >>>>>>>> >>>>>>>> > Included runtime dev to see whether they have some idea to >>>>>>>> handle >>>>>>>> > the compilation choices. >>>>>>>> >>>>>>>> And I still have no idea what you're asking here? What compilation >>>>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>>>> to have mis-matched pieces and be unstable? >>>>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>>>> instead of just one object (ostream.o)?" is my main question I >>>>>>> originally tried to ask. >>>>>>> >>>>>>>> >>>>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>>>> change has too many open questions (for now)... >>>>>>>> >>>>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>>>> 2GB+ >>>>>>>> GC log or not. The conversation below didn't help me at all. >>>>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>>>> >>>>>>>> Dan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>>>> Since the changeset touched makefiles, I've included >>>>>>>>> build-dev at openjdk.java.net . >>>>>>>>> >>>>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Tao >>>>>>>>> >>>>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Need reviews to catch RDP2. >>>>>>>>>> >>>>>>>>>> The current webrev is a working solution to all platforms, >>>>>>>>>> Linux, >>>>>>>>>> Windows, and Solaris. >>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>>>> Included runtime dev to see whether they have some idea to >>>>>>>>>>> handle >>>>>>>>>>> the compilation choices. >>>>>>>>>>> >>>>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>>>> sufficient. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>>>> Thank you, Mikael. >>>>>>>>>>>> >>>>>>>>>>>> Please see inline. >>>>>>>>>>>> >>>>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>>>> observation. >>>>>>>>>>>> >>>>>>>>>>>> Tao >>>>>>>>>>>> >>>>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>>>> Tao, >>>>>>>>>>>>> >>>>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>>>> 7ux bug >>>>>>>>>>>>>> >>>>>>>>>>>>>> webrev: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> changeset: >>>>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>>>> ostream.o >>>>>>>>>>>>>> >>>>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>>>> globally >>>>>>>>>>>>>> applicable? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works >>>>>>>>>>>>>> fine; >>>>>>>>>>>>>> however, >>>>>>>>>>>>>> there are at least five code conflicts if introducing the >>>>>>>>>>>>>> flag >>>>>>>>>>>>>> globally >>>>>>>>>>>>>> to Solaris. >>>>>>>>>>>>>> >>>>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>>>> four >>>>>>>>>>>>>> files >>>>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>>>> from >>>>>>>>>>>>>> setting >>>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>>>> I haven't found any clean solution for solving this >>>>>>>>>>>>>> problem on >>>>>>>>>>>>>> Windows. >>>>>>>>>>>>> >>>>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>>>> work on >>>>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>>>> problem >>>>>>>>>>>>> there. >>>>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>>>> That's >>>>>>>>>>>> why I chose to apply the flag only to the files >>>>>>>>>>>> (ostream.cpp and >>>>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>>>> >>>>>>>>>>>>> Instead I suggest that you use the compatibility API >>>>>>>>>>>>> described >>>>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>>>> and >>>>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>>>> >>>>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and >>>>>>>>>>>>> has >>>>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>>>> ostream >>>>>>>>>>>>> >>>>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>>>> resembles the changes which will be needed for Windows >>>>>>>>>>>>> anyway. >>>>>>>>>>>>> Those changes would consist of changing calls to >>>>>>>>>>>>> ftell/fseek to >>>>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on >>>>>>>>>>>>> Solaris/Linux. >>>>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>>>> excludes >>>>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>>>> provides other benefits. >>>>>>>>>>>> >>>>>>>>>>>> This Sun White Paper >>>>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> summarizes >>>>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Since there is no fopen64 on Windows it seems that the >>>>>>>>>>>>> default >>>>>>>>>>>>> fopen already supports large files. >>>>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows >>>>>>>>>>>> can >>>>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved >>>>>>>>>>>> "half >>>>>>>>>>>> of my problem" :) >>>>>>>>>>>>> >>>>>>>>>>>>> /Mikael >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> test: >>>>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>>>> verified on the >>>>>>>>>>>>>> following configurations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Linux * i586 >>>>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>>>> >>>>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>>>> >>>>>>>> >>>>>> > From jon.masamitsu at oracle.com Thu Jun 6 14:40:35 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 06 Jun 2013 07:40:35 -0700 Subject: Request for review (vs) - 8014546: MetaspaceAux print_metaspace_change() should print "used" after GC not capacity In-Reply-To: <1370508403.2590.6.camel@cirrus> References: <51AF9D2C.3030005@oracle.com> <1370508403.2590.6.camel@cirrus> Message-ID: <51B09F63.6010202@oracle.com> Thanks On 6/6/13 1:46 AM, Thomas Schatzl wrote: > Hi, > > On Wed, 2013-06-05 at 13:18 -0700, Jon Masamitsu wrote: >> As the summary says, print the used metaspace in >> print_metaspace_change() instead >> of capacity. The output used to be >> >> previous_used -> capacity (reserved) >> >> and now is >> >> previous_used -> used_after_GC (reserved) >> >> This makes the print_metaspace_change() more similar to the >> GC print_heap_change(). >> >> http://cr.openjdk.java.net/~jmasa/8014546/webrev.00/ >> >> Two lines changed. > Looks good :) > > Thomas > > From jon.masamitsu at oracle.com Thu Jun 6 14:50:34 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 06 Jun 2013 07:50:34 -0700 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51B054F2.7000807@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> <51B054F2.7000807@oracle.com> Message-ID: <51B0A1BA.9070308@oracle.com> Mikael, Thanks. I'd be interested in seeing the instrumentation you add. Might be worth adding as an enhancement in a later changeset. Jon On 6/6/13 2:22 AM, Mikael Gerdin wrote: > Jon, > > On 2013-06-06 04:41, Jon Masamitsu wrote: >> >> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >>> Hi, >>> >>> Can I have some reviews of this small fix to the Metaspace memory >>> allocation path. >>> >>> Problem: >>> When a Metaspace allocation request cannot be satisfied by the current >>> chunk the chunk is retired and a new chunk is requested. This causes >>> whatever is left in the chunk to be effectively leaked. >>> >>> Suggested fix: >>> Put the remaining memory in each chunk on the Metablock freelist so it >>> can be used to satisfy future allocations. >>> >>> Possible addition: >>> When allocating from the block free list, use >>> FreeBlockDictionary::atLeast instead of >>> FreeBlockDictionary::exactly and split the Metablock if >>> it's large enough. >>> >>> One might argue that this increases the fragmentation of the memory on >>> the block free list but I think that we primarily want to use the >>> block free list for small allocations and allocate from chunks for >>> large allocations. >>> >>> Webrev: >>> Only fix: >>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ >> >> The "Only fix" looks good. Did you test with >> metaspace_slow_verify=true? >> >>> >>> Incremental webrev for splitting blocks: >>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ >> >> Change looks good. >> >> Did you do any long running tests with the block splitting? Such as >> 24hours with kitchensink? Something that would reuse Metablocks >> so that we can see if we are fragmenting instead of reusing? >> > > I did some runs earlier but I don't have any data from them. > I can try to get an instrumented build together and run KS over the > weekend. > > /Mikael > >> Jon >> >> >>> >>> Bug links: >>> https://jbs.oracle.com/bugs/browse/JDK-8009561 >>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >>> >>> Thanks >>> /Mikael >> > From thomas.schatzl at oracle.com Thu Jun 6 16:02:05 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Jun 2013 18:02:05 +0200 Subject: CMS parallel initial mark In-Reply-To: References: Message-ID: <1370534525.2590.43.camel@cirrus> Hi, On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: > Hi, > > I'd like to have the following contributed if it makes sense. > > 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ > > that implements a parallel version of the initial mark phase of the > CMS collector. It's relatively a straightforward parallelization of > the existing single-threaded code. With the above patch, I see about > ~3-6x speedup in the initial mark pause times. had a look at this patch in the version at http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.01/ Looks good so far, some comments: - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new Par_MarkRefsIntoClosure could be refactored slightly as they have exactly the same member variables. Not sure how this situation is handled in other code though, and what others (Jon) think. - in concurrentMarkSweepGeneration.cpp in CMSCollector::checkpointRootsInitialWork() the new code seems a little strange to me. I.e. roughly it does the following: [...] if (CMSParallelInitialMarkEnabled) { [ set up parallel stuff ] if (n_workers > 1) { // do parallel thread } else { // do serial work using the current thread } } else { // do serial work } it does not feel good to have two serial paths here; maybe use the serial/original code if n_workers == 1 and guarantee(n_workers > 1, ) in the first part of the if? Or look if the old serial path could be removed? (or the new one). - I think Jon already mentioned that it might be better to use an int for worker_id in CMSParMarkTask::do_young_space_rescan() (if it is actually needed) to keep that code in line with current use. - the comments for at least CMSCollector:: initialize_sequential_subtasks_for_young_gen_rescan() and CMSCollector::merge_survivor_plab_arrays() should be reworked - they only mention parallel rescan as entry points. - the change in the assert in CMSCollector::merge_survivor_plab_arrays() looks a little odd now. I mean, looking at that assert in isolation poses the question why is this method only called when doing initial mark in parallel? It somehow seems that the same work is done somewhere else just for the initial mark and serial case, which just looks strange to me. That's just an observation from me. assert(_collectorState == FinalMarking || (CMSParallelInitialMarkEnabled && _collectorState == InitialMarking), "Error"); - in CMSCollector::CMSCollector() there is this unconditional setup of a few data structures used for parallel processing if any of the CMSParallel* switches is enabled. Hth and thanks for the contribution :), Thomas From jon.masamitsu at oracle.com Thu Jun 6 17:06:08 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 06 Jun 2013 10:06:08 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370534525.2590.43.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> Message-ID: <51B0C180.4020106@oracle.com> On 6/6/13 9:02 AM, Thomas Schatzl wrote: > Hi, > > On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: >> Hi, >> >> I'd like to have the following contributed if it makes sense. >> >> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >> >> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ >> >> that implements a parallel version of the initial mark phase of the >> CMS collector. It's relatively a straightforward parallelization of >> the existing single-threaded code. With the above patch, I see about >> ~3-6x speedup in the initial mark pause times. > had a look at this patch in the version at > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.01/ > > Looks good so far, some comments: > - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new > Par_MarkRefsIntoClosure could be refactored slightly as they have > exactly the same member variables. Not sure how this situation is > handled in other code though, and what others (Jon) think. Thomas, If you don't mind I'd like to keep this changeset to a minimum so not do any additional refactoring. That's a good suggestion but since this is the first sizable contribution I'm sponsoring, simpler is better for me. > > - in concurrentMarkSweepGeneration.cpp in > CMSCollector::checkpointRootsInitialWork() the new code seems a little > strange to me. I.e. roughly it does the following: > > [...] > if (CMSParallelInitialMarkEnabled) { > [ set up parallel stuff ] > if (n_workers > 1) { > // do parallel thread > } else { > // do serial work using the current thread > } > } else { > // do serial work > } > > it does not feel good to have two serial paths here; maybe use the > serial/original code if n_workers == 1 and guarantee(n_workers > 1, ) in > the first part of the if? Or look if the old serial path could be > removed? (or the new one). Is the inner serial path one thread executing a single chunk that does all the work (i.e., executes the parallel code but by only one thread)? > > - I think Jon already mentioned that it might be better to use an int > for worker_id in CMSParMarkTask::do_young_space_rescan() (if it is > actually needed) to keep that code in line with current use. > > - the comments for at least CMSCollector:: > initialize_sequential_subtasks_for_young_gen_rescan() and > CMSCollector::merge_survivor_plab_arrays() should be reworked - they > only mention parallel rescan as entry points. > > - the change in the assert in > CMSCollector::merge_survivor_plab_arrays() looks a little odd now. I > mean, looking at that assert in isolation poses the question why is this > method only called when doing initial mark in parallel? It somehow seems > that the same work is done somewhere else just for the initial mark and > serial case, which just looks strange to me. That's just an observation > from me. > > assert(_collectorState == FinalMarking || > (CMSParallelInitialMarkEnabled && _collectorState == > InitialMarking), "Error"); > > - in CMSCollector::CMSCollector() there is this unconditional setup of > a few data structures used for parallel processing if any of the > CMSParallel* switches is enabled. I'm not sure what code this is. What lines in the webrev? Jon > Hth and thanks for the contribution :), > Thomas > From jon.masamitsu at oracle.com Thu Jun 6 17:12:24 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 6 Jun 2013 10:12:24 -0700 (PDT) Subject: CMS parallel initial mark In-Reply-To: References: <51A6AD6D.8010309@oracle.com> Message-ID: <51B0C2F8.6020303@oracle.com> Hiroshi, The CR for this changeset will be 6412968. CMS: Long initial mark pauses Jon On 5/30/13 10:56 AM, Hiroshi Yamauchi wrote: > Thanks, Jon. Please let me know when you know more. > > On Wed, May 29, 2013 at 6:37 PM, Jon Masamitsu wrote: >> Hiroshi, >> >> I'm still reviewing the changes but so far this looks >> very promising. I've patched you changes into a repository >> and started running a few tests. I've turned on >> >> -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways >> >> Thanks. >> >> Jon >> >> >> On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: >>> Hi, >>> >>> I'd like to have the following contributed if it makes sense. >>> >>> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >>> >>> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ >>> >>> that implements a parallel version of the initial mark phase of the >>> CMS collector. It's relatively a straightforward parallelization of >>> the existing single-threaded code. With the above patch, I see about >>> ~3-6x speedup in the initial mark pause times. >>> >>> 2) Now, here's a related issue and a suggested fix/patch for it: >>> >>> I see that the initial mark and remark pause times sometimes spike >>> with a large young generation. For example, under a 1 GB young gen / 3 >>> GB heap setting, they occasionally spike up to ~500 milliseconds from >>> the normal < 100 ms range, on my machine. As far as I can tell, this >>> happens when the eden is fairly occupied (> 700 MB full) and not >>> sufficiently divided up and the parallelism decreases (at the worst >>> case it becomes almost single-threaded.) >>> >>> Here's a suggested patch in a separate patch: >>> >>> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ >>> >>> that attempts to improve on this issue by implementing an alternative >>> way of dividing up the eden into chunks for an increased parallelism >>> (or better load balancing between the GC threads) for the young gen >>> scan portion of the remark phase (and the now-parallelized initial >>> mark phase.) It uses a CAS-based mechanism that samples the object >>> boundaries in the eden space on the slow allocation code paths (eg. at >>> the TLAB refill and large object allocation times) at all times. >>> >>> This approach is in contrast to the original mechanism that samples >>> object boundaries in the eden space asynchronously during the preclean >>> phase. I think the reason that the above issue happens is that when >>> the young generation is large, a large portion of the eden space could >>> get filled/allocated outside of the preclean phase (or a concurrent >>> collection) and the object boundaries do not get sampled >>> often/regularly enough. Also, it isn't very suited for the parallel >>> initial mark because the initial mark phase isn't preceded by the >>> preclean phase unlike the remark phase. According to the Dacapo >>> benchmarks, this alternative sampling mechanism does not have >>> noticeable runtime overhead despite it is engaged at all times. >>> >>> With this patch, I see that the (parallel) initial mark and remark >>> pause times stay below 100 ms (no spikes) under the same setting. >>> >>> Both of these features/flags are disabled by default. You're welcome >>> to handle the two patches separately. >>> >>> Thanks, >>> Hiroshi >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Thu Jun 6 18:46:46 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Thu, 06 Jun 2013 11:46:46 -0700 Subject: RFR (XS): 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 In-Reply-To: <51AFB2AE.7040604@oracle.com> References: <51AFB2AE.7040604@oracle.com> Message-ID: <51B0D916.9030006@oracle.com> Hi Jesper, The change is psAdaptiveSizePolicy.cpp looks good. I don't, however, think you should remove the line break in psScavenge.cpp. I _that_ that line break is meant to be there. The output beginning with "avg_survived_padded_avg" is slightly indented. Compare the other places where "AdaptiveSizeStart" is printed in psMarkSweep.cpp and psParallelCompact.cpp. Leaving line break in will make the output: AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 target_size: 1048576 JohnC On 6/5/2013 2:50 PM, Jesper Wilhelmsson wrote: > Hi, > > Can I have a couple of reviews for this really minor fix? > > 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 > > A missing line break in the adaptive size verbose logging has been added. > > When looking at this I also found that an existing line break caused a > slightly ugly verbose output when breaking a line causing a second > line without a identifier at the beginning. I removed that line break > as well in this change. > > > webrev: > http://cr.openjdk.java.net/~jwilhelm/8015903/hotspot-webrev/ > > > > Output before: > > AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 > overflow: falseAdaptiveSizeStart: 23.548 collection: 67 > avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: > 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 > tenuring_thresh: 14 target_size: 1048576 > > > Output after: > > AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 > overflow: false > AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: > 728176.250000 avg_promoted_padded_avg: 223962144.000000 > avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 > target_size: 1048576 > > > Thanks, > /Jesper From mikael.gerdin at oracle.com Thu Jun 6 19:06:17 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Thu, 06 Jun 2013 21:06:17 +0200 Subject: URGENT: Request for review: 7122222: GC log is limited to 2G for 32-bit In-Reply-To: <51B09541.8090503@oracle.com> References: <51A0037B.9070508@oracle.com> <51A321CB.3060906@oracle.com> <51A69D07.9090005@oracle.com> <51A78AAD.8060102@oracle.com> <51AE5E09.9080808@oracle.com> <51AE7256.1070609@oracle.com> <51AE79AF.9070706@oracle.com> <51AE8114.80600@oracle.com> <51AE8488.5050101@oracle.com> <51AEF4A8.4030500@oracle.com> <51AF5DBC.3080107@oracle.com> <51AF6126.5050106@oracle.com> <51AF6F63.3010906@oracle.com> <51B05441.60402@oracle.com> <51B09541.8090503@oracle.com> Message-ID: <51B0DDA9.20403@oracle.com> Dan, On 2013-06-06 15:57, Daniel D. Daugherty wrote: > Mikael, > > Sorry for top posting a reply here, but the thread below is just > getting too muddled. > > I'm not quite clear on why you keep trying to distinguish between > something being a "#define" or being a bound linkage symbol. In > either case, the end result will be the same. The ostream.o module > will call foo64() versions of functions instead of foo() versions. > > Consider this: > > - ostream.cpp can include a great many .hpp files or not > - those .hpp files may have functions affected by _FILE_OFFSET_BITS > or not > - the only functions in any of the .hpp files that matter to ostream.o > are the ones that code in ostream.o actually calls > - if ostream.o doesn't attempt to use/bind to a function then it > doesn't matter whether it has been #define'ed or rebound to a new > name > > What really matters here is that ostream.o will call foo64() and some > other object file, bar.o, will call foo(). If ostream.o and bar.o are > operating on the same FILE or file, then we will have problems. How > we got there (#define or binding) is not really the critical point. I agree with your summary. I don't plan to push my opinions on this any further. /Mikael > > Dan > > P.S. > There is a reason the "#pragma redefine_extname" mechanism is > preferred to the "#define" mechanism that is discussed in the > largefiles.pdf, but that point is not critical to this discussion. > > > On 6/6/13 3:20 AM, Mikael Gerdin wrote: >> Tao, >> >> On 2013-06-05 19:03, Tao Mao wrote: >>> Hi Mikael, >>> >>> Let me explain the changeset first before I go into your comments. >> >> Ok. >> >>> >>> (1) make/linux/makefiles/vm.make >>> >>> # Large File Support >>> ifneq ($(LP64), 1) >>> CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 >>> endif # ifneq ($(LP64), 1) >>> >>> For "Linux" code, set _FILE_OFFSET_BITS=64 "only" to ostream.o (i.e. >>> ostream.{c,h}pp) in order to bind all foo() functions to foo64() >>> functions when linking libraries. This will enable functions in >>> ostream.{c,h}pp to deal with large files while all other files won't >>> have the ability. And that's what we want at this point. >> >> This is slightly incorrect. Your change has no thing to do with >> ostream.hpp other than the fact that the version of ostream.hpp which >> is preprocessed into ostream.cpp will have different preprocessor >> behavior. >> >> In reality, your change will affect all the header files transitively >> included from ostream.cpp during its compilation. >> >> I agree that in effect this will cause references to fopen() in >> ostream.cpp to become "call fopen64" in the .o file. >> >> >>> >>> (2) make/solaris/makefiles/vm.make >>> >>> # Large File Support >>> ifneq ($(LP64), 1) >>> CXXFLAGS/ostream.o += -D_FILE_OFFSET_BITS=64 >>> endif # ifneq ($(LP64), 1) >>> >>> Similarly, for "Solaris" code, set _FILE_OFFSET_BITS=64 "only" to >>> ostream.o (i.e. ostream.{c,h}pp) in order to bind all foo() functions to >>> foo64() functions when linking libraries. This will enable functions in >>> ostream.{c,h}pp to deal with large files while all other files won't >>> have the ability. >>> >>> (1) & (2) altogether set _FILE_OFFSET_BITS=64 "only" to ostream.o in >>> "Linux" and "Solaris" code base and have not changed Windows and BSD >>> code, as they can handle large files currently. No need to change. >>> >>> Remember the current implementation has a limited affected range (i.e. >>> ostream.o for Linux and Solaris). No more no less. >>> >>> (3) src/os/solaris/vm/os_solaris.inline.hpp >>> >>> #if defined(_LP64) || defined(_GNU_SOURCE) || _FILE_OFFSET_BITS==64 >>> dirent* p; >>> int status; >>> >>> if((status = ::readdir_r(dirp, dbuf, &p)) != 0) { >>> errno = status; >>> return NULL; >>> } else >>> return p; >>> #else // defined(_LP64) || defined(_GNU_SOURCE) || >>> _FILE_OFFSET_BITS==64 >>> return ::readdir_r(dirp, dbuf); >>> #endif // defined(_LP64) || defined(_GNU_SOURCE) || >>> _FILE_OFFSET_BITS==64 >>> >>> >>> This piece of code handles some exception I need to fix. It is not what >>> I want to change but rather what I have to change. >>> >>> Please see inline. >>> >>> Thanks. >>> Tao >>> >>> >>> On 6/5/13 9:02 AM, Mikael Gerdin wrote: >>>> Tao, >>>> >>>> On 2013-06-05 17:48, Tao Mao wrote: >>>>> Thank you for comments. One thing I want to point out is that the >>>>> current change has not touched on Windows code. >>>>> >>>>> Please see inline. >>>>> >>>>> Tao >>>>> >>>>> On 6/5/13 1:19 AM, Mikael Gerdin wrote: >>>>>> >>>>>> >>>>>> On 2013-06-05 02:21, Daniel D. Daugherty wrote: >>>>>>> OK, based on the largefiles.pdf write-up, your use of >>>>>>> _FILE_OFFSET_BITS=64 >>>>>>> is going to cause ostream.o to bind to various 64-bit versions of >>>>>>> some >>>>>>> functions. Based on my very hazy memory of my days in file system >>>>>>> code, >>>>>>> this binding is going to affect ostream.o only unless ostream.o is >>>>>>> writing to some file with the foo64() function and some other >>>>>>> code is >>>>>>> also writing to the same file at the same time with foo(). However, >>>>>>> if we >>>>>>> have two different functions writing to the same open file at the >>>>>>> same >>>>>>> time, then we have bigger issues. :-) >>>>>>> >>>>>>> I'm good with these changes now. I agree that solving the problem of >>>>>>> setting _FILE_OFFSET_BITS=64 for the entire VM build doesn't have to >>>>>>> solved right now. >>>>>> >>>>>> I think this change is really scary, setting _FILE_OFFSET_BITS=64 >>>>>> when >>>>>> compiling ostream.cpp will effectively cause the headers to swap out >>>>>> the implementations of the f[open,tell,seek] to 64 bit versions for >>>>>> all headers that are included and inlined in ostream.cpp. >>>>>> Other parts of the code using the same headers will see different >>>>>> versions of the functions with different parameters due to off_t >>>>>> changing sizes. >>>>> The change is currently effective for Linux and Solaris if you look at >>>>> the file directories. Nothing changed for Windows and BSD, as they >>>>> don't >>>>> need such change. >>>> >>>> Right. >>>> But if you use my suggested approach you would need to change calls to >>>> fopen() in ostream.cpp to fopen_pd where >>>> >>>> if (linux || solaris) && 32bit >>>> #define fopen_pd fopen64 >>>> else >>>> #define fopen_pd fopen >>> I saw what you suggested. >>> >>> But, what's the benefit of doing so, or, what's the severe problem about >>> the current changeset? I don't see that one's better than another. >> >> Because I want to avoid the situation I described in my mail to Dan. >> I wrote a small code example to describe the situation I want to avoid: >> https://gist.github.com/anonymous/b690f753401504f1f096 >> >> I know that we currently don't have calls to fopen() from .hpp files >> in the VM but if anyone ever was to add one strange, hard to track >> down bugs will happen. >> >>> >>>> >>>>>> >>>>>> I think that what we should do is to use the "Transitional >>>>>> compilation >>>>>> environment" detailed in ?2.6.2 in largefiles.pdf and change the >>>>>> calls >>>>>> in ostream.cpp to use the appropriate f[open,tell,seek]64 functions >>>>>> directly. I feel this is especially important at this late stage in >>>>>> the release to make an explicit change instead of setting a #define >>>>>> which has propagating side-effects. >>>>> How do you see "propagating side-effects" and to where? >>>> >>>> _FILE_OFFSET_BITS=64 changes the definition of fopen for every file >>>> including stdio.h. >>> The current implementation has a limited affected range (i.e. ostream.o >>> for Linux and Solaris). No more no less. >>> >>> How do you see this flag will affect every file? >> >> Right, I meant that it will transitively affect all files included >> from ostream.cpp, as I mentioned above. >> >> /Mikael >> >>>> >>>> It's confusing when a call to "fopen()" in one file calls the 64 bit >>>> version and in other files it doesn't. >>>> >>>> How will this work with precompiled headers? Which version of fopen >>>> will be in the precompiled header file? >>>> >>>>>> >>>>>> As Tao mentioned this will require us to handle the fact that >>>>>> there is >>>>>> no fopen64() call on Windows, that the CRT fopen() already seems to >>>>>> handle large files and that ftell64() and fseek64() have slightly >>>>>> different names on Windows. I don't think this is a large hurdle >>>>>> and I >>>>>> think we know how to solve this problem. >>>>> As I said, nothing was changed for Windows code. >>>> >>>> No, but to be consistent you'd need to update the ftell* fseek* to use >>>> 64 bit versions, right? >>>> >>>> /Mikael >>>> >>>>>> >>>>>> /Mikael >>>>>> >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> On 6/4/13 6:06 PM, Tao Mao wrote: >>>>>>>> Thank you for review, Dan. >>>>>>>> >>>>>>>> I'll try to answer as much as I can. Please see inline. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Tao >>>>>>>> >>>>>>>> On 6/4/13 4:35 PM, Daniel D. Daugherty wrote: >>>>>>>>> > http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>> >>>>>>>>> Tao, >>>>>>>>> >>>>>>>>> I think the lack of response to this review request is the >>>>>>>>> absolutely >>>>>>>>> strange nature of these changes. And I thought I put out some >>>>>>>>> weird >>>>>>>>> code reviews... :-) >>>>>>>>> >>>>>>>>> make/linux/makefiles/vm.make >>>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Linux. Nothing >>>>>>>>> obvious in this webrev about what this will mean so I took a >>>>>>>>> look at src/share/vm/utilities/ostream.{c,h}pp and I see no >>>>>>>>> use of _FILE_OFFSET_BITS in either of those source files. >>>>>>>>> >>>>>>>>> Must be in the source files somewhere, but I can't find any >>>>>>>>> use of _FILE_OFFSET_BITS in the entire hotspot source base. >>>>>>>>> >>>>>>>>> make/solaris/makefiles/vm.make >>>>>>>>> Build ostream.o with _FILE_OFFSET_BITS==64 on Solaris. >>>>>>>>> >>>>>>>>> OK, I looked for _FILE_OFFSET_BITS in /usr/include on my >>>>>>>>> Solaris box. Lots of references, but nothing that helps me >>>>>>>>> understand what you're doing here. >>>>>>>>> src/os/solaris/vm/os_solaris.inline.hpp >>>>>>>>> The addition of _FILE_OFFSET_BITS==64 means that the >>>>>>>>> os::readdir() function will use the safer, multi-threaded >>>>>>>>> version of readdir_r(). Seems fine to me. >>>>>>>>> >>>>>>>>> Here's what I need to know: >>>>>>>>> >>>>>>>>> - what effect does _FILE_OFFSET_BITS have on building >>>>>>>>> ostream.{c,h}pp? >>>>>>>> _FILE_OFFSET_BITS is set to be picked by c++ compiler. >>>>>>>> >>>>>>>> For why we need to set _FILE_OFFSET_BITS==64 in this case, please >>>>>>>> refer to the following document >>>>>>>> >>>>>>>> This Sun White Paper >>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>> summarizes the usage of the flags on solaris (page "5-26"). And, it >>>>>>>> should apply to Linux the same way as was agreed across platforms >>>>>>>> (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-limit.html). >>>>>>>> >>>>>>>>> - if ostream.o has one idea about the value of _FILE_OFFSET_BITS >>>>>>>>> what happens if another part of the VM has a different idea >>>>>>>>> about >>>>>>>>> the value of _FILE_OFFSET_BITS? >>>>>>>> _FILE_OFFSET_BITS is not set for other particular effects, but for >>>>>>>> extending the ability to deal with large files in ostream.{c,h}pp. >>>>>>>> So, >>>>>>>> if other files have a different idea about _FILE_OFFSET_BITS, they >>>>>>>> can't deal with large files. No more no less. >>>>>>>> >>>>>>>>> >>>>>>>>> I saw this in the post to the Runtime alias: >>>>>>>>> >>>>>>>>> > Included runtime dev to see whether they have some idea to >>>>>>>>> handle >>>>>>>>> > the compilation choices. >>>>>>>>> >>>>>>>>> And I still have no idea what you're asking here? What compilation >>>>>>>>> choices? Are you asking about your Makefile changes? Are asking >>>>>>>>> about defining _FILE_OFFSET_BITS for the entire build instead of >>>>>>>>> just one object (ostream.o)? Are you worried that this VM is going >>>>>>>>> to have mis-matched pieces and be unstable? >>>>>>>> "Are asking about defining _FILE_OFFSET_BITS for the entire build >>>>>>>> instead of just one object (ostream.o)?" is my main question I >>>>>>>> originally tried to ask. >>>>>>>> >>>>>>>>> >>>>>>>>> So I reviewed it, but I definitely can't approve it without more >>>>>>>>> info. I realize that you're up against the RDP2 limit, but this >>>>>>>>> change has too many open questions (for now)... >>>>>>>>> >>>>>>>>> BTW, it is not at all clear whether Win32 will be able to write a >>>>>>>>> 2GB+ >>>>>>>>> GC log or not. The conversation below didn't help me at all. >>>>>>>> I used a jdk7 (just any) to successfully generate a log file larger >>>>>>>> than 4GB. So, it shouldn't be a problem for Win32. >>>>>>>>> >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 6/4/13 5:03 PM, Tao Mao wrote: >>>>>>>>>> Since the changeset touched makefiles, I've included >>>>>>>>>> build-dev at openjdk.java.net . >>>>>>>>>> >>>>>>>>>> I need to push the hsx24 bug asap. Please review it. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Tao >>>>>>>>>> >>>>>>>>>> On 6/4/13 2:37 PM, Tao Mao wrote: >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> Need reviews to catch RDP2. >>>>>>>>>>> >>>>>>>>>>> The current webrev is a working solution to all platforms, >>>>>>>>>>> Linux, >>>>>>>>>>> Windows, and Solaris. >>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> Tao >>>>>>>>>>> >>>>>>>>>>> On 5/30/13 10:21 AM, Tao Mao wrote: >>>>>>>>>>>> Included runtime dev to see whether they have some idea to >>>>>>>>>>>> handle >>>>>>>>>>>> the compilation choices. >>>>>>>>>>>> >>>>>>>>>>>> For now, it's been verified that the fix is functionally >>>>>>>>>>>> sufficient. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> Tao >>>>>>>>>>>> >>>>>>>>>>>> On 5/29/13 5:27 PM, Tao Mao wrote: >>>>>>>>>>>>> Thank you, Mikael. >>>>>>>>>>>>> >>>>>>>>>>>>> Please see inline. >>>>>>>>>>>>> >>>>>>>>>>>>> Reviewers, please review it based on the following new >>>>>>>>>>>>> observation. >>>>>>>>>>>>> >>>>>>>>>>>>> Tao >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/27/13 2:05 AM, Mikael Gerdin wrote: >>>>>>>>>>>>>> Tao, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2013-05-25 02:19, Tao Mao wrote: >>>>>>>>>>>>>>> 7ux bug >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> webrev: >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~tamao/7122222/webrev.00/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> changeset: >>>>>>>>>>>>>>> (1) make -D_FILE_OFFSET_BITS=64 only available to generating >>>>>>>>>>>>>>> ostream.o >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Why conservative rather than making -D_FILE_OFFSET_BITS=64 >>>>>>>>>>>>>>> globally >>>>>>>>>>>>>>> applicable? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Global setting of -D_FILE_OFFSET_BITS=64 on linux works >>>>>>>>>>>>>>> fine; >>>>>>>>>>>>>>> however, >>>>>>>>>>>>>>> there are at least five code conflicts if introducing the >>>>>>>>>>>>>>> flag >>>>>>>>>>>>>>> globally >>>>>>>>>>>>>>> to Solaris. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> One was resolved as in os_solaris.inline.hpp, but the rest >>>>>>>>>>>>>>> four >>>>>>>>>>>>>>> files >>>>>>>>>>>>>>> had conflicts deep in c library. Even if they are excluded >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>> setting >>>>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64, the compiled VM is corrupted. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (2) For now, no Windows solution. >>>>>>>>>>>>>>> I haven't found any clean solution for solving this >>>>>>>>>>>>>>> problem on >>>>>>>>>>>>>>> Windows. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This seems like an insufficient fix if you can't make it >>>>>>>>>>>>>> work on >>>>>>>>>>>>>> all platforms. I tried building with "-D_LARGEFILE64_SOURCE >>>>>>>>>>>>>> -D_FILE_OFFSET_BITS=64" ons Solaris and hit an #error in >>>>>>>>>>>>>> libelf.h saying it wasn't supported so I understand your >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> there. >>>>>>>>>>>>> Yes, that's my grief :( you touched them, a bunch of them. >>>>>>>>>>>>> That's >>>>>>>>>>>>> why I chose to apply the flag only to the files >>>>>>>>>>>>> (ostream.cpp and >>>>>>>>>>>>> ostream.hpp) I want the effect. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Instead I suggest that you use the compatibility API >>>>>>>>>>>>>> described >>>>>>>>>>>>>> in lf64(5) on Solaris. This API consists of fopen64, ftell64 >>>>>>>>>>>>>> and >>>>>>>>>>>>>> friends and is exposed when "-D_LARGEFILE64_SOURCE" is set. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The same "-D_LARGEFILE64_SOURCE" is available on Linux and >>>>>>>>>>>>>> has >>>>>>>>>>>>>> the added advantage of not changing any existing symbols and >>>>>>>>>>>>>> therefore we can set the define for all files instead of just >>>>>>>>>>>>>> ostream >>>>>>>>>>>>>> >>>>>>>>>>>>>> This approach has the added advantage that it more closely >>>>>>>>>>>>>> resembles the changes which will be needed for Windows >>>>>>>>>>>>>> anyway. >>>>>>>>>>>>>> Those changes would consist of changing calls to >>>>>>>>>>>>>> ftell/fseek to >>>>>>>>>>>>>> 64-bit versions and changing fopen to fopen64 on >>>>>>>>>>>>>> Solaris/Linux. >>>>>>>>>>>>> Both ways have pros and cons. The current implementation >>>>>>>>>>>>> excludes >>>>>>>>>>>>> the usage of fopen64, providing portability (since there's no >>>>>>>>>>>>> fopen64 for Windows). Meanwhile, I understand your suggestion >>>>>>>>>>>>> provides other benefits. >>>>>>>>>>>>> >>>>>>>>>>>>> This Sun White Paper >>>>>>>>>>>>> (http://unix.business.utah.edu/doc/os/solaris/misc/largefiles.pdf) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> summarizes >>>>>>>>>>>>> the usage of the flags on solaris (Page 5-26). And, it should >>>>>>>>>>>>> apply to Linux the same way as was agreed across platforms. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Since there is no fopen64 on Windows it seems that the >>>>>>>>>>>>>> default >>>>>>>>>>>>>> fopen already supports large files. >>>>>>>>>>>>> I tested, and you are correct that the 32-bit VM on Windows >>>>>>>>>>>>> can >>>>>>>>>>>>> write beyond 2GB (and beyond 4GB). Thank you, it's solved >>>>>>>>>>>>> "half >>>>>>>>>>>>> of my problem" :) >>>>>>>>>>>>>> >>>>>>>>>>>>>> /Mikael >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> test: >>>>>>>>>>>>>>> (1) Ability to write over 2g file for 32-bit builds were >>>>>>>>>>>>>>> verified on the >>>>>>>>>>>>>>> following configurations. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Linux * i586 >>>>>>>>>>>>>>> Solaris * i586 >>>>>>>>>>>>>>> Solaris * sparc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (2) Need a JPRT test for sanity check. >>>>>>>>>>>>>> >>>>>>>>> >>>>>>> >> > From thomas.schatzl at oracle.com Thu Jun 6 19:25:40 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 6 Jun 2013 12:25:40 -0700 (PDT) Subject: CMS parallel initial mark In-Reply-To: <51B0C180.4020106@oracle.com> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> Message-ID: <1370546740.2445.31.camel@cirrus> Hi all, On Thu, 2013-06-06 at 10:06 -0700, Jon Masamitsu wrote: > On 6/6/13 9:02 AM, Thomas Schatzl wrote: > > Hi, > > > > On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: > >> Hi, > >> > >> I'd like to have the following contributed if it makes sense. > >> > >> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): > >> > >> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ > >> > >> that implements a parallel version of the initial mark phase of the > >> CMS collector. It's relatively a straightforward parallelization of > >> the existing single-threaded code. With the above patch, I see about > >> ~3-6x speedup in the initial mark pause times. > > had a look at this patch in the version at > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.01/ > > > > Looks good so far, some comments: > > - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new > > Par_MarkRefsIntoClosure could be refactored slightly as they have > > exactly the same member variables. Not sure how this situation is > > handled in other code though, and what others (Jon) think. > Thomas, > > If you don't mind I'd like to keep this changeset to a minimum so > not do any additional refactoring. That's a good suggestion but > since this is the first sizable contribution I'm sponsoring, simpler > is better for me. Okay. It would be a tiny additional change though, which has only been enabled by the addition of the Par_MarkRefsIntoClosure, and of course depends on whether the old serial initial marking code is kept. (Maybe it is also possible to remove the underscore in the class name? It's not exactly customary to have class names with underscores in the Hotspot code - I think) > > > > - in concurrentMarkSweepGeneration.cpp in > > CMSCollector::checkpointRootsInitialWork() the new code seems a little > > strange to me. I.e. roughly it does the following: > > > > [...] > > if (CMSParallelInitialMarkEnabled) { > > [ set up parallel stuff ] > > if (n_workers > 1) { > > // do parallel thread > > } else { > > // do serial work using the current thread > > } > > } else { > > // do serial work > > } > > > > it does not feel good to have two serial paths here; maybe use the > > serial/original code if n_workers == 1 and guarantee(n_workers > 1, ) in > > the first part of the if? Or look if the old serial path could be > > removed? (or the new one). > > Is the inner serial path one thread executing a single chunk that > does all the work (i.e., executes the parallel code but by only > one thread)? Yes, that is in my opinion a little awkward. It would be nice if we had only one serial initial marking code path to simplify debugging/maintenance if there are issues. Removal of one or the other (if so) probably depends on how different the performance between the two paths is. > > - the comments for at least CMSCollector:: > > initialize_sequential_subtasks_for_young_gen_rescan() and > > CMSCollector::merge_survivor_plab_arrays() should be reworked - they > > only mention parallel rescan as entry points. I recommend fixing the comments though - wrong comments are not exactly helpful, although it's just an omission here. > > - the change in the assert in > > CMSCollector::merge_survivor_plab_arrays() looks a little odd now. I > > mean, looking at that assert in isolation poses the question why is this > > method only called when doing initial mark in parallel? It somehow seems > > that the same work is done somewhere else just for the initial mark and > > serial case, which just looks strange to me. That's just an observation > > from me. > > > > assert(_collectorState == FinalMarking || > > (CMSParallelInitialMarkEnabled && _collectorState == > > InitialMarking), "Error"); > > > > - in CMSCollector::CMSCollector() there is this unconditional setup of > > a few data structures used for parallel processing if any of the > > CMSParallel* switches is enabled. > I'm not sure what code this is. What lines in the webrev? I hit the send button too eagerly (was at leaving the office), sorry. I meant ConcurrentMarkSweepGeneration.cpp:727, and was thinking about removing on of the serial initial marking paths. If the old serial version (old code) is kept, this condition needs to be adapted, as data structures within the if-block started at that line are only used in the parallel case. But that was just a thought as I am not sure how far you are with testing it. Thanks, Thomas From yamauchi at google.com Thu Jun 6 22:13:46 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 6 Jun 2013 15:13:46 -0700 Subject: CMS parallel initial mark In-Reply-To: <51AFD40B.8020300@oracle.com> References: <51A6AD6D.8010309@oracle.com> <51AFD40B.8020300@oracle.com> Message-ID: On Wed, Jun 5, 2013 at 5:12 PM, Jon Masamitsu wrote: > Hiroshi, > > I've looked at the parallel initial mark changeset so far. Looks good. > Would you says that the FlexibleWorkerGang > > // Parallel initial mark task > class CMSParInitialMarkTask: public CMSParMarkTask { > FlexibleWorkGang* _workers; > > is unused above just as it is unnecessary in the constructor for > CMSParRemarkTask? > Yes, it is unused and unnecessary. I'll remove it. > > I'll look at the second changeset in a bit. > > Jon > > > > > On 5/30/2013 10:56 AM, Hiroshi Yamauchi wrote: > >> Thanks, Jon. Please let me know when you know more. >> >> On Wed, May 29, 2013 at 6:37 PM, Jon Masamitsu >> wrote: >> >>> Hiroshi, >>> >>> I'm still reviewing the changes but so far this looks >>> very promising. I've patched you changes into a repository >>> and started running a few tests. I've turned on >>> >>> -XX:+**CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways >>> >>> Thanks. >>> >>> Jon >>> >>> >>> On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: >>> >>>> Hi, >>>> >>>> I'd like to have the following contributed if it makes sense. >>>> >>>> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc >>>> repo): >>>> >>>> http://cr.openjdk.java.net/~**hiroshi/webrevs/** >>>> cmsparinitmark/webrev.00/ >>>> >>>> that implements a parallel version of the initial mark phase of the >>>> CMS collector. It's relatively a straightforward parallelization of >>>> the existing single-threaded code. With the above patch, I see about >>>> ~3-6x speedup in the initial mark pause times. >>>> >>>> 2) Now, here's a related issue and a suggested fix/patch for it: >>>> >>>> I see that the initial mark and remark pause times sometimes spike >>>> with a large young generation. For example, under a 1 GB young gen / 3 >>>> GB heap setting, they occasionally spike up to ~500 milliseconds from >>>> the normal < 100 ms range, on my machine. As far as I can tell, this >>>> happens when the eden is fairly occupied (> 700 MB full) and not >>>> sufficiently divided up and the parallelism decreases (at the worst >>>> case it becomes almost single-threaded.) >>>> >>>> Here's a suggested patch in a separate patch: >>>> >>>> http://cr.openjdk.java.net/~**hiroshi/webrevs/edenchunks/** >>>> webrev.00/ >>>> >>>> that attempts to improve on this issue by implementing an alternative >>>> way of dividing up the eden into chunks for an increased parallelism >>>> (or better load balancing between the GC threads) for the young gen >>>> scan portion of the remark phase (and the now-parallelized initial >>>> mark phase.) It uses a CAS-based mechanism that samples the object >>>> boundaries in the eden space on the slow allocation code paths (eg. at >>>> the TLAB refill and large object allocation times) at all times. >>>> >>>> This approach is in contrast to the original mechanism that samples >>>> object boundaries in the eden space asynchronously during the preclean >>>> phase. I think the reason that the above issue happens is that when >>>> the young generation is large, a large portion of the eden space could >>>> get filled/allocated outside of the preclean phase (or a concurrent >>>> collection) and the object boundaries do not get sampled >>>> often/regularly enough. Also, it isn't very suited for the parallel >>>> initial mark because the initial mark phase isn't preceded by the >>>> preclean phase unlike the remark phase. According to the Dacapo >>>> benchmarks, this alternative sampling mechanism does not have >>>> noticeable runtime overhead despite it is engaged at all times. >>>> >>>> With this patch, I see that the (parallel) initial mark and remark >>>> pause times stay below 100 ms (no spikes) under the same setting. >>>> >>>> Both of these features/flags are disabled by default. You're welcome >>>> to handle the two patches separately. >>>> >>>> Thanks, >>>> Hiroshi >>>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 6 23:19:17 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 6 Jun 2013 16:19:17 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B00B03.1020503@oracle.com> References: <51B00B03.1020503@oracle.com> Message-ID: On Wed, Jun 5, 2013 at 9:07 PM, Jon Masamitsu wrote: > Hiroshi, > > For the sampling change. > > I appreciate that you allow for reverting to the old behavior of > sampling during precleaning but am curious about whether > you've seen an occasion where it was preferable. > I assume you are referring to an occasion where the old behavior was preferable than the new behavior. No, I haven't seen such a case. As far as I can tell, there's no noticeable runtime overhead due to the new way of sampling, and I haven't seen a case where the remark pause time was better with the old behavior. The new behavior is disabled by default just for conservatism. If it's preferred to adopt the new behavior without a flag, there's no problem with me. > http://cr.openjdk.java.net/~**hiroshi/webrevs/edenchunks/** > webrev.00/src/share/vm/gc_**implementation/**concurrentMarkSweep/** > concurrentMarkSweepGeneration.**hpp.frames.html > > 739 // This is meant to be a boolean flag, but jbyte for CAS. > 740 jbyte _eden_chunk_sampling_active; > > Other than the card table I'm used to seeing the atomic operations > on word sized variables. Would jint work as well and be simpler to > think about? > Sure. jint would be fine, too. > > Maybe more later. > > > Jon > > > > On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: > >> Hi, >> >> I'd like to have the following contributed if it makes sense. >> >> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >> >> http://cr.openjdk.java.net/~**hiroshi/webrevs/** >> cmsparinitmark/webrev.00/ >> >> that implements a parallel version of the initial mark phase of the >> CMS collector. It's relatively a straightforward parallelization of >> the existing single-threaded code. With the above patch, I see about >> ~3-6x speedup in the initial mark pause times. >> >> 2) Now, here's a related issue and a suggested fix/patch for it: >> >> I see that the initial mark and remark pause times sometimes spike >> with a large young generation. For example, under a 1 GB young gen / 3 >> GB heap setting, they occasionally spike up to ~500 milliseconds from >> the normal < 100 ms range, on my machine. As far as I can tell, this >> happens when the eden is fairly occupied (> 700 MB full) and not >> sufficiently divided up and the parallelism decreases (at the worst >> case it becomes almost single-threaded.) >> >> Here's a suggested patch in a separate patch: >> >> http://cr.openjdk.java.net/~**hiroshi/webrevs/edenchunks/**webrev.00/ >> >> that attempts to improve on this issue by implementing an alternative >> way of dividing up the eden into chunks for an increased parallelism >> (or better load balancing between the GC threads) for the young gen >> scan portion of the remark phase (and the now-parallelized initial >> mark phase.) It uses a CAS-based mechanism that samples the object >> boundaries in the eden space on the slow allocation code paths (eg. at >> the TLAB refill and large object allocation times) at all times. >> >> This approach is in contrast to the original mechanism that samples >> object boundaries in the eden space asynchronously during the preclean >> phase. I think the reason that the above issue happens is that when >> the young generation is large, a large portion of the eden space could >> get filled/allocated outside of the preclean phase (or a concurrent >> collection) and the object boundaries do not get sampled >> often/regularly enough. Also, it isn't very suited for the parallel >> initial mark because the initial mark phase isn't preceded by the >> preclean phase unlike the remark phase. According to the Dacapo >> benchmarks, this alternative sampling mechanism does not have >> noticeable runtime overhead despite it is engaged at all times. >> >> With this patch, I see that the (parallel) initial mark and remark >> pause times stay below 100 ms (no spikes) under the same setting. >> >> Both of these features/flags are disabled by default. You're welcome >> to handle the two patches separately. >> >> Thanks, >> Hiroshi >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.rutisson at oracle.com Fri Jun 7 05:21:02 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Fri, 07 Jun 2013 07:21:02 +0200 Subject: RFR (XS): 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 In-Reply-To: <51B0D916.9030006@oracle.com> References: <51AFB2AE.7040604@oracle.com> <51B0D916.9030006@oracle.com> Message-ID: <51B16DBE.4010300@oracle.com> Hi Jesper, I agree with John, the line break in psScavenge.cpp looks intentional and should probably not be changed. Also, please use gclog_or_tty->print_cr() in psAdaptiveSizePolicy.cpp rather than adding extra calls to gclog_or_tty->cr(); Bengt On 6/6/13 8:46 PM, John Cuthbertson wrote: > Hi Jesper, > > The change is psAdaptiveSizePolicy.cpp looks good. > > I don't, however, think you should remove the line break in > psScavenge.cpp. I _that_ that line break is meant to be there. The > output beginning with "avg_survived_padded_avg" is slightly indented. > Compare the other places where "AdaptiveSizeStart" is printed in > psMarkSweep.cpp and psParallelCompact.cpp. Leaving line break in will > make the output: > > AdaptiveSizeStart: 23.548 collection: 67 > avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: > 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 > tenuring_thresh: 14 target_size: 1048576 > > JohnC > > On 6/5/2013 2:50 PM, Jesper Wilhelmsson wrote: >> Hi, >> >> Can I have a couple of reviews for this really minor fix? >> >> 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 >> >> A missing line break in the adaptive size verbose logging has been >> added. >> >> When looking at this I also found that an existing line break caused >> a slightly ugly verbose output when breaking a line causing a second >> line without a identifier at the beginning. I removed that line break >> as well in this change. >> >> >> webrev: >> http://cr.openjdk.java.net/~jwilhelm/8015903/hotspot-webrev/ >> >> >> >> Output before: >> >> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 >> overflow: falseAdaptiveSizeStart: 23.548 collection: 67 >> avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: >> 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 >> tenuring_thresh: 14 target_size: 1048576 >> >> >> Output after: >> >> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 >> overflow: false >> AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: >> 728176.250000 avg_promoted_padded_avg: 223962144.000000 >> avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 >> target_size: 1048576 >> >> >> Thanks, >> /Jesper > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Fri Jun 7 08:52:08 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 7 Jun 2013 01:52:08 -0700 (PDT) Subject: Request for review (s) - 8015851 :UseAdaptiveGCBoundary is broken In-Reply-To: <51AFD701.90400@oracle.com> References: <51AF9F40.6040204@oracle.com> <51AFD701.90400@oracle.com> Message-ID: <1370595128.2671.5.camel@cirrus> Hi, On Wed, 2013-06-05 at 17:25 -0700, Jon Masamitsu wrote: > Fixed the CR # in the subject. > > On 6/5/2013 1:27 PM, Jon Masamitsu wrote: > > A recent changeset effectively removed the initialization of > > performance counters used by the option UseAdaptiveGCBoundary > > > > This changeset is built on top of the one for 8014546 and the > > webrev comes in two parts. > > > > The fix plus the test > > > > http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ Looks good. About the test: is there a way to actually test performance counter initialization? So far it only checks whether the VM crashes, and not whether initialization is okay. > > > > Some cleanups of the code > > > > http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ > > Just to make sure what this cleanup is about: move the call to initialize_work() into the constructors of the generations, and clean up some unused constructors. If so, looks okay. Hth, Thomas From bengt.rutisson at oracle.com Fri Jun 7 09:21:20 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Fri, 07 Jun 2013 11:21:20 +0200 Subject: Request for review (s) - 8015851 :UseAdaptiveGCBoundary is broken In-Reply-To: <51AFD701.90400@oracle.com> References: <51AF9F40.6040204@oracle.com> <51AFD701.90400@oracle.com> Message-ID: <51B1A610.90007@oracle.com> Hi Jon, On 6/6/13 2:25 AM, Jon Masamitsu wrote: > Fixed the CR # in the subject. > > On 6/5/2013 1:27 PM, Jon Masamitsu wrote: >> A recent changeset effectively removed the initialization of >> performance counters used by the option UseAdaptiveGCBoundary Which change broke the initialization of the performance counters? One comment: The code you add in ASPSOldGen::initialize_work() is very similar to the code in PSOldGen::initialize(). The PSOldGen::initialize() method is called from one (why not both?!?!) of the constructors for PSOldGen, which means that it is called from one of the ASPSOldGen constructors. But these constructors seem to be unused. Would it be possible to move your code from ASPSOldGen::initialize_work() in to the constructor of PSOldGen that is actually used? Thanks, Bengt >> >> This changeset is built on top of the one for 8014546 and the >> webrev comes in two parts. >> >> The fix plus the test >> >> http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ >> >> Some cleanups of the code >> >> http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ >> >> Thanks. >> >> Jon > From john.coomes at oracle.com Fri Jun 7 10:17:17 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:17:17 +0000 Subject: hg: hsx/hotspot-gc/corba: Added tag jdk8-b93 for changeset 8dc9d7ccbb2d Message-ID: <20130607101721.5412348088@hg.openjdk.java.net> Changeset: 22f5d7f261d9 Author: katleman Date: 2013-06-06 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/corba/rev/22f5d7f261d9 Added tag jdk8-b93 for changeset 8dc9d7ccbb2d ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:17:11 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:17:11 +0000 Subject: hg: hsx/hotspot-gc: 12 new changesets Message-ID: <20130607101713.A706248087@hg.openjdk.java.net> Changeset: 78852ce176db Author: jqzuo Date: 2013-05-28 20:03 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/78852ce176db 8014762: Add JMC configure option mapping to Jprt.gmk Summary: Need to add the mapping between JPRT env var and configure flag for JMC, from ALT_JMC_ZIP_DIR to --with-jmc-zip-dir (same pattern as for Javafx) Reviewed-by: tbell, erikj Contributed-by: klara.ward at oracle.com ! common/autoconf/generated-configure.sh ! common/makefiles/Jprt.gmk Changeset: c22d59e3f06e Author: pbhat Date: 2013-05-29 11:02 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/c22d59e3f06e Merge Changeset: ea6f3bf82903 Author: jqzuo Date: 2013-06-04 00:12 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/ea6f3bf82903 Merge ! common/autoconf/generated-configure.sh Changeset: 33b6df33a2b7 Author: erikj Date: 2013-05-29 13:58 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/33b6df33a2b7 8013920: Configure sets JOBS to 0 if memory is too low. Reviewed-by: tbell ! common/autoconf/build-performance.m4 ! common/autoconf/generated-configure.sh Changeset: 03e60e87d92a Author: erikj Date: 2013-05-29 14:01 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/03e60e87d92a 8013489: New build system does not run codesign on SA-related launchers on OS X Reviewed-by: sla, tbell ! common/autoconf/basics.m4 ! common/autoconf/generated-configure.sh ! common/autoconf/spec.gmk.in ! common/makefiles/MakeBase.gmk ! common/makefiles/NativeCompilation.gmk Changeset: c31e9dc1fe3d Author: erikj Date: 2013-05-31 14:07 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/c31e9dc1fe3d 8014003: New build does not handle symlinks in workspace path Reviewed-by: tbell ! common/autoconf/basics.m4 ! common/autoconf/basics_windows.m4 ! common/autoconf/generated-configure.sh Changeset: 44259699e0b5 Author: erikj Date: 2013-06-04 10:23 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/44259699e0b5 8015784: Add configure parameter --with-update-version Reviewed-by: tbell, katleman, erikj Contributed-by: tristan.yan at oracle.com ! common/autoconf/generated-configure.sh ! common/autoconf/jdk-options.m4 Changeset: db3144e1f89b Author: mduigou Date: 2013-06-04 10:36 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/db3144e1f89b 8015510: (s) Improve JTReg location detection and provide location to test/Makefile Reviewed-by: erikj ! common/autoconf/generated-configure.sh ! common/autoconf/toolchain.m4 ! common/makefiles/Main.gmk Changeset: 9b8e8098172c Author: katleman Date: 2013-06-04 11:02 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/9b8e8098172c Merge Changeset: f55734874c4f Author: katleman Date: 2013-06-04 15:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/f55734874c4f Merge ! common/autoconf/generated-configure.sh Changeset: 27c51c6e31c1 Author: katleman Date: 2013-06-05 15:20 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/27c51c6e31c1 6983966: remove lzma and upx from repository JDK8 Reviewed-by: tbell, paulk, ngthomas ! common/autoconf/generated-configure.sh ! common/makefiles/Jprt.gmk ! make/deploy-rules.gmk Changeset: 8dfb6ee04114 Author: katleman Date: 2013-06-06 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/rev/8dfb6ee04114 Added tag jdk8-b93 for changeset 27c51c6e31c1 ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:17:26 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:17:26 +0000 Subject: hg: hsx/hotspot-gc/jaxp: 6 new changesets Message-ID: <20130607101752.5B6C148089@hg.openjdk.java.net> Changeset: a7cec93e4682 Author: joehw Date: 2013-05-20 16:11 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/a7cec93e4682 8014891: Redundant setting of external access properties in setFeatures Reviewed-by: lancea ! src/com/sun/org/apache/xerces/internal/jaxp/DocumentBuilderImpl.java ! src/com/sun/org/apache/xerces/internal/jaxp/SAXParserImpl.java Changeset: 37b73984640a Author: joehw Date: 2013-05-20 23:46 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/37b73984640a 8012683: Remove unused, obsolete ObjectFactory classes Reviewed-by: lancea - src/com/sun/org/apache/xerces/internal/xinclude/ObjectFactory.java - src/com/sun/org/apache/xml/internal/serialize/ObjectFactory.java Changeset: 0765806dcc58 Author: lana Date: 2013-05-22 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/0765806dcc58 Merge Changeset: 627c265d6e0c Author: lana Date: 2013-05-29 16:57 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/627c265d6e0c Merge - src/com/sun/org/apache/xerces/internal/xinclude/ObjectFactory.java - src/com/sun/org/apache/xml/internal/serialize/ObjectFactory.java Changeset: d583a491d63c Author: lana Date: 2013-06-03 23:19 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/d583a491d63c Merge - src/com/sun/org/apache/xerces/internal/xinclude/ObjectFactory.java - src/com/sun/org/apache/xml/internal/serialize/ObjectFactory.java Changeset: 40da96cab40e Author: katleman Date: 2013-06-06 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxp/rev/40da96cab40e Added tag jdk8-b93 for changeset d583a491d63c ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:17:59 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:17:59 +0000 Subject: hg: hsx/hotspot-gc/jaxws: Added tag jdk8-b93 for changeset 7386eca865e1 Message-ID: <20130607101805.EF2744808A@hg.openjdk.java.net> Changeset: 254c53fd97ab Author: katleman Date: 2013-06-06 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jaxws/rev/254c53fd97ab Added tag jdk8-b93 for changeset 7386eca865e1 ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:21:06 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:21:06 +0000 Subject: hg: hsx/hotspot-gc/jdk: 68 new changesets Message-ID: <20130607104419.933574808B@hg.openjdk.java.net> Changeset: 93de1ab38793 Author: jchen Date: 2013-05-17 10:04 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/93de1ab38793 8003444: Fix potential NULL pointer dereference Reviewed-by: jgodinez, prr ! src/share/native/sun/java2d/cmm/lcms/cmscgats.c ! src/share/native/sun/java2d/cmm/lcms/cmslut.c Changeset: 0cec8dc2bcf8 Author: lana Date: 2013-05-22 19:35 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/0cec8dc2bcf8 Merge - make/com/sun/script/Makefile - make/sun/org/Makefile - make/sun/org/mozilla/Makefile - make/sun/org/mozilla/javascript/Makefile - src/share/classes/com/sun/script/javascript/ExternalScriptable.java - src/share/classes/com/sun/script/javascript/JSAdapter.java - src/share/classes/com/sun/script/javascript/JavaAdapter.java - src/share/classes/com/sun/script/javascript/META-INF/services/javax.script.ScriptEngineFactory - src/share/classes/com/sun/script/javascript/RhinoClassShutter.java - src/share/classes/com/sun/script/javascript/RhinoCompiledScript.java - src/share/classes/com/sun/script/javascript/RhinoScriptEngine.java - src/share/classes/com/sun/script/javascript/RhinoScriptEngineFactory.java - src/share/classes/com/sun/script/javascript/RhinoTopLevel.java - src/share/classes/com/sun/script/javascript/RhinoWrapFactory.java - src/share/classes/com/sun/script/util/BindingsBase.java - src/share/classes/com/sun/script/util/BindingsEntrySet.java - src/share/classes/com/sun/script/util/BindingsImpl.java - src/share/classes/com/sun/script/util/InterfaceImplementor.java - src/share/classes/com/sun/script/util/ScriptEngineFactoryBase.java - src/share/classes/java/time/format/DateTimeFormatSymbols.java - src/share/classes/sun/nio/cs/ext/META-INF/services/java.nio.charset.spi.CharsetProvider - test/java/lang/Thread/StackTraces.java - test/java/time/tck/java/time/format/TCKDateTimeFormatSymbols.java - test/java/time/test/java/time/format/TestDateTimeFormatSymbols.java - test/java/util/logging/bundlesearch/LoadItUp.java - test/sun/security/provider/certpath/X509CertPath/ForwardBuildCompromised.java - test/sun/security/provider/certpath/X509CertPath/ReverseBuildCompromised.java - test/sun/security/provider/certpath/X509CertPath/ValidateCompromised.java Changeset: 0208f5f12dc3 Author: jchen Date: 2013-05-23 12:16 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/0208f5f12dc3 8012629: java.lang.UnsatisfiedLinkError exception throw by getAllFonts() on MacOSX Reviewed-by: bae, prr ! make/sun/awt/FILES_c_unix.gmk ! make/sun/awt/FILES_export_unix.gmk ! make/sun/awt/mawt.gmk ! makefiles/CompileNativeLibraries.gmk ! src/macosx/native/sun/font/AWTFont.m Changeset: f24f9038e050 Author: prr Date: 2013-05-24 09:31 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/f24f9038e050 8008535: JDK7 Printing : CJK and Latin Text in a string overlap Reviewed-by: bae, jgodinez ! src/windows/classes/sun/awt/windows/WPathGraphics.java + test/java/awt/print/PrinterJob/PrintLatinCJKTest.java Changeset: f4ad2fa22474 Author: jgodinez Date: 2013-05-29 09:18 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/f4ad2fa22474 7183520: [macosx]Unable to print out the defined page for 2D_PrintingTiger/JTablePrintPageRangesTest. Reviewed-by: bae, prr ! src/macosx/classes/sun/lwawt/macosx/CPrinterJob.java Changeset: 7e2a887a069e Author: jgodinez Date: 2013-05-29 09:46 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7e2a887a069e 8012381: [macosx]Unable to print out the defined page for 2D_PrintingTiger/JTablePrintPageRangesTest Reviewed-by: jchen, prr ! src/solaris/classes/sun/print/IPPPrintService.java ! test/java/awt/print/PrinterJob/Collate2DPrintingTest.java Changeset: 8ac29ee867fd Author: lana Date: 2013-05-29 16:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/8ac29ee867fd Merge Changeset: 85df65495177 Author: mcherkas Date: 2013-05-21 03:20 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/85df65495177 7011777: JDK 6 parses html text with script tags within comments differently from previous releases Reviewed-by: alexsch Contributed-by: Dmitry Markov ! src/share/classes/javax/swing/text/html/parser/Parser.java + test/javax/swing/text/html/parser/Parser/7011777/bug7011777.java Changeset: e36d0b9ed018 Author: alitvinov Date: 2013-05-21 05:02 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/e36d0b9ed018 8005607: Recursion in J2DXErrHandler() Causes a Stack Overflow on Linux Reviewed-by: art, anthony, prr ! src/solaris/classes/sun/awt/X11/MotifDnDConstants.java ! src/solaris/classes/sun/awt/X11/MotifDnDDropTargetProtocol.java ! src/solaris/classes/sun/awt/X11/WindowPropertyGetter.java ! src/solaris/classes/sun/awt/X11/XConstants.java ! src/solaris/classes/sun/awt/X11/XDnDDragSourceProtocol.java ! src/solaris/classes/sun/awt/X11/XDnDDropTargetProtocol.java ! src/solaris/classes/sun/awt/X11/XDragSourceProtocol.java ! src/solaris/classes/sun/awt/X11/XDropTargetRegistry.java ! src/solaris/classes/sun/awt/X11/XEmbedCanvasPeer.java ! src/solaris/classes/sun/awt/X11/XErrorHandler.java + src/solaris/classes/sun/awt/X11/XErrorHandlerUtil.java ! src/solaris/classes/sun/awt/X11/XQueryTree.java ! src/solaris/classes/sun/awt/X11/XToolkit.java ! src/solaris/classes/sun/awt/X11/XTranslateCoordinates.java ! src/solaris/classes/sun/awt/X11/XWM.java ! src/solaris/classes/sun/awt/X11/XlibUtil.java ! src/solaris/classes/sun/awt/X11/generator/WrapperGenerator.java ! src/solaris/native/sun/awt/awt_GraphicsEnv.c ! src/solaris/native/sun/awt/awt_GraphicsEnv.h ! src/solaris/native/sun/awt/awt_util.c ! src/solaris/native/sun/awt/awt_util.h ! src/solaris/native/sun/java2d/opengl/GLXSurfaceData.c ! src/solaris/native/sun/java2d/x11/X11SurfaceData.c ! src/solaris/native/sun/xawt/XlibWrapper.c Changeset: 73d3bed5f8c8 Author: lana Date: 2013-05-22 17:37 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/73d3bed5f8c8 Merge - make/com/sun/script/Makefile - make/sun/org/Makefile - make/sun/org/mozilla/Makefile - make/sun/org/mozilla/javascript/Makefile - src/share/classes/com/sun/script/javascript/ExternalScriptable.java - src/share/classes/com/sun/script/javascript/JSAdapter.java - src/share/classes/com/sun/script/javascript/JavaAdapter.java - src/share/classes/com/sun/script/javascript/META-INF/services/javax.script.ScriptEngineFactory - src/share/classes/com/sun/script/javascript/RhinoClassShutter.java - src/share/classes/com/sun/script/javascript/RhinoCompiledScript.java - src/share/classes/com/sun/script/javascript/RhinoScriptEngine.java - src/share/classes/com/sun/script/javascript/RhinoScriptEngineFactory.java - src/share/classes/com/sun/script/javascript/RhinoTopLevel.java - src/share/classes/com/sun/script/javascript/RhinoWrapFactory.java - src/share/classes/com/sun/script/util/BindingsBase.java - src/share/classes/com/sun/script/util/BindingsEntrySet.java - src/share/classes/com/sun/script/util/BindingsImpl.java - src/share/classes/com/sun/script/util/InterfaceImplementor.java - src/share/classes/com/sun/script/util/ScriptEngineFactoryBase.java - src/share/classes/java/time/format/DateTimeFormatSymbols.java - src/share/classes/sun/nio/cs/ext/META-INF/services/java.nio.charset.spi.CharsetProvider - test/java/lang/Thread/StackTraces.java - test/java/time/tck/java/time/format/TCKDateTimeFormatSymbols.java - test/java/time/test/java/time/format/TestDateTimeFormatSymbols.java - test/java/util/logging/bundlesearch/LoadItUp.java - test/sun/security/provider/certpath/X509CertPath/ForwardBuildCompromised.java - test/sun/security/provider/certpath/X509CertPath/ReverseBuildCompromised.java - test/sun/security/provider/certpath/X509CertPath/ValidateCompromised.java Changeset: 6261e94e9869 Author: alexsch Date: 2013-05-23 15:52 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/6261e94e9869 8014924: JToolTip#setTipText() sometimes (very often) not repaints component. Reviewed-by: serb ! src/share/classes/javax/swing/JToolTip.java Changeset: e8cacde33d27 Author: ant Date: 2013-05-24 18:01 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/e8cacde33d27 8013437: Test sun/awt/datatransfer/SuplementaryCharactersTransferTest.java fails to compile since 8b86 Reviewed-by: alexsch ! test/sun/awt/datatransfer/SuplementaryCharactersTransferTest.java Changeset: 6b29c27d0807 Author: malenkov Date: 2013-05-24 19:41 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/6b29c27d0807 8013416: Java Bean Persistence with XMLEncoder Reviewed-by: alexsch ! src/share/classes/com/sun/beans/finder/AbstractFinder.java ! src/share/classes/com/sun/beans/finder/ConstructorFinder.java ! src/share/classes/com/sun/beans/finder/MethodFinder.java + test/java/beans/XMLEncoder/Test8013416.java Changeset: c36626831f07 Author: vkarnauk Date: 2013-05-27 12:47 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/c36626831f07 8010721: [macosx] In JDK7 the menu bar disappears when a Dialog is shown Reviewed-by: anthony, serb ! src/macosx/classes/sun/lwawt/macosx/CPlatformWindow.java ! src/macosx/native/sun/awt/AWTWindow.m Changeset: 70ac1bf74865 Author: serb Date: 2013-05-27 22:31 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/70ac1bf74865 8014726: TEST_BUG: java/awt/WMSpecificTests/Metacity/FullscreenDialogModality.java should be modified Reviewed-by: serb, anthony Contributed-by: alexander.zvegintsev at oracle.com ! test/java/awt/WMSpecificTests/Metacity/FullscreenDialogModality.java Changeset: ff1c2e379f27 Author: pchelko Date: 2013-05-28 12:37 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/ff1c2e379f27 8000422: [macosx] Views keep scrolling back to the drag position after DnD Reviewed-by: serb, anthony ! src/macosx/classes/sun/lwawt/macosx/CDropTargetContextPeer.java Changeset: 4f24a4f65a07 Author: anthony Date: 2013-05-28 16:38 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/4f24a4f65a07 7039616: java/awt/Window/TranslucentJAppletTest/TranslucentJAppletTest.java should be updated Summary: Consider the test passed if the system does not support translucency Reviewed-by: art ! test/java/awt/Window/TranslucentJAppletTest/TranslucentJAppletTest.java Changeset: 1f0628078531 Author: pchelko Date: 2013-05-29 12:10 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/1f0628078531 8009911: [macosx] SWT app freeze when going full screen using Java 7 on Mac Reviewed-by: anthony, ksrini ! src/macosx/bin/java_md_macosx.c Changeset: c8a0abc1fd2d Author: mcherkas Date: 2013-05-29 18:40 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/c8a0abc1fd2d 8014863: Line break calculations in Java 7 are incorrect. Reviewed-by: alexp, alexsch Contributed-by: Dmitry Markov ! src/share/classes/javax/swing/text/View.java + test/javax/swing/text/View/8014863/bug8014863.java Changeset: aae7b96a350e Author: lana Date: 2013-05-29 16:56 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/aae7b96a350e Merge Changeset: 3b1450ee2bb9 Author: dxu Date: 2013-05-17 12:04 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/3b1450ee2bb9 8011136: FileInputStream.available and skip inconsistencies Summary: Correct the behavior of available() and update related java specs for available() and skip() in InputStream and FileInputStream classes. Reviewed-by: alanb ! src/share/classes/java/io/FileInputStream.java ! src/share/classes/java/io/InputStream.java ! src/share/native/java/io/FileInputStream.c ! test/java/io/FileInputStream/LargeFileAvailable.java ! test/java/io/FileInputStream/NegativeAvailable.java Changeset: 0f7aaabed25f Author: weijun Date: 2013-05-18 10:15 +0800 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/0f7aaabed25f 8012261: update policytool to support java.net.HttpURLPermission Reviewed-by: mullan ! src/share/classes/sun/security/tools/policytool/PolicyTool.java ! src/share/classes/sun/security/tools/policytool/Resources.java Changeset: e8b40b034fcd Author: psandoz Date: 2013-05-15 10:15 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/e8b40b034fcd 8013334: Spliterator behavior for LinkedList contradicts Spliterator.trySplit Summary: this changeset also contains some minor, non spec, related fixes to tidy up other areas of the JavaDoc. Reviewed-by: plevart, darcy Contributed-by: John Rose , Mike Duigou , Paul Sandoz ! src/share/classes/java/util/Spliterator.java Changeset: 6bbc2816d936 Author: psandoz Date: 2013-05-15 10:25 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/6bbc2816d936 8014133: Spliterator.OfPrimitive Reviewed-by: mduigou, forax Contributed-by: Paul Sandoz , Brian Goetz ! src/share/classes/java/util/Spliterator.java Changeset: dc5cf74c8c9c Author: mduigou Date: 2013-05-17 10:36 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/dc5cf74c8c9c 8004015: Additional static and instance utils for functional interfaces. 8011010: Spec j.u.f.Predicate doesn't specify NPEs thrown by the SE8's Reference Implementation Reviewed-by: briangoetz, dholmes, chegar ! src/share/classes/java/util/function/BiConsumer.java ! src/share/classes/java/util/function/BiFunction.java ! src/share/classes/java/util/function/BiPredicate.java ! src/share/classes/java/util/function/BooleanSupplier.java ! src/share/classes/java/util/function/Consumer.java ! src/share/classes/java/util/function/DoubleBinaryOperator.java ! src/share/classes/java/util/function/DoubleConsumer.java ! src/share/classes/java/util/function/DoubleFunction.java ! src/share/classes/java/util/function/DoublePredicate.java ! src/share/classes/java/util/function/DoubleSupplier.java ! src/share/classes/java/util/function/DoubleUnaryOperator.java ! src/share/classes/java/util/function/Function.java ! src/share/classes/java/util/function/IntBinaryOperator.java ! src/share/classes/java/util/function/IntConsumer.java ! src/share/classes/java/util/function/IntFunction.java ! src/share/classes/java/util/function/IntPredicate.java ! src/share/classes/java/util/function/IntSupplier.java ! src/share/classes/java/util/function/IntUnaryOperator.java ! src/share/classes/java/util/function/LongBinaryOperator.java ! src/share/classes/java/util/function/LongConsumer.java ! src/share/classes/java/util/function/LongFunction.java ! src/share/classes/java/util/function/LongPredicate.java ! src/share/classes/java/util/function/LongSupplier.java ! src/share/classes/java/util/function/LongUnaryOperator.java ! src/share/classes/java/util/function/ObjDoubleConsumer.java ! src/share/classes/java/util/function/ObjIntConsumer.java ! src/share/classes/java/util/function/ObjLongConsumer.java ! src/share/classes/java/util/function/Predicate.java ! src/share/classes/java/util/function/Supplier.java ! src/share/classes/java/util/function/ToDoubleBiFunction.java ! src/share/classes/java/util/function/ToDoubleFunction.java ! src/share/classes/java/util/function/ToIntBiFunction.java ! src/share/classes/java/util/function/ToIntFunction.java ! src/share/classes/java/util/function/ToLongBiFunction.java ! src/share/classes/java/util/function/ToLongFunction.java ! src/share/classes/java/util/function/UnaryOperator.java Changeset: 23e75751554a Author: henryjen Date: 2013-05-09 14:44 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/23e75751554a 8006884: (fs) Add Files.list, lines and find Reviewed-by: briangoetz, mduigou Contributed-by: alan.bateman at oracle.com, henry.jen at oracle.com + src/share/classes/java/nio/file/FileTreeIterator.java ! src/share/classes/java/nio/file/FileTreeWalker.java ! src/share/classes/java/nio/file/Files.java + test/java/nio/file/Files/FaultyFileSystem.java ! test/java/nio/file/Files/PassThroughFileSystem.java + test/java/nio/file/Files/StreamTest.java Changeset: b9b26b424bfc Author: mduigou Date: 2013-05-18 18:55 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/b9b26b424bfc Merge Changeset: 08ebdb2b53cc Author: plevart Date: 2013-05-17 14:41 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/08ebdb2b53cc 8014477: (str) Race condition in String.contentEquals when comparing with StringBuffer Reviewed-by: alanb, mduigou, dholmes ! src/share/classes/java/lang/String.java + test/java/lang/String/StringContentEqualsBug.java Changeset: 6a9148865139 Author: sherman Date: 2013-05-20 11:56 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/6a9148865139 8004789: (zipfs) zip provider doesn't work correctly with file systems providers rather than the default Summary: to use Files.createTempFile(...) to create the temp file on the same fs as the targeted path. Reviewed-by: alanb, sherman Contributed-by: philippe.marschall at gmail.com ! src/share/demo/nio/zipfs/src/com/sun/nio/zipfs/ZipFileSystem.java Changeset: 1baf3d7fe2f1 Author: dholmes Date: 2013-05-21 01:17 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/1baf3d7fe2f1 8014857: Enable ergonomic VM selection in arm/jvm.cfg Reviewed-by: darcy ! src/solaris/bin/arm/jvm.cfg Changeset: 20925206aef8 Author: alanb Date: 2013-05-21 08:53 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/20925206aef8 8014892: More ProblemList.txt updates (5/2013) Reviewed-by: alanb Contributed-by: amy.lu at oracle.com ! test/ProblemList.txt Changeset: 63c7e92e5e6d Author: yhuang Date: 2013-05-20 23:03 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/63c7e92e5e6d 7074882: Locale data needs correction (Month names for Maltese language) Reviewed-by: naoto ! src/share/classes/sun/text/resources/mt/FormatData_mt.java ! test/sun/text/resources/LocaleData ! test/sun/text/resources/LocaleDataTest.java Changeset: 1fba35ef4360 Author: yhuang Date: 2013-05-21 01:50 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/1fba35ef4360 Merge Changeset: 48e8a6e0c805 Author: chegar Date: 2013-05-22 13:50 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/48e8a6e0c805 8010182: Thread safety of Thread get/setName() Reviewed-by: dholmes, alanb, mduigou ! src/share/classes/java/lang/Thread.java Changeset: 4b555b53dc57 Author: mduigou Date: 2013-05-22 09:59 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/4b555b53dc57 8014819: set max size for jtreg testvms Reviewed-by: alanb, darcy ! test/Makefile Changeset: bcfab7056195 Author: lana Date: 2013-05-22 09:57 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/bcfab7056195 Merge Changeset: 760d4187597a Author: lana Date: 2013-05-22 12:09 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/760d4187597a Merge Changeset: 50fde3eeb48c Author: naoto Date: 2013-05-22 16:43 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/50fde3eeb48c 7056126: DateFormatSymbols documentation has incorrect description about DateFormat 7083668: Sample code in ListResourceBundle is still not correct Reviewed-by: okutsu ! src/share/classes/java/text/DateFormatSymbols.java ! src/share/classes/java/util/ListResourceBundle.java Changeset: a1a8e71e130a Author: dholmes Date: 2013-05-22 20:21 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/a1a8e71e130a 8014814: (str) StringBuffer "null" is not appended Reviewed-by: alanb ! src/share/classes/java/lang/StringBuffer.java ! test/java/lang/StringBuffer/ToStringCache.java Changeset: e764bb01567e Author: darcy Date: 2013-05-22 20:03 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/e764bb01567e 8014836: Have GenericDeclaration extend AnnotatedElement Reviewed-by: abuckley, jfranck ! src/share/classes/java/lang/Class.java ! src/share/classes/java/lang/reflect/GenericDeclaration.java Changeset: 0da6485cf656 Author: nloodin Date: 2013-05-23 15:50 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/0da6485cf656 8014048: Online user guide of jconsole points incorrect link Reviewed-by: mchung, sla, jbachorik ! src/share/classes/sun/tools/jconsole/AboutDialog.java ! src/share/classes/sun/tools/jconsole/resources/messages.properties ! src/share/classes/sun/tools/jconsole/resources/messages_ja.properties ! src/share/classes/sun/tools/jconsole/resources/messages_zh_CN.properties Changeset: 3b23e3529ab3 Author: dl Date: 2013-05-23 18:34 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/3b23e3529ab3 8014076: Arrays parallel and serial sorting improvements Reviewed-by: chegar, mduigou ! src/share/classes/java/util/Arrays.java ! src/share/classes/java/util/ArraysParallelSortHelpers.java ! src/share/classes/java/util/ComparableTimSort.java ! src/share/classes/java/util/DualPivotQuicksort.java ! src/share/classes/java/util/TimSort.java ! test/java/util/Arrays/ParallelSorting.java Changeset: 6816afd70a68 Author: weijun Date: 2013-05-24 17:15 +0800 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/6816afd70a68 8014196: ktab creates a file with zero kt_vno Reviewed-by: valeriep ! src/share/classes/sun/security/krb5/internal/ktab/KeyTab.java ! src/windows/classes/sun/security/krb5/internal/tools/Ktab.java + test/sun/security/krb5/tools/KtabZero.java + test/sun/security/krb5/tools/ktzero.sh Changeset: 5e769206f036 Author: ksrini Date: 2013-05-24 17:01 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/5e769206f036 8007333: [launcher] removes multiple back slashes Reviewed-by: alanb, akhil ! src/windows/bin/cmdtoargs.c ! test/tools/launcher/Arrrghs.java Changeset: d78f91ab0e96 Author: uta Date: 2013-05-27 15:18 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/d78f91ab0e96 8014394: (fs) WatchService failing when watching \\server\$d Reviewed-by: alanb ! src/windows/classes/sun/nio/fs/WindowsConstants.java ! src/windows/classes/sun/nio/fs/WindowsNativeDispatcher.java ! src/windows/classes/sun/nio/fs/WindowsWatchService.java ! src/windows/native/sun/nio/fs/WindowsNativeDispatcher.c Changeset: 0b8dab7fec54 Author: plevart Date: 2013-05-27 09:41 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/0b8dab7fec54 7038914: VM could throw uncaught OOME in ReferenceHandler thread Summary: Catch OutOfMemoryError in reference handler thread if caused by allocation of an InterruptedException Reviewed-by: dholmes, alanb ! src/share/classes/java/lang/ref/Reference.java + test/java/lang/ref/OOMEInReferenceHandler.java Changeset: a2dc42667df3 Author: chegar Date: 2013-05-27 14:00 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/a2dc42667df3 8015439: Minor/sync/cleanup of ConcurrentHashMap Reviewed-by: chegar Contributed-by: Doug Lea
, Chris Hegarty ! src/share/classes/java/util/concurrent/ConcurrentHashMap.java Changeset: 9bbf2237071e Author: chegar Date: 2013-05-27 15:24 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/9bbf2237071e Merge Changeset: bbf6e6222726 Author: nloodin Date: 2013-05-27 17:10 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/bbf6e6222726 6470730: Disconnect button leads to wrong popup message Reviewed-by: dcubed, sla, egahlin ! src/share/classes/sun/tools/jconsole/VMPanel.java Changeset: 7d9fab5d86cd Author: jbachorik Date: 2013-05-28 15:57 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7d9fab5d86cd 8005472: com/sun/jmx/remote/NotificationMarshalVersions/TestSerializationMismatch.sh failed on windows Reviewed-by: chegar, smarks, dfuchs ! test/ProblemList.txt ! test/com/sun/jmx/remote/NotificationMarshalVersions/Client/Client.java ! test/com/sun/jmx/remote/NotificationMarshalVersions/Server/Server.java + test/com/sun/jmx/remote/NotificationMarshalVersions/TestSerializationMismatch.java - test/com/sun/jmx/remote/NotificationMarshalVersions/TestSerializationMismatch.sh Changeset: 7d7bfce34a79 Author: dsamersoff Date: 2013-05-28 18:46 +0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7d7bfce34a79 8014420: Default JDP address does not match the one assigned by IANA Summary: JDP protocol defaults changed to IANA assigned values Reviewed-by: dholmes, jbachorik, hirt Contributed-by: fweimer at redhat.com ! src/share/classes/sun/management/Agent.java ! src/share/classes/sun/management/jdp/package-info.java ! test/sun/management/jdp/JdpTest.sh Changeset: b16a8b4ae6b4 Author: robm Date: 2013-05-28 16:35 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/b16a8b4ae6b4 7038105: File.isHidden() should return true for pagefile.sys and hiberfil.sys Reviewed-by: alanb ! src/windows/native/java/io/WinNTFileSystem_md.c ! test/java/io/File/IsHidden.java Changeset: 7fa2d1dcb8f6 Author: sherman Date: 2013-05-28 10:42 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7fa2d1dcb8f6 8001750: CharsetDecoder.replacement should not be changeable except via replaceWith method Summary: to make defensive copy for set/get replacement byte array Reviewed-by: martin ! src/share/classes/java/nio/charset/Charset-X-Coder.java.template ! src/share/classes/sun/nio/cs/UTF_8.java ! src/share/classes/sun/nio/cs/ext/DoubleByte.java ! src/share/classes/sun/nio/cs/ext/HKSCS.java Changeset: b99d56d1aa3f Author: naoto Date: 2013-05-28 14:02 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/b99d56d1aa3f 6251788: (rb) PropertyResourceBundle doesn't document exceptions Reviewed-by: okutsu ! src/share/classes/java/util/PropertyResourceBundle.java Changeset: 1652a22cf6e7 Author: xuelei Date: 2013-05-28 18:47 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/1652a22cf6e7 8010815: some constructors issues in com.sun.jndi.toolkit Reviewed-by: alanb ! src/share/classes/com/sun/jndi/toolkit/ctx/Continuation.java ! src/share/classes/com/sun/jndi/toolkit/dir/LazySearchEnumerationImpl.java ! src/share/classes/com/sun/jndi/toolkit/url/GenericURLContext.java Changeset: e59d7f0f36f7 Author: ewang Date: 2013-05-28 22:22 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/e59d7f0f36f7 8009258: TEST_BUG:java/io/pathNames/GeneralWin32.java fails intermittently Reviewed-by: dxu, alanb Contributed-by: yiming.wang at oracle.com ! test/java/io/pathNames/General.java ! test/java/io/pathNames/GeneralWin32.java Changeset: bd6d3801347b Author: sla Date: 2013-05-29 09:42 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/bd6d3801347b 8015440: java/lang/management/MemoryMXBean/ResetPeakMemoryUsage.java fails with RuntimeException Summary: Make sure serial gc compacts heap every time Reviewed-by: mchung, brutisso, nloodin ! test/java/lang/management/MemoryMXBean/ResetPeakMemoryUsage.java Changeset: 2b3242a69a44 Author: alanb Date: 2013-05-29 10:24 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/2b3242a69a44 8014928: (fs) Files.readAllBytes() copies content to new array when content completely read Reviewed-by: martin ! src/share/classes/java/nio/file/Files.java Changeset: 00ad19610e75 Author: vinnie Date: 2013-05-29 14:57 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/00ad19610e75 7174966: With OCSP enabled on Java 7 get error 'Wrong key usage' with Comodo certificate Reviewed-by: xuelei ! src/share/classes/sun/security/provider/certpath/OCSPResponse.java Changeset: 5d9273a5a84e Author: lana Date: 2013-05-29 16:58 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/5d9273a5a84e Merge - test/com/sun/jmx/remote/NotificationMarshalVersions/TestSerializationMismatch.sh Changeset: 7eae7c89dab4 Author: lana Date: 2013-06-03 23:23 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7eae7c89dab4 Merge - test/com/sun/jmx/remote/NotificationMarshalVersions/TestSerializationMismatch.sh Changeset: 583e6dec1ed7 Author: erikj Date: 2013-05-29 14:01 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/583e6dec1ed7 8013489: New build system does not run codesign on SA-related launchers on OS X Reviewed-by: sla, tbell ! makefiles/CompileLaunchers.gmk Changeset: d8c97d6772cd Author: erikj Date: 2013-05-30 09:29 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/d8c97d6772cd Merge Changeset: bc3a17982aae Author: erikj Date: 2013-05-31 14:05 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/bc3a17982aae 7195481: FDS: debuginfo file for libjdwp.so is missed Reviewed-by: tbell ! make/jpda/back/Makefile ! makefiles/CompileNativeLibraries.gmk Changeset: c50add191a39 Author: katleman Date: 2013-06-04 11:03 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/c50add191a39 Merge ! makefiles/CompileNativeLibraries.gmk Changeset: 16003f414ca3 Author: katleman Date: 2013-06-04 14:11 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/16003f414ca3 8015644: makefile changes to allow integration of new features Reviewed-by: tbell, erikj, dholmes Contributed-by: amy.y.wang at oracle.com ! makefiles/Images.gmk Changeset: 691d6c6cd332 Author: katleman Date: 2013-06-05 15:25 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/691d6c6cd332 6983966: remove lzma and upx from repository JDK8 Reviewed-by: tbell, paulk, ngthomas ! make/common/Defs-windows.gmk Changeset: 7b757d567346 Author: katleman Date: 2013-06-06 09:55 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/jdk/rev/7b757d567346 Added tag jdk8-b93 for changeset 691d6c6cd332 ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:46:44 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:46:44 +0000 Subject: hg: hsx/hotspot-gc/langtools: 20 new changesets Message-ID: <20130607104929.9D6C44808D@hg.openjdk.java.net> Changeset: 0928f2cfbf8e Author: jjg Date: 2013-05-17 13:48 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/0928f2cfbf8e 6885876: add comments to javac/util/Convert.java Reviewed-by: mduigou ! src/share/classes/com/sun/tools/javac/util/Convert.java Changeset: 67cbd6d756f4 Author: jfranck Date: 2013-05-21 12:00 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/67cbd6d756f4 8013180: Qualified type reference with annotations in throws list crashes compiler Reviewed-by: jjg + test/tools/javac/annotations/typeAnnotations/8013180/QualifiedName.java Changeset: 824932ecdbc8 Author: vromero Date: 2013-05-21 11:41 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/824932ecdbc8 7177168: Redundant array copy in UnsharedNameTable Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/util/UnsharedNameTable.java Changeset: 3d9750039fff Author: vromero Date: 2013-05-21 12:17 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/3d9750039fff 7060779: test/tools/javac/diags/Example.java leaves directories in tempdir Reviewed-by: mcimadamore ! test/tools/javac/diags/Example.java Changeset: 37295244f534 Author: vromero Date: 2013-05-21 13:50 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/37295244f534 8005207: test has 2 @bug tags Reviewed-by: mcimadamore ! test/tools/doclint/RunTest.java ! test/tools/javac/5045412/Bar.java ! test/tools/javac/5045412/Foo.java ! test/tools/javac/lambda/MethodReferenceParserTest.java ! test/tools/javac/lambda/TestInvokeDynamic.java ! test/tools/javac/mandatoryWarnings/deprecated/Test.java ! test/tools/javac/mandatoryWarnings/unchecked/Test.java ! test/tools/javac/policy/test3/Test.java Changeset: 08daea43a7f8 Author: vromero Date: 2013-05-21 14:33 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/08daea43a7f8 7164114: Two jtreg tests are not run due to no file extension on the test files Reviewed-by: mcimadamore - test/tools/javac/HiddenAbstractMethod/Test + test/tools/javac/HiddenAbstractMethod/Test.java - test/tools/javac/NonAmbiguousField/Test + test/tools/javac/NonAmbiguousField/Test.java ! test/tools/javac/NonAmbiguousField/two/Child2.java Changeset: 31344e8e3343 Author: lana Date: 2013-05-22 09:59 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/31344e8e3343 Merge Changeset: 3bd22f99d408 Author: darcy Date: 2013-05-22 13:34 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/3bd22f99d408 8010680: Clarify "present" and annotation ordering in javax.lang.model Reviewed-by: abuckley, jjg ! src/share/classes/javax/lang/model/AnnotatedConstruct.java ! src/share/classes/javax/lang/model/util/Elements.java Changeset: 58329d9f6b68 Author: mcimadamore Date: 2013-05-24 15:26 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/58329d9f6b68 8014643: Parser regression in JDK 8 when compiling super.x Summary: Fixed latent bug in JavacParser.analyzeParens() Reviewed-by: jjg, vromero ! src/share/classes/com/sun/tools/javac/parser/JavacParser.java + test/tools/javac/parser/8014643/T8014643.java Changeset: 97a9b4b3e63a Author: mcimadamore Date: 2013-05-24 15:27 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/97a9b4b3e63a 8014649: Regression: bug in Resolve.resolveOperator Summary: Missing curly braces causes Resolve.findMethod to be called spuriously Reviewed-by: jjg, vromero ! src/share/classes/com/sun/tools/javac/comp/Resolve.java ! test/tools/javac/resolve/ResolveHarness.java + test/tools/javac/resolve/tests/PrimitiveBinopOverload.java Changeset: 6e5076af4660 Author: mcimadamore Date: 2013-05-24 15:27 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/6e5076af4660 8014494: javac crashes when varargs element of a method reference is inferred from the context Summary: varargs element is not refreshed after type-inference Reviewed-by: jjg, vromero ! src/share/classes/com/sun/tools/javac/comp/Attr.java ! src/share/classes/com/sun/tools/javac/comp/Check.java + test/tools/javac/lambda/TargetType73.java Changeset: 0f8e9a0e5d9a Author: darcy Date: 2013-05-24 11:26 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/0f8e9a0e5d9a 8014836: Have GenericDeclaration extend AnnotatedElement Reviewed-by: jfranck ! src/share/sample/language/model/CoreReflectionFactory.java Changeset: b391ecea538e Author: vromero Date: 2013-05-27 13:44 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/b391ecea538e 7030476: Fix conflicting use of JCTree/JCExpression Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/comp/Lower.java ! src/share/classes/com/sun/tools/javac/tree/JCTree.java ! src/share/classes/com/sun/tools/javac/tree/TreeCopier.java ! src/share/classes/com/sun/tools/javac/tree/TreeMaker.java Changeset: c6df5b20f9eb Author: vromero Date: 2013-05-28 12:46 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/c6df5b20f9eb 6970173: Debug pointer at bad position Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/comp/Lower.java + test/tools/javac/T6970173/DebugPointerAtBadPositionTest.java Changeset: d042cba65eab Author: vromero Date: 2013-05-28 17:39 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/d042cba65eab 8012333: javac, ClassFile should have a read(Path) method Reviewed-by: jjg ! src/share/classes/com/sun/tools/classfile/ClassFile.java Changeset: 92e420e9807d Author: vromero Date: 2013-05-29 10:56 +0100 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/92e420e9807d 7053059: VerifyError with double Assignment using a Generic Member of a Superclass Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/comp/TransTypes.java + test/tools/javac/T7053059/VerifyErrorWithDoubleAssignmentTest.java Changeset: d685b12b62a4 Author: jjg Date: 2013-05-29 15:34 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/d685b12b62a4 8015641: genstubs needs to cope with static interface methods Reviewed-by: ksrini ! make/tools/genstubs/GenStubs.java Changeset: 18943a1b7a47 Author: lana Date: 2013-05-29 16:59 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/18943a1b7a47 Merge - test/tools/javac/HiddenAbstractMethod/Test - test/tools/javac/NonAmbiguousField/Test Changeset: 2c5a568ee36e Author: lana Date: 2013-06-03 23:24 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/2c5a568ee36e Merge - test/tools/javac/HiddenAbstractMethod/Test - test/tools/javac/NonAmbiguousField/Test Changeset: 888386fddc09 Author: katleman Date: 2013-06-06 09:55 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/langtools/rev/888386fddc09 Added tag jdk8-b93 for changeset 2c5a568ee36e ! .hgtags From john.coomes at oracle.com Fri Jun 7 10:49:49 2013 From: john.coomes at oracle.com (john.coomes at oracle.com) Date: Fri, 07 Jun 2013 10:49:49 +0000 Subject: hg: hsx/hotspot-gc/nashorn: 44 new changesets Message-ID: <20130607105040.4551C4808E@hg.openjdk.java.net> Changeset: 80d4db063d5a Author: jlaskey Date: 2013-05-14 11:15 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/80d4db063d5a 8014512: Exclude testing and infrastructure packages from code coverage Reviewed-by: sundar Contributed-by: james.laskey at oracle.com ! make/code_coverage.xml Changeset: eeed4db61215 Author: jlaskey Date: 2013-05-14 11:16 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/eeed4db61215 Merge - src/jdk/nashorn/internal/ir/LineNumberNode.java - src/jdk/nashorn/internal/ir/Location.java - test/script/trusted/logcoverage.js Changeset: fc20983ef38e Author: attila Date: 2013-05-14 19:18 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/fc20983ef38e 8011718: binding already bound function with extra arguments fails Reviewed-by: jlaskey, sundar ! src/jdk/nashorn/internal/runtime/ScriptFunctionData.java + test/script/basic/JDK-8011718.js + test/script/basic/JDK-8011718.js.EXPECTED Changeset: f88a4818a4dc Author: lagergren Date: 2013-05-14 19:56 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/f88a4818a4dc 8014426: Original exception no longer thrown away when a finally rethrows Reviewed-by: attila, jlaskey ! src/jdk/nashorn/internal/codegen/CodeGenerator.java ! src/jdk/nashorn/internal/codegen/Lower.java ! src/jdk/nashorn/internal/ir/CatchNode.java ! src/jdk/nashorn/internal/ir/ThrowNode.java ! src/jdk/nashorn/internal/parser/Parser.java + test/script/basic/JDK-8014426.js + test/script/basic/JDK-8014426.js.EXPECTED Changeset: 64ef1aeaeb4e Author: attila Date: 2013-05-15 10:28 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/64ef1aeaeb4e 8014639: Remove debug flag from test runs Reviewed-by: hannesw, lagergren ! make/project.properties Changeset: b37eb709ae27 Author: attila Date: 2013-05-15 14:54 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/b37eb709ae27 8014646: Update the Java interop documentation in the Java Scripting Programmer's Guide Reviewed-by: jlaskey, hannesw, lagergren ! docs/JavaScriptingProgrammersGuide.html Changeset: 1eaa542cc8e2 Author: sundar Date: 2013-05-15 19:45 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/1eaa542cc8e2 8012305: Function.bind can't be called on prototype function inside constructor Reviewed-by: lagergren, attila + test/script/basic/JDK-8012305.js + test/script/basic/JDK-8012305.js.EXPECTED Changeset: 6344644b81ec Author: jlaskey Date: 2013-05-15 12:09 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/6344644b81ec 8014648: Exclude testing and infrastructure packages from code coverage, round two Reviewed-by: sundar Contributed-by: james.laskey at oracle.com ! make/code_coverage.xml ! src/jdk/nashorn/internal/runtime/options/Option.java ! src/jdk/nashorn/internal/runtime/options/Options.java - src/jdk/nashorn/internal/runtime/options/ValueOption.java ! test/script/basic/allgettersetters.js Changeset: 19e9cd9c7010 Author: attila Date: 2013-05-15 20:21 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/19e9cd9c7010 8014647: Allow class-based overrides to be initialized with a ScriptFunction Reviewed-by: hannesw, jlaskey, sundar ! src/jdk/nashorn/internal/runtime/linker/JavaAdapterBytecodeGenerator.java + test/script/basic/JDK-8014647.js + test/script/basic/JDK-8014647.js.EXPECTED Changeset: ac14a1fb0cab Author: sundar Date: 2013-05-16 14:52 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/ac14a1fb0cab 8009141: Avoid netscape.javascript.JSObject in nashorn code Reviewed-by: lagergren, hannesw + src/jdk/nashorn/api/scripting/JSObject.java ! src/jdk/nashorn/api/scripting/ScriptObjectMirror.java ! src/jdk/nashorn/internal/runtime/linker/JSObjectLinker.java - src/netscape/javascript/JSObject.java ! test/src/jdk/nashorn/api/scripting/ScriptEngineTest.java Changeset: 4c67a692ef97 Author: lagergren Date: 2013-05-16 13:44 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/4c67a692ef97 8013919: Original exception no longer thrown away when a finally rethrows Reviewed-by: jlaskey, sundar ! src/jdk/nashorn/internal/codegen/Lower.java ! src/jdk/nashorn/internal/ir/FunctionNode.java + test/script/basic/JDK-8013919.js + test/script/basic/JDK-8013919.js.EXPECTED Changeset: 98798a6336de Author: hannesw Date: 2013-05-16 19:52 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/98798a6336de 8012359: Increase code coverage in Joni Reviewed-by: jlaskey, lagergren ! make/build.xml - src/jdk/nashorn/internal/runtime/regexp/DefaultRegExp.java + src/jdk/nashorn/internal/runtime/regexp/JdkRegExp.java ! src/jdk/nashorn/internal/runtime/regexp/JoniRegExp.java ! src/jdk/nashorn/internal/runtime/regexp/RegExpFactory.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Analyser.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ArrayCompiler.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompiler.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompilerSupport.java ! src/jdk/nashorn/internal/runtime/regexp/joni/BitSet.java ! src/jdk/nashorn/internal/runtime/regexp/joni/BitStatus.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ByteCodeMachine.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ByteCodePrinter.java - src/jdk/nashorn/internal/runtime/regexp/joni/CaptureTreeNode.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Compiler.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Config.java ! src/jdk/nashorn/internal/runtime/regexp/joni/EncodingHelper.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Lexer.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Matcher.java - src/jdk/nashorn/internal/runtime/regexp/joni/NameEntry.java - src/jdk/nashorn/internal/runtime/regexp/joni/NativeMachine.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Parser.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Regex.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Region.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ScanEnvironment.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ScannerSupport.java ! src/jdk/nashorn/internal/runtime/regexp/joni/StackMachine.java ! src/jdk/nashorn/internal/runtime/regexp/joni/Syntax.java - src/jdk/nashorn/internal/runtime/regexp/joni/UnsetAddrList.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ast/CClassNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CTypeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CallNode.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ast/EncloseNode.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ast/QuantifierNode.java ! src/jdk/nashorn/internal/runtime/regexp/joni/ast/StateNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/AbstractBench.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchGreedyBacktrack.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchRailsRegs.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchSeveralRegexps.java ! src/jdk/nashorn/internal/runtime/regexp/joni/constants/OPCode.java - src/jdk/nashorn/internal/runtime/regexp/joni/constants/Reduce.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/AsciiTables.java ! src/jdk/nashorn/internal/runtime/regexp/joni/encoding/ObjPtr.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/PosixBracket.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/Ptr.java ! src/jdk/nashorn/internal/runtime/regexp/joni/exception/ErrorMessages.java ! src/jdk/nashorn/internal/runtime/regexp/joni/exception/ValueException.java + test/src/jdk/nashorn/internal/runtime/regexp/JdkRegExpTest.java + test/src/jdk/nashorn/internal/runtime/regexp/joni/JoniTest.java Changeset: aa1b6e8c51a0 Author: jlaskey Date: 2013-05-17 14:30 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/aa1b6e8c51a0 8012694: Smoke test fail: Windows JDK-8008554.js - access denied ("java.io.FilePermission" "//C/aurora/sandbox/nashorn~source/test/script/basic/NASHORN-99.js" "read") Reviewed-by: jlaskey Contributed-by: konstantin.shefov at oracle.com Changeset: a92be4c0063b Author: jlaskey Date: 2013-05-17 16:12 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/a92be4c0063b Merge - src/jdk/nashorn/internal/runtime/regexp/DefaultRegExp.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompiler.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompilerSupport.java - src/jdk/nashorn/internal/runtime/regexp/joni/CaptureTreeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/NameEntry.java - src/jdk/nashorn/internal/runtime/regexp/joni/NativeMachine.java - src/jdk/nashorn/internal/runtime/regexp/joni/UnsetAddrList.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CTypeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CallNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/AbstractBench.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchGreedyBacktrack.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchRailsRegs.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchSeveralRegexps.java - src/jdk/nashorn/internal/runtime/regexp/joni/constants/Reduce.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/AsciiTables.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/PosixBracket.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/Ptr.java - src/netscape/javascript/JSObject.java Changeset: 1d5a8f1f416e Author: jlaskey Date: 2013-05-17 16:44 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/1d5a8f1f416e 8014823: Reprise - Smoke test fail: Windows JDK-8008554.js - access denied ("java.io.FilePermission" "//C/aurora/sandbox/nashorn~source/test/script/basic/NASHORN-99.js" "read") Reviewed-by: jlaskey Contributed-by: konstantin.shefov at oracle.com ! test/script/basic/JDK-8008554.js Changeset: 92164a5742db Author: lagergren Date: 2013-05-20 16:38 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/92164a5742db 8006069: Range analysis first iteration, runtime specializations Reviewed-by: jlaskey, sundar ! src/jdk/nashorn/internal/codegen/CompilationPhase.java ! src/jdk/nashorn/internal/codegen/Compiler.java ! src/jdk/nashorn/internal/codegen/CompilerConstants.java ! src/jdk/nashorn/internal/codegen/MethodEmitter.java + src/jdk/nashorn/internal/codegen/RangeAnalyzer.java + src/jdk/nashorn/internal/codegen/types/Range.java ! src/jdk/nashorn/internal/codegen/types/Type.java ! src/jdk/nashorn/internal/ir/BinaryNode.java ! src/jdk/nashorn/internal/ir/FunctionNode.java ! src/jdk/nashorn/internal/ir/LexicalContext.java ! src/jdk/nashorn/internal/ir/Node.java ! src/jdk/nashorn/internal/ir/Symbol.java ! src/jdk/nashorn/internal/runtime/CompiledFunction.java ! src/jdk/nashorn/internal/runtime/FinalScriptFunctionData.java ! src/jdk/nashorn/internal/runtime/RecompilableScriptFunctionData.java ! src/jdk/nashorn/internal/runtime/ScriptEnvironment.java ! src/jdk/nashorn/internal/runtime/ScriptFunctionData.java ! src/jdk/nashorn/internal/runtime/resources/Options.properties + test/script/basic/ranges_disabled.js + test/script/basic/ranges_disabled.js.EXPECTED + test/script/basic/ranges_enabled.js + test/script/basic/ranges_enabled.js.EXPECTED + test/script/basic/ranges_payload.js Changeset: b558e19d5de5 Author: sundar Date: 2013-05-20 23:04 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/b558e19d5de5 8014909: ant test compilation error with JoniTest.java Reviewed-by: jlaskey ! make/build.xml Changeset: 1fd18f40ab52 Author: attila Date: 2013-05-20 21:25 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/1fd18f40ab52 8014797: rename Java.toJavaArray/toJavaScriptArray to Java.to/from, respectively. Reviewed-by: jlaskey, sundar ! docs/JavaScriptingProgrammersGuide.html ! docs/source/javaarray.js ! src/jdk/nashorn/api/scripting/resources/engine.js ! src/jdk/nashorn/internal/objects/NativeJava.java ! src/jdk/nashorn/internal/runtime/resources/Messages.properties ! test/script/basic/NASHORN-556.js ! test/script/basic/javaarrayconversion.js ! test/script/currently-failing/logcoverage.js ! test/script/trusted/NASHORN-638.js ! test/script/trusted/NASHORN-653.js Changeset: e955e64fd15d Author: lana Date: 2013-05-22 09:59 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/e955e64fd15d Merge Changeset: 833a9a584b64 Author: attila Date: 2013-05-21 13:40 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/833a9a584b64 8014953: Have NativeJavaPackage throw a ClassNotFoundException when invoked Reviewed-by: jlaskey, sundar ! src/jdk/nashorn/internal/runtime/NativeJavaPackage.java + test/script/basic/JDK-8014953.js + test/script/basic/JDK-8014953.js.EXPECTED Changeset: 288ff54da2a5 Author: jlaskey Date: 2013-05-21 10:17 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/288ff54da2a5 8014827: readLine should accept a prompt as an argument Reviewed-by: sundar, hannesw Contributed-by: james.laskey at oracle.com ! src/jdk/nashorn/internal/runtime/ScriptingFunctions.java Changeset: 07cefc062032 Author: sundar Date: 2013-05-22 16:39 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/07cefc062032 8008947: ScriptEnvironment ctor should be public Reviewed-by: lagergren, attila ! .hgignore ! src/jdk/nashorn/internal/runtime/ScriptEnvironment.java Changeset: 66685c69bdb3 Author: sundar Date: 2013-05-22 19:33 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/66685c69bdb3 8014735: Typed Array, BYTES_PER_ELEMENT should be a class property Reviewed-by: lagergren, jlaskey ! src/jdk/nashorn/internal/objects/ArrayBufferView.java ! src/jdk/nashorn/internal/objects/NativeFloat32Array.java ! src/jdk/nashorn/internal/objects/NativeFloat64Array.java ! src/jdk/nashorn/internal/objects/NativeInt16Array.java ! src/jdk/nashorn/internal/objects/NativeInt32Array.java ! src/jdk/nashorn/internal/objects/NativeInt8Array.java ! src/jdk/nashorn/internal/objects/NativeUint16Array.java ! src/jdk/nashorn/internal/objects/NativeUint32Array.java ! src/jdk/nashorn/internal/objects/NativeUint8Array.java ! src/jdk/nashorn/internal/objects/NativeUint8ClampedArray.java + test/script/basic/JDK-8014735.js + test/script/basic/JDK-8014735.js.EXPECTED ! test/script/basic/NASHORN-377.js Changeset: 8f7553df4503 Author: hannesw Date: 2013-05-22 16:43 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/8f7553df4503 8010804: Review long and integer usage conventions Reviewed-by: jlaskey, sundar ! src/jdk/nashorn/internal/codegen/FoldConstants.java ! src/jdk/nashorn/internal/objects/NativeArray.java ! src/jdk/nashorn/internal/objects/NativeDate.java ! src/jdk/nashorn/internal/runtime/JSType.java + test/script/basic/JDK-8010804.js + test/script/basic/JDK-8010804.js.EXPECTED Changeset: 1c1453863ea8 Author: attila Date: 2013-05-23 12:01 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/1c1453863ea8 8015267: Allow conversion of JS arrays to Java List/Deque Reviewed-by: lagergren, sundar ! make/build.xml ! src/jdk/nashorn/internal/objects/NativeJava.java + src/jdk/nashorn/internal/runtime/ListAdapter.java ! src/jdk/nashorn/internal/runtime/linker/InvokeByName.java ! src/jdk/nashorn/internal/runtime/resources/Messages.properties + test/script/basic/JDK-8015267.js + test/script/basic/JDK-8015267.js.EXPECTED Changeset: f7eb4436410e Author: lagergren Date: 2013-05-23 13:10 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/f7eb4436410e 8012083: Array literal constant folding issue Reviewed-by: attila, jlaskey ! src/jdk/nashorn/internal/codegen/CodeGenerator.java ! src/jdk/nashorn/internal/codegen/FinalizeTypes.java ! src/jdk/nashorn/internal/codegen/FoldConstants.java + test/script/basic/JDK-8012083.js + test/script/basic/JDK-8012083.js.EXPECTED Changeset: 704bc91a0c41 Author: attila Date: 2013-05-23 13:36 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/704bc91a0c41 8015278: Revert accidental changes to build.xml Reviewed-by: jlaskey, lagergren ! make/build.xml Changeset: 8af550dee961 Author: jlaskey Date: 2013-05-23 09:49 -0300 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/8af550dee961 Merge Changeset: 6fc7b51e83d6 Author: lagergren Date: 2013-05-23 15:51 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/6fc7b51e83d6 8012522: Clean up lexical contexts - split out stack based functionality in CodeGenerator and generify NodeVisitors based on their LexicalContext type to avoid casts Reviewed-by: attila, jlaskey ! src/jdk/nashorn/internal/codegen/Attr.java ! src/jdk/nashorn/internal/codegen/CodeGenerator.java + src/jdk/nashorn/internal/codegen/CodeGeneratorLexicalContext.java ! src/jdk/nashorn/internal/codegen/CompilationPhase.java ! src/jdk/nashorn/internal/codegen/FinalizeTypes.java ! src/jdk/nashorn/internal/codegen/FoldConstants.java ! src/jdk/nashorn/internal/codegen/Lower.java ! src/jdk/nashorn/internal/codegen/RangeAnalyzer.java ! src/jdk/nashorn/internal/codegen/SharedScopeCall.java ! src/jdk/nashorn/internal/codegen/Splitter.java ! src/jdk/nashorn/internal/codegen/WeighNodes.java ! src/jdk/nashorn/internal/ir/AccessNode.java ! src/jdk/nashorn/internal/ir/BinaryNode.java ! src/jdk/nashorn/internal/ir/Block.java ! src/jdk/nashorn/internal/ir/BreakNode.java ! src/jdk/nashorn/internal/ir/CallNode.java ! src/jdk/nashorn/internal/ir/CaseNode.java ! src/jdk/nashorn/internal/ir/CatchNode.java ! src/jdk/nashorn/internal/ir/ContinueNode.java ! src/jdk/nashorn/internal/ir/EmptyNode.java ! src/jdk/nashorn/internal/ir/ExecuteNode.java ! src/jdk/nashorn/internal/ir/ForNode.java ! src/jdk/nashorn/internal/ir/FunctionNode.java ! src/jdk/nashorn/internal/ir/IdentNode.java ! src/jdk/nashorn/internal/ir/IfNode.java ! src/jdk/nashorn/internal/ir/IndexNode.java ! src/jdk/nashorn/internal/ir/LabelNode.java ! src/jdk/nashorn/internal/ir/LexicalContextNode.java ! src/jdk/nashorn/internal/ir/LiteralNode.java ! src/jdk/nashorn/internal/ir/Node.java ! src/jdk/nashorn/internal/ir/ObjectNode.java ! src/jdk/nashorn/internal/ir/PropertyNode.java ! src/jdk/nashorn/internal/ir/ReturnNode.java ! src/jdk/nashorn/internal/ir/RuntimeNode.java ! src/jdk/nashorn/internal/ir/SplitNode.java ! src/jdk/nashorn/internal/ir/SwitchNode.java ! src/jdk/nashorn/internal/ir/TernaryNode.java ! src/jdk/nashorn/internal/ir/ThrowNode.java ! src/jdk/nashorn/internal/ir/TryNode.java ! src/jdk/nashorn/internal/ir/UnaryNode.java ! src/jdk/nashorn/internal/ir/VarNode.java ! src/jdk/nashorn/internal/ir/WhileNode.java ! src/jdk/nashorn/internal/ir/WithNode.java ! src/jdk/nashorn/internal/ir/debug/JSONWriter.java ! src/jdk/nashorn/internal/ir/debug/PrintVisitor.java ! src/jdk/nashorn/internal/ir/visitor/NodeOperatorVisitor.java ! src/jdk/nashorn/internal/ir/visitor/NodeVisitor.java ! src/jdk/nashorn/internal/objects/ArrayBufferView.java ! src/jdk/nashorn/internal/runtime/DebugLogger.java ! src/jdk/nashorn/internal/runtime/linker/JavaAdapterClassLoader.java ! src/jdk/nashorn/internal/runtime/linker/JavaAdapterFactory.java ! src/jdk/nashorn/internal/runtime/linker/JavaAdapterGeneratorBase.java Changeset: fdfb4edd78d6 Author: hannesw Date: 2013-05-24 13:54 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/fdfb4edd78d6 8011630: JSON parsing performance issue Reviewed-by: lagergren, sundar ! src/jdk/nashorn/internal/objects/NativeArguments.java ! src/jdk/nashorn/internal/runtime/AccessorProperty.java ! src/jdk/nashorn/internal/runtime/FindProperty.java ! src/jdk/nashorn/internal/runtime/Property.java ! src/jdk/nashorn/internal/runtime/ScriptObject.java ! src/jdk/nashorn/internal/runtime/SetMethodCreator.java ! src/jdk/nashorn/internal/runtime/UserAccessorProperty.java Changeset: 4d2eca4d4d66 Author: sundar Date: 2013-05-24 18:39 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/4d2eca4d4d66 8015354: JSON.parse should not use [[Put]] but use [[DefineOwnProperty]] instead Reviewed-by: lagergren, hannesw ! src/jdk/nashorn/internal/objects/NativeFloat32Array.java ! src/jdk/nashorn/internal/objects/NativeFloat64Array.java ! src/jdk/nashorn/internal/objects/NativeInt16Array.java ! src/jdk/nashorn/internal/objects/NativeInt32Array.java ! src/jdk/nashorn/internal/objects/NativeInt8Array.java ! src/jdk/nashorn/internal/objects/NativeUint16Array.java ! src/jdk/nashorn/internal/objects/NativeUint32Array.java ! src/jdk/nashorn/internal/objects/NativeUint8Array.java ! src/jdk/nashorn/internal/objects/NativeUint8ClampedArray.java ! src/jdk/nashorn/internal/runtime/JSONFunctions.java ! src/jdk/nashorn/internal/runtime/Property.java + test/script/basic/JDK-8015354.js Changeset: 751cfefff5eb Author: sundar Date: 2013-05-24 23:27 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/751cfefff5eb 8015351: Nashorn shell does not start with Turkish locale Reviewed-by: jlaskey ! make/project.properties ! src/jdk/nashorn/internal/runtime/options/OptionTemplate.java Changeset: 0bf451c0678d Author: hannesw Date: 2013-05-27 12:26 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/0bf451c0678d 8015348: RegExp("[") results in StackOverflowError Reviewed-by: sundar, attila ! src/jdk/nashorn/internal/runtime/regexp/RegExpScanner.java + test/script/basic/JDK-8015348.js + test/script/basic/JDK-8015348.js.EXPECTED Changeset: 1f57afd14cc1 Author: lagergren Date: 2013-05-27 13:11 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/1f57afd14cc1 8014219: Make the run-octane harness more deterministic by not measuring elapsed time every iteration. Also got rid of most of the run logic in base.js and call benchmarks directly for the same purpose Reviewed-by: jlaskey, attila ! make/build-benchmark.xml ! src/jdk/nashorn/internal/runtime/AccessorProperty.java ! src/jdk/nashorn/internal/runtime/Property.java ! src/jdk/nashorn/internal/runtime/UserAccessorProperty.java ! test/script/basic/compile-octane.js.EXPECTED ! test/script/basic/run-octane.js Changeset: 910fd2849c4c Author: lagergren Date: 2013-05-27 13:12 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/910fd2849c4c Merge Changeset: 343fd0450802 Author: sundar Date: 2013-05-27 20:41 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/343fd0450802 8015352: "i".toUpperCase() => currently returns "??", but should be "I" (with Turkish locale) Reviewed-by: jlaskey, lagergren ! src/jdk/nashorn/internal/objects/NativeString.java ! src/jdk/nashorn/internal/runtime/ScriptEnvironment.java ! src/jdk/nashorn/internal/runtime/options/OptionTemplate.java ! src/jdk/nashorn/internal/runtime/options/Options.java ! src/jdk/nashorn/internal/runtime/resources/Options.properties + test/script/basic/JDK-8015352.js Changeset: e6193dcfe36c Author: lagergren Date: 2013-05-27 17:57 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/e6193dcfe36c 8015447: Octane harness fixes for rhino and entire test runs: ant octane, ant octane-v8, ant octane-rhino Reviewed-by: sundar, jlaskey ! make/build-benchmark.xml ! test/script/basic/run-octane.js Changeset: d56168970de1 Author: sundar Date: 2013-05-28 16:37 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/d56168970de1 8015459: Octane test run fails on Turkish locale Reviewed-by: lagergren, attila ! src/jdk/nashorn/internal/codegen/CodeGenerator.java ! src/jdk/nashorn/internal/objects/DateParser.java ! src/jdk/nashorn/internal/parser/TokenType.java ! src/jdk/nashorn/internal/runtime/GlobalFunctions.java ! src/jdk/nashorn/internal/runtime/JSType.java ! src/jdk/nashorn/internal/runtime/Logging.java ! src/jdk/nashorn/internal/runtime/ScriptRuntime.java ! src/jdk/nashorn/internal/runtime/options/OptionTemplate.java Changeset: f472f7046ec9 Author: sundar Date: 2013-05-29 15:41 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/f472f7046ec9 8005979: A lot of tests are named "runTest" in reports Reviewed-by: jlaskey ! make/project.properties Changeset: f69e76417211 Author: lagergren Date: 2013-05-29 14:08 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/f69e76417211 8011023: Math round didn't conform to ECMAScript 5 spec Reviewed-by: jlaskey, attila ! src/jdk/nashorn/internal/objects/NativeMath.java + test/script/basic/JDK-8011023.js + test/script/basic/JDK-8011023.js.EXPECTED Changeset: a2e2797392b3 Author: sundar Date: 2013-05-29 21:27 +0530 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/a2e2797392b3 8015349: "abc".lastIndexOf("a",-1) should evaluate to 0 and not -1 Reviewed-by: lagergren, attila, jlaskey ! src/jdk/nashorn/internal/objects/NativeString.java + test/script/basic/JDK-8015349.js + test/script/basic/JDK-8015349.js.EXPECTED Changeset: 4463e94d9b0d Author: lana Date: 2013-05-29 16:59 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/4463e94d9b0d Merge - src/jdk/nashorn/internal/runtime/options/ValueOption.java - src/jdk/nashorn/internal/runtime/regexp/DefaultRegExp.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompiler.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompilerSupport.java - src/jdk/nashorn/internal/runtime/regexp/joni/CaptureTreeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/NameEntry.java - src/jdk/nashorn/internal/runtime/regexp/joni/NativeMachine.java - src/jdk/nashorn/internal/runtime/regexp/joni/UnsetAddrList.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CTypeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CallNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/AbstractBench.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchGreedyBacktrack.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchRailsRegs.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchSeveralRegexps.java - src/jdk/nashorn/internal/runtime/regexp/joni/constants/Reduce.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/AsciiTables.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/PosixBracket.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/Ptr.java - src/netscape/javascript/JSObject.java Changeset: ddbf41575a2b Author: lana Date: 2013-06-03 23:24 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/ddbf41575a2b Merge - src/jdk/nashorn/internal/runtime/options/ValueOption.java - src/jdk/nashorn/internal/runtime/regexp/DefaultRegExp.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompiler.java - src/jdk/nashorn/internal/runtime/regexp/joni/AsmCompilerSupport.java - src/jdk/nashorn/internal/runtime/regexp/joni/CaptureTreeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/NameEntry.java - src/jdk/nashorn/internal/runtime/regexp/joni/NativeMachine.java - src/jdk/nashorn/internal/runtime/regexp/joni/UnsetAddrList.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CTypeNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/ast/CallNode.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/AbstractBench.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchGreedyBacktrack.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchRailsRegs.java - src/jdk/nashorn/internal/runtime/regexp/joni/bench/BenchSeveralRegexps.java - src/jdk/nashorn/internal/runtime/regexp/joni/constants/Reduce.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/AsciiTables.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/PosixBracket.java - src/jdk/nashorn/internal/runtime/regexp/joni/encoding/Ptr.java - src/netscape/javascript/JSObject.java Changeset: e857ab684db0 Author: cl Date: 2013-06-06 20:48 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/nashorn/rev/e857ab684db0 Added tag jdk8-b93 for changeset ddbf41575a2b ! .hgtags From thomas.schatzl at oracle.com Fri Jun 7 11:23:14 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Jun 2013 13:23:14 +0200 Subject: CMS parallel initial mark In-Reply-To: References: Message-ID: <1370604194.2671.55.camel@cirrus> Hi all, some notes about the second patch, the modified eden boundary sampling; in general I think the idea is good (Jon suggested that it should become default), but imho there are some issues with the implementation. On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: > Hi, > > I'd like to have the following contributed if it makes sense. > > > 2) Now, here's a related issue and a suggested fix/patch for it: > > I see that the initial mark and remark pause times sometimes spike > with a large young generation. For example, under a 1 GB young gen / 3 > GB heap setting, they occasionally spike up to ~500 milliseconds from > the normal < 100 ms range, on my machine. As far as I can tell, this > happens when the eden is fairly occupied (> 700 MB full) and not > sufficiently divided up and the parallelism decreases (at the worst > case it becomes almost single-threaded.) > > Here's a suggested patch in a separate patch: > > http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ > > that attempts to improve on this issue by implementing an alternative > way of dividing up the eden into chunks for an increased parallelism > (or better load balancing between the GC threads) for the young gen > scan portion of the remark phase (and the now-parallelized initial > mark phase.) It uses a CAS-based mechanism that samples the object > boundaries in the eden space on the slow allocation code paths (eg. at > the TLAB refill and large object allocation times) at all times. - this patch is for DefNew/CMS only it seems. Is this correct? - in the Hotspot code base explicit != NULL checks are done for whatever reason. To keep the same style it would be nice to update the checks whether to do the sampling ("if (CMSEdenChunksRecordAlways && _next_gen)") accordingly. (the next point is not an issue of the patch) - the check whether there is a _next_gen is odd - I do not think that DefNew works as a generation without an older generation, but I'm not sure. Looking at other similar code checking for _next_gen != NULL, the response is typically to update _next_gen and then asserting that _next_gen is != NULL. E.g. if (_next_gen == NULL) { GenCollectedHeap* gch = GenCollectedHeap::heap(); _next_gen = gch->next_gen(this); assert(_next_gen != NULL, "This must be the youngest gen, and not the only gen"); } Jon? - I think CMSCollector::_eden_chunk_sampling_active should be volatile to avoid any clever compilers skipping the read in sample_eden_chunk(). (Although I don't think it's possible here). See also below for avoiding that variable completely. - in the CMSCollector::CMSCollector() initialization list, the initialization of _eden_chunk_sampling_active breaks the "-- ditto --" list, which looks strange. (Actually I think the comment that a variable may eventually be set in the ctor again later seems questionable anyway) - instead of checking whether _eden_chunk_array is NULL to detect whether you can sample I would prefer if the same predicate as in the initialization (gch->supports_inline_contig_alloc()) were used. (I think the change just copied what was done in the previous version of concurrentMarkSweepGeneration.cpp:4515 has been done) Maybe create a new predicate for that, used everywhere. (btw, throughout this code there is a lot of checking whether some NEW_C_HEAP_ARRAY returned NULL or not, and does elaborate recovery - however by default NEW_C_HEAP_ARRAY fails with an OOM anyway... - but that is certainly out of scope) Same for checking whether _survivor_chunk_array is NULL - at the moment sometimes _survivor_chunk_array is checked against NULL to avoid entering code (in print_eden_and_survivor_chunk_arrays()). The condition that could be used is the one that triggers allocation of _survivor_chunk_array: (CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) in concurrentMarkSweepGeneration.cpp:728. - the code to get the lock on _eden_chunk_sampling_active seems to be needlessly complicated. Please consider using a dedicated Monitor in conjunction with try_lock(). I.e. in CMSCollector add a Monitor * _cms_eden_sampling_lock; // initialize in the constructor and then if (_cms_eden_sampling_lock->try_lock()) { // do sampling _cms_eden_sampling_lock->unlock(); } Monitor::try_lock() boils down to the exactly same code, uses a CAS to obtain the lock. I cannot see a reason to roll an own lock implementation here. Hiroshi, is it possible for you to take another look at this change? It would help us a lot. If there are questions, please do not hesitate to ask. Thanks a lot for your contributions so far, Thomas From erik.helin at oracle.com Fri Jun 7 13:42:40 2013 From: erik.helin at oracle.com (Erik Helin) Date: Fri, 7 Jun 2013 15:42:40 +0200 Subject: RFR (S): 8015972: Refactor the sending of the object count after GC event In-Reply-To: <20130605171108.GA3096@ehelin-thinkpad> References: <20130605171108.GA3096@ehelin-thinkpad> Message-ID: <20130607134240.GC2162@ehelin-thinkpad> All, based on internal feedback, I've updated the change, please see new webrev at: http://cr.openjdk.java.net/~ehelin/8015972/webrev.01/ The changes from webrev.00 are: - Using the typedef GCId instead of jlong - Remove the #include "utilities/macro.hpp" from objectCountEventSender.hpp - Use public inheritance from AllStatic - Sort includes in objectCountEventSender.hpp - Remove the "should_commit" check from ObjectCountEventSender::send, add an assert instead - Use >= instead of > in ObjectCountEventSenderClosure::should_send_event - Remove the "is_alive" variable from ObjectCountFilter::do_object_b Thanks, Erik On 2013-06-05, Erik Helin wrote: > Hi all, > > this small change refactors the way the object count after GC trace > event is sent. > > Webrev: > http://cr.openjdk.java.net/~ehelin/8015972/webrev.00/ > > Bug: > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8015972 > > Testing: > JPRT > > Thanks, > Erik From vladimir.kempik at oracle.com Fri Jun 7 13:46:20 2013 From: vladimir.kempik at oracle.com (Vladimir Kempik) Date: Fri, 07 Jun 2013 17:46:20 +0400 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 Message-ID: <51B1E42C.2080801@oracle.com> Hi all, Could I have a couple of reviews for this change? http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: AdaptiveSizePolicy::update_averages: survived: 479272 promoted: 251648 overflow: falseAdaptiveSizeStart: 4,136 collection: 16 avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: 1300947,750000 avg_pretenured_padded_avg: 286597,000000 tenuring_thresh: 1 target_size: 1835008 and there is no space between overflow: false and AdaptiveSizeStart: The patch adds line break after overflow: false. Thanks, Vladimir From mikael.gerdin at oracle.com Fri Jun 7 14:32:42 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Fri, 7 Jun 2013 07:32:42 -0700 (PDT) Subject: RFR (S): 8015972: Refactor the sending of the object count after GC event In-Reply-To: <20130607134240.GC2162@ehelin-thinkpad> References: <20130605171108.GA3096@ehelin-thinkpad> <20130607134240.GC2162@ehelin-thinkpad> Message-ID: <51B1EF0A.2010603@oracle.com> Erik, On 2013-06-07 15:42, Erik Helin wrote: > All, > > based on internal feedback, I've updated the change, please see new > webrev at: > > http://cr.openjdk.java.net/~ehelin/8015972/webrev.01/ Looks good to me. /Mikael > > The changes from webrev.00 are: > - Using the typedef GCId instead of jlong > - Remove the #include "utilities/macro.hpp" from > objectCountEventSender.hpp > - Use public inheritance from AllStatic > - Sort includes in objectCountEventSender.hpp > - Remove the "should_commit" check from ObjectCountEventSender::send, > add an assert instead > - Use >= instead of > in ObjectCountEventSenderClosure::should_send_event > - Remove the "is_alive" variable from ObjectCountFilter::do_object_b > > Thanks, > Erik > > On 2013-06-05, Erik Helin wrote: >> Hi all, >> >> this small change refactors the way the object count after GC trace >> event is sent. >> >> Webrev: >> http://cr.openjdk.java.net/~ehelin/8015972/webrev.00/ >> >> Bug: >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8015972 >> >> Testing: >> JPRT >> >> Thanks, >> Erik From jesper.wilhelmsson at oracle.com Fri Jun 7 14:37:26 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Fri, 7 Jun 2013 07:37:26 -0700 (PDT) Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B1E42C.2080801@oracle.com> References: <51B1E42C.2080801@oracle.com> Message-ID: <51B1F026.4030507@oracle.com> Vladimir, Looks good, When I fixed the same thing yesterday I noticed that the method above, PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, writes its cr() to tty instead of gclog_or_tty. Do you mind fixing that in the same change since it is slightly related? And I don't know how to stress this enough, you are not the first one who did it, never ever work on a bug without assigning it to yourself, or talk to the assigned engineer first. Having two engineers working on the same bug is not efficient use of our time. /Jesper Vladimir Kempik skrev 7/6/13 3:46 PM: > Hi all, > > Could I have a couple of reviews for this change? > > http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ > > With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: > > AdaptiveSizePolicy::update_averages: survived: 479272 promoted: 251648 overflow: > falseAdaptiveSizeStart: 4,136 collection: 16 > avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: > 1300947,750000 avg_pretenured_padded_avg: 286597,000000 tenuring_thresh: 1 > target_size: 1835008 > > and there is no space between overflow: false and AdaptiveSizeStart: > > The patch adds line break after overflow: false. > > Thanks, > Vladimir > From erik.helin at oracle.com Fri Jun 7 14:42:31 2013 From: erik.helin at oracle.com (Erik Helin) Date: Fri, 7 Jun 2013 16:42:31 +0200 Subject: RFR (S): 8015972: Refactor the sending of the object count after GC event In-Reply-To: <51B1EF0A.2010603@oracle.com> References: <20130605171108.GA3096@ehelin-thinkpad> <20130607134240.GC2162@ehelin-thinkpad> <51B1EF0A.2010603@oracle.com> Message-ID: <20130607144230.GB2168@ehelin-thinkpad> On 2013-06-07, Mikael Gerdin wrote: > Erik, > > On 2013-06-07 15:42, Erik Helin wrote: > >All, > > > >based on internal feedback, I've updated the change, please see new > >webrev at: > > > >http://cr.openjdk.java.net/~ehelin/8015972/webrev.01/ > > Looks good to me. Thanks! Erik > /Mikael > > > > >The changes from webrev.00 are: > >- Using the typedef GCId instead of jlong > >- Remove the #include "utilities/macro.hpp" from > > objectCountEventSender.hpp > >- Use public inheritance from AllStatic > >- Sort includes in objectCountEventSender.hpp > >- Remove the "should_commit" check from ObjectCountEventSender::send, > > add an assert instead > >- Use >= instead of > in ObjectCountEventSenderClosure::should_send_event > >- Remove the "is_alive" variable from ObjectCountFilter::do_object_b > > > >Thanks, > >Erik > > > >On 2013-06-05, Erik Helin wrote: > >>Hi all, > >> > >>this small change refactors the way the object count after GC trace > >>event is sent. > >> > >>Webrev: > >>http://cr.openjdk.java.net/~ehelin/8015972/webrev.00/ > >> > >>Bug: > >>http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8015972 > >> > >>Testing: > >>JPRT > >> > >>Thanks, > >>Erik From jon.masamitsu at oracle.com Fri Jun 7 14:56:32 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 07:56:32 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370546740.2445.31.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> Message-ID: <51B1F4A0.10902@oracle.com> On 6/6/2013 12:25 PM, Thomas Schatzl wrote: > Hi all, > > On Thu, 2013-06-06 at 10:06 -0700, Jon Masamitsu wrote: >> On 6/6/13 9:02 AM, Thomas Schatzl wrote: >>> Hi, >>> >>> On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: >>>> Hi, >>>> >>>> I'd like to have the following contributed if it makes sense. >>>> >>>> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >>>> >>>> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ >>>> >>>> that implements a parallel version of the initial mark phase of the >>>> CMS collector. It's relatively a straightforward parallelization of >>>> the existing single-threaded code. With the above patch, I see about >>>> ~3-6x speedup in the initial mark pause times. >>> had a look at this patch in the version at >>> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.01/ >>> >>> Looks good so far, some comments: >>> - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new >>> Par_MarkRefsIntoClosure could be refactored slightly as they have >>> exactly the same member variables. Not sure how this situation is >>> handled in other code though, and what others (Jon) think. >> Thomas, >> >> If you don't mind I'd like to keep this changeset to a minimum so >> not do any additional refactoring. That's a good suggestion but >> since this is the first sizable contribution I'm sponsoring, simpler >> is better for me. > Okay. It would be a tiny additional change though, which has only been > enabled by the addition of the Par_MarkRefsIntoClosure, and of course > depends on whether the old serial initial marking code is kept. Thanks. > > (Maybe it is also possible to remove the underscore in the class name? > It's not exactly customary to have class names with underscores in the > Hotspot code - I think) > >>> - in concurrentMarkSweepGeneration.cpp in >>> CMSCollector::checkpointRootsInitialWork() the new code seems a little >>> strange to me. I.e. roughly it does the following: >>> >>> [...] >>> if (CMSParallelInitialMarkEnabled) { >>> [ set up parallel stuff ] >>> if (n_workers > 1) { >>> // do parallel thread >>> } else { >>> // do serial work using the current thread >>> } >>> } else { >>> // do serial work >>> } >>> >>> it does not feel good to have two serial paths here; maybe use the >>> serial/original code if n_workers == 1 and guarantee(n_workers > 1, ) in >>> the first part of the if? Or look if the old serial path could be >>> removed? (or the new one). >> Is the inner serial path one thread executing a single chunk that >> does all the work (i.e., executes the parallel code but by only >> one thread)? > Yes, that is in my opinion a little awkward. It would be nice if we had > only one serial initial marking code path to simplify > debugging/maintenance if there are issues. Removal of one or the other > (if so) probably depends on how different the performance between the > two paths is. I would prefer not to remove the path that executes the parallel code with a single thread. It is a tool for looking for performance anomalies in the parallel code. I've used code like it (i.e., executes parallel code in one thread in UseParallelGC and UseParNewGC) this week while looking for a performance problem. The strictly serial path (which is the code we have today) would be nice to have for a while as a workaround for any problems. > >>> - the comments for at least CMSCollector:: >>> initialize_sequential_subtasks_for_young_gen_rescan() and >>> CMSCollector::merge_survivor_plab_arrays() should be reworked - they >>> only mention parallel rescan as entry points. > I recommend fixing the comments though - wrong comments are not exactly > helpful, although it's just an omission here. > >>> - the change in the assert in >>> CMSCollector::merge_survivor_plab_arrays() looks a little odd now. I >>> mean, looking at that assert in isolation poses the question why is this >>> method only called when doing initial mark in parallel? It somehow seems >>> that the same work is done somewhere else just for the initial mark and >>> serial case, which just looks strange to me. That's just an observation >>> from me. >>> >>> assert(_collectorState == FinalMarking || >>> (CMSParallelInitialMarkEnabled && _collectorState == >>> InitialMarking), "Error"); >>> >>> - in CMSCollector::CMSCollector() there is this unconditional setup of >>> a few data structures used for parallel processing if any of the >>> CMSParallel* switches is enabled. >> I'm not sure what code this is. What lines in the webrev? > I hit the send button too eagerly (was at leaving the office), sorry. I > meant ConcurrentMarkSweepGeneration.cpp:727, and was thinking about > removing on of the serial initial marking paths. > > If the old serial version (old code) is kept, this condition needs to be > adapted, as data structures within the if-block started at that line are > only used in the parallel case. Where talking about this like if ((CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) || CMSParallelInitialMarkEnabled) { and you're saying the > this condition needs to be > adapted, as data structures within the if-block started at that line are > only used in the parallel case. Sorry, I still don't get it. Are you saying the condition in the test is not correct? By "parallel case" do you mean the case where more than one GC thread does the work? Or are you counting one GC thread executing the parallel code as part of the "parallel case"? Jon > > But that was just a thought as I am not sure how far you are with > testing it. > > Thanks, > Thomas > > From jon.masamitsu at oracle.com Fri Jun 7 15:09:00 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 08:09:00 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <51B00B03.1020503@oracle.com> Message-ID: <51B1F78C.3020509@oracle.com> On 6/6/2013 4:19 PM, Hiroshi Yamauchi wrote: > On Wed, Jun 5, 2013 at 9:07 PM, Jon Masamitsu wrote: > >> Hiroshi, >> >> For the sampling change. >> >> I appreciate that you allow for reverting to the old behavior of >> sampling during precleaning but am curious about whether >> you've seen an occasion where it was preferable. >> > I assume you are referring to an occasion where the old behavior was > preferable than the new behavior. No, I haven't seen such a case. As far as > I can tell, there's no noticeable runtime overhead due to the new way of > sampling, and I haven't seen a case where the remark pause time was better > with the old behavior. The new behavior is disabled by default just for > conservatism. If it's preferred to adopt the new behavior without a flag, > there's no problem with me. I'd suggest changing the default for both to true CMSParallelInitialMarkEnabled CMSEdenChunksRecordAlways That will exercise the new code, right? Jon > > >> http://cr.openjdk.java.net/~**hiroshi/webrevs/edenchunks/** >> webrev.00/src/share/vm/gc_**implementation/**concurrentMarkSweep/** >> concurrentMarkSweepGeneration.**hpp.frames.html >> >> 739 // This is meant to be a boolean flag, but jbyte for CAS. >> 740 jbyte _eden_chunk_sampling_active; >> >> Other than the card table I'm used to seeing the atomic operations >> on word sized variables. Would jint work as well and be simpler to >> think about? >> > Sure. jint would be fine, too. > > > >> Maybe more later. >> >> >> Jon >> >> >> >> On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: >> >>> Hi, >>> >>> I'd like to have the following contributed if it makes sense. >>> >>> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >>> >>> http://cr.openjdk.java.net/~**hiroshi/webrevs/** >>> cmsparinitmark/webrev.00/ >>> >>> that implements a parallel version of the initial mark phase of the >>> CMS collector. It's relatively a straightforward parallelization of >>> the existing single-threaded code. With the above patch, I see about >>> ~3-6x speedup in the initial mark pause times. >>> >>> 2) Now, here's a related issue and a suggested fix/patch for it: >>> >>> I see that the initial mark and remark pause times sometimes spike >>> with a large young generation. For example, under a 1 GB young gen / 3 >>> GB heap setting, they occasionally spike up to ~500 milliseconds from >>> the normal < 100 ms range, on my machine. As far as I can tell, this >>> happens when the eden is fairly occupied (> 700 MB full) and not >>> sufficiently divided up and the parallelism decreases (at the worst >>> case it becomes almost single-threaded.) >>> >>> Here's a suggested patch in a separate patch: >>> >>> http://cr.openjdk.java.net/~**hiroshi/webrevs/edenchunks/**webrev.00/ >>> >>> that attempts to improve on this issue by implementing an alternative >>> way of dividing up the eden into chunks for an increased parallelism >>> (or better load balancing between the GC threads) for the young gen >>> scan portion of the remark phase (and the now-parallelized initial >>> mark phase.) It uses a CAS-based mechanism that samples the object >>> boundaries in the eden space on the slow allocation code paths (eg. at >>> the TLAB refill and large object allocation times) at all times. >>> >>> This approach is in contrast to the original mechanism that samples >>> object boundaries in the eden space asynchronously during the preclean >>> phase. I think the reason that the above issue happens is that when >>> the young generation is large, a large portion of the eden space could >>> get filled/allocated outside of the preclean phase (or a concurrent >>> collection) and the object boundaries do not get sampled >>> often/regularly enough. Also, it isn't very suited for the parallel >>> initial mark because the initial mark phase isn't preceded by the >>> preclean phase unlike the remark phase. According to the Dacapo >>> benchmarks, this alternative sampling mechanism does not have >>> noticeable runtime overhead despite it is engaged at all times. >>> >>> With this patch, I see that the (parallel) initial mark and remark >>> pause times stay below 100 ms (no spikes) under the same setting. >>> >>> Both of these features/flags are disabled by default. You're welcome >>> to handle the two patches separately. >>> >>> Thanks, >>> Hiroshi >>> >> From vladimir.kempik at oracle.com Fri Jun 7 15:13:01 2013 From: vladimir.kempik at oracle.com (Vladimir Kempik) Date: Fri, 7 Jun 2013 08:13:01 -0700 (PDT) Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B1F026.4030507@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> Message-ID: <51B1F87D.7060105@oracle.com> Thanks for comments Here is updated webrev - http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.01/ On 07.06.2013 18:37, Jesper Wilhelmsson wrote: > Vladimir, > > Looks good, > > When I fixed the same thing yesterday I noticed that the method above, > PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, > writes its cr() to tty instead of gclog_or_tty. Do you mind fixing > that in the same change since it is slightly related? > > And I don't know how to stress this enough, you are not the first one > who did it, never ever work on a bug without assigning it to yourself, > or talk to the assigned engineer first. Having two engineers working > on the same bug is not efficient use of our time. Sorry about that. Vladimir. > /Jesper > > > > Vladimir Kempik skrev 7/6/13 3:46 PM: >> Hi all, >> >> Could I have a couple of reviews for this change? >> >> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >> >> With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: >> >> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: >> 251648 overflow: >> falseAdaptiveSizeStart: 4,136 collection: 16 >> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 >> tenuring_thresh: 1 >> target_size: 1835008 >> >> and there is no space between overflow: false and AdaptiveSizeStart: >> >> The patch adds line break after overflow: false. >> >> Thanks, >> Vladimir >> From jon.masamitsu at oracle.com Fri Jun 7 15:13:45 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 08:13:45 -0700 Subject: RFR (XS): 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 In-Reply-To: <51B16DBE.4010300@oracle.com> References: <51AFB2AE.7040604@oracle.com> <51B0D916.9030006@oracle.com> <51B16DBE.4010300@oracle.com> Message-ID: <51B1F8A9.1030208@oracle.com> Sorry guys. I probably mislead Jesper on this in an IM exchange we had. As John's example shows the line break is preferable. Jon On 6/6/2013 10:21 PM, Bengt Rutisson wrote: > > Hi Jesper, > > I agree with John, the line break in psScavenge.cpp looks intentional > and should probably not be changed. > > Also, please use gclog_or_tty->print_cr() in psAdaptiveSizePolicy.cpp > rather than adding extra calls to gclog_or_tty->cr(); > > Bengt > > On 6/6/13 8:46 PM, John Cuthbertson wrote: >> Hi Jesper, >> >> The change is psAdaptiveSizePolicy.cpp looks good. >> >> I don't, however, think you should remove the line break in >> psScavenge.cpp. I _that_ that line break is meant to be there. The >> output beginning with "avg_survived_padded_avg" is slightly indented. >> Compare the other places where "AdaptiveSizeStart" is printed in >> psMarkSweep.cpp and psParallelCompact.cpp. Leaving line break in will >> make the output: >> >> AdaptiveSizeStart: 23.548 collection: 67 >> avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: >> 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 >> tenuring_thresh: 14 target_size: 1048576 >> >> JohnC >> >> On 6/5/2013 2:50 PM, Jesper Wilhelmsson wrote: >>> Hi, >>> >>> Can I have a couple of reviews for this really minor fix? >>> >>> 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 >>> >>> A missing line break in the adaptive size verbose logging has been >>> added. >>> >>> When looking at this I also found that an existing line break caused >>> a slightly ugly verbose output when breaking a line causing a second >>> line without a identifier at the beginning. I removed that line >>> break as well in this change. >>> >>> >>> webrev: >>> http://cr.openjdk.java.net/~jwilhelm/8015903/hotspot-webrev/ >>> >>> >>> >>> Output before: >>> >>> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 >>> overflow: falseAdaptiveSizeStart: 23.548 collection: 67 >>> avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: >>> 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 >>> tenuring_thresh: 14 target_size: 1048576 >>> >>> >>> Output after: >>> >>> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 >>> overflow: false >>> AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: >>> 728176.250000 avg_promoted_padded_avg: 223962144.000000 >>> avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 >>> target_size: 1048576 >>> >>> >>> Thanks, >>> /Jesper >> > > From erik.helin at oracle.com Fri Jun 7 15:23:39 2013 From: erik.helin at oracle.com (Erik Helin) Date: Fri, 7 Jun 2013 17:23:39 +0200 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B1E42C.2080801@oracle.com> References: <51B1E42C.2080801@oracle.com> Message-ID: <20130607152339.GC2168@ehelin-thinkpad> Hi Vladimir, I am gatekeeper this month for the hotspot-gc repository. As you might know, JFR might be merged on Monday to the hotspot-rt repository. Due to this, we have locked the hotspot-gc repository, meaning that no one are allowed to push to the hotspot-gc repository. I will send out more details about once I get some feedback from the group. Please reply to this email so that I know that you have seen this. Thanks, and thanks for fixing the bug! Erik PS. I did not look at the change, this is not a review :) On 2013-06-07, Vladimir Kempik wrote: > Hi all, > > Could I have a couple of reviews for this change? > > http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ > > With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: > > AdaptiveSizePolicy::update_averages: survived: 479272 promoted: > 251648 overflow: falseAdaptiveSizeStart: 4,136 collection: 16 > avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: > 1300947,750000 avg_pretenured_padded_avg: 286597,000000 > tenuring_thresh: 1 target_size: 1835008 > > and there is no space between overflow: false and AdaptiveSizeStart: > > The patch adds line break after overflow: false. > > Thanks, > Vladimir > From jon.masamitsu at oracle.com Fri Jun 7 15:25:10 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 7 Jun 2013 08:25:10 -0700 (PDT) Subject: Request for review (s) - 8015851 :UseAdaptiveGCBoundary is broken In-Reply-To: <1370595128.2671.5.camel@cirrus> References: <51AF9F40.6040204@oracle.com> <51AFD701.90400@oracle.com> <1370595128.2671.5.camel@cirrus> Message-ID: <51B1FB56.7030905@oracle.com> On 6/7/2013 1:52 AM, Thomas Schatzl wrote: > Hi, > > On Wed, 2013-06-05 at 17:25 -0700, Jon Masamitsu wrote: >> Fixed the CR # in the subject. >> >> On 6/5/2013 1:27 PM, Jon Masamitsu wrote: >>> A recent changeset effectively removed the initialization of >>> performance counters used by the option UseAdaptiveGCBoundary >>> >>> This changeset is built on top of the one for 8014546 and the >>> webrev comes in two parts. >>> >>> The fix plus the test >>> >>> http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ > Looks good. > > About the test: is there a way to actually test performance counter > initialization? So far it only checks whether the VM crashes, and not > whether initialization is okay. The call to System.gc() will cause the performance counters to be updated. At least the performance counters that are explicitly updated after a GC. There could be performance counters that are updated periodicially instead. Those latter won't necessarily be updated. I think we would need to use something external to the JVM like jstat to see if the counters made sense. I think that is more than a regression test. > >>> Some cleanups of the code >>> >>> http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ >>> > Just to make sure what this cleanup is about: move the call to > initialize_work() into the constructors of the generations, and clean up > some unused constructors. Yes. Thanks. Jon > > If so, looks okay. > > Hth, > Thomas > From vladimir.kempik at oracle.com Fri Jun 7 15:25:53 2013 From: vladimir.kempik at oracle.com (Vladimir Kempik) Date: Fri, 07 Jun 2013 19:25:53 +0400 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <20130607152339.GC2168@ehelin-thinkpad> References: <51B1E42C.2080801@oracle.com> <20130607152339.GC2168@ehelin-thinkpad> Message-ID: <51B1FB81.8000405@oracle.com> Hello. I understood. On 07.06.2013 19:23, Erik Helin wrote: > Hi Vladimir, > > I am gatekeeper this month for the hotspot-gc repository. As you might > know, JFR might be merged on Monday to the hotspot-rt repository. Due to > this, we have locked the hotspot-gc repository, meaning that no one are > allowed to push to the hotspot-gc repository. > > I will send out more details about once I get some feedback from the > group. > > Please reply to this email so that I know that you have seen this. > > Thanks, and thanks for fixing the bug! > > Erik > > PS. I did not look at the change, this is not a review :) > > On 2013-06-07, Vladimir Kempik wrote: >> Hi all, >> >> Could I have a couple of reviews for this change? >> >> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >> >> With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: >> >> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: >> 251648 overflow: falseAdaptiveSizeStart: 4,136 collection: 16 >> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 >> tenuring_thresh: 1 target_size: 1835008 >> >> and there is no space between overflow: false and AdaptiveSizeStart: >> >> The patch adds line break after overflow: false. >> >> Thanks, >> Vladimir >> From mikael.gerdin at oracle.com Fri Jun 7 15:28:44 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Fri, 7 Jun 2013 08:28:44 -0700 (PDT) Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51B0A1BA.9070308@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> <51B054F2.7000807@oracle.com> <51B0A1BA.9070308@oracle.com> Message-ID: <51B1FC2C.9030105@oracle.com> Jon, On 2013-06-06 16:50, Jon Masamitsu wrote: > Mikael, > > Thanks. I'd be interested in seeing the instrumentation you > add. Might be worth adding as an enhancement in a later > changeset. I did a 1hr KS run today with and without block splitting, here's what I came up with (in an entirely non-scientific way) http://cr.openjdk.java.net/~mgerdin/8009561/splitting.txt http://cr.openjdk.java.net/~mgerdin/8009561/splitting.png We hit the HWM 4 times with splitting and 5 times without splitting. On the other hand: splitting did lead us with more metaspace memory committed in the end. I put up the very simple instrumentation at: http://cr.openjdk.java.net/~mgerdin/8009561/instrument/webrev I also changed the allocation_from_dictionary_limit to 4k to force us to make more freelist allocations. /Mikael > > Jon > > > On 6/6/13 2:22 AM, Mikael Gerdin wrote: >> Jon, >> >> On 2013-06-06 04:41, Jon Masamitsu wrote: >>> >>> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >>>> Hi, >>>> >>>> Can I have some reviews of this small fix to the Metaspace memory >>>> allocation path. >>>> >>>> Problem: >>>> When a Metaspace allocation request cannot be satisfied by the current >>>> chunk the chunk is retired and a new chunk is requested. This causes >>>> whatever is left in the chunk to be effectively leaked. >>>> >>>> Suggested fix: >>>> Put the remaining memory in each chunk on the Metablock freelist so it >>>> can be used to satisfy future allocations. >>>> >>>> Possible addition: >>>> When allocating from the block free list, use >>>> FreeBlockDictionary::atLeast instead of >>>> FreeBlockDictionary::exactly and split the Metablock if >>>> it's large enough. >>>> >>>> One might argue that this increases the fragmentation of the memory on >>>> the block free list but I think that we primarily want to use the >>>> block free list for small allocations and allocate from chunks for >>>> large allocations. >>>> >>>> Webrev: >>>> Only fix: >>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ >>> >>> The "Only fix" looks good. Did you test with >>> metaspace_slow_verify=true? >>> >>>> >>>> Incremental webrev for splitting blocks: >>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ >>> >>> Change looks good. >>> >>> Did you do any long running tests with the block splitting? Such as >>> 24hours with kitchensink? Something that would reuse Metablocks >>> so that we can see if we are fragmenting instead of reusing? >>> >> >> I did some runs earlier but I don't have any data from them. >> I can try to get an instrumented build together and run KS over the >> weekend. >> >> /Mikael >> >>> Jon >>> >>> >>>> >>>> Bug links: >>>> https://jbs.oracle.com/bugs/browse/JDK-8009561 >>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >>>> >>>> Thanks >>>> /Mikael >>> >> > From thomas.schatzl at oracle.com Fri Jun 7 15:53:23 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Jun 2013 17:53:23 +0200 Subject: CMS parallel initial mark In-Reply-To: <51B1F4A0.10902@oracle.com> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> Message-ID: <1370620403.3085.2.camel@cirrus> Hi Jon, On Fri, 2013-06-07 at 07:56 -0700, Jon Masamitsu wrote: > On 6/6/2013 12:25 PM, Thomas Schatzl wrote: > > Hi all, > > > > On Thu, 2013-06-06 at 10:06 -0700, Jon Masamitsu wrote: > >> On 6/6/13 9:02 AM, Thomas Schatzl wrote: > >>> Hi, > >>> > >>> On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: > > > > (Maybe it is also possible to remove the underscore in the class name? > > It's not exactly customary to have class names with underscores in the > > Hotspot code - I think) > > > >>> - in concurrentMarkSweepGeneration.cpp in > >>> CMSCollector::checkpointRootsInitialWork() the new code seems a little > >>> strange to me. I.e. roughly it does the following: > >>> > >>> [...] > >>> if (CMSParallelInitialMarkEnabled) { > >>> [ set up parallel stuff ] > >>> if (n_workers > 1) { > >>> // do parallel thread > >>> } else { > >>> // do serial work using the current thread > >>> } > >>> } else { > >>> // do serial work > >>> } > >>> > >>> it does not feel good to have two serial paths here; maybe use the > >>> serial/original code if n_workers == 1 and guarantee(n_workers > 1, ) in > >>> the first part of the if? Or look if the old serial path could be > >>> removed? (or the new one). > >> Is the inner serial path one thread executing a single chunk that > >> does all the work (i.e., executes the parallel code but by only > >> one thread)? > > Yes, that is in my opinion a little awkward. It would be nice if we had > > only one serial initial marking code path to simplify > > debugging/maintenance if there are issues. Removal of one or the other > > (if so) probably depends on how different the performance between the > > two paths is. > > I would prefer not to remove the path that executes the parallel code > with a single thread. It is a tool for looking for performance anomalies > in the parallel code. I've used code like it (i.e., executes parallel > code in > one thread in UseParallelGC and UseParNewGC) this week while looking > for a performance problem. > > The strictly serial path (which is the code we have today) would be nice > to have for a while as a workaround for any problems. Okay, fair enough. > > If the old serial version (old code) is kept, this condition needs to be > > adapted, as data structures within the if-block started at that line are > > only used in the parallel case. > > Where talking about this like > > if ((CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) || > CMSParallelInitialMarkEnabled) { > > and you're saying the > > this condition needs to be > > adapted, as data structures within the if-block started at that line are > > only used in the parallel case. > Sorry, I still don't get it. Are you saying the condition in the test is > not correct? By "parallel case" do you mean the case where > more than one GC thread does the work? Or are you counting > one GC thread executing the parallel code as part of the > "parallel case"? Sorry for confusing you. This was just some suggestion for if one of the serial paths is removed (e.g. in the sense of if we remove that, we need to also remove this). Since it has been decided to keep both for now (I think), this comment is obsolete. Thanks, Thomas From jesper.wilhelmsson at oracle.com Fri Jun 7 16:20:49 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Fri, 07 Jun 2013 18:20:49 +0200 Subject: RFR (XS): 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 In-Reply-To: <51B1F8A9.1030208@oracle.com> References: <51AFB2AE.7040604@oracle.com> <51B0D916.9030006@oracle.com> <51B16DBE.4010300@oracle.com> <51B1F8A9.1030208@oracle.com> Message-ID: <51B20861.6020601@oracle.com> All, Looks like Vladimir Kempik was already working on this bug. His latest change looks like mine would have after your comments, so give him a review and we'll forget that I worked on it. :-/ /Jesper Jon Masamitsu skrev 7/6/13 5:13 PM: > Sorry guys. I probably mislead Jesper on this in an IM exchange > we had. As John's example shows the line break is preferable. > > Jon > > On 6/6/2013 10:21 PM, Bengt Rutisson wrote: >> >> Hi Jesper, >> >> I agree with John, the line break in psScavenge.cpp looks intentional and >> should probably not be changed. >> >> Also, please use gclog_or_tty->print_cr() in psAdaptiveSizePolicy.cpp rather >> than adding extra calls to gclog_or_tty->cr(); >> >> Bengt >> >> On 6/6/13 8:46 PM, John Cuthbertson wrote: >>> Hi Jesper, >>> >>> The change is psAdaptiveSizePolicy.cpp looks good. >>> >>> I don't, however, think you should remove the line break in psScavenge.cpp. I >>> _that_ that line break is meant to be there. The output beginning with >>> "avg_survived_padded_avg" is slightly indented. Compare the other places >>> where "AdaptiveSizeStart" is printed in psMarkSweep.cpp and >>> psParallelCompact.cpp. Leaving line break in will make the output: >>> >>> AdaptiveSizeStart: 23.548 collection: 67 >>> avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: >>> 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: >>> 14 target_size: 1048576 >>> >>> JohnC >>> >>> On 6/5/2013 2:50 PM, Jesper Wilhelmsson wrote: >>>> Hi, >>>> >>>> Can I have a couple of reviews for this really minor fix? >>>> >>>> 8015903: Fomat issue with -XX:+PRINTADAPTIVESIZEPOLICY on JDK8 >>>> >>>> A missing line break in the adaptive size verbose logging has been added. >>>> >>>> When looking at this I also found that an existing line break caused a >>>> slightly ugly verbose output when breaking a line causing a second line >>>> without a identifier at the beginning. I removed that line break as well in >>>> this change. >>>> >>>> >>>> webrev: >>>> http://cr.openjdk.java.net/~jwilhelm/8015903/hotspot-webrev/ >>>> >>>> >>>> >>>> Output before: >>>> >>>> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 overflow: >>>> falseAdaptiveSizeStart: 23.548 collection: 67 >>>> avg_survived_padded_avg: 728176.250000 avg_promoted_padded_avg: >>>> 223962144.000000 avg_pretenured_padded_avg: 187235776.000000 >>>> tenuring_thresh: 14 target_size: 1048576 >>>> >>>> >>>> Output after: >>>> >>>> AdaptiveSizePolicy::update_averages: survived: 163840 promoted: 0 overflow: >>>> false >>>> AdaptiveSizeStart: 23.548 collection: 67 avg_survived_padded_avg: >>>> 728176.250000 avg_promoted_padded_avg: 223962144.000000 >>>> avg_pretenured_padded_avg: 187235776.000000 tenuring_thresh: 14 target_size: >>>> 1048576 >>>> >>>> >>>> Thanks, >>>> /Jesper >>> >> >> > From jesper.wilhelmsson at oracle.com Fri Jun 7 16:23:20 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Fri, 07 Jun 2013 18:23:20 +0200 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B1F87D.7060105@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> <51B1F87D.7060105@oracle.com> Message-ID: <51B208F8.6030200@oracle.com> Vladimir Kempik skrev 7/6/13 5:13 PM: > Thanks for comments > > Here is updated webrev - > http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.01/ Thanks for fixing this! Looks good, ship it (once the gc repo is open for pushes again). /Jesper > On 07.06.2013 18:37, Jesper Wilhelmsson wrote: >> Vladimir, >> >> Looks good, >> >> When I fixed the same thing yesterday I noticed that the method above, >> PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, writes its >> cr() to tty instead of gclog_or_tty. Do you mind fixing that in the same >> change since it is slightly related? >> >> And I don't know how to stress this enough, you are not the first one who did >> it, never ever work on a bug without assigning it to yourself, or talk to the >> assigned engineer first. Having two engineers working on the same bug is not >> efficient use of our time. > Sorry about that. > > Vladimir. >> /Jesper >> >> >> >> Vladimir Kempik skrev 7/6/13 3:46 PM: >>> Hi all, >>> >>> Could I have a couple of reviews for this change? >>> >>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >>> >>> With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: >>> >>> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: 251648 overflow: >>> falseAdaptiveSizeStart: 4,136 collection: 16 >>> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >>> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 tenuring_thresh: 1 >>> target_size: 1835008 >>> >>> and there is no space between overflow: false and AdaptiveSizeStart: >>> >>> The patch adds line break after overflow: false. >>> >>> Thanks, >>> Vladimir >>> > From jon.masamitsu at oracle.com Fri Jun 7 17:02:38 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 10:02:38 -0700 Subject: Request for review (s) - 8015851 :UseAdaptiveGCBoundary is broken In-Reply-To: <51B1A610.90007@oracle.com> References: <51AF9F40.6040204@oracle.com> <51AFD701.90400@oracle.com> <51B1A610.90007@oracle.com> Message-ID: <51B2122E.3000003@oracle.com> On 6/7/2013 2:21 AM, Bengt Rutisson wrote: > > Hi Jon, > > On 6/6/13 2:25 AM, Jon Masamitsu wrote: >> Fixed the CR # in the subject. >> >> On 6/5/2013 1:27 PM, Jon Masamitsu wrote: >>> A recent changeset effectively removed the initialization of >>> performance counters used by the option UseAdaptiveGCBoundary > > Which change broke the initialization of the performance counters? I didn't track down the particular changeset. The breakage was reported to me on the latest build. I can hunt it down if you like. > > One comment: > > The code you add in ASPSOldGen::initialize_work() is very similar to > the code in PSOldGen::initialize(). The PSOldGen::initialize() method > is called from one (why not both?!?!) of the constructors for > PSOldGen, which means that it is called from one of the ASPSOldGen > constructors. But these constructors seem to be unused. Would it be > possible to move your code from ASPSOldGen::initialize_work() in to > the constructor of PSOldGen that is actually used? The short answer is that I can move the code in ASPSOldGen::initialize_work() into the 2nd PSOldGen that does no initialization. And will. For why ASPSOldGen::initialize_work() and PSOldGen::initialize() are similar but different and about the PSOldGen constructors (which I think are both actually used). Many (maybe most) of the Generation types have constructors that take a ReservedSpace and initialize based on the ReserveSpace. ASPSOldGen and ASPSYoungGen share a ReservedSpace. I thought that passing in the addresses and sizes of the part of the ReservedSpace to be used by each generation (ASPSOldGen and ASPSYoungGen) was more transparent than to pass the ReservedSpace to each generation and let each generation take a piece. That is, I thought that doing the division of the space in the ReservedSpace intended for each generation in the code where each generation was created was better (a single spot) than having it in two separate places (the constructors for the generation). So there is a difference between how PSOldGen and ASPSOldGen are initialized so using PSOldGen::initialize() with ASPSOldGen doesn't work without so additional refactoring. I removed one of the constructors for ASPSOldGen in the cleanups so there is only ASPSOldGen(PSVirtualSpace* vs, size_t initial_byte_size, size_t minimum_byte_size, size_t byte_size_limit, const char* gen_name, int level); which explicitly uses PSOldGen(size_t initial_size, size_t min_size, size_t max_size, const char* perf_data_name, int level); which is the constructor that does not do the initialization. Thanks for looking at this and for the suggection about moving the code into the 2nd PSOldGen constructor. Jon > > Thanks, > Bengt > >>> >>> This changeset is built on top of the one for 8014546 and the >>> webrev comes in two parts. >>> >>> The fix plus the test >>> >>> http://cr.openjdk.java.net/~jmasa/8015851/webrev.00/ >>> >>> Some cleanups of the code >>> >>> http://cr.openjdk.java.net/~jmasa/8015851/webrev_refactor.00/ >>> >>> Thanks. >>> >>> Jon >> > From jon.masamitsu at oracle.com Fri Jun 7 17:36:11 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 7 Jun 2013 10:36:11 -0700 (PDT) Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51B1FC2C.9030105@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> <51B054F2.7000807@oracle.com> <51B0A1BA.9070308@oracle.com> <51B1FC2C.9030105@oracle.com> Message-ID: <51B21A0B.2030800@oracle.com> On 6/7/2013 8:28 AM, Mikael Gerdin wrote: > Jon, > > > On 2013-06-06 16:50, Jon Masamitsu wrote: >> Mikael, >> >> Thanks. I'd be interested in seeing the instrumentation you >> add. Might be worth adding as an enhancement in a later >> changeset. > > I did a 1hr KS run today with and without block splitting, here's what > I came up with (in an entirely non-scientific way) > > http://cr.openjdk.java.net/~mgerdin/8009561/splitting.txt > http://cr.openjdk.java.net/~mgerdin/8009561/splitting.png Good graphs. The behavior is what we expect (I think). With splitting we are able to do more small allocations from the dictionary (where we split a larger block to get a smaller block) and get fewer larger blocks allocated (some have been split). > > We hit the HWM 4 times with splitting and 5 times without splitting. Because we don't have to expand (get new chunks as often, which is good) I would surmise. > On the other hand: splitting did lead us with more metaspace memory > committed in the end. One explanation would be that allocations of larger block need to come out of newly committed space instead of the dictionary (where the large blocks have been broken up). Is there a policy that we could use that says "break up a larger block for a smaller block allocation only if ..." You fill in the blank? > > I put up the very simple instrumentation at: > http://cr.openjdk.java.net/~mgerdin/8009561/instrument/webrev > > I also changed the allocation_from_dictionary_limit to 4k to force us > to make more freelist allocations. Does it really make sense to have any allocation_from_dictionary_limit? I know it was initially added because allocation from a freelist takes longer but to have a static limit like that just seems to put that space forever beyond reach. Thanks for the numbers. Jon > > /Mikael > >> >> Jon >> >> >> On 6/6/13 2:22 AM, Mikael Gerdin wrote: >>> Jon, >>> >>> On 2013-06-06 04:41, Jon Masamitsu wrote: >>>> >>>> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >>>>> Hi, >>>>> >>>>> Can I have some reviews of this small fix to the Metaspace memory >>>>> allocation path. >>>>> >>>>> Problem: >>>>> When a Metaspace allocation request cannot be satisfied by the >>>>> current >>>>> chunk the chunk is retired and a new chunk is requested. This causes >>>>> whatever is left in the chunk to be effectively leaked. >>>>> >>>>> Suggested fix: >>>>> Put the remaining memory in each chunk on the Metablock freelist >>>>> so it >>>>> can be used to satisfy future allocations. >>>>> >>>>> Possible addition: >>>>> When allocating from the block free list, use >>>>> FreeBlockDictionary::atLeast instead of >>>>> FreeBlockDictionary::exactly and split the Metablock if >>>>> it's large enough. >>>>> >>>>> One might argue that this increases the fragmentation of the >>>>> memory on >>>>> the block free list but I think that we primarily want to use the >>>>> block free list for small allocations and allocate from chunks for >>>>> large allocations. >>>>> >>>>> Webrev: >>>>> Only fix: >>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ >>>> >>>> The "Only fix" looks good. Did you test with >>>> metaspace_slow_verify=true? >>>> >>>>> >>>>> Incremental webrev for splitting blocks: >>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ >>>> >>>> Change looks good. >>>> >>>> Did you do any long running tests with the block splitting? Such as >>>> 24hours with kitchensink? Something that would reuse Metablocks >>>> so that we can see if we are fragmenting instead of reusing? >>>> >>> >>> I did some runs earlier but I don't have any data from them. >>> I can try to get an instrumented build together and run KS over the >>> weekend. >>> >>> /Mikael >>> >>>> Jon >>>> >>>> >>>>> >>>>> Bug links: >>>>> https://jbs.oracle.com/bugs/browse/JDK-8009561 >>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >>>>> >>>>> Thanks >>>>> /Mikael >>>> >>> >> From jon.masamitsu at oracle.com Fri Jun 7 18:17:42 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 11:17:42 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370604194.2671.55.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> Message-ID: <51B223C6.7080805@oracle.com> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: > Hi all, > > some notes about the second patch, the modified eden boundary > sampling; in general I think the idea is good (Jon suggested that it > should become default), but imho there are some issues with the > implementation. > > On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: >> Hi, >> >> I'd like to have the following contributed if it makes sense. >> >> >> 2) Now, here's a related issue and a suggested fix/patch for it: >> >> I see that the initial mark and remark pause times sometimes spike >> with a large young generation. For example, under a 1 GB young gen / 3 >> GB heap setting, they occasionally spike up to ~500 milliseconds from >> the normal < 100 ms range, on my machine. As far as I can tell, this >> happens when the eden is fairly occupied (> 700 MB full) and not >> sufficiently divided up and the parallelism decreases (at the worst >> case it becomes almost single-threaded.) >> >> Here's a suggested patch in a separate patch: >> >> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ >> >> that attempts to improve on this issue by implementing an alternative >> way of dividing up the eden into chunks for an increased parallelism >> (or better load balancing between the GC threads) for the young gen >> scan portion of the remark phase (and the now-parallelized initial >> mark phase.) It uses a CAS-based mechanism that samples the object >> boundaries in the eden space on the slow allocation code paths (eg. at >> the TLAB refill and large object allocation times) at all times. > - this patch is for DefNew/CMS only it seems. Is this correct? > > - in the Hotspot code base explicit != NULL checks are done for whatever > reason. To keep the same style it would be nice to update the checks > whether to do the sampling ("if (CMSEdenChunksRecordAlways && > _next_gen)") accordingly. > > (the next point is not an issue of the patch) > - the check whether there is a _next_gen is odd - I do not think that > DefNew works as a generation without an older generation, but I'm not > sure. You're correct that DefNew needs to have a _next_gen. > Looking at other similar code checking for _next_gen != NULL, the > response is typically to update _next_gen and then asserting that > _next_gen is != NULL. E.g. > > if (_next_gen == NULL) { > GenCollectedHeap* gch = GenCollectedHeap::heap(); > _next_gen = gch->next_gen(this); > assert(_next_gen != NULL, > "This must be the youngest gen, and not the only gen"); > } > > Jon? Yes, except in a very few situations, _next_gen should be set. Exceptions I would expect is during initialization. At the point Thomas indicates and assertion would be sufficient. assert(_next_gen != NULL, "..."); > > - I think CMSCollector::_eden_chunk_sampling_active should be volatile > to avoid any clever compilers skipping the read in sample_eden_chunk(). > (Although I don't think it's possible here). See also below for avoiding > that variable completely. Could you explain you thinking for making it volatile. As you have, I didn't think it was necessary and thought ,"If not necessary, don't do it". > > - in the CMSCollector::CMSCollector() initialization list, the > initialization of _eden_chunk_sampling_active breaks the "-- ditto --" > list, which looks strange. (Actually I think the comment that a variable > may eventually be set in the ctor again later seems questionable anyway) > > - instead of checking whether _eden_chunk_array is NULL to detect > whether you can sample I would prefer if the same predicate as in the > initialization (gch->supports_inline_contig_alloc()) were used. > (I think the change just copied what was done in the previous version of > concurrentMarkSweepGeneration.cpp:4515 has been done) > > Maybe create a new predicate for that, used everywhere. > > (btw, throughout this code there is a lot of checking whether some > NEW_C_HEAP_ARRAY returned NULL or not, and does elaborate recovery - > however by default NEW_C_HEAP_ARRAY fails with an OOM anyway... - but > that is certainly out of scope) Thomas, could you file a CR for fixing that? Thanks. > > Same for checking whether _survivor_chunk_array is NULL - at the moment > sometimes _survivor_chunk_array is checked against NULL to avoid > entering code (in print_eden_and_survivor_chunk_arrays()). > > The condition that could be used is the one that triggers allocation of > _survivor_chunk_array: (CMSParallelRemarkEnabled && > CMSParallelSurvivorRemarkEnabled) in > concurrentMarkSweepGeneration.cpp:728. Personally, I like a check like if (_eden_chunk_array != NULL) better. I think it is more future proof. For whatever reason we don't try to dereference _eden_chunk_array if it is NULL. An assertion checking that (CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) is true would we be appropriate. Jon > > - the code to get the lock on _eden_chunk_sampling_active seems to be > needlessly complicated. > > Please consider using a dedicated Monitor in conjunction with > try_lock(). I.e. > > in CMSCollector add a > > Monitor * _cms_eden_sampling_lock; // initialize in the constructor > > and then > > if (_cms_eden_sampling_lock->try_lock()) { > > // do sampling > > _cms_eden_sampling_lock->unlock(); > } > > Monitor::try_lock() boils down to the exactly same code, uses a CAS to > obtain the lock. I cannot see a reason to roll an own lock > implementation here. > > Hiroshi, is it possible for you to take another look at this change? It > would help us a lot. If there are questions, please do not hesitate to > ask. > > Thanks a lot for your contributions so far, > Thomas > From jon.masamitsu at oracle.com Fri Jun 7 18:28:09 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 7 Jun 2013 11:28:09 -0700 (PDT) Subject: CMS parallel initial mark In-Reply-To: <1370604194.2671.55.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> Message-ID: <51B22639.7040004@oracle.com> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: > > > > > > - the code to get the lock on _eden_chunk_sampling_active seems to be > needlessly complicated. > > Please consider using a dedicated Monitor in conjunction with > try_lock(). I.e. Thomas, Any concerns about the "sneak" code in try_lock()? That would only matter if the VM thread were involved in the eden sampling along with other GC threads, yes? Jon > > in CMSCollector add a > > Monitor * _cms_eden_sampling_lock; // initialize in the constructor > > and then > > if (_cms_eden_sampling_lock->try_lock()) { > > // do sampling > > _cms_eden_sampling_lock->unlock(); > } > > Monitor::try_lock() boils down to the exactly same code, uses a CAS to > obtain the lock. I cannot see a reason to roll an own lock > implementation here. > > Hiroshi, is it possible for you to take another look at this change? It > would help us a lot. If there are questions, please do not hesitate to > ask. > > Thanks a lot for your contributions so far, > Thomas > From yamauchi at google.com Fri Jun 7 18:39:20 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 11:39:20 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B1F4A0.10902@oracle.com> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> Message-ID: Hi Thomas, Thanks for your comments. I'll be catching up with them. >>>> - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new >>>> Par_MarkRefsIntoClosure could be refactored slightly as they have >>>> exactly the same member variables. Not sure how this situation is >>>> handled in other code though, and what others (Jon) think. >>>> >>> Thomas, >>> >>> If you don't mind I'd like to keep this changeset to a minimum so >>> not do any additional refactoring. That's a good suggestion but >>> since this is the first sizable contribution I'm sponsoring, simpler >>> is better for me. >>> >> Okay. It would be a tiny additional change though, which has only been >> enabled by the addition of the Par_MarkRefsIntoClosure, and of course >> depends on whether the old serial initial marking code is kept. >> > > Thanks. > > Regarding whether to refactor MarkRefsIntoClosure and Par_MarkRefsIntoClosure, it's a valid point. I assume you are referring to factoring out the common parts into a common super class. I don't have a strong opinion. Looking at the exchanges, I'm interpreting it as "let's not do it right now." Let me know if it's not the case. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 18:46:58 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 11:46:58 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370620403.3085.2.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> <1370620403.3085.2.camel@cirrus> Message-ID: > >>> - in concurrentMarkSweepGeneration.cpp in > > >>> CMSCollector::checkpointRootsInitialWork() the new code seems a > little > > >>> strange to me. I.e. roughly it does the following: > > >>> > > >>> [...] > > >>> if (CMSParallelInitialMarkEnabled) { > > >>> [ set up parallel stuff ] > > >>> if (n_workers > 1) { > > >>> // do parallel thread > > >>> } else { > > >>> // do serial work using the current thread > > >>> } > > >>> } else { > > >>> // do serial work > > >>> } > > >>> > > >>> it does not feel good to have two serial paths here; maybe use the > > >>> serial/original code if n_workers == 1 and guarantee(n_workers > 1, > ) in > > >>> the first part of the if? Or look if the old serial path could be > > >>> removed? (or the new one). > > >> Is the inner serial path one thread executing a single chunk that > > >> does all the work (i.e., executes the parallel code but by only > > >> one thread)? > > > Yes, that is in my opinion a little awkward. It would be nice if we had > > > only one serial initial marking code path to simplify > > > debugging/maintenance if there are issues. Removal of one or the other > > > (if so) probably depends on how different the performance between the > > > two paths is. > > > > I would prefer not to remove the path that executes the parallel code > > with a single thread. It is a tool for looking for performance anomalies > > in the parallel code. I've used code like it (i.e., executes parallel > > code in > > one thread in UseParallelGC and UseParNewGC) this week while looking > > for a performance problem. > > > > The strictly serial path (which is the code we have today) would be nice > > to have for a while as a workaround for any problems. > > Okay, fair enough. > > I'll keep the code as is, keeping the two serial code paths. To add my 2 cents, I think a similar pattern existed in the CMS code (eg. CMSCollector::do_remark_parallel()), and that's probably why I followed the same pattern there. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 18:51:17 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 11:51:17 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370534525.2590.43.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> Message-ID: > - I think Jon already mentioned that it might be better to use an int > for worker_id in CMSParMarkTask::do_young_space_rescan() (if it is > actually needed) to keep that code in line with current use. > I thought there was a mass conversion from "work(int i)" to "work(uint worker_id)" around Hotspot 24 or so. Given that understanding, this sounds like going backwards from uint to int. Am I misunderstanding? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 18:58:55 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 11:58:55 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370546740.2445.31.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> Message-ID: > > > - the comments for at least CMSCollector:: > > > initialize_sequential_subtasks_for_young_gen_rescan() and > > > CMSCollector::merge_survivor_plab_arrays() should be reworked - they > > > only mention parallel rescan as entry points. > > I recommend fixing the comments though - wrong comments are not exactly > helpful, although it's just an omission here. > A minimum amount of comment edit seems fine, unless Jon has another idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 19:06:42 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 12:06:42 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370534525.2590.43.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> Message-ID: > Hth and thanks for the contribution :), You're welcome :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 19:08:33 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 12:08:33 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370620403.3085.2.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> <1370620403.3085.2.camel@cirrus> Message-ID: > > > If the old serial version (old code) is kept, this condition needs to > be > > > adapted, as data structures within the if-block started at that line > are > > > only used in the parallel case. > > > > Where talking about this like > > > > if ((CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) || > > CMSParallelInitialMarkEnabled) { > > > > and you're saying the > > > this condition needs to be > > > adapted, as data structures within the if-block started at that line > are > > > only used in the parallel case. > > Sorry, I still don't get it. Are you saying the condition in the test is > > not correct? By "parallel case" do you mean the case where > > more than one GC thread does the work? Or are you counting > > one GC thread executing the parallel code as part of the > > "parallel case"? > > Sorry for confusing you. This was just some suggestion for if one of the > serial paths is removed (e.g. in the sense of if we remove that, we need > to also remove this). Since it has been decided to keep both for now (I > think), this comment is obsolete. I take this point as a non-issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 19:09:25 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 12:09:25 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B0C2F8.6020303@oracle.com> References: <51A6AD6D.8010309@oracle.com> <51B0C2F8.6020303@oracle.com> Message-ID: Thanks for creating that. On Thu, Jun 6, 2013 at 10:12 AM, Jon Masamitsu wrote: > Hiroshi, > > The CR for this changeset will be 6412968. > > CMS: Long initial mark pauses > > Jon > > > On 5/30/13 10:56 AM, Hiroshi Yamauchi wrote: > > Thanks, Jon. Please let me know when you know more. > > On Wed, May 29, 2013 at 6:37 PM, Jon Masamitsu wrote: > > Hiroshi, > > I'm still reviewing the changes but so far this looks > very promising. I've patched you changes into a repository > and started running a few tests. I've turned on > > -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways > > Thanks. > > Jon > > > On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: > > Hi, > > I'd like to have the following contributed if it makes sense. > > 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ > > that implements a parallel version of the initial mark phase of the > CMS collector. It's relatively a straightforward parallelization of > the existing single-threaded code. With the above patch, I see about > ~3-6x speedup in the initial mark pause times. > > 2) Now, here's a related issue and a suggested fix/patch for it: > > I see that the initial mark and remark pause times sometimes spike > with a large young generation. For example, under a 1 GB young gen / 3 > GB heap setting, they occasionally spike up to ~500 milliseconds from > the normal < 100 ms range, on my machine. As far as I can tell, this > happens when the eden is fairly occupied (> 700 MB full) and not > sufficiently divided up and the parallelism decreases (at the worst > case it becomes almost single-threaded.) > > Here's a suggested patch in a separate patch: > > http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ > > that attempts to improve on this issue by implementing an alternative > way of dividing up the eden into chunks for an increased parallelism > (or better load balancing between the GC threads) for the young gen > scan portion of the remark phase (and the now-parallelized initial > mark phase.) It uses a CAS-based mechanism that samples the object > boundaries in the eden space on the slow allocation code paths (eg. at > the TLAB refill and large object allocation times) at all times. > > This approach is in contrast to the original mechanism that samples > object boundaries in the eden space asynchronously during the preclean > phase. I think the reason that the above issue happens is that when > the young generation is large, a large portion of the eden space could > get filled/allocated outside of the preclean phase (or a concurrent > collection) and the object boundaries do not get sampled > often/regularly enough. Also, it isn't very suited for the parallel > initial mark because the initial mark phase isn't preceded by the > preclean phase unlike the remark phase. According to the Dacapo > benchmarks, this alternative sampling mechanism does not have > noticeable runtime overhead despite it is engaged at all times. > > With this patch, I see that the (parallel) initial mark and remark > pause times stay below 100 ms (no spikes) under the same setting. > > Both of these features/flags are disabled by default. You're welcome > to handle the two patches separately. > > Thanks, > Hiroshi > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Fri Jun 7 19:13:33 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 12:13:33 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B1F78C.3020509@oracle.com> References: <51B00B03.1020503@oracle.com> <51B1F78C.3020509@oracle.com> Message-ID: > I'd suggest changing the default for both to true > > CMSParallelInitialMarkEnabled > CMSEdenChunksRecordAlways > > That will exercise the new code, right? Sure. Yes, that will exercise the new code with the old behavior kept behind the flags. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Fri Jun 7 19:53:23 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 12:53:23 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> Message-ID: <51B23A33.4070406@oracle.com> On 6/7/2013 11:58 AM, Hiroshi Yamauchi wrote: >>>> - the comments for at least CMSCollector:: >>>> initialize_sequential_subtasks_for_young_gen_rescan() and >>>> CMSCollector::merge_survivor_plab_arrays() should be reworked - they >>>> only mention parallel rescan as entry points. >> I recommend fixing the comments though - wrong comments are not exactly >> helpful, although it's just an omission here. >> > A minimum amount of comment edit seems fine, unless Jon has another idea. > Fixing the comments would be good. From jon.masamitsu at oracle.com Fri Jun 7 20:12:15 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 13:12:15 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> Message-ID: <51B23E9F.8050605@oracle.com> I've created the CR 8016184 Consolidate common code between MarkRefsIntoClosure and Par_MarkRefsIntoClosure Hiroshi, The renaming of Par_MarkRefsIntoClosure to ParMarkRefsIntoClosure can wait (unless you've already done it). As you've seen the use of Par_ is wide spread in CMS and should be fixed comprehensively in one changeset. Jon On 6/7/2013 11:39 AM, Hiroshi Yamauchi wrote: > Hi Thomas, > > Thanks for your comments. I'll be catching up with them. > > >>>>> - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new >>>>> Par_MarkRefsIntoClosure could be refactored slightly as they have >>>>> exactly the same member variables. Not sure how this situation is >>>>> handled in other code though, and what others (Jon) think. >>>>> >>>> Thomas, >>>> >>>> If you don't mind I'd like to keep this changeset to a minimum so >>>> not do any additional refactoring. That's a good suggestion but >>>> since this is the first sizable contribution I'm sponsoring, simpler >>>> is better for me. >>>> >>> Okay. It would be a tiny additional change though, which has only been >>> enabled by the addition of the Par_MarkRefsIntoClosure, and of course >>> depends on whether the old serial initial marking code is kept. >>> >> Thanks. >> >> > Regarding whether to refactor MarkRefsIntoClosure and > Par_MarkRefsIntoClosure, it's a valid point. I assume you are referring to > factoring out the common parts into a common super class. > > I don't have a strong opinion. Looking at the exchanges, I'm interpreting > it as "let's not do it right now." Let me know if it's not the case. > From thomas.schatzl at oracle.com Fri Jun 7 21:47:43 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Jun 2013 23:47:43 +0200 Subject: CMS parallel initial mark In-Reply-To: References: <1370534525.2590.43.camel@cirrus> Message-ID: <1370641663.7429.8.camel@cirrus> Hi, On Fri, 2013-06-07 at 11:51 -0700, Hiroshi Yamauchi wrote: > > - I think Jon already mentioned that it might be better to > use an int > for worker_id in CMSParMarkTask::do_young_space_rescan() (if > it is > actually needed) to keep that code in line with current use. > > > I thought there was a mass conversion from "work(int i)" to "work(uint > worker_id)" around Hotspot 24 or so. Given that understanding, this > sounds like going backwards from uint to int. Am I misunderstanding? No, you are right - when checking that statement I happened to look at exactly the files where for some reason this apparently not been changed - i.e. concurrentG1RefineThread.?pp and concurrentG1Refine.cpp. :( All others seem to be uint (grepping through sources), so keep it. I will file a CR for that unless somebody else knows a good reason to keep them ints in these files. Sorry. Thomas From jon.masamitsu at oracle.com Fri Jun 7 21:52:16 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 07 Jun 2013 14:52:16 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <51A6AD6D.8010309@oracle.com> <51B0C2F8.6020303@oracle.com> Message-ID: <51B25610.2030900@oracle.com> On 6/7/2013 12:09 PM, Hiroshi Yamauchi wrote: > Thanks for creating that. It's an old bug really. For the partitioning during young allocation, we can use 6990419 CMS: Remaining work for 6572569: consistently skewed work distribution in (long) re-mark pauses Another old one. Were these changes reviewed internally by any openjdk hotspot members? Jon > On Thu, Jun 6, 2013 at 10:12 AM, Jon Masamitsu wrote: > >> Hiroshi, >> >> The CR for this changeset will be 6412968. >> >> CMS: Long initial mark pauses >> >> Jon >> >> >> On 5/30/13 10:56 AM, Hiroshi Yamauchi wrote: >> >> Thanks, Jon. Please let me know when you know more. >> >> On Wed, May 29, 2013 at 6:37 PM, Jon Masamitsu wrote: >> >> Hiroshi, >> >> I'm still reviewing the changes but so far this looks >> very promising. I've patched you changes into a repository >> and started running a few tests. I've turned on >> >> -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways >> >> Thanks. >> >> Jon >> >> >> On 5/28/2013 5:24 PM, Hiroshi Yamauchi wrote: >> >> Hi, >> >> I'd like to have the following contributed if it makes sense. >> >> 1) Here's a patch (against a recent revision of the hsx/hotspot-gc repo): >> >> http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.00/ >> >> that implements a parallel version of the initial mark phase of the >> CMS collector. It's relatively a straightforward parallelization of >> the existing single-threaded code. With the above patch, I see about >> ~3-6x speedup in the initial mark pause times. >> >> 2) Now, here's a related issue and a suggested fix/patch for it: >> >> I see that the initial mark and remark pause times sometimes spike >> with a large young generation. For example, under a 1 GB young gen / 3 >> GB heap setting, they occasionally spike up to ~500 milliseconds from >> the normal < 100 ms range, on my machine. As far as I can tell, this >> happens when the eden is fairly occupied (> 700 MB full) and not >> sufficiently divided up and the parallelism decreases (at the worst >> case it becomes almost single-threaded.) >> >> Here's a suggested patch in a separate patch: >> >> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ >> >> that attempts to improve on this issue by implementing an alternative >> way of dividing up the eden into chunks for an increased parallelism >> (or better load balancing between the GC threads) for the young gen >> scan portion of the remark phase (and the now-parallelized initial >> mark phase.) It uses a CAS-based mechanism that samples the object >> boundaries in the eden space on the slow allocation code paths (eg. at >> the TLAB refill and large object allocation times) at all times. >> >> This approach is in contrast to the original mechanism that samples >> object boundaries in the eden space asynchronously during the preclean >> phase. I think the reason that the above issue happens is that when >> the young generation is large, a large portion of the eden space could >> get filled/allocated outside of the preclean phase (or a concurrent >> collection) and the object boundaries do not get sampled >> often/regularly enough. Also, it isn't very suited for the parallel >> initial mark because the initial mark phase isn't preceded by the >> preclean phase unlike the remark phase. According to the Dacapo >> benchmarks, this alternative sampling mechanism does not have >> noticeable runtime overhead despite it is engaged at all times. >> >> With this patch, I see that the (parallel) initial mark and remark >> pause times stay below 100 ms (no spikes) under the same setting. >> >> Both of these features/flags are disabled by default. You're welcome >> to handle the two patches separately. >> >> Thanks, >> Hiroshi >> >> >> From yamauchi at google.com Fri Jun 7 22:20:27 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Fri, 7 Jun 2013 15:20:27 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B23E9F.8050605@oracle.com> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> <51B23E9F.8050605@oracle.com> Message-ID: Here's an update version of the first patch based on what's been discussed so far: http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.02/ I'll catch up with the comments on the other patch later. On Fri, Jun 7, 2013 at 1:12 PM, Jon Masamitsu wrote: > I've created the CR 8016184 > > Consolidate common code between MarkRefsIntoClosure and > Par_MarkRefsIntoClosure > > Hiroshi, > > The renaming of Par_MarkRefsIntoClosure to ParMarkRefsIntoClosure can > wait (unless you've already done it). As you've seen the use of > Par_ is wide spread in CMS and should be fixed > comprehensively in one changeset. > > Jon > > > > On 6/7/2013 11:39 AM, Hiroshi Yamauchi wrote: > >> Hi Thomas, >> >> Thanks for your comments. I'll be catching up with them. >> >> >> - in cmsOopClosures.hpp, MarkRefsIntoClosure and the new >>>>>> Par_MarkRefsIntoClosure could be refactored slightly as they have >>>>>> exactly the same member variables. Not sure how this situation is >>>>>> handled in other code though, and what others (Jon) think. >>>>>> >>>>>> Thomas, >>>>> >>>>> If you don't mind I'd like to keep this changeset to a minimum so >>>>> not do any additional refactoring. That's a good suggestion but >>>>> since this is the first sizable contribution I'm sponsoring, simpler >>>>> is better for me. >>>>> >>>>> Okay. It would be a tiny additional change though, which has only been >>>> enabled by the addition of the Par_MarkRefsIntoClosure, and of course >>>> depends on whether the old serial initial marking code is kept. >>>> >>>> Thanks. >>> >>> >>> Regarding whether to refactor MarkRefsIntoClosure and >> Par_MarkRefsIntoClosure, it's a valid point. I assume you are referring to >> factoring out the common parts into a common super class. >> >> I don't have a strong opinion. Looking at the exchanges, I'm interpreting >> it as "let's not do it right now." Let me know if it's not the case. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alejandro.murillo at oracle.com Sat Jun 8 00:45:26 2013 From: alejandro.murillo at oracle.com (alejandro.murillo at oracle.com) Date: Sat, 08 Jun 2013 00:45:26 +0000 Subject: hg: hsx/hotspot-gc/hotspot: 29 new changesets Message-ID: <20130608004630.F07C4480B8@hg.openjdk.java.net> Changeset: 61dcf187a198 Author: katleman Date: 2013-06-06 09:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/61dcf187a198 Added tag jdk8-b93 for changeset 573d86d412cd ! .hgtags Changeset: a1ebd310d5c1 Author: iklam Date: 2013-05-28 16:36 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/a1ebd310d5c1 8014912: Restore PrintSharedSpaces functionality after NPG Summary: Added dumping of object sizes in CDS archive, sorted by MetaspaceObj::Type Reviewed-by: coleenp, acorn ! src/share/vm/memory/allocation.cpp ! src/share/vm/memory/allocation.hpp ! src/share/vm/memory/metaspace.cpp ! src/share/vm/memory/metaspace.hpp ! src/share/vm/memory/metaspaceShared.cpp ! src/share/vm/oops/annotations.cpp ! src/share/vm/oops/constMethod.cpp ! src/share/vm/oops/constantPool.cpp ! src/share/vm/oops/cpCache.cpp ! src/share/vm/oops/klass.cpp ! src/share/vm/oops/method.cpp ! src/share/vm/oops/methodCounters.cpp ! src/share/vm/oops/methodData.cpp ! src/share/vm/oops/symbol.cpp ! src/share/vm/utilities/array.hpp Changeset: fe00365c8f31 Author: sspitsyn Date: 2013-05-30 11:46 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/fe00365c8f31 8015436: compiler/ciReplay/TestSA.sh fails with assert() index is out of bounds Summary: The InstanceKlass _initial_method_idnum value must be adjusted if overpass methods are added. Reviewed-by: twisti, kvn Contributed-by: serguei.spitsyn at oracle.com ! src/share/vm/classfile/defaultMethods.cpp + test/compiler/8015436/Test8015436.java Changeset: a589c78a8811 Author: rbackman Date: 2013-05-31 13:02 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/a589c78a8811 8014709: Constructor.getAnnotatedReturnType() returns empty AnnotatedType Reviewed-by: stefank, rbackman Contributed-by: Joel Borggren-Franck ! src/share/vm/runtime/reflection.cpp ! test/runtime/8007320/ConstMethodTest.java Changeset: efe8b7d64424 Author: ctornqvi Date: 2013-05-31 20:24 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/efe8b7d64424 6726963: multi_allocate() call does not CHECK_NULL and causes crash in fastdebug bits Summary: Using CHECK_NULL when calling multi_allocate() from the corresponding reflection code; added test for this condition Reviewed-by: dholmes, minqi Contributed-by: Mikhailo Seledtsov ! src/share/vm/runtime/reflection.cpp + test/runtime/memory/MultiAllocateNullCheck.java Changeset: 532c55335fb6 Author: dcubed Date: 2013-06-01 09:28 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/532c55335fb6 Merge Changeset: 4552a7633a07 Author: hseigel Date: 2013-06-03 10:00 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/4552a7633a07 8015385: Remove RelaxAccessControlCheck for JDK 8 bytecodes Summary: Check bytecode versions along with RelaxAccessControlCheck version Reviewed-by: dholmes, acorn ! src/share/vm/classfile/verifier.hpp ! src/share/vm/runtime/reflection.cpp Changeset: e7d29a019a3c Author: sspitsyn Date: 2013-06-03 14:28 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/e7d29a019a3c 8014052: JSR292: assert(end_offset == next_offset) failed: matched ending Summary: A call to the finalize_operands_merge() must be unconditional Reviewed-by: kvn, twisti Contributed-by: serguei.spitsyn at oracle.com ! src/share/vm/prims/jvmtiRedefineClasses.cpp Changeset: 2f004f9dc9e1 Author: sspitsyn Date: 2013-06-04 01:06 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/2f004f9dc9e1 8015803: Test8015436.java fails 'can not access a member of class Test8015436 with modifiers "public static"' Summary: Newly added test has an issue: the main class must be public Reviewed-by: kvn, jbachorik, coleenp Contributed-by: serguei.spitsyn at oracle.com ! test/compiler/8015436/Test8015436.java Changeset: 04551f4dbdb9 Author: nloodin Date: 2013-06-05 09:47 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/04551f4dbdb9 Merge Changeset: 62e7bac9524f Author: dcubed Date: 2013-06-04 19:39 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/62e7bac9524f 8010257: remove unused thread-local variables _ScratchA and _ScratchB Summary: Remove dead code. Reviewed-by: twisti, coleenp ! src/share/vm/runtime/thread.hpp Changeset: 6bf8b8bb7c19 Author: hseigel Date: 2013-06-05 14:12 -0400 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/6bf8b8bb7c19 8009302: Mac OS X: JVM crash on infinite recursion on Appkit Thread Summary: Use SA_ONSTACK flag to ensure signal gets delivered properly. Reviewed-by: dholmes, coleenp Contributed-by: gerard.ziemski at oracle.com ! src/os/bsd/vm/os_bsd.cpp Changeset: f8c8cace25ad Author: dcubed Date: 2013-06-06 05:56 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/f8c8cace25ad Merge ! src/os/bsd/vm/os_bsd.cpp Changeset: 320b4e0f0892 Author: roland Date: 2013-05-30 11:21 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/320b4e0f0892 8015585: Missing regression test for 8011771 Summary: missing regression test Reviewed-by: kvn + test/compiler/8011771/Test8011771.java Changeset: f15fe46d8c00 Author: twisti Date: 2013-05-30 08:37 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/f15fe46d8c00 8015266: fix some -Wsign-compare warnings in adlc Reviewed-by: kvn ! src/share/vm/adlc/archDesc.cpp ! src/share/vm/adlc/dict2.cpp ! src/share/vm/adlc/formssel.cpp ! src/share/vm/adlc/formssel.hpp ! src/share/vm/adlc/output_c.cpp Changeset: 28e5aed7f3a6 Author: roland Date: 2013-05-31 14:40 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/28e5aed7f3a6 8009981: nashorn tests fail with -XX:+VerifyStack Summary: nmethod::preserve_callee_argument_oops() must take appendix into account. Reviewed-by: kvn, twisti ! src/share/vm/code/nmethod.cpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/frame.cpp ! src/share/vm/runtime/frame.hpp ! src/share/vm/runtime/sharedRuntime.cpp ! src/share/vm/runtime/sharedRuntime.hpp Changeset: 83dcb116fdb1 Author: kvn Date: 2013-05-31 13:54 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/83dcb116fdb1 8015441: runThese crashed with assert(opcode == Op_ConP || opcode == Op_ThreadLocal || opcode == Op_CastX2P ..) failed: sanity Summary: Relax the assert to accept any raw ptr types. Reviewed-by: roland ! src/share/vm/opto/escape.cpp Changeset: c07dd9be16e8 Author: anoll Date: 2013-05-31 06:41 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/c07dd9be16e8 8013496: Code cache management command line options work only in special order. Another order of arguments does not deliver the second parameter to the jvm. Summary: Moved check that ReservedCodeCacheSize >= InitialCodeCacheSize to Arguments::check_vm_args_consistency(). As a result, the ordering in which the two parameters are given to the VM is not relevant. Added a regression test. Reviewed-by: kvn, twisti ! src/share/vm/runtime/arguments.cpp + test/compiler/8013496/Test8013496.sh Changeset: 603ca7e51354 Author: roland Date: 2013-04-24 11:49 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/603ca7e51354 8010460: Interpreter on some platforms loads ConstMethod::_max_stack and misses extra stack slots for JSR 292 Summary: ConstMethod::max_stack() doesn't account for JSR 292 appendix. Reviewed-by: kvn ! src/cpu/sparc/vm/cppInterpreter_sparc.cpp ! src/cpu/sparc/vm/interp_masm_sparc.cpp ! src/cpu/sparc/vm/templateInterpreter_sparc.cpp ! src/cpu/x86/vm/cppInterpreter_x86.cpp ! src/cpu/x86/vm/templateInterpreter_x86_32.cpp ! src/cpu/x86/vm/templateInterpreter_x86_64.cpp ! src/share/vm/interpreter/bytecodeInterpreter.cpp ! src/share/vm/oops/method.hpp ! src/share/vm/opto/matcher.cpp Changeset: 813f26e34135 Author: anoll Date: 2013-06-03 08:52 +0200 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/813f26e34135 8013329: File leak in hotspot/src/share/vm/compiler/compileBroker.cpp Summary: Added calling of the destructor of CompileLog so that files are closed. Added/moved memory allocation/deallocation of the string that contains the name of the log file to class CompileLog. Reviewed-by: kvn, roland ! src/share/vm/compiler/compileBroker.cpp ! src/share/vm/compiler/compileLog.cpp ! src/share/vm/compiler/compileLog.hpp Changeset: b274ac1dbe11 Author: adlertz Date: 2013-06-03 12:39 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/b274ac1dbe11 8005956: C2: assert(!def_outside->member(r)) failed: Use of external LRG overlaps the same LRG defined in this block Summary: Disable re-materialization of reaching definitions (which have live inputs) for phi nodes when spilling. Reviewed-by: twisti, kvn ! src/share/vm/opto/reg_split.cpp Changeset: 770e91e578a6 Author: kvn Date: 2013-06-03 14:02 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/770e91e578a6 Merge Changeset: 075ea888b039 Author: morris Date: 2013-06-04 12:06 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/075ea888b039 8010724: [parfait] Null pointer dereference in hotspot/src/share/vm/c1/c1_LIRGenerator.cpp Summary: added guarantee() Reviewed-by: kvn ! src/share/vm/c1/c1_LIRGenerator.cpp Changeset: 2cb5d5f6d5e5 Author: simonis Date: 2013-06-04 22:16 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/2cb5d5f6d5e5 8015252: Enable HotSpot build with Clang Reviewed-by: twisti, dholmes, kvn ! make/bsd/makefiles/adlc.make ! make/bsd/makefiles/gcc.make ! make/bsd/makefiles/vm.make ! make/linux/makefiles/adlc.make ! make/linux/makefiles/gcc.make ! src/os/bsd/vm/os_bsd.cpp ! src/os_cpu/linux_x86/vm/linux_x86_32.s ! src/os_cpu/linux_x86/vm/os_linux_x86.cpp Changeset: 609aad72004a Author: anoll Date: 2013-06-06 09:29 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/609aad72004a 8014246: remove assert to catch access to object headers in index_oop_from_field_offset_long Reviewed-by: twisti, jrose ! src/share/vm/prims/unsafe.cpp Changeset: ef1818846c22 Author: kvn Date: 2013-06-06 11:02 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/ef1818846c22 Merge ! src/os/bsd/vm/os_bsd.cpp Changeset: 3c78a14da19d Author: amurillo Date: 2013-06-07 09:25 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/3c78a14da19d Merge ! .hgtags Changeset: 1beed1f6f9ed Author: amurillo Date: 2013-06-07 09:25 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/1beed1f6f9ed Added tag hs25-b36 for changeset 3c78a14da19d ! .hgtags Changeset: d0add7016434 Author: amurillo Date: 2013-06-07 09:33 -0700 URL: http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/d0add7016434 8016078: new hotspot build - hs25-b37 Reviewed-by: jcoomes ! make/hotspot_version From bengt.rutisson at oracle.com Sat Jun 8 11:07:11 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Sat, 8 Jun 2013 13:07:11 +0200 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B208F8.6030200@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> <51B1F87D.7060105@oracle.com> <51B208F8.6030200@oracle.com> Message-ID: <228A79C9-6839-4EED-B8DB-2F61E669E047@oracle.com> Vladimir, I would prefer to remove the tty->cr() call you changed and instead use gclog_or_tty->print_cr() in the gclog_or_tty->print() statement before. Other than that it looks good. Bengt 7 jun 2013 kl. 18:23 skrev Jesper Wilhelmsson : > Vladimir Kempik skrev 7/6/13 5:13 PM: >> Thanks for comments >> >> Here is updated webrev - >> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.01/ > > Thanks for fixing this! > Looks good, ship it (once the gc repo is open for pushes again). > /Jesper > > >> On 07.06.2013 18:37, Jesper Wilhelmsson wrote: >>> Vladimir, >>> >>> Looks good, >>> >>> When I fixed the same thing yesterday I noticed that the method above, >>> PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, writes its >>> cr() to tty instead of gclog_or_tty. Do you mind fixing that in the same >>> change since it is slightly related? >>> >>> And I don't know how to stress this enough, you are not the first one who did >>> it, never ever work on a bug without assigning it to yourself, or talk to the >>> assigned engineer first. Having two engineers working on the same bug is not >>> efficient use of our time. >> Sorry about that. >> >> Vladimir. >>> /Jesper >>> >>> >>> >>> Vladimir Kempik skrev 7/6/13 3:46 PM: >>>> Hi all, >>>> >>>> Could I have a couple of reviews for this change? >>>> >>>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >>>> >>>> With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: >>>> >>>> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: 251648 overflow: >>>> falseAdaptiveSizeStart: 4,136 collection: 16 >>>> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >>>> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 tenuring_thresh: 1 >>>> target_size: 1835008 >>>> >>>> and there is no space between overflow: false and AdaptiveSizeStart: >>>> >>>> The patch adds line break after overflow: false. >>>> >>>> Thanks, >>>> Vladimir >> From vladimir.kempik at oracle.com Mon Jun 10 15:24:42 2013 From: vladimir.kempik at oracle.com (Vladimir Kempik) Date: Mon, 10 Jun 2013 19:24:42 +0400 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <228A79C9-6839-4EED-B8DB-2F61E669E047@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> <51B1F87D.7060105@oracle.com> <51B208F8.6030200@oracle.com> <228A79C9-6839-4EED-B8DB-2F61E669E047@oracle.com> Message-ID: <51B5EFBA.9090202@oracle.com> Hello. Thanks for comment. Here is updated webrev - http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.02/ Vladimir. On 08.06.2013 15:07, Bengt Rutisson wrote: > Vladimir, > > I would prefer to remove the tty->cr() call you changed and instead use gclog_or_tty->print_cr() in the gclog_or_tty->print() statement before. > > Other than that it looks good. > > Bengt > > 7 jun 2013 kl. 18:23 skrev Jesper Wilhelmsson : > >> Vladimir Kempik skrev 7/6/13 5:13 PM: >>> Thanks for comments >>> >>> Here is updated webrev - >>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.01/ >> Thanks for fixing this! >> Looks good, ship it (once the gc repo is open for pushes again). >> /Jesper >> >> >>> On 07.06.2013 18:37, Jesper Wilhelmsson wrote: >>>> Vladimir, >>>> >>>> Looks good, >>>> >>>> When I fixed the same thing yesterday I noticed that the method above, >>>> PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, writes its >>>> cr() to tty instead of gclog_or_tty. Do you mind fixing that in the same >>>> change since it is slightly related? >>>> >>>> And I don't know how to stress this enough, you are not the first one who did >>>> it, never ever work on a bug without assigning it to yourself, or talk to the >>>> assigned engineer first. Having two engineers working on the same bug is not >>>> efficient use of our time. >>> Sorry about that. >>> >>> Vladimir. >>>> /Jesper >>>> >>>> >>>> >>>> Vladimir Kempik skrev 7/6/13 3:46 PM: >>>>> Hi all, >>>>> >>>>> Could I have a couple of reviews for this change? >>>>> >>>>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >>>>> >>>>> With next option -XX:+PrintAdapriveSizePolicy, java log looks like this: >>>>> >>>>> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: 251648 overflow: >>>>> falseAdaptiveSizeStart: 4,136 collection: 16 >>>>> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >>>>> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 tenuring_thresh: 1 >>>>> target_size: 1835008 >>>>> >>>>> and there is no space between overflow: false and AdaptiveSizeStart: >>>>> >>>>> The patch adds line break after overflow: false. >>>>> >>>>> Thanks, >>>>> Vladimir From erik.helin at oracle.com Sun Jun 9 07:43:15 2013 From: erik.helin at oracle.com (Erik Helin) Date: Sun, 9 Jun 2013 09:43:15 +0200 Subject: RFR (XXS): 8016170: GC id variable in gcTrace.cpp should use typedef GCId Message-ID: <20130609074315.GC1963@ehelin-thinkpad> Hi all, this very small change ensures that variable GCTracer_next_gc_id in gcTrace.cpp always has the same type as the typedef GCId in gcTrace.hpp. Webrev: http://cr.openjdk.java.net/~ehelin/8016170/webrev.00/ Testing: JPRT Thanks, Erik From erik.helin at oracle.com Sun Jun 9 07:28:49 2013 From: erik.helin at oracle.com (Erik Helin) Date: Sun, 9 Jun 2013 09:28:49 +0200 Subject: RFR (S): 8015683: object_count_after_gc should have the same timestamp for all events Message-ID: <20130609072848.GA1963@ehelin-thinkpad> Hi all, this small change ensures that all vm/gc/detailed/object_count_after_gc trace events that are sent during the same GC will have identical timestamps. Please note that this change is based upon another change that is out for review - "8015972: Refactor the sending of the object count after GC event". Webrev: http://cr.openjdk.java.net/~ehelin/8015683/webrev.00/ Testing: JPRT Thanks, Erik From bengt.rutisson at oracle.com Mon Jun 10 06:28:52 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 10 Jun 2013 08:28:52 +0200 Subject: RFR (S): 8015972: Refactor the sending of the object count after GC event In-Reply-To: <20130607134240.GC2162@ehelin-thinkpad> References: <20130605171108.GA3096@ehelin-thinkpad> <20130607134240.GC2162@ehelin-thinkpad> Message-ID: <51B57224.3000307@oracle.com> Erik, Looks good. Bengt On 6/7/13 3:42 PM, Erik Helin wrote: > All, > > based on internal feedback, I've updated the change, please see new > webrev at: > > http://cr.openjdk.java.net/~ehelin/8015972/webrev.01/ > > The changes from webrev.00 are: > - Using the typedef GCId instead of jlong > - Remove the #include "utilities/macro.hpp" from > objectCountEventSender.hpp > - Use public inheritance from AllStatic > - Sort includes in objectCountEventSender.hpp > - Remove the "should_commit" check from ObjectCountEventSender::send, > add an assert instead > - Use >= instead of > in ObjectCountEventSenderClosure::should_send_event > - Remove the "is_alive" variable from ObjectCountFilter::do_object_b > > Thanks, > Erik > > On 2013-06-05, Erik Helin wrote: >> Hi all, >> >> this small change refactors the way the object count after GC trace >> event is sent. >> >> Webrev: >> http://cr.openjdk.java.net/~ehelin/8015972/webrev.00/ >> >> Bug: >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8015972 >> >> Testing: >> JPRT >> >> Thanks, >> Erik From jesper.wilhelmsson at oracle.com Mon Jun 10 16:57:45 2013 From: jesper.wilhelmsson at oracle.com (Jesper Wilhelmsson) Date: Mon, 10 Jun 2013 18:57:45 +0200 Subject: RFR (XXS): 8016170: GC id variable in gcTrace.cpp should use typedef GCId In-Reply-To: <20130609074315.GC1963@ehelin-thinkpad> References: <20130609074315.GC1963@ehelin-thinkpad> Message-ID: <51B60589.7080400@oracle.com> Looks good. Ship it! /Jesper Erik Helin skrev 9/6/13 9:43 AM: > Hi all, > > this very small change ensures that variable GCTracer_next_gc_id in > gcTrace.cpp always has the same type as the typedef GCId in gcTrace.hpp. > > Webrev: > http://cr.openjdk.java.net/~ehelin/8016170/webrev.00/ > > Testing: > JPRT > > Thanks, > Erik > From jon.masamitsu at oracle.com Mon Jun 10 19:21:32 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 10 Jun 2013 12:21:32 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> <51B23E9F.8050605@oracle.com> Message-ID: <51B6273C.1040301@oracle.com> On 6/7/13 3:20 PM, Hiroshi Yamauchi wrote: > Here's an update version of the first patch based on what's been > discussed so far: > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.02/ > > > I'll catch up with the comments on the other patch later. Changes look good. Thanks. Jon > > > On Fri, Jun 7, 2013 at 1:12 PM, Jon Masamitsu > > wrote: > > I've created the CR 8016184 > > Consolidate common code between MarkRefsIntoClosure and > Par_MarkRefsIntoClosure > > Hiroshi, > > The renaming of Par_MarkRefsIntoClosure to ParMarkRefsIntoClosure can > wait (unless you've already done it). As you've seen the use of > Par_ is wide spread in CMS and should be fixed > comprehensively in one changeset. > > Jon > > > > On 6/7/2013 11:39 AM, Hiroshi Yamauchi wrote: > > Hi Thomas, > > Thanks for your comments. I'll be catching up with them. > > > - in cmsOopClosures.hpp, > MarkRefsIntoClosure and the new > Par_MarkRefsIntoClosure could be refactored > slightly as they have > exactly the same member variables. Not sure > how this situation is > handled in other code though, and what others > (Jon) think. > > Thomas, > > If you don't mind I'd like to keep this changeset > to a minimum so > not do any additional refactoring. That's a good > suggestion but > since this is the first sizable contribution I'm > sponsoring, simpler > is better for me. > > Okay. It would be a tiny additional change though, > which has only been > enabled by the addition of the > Par_MarkRefsIntoClosure, and of course > depends on whether the old serial initial marking code > is kept. > > Thanks. > > > Regarding whether to refactor MarkRefsIntoClosure and > Par_MarkRefsIntoClosure, it's a valid point. I assume you are > referring to > factoring out the common parts into a common super class. > > I don't have a strong opinion. Looking at the exchanges, I'm > interpreting > it as "let's not do it right now." Let me know if it's not the > case. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Mon Jun 10 19:45:38 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 10 Jun 2013 21:45:38 +0200 Subject: CMS parallel initial mark In-Reply-To: <51B223C6.7080805@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> Message-ID: <1370893538.2674.9.camel@cirrus> Hi, On Fri, 2013-06-07 at 11:17 -0700, Jon Masamitsu wrote: > On 6/7/2013 4:23 AM, Thomas Schatzl wrote: > > Hi all, > > > > some notes about the second patch, the modified eden boundary > > sampling; in general I think the idea is good (Jon suggested that it > > should become default), but imho there are some issues with the > > implementation. > > > > On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: > >> Hi, > >> > >> Here's a suggested patch in a separate patch: > >> > >> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ > >> > >> that attempts to improve on this issue by implementing an alternative > >> way of dividing up the eden into chunks for an increased parallelism > >> (or better load balancing between the GC threads) for the young gen > >> scan portion of the remark phase (and the now-parallelized initial > >> mark phase.) It uses a CAS-based mechanism that samples the object > >> boundaries in the eden space on the slow allocation code paths (eg. at > >> the TLAB refill and large object allocation times) at all times. > > - this patch is for DefNew/CMS only it seems. Is this correct? > > > > - in the Hotspot code base explicit != NULL checks are done for whatever > > reason. To keep the same style it would be nice to update the checks > > whether to do the sampling ("if (CMSEdenChunksRecordAlways && > > _next_gen)") accordingly. > > > > (the next point is not an issue of the patch) > > - the check whether there is a _next_gen is odd - I do not think that > > DefNew works as a generation without an older generation, but I'm not > > sure. > You're correct that DefNew needs to have a _next_gen. > > > Looking at other similar code checking for _next_gen != NULL, the > > response is typically to update _next_gen and then asserting that > > _next_gen is != NULL. E.g. > > > > if (_next_gen == NULL) { > > GenCollectedHeap* gch = GenCollectedHeap::heap(); > > _next_gen = gch->next_gen(this); > > assert(_next_gen != NULL, > > "This must be the youngest gen, and not the only gen"); > > } > > > > Jon? > > Yes, except in a very few situations, _next_gen should be set. > Exceptions I would expect is during initialization. At the point > Thomas indicates and assertion would be sufficient. > > assert(_next_gen != NULL, "..."); Thanks. > > > > > > - I think CMSCollector::_eden_chunk_sampling_active should be volatile > > to avoid any clever compilers skipping the read in sample_eden_chunk(). > > (Although I don't think it's possible here). See also below for avoiding > > that variable completely. > > Could you explain you thinking for making it volatile. As you have, I > didn't think > it was necessary and thought ,"If not necessary, don't do it". It's not necessary I think, just as a reminder to the compiler to not optimize that load away. I think the Atomic instruction contain the necessary instructions though. > > (btw, throughout this code there is a lot of checking whether some > > NEW_C_HEAP_ARRAY returned NULL or not, and does elaborate recovery - > > however by default NEW_C_HEAP_ARRAY fails with an OOM anyway... - but > > that is certainly out of scope) > > Thomas, could you file a CR for fixing that? Thanks. CR 8016276: CMS concurrentMarkSweepGeneration contains lots of unnecessary allocation failure handling > > Same for checking whether _survivor_chunk_array is NULL - at the moment > > sometimes _survivor_chunk_array is checked against NULL to avoid > > entering code (in print_eden_and_survivor_chunk_arrays()). > > > > The condition that could be used is the one that triggers allocation of > > _survivor_chunk_array: (CMSParallelRemarkEnabled && > > CMSParallelSurvivorRemarkEnabled) in > > concurrentMarkSweepGeneration.cpp:728. > > Personally, I like a check like if (_eden_chunk_array != NULL) > better. I think it is more future proof. For whatever reason we don't > try to dereference _eden_chunk_array if it is NULL. An assertion > checking that > > (CMSParallelRemarkEnabled && > CMSParallelSurvivorRemarkEnabled) > > is true would we be appropriate. Also good. Thomas From thomas.schatzl at oracle.com Mon Jun 10 19:49:54 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 10 Jun 2013 21:49:54 +0200 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <51B22639.7040004@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> Message-ID: <1370893794.2674.12.camel@cirrus> Hi, On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: > On 6/7/2013 4:23 AM, Thomas Schatzl wrote: > > > > - the code to get the lock on _eden_chunk_sampling_active seems to be > > needlessly complicated. > > > > Please consider using a dedicated Monitor in conjunction with > > try_lock(). I.e. > Thomas, > > Any concerns about the "sneak" code in try_lock()? That would only > matter if the VM thread were involved in the eden sampling along with > other GC threads, yes? I do not think so in this case. Only the VM thread can sneak, and only during safepoint. The sampling is done only for eden space and eventually from-space, which are never allocated to during gc (only by java threads outside of stw pauses), aren't they? The code could assert that in the sampling too. Even then, this sampling is only active for defnew at the moment. The only thread that is active in a defnew collection is the vm thread where sneaking cannot happen. Thomas From john.cuthbertson at oracle.com Mon Jun 10 22:38:03 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Mon, 10 Jun 2013 15:38:03 -0700 Subject: RFR (XXS): 8016170: GC id variable in gcTrace.cpp should use typedef GCId In-Reply-To: <20130609074315.GC1963@ehelin-thinkpad> References: <20130609074315.GC1963@ehelin-thinkpad> Message-ID: <51B6554B.2040404@oracle.com> Hi Erik, Looks good. Ship it. JohnC On 6/9/2013 12:43 AM, Erik Helin wrote: > Hi all, > > this very small change ensures that variable GCTracer_next_gc_id in > gcTrace.cpp always has the same type as the typedef GCId in gcTrace.hpp. > > Webrev: > http://cr.openjdk.java.net/~ehelin/8016170/webrev.00/ > > Testing: > JPRT > > Thanks, > Erik From thomas.schatzl at oracle.com Tue Jun 11 07:33:33 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 11 Jun 2013 09:33:33 +0200 Subject: int vs. uint for worker id [Was: Re: CMS parallel initial mark] In-Reply-To: <1370641663.7429.8.camel@cirrus> References: <1370534525.2590.43.camel@cirrus> <1370641663.7429.8.camel@cirrus> Message-ID: <1370936013.2648.13.camel@cirrus> Hi again, On Fri, 2013-06-07 at 23:47 +0200, Thomas Schatzl wrote: > Hi, > > On Fri, 2013-06-07 at 11:51 -0700, Hiroshi Yamauchi wrote: > > > > - I think Jon already mentioned that it might be better to > > use an int > > for worker_id in CMSParMarkTask::do_young_space_rescan() (if > > it is > > actually needed) to keep that code in line with current use. > > > > > > I thought there was a mass conversion from "work(int i)" to "work(uint > > worker_id)" around Hotspot 24 or so. Given that understanding, this > > sounds like going backwards from uint to int. Am I misunderstanding? > > No, you are right - when checking that statement I happened to look at > exactly the files where for some reason this apparently not been changed > - i.e. concurrentG1RefineThread.?pp and concurrentG1Refine.cpp. :( Fyi, the original changed happened in CR 7121618 ( http://bugs.sun.com/view_bug.do?bug_id=7121618 ) I added a new CR "8016302 Change type of the number of GC workers to unsigned int (2)" to track that. Thomas From erik.helin at oracle.com Tue Jun 11 15:27:59 2013 From: erik.helin at oracle.com (Erik Helin) Date: Tue, 11 Jun 2013 17:27:59 +0200 Subject: RFR: 8013590: NPG: Add a memory pool MXBean for Metaspace In-Reply-To: <51A89305.3070005@oracle.com> References: <20130529084430.GD1956@ehelin-thinkpad> <51A89305.3070005@oracle.com> Message-ID: <20130611152759.GD3566@ehelin-thinkpad> Hi Mikael and Jon, thanks for reviewing! I have updated the webrev. The changes from webrev.00 are: - Only exposing the compressed class space memory pool if compressed classes are used. - Renamed the name of the pool to "Compressed Class Space" (was "Compressed Klass Space", note the 'K'). - Renamed the variable _cks_pool to _compressed_class_pool. - Using max_uintx instead of _undefined_size. - Updated the test and the rest of the code to take these new change into account. Please see new webrev at: http://cr.openjdk.java.net/~ehelin/8013590/webrev.01/ On 2013-06-04, Jon Masamitsu wrote: > http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/test/gc/metaspace/TestMetaspaceMemoryPool.java.html > > The functions assertEquals(), assertTrue(), and assertDefined() seem > useful. Is there > a more global place where they can be put so other can use them? I agree that assertTrue and assertEquals should be move to a separate file, but I think that assertDefined is mostly usable for tests dealing with MXMemoryBeans (since defined is not always the same as "differ from -1"). I would like to do these changes as a separate change. Jon, what do you about that? Thanks, Erik On 2013-05-31, Mikael Gerdin wrote: > (merging the gc-dev and svc-dev threads) > > Hi Erik, > > On 2013-05-29 10:44, Erik Helin wrote: > >Hi all, > > > >this want sent to hotspot-gc-dev at openjdk.java.net, sending to > >serviceability-dev at openjdk.java.net as well since the change is about > >memory pools. > > > >This change adds two memory pools for metaspace, one for Metaspace and > >one for compressed klass space. The memory pool for compressed klass > >space will only have valus that differ from 0 or -1 (undefined) if > >compressed klass pointers are used. > > > >Question: Should I use an empty pool when compressed klass pointers are > >*not* used or should I just not expose the pool? > > > >This change also adds a manager for the pools: Metaspace Manager. > > > >I have also added a test that checks that the metaspace manager is > >present and that the two pools are present. The test also verifies that > >the compressed klass space pool act correct according to the > >UseCompressedKlass flag. > > > >The last time I added metaspace memory pools, it triggered some > >unforeseen bugs: > >- Two asserts in jmm_GetMemoryUsage that asserted that a memory pool was > > either of heap type or had an undefined init/max size. > >- The jdk/test/java/lang/management/MemoryMXBean/MemoryTest.java failed > >- The service lock was taken out of order with the metaspace locks > > > >These bugs have all been fixed: > >- The asserts have been removed since they are no longer true > >- The test has been updated but is not part of this change since it is a > > JDK change > >- This change does not make use of functions requiring the metaspace > > lock. I had to remove some verification code in free_chunks_total to > > ensure this. > > > >Webrev: > >http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/ > > Overall I think your fix looks good but I disagree with your choice > of exposing an EmptyCompressedKlassSpacePool for the case of > -UseCompressedClassPointers. > > Given the dynamic nature of the MemoryManagerMXBeans and > MemoryPoolMXBeans it seems more true to the intentions of the API to > ony expose a MemoryPool if it contains interesting information. > > Generally I don't think users of the management APIs can (or should) > depend on the exact set of MemoryManagerMXBeans and > MemoryPoolMXBeans > available in a given VM instance. > > /Mikael > > > > >Testing: > >- One new jtreg test > >- JPRT > >- All the tests that failed in nighly testing last time now pass > > > >Thanks, > >Erik > > From john.cuthbertson at oracle.com Tue Jun 11 19:30:45 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 11 Jun 2013 12:30:45 -0700 Subject: RFR(S): Backport of changes for 8015237 to hs24 Message-ID: <51B77AE5.8040708@oracle.com> Hi Everyone, The hs25 changes for 8015237 did not apply cleanly to hs24 (no surprise there) so can I have a couple of volunteers look over these backported changes? The backported changes including Chris' comments about changing the name of StringTable::_par_claimed_index can be found at: http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ The original hsx25 webrev (minus Chris' comments) can be found at: http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ Testing: GC testsuite with G1 and CMS with ParallelGCThreads=4; jprt. Thanks, JohnC From stefan.karlsson at oracle.com Tue Jun 11 19:52:56 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 11 Jun 2013 21:52:56 +0200 Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <51B77AE5.8040708@oracle.com> References: <51B77AE5.8040708@oracle.com> Message-ID: <51B78018.9090207@oracle.com> On 2013-06-11 21:30, John Cuthbertson wrote: > Hi Everyone, > > The hs25 changes for 8015237 did not apply cleanly to hs24 (no > surprise there) so can I have a couple of volunteers look over these > backported changes? > > The backported changes including Chris' comments about changing the > name of StringTable::_par_claimed_index can be found at: > http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ The backport looks good. thanks, StefanK > > > The original hsx25 webrev (minus Chris' comments) can be found at: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ > > Testing: GC testsuite with G1 and CMS with ParallelGCThreads=4; jprt. > > Thanks, > > JohnC From tao.mao at oracle.com Tue Jun 11 20:04:44 2013 From: tao.mao at oracle.com (Tao Mao) Date: Tue, 11 Jun 2013 13:04:44 -0700 Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <51B77AE5.8040708@oracle.com> References: <51B77AE5.8040708@oracle.com> Message-ID: <51B782DC.4080004@oracle.com> Just a thing to point out: the common practice for dev's updating copyright date is not to update, or, at least, keep it consistent across files. Thanks. Tao On 6/11/13 12:30 PM, John Cuthbertson wrote: > Hi Everyone, > > The hs25 changes for 8015237 did not apply cleanly to hs24 (no > surprise there) so can I have a couple of volunteers look over these > backported changes? > > The backported changes including Chris' comments about changing the > name of StringTable::_par_claimed_index can be found at: > http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ > > The original hsx25 webrev (minus Chris' comments) can be found at: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ > > Testing: GC testsuite with G1 and CMS with ParallelGCThreads=4; jprt. > > Thanks, > > JohnC From john.cuthbertson at oracle.com Tue Jun 11 20:13:38 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 11 Jun 2013 13:13:38 -0700 Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <51B782DC.4080004@oracle.com> References: <51B77AE5.8040708@oracle.com> <51B782DC.4080004@oracle.com> Message-ID: <51B784F2.1030406@oracle.com> Hi Tao, I believe the policy is it is up to the individual developer. In my case I sometimes use an updated copyright year to indicate (to me) that I'm happy with that file. What happened here is that one of the files in hs25 already had an updated copyright year and hence was not included in the generated patch. Thanks for pointing out the inconsistency in this webrev. I'll fix it before pushing. JohnC On 6/11/2013 1:04 PM, Tao Mao wrote: > Just a thing to point out: the common practice for dev's updating > copyright date is not to update, or, at least, keep it consistent > across files. > > Thanks. > Tao > > On 6/11/13 12:30 PM, John Cuthbertson wrote: >> Hi Everyone, >> >> The hs25 changes for 8015237 did not apply cleanly to hs24 (no >> surprise there) so can I have a couple of volunteers look over these >> backported changes? >> >> The backported changes including Chris' comments about changing the >> name of StringTable::_par_claimed_index can be found at: >> http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ >> >> The original hsx25 webrev (minus Chris' comments) can be found at: >> http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ >> >> Testing: GC testsuite with G1 and CMS with ParallelGCThreads=4; jprt. >> >> Thanks, >> >> JohnC From john.cuthbertson at oracle.com Tue Jun 11 20:13:27 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 11 Jun 2013 13:13:27 -0700 (PDT) Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <51B78018.9090207@oracle.com> References: <51B77AE5.8040708@oracle.com> <51B78018.9090207@oracle.com> Message-ID: <51B784E7.4090401@oracle.com> Hi Stefan, Thanks for sanity checking me. Cheers, JohnC On 6/11/2013 12:52 PM, Stefan Karlsson wrote: > On 2013-06-11 21:30, John Cuthbertson wrote: >> Hi Everyone, >> >> The hs25 changes for 8015237 did not apply cleanly to hs24 (no >> surprise there) so can I have a couple of volunteers look over these >> backported changes? >> >> The backported changes including Chris' comments about changing the >> name of StringTable::_par_claimed_index can be found at: >> http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ > > The backport looks good. > > thanks, > StefanK >> >> >> The original hsx25 webrev (minus Chris' comments) can be found at: >> http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ >> >> Testing: GC testsuite with G1 and CMS with ParallelGCThreads=4; jprt. >> >> Thanks, >> >> JohnC > From thomas.schatzl at oracle.com Tue Jun 11 20:45:39 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 11 Jun 2013 22:45:39 +0200 Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <51B77AE5.8040708@oracle.com> References: <51B77AE5.8040708@oracle.com> Message-ID: <1370983539.5102.1.camel@cirrus> On Tue, 2013-06-11 at 12:30 -0700, John Cuthbertson wrote: > Hi Everyone, > > The hs25 changes for 8015237 did not apply cleanly to hs24 (no surprise > there) so can I have a couple of volunteers look over these backported > changes? > > The backported changes including Chris' comments about changing the name > of StringTable::_par_claimed_index can be found at: > http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ > > The original hsx25 webrev (minus Chris' comments) can be found at: > http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ Looks good. Just asking, could you also apply Chris' changes when pushing the hsx25 change? I think it has not been pushed yet, and it would be nice to keep them in sync as much as possible. Thanks, Thomas From john.cuthbertson at oracle.com Tue Jun 11 21:28:34 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Tue, 11 Jun 2013 14:28:34 -0700 Subject: RFR(S): Backport of changes for 8015237 to hs24 In-Reply-To: <1370983539.5102.1.camel@cirrus> References: <51B77AE5.8040708@oracle.com> <1370983539.5102.1.camel@cirrus> Message-ID: <51B79682.7010809@oracle.com> Hi Thomas, Apologies for the confusion. Chris' suggestion has already been included in the (unpushed) hs25 changes. I just didn't update the hs25 webrev. JohnC On 6/11/2013 1:45 PM, Thomas Schatzl wrote: > On Tue, 2013-06-11 at 12:30 -0700, John Cuthbertson wrote: >> Hi Everyone, >> >> The hs25 changes for 8015237 did not apply cleanly to hs24 (no surprise >> there) so can I have a couple of volunteers look over these backported >> changes? >> >> The backported changes including Chris' comments about changing the name >> of StringTable::_par_claimed_index can be found at: >> http://cr.openjdk.java.net/~johnc/8015237-hsx24-backport/webrev.0/ >> >> The original hsx25 webrev (minus Chris' comments) can be found at: >> http://cr.openjdk.java.net/~johnc/8015237/webrev.1/ > Looks good. > > Just asking, could you also apply Chris' changes when pushing the hsx25 > change? I think it has not been pushed yet, and it would be nice to keep > them in sync as much as possible. > > Thanks, > Thomas > From mikael.gerdin at oracle.com Tue Jun 11 21:46:46 2013 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Tue, 11 Jun 2013 23:46:46 +0200 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51B21A0B.2030800@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> <51B054F2.7000807@oracle.com> <51B0A1BA.9070308@oracle.com> <51B1FC2C.9030105@oracle.com> <51B21A0B.2030800@oracle.com> Message-ID: <51B79AC6.7040107@oracle.com> Jon On 06/07/2013 07:36 PM, Jon Masamitsu wrote: > > On 6/7/2013 8:28 AM, Mikael Gerdin wrote: >> Jon, >> >> >> On 2013-06-06 16:50, Jon Masamitsu wrote: >>> Mikael, >>> >>> Thanks. I'd be interested in seeing the instrumentation you >>> add. Might be worth adding as an enhancement in a later >>> changeset. >> >> I did a 1hr KS run today with and without block splitting, here's what >> I came up with (in an entirely non-scientific way) >> >> http://cr.openjdk.java.net/~mgerdin/8009561/splitting.txt >> http://cr.openjdk.java.net/~mgerdin/8009561/splitting.png > > Good graphs. > > The behavior is what we expect (I think). With splitting we are able to > do more > small allocations from the dictionary (where we split a larger block to > get a smaller > block) and get fewer larger blocks allocated (some have been split). > > >> >> We hit the HWM 4 times with splitting and 5 times without splitting. > > Because we don't have to expand (get new chunks as often, which is good) I > would surmise. > >> On the other hand: splitting did lead us with more metaspace memory >> committed in the end. > > One explanation would be that allocations of larger block need to come > out of newly committed space instead of the dictionary (where the large > blocks have been broken up). > > Is there a policy that we could use that says > > "break up a larger block for a smaller block allocation only if ..." > > You fill in the blank? ...only if the larger block is less than 4 times larger than the allocation? 2 times? 8 times? I could try some more KS runs but I'm unsure if the figures I come up with are actually relevant. > > >> >> I put up the very simple instrumentation at: >> http://cr.openjdk.java.net/~mgerdin/8009561/instrument/webrev >> >> I also changed the allocation_from_dictionary_limit to 4k to force us >> to make more freelist allocations. > > Does it really make sense to have any allocation_from_dictionary_limit? > I know it was initially added because allocation from a freelist takes > longer > but to have a static limit like that just seems to put that space forever > beyond reach. I thought you had added the limit. I sort of feel that 64k is a bit much but the code would definitely be simpler if there was none. We already take the hit of acquiring a Mutex for each Metaspace allocation so maybe the dictionary lookup isn't that expensive? > > Thanks for the numbers. You're welcome. /Mikael > > Jon > >> >> /Mikael >> >>> >>> Jon >>> >>> >>> On 6/6/13 2:22 AM, Mikael Gerdin wrote: >>>> Jon, >>>> >>>> On 2013-06-06 04:41, Jon Masamitsu wrote: >>>>> >>>>> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >>>>>> Hi, >>>>>> >>>>>> Can I have some reviews of this small fix to the Metaspace memory >>>>>> allocation path. >>>>>> >>>>>> Problem: >>>>>> When a Metaspace allocation request cannot be satisfied by the >>>>>> current >>>>>> chunk the chunk is retired and a new chunk is requested. This causes >>>>>> whatever is left in the chunk to be effectively leaked. >>>>>> >>>>>> Suggested fix: >>>>>> Put the remaining memory in each chunk on the Metablock freelist >>>>>> so it >>>>>> can be used to satisfy future allocations. >>>>>> >>>>>> Possible addition: >>>>>> When allocating from the block free list, use >>>>>> FreeBlockDictionary::atLeast instead of >>>>>> FreeBlockDictionary::exactly and split the Metablock if >>>>>> it's large enough. >>>>>> >>>>>> One might argue that this increases the fragmentation of the >>>>>> memory on >>>>>> the block free list but I think that we primarily want to use the >>>>>> block free list for small allocations and allocate from chunks for >>>>>> large allocations. >>>>>> >>>>>> Webrev: >>>>>> Only fix: >>>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ >>>>> >>>>> The "Only fix" looks good. Did you test with >>>>> metaspace_slow_verify=true? >>>>> >>>>>> >>>>>> Incremental webrev for splitting blocks: >>>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ >>>>> >>>>> Change looks good. >>>>> >>>>> Did you do any long running tests with the block splitting? Such as >>>>> 24hours with kitchensink? Something that would reuse Metablocks >>>>> so that we can see if we are fragmenting instead of reusing? >>>>> >>>> >>>> I did some runs earlier but I don't have any data from them. >>>> I can try to get an instrumented build together and run KS over the >>>> weekend. >>>> >>>> /Mikael >>>> >>>>> Jon >>>>> >>>>> >>>>>> >>>>>> Bug links: >>>>>> https://jbs.oracle.com/bugs/browse/JDK-8009561 >>>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >>>>>> >>>>>> Thanks >>>>>> /Mikael >>>>> >>>> >>> > From jon.masamitsu at oracle.com Tue Jun 11 22:22:03 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 11 Jun 2013 15:22:03 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370893538.2674.9.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> <1370893538.2674.9.camel@cirrus> Message-ID: <51B7A30B.2060202@oracle.com> On 6/10/13 12:45 PM, Thomas Schatzl wrote: > Hi, > > On Fri, 2013-06-07 at 11:17 -0700, Jon Masamitsu wrote: >> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >>> Hi all, >>> >>> some notes about the second patch, the modified eden boundary >>> sampling; in general I think the idea is good (Jon suggested that it >>> should become default), but imho there are some issues with the >>> implementation. >>> >>> On Tue, 2013-05-28 at 17:24 -0700, Hiroshi Yamauchi wrote: >>>> Hi, >>>> >>>> Here's a suggested patch in a separate patch: >>>> >>>> http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/ >>>> >>>> that attempts to improve on this issue by implementing an alternative >>>> way of dividing up the eden into chunks for an increased parallelism >>>> (or better load balancing between the GC threads) for the young gen >>>> scan portion of the remark phase (and the now-parallelized initial >>>> mark phase.) It uses a CAS-based mechanism that samples the object >>>> boundaries in the eden space on the slow allocation code paths (eg. at >>>> the TLAB refill and large object allocation times) at all times. >>> - this patch is for DefNew/CMS only it seems. Is this correct? >>> >>> - in the Hotspot code base explicit != NULL checks are done for whatever >>> reason. To keep the same style it would be nice to update the checks >>> whether to do the sampling ("if (CMSEdenChunksRecordAlways && >>> _next_gen)") accordingly. >>> >>> (the next point is not an issue of the patch) >>> - the check whether there is a _next_gen is odd - I do not think that >>> DefNew works as a generation without an older generation, but I'm not >>> sure. >> You're correct that DefNew needs to have a _next_gen. >> >>> Looking at other similar code checking for _next_gen != NULL, the >>> response is typically to update _next_gen and then asserting that >>> _next_gen is != NULL. E.g. >>> >>> if (_next_gen == NULL) { >>> GenCollectedHeap* gch = GenCollectedHeap::heap(); >>> _next_gen = gch->next_gen(this); >>> assert(_next_gen != NULL, >>> "This must be the youngest gen, and not the only gen"); >>> } >>> >>> Jon? >> Yes, except in a very few situations, _next_gen should be set. >> Exceptions I would expect is during initialization. At the point >> Thomas indicates and assertion would be sufficient. >> >> assert(_next_gen != NULL, "..."); > Thanks. > >> >>> - I think CMSCollector::_eden_chunk_sampling_active should be volatile >>> to avoid any clever compilers skipping the read in sample_eden_chunk(). >>> (Although I don't think it's possible here). See also below for avoiding >>> that variable completely. >> Could you explain you thinking for making it volatile. As you have, I >> didn't think >> it was necessary and thought ,"If not necessary, don't do it". > It's not necessary I think, just as a reminder to the compiler to not > optimize that load away. I think the Atomic instruction contain the > necessary instructions though. Ok. I might be a bit of a distraction to include volatile when not necessary. >>> (btw, throughout this code there is a lot of checking whether some >>> NEW_C_HEAP_ARRAY returned NULL or not, and does elaborate recovery - >>> however by default NEW_C_HEAP_ARRAY fails with an OOM anyway... - but >>> that is certainly out of scope) >> Thomas, could you file a CR for fixing that? Thanks. > CR 8016276: CMS concurrentMarkSweepGeneration contains lots of > unnecessary allocation failure handling Thanks. Jon > >>> Same for checking whether _survivor_chunk_array is NULL - at the moment >>> sometimes _survivor_chunk_array is checked against NULL to avoid >>> entering code (in print_eden_and_survivor_chunk_arrays()). >>> >>> The condition that could be used is the one that triggers allocation of >>> _survivor_chunk_array: (CMSParallelRemarkEnabled && >>> CMSParallelSurvivorRemarkEnabled) in >>> concurrentMarkSweepGeneration.cpp:728. >> Personally, I like a check like if (_eden_chunk_array != NULL) >> better. I think it is more future proof. For whatever reason we don't >> try to dereference _eden_chunk_array if it is NULL. An assertion >> checking that >> >> (CMSParallelRemarkEnabled && >> CMSParallelSurvivorRemarkEnabled) >> >> is true would we be appropriate. > Also good. > > Thomas > > From jon.masamitsu at oracle.com Tue Jun 11 22:30:29 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 11 Jun 2013 15:30:29 -0700 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1370893794.2674.12.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> Message-ID: <51B7A505.2000300@oracle.com> On 6/10/13 12:49 PM, Thomas Schatzl wrote: > Hi, > > On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: >> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >>> - the code to get the lock on _eden_chunk_sampling_active seems to be >>> needlessly complicated. >>> >>> Please consider using a dedicated Monitor in conjunction with >>> try_lock(). I.e. >> Thomas, >> >> Any concerns about the "sneak" code in try_lock()? That would only >> matter if the VM thread were involved in the eden sampling along with >> other GC threads, yes? > I do not think so in this case. Only the VM thread can sneak, and only > during safepoint. The sampling is done only for eden space and eventually > from-space, which are never allocated to during gc (only by java threads > outside of stw pauses), aren't they? > > The code could assert that in the sampling too. Where would the assert go that asserts sampling is being done? Or do you mean at the use of try_lock() assert that we're not the VM thread? Jon > > Even then, this sampling is only active for defnew at the moment. The > only thread that is active in a defnew collection is the vm thread where > sneaking cannot happen. > > Thomas > > From jon.masamitsu at oracle.com Tue Jun 11 22:51:20 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 11 Jun 2013 15:51:20 -0700 Subject: Request for review: JDK-8009561 NPG: Metaspace fragmentation when retiring a Metachunk In-Reply-To: <51B79AC6.7040107@oracle.com> References: <51AF4576.3080203@oracle.com> <51AFF6BD.60501@oracle.com> <51B054F2.7000807@oracle.com> <51B0A1BA.9070308@oracle.com> <51B1FC2C.9030105@oracle.com> <51B21A0B.2030800@oracle.com> <51B79AC6.7040107@oracle.com> Message-ID: <51B7A9E8.2030909@oracle.com> On 6/11/13 2:46 PM, Mikael Gerdin wrote: > Jon > > On 06/07/2013 07:36 PM, Jon Masamitsu wrote: >> >> On 6/7/2013 8:28 AM, Mikael Gerdin wrote: >>> Jon, >>> >>> >>> On 2013-06-06 16:50, Jon Masamitsu wrote: >>>> Mikael, >>>> >>>> Thanks. I'd be interested in seeing the instrumentation you >>>> add. Might be worth adding as an enhancement in a later >>>> changeset. >>> >>> I did a 1hr KS run today with and without block splitting, here's what >>> I came up with (in an entirely non-scientific way) >>> >>> http://cr.openjdk.java.net/~mgerdin/8009561/splitting.txt >>> http://cr.openjdk.java.net/~mgerdin/8009561/splitting.png >> >> Good graphs. >> >> The behavior is what we expect (I think). With splitting we are able to >> do more >> small allocations from the dictionary (where we split a larger block to >> get a smaller >> block) and get fewer larger blocks allocated (some have been split). >> >> >>> >>> We hit the HWM 4 times with splitting and 5 times without splitting. >> >> Because we don't have to expand (get new chunks as often, which is >> good) I >> would surmise. >> >>> On the other hand: splitting did lead us with more metaspace memory >>> committed in the end. >> >> One explanation would be that allocations of larger block need to come >> out of newly committed space instead of the dictionary (where the large >> blocks have been broken up). >> >> Is there a policy that we could use that says >> >> "break up a larger block for a smaller block allocation only if ..." >> >> You fill in the blank? > > ...only if the larger block is less than 4 times larger than the > allocation? 2 times? 8 times? > > I could try some more KS runs but I'm unsure if the figures I come up > with are actually relevant. I also don't know if more KS runs would be relevant. Can you ask the dictionary how many blocks there are of the size you're going to split? If we only split if there are more than 4 blocks of that size, that would moderate the splitting a bit. Jon > > >> >> >>> >>> I put up the very simple instrumentation at: >>> http://cr.openjdk.java.net/~mgerdin/8009561/instrument/webrev >>> >>> I also changed the allocation_from_dictionary_limit to 4k to force us >>> to make more freelist allocations. >> >> Does it really make sense to have any allocation_from_dictionary_limit? >> I know it was initially added because allocation from a freelist takes >> longer >> but to have a static limit like that just seems to put that space >> forever >> beyond reach. > > I thought you had added the limit. I sort of feel that 64k is a bit > much but the code would definitely be simpler if there was none. > We already take the hit of acquiring a Mutex for each Metaspace > allocation so maybe the dictionary lookup isn't that expensive? > >> >> Thanks for the numbers. > > You're welcome. > > /Mikael > >> >> Jon >> >>> >>> /Mikael >>> >>>> >>>> Jon >>>> >>>> >>>> On 6/6/13 2:22 AM, Mikael Gerdin wrote: >>>>> Jon, >>>>> >>>>> On 2013-06-06 04:41, Jon Masamitsu wrote: >>>>>> >>>>>> On 6/5/2013 7:04 AM, Mikael Gerdin wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Can I have some reviews of this small fix to the Metaspace memory >>>>>>> allocation path. >>>>>>> >>>>>>> Problem: >>>>>>> When a Metaspace allocation request cannot be satisfied by the >>>>>>> current >>>>>>> chunk the chunk is retired and a new chunk is requested. This >>>>>>> causes >>>>>>> whatever is left in the chunk to be effectively leaked. >>>>>>> >>>>>>> Suggested fix: >>>>>>> Put the remaining memory in each chunk on the Metablock freelist >>>>>>> so it >>>>>>> can be used to satisfy future allocations. >>>>>>> >>>>>>> Possible addition: >>>>>>> When allocating from the block free list, use >>>>>>> FreeBlockDictionary::atLeast instead of >>>>>>> FreeBlockDictionary::exactly and split the Metablock if >>>>>>> it's large enough. >>>>>>> >>>>>>> One might argue that this increases the fragmentation of the >>>>>>> memory on >>>>>>> the block free list but I think that we primarily want to use the >>>>>>> block free list for small allocations and allocate from chunks for >>>>>>> large allocations. >>>>>>> >>>>>>> Webrev: >>>>>>> Only fix: >>>>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0/ >>>>>> >>>>>> The "Only fix" looks good. Did you test with >>>>>> metaspace_slow_verify=true? >>>>>> >>>>>>> >>>>>>> Incremental webrev for splitting blocks: >>>>>>> http://cr.openjdk.java.net/~mgerdin/8009561/webrev.0%2b/ >>>>>> >>>>>> Change looks good. >>>>>> >>>>>> Did you do any long running tests with the block splitting? Such as >>>>>> 24hours with kitchensink? Something that would reuse Metablocks >>>>>> so that we can see if we are fragmenting instead of reusing? >>>>>> >>>>> >>>>> I did some runs earlier but I don't have any data from them. >>>>> I can try to get an instrumented build together and run KS over the >>>>> weekend. >>>>> >>>>> /Mikael >>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>>> >>>>>>> Bug links: >>>>>>> https://jbs.oracle.com/bugs/browse/JDK-8009561 >>>>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8009561 >>>>>>> >>>>>>> Thanks >>>>>>> /Mikael >>>>>> >>>>> >>>> >> > From thomas.schatzl at oracle.com Tue Jun 11 23:07:29 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 12 Jun 2013 01:07:29 +0200 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <51B7A505.2000300@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> Message-ID: <1370992049.5102.40.camel@cirrus> Hi, On Tue, 2013-06-11 at 15:30 -0700, Jon Masamitsu wrote: > On 6/10/13 12:49 PM, Thomas Schatzl wrote: > > Hi, > > > > On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: > >> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: > >>> - the code to get the lock on _eden_chunk_sampling_active seems to be > >>> needlessly complicated. > >>> > >>> Please consider using a dedicated Monitor in conjunction with > >>> try_lock(). I.e. > >> Thomas, > >> > >> Any concerns about the "sneak" code in try_lock()? That would only > >> matter if the VM thread were involved in the eden sampling along with > >> other GC threads, yes? > > I do not think so in this case. Only the VM thread can sneak, and only > > during safepoint. The sampling is done only for eden space and eventually > > from-space, which are never allocated to during gc (only by java threads > > outside of stw pauses), aren't they? > > > > The code could assert that in the sampling too. > Where would the assert go that asserts sampling is > being done? > > Or do you mean at the use of try_lock() assert that > we're not the VM thread? > Sorry, I was unclear. In the sampling code itself: sampling should not be performed during a safepoint as far as I can see. So e.g. add to the sampling method an assert(!SafepointSynchronize::is_at_safepoint(), "..."); There is also the option to add a parameter to try_lock() that prohibits sneaking. E.g. bool try_lock(bool allow_vm_thread_to_sneak = true) { ... bool can_sneak = allow_vm_thread_to_sneak && ... ... } Then in the code if (...->try_lock(false)) { ... ...->unlock(); } All other uses would be unaffected. I would simply prefer using the existing synchronization primitives instead of rolling our own just for that particular use. They make the code also much more readable. Another reason I am not worried about vm thread sneaking in this case is that during gc only the vm thread runs (and try_lock() does not seem to be a synchronization point for the safepoint), which cannot sneak itself (or is at least benign). Thomas From bengt.rutisson at oracle.com Wed Jun 12 07:19:03 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Wed, 12 Jun 2013 00:19:03 -0700 (PDT) Subject: RFR (S): 8015683: object_count_after_gc should have the same timestamp for all events In-Reply-To: <20130609072848.GA1963@ehelin-thinkpad> References: <20130609072848.GA1963@ehelin-thinkpad> Message-ID: <51B820E7.2000801@oracle.com> Looks good. Bengt On 6/9/13 9:28 AM, Erik Helin wrote: > Hi all, > > this small change ensures that all vm/gc/detailed/object_count_after_gc trace > events that are sent during the same GC will have identical timestamps. > > Please note that this change is based upon another change that is out > for review - "8015972: Refactor the sending of the object count after GC event". > > Webrev: > http://cr.openjdk.java.net/~ehelin/8015683/webrev.00/ > > Testing: > JPRT > > Thanks, > Erik From stefan.karlsson at oracle.com Wed Jun 12 07:43:41 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 12 Jun 2013 09:43:41 +0200 Subject: RFR (S): 8015683: object_count_after_gc should have the same timestamp for all events In-Reply-To: <20130609072848.GA1963@ehelin-thinkpad> References: <20130609072848.GA1963@ehelin-thinkpad> Message-ID: <51B826AD.5070105@oracle.com> On 06/09/2013 09:28 AM, Erik Helin wrote: > Hi all, > > this small change ensures that all vm/gc/detailed/object_count_after_gc trace > events that are sent during the same GC will have identical timestamps. > > Please note that this change is based upon another change that is out > for review - "8015972: Refactor the sending of the object count after GC event". > > Webrev: > http://cr.openjdk.java.net/~ehelin/8015683/webrev.00/ I think it would be better to convert this event into an Instant event, then you wouldn't have to store two identical timestamps in the event. thanks, StefanK > > Testing: > JPRT > > Thanks, > Erik From erik.helin at oracle.com Wed Jun 12 09:27:40 2013 From: erik.helin at oracle.com (Erik Helin) Date: Wed, 12 Jun 2013 11:27:40 +0200 Subject: RFR (S): 8015683: object_count_after_gc should have the same timestamp for all events In-Reply-To: <51B826AD.5070105@oracle.com> References: <20130609072848.GA1963@ehelin-thinkpad> <51B826AD.5070105@oracle.com> Message-ID: <20130612092740.GC2015@ehelin-thinkpad> On 2013-06-12, Stefan Karlsson wrote: > On 06/09/2013 09:28 AM, Erik Helin wrote: > >Hi all, > > > >this small change ensures that all vm/gc/detailed/object_count_after_gc trace > >events that are sent during the same GC will have identical timestamps. > > > >Please note that this change is based upon another change that is out > >for review - "8015972: Refactor the sending of the object count after GC event". > > > >Webrev: > >http://cr.openjdk.java.net/~ehelin/8015683/webrev.00/ > > I think it would be better to convert this event into an Instant > event, then you wouldn't have to store two identical timestamps in > the event. Agree, I've updated the webrev to take this into account. Please see new webrev at: http://cr.openjdk.java.net/~ehelin/8015683/webrev.01/ Thanks, Erik > thanks, > StefanK > > > > >Testing: > >JPRT > > > >Thanks, > >Erik > From stefan.karlsson at oracle.com Wed Jun 12 11:17:38 2013 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 12 Jun 2013 13:17:38 +0200 Subject: RFR (S): 8015683: object_count_after_gc should have the same timestamp for all events In-Reply-To: <20130612092740.GC2015@ehelin-thinkpad> References: <20130609072848.GA1963@ehelin-thinkpad> <51B826AD.5070105@oracle.com> <20130612092740.GC2015@ehelin-thinkpad> Message-ID: <51B858D2.30305@oracle.com> On 06/12/2013 11:27 AM, Erik Helin wrote: > On 2013-06-12, Stefan Karlsson wrote: >> On 06/09/2013 09:28 AM, Erik Helin wrote: >>> Hi all, >>> >>> this small change ensures that all vm/gc/detailed/object_count_after_gc trace >>> events that are sent during the same GC will have identical timestamps. >>> >>> Please note that this change is based upon another change that is out >>> for review - "8015972: Refactor the sending of the object count after GC event". >>> >>> Webrev: >>> http://cr.openjdk.java.net/~ehelin/8015683/webrev.00/ >> I think it would be better to convert this event into an Instant >> event, then you wouldn't have to store two identical timestamps in >> the event. > Agree, I've updated the webrev to take this into account. Please see new > webrev at: > > http://cr.openjdk.java.net/~ehelin/8015683/webrev.01/ Looks good. StefanK > > Thanks, > Erik > >> thanks, >> StefanK >> >>> Testing: >>> JPRT >>> >>> Thanks, >>> Erik From jon.masamitsu at oracle.com Wed Jun 12 14:11:35 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 12 Jun 2013 07:11:35 -0700 Subject: RFR: 8013590: NPG: Add a memory pool MXBean for Metaspace In-Reply-To: <20130611152759.GD3566@ehelin-thinkpad> References: <20130529084430.GD1956@ehelin-thinkpad> <51A89305.3070005@oracle.com> <20130611152759.GD3566@ehelin-thinkpad> Message-ID: <51B88197.4050700@oracle.com> On 6/11/2013 8:27 AM, Erik Helin wrote: > Hi Mikael and Jon, > > thanks for reviewing! > > I have updated the webrev. The changes from webrev.00 are: > - Only exposing the compressed class space memory pool if compressed > classes are used. > - Renamed the name of the pool to "Compressed Class Space" (was > "Compressed Klass Space", note the 'K'). There was a discussion about the way the flag UseCompressedKlassPointers was spelled. I had advocated for keeping the "K" in Klass but after the other responses I thought I had lost. In particular I had thought the flag had existed before the NPG changes and had been exposed to users and was wrong. Why did you decide to keep the flag (and internal variable names) with the "K" but change the name of the space to "Compressed Class Space"? > - Renamed the variable _cks_pool to _compressed_class_pool. > - Using max_uintx instead of _undefined_size. > - Updated the test and the rest of the code to take these new change > into account. > > Please see new webrev at: > http://cr.openjdk.java.net/~ehelin/8013590/webrev.01/ Rest looks fine. Jon > > On 2013-06-04, Jon Masamitsu wrote: >> http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/test/gc/metaspace/TestMetaspaceMemoryPool.java.html >> >> The functions assertEquals(), assertTrue(), and assertDefined() seem >> useful. Is there >> a more global place where they can be put so other can use them? > I agree that assertTrue and assertEquals should be move to a separate > file, but I think that assertDefined is mostly usable for tests dealing > with MXMemoryBeans (since defined is not always the same as "differ from > -1"). > > I would like to do these changes as a separate change. Jon, what do you > about that? > > Thanks, > Erik > > On 2013-05-31, Mikael Gerdin wrote: >> (merging the gc-dev and svc-dev threads) >> >> Hi Erik, >> >> On 2013-05-29 10:44, Erik Helin wrote: >>> Hi all, >>> >>> this want sent to hotspot-gc-dev at openjdk.java.net, sending to >>> serviceability-dev at openjdk.java.net as well since the change is about >>> memory pools. >>> >>> This change adds two memory pools for metaspace, one for Metaspace and >>> one for compressed klass space. The memory pool for compressed klass >>> space will only have valus that differ from 0 or -1 (undefined) if >>> compressed klass pointers are used. >>> >>> Question: Should I use an empty pool when compressed klass pointers are >>> *not* used or should I just not expose the pool? >>> >>> This change also adds a manager for the pools: Metaspace Manager. >>> >>> I have also added a test that checks that the metaspace manager is >>> present and that the two pools are present. The test also verifies that >>> the compressed klass space pool act correct according to the >>> UseCompressedKlass flag. >>> >>> The last time I added metaspace memory pools, it triggered some >>> unforeseen bugs: >>> - Two asserts in jmm_GetMemoryUsage that asserted that a memory pool was >>> either of heap type or had an undefined init/max size. >>> - The jdk/test/java/lang/management/MemoryMXBean/MemoryTest.java failed >>> - The service lock was taken out of order with the metaspace locks >>> >>> These bugs have all been fixed: >>> - The asserts have been removed since they are no longer true >>> - The test has been updated but is not part of this change since it is a >>> JDK change >>> - This change does not make use of functions requiring the metaspace >>> lock. I had to remove some verification code in free_chunks_total to >>> ensure this. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/ >> Overall I think your fix looks good but I disagree with your choice >> of exposing an EmptyCompressedKlassSpacePool for the case of >> -UseCompressedClassPointers. >> >> Given the dynamic nature of the MemoryManagerMXBeans and >> MemoryPoolMXBeans it seems more true to the intentions of the API to >> ony expose a MemoryPool if it contains interesting information. >> >> Generally I don't think users of the management APIs can (or should) >> depend on the exact set of MemoryManagerMXBeans and >> MemoryPoolMXBeans >> available in a given VM instance. >> >> /Mikael >> >>> Testing: >>> - One new jtreg test >>> - JPRT >>> - All the tests that failed in nighly testing last time now pass >>> >>> Thanks, >>> Erik >>> From jon.masamitsu at oracle.com Wed Jun 12 14:25:28 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 12 Jun 2013 07:25:28 -0700 (PDT) Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1370992049.5102.40.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> Message-ID: <51B884D8.4000103@oracle.com> On 6/11/2013 4:07 PM, Thomas Schatzl wrote: > Hi, > > On Tue, 2013-06-11 at 15:30 -0700, Jon Masamitsu wrote: >> On 6/10/13 12:49 PM, Thomas Schatzl wrote: >>> Hi, >>> >>> On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: >>>> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >>>>> - the code to get the lock on _eden_chunk_sampling_active seems to be >>>>> needlessly complicated. >>>>> >>>>> Please consider using a dedicated Monitor in conjunction with >>>>> try_lock(). I.e. >>>> Thomas, >>>> >>>> Any concerns about the "sneak" code in try_lock()? That would only >>>> matter if the VM thread were involved in the eden sampling along with >>>> other GC threads, yes? >>> I do not think so in this case. Only the VM thread can sneak, and only >>> during safepoint. The sampling is done only for eden space and eventually >>> from-space, which are never allocated to during gc (only by java threads >>> outside of stw pauses), aren't they? >>> >>> The code could assert that in the sampling too. >> Where would the assert go that asserts sampling is >> being done? >> >> Or do you mean at the use of try_lock() assert that >> we're not the VM thread? >> > Sorry, I was unclear. In the sampling code itself: sampling should not > be performed during a safepoint as far as I can see. So e.g. add to the > sampling method an > > assert(!SafepointSynchronize::is_at_safepoint(), "..."); OK. That would be sufficient. > > There is also the option to add a parameter to try_lock() that prohibits > sneaking. Or a try_lock_no_sneak() that is called from try_lock() and could be used here. I hate that name. And I wouldn't touch mutex.cpp myself (would worry about unintended performance consequences). Jon > > E.g. > > bool try_lock(bool allow_vm_thread_to_sneak = true) { > ... > bool can_sneak = allow_vm_thread_to_sneak && ... > ... > } > > Then in the code > > if (...->try_lock(false)) { > ... > ...->unlock(); > } > > All other uses would be unaffected. > > I would simply prefer using the existing synchronization primitives > instead of rolling our own just for that particular use. > They make the code also much more readable. > > Another reason I am not worried about vm thread sneaking in this case is > that during gc only the vm thread runs (and try_lock() does not seem to > be a synchronization point for the safepoint), which cannot sneak itself > (or is at least benign). > > Thomas > > From thomas.schatzl at oracle.com Thu Jun 13 07:46:23 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Jun 2013 09:46:23 +0200 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <51B884D8.4000103@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> Message-ID: <1371109583.2679.23.camel@cirrus> Hi, On Wed, 2013-06-12 at 07:25 -0700, Jon Masamitsu wrote: > On 6/11/2013 4:07 PM, Thomas Schatzl wrote: > > Hi, > > > > On Tue, 2013-06-11 at 15:30 -0700, Jon Masamitsu wrote: > >> On 6/10/13 12:49 PM, Thomas Schatzl wrote: > >>> Hi, > >>> > >>> On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: > >>>> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >[...] > >> Or do you mean at the use of try_lock() assert that > >> we're not the VM thread? > >> > > Sorry, I was unclear. In the sampling code itself: sampling should not > > be performed during a safepoint as far as I can see. So e.g. add to the > > sampling method an > > > > assert(!SafepointSynchronize::is_at_safepoint(), "..."); > > OK. That would be sufficient. > > > > > There is also the option to add a parameter to try_lock() that prohibits > > sneaking. > > Or a try_lock_no_sneak() that is called from try_lock() and could be used > here. I hate that name. And I wouldn't touch mutex.cpp myself (would worry > about unintended performance consequences). I would almost prefer the try_lock_no_sneak() approach you suggested - the assert would need some additional explanation while imo it would be immediately clear with the try_lock_no_sneak(). (But doing both is fine too). (Actually I'd expect that a try_lock() does not do unexpected things like sneaking in the first place. Try_lock() is a common name for that functionality in libraries with some expected behavior. That likely does not include something like sneaking the lock... i.e. you should be required to call try_lock_sneak() in the places where you want that, but, oh well...) As for that change to add try_lock_no_sneak() in mutex code, this is a minor structural change only. Performance wise I do not expect a big difference: the CAS isn't too fast, the surrounding loop does not give a real performance guarantee in the first place, and the try_lock_no_sneak() call would be the only call within mutex.cpp, i.e. most likely the compiler would inline it anyway into the try_lock() method. Thanks, Thomas From erik.helin at oracle.com Thu Jun 13 08:07:08 2013 From: erik.helin at oracle.com (Erik Helin) Date: Thu, 13 Jun 2013 10:07:08 +0200 Subject: RFR: 8013590: NPG: Add a memory pool MXBean for Metaspace In-Reply-To: <51B88197.4050700@oracle.com> References: <20130529084430.GD1956@ehelin-thinkpad> <51A89305.3070005@oracle.com> <20130611152759.GD3566@ehelin-thinkpad> <51B88197.4050700@oracle.com> Message-ID: <20130613080708.GA8870@ehelin-thinkpad> On 2013-06-12, Jon Masamitsu wrote: > > On 6/11/2013 8:27 AM, Erik Helin wrote: > >Hi Mikael and Jon, > > > >thanks for reviewing! > > > >I have updated the webrev. The changes from webrev.00 are: > >- Only exposing the compressed class space memory pool if compressed > > classes are used. > >- Renamed the name of the pool to "Compressed Class Space" (was > > "Compressed Klass Space", note the 'K'). > > There was a discussion about the way the flag > UseCompressedKlassPointers was spelled. > I had advocated for keeping the "K" in Klass but after the other > responses I thought > I had lost. In particular I had thought the flag had existed before > the NPG changes and > had been exposed to users and was wrong. > > Why did you decide to keep the flag (and internal variable names) > with the "K" but > change the name of the space to "Compressed Class Space"? I'm sorry, I should be more explicit when I write my emails :( Yes, we've decided to use 'C' instead of 'K' for all external names regarding compressed klass space (and compressed klass pointers). I'm going to send out a separate patch with these changes, but I should have mentioned that in this email to avoid any confusion. If you want, I can continue to keep 'K' for this change and then change all of the naming in a separate patch (that would be more consistent). Would you prefer to do it this way? Thanks, Erik > >- Renamed the variable _cks_pool to _compressed_class_pool. > >- Using max_uintx instead of _undefined_size. > >- Updated the test and the rest of the code to take these new change > > into account. > > > >Please see new webrev at: > >http://cr.openjdk.java.net/~ehelin/8013590/webrev.01/ > > Rest looks fine. > > Jon > > > > >On 2013-06-04, Jon Masamitsu wrote: > >>http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/test/gc/metaspace/TestMetaspaceMemoryPool.java.html > >> > >>The functions assertEquals(), assertTrue(), and assertDefined() seem > >>useful. Is there > >>a more global place where they can be put so other can use them? > >I agree that assertTrue and assertEquals should be move to a separate > >file, but I think that assertDefined is mostly usable for tests dealing > >with MXMemoryBeans (since defined is not always the same as "differ from > >-1"). > > > >I would like to do these changes as a separate change. Jon, what do you > >about that? > > > >Thanks, > >Erik > > > >On 2013-05-31, Mikael Gerdin wrote: > >>(merging the gc-dev and svc-dev threads) > >> > >>Hi Erik, > >> > >>On 2013-05-29 10:44, Erik Helin wrote: > >>>Hi all, > >>> > >>>this want sent to hotspot-gc-dev at openjdk.java.net, sending to > >>>serviceability-dev at openjdk.java.net as well since the change is about > >>>memory pools. > >>> > >>>This change adds two memory pools for metaspace, one for Metaspace and > >>>one for compressed klass space. The memory pool for compressed klass > >>>space will only have valus that differ from 0 or -1 (undefined) if > >>>compressed klass pointers are used. > >>> > >>>Question: Should I use an empty pool when compressed klass pointers are > >>>*not* used or should I just not expose the pool? > >>> > >>>This change also adds a manager for the pools: Metaspace Manager. > >>> > >>>I have also added a test that checks that the metaspace manager is > >>>present and that the two pools are present. The test also verifies that > >>>the compressed klass space pool act correct according to the > >>>UseCompressedKlass flag. > >>> > >>>The last time I added metaspace memory pools, it triggered some > >>>unforeseen bugs: > >>>- Two asserts in jmm_GetMemoryUsage that asserted that a memory pool was > >>> either of heap type or had an undefined init/max size. > >>>- The jdk/test/java/lang/management/MemoryMXBean/MemoryTest.java failed > >>>- The service lock was taken out of order with the metaspace locks > >>> > >>>These bugs have all been fixed: > >>>- The asserts have been removed since they are no longer true > >>>- The test has been updated but is not part of this change since it is a > >>> JDK change > >>>- This change does not make use of functions requiring the metaspace > >>> lock. I had to remove some verification code in free_chunks_total to > >>> ensure this. > >>> > >>>Webrev: > >>>http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/ > >>Overall I think your fix looks good but I disagree with your choice > >>of exposing an EmptyCompressedKlassSpacePool for the case of > >>-UseCompressedClassPointers. > >> > >>Given the dynamic nature of the MemoryManagerMXBeans and > >>MemoryPoolMXBeans it seems more true to the intentions of the API to > >>ony expose a MemoryPool if it contains interesting information. > >> > >>Generally I don't think users of the management APIs can (or should) > >>depend on the exact set of MemoryManagerMXBeans and > >>MemoryPoolMXBeans > >>available in a given VM instance. > >> > >>/Mikael > >> > >>>Testing: > >>>- One new jtreg test > >>>- JPRT > >>>- All the tests that failed in nighly testing last time now pass > >>> > >>>Thanks, > >>>Erik > >>> > From bengt.rutisson at oracle.com Thu Jun 13 09:53:43 2013 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Thu, 13 Jun 2013 11:53:43 +0200 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B5EFBA.9090202@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> <51B1F87D.7060105@oracle.com> <51B208F8.6030200@oracle.com> <228A79C9-6839-4EED-B8DB-2F61E669E047@oracle.com> <51B5EFBA.9090202@oracle.com> Message-ID: <51B996A7.8090702@oracle.com> Looks good. Bengt On 6/10/13 5:24 PM, Vladimir Kempik wrote: > Hello. > > Thanks for comment. > > Here is updated webrev - > http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.02/ > > Vladimir. > > On 08.06.2013 15:07, Bengt Rutisson wrote: >> Vladimir, >> >> I would prefer to remove the tty->cr() call you changed and instead >> use gclog_or_tty->print_cr() in the gclog_or_tty->print() statement >> before. >> >> Other than that it looks good. >> >> Bengt >> >> 7 jun 2013 kl. 18:23 skrev Jesper Wilhelmsson >> : >> >>> Vladimir Kempik skrev 7/6/13 5:13 PM: >>>> Thanks for comments >>>> >>>> Here is updated webrev - >>>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.01/ >>> Thanks for fixing this! >>> Looks good, ship it (once the gc repo is open for pushes again). >>> /Jesper >>> >>> >>>> On 07.06.2013 18:37, Jesper Wilhelmsson wrote: >>>>> Vladimir, >>>>> >>>>> Looks good, >>>>> >>>>> When I fixed the same thing yesterday I noticed that the method >>>>> above, >>>>> PSAdaptiveSizePolicy::compute_survivor_space_size_and_threshold, >>>>> writes its >>>>> cr() to tty instead of gclog_or_tty. Do you mind fixing that in >>>>> the same >>>>> change since it is slightly related? >>>>> >>>>> And I don't know how to stress this enough, you are not the first >>>>> one who did >>>>> it, never ever work on a bug without assigning it to yourself, or >>>>> talk to the >>>>> assigned engineer first. Having two engineers working on the same >>>>> bug is not >>>>> efficient use of our time. >>>> Sorry about that. >>>> >>>> Vladimir. >>>>> /Jesper >>>>> >>>>> >>>>> >>>>> Vladimir Kempik skrev 7/6/13 3:46 PM: >>>>>> Hi all, >>>>>> >>>>>> Could I have a couple of reviews for this change? >>>>>> >>>>>> http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.00/ >>>>>> >>>>>> With next option -XX:+PrintAdapriveSizePolicy, java log looks >>>>>> like this: >>>>>> >>>>>> AdaptiveSizePolicy::update_averages: survived: 479272 promoted: >>>>>> 251648 overflow: >>>>>> falseAdaptiveSizeStart: 4,136 collection: 16 >>>>>> avg_survived_padded_avg: 2223544,750000 avg_promoted_padded_avg: >>>>>> 1300947,750000 avg_pretenured_padded_avg: 286597,000000 >>>>>> tenuring_thresh: 1 >>>>>> target_size: 1835008 >>>>>> >>>>>> and there is no space between overflow: false and AdaptiveSizeStart: >>>>>> >>>>>> The patch adds line break after overflow: false. >>>>>> >>>>>> Thanks, >>>>>> Vladimir > From thomas.schatzl at oracle.com Thu Jun 13 10:23:14 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Jun 2013 12:23:14 +0200 Subject: Request for review: 8015903: Format issue with -XX:+PrintAdaptiveSizePolicy on JDK8 In-Reply-To: <51B5EFBA.9090202@oracle.com> References: <51B1E42C.2080801@oracle.com> <51B1F026.4030507@oracle.com> <51B1F87D.7060105@oracle.com> <51B208F8.6030200@oracle.com> <228A79C9-6839-4EED-B8DB-2F61E669E047@oracle.com> <51B5EFBA.9090202@oracle.com> Message-ID: <1371118994.2679.42.camel@cirrus> Hi, On Mon, 2013-06-10 at 19:24 +0400, Vladimir Kempik wrote: > Hello. > > Thanks for comment. > > Here is updated webrev - > http://cr.openjdk.java.net/~mcherkas/vladimir/8015903/webrev.02/ > > Vladimir. looks good. Thomas From david.holmes at oracle.com Thu Jun 13 10:33:19 2013 From: david.holmes at oracle.com (David Holmes) Date: Thu, 13 Jun 2013 20:33:19 +1000 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1371109583.2679.23.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> <1371109583.2679.23.camel@cirrus> Message-ID: <51B99FEF.7050302@oracle.com> Thomas, lock-sneaking is something very specific to the VMThread to avoid deadlocks at safepoints. I really don't think we want to see it promoted into the API for mutexes/monitors. I hadn't been paying this thread too much attention but now I'm starting to think I need to. David On 13/06/2013 5:46 PM, Thomas Schatzl wrote: > Hi, > > On Wed, 2013-06-12 at 07:25 -0700, Jon Masamitsu wrote: >> On 6/11/2013 4:07 PM, Thomas Schatzl wrote: >>> Hi, >>> >>> On Tue, 2013-06-11 at 15:30 -0700, Jon Masamitsu wrote: >>>> On 6/10/13 12:49 PM, Thomas Schatzl wrote: >>>>> Hi, >>>>> >>>>> On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: >>>>>> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >> [...] >>>> Or do you mean at the use of try_lock() assert that >>>> we're not the VM thread? >>>> >>> Sorry, I was unclear. In the sampling code itself: sampling should not >>> be performed during a safepoint as far as I can see. So e.g. add to the >>> sampling method an >>> >>> assert(!SafepointSynchronize::is_at_safepoint(), "..."); >> >> OK. That would be sufficient. >> >>> >>> There is also the option to add a parameter to try_lock() that prohibits >>> sneaking. >> >> Or a try_lock_no_sneak() that is called from try_lock() and could be used >> here. I hate that name. And I wouldn't touch mutex.cpp myself (would worry >> about unintended performance consequences). > > I would almost prefer the try_lock_no_sneak() approach you suggested - > the assert would need some additional explanation while imo it would be > immediately clear with the try_lock_no_sneak(). (But doing both is fine > too). > > (Actually I'd expect that a try_lock() does not do unexpected things > like sneaking in the first place. Try_lock() is a common name for that > functionality in libraries with some expected behavior. That likely does > not include something like sneaking the lock... i.e. you should be > required to call try_lock_sneak() in the places where you want that, > but, oh well...) > > As for that change to add try_lock_no_sneak() in mutex code, this is a > minor structural change only. Performance wise I do not expect a big > difference: the CAS isn't too fast, the surrounding loop does not give a > real performance guarantee in the first place, and the > try_lock_no_sneak() call would be the only call within mutex.cpp, i.e. > most likely the compiler would inline it anyway into the try_lock() > method. > > Thanks, > Thomas > From thomas.schatzl at oracle.com Thu Jun 13 11:45:09 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Jun 2013 13:45:09 +0200 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <51B99FEF.7050302@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> <1371109583.2679.23.camel@cirrus> <51B99FEF.7050302@oracle.com> Message-ID: <1371123909.2679.91.camel@cirrus> Hi, On Thu, 2013-06-13 at 20:33 +1000, David Holmes wrote: > Thomas, > > lock-sneaking is something very specific to the VMThread to avoid > deadlocks at safepoints. I really don't think we want to see it promoted > into the API for mutexes/monitors. > > I hadn't been paying this thread too much attention but now I'm starting > to think I need to. the situation is (and I should have asked the runtime team more directly, and cc'ed the thread earlier, apologies) that in CMS there is the need to frequently record object boundaries into a buffer for performance reasons. There is a contribution that changes this sampling from being done concurrently to being done every time after a tlab refill. The problem is that this operation involves a few reads that must be atomic; so the change guards that operation with a critical section. It is also not critical that the operation is executed always, it may be skipped some times. So the change manually implements a try_lock()/unlock() idiom using a CAS and a global variable. The suggestion that tripped this thread has been to use existing VM code, in particular monitors, as they seem to have the requested functionality. There does not seem to be need for rolling own code here. The code in that critical section is very short, and straight line code; the critical section boiler plate code is larger than that... Then there has been the concern about that try_lock() may be sneaked by the vm thread; and some suggestions to put these concerns to rest. See http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp.udiff.html , in the sample_eden_chunk() method. So maybe could you confirm the statements made so far: - Monitor::try_lock()/::unlock() is equivalent to what the change does in this case. (I had a look at the unlock() code, and it's a lot more complicated than the equivalent code in the change. It boils down to the same behavior however?) - that new sampling code is supposed to be only every executed while the mutator is running. Sneaking can only occur during safepoints; so it's use here would be "safe" in that there will be no sneaking at all. There is no case when the suggested assert(! SafepointSynchronize::is_at_safepoint()) would fire? Other than that some other ideas to avoid execution of the sneaking code were thrown into the room. The assert should be fine, but it would be better to avoid the possibility of sneaking directly. The latest suggestion involved adding a no-sneaking-variant of try_lock() ("try_lock_no_sneak()") that is called by the original one to keep exact semantics (except for the additional call). Nobody intends to change anything in that code without your input :) everyone is aware that this is tricky code with possible unintended side effects. The relevant thread starts here, most of it having been cc'ed to runtime-dev: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-June/007510.html Thanks for looking at this issue, Thomas From yamauchi at google.com Thu Jun 13 18:54:01 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 11:54:01 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370893538.2674.9.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> <1370893538.2674.9.camel@cirrus> Message-ID: > > > - this patch is for DefNew/CMS only it seems. Is this correct? > Yes. > > > > > > - in the Hotspot code base explicit != NULL checks are done for > whatever > > > reason. To keep the same style it would be nice to update the checks > > > whether to do the sampling ("if (CMSEdenChunksRecordAlways && > > > _next_gen)") accordingly. > I'll fix these. > > > > > (the next point is not an issue of the patch) > > > - the check whether there is a _next_gen is odd - I do not think that > > > DefNew works as a generation without an older generation, but I'm not > > > sure. > > You're correct that DefNew needs to have a _next_gen. > > > > > Looking at other similar code checking for _next_gen != NULL, the > > > response is typically to update _next_gen and then asserting that > > > _next_gen is != NULL. E.g. > > > > > > if (_next_gen == NULL) { > > > GenCollectedHeap* gch = GenCollectedHeap::heap(); > > > _next_gen = gch->next_gen(this); > > > assert(_next_gen != NULL, > > > "This must be the youngest gen, and not the only gen"); > > > } > > > > > > Jon? > > > > Yes, except in a very few situations, _next_gen should be set. > > Exceptions I would expect is during initialization. At the point > > Thomas indicates and assertion would be sufficient. > > > > assert(_next_gen != NULL, "..."); > > Thanks. I assume this discussion is out of scope of this patch. Let me know otherwise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 13 19:00:05 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 12:00:05 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B7A30B.2060202@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> <1370893538.2674.9.camel@cirrus> <51B7A30B.2060202@oracle.com> Message-ID: > - I think CMSCollector::_eden_chunk_**sampling_active should be volatile >>>> to avoid any clever compilers skipping the read in sample_eden_chunk(). >>>> (Although I don't think it's possible here). See also below for avoiding >>>> that variable completely. >>>> >>> Could you explain you thinking for making it volatile. As you have, I >>> didn't think >>> it was necessary and thought ,"If not necessary, don't do it". >>> >> It's not necessary I think, just as a reminder to the compiler to not >> optimize that load away. I think the Atomic instruction contain the >> necessary instructions though. >> > > Ok. I might be a bit of a distraction to include volatile when > not necessary. I haven't thought about this volatile thing too much. But it seems unnecessary, even if we keep this _eden_chunk_**sampling_active, as opposed to using try_lock() which I haven't thought about yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 13 19:10:46 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 12:10:46 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370604194.2671.55.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> Message-ID: > - in the CMSCollector::CMSCollector() initialization list, the > initialization of _eden_chunk_sampling_active breaks the "-- ditto --" > list, which looks strange. (Actually I think the comment that a variable > may eventually be set in the ctor again later seems questionable anyway) > Right. It could be moved below _survivor_chunk_index so it won't interrupt the "-- ditto --" list. Or, it could be moved above the list along with its declaration in the .hpp file. But let's hold off until the try_lock discussion develops. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 13 19:27:31 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 12:27:31 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B223C6.7080805@oracle.com> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> Message-ID: > - instead of checking whether _eden_chunk_array is NULL to detect >> whether you can sample I would prefer if the same predicate as in the >> initialization (gch->supports_inline_contig_**alloc()) were used. >> (I think the change just copied what was done in the previous version of >> concurrentMarkSweepGeneration.**cpp:4515 has been done) >> >> Maybe create a new predicate for that, used everywhere. >> >> (btw, throughout this code there is a lot of checking whether some >> NEW_C_HEAP_ARRAY returned NULL or not, and does elaborate recovery - >> however by default NEW_C_HEAP_ARRAY fails with an OOM anyway... - but >> that is certainly out of scope) >> > > Thomas, could you file a CR for fixing that? Thanks. > > Thomas, by a new predicate, do you mean checking if (gch->supports_inline_contig_**alloc()) as opposed to if (_eden_chunk_array != NULL) assuming that the NEW_C_HEAP_ARRAY succeeded? You're right that the code checks the null-ness of _eden_chunk_array as CMSCollector::sample_eden() did so. I see that CR 8016276 was filed. Given this, but without knowing the details of the CR, it seems OK for this patch to keep the null check against _eden_chunk_array as it is? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 13 19:39:13 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 12:39:13 -0700 Subject: CMS parallel initial mark In-Reply-To: <1370893538.2674.9.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> <1370893538.2674.9.camel@cirrus> Message-ID: > > > Same for checking whether _survivor_chunk_array is NULL - at the moment > > > sometimes _survivor_chunk_array is checked against NULL to avoid > > > entering code (in print_eden_and_survivor_chunk_arrays()). > > > > > > The condition that could be used is the one that triggers allocation of > > > _survivor_chunk_array: (CMSParallelRemarkEnabled && > > > CMSParallelSurvivorRemarkEnabled) in > > > concurrentMarkSweepGeneration.cpp:728. > > > > Personally, I like a check like if (_eden_chunk_array != NULL) > > better. I think it is more future proof. For whatever reason we don't > > try to dereference _eden_chunk_array if it is NULL. An assertion > > checking that > > > > (CMSParallelRemarkEnabled && > > CMSParallelSurvivorRemarkEnabled) > > > > is true would we be appropriate. > > Also good. > Do you mean adding an assert in CMSCollector::print_eden_and_survivor_chunk_arrays() like the following: if (_survivor_chunk_array != NULL) { assert(CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled, "Error"); ... } ? That can be easily inserted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamauchi at google.com Thu Jun 13 19:45:11 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 12:45:11 -0700 Subject: CMS parallel initial mark In-Reply-To: <51B6273C.1040301@oracle.com> References: <1370534525.2590.43.camel@cirrus> <51B0C180.4020106@oracle.com> <1370546740.2445.31.camel@cirrus> <51B1F4A0.10902@oracle.com> <51B23E9F.8050605@oracle.com> <51B6273C.1040301@oracle.com> Message-ID: On Mon, Jun 10, 2013 at 12:21 PM, Jon Masamitsu wrote: > > On 6/7/13 3:20 PM, Hiroshi Yamauchi wrote: > > Here's an update version of the first patch based on what's been discussed > so far: > > http://cr.openjdk.java.net/~hiroshi/webrevs/cmsparinitmark/webrev.02/ > > I'll catch up with the comments on the other patch later. > > Changes look good. > > Thanks. > Thanks. I'll focus on the second patch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Thu Jun 13 19:55:28 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 13 Jun 2013 12:55:28 -0700 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1371109583.2679.23.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> <1371109583.2679.23.camel@cirrus> Message-ID: <51BA23B0.20503@oracle.com> On 6/13/13 12:46 AM, Thomas Schatzl wrote: > Hi, > > On Wed, 2013-06-12 at 07:25 -0700, Jon Masamitsu wrote: >> On 6/11/2013 4:07 PM, Thomas Schatzl wrote: >>> Hi, >>> >>> On Tue, 2013-06-11 at 15:30 -0700, Jon Masamitsu wrote: >>>> On 6/10/13 12:49 PM, Thomas Schatzl wrote: >>>>> Hi, >>>>> >>>>> On Fri, 2013-06-07 at 11:28 -0700, Jon Masamitsu wrote: >>>>>> On 6/7/2013 4:23 AM, Thomas Schatzl wrote: >> [...] >>>> Or do you mean at the use of try_lock() assert that >>>> we're not the VM thread? >>>> >>> Sorry, I was unclear. In the sampling code itself: sampling should not >>> be performed during a safepoint as far as I can see. So e.g. add to the >>> sampling method an >>> >>> assert(!SafepointSynchronize::is_at_safepoint(), "..."); >> OK. That would be sufficient. >> >>> There is also the option to add a parameter to try_lock() that prohibits >>> sneaking. >> Or a try_lock_no_sneak() that is called from try_lock() and could be used >> here. I hate that name. And I wouldn't touch mutex.cpp myself (would worry >> about unintended performance consequences). > I would almost prefer the try_lock_no_sneak() approach you suggested - > the assert would need some additional explanation while imo it would be > immediately clear with the try_lock_no_sneak(). (But doing both is fine > too). > > (Actually I'd expect that a try_lock() does not do unexpected things > like sneaking in the first place. Try_lock() is a common name for that > functionality in libraries with some expected behavior. That likely does > not include something like sneaking the lock... i.e. you should be > required to call try_lock_sneak() in the places where you want that, > but, oh well...) > > As for that change to add try_lock_no_sneak() in mutex code, this is a > minor structural change only. Performance wise I do not expect a big > difference: the CAS isn't too fast, the surrounding loop does not give a > real performance guarantee in the first place, and the > try_lock_no_sneak() call would be the only call within mutex.cpp, i.e. > most likely the compiler would inline it anyway into the try_lock() > method. I was thinking more of the ripple effect of a larger body for try_lock() and in-lining. Different on every platform, hard to diagnose. Jon > > Thanks, > Thomas > From jon.masamitsu at oracle.com Thu Jun 13 19:58:00 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 13 Jun 2013 12:58:00 -0700 Subject: RFR: 8013590: NPG: Add a memory pool MXBean for Metaspace In-Reply-To: <20130613080708.GA8870@ehelin-thinkpad> References: <20130529084430.GD1956@ehelin-thinkpad> <51A89305.3070005@oracle.com> <20130611152759.GD3566@ehelin-thinkpad> <51B88197.4050700@oracle.com> <20130613080708.GA8870@ehelin-thinkpad> Message-ID: <51BA2448.4040304@oracle.com> On 6/13/13 1:07 AM, Erik Helin wrote: > On 2013-06-12, Jon Masamitsu wrote: >> On 6/11/2013 8:27 AM, Erik Helin wrote: >>> Hi Mikael and Jon, >>> >>> thanks for reviewing! >>> >>> I have updated the webrev. The changes from webrev.00 are: >>> - Only exposing the compressed class space memory pool if compressed >>> classes are used. >>> - Renamed the name of the pool to "Compressed Class Space" (was >>> "Compressed Klass Space", note the 'K'). >> There was a discussion about the way the flag >> UseCompressedKlassPointers was spelled. >> I had advocated for keeping the "K" in Klass but after the other >> responses I thought >> I had lost. In particular I had thought the flag had existed before >> the NPG changes and >> had been exposed to users and was wrong. >> >> Why did you decide to keep the flag (and internal variable names) >> with the "K" but >> change the name of the space to "Compressed Class Space"? > I'm sorry, I should be more explicit when I write my emails :( > > Yes, we've decided to use 'C' instead of 'K' for all external names > regarding compressed klass space (and compressed klass pointers). I'm > going to send out a separate patch with these changes, but I should have > mentioned that in this email to avoid any confusion. > > If you want, I can continue to keep 'K' for this change and then change > all of the naming in a separate patch (that would be more consistent). > Would you prefer to do it this way? I don't have a preference since very quickly it will all become consistent. Jon > > Thanks, > Erik > >>> - Renamed the variable _cks_pool to _compressed_class_pool. >>> - Using max_uintx instead of _undefined_size. >>> - Updated the test and the rest of the code to take these new change >>> into account. >>> >>> Please see new webrev at: >>> http://cr.openjdk.java.net/~ehelin/8013590/webrev.01/ >> Rest looks fine. >> >> Jon >> >>> On 2013-06-04, Jon Masamitsu wrote: >>>> http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/test/gc/metaspace/TestMetaspaceMemoryPool.java.html >>>> >>>> The functions assertEquals(), assertTrue(), and assertDefined() seem >>>> useful. Is there >>>> a more global place where they can be put so other can use them? >>> I agree that assertTrue and assertEquals should be move to a separate >>> file, but I think that assertDefined is mostly usable for tests dealing >>> with MXMemoryBeans (since defined is not always the same as "differ from >>> -1"). >>> >>> I would like to do these changes as a separate change. Jon, what do you >>> about that? >>> >>> Thanks, >>> Erik >>> >>> On 2013-05-31, Mikael Gerdin wrote: >>>> (merging the gc-dev and svc-dev threads) >>>> >>>> Hi Erik, >>>> >>>> On 2013-05-29 10:44, Erik Helin wrote: >>>>> Hi all, >>>>> >>>>> this want sent to hotspot-gc-dev at openjdk.java.net, sending to >>>>> serviceability-dev at openjdk.java.net as well since the change is about >>>>> memory pools. >>>>> >>>>> This change adds two memory pools for metaspace, one for Metaspace and >>>>> one for compressed klass space. The memory pool for compressed klass >>>>> space will only have valus that differ from 0 or -1 (undefined) if >>>>> compressed klass pointers are used. >>>>> >>>>> Question: Should I use an empty pool when compressed klass pointers are >>>>> *not* used or should I just not expose the pool? >>>>> >>>>> This change also adds a manager for the pools: Metaspace Manager. >>>>> >>>>> I have also added a test that checks that the metaspace manager is >>>>> present and that the two pools are present. The test also verifies that >>>>> the compressed klass space pool act correct according to the >>>>> UseCompressedKlass flag. >>>>> >>>>> The last time I added metaspace memory pools, it triggered some >>>>> unforeseen bugs: >>>>> - Two asserts in jmm_GetMemoryUsage that asserted that a memory pool was >>>>> either of heap type or had an undefined init/max size. >>>>> - The jdk/test/java/lang/management/MemoryMXBean/MemoryTest.java failed >>>>> - The service lock was taken out of order with the metaspace locks >>>>> >>>>> These bugs have all been fixed: >>>>> - The asserts have been removed since they are no longer true >>>>> - The test has been updated but is not part of this change since it is a >>>>> JDK change >>>>> - This change does not make use of functions requiring the metaspace >>>>> lock. I had to remove some verification code in free_chunks_total to >>>>> ensure this. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~ehelin/8013590/webrev.00/ >>>> Overall I think your fix looks good but I disagree with your choice >>>> of exposing an EmptyCompressedKlassSpacePool for the case of >>>> -UseCompressedClassPointers. >>>> >>>> Given the dynamic nature of the MemoryManagerMXBeans and >>>> MemoryPoolMXBeans it seems more true to the intentions of the API to >>>> ony expose a MemoryPool if it contains interesting information. >>>> >>>> Generally I don't think users of the management APIs can (or should) >>>> depend on the exact set of MemoryManagerMXBeans and >>>> MemoryPoolMXBeans >>>> available in a given VM instance. >>>> >>>> /Mikael >>>> >>>>> Testing: >>>>> - One new jtreg test >>>>> - JPRT >>>>> - All the tests that failed in nighly testing last time now pass >>>>> >>>>> Thanks, >>>>> Erik >>>>> From yamauchi at google.com Thu Jun 13 20:46:50 2013 From: yamauchi at google.com (Hiroshi Yamauchi) Date: Thu, 13 Jun 2013 13:46:50 -0700 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1371123909.2679.91.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> <1371109583.2679.23.camel@cirrus> <51B99FEF.7050302@oracle.com> <1371123909.2679.91.camel@cirrus> Message-ID: On Thu, Jun 13, 2013 at 4:45 AM, Thomas Schatzl wrote: > Hi, > > On Thu, 2013-06-13 at 20:33 +1000, David Holmes wrote: > > Thomas, > > > > lock-sneaking is something very specific to the VMThread to avoid > > deadlocks at safepoints. I really don't think we want to see it promoted > > into the API for mutexes/monitors. > > > > I hadn't been paying this thread too much attention but now I'm starting > > to think I need to. > > the situation is (and I should have asked the runtime team more > directly, and cc'ed the thread earlier, apologies) that in CMS there is > the need to frequently record object boundaries into a buffer for > performance reasons. > > There is a contribution that changes this sampling from being done > concurrently to being done every time after a tlab refill. The problem > is that this operation involves a few reads that must be atomic; so the > change guards that operation with a critical section. It is also not > critical that the operation is executed always, it may be skipped some > times. > > So the change manually implements a try_lock()/unlock() idiom using a > CAS and a global variable. > > The suggestion that tripped this thread has been to use existing VM > code, in particular monitors, as they seem to have the requested > functionality. There does not seem to be need for rolling own code here. > > The code in that critical section is very short, and straight line code; > the critical section boiler plate code is larger than that... > > Then there has been the concern about that try_lock() may be sneaked by > the vm thread; and some suggestions to put these concerns to rest. > > See > > http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp.udiff.html, > in the sample_eden_chunk() method. > > So maybe could you confirm the statements made so far: > - Monitor::try_lock()/::unlock() is equivalent to what the change does > in this case. > > (I had a look at the unlock() code, and it's a lot more complicated than > the equivalent code in the change. It boils down to the same behavior > however?) > > - that new sampling code is supposed to be only every executed while the > mutator is running. Sneaking can only occur during safepoints; so it's > use here would be "safe" in that there will be no sneaking at all. There > is no case when the suggested assert(! > SafepointSynchronize::is_at_safepoint()) would fire? > > Other than that some other ideas to avoid execution of the sneaking code > were thrown into the room. The assert should be fine, but it would be > better to avoid the possibility of sneaking directly. The latest > suggestion involved adding a no-sneaking-variant of try_lock() > ("try_lock_no_sneak()") that is called by the original one to keep exact > semantics (except for the additional call). > Nobody intends to change anything in that code without your input :) > everyone is aware that this is tricky code with possible unintended side > effects. > > The relevant thread starts here, most of it having been cc'ed to > runtime-dev: > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-June/007510.html > > Thanks for looking at this issue, > Thomas > > > Thomas, thanks for summarizing the discussion so far. I agree that if try_lock()/unlock() does what the manual/custom synchronization does (and as efficiently), it should suffice. No need to roll one's own. I also agree that adding the assert(!SafepointSynchronize::is_at_safepoint()) (or assert(!Thread->current()->is_VM_thread())) may be good enough and if sneaking can be turned off for a case like this, it'd be safer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Thu Jun 13 21:31:43 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 13 Jun 2013 14:31:43 -0700 Subject: CMS parallel initial mark In-Reply-To: References: <1370604194.2671.55.camel@cirrus> <51B223C6.7080805@oracle.com> <1370893538.2674.9.camel@cirrus> Message-ID: <51BA3A3F.10704@oracle.com> On 6/13/13 11:54 AM, Hiroshi Yamauchi wrote: > > > > - this patch is for DefNew/CMS only it seems. Is this correct? > > > Yes. > > > > > > > - in the Hotspot code base explicit != NULL checks are done > for whatever > > > reason. To keep the same style it would be nice to update the > checks > > > whether to do the sampling ("if (CMSEdenChunksRecordAlways && > > > _next_gen)") accordingly. > > > I'll fix these. > > > > > > > (the next point is not an issue of the patch) > > > - the check whether there is a _next_gen is odd - I do not > think that > > > DefNew works as a generation without an older generation, but > I'm not > > > sure. > > You're correct that DefNew needs to have a _next_gen. > > > > > Looking at other similar code checking for _next_gen != NULL, the > > > response is typically to update _next_gen and then asserting that > > > _next_gen is != NULL. E.g. > > > > > > if (_next_gen == NULL) { > > > GenCollectedHeap* gch = GenCollectedHeap::heap(); > > > _next_gen = gch->next_gen(this); > > > assert(_next_gen != NULL, > > > "This must be the youngest gen, and not the only > gen"); > > > } > > > > > > Jon? > > > > Yes, except in a very few situations, _next_gen should be set. > > Exceptions I would expect is during initialization. At the point > > Thomas indicates and assertion would be sufficient. > > > > assert(_next_gen != NULL, "..."); > > Thanks. > > > I assume this discussion is out of scope of this patch. Let me know > otherwise. > > Yes, not your code so fixing should not be part of your patch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.cuthbertson at oracle.com Thu Jun 13 22:30:59 2013 From: john.cuthbertson at oracle.com (John Cuthbertson) Date: Thu, 13 Jun 2013 15:30:59 -0700 Subject: RFR(M/L): 7145569: G1: optimize nmethods scanning Message-ID: <51BA4823.7000509@oracle.com> Hi Everyone, Can I have a few volunteers look over the changes for this CR - ideally I would like at least one person from the compiler team and one from the GC team. Summary: For some workloads with G1, the scanning of the Code cache cache can take quite a long time. In fact there have been cases where the scanning of the code cache is the dominator of the pause and the other worker threads sit an spin in the termination code while the worker that claimed the code cache scanning task chugs along. Part of the reason is that the list of nmethods with scavengable oops is a single root task. Another part, specifically affecting G1, is that the presence of an nmethod in the list depends on the definition of is_scavengable. For the other collectors, when the oops in an nmethod are promoted to the old generation that nmethod ceases to be scavengable and is pruned from the scavengable list. For G1, because we can collect "old" data during a mixed evacuation pause we can't prune the scavengable nmethod list. The changes are my attempt at a solution outlined by Igor Veresov. I have added a list of nmethods to each HeapRegion. That list is populated by nmethods that contain roots that point into that heap region. The lists are populated when an nmethod is created (or in the case of C1 when it is patched). During an evacuation pause we skip scanning the entire code cache and instead scan the oops in each nmethod attached to a region when we scan that regions RSet. Near the end of the pause we have migrate the nmethods attached to regions in the collection to regions in to-space. During an initial mark pause, we walk the nmethods attached to all heap regions (other than those in the collection set - they're marked when their nmethod lists are scanned). During a full GC we empty the nmethod list attached to each region and then rebuild it - similar to what happens to the RSets. To verify the integrity of these nmethod lists I've extended the heap verification. When scanning the code cache we check that an nmethod is attached to heap regions that it points into. When verifying the heap regions we ensure that the nmethods attached to a region contain at least one pointer into that region. This additional verification can add some time so I recently put it under control of a diagnostic flag (testing was performed with it enabled). As part of these changes I moved some code around (specifically the verification code) in g1CollectedHeap.[ch]pp and heapRegion.[ch]pp. The purely clean up changes can be found at: http://cr.openjdk.java.net/~johnc/7145569/webrev.0.code-movement/ The webrev containing just the functionality can be found at: http://cr.openjdk.java.net/~johnc/7145569/webrev.1.optimize-nmethod-scanning/ The whole shebang can be found at: http://cr.openjdk.java.net/~johnc/7145569/webrev.all Testing: GC test suite with C1, C2, and Tiered on x64 and sparc (Xmixed and Xcomp) - with and without verification (including the extra verification); a few jprt runs Kitchensink (4 hour runs) with C1, C2, Tiered on x64 (Xmixed and Xcomp) - with and without verification (including the extra verification). Future enhancements: * Currently I'm using a growable array for the nmethod lists but in the long term I want to move to a more suitable data structure. Perhaps a specialized version of Stack. * Currently I migrate nmethods from regions in the collection set to regions in to-space in a separate phase. Ideally this should be done when we're finished scanning a region's RSet. When we do this, migration will be performed in parallel and two threads could be pushing to a to-space regions nmethod list at the same time - we will need an "atomic" push operation. When the compilers push an nmethod, the do it while holding the CodeCache_lock so such an operation is, currently, unnecessary. Thanks, JohnC From david.holmes at oracle.com Fri Jun 14 07:17:51 2013 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Jun 2013 17:17:51 +1000 Subject: CMS parallel initial mark; vm thread sneaking in Thread::try_lock() In-Reply-To: <1371123909.2679.91.camel@cirrus> References: <1370604194.2671.55.camel@cirrus> <51B22639.7040004@oracle.com> <1370893794.2674.12.camel@cirrus> <51B7A505.2000300@oracle.com> <1370992049.5102.40.camel@cirrus> <51B884D8.4000103@oracle.com> <1371109583.2679.23.camel@cirrus> <51B99FEF.7050302@oracle.com> <1371123909.2679.91.camel@cirrus> Message-ID: <51BAC39F.1060004@oracle.com> Hi Thomas, On 13/06/2013 9:45 PM, Thomas Schatzl wrote: > On Thu, 2013-06-13 at 20:33 +1000, David Holmes wrote: >> Thomas, >> >> lock-sneaking is something very specific to the VMThread to avoid >> deadlocks at safepoints. I really don't think we want to see it promoted >> into the API for mutexes/monitors. >> >> I hadn't been paying this thread too much attention but now I'm starting >> to think I need to. > > the situation is (and I should have asked the runtime team more > directly, and cc'ed the thread earlier, apologies) that in CMS there is > the need to frequently record object boundaries into a buffer for > performance reasons. > > There is a contribution that changes this sampling from being done > concurrently to being done every time after a tlab refill. The problem > is that this operation involves a few reads that must be atomic; so the > change guards that operation with a critical section. It is also not > critical that the operation is executed always, it may be skipped some > times. > > So the change manually implements a try_lock()/unlock() idiom using a > CAS and a global variable. This is a simple and efficient approach if you are happy to skip sampling when others are doing it. A couple of things about the way you implemented this though: a) the variable should be a jint. Using a jbyte is unlikely to save space because it will need to be correctly aligned on some platforms. Conversely atomic ops on unaligned subwords, if allowed, can take a performance hit. And in the worst case attempting a CAS on an unaligned jbyte could cause a crash! Also it should be marked volatile. b) there is no need to use CAS to restore the variable to zero. But you would need to use a store_fence, as done in the sync code. > The suggestion that tripped this thread has been to use existing VM > code, in particular monitors, as they seem to have the requested > functionality. There does not seem to be need for rolling own code here. True, try_lock and unlock are functionally equivalent but may be less performant due to the added support for blocking that you don't need. > The code in that critical section is very short, and straight line code; > the critical section boiler plate code is larger than that... > > Then there has been the concern about that try_lock() may be sneaked by > the vm thread; and some suggestions to put these concerns to rest. > > See > http://cr.openjdk.java.net/~hiroshi/webrevs/edenchunks/webrev.00/src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp.udiff.html , > in the sample_eden_chunk() method. > > So maybe could you confirm the statements made so far: > - Monitor::try_lock()/::unlock() is equivalent to what the change does > in this case. > > (I had a look at the unlock() code, and it's a lot more complicated than > the equivalent code in the change. It boils down to the same behavior > however?) Yes. > - that new sampling code is supposed to be only every executed while the > mutator is running. Sneaking can only occur during safepoints; so it's > use here would be "safe" in that there will be no sneaking at all. There > is no case when the suggested assert(! > SafepointSynchronize::is_at_safepoint()) would fire? That seems reasonable. But I think you are misunderstanding what sneaking does. It isn't dangerous, you don't get multiple threads inside a critical region or anything strange like that - as long as the other threads using it will stop at safepoints of course! With monitors/mutexes a point is reached where the lock has been claimed but the new owner then blocks at a safepoint. So externally it looks like the lock is owned but the owner has not yet entered the critical region. If the VMThread needed that same lock there would be a deadlock. So the VMThread is allowed to "sneak" the lock, effectively taking ownership of it as-if the real owner blocked just before it could take ownership. The real owner can't wake up and enter the critical section until the safepoint is over and the VMThread will release the lock before it ends the safepoint. Note that any lock needed by the VMThread at a safepoint should not be used to guard critical sections in which other safepoint checks occur, else you would again get a deadlock. (And these locks are not reentrant.) Cheers, David ------ > Other than that some other ideas to avoid execution of the sneaking code > were thrown into the room. The assert should be fine, but it would be > better to avoid the possibility of sneaking directly. The latest > suggestion involved adding a no-sneaking-variant of try_lock() > ("try_lock_no_sneak()") that is called by the original one to keep exact > semantics (except for the additional call). > Nobody intends to change anything in that code without your input :) > everyone is aware that this is tricky code with possible unintended side > effects. > > The relevant thread starts here, most of it having been cc'ed to > runtime-dev: > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-June/007510.html > > Thanks for looking at this issue, > Thomas > > From thomas.schatzl at oracle.com Fri Jun 14 10:15:51 2013 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 14 Jun 2013 12:15:51 +0200 Subject: RFR(M/L): 7145569: G1: optimize nmethods scanning In-Reply-To: <51BA4823.7000509@oracle.com> References: <51BA4823.7000509@oracle.com> Message-ID: <1371204951.2690.5.camel@cirrus> Hi John, I had some initial look at the code, first comments (mostly about typos, comments) below: On Thu, 2013-06-13 at 15:30 -0700, John Cuthbertson wrote: > Hi Everyone, > > Can I have a few volunteers look over the changes for this CR - ideally > I would like at least one person from the compiler team and one from the > GC team. - the patch does not seem to apply to latest hotspot-gc repository here, getting a few failed hunks. - I cannot really comment on the completeness of compiler changes. g1_globals.hpp: - I am not a particular fan of the "If true, ..." prefix for globals descriptions, it seems redundant. But this is really imo. nmethod.cpp: - Universe::heap()->register_nmethod(this); seems to be always paired with CodeCache::add_scavenge_root_nmethod(this). Maybe it is useful to put the nmethod registration in that method. heapRegion.hpp - push_strong_code_root vs. add_strong_code_root - comment for migrate_strong_code_roots(): "... they *ppoint* into" - comment for strong_code_roots_do(): "Applied" -> "Apply" or "Applies"? - comment for verify_strong_code_roots(): "... include at *lease*..."; The method does not return the number of failures as the failures parameter is a bool. g1RemSet.hpp - worker_i should be an uint; there has been a pre-hs24 (7121618) issue that changed worker_i variables across all collectors from int to uint. Recently the use of "int" has crept in into g1 code again. If it causes trouble (because int is used in g1), leave it - there is a new CR (8016302) to fix that at once. - comment for oops_into_collection_set_do(): " .. this param will be *ingored*" heapRegion.cpp: - In HeapRegion::hr_clear(), it may be interesting to not only clear the code root list, but deallocate it. Depending on the size of this list (do you have numbers?) this may lead to excessive memory use of free regions' HeapRegions. I.e. there have been too many issues recently about G1 taking too much space by default :) In the same direction, maybe the default size of 10 could be lowered. Eg. with 60k+ region heaps, this will add 4M+ to the heap at startup. Then again, this could be done in another cr when fixing the general issue. - about use of another data structure for the per-region strong roots list: I agree, but depending on the typical size of these arrays. - at the end of the file there is this change that (forward?) declares the "template class GrowableArray;". As it's visible only in that cpp file, and afterwards there is no code, it seems superfluous. - in remove_strong_code_root() the code tries to remove multiple entries of a given nmethod. How is it possible to get multiple duplicate entries, assuming that push_strong_code_root() already does not push an nmethod if it is already in the list? g1CollectedHeap.cpp: - in the line "G1VerifyCodeRootBlobClosure blobsCl(&codeRootsCl, /*do_marking=*/ false);" I think common style is to do the comment with spaces around and no equals sign, e.g. /* do_marking */, and after the actual parameter. - line 3884, the text for the assert has wrong indentation (compared to the assert above) - in the comment in line 4990, maybe explicitly mention that this is about regions in the collection set once. E.g. "... that point into a region *in the collection set* are scanned when we scan ...". Not absolutely necessary I think. - after trying to fix mentioned patch import errors, I get warnings/errors when compiling this file. It seems that in RegisterNMethodOopClosure/UnregisterNMethodOopClosure ::do_oop_work() a decode_heap_oop_not_null() is missing. I.e. change template void do_oop_work(T* p) { T heap_oop = oopDesc::load_heap_oop(p); if (!oopDesc::is_null(heap_oop)) { HeapRegion* hr = _g1h->heap_region_containing(heap_oop); to template void do_oop_work(T* p) { T heap_oop = oopDesc::load_heap_oop(p); if (!oopDesc::is_null(heap_oop)) { oop obj = oopDesc::decode_heap_oop_not_null(heap_oop);