From timur.akhmadeev at gmail.com Thu Feb 4 13:13:20 2016 From: timur.akhmadeev at gmail.com (Timur Akhmadeev) Date: Thu, 4 Feb 2016 16:13:20 +0300 Subject: Metaspace - memory allocation Message-ID: Hi, I'm trying to figure out which parts of memory Hotspot can allocate in HugePages on Linux (appears to be heap and code cache by default, plus Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't explain. There's 1G allocation on the OS level in 4K pages that is apparently attributed to Class metadata, although Metaspace is maxed at a value much lower than 1G. Question: why? [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' /proc/18906/smaps | grep '^Size:' | head Size: 4 kB Size: 4 kB Size: 2764 kB Size: 12288 kB Size: 1036288 kB Size: 128 kB Size: 4 kB Size: 4 kB Size: 4 kB Size: 8 kB [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 VM.native_memory summary 18906: Native Memory Tracking: Total: reserved=2069703KB, committed=1032559KB - Java Heap (reserved=786432KB, committed=786432KB) (mmap: reserved=786432KB, committed=786432KB) - Class (reserved=1142808KB, committed=106520KB) (classes #13624) (malloc=2072KB #22966) (mmap: reserved=1140736KB, committed=104448KB) - Thread (reserved=56009KB, committed=56009KB) (thread #55) (stack: reserved=55512KB, committed=55512KB) (malloc=178KB #272) (arena=319KB #108) - Code (reserved=55407KB, committed=55407KB) (malloc=5487KB #7167) (mmap: reserved=49920KB, committed=49920KB) - GC (reserved=2600KB, committed=1748KB) (malloc=32KB #355) (mmap: reserved=2568KB, committed=1716KB) - Compiler (reserved=213KB, committed=213KB) (malloc=82KB #331) (arena=131KB #3) - Internal (reserved=4921KB, committed=4917KB) (malloc=4885KB #16137) (mmap: reserved=36KB, committed=32KB) - Symbol (reserved=17518KB, committed=17518KB) (malloc=14377KB #172159) (arena=3141KB #1) - Native Memory Tracking (reserved=3617KB, committed=3617KB) (malloc=149KB #2329) (tracking overhead=3468KB) - Arena Chunk (reserved=178KB, committed=178KB) (malloc=178KB) [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 VM.native_memory detail | grep -A 10 'for Class' [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class from [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, unsigned long, bool, char*, unsigned long, bool)+0xc2 [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, unsigned long, bool, char*, unsigned long)+0x1b [0x00007f7194ec8300] Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned char*)+0x40 [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, bool)+0x199 [0x00007f7194ec6766] VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned long)+0x76 [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned long, unsigned long)+0xf0 [0x00007f7194ec9713] VirtualSpaceList::get_new_chunk(unsigned long, unsigned long, unsigned long)+0xb3 ... [oracle at oel6u7 bin]$ head ../logs/gc.log Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8) Memory: 4k page, physical 3082784k(506980k free), swap 1257468k(1148276k free) CommandLine flags: -XX:InitialCodeCacheSize=50331648 -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace -- Regards Timur Akhmadeev -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Feb 4 14:41:37 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 4 Feb 2016 15:41:37 +0100 Subject: Metaspace - memory allocation In-Reply-To: References: Message-ID: Hi Timur, I believe you see the space for the compressed class data (Compressed class space), which is by default 1GB and controlled by -XX:CompressedClassSpaceSize . I think this area is semantically one subarea of the metaspace, the other being non-class data. I am unsure about the difference between MaxMetaSpaceSize and CompressedClassSpaceSize, but what I believe is that: - MaxMetaSpaceSize limits the amount of total committed (not reserved) space - CompressedClassSpaceSize sets the amount of space to be reserved (not committed) for compressed classes only. This would fit your NMT output, which says: "Class (reserved=1142808KB, committed=106520KB)" so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you specified - you would not be able to commit the whole 1GB, but hit the limit far earlier. See also: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm (the section about UseCompressedClassesPointers) Please take this with a grain of salt. I am not sure I understand this correctly and would value input from other developers. Kind Regards, Thomas On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev wrote: > Hi, > > I'm trying to figure out which parts of memory Hotspot can allocate in > HugePages on Linux (appears to be heap and code cache by default, plus > Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't > explain. > > There's 1G allocation on the OS level in 4K pages that is apparently > attributed to Class metadata, although Metaspace is maxed at a value much > lower than 1G. Question: why? > > [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' > /proc/18906/smaps | grep '^Size:' | head > Size: 4 kB > Size: 4 kB > Size: 2764 kB > Size: 12288 kB > Size: 1036288 kB > Size: 128 kB > Size: 4 kB > Size: 4 kB > Size: 4 kB > Size: 8 kB > > [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 > VM.native_memory summary > 18906: > > Native Memory Tracking: > > Total: reserved=2069703KB, committed=1032559KB > - Java Heap (reserved=786432KB, committed=786432KB) > (mmap: reserved=786432KB, committed=786432KB) > > - Class (reserved=1142808KB, committed=106520KB) > (classes #13624) > (malloc=2072KB #22966) > (mmap: reserved=1140736KB, committed=104448KB) > > - Thread (reserved=56009KB, committed=56009KB) > (thread #55) > (stack: reserved=55512KB, committed=55512KB) > (malloc=178KB #272) > (arena=319KB #108) > > - Code (reserved=55407KB, committed=55407KB) > (malloc=5487KB #7167) > (mmap: reserved=49920KB, committed=49920KB) > > - GC (reserved=2600KB, committed=1748KB) > (malloc=32KB #355) > (mmap: reserved=2568KB, committed=1716KB) > > - Compiler (reserved=213KB, committed=213KB) > (malloc=82KB #331) > (arena=131KB #3) > > - Internal (reserved=4921KB, committed=4917KB) > (malloc=4885KB #16137) > (mmap: reserved=36KB, committed=32KB) > > - Symbol (reserved=17518KB, committed=17518KB) > (malloc=14377KB #172159) > (arena=3141KB #1) > > - Native Memory Tracking (reserved=3617KB, committed=3617KB) > (malloc=149KB #2329) > (tracking overhead=3468KB) > > - Arena Chunk (reserved=178KB, committed=178KB) > (malloc=178KB) > > [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 > VM.native_memory detail | grep -A 10 'for Class' > [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class from > [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, unsigned > long, bool, char*, unsigned long, bool)+0xc2 > [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, > unsigned long, bool, char*, unsigned long)+0x1b > [0x00007f7194ec8300] > Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned > char*)+0x40 > [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf > > [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from > [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, > bool)+0x199 > [0x00007f7194ec6766] > VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned > long)+0x76 > [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned > long, unsigned long)+0xf0 > [0x00007f7194ec9713] VirtualSpaceList::get_new_chunk(unsigned > long, unsigned long, unsigned long)+0xb3 > ... > > [oracle at oel6u7 bin]$ head ../logs/gc.log > Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE > (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 > 20080428 (Red Hat 4.3.0-8) > Memory: 4k page, physical 3082784k(506980k free), swap 1257468k(1148276k > free) > CommandLine flags: -XX:InitialCodeCacheSize=50331648 > -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 > -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 > -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps > -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers > -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace > > -- > Regards > Timur Akhmadeev > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Thu Feb 4 19:02:09 2016 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 4 Feb 2016 11:02:09 -0800 Subject: Metaspace - memory allocation In-Reply-To: References: Message-ID: <56B3A031.30605@oracle.com> On 02/04/2016 06:41 AM, Thomas St?fe wrote: > Hi Timur, > > I believe you see the space for the compressed class data (Compressed > class space), which is by default 1GB and controlled by > -XX:CompressedClassSpaceSize . I think this area is semantically one > subarea of the metaspace, the other being non-class data. > > I am unsure about the difference between MaxMetaSpaceSize and > CompressedClassSpaceSize, but what I believe is that: > > - MaxMetaSpaceSize limits the amount of total committed (not reserved) > space > - CompressedClassSpaceSize sets the amount of space to be reserved > (not committed) for compressed classes only. > > This would fit your NMT output, which says: > > "Class (reserved=1142808KB, committed=106520KB)" > > so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you > specified - you would not be able to commit the whole 1GB, but hit the > limit far earlier. > > See also: > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm > (the section about UseCompressedClassesPointers) > > Please take this with a grain of salt. I am not sure I understand this > correctly and would value input from other developers. Thomas, No disagreement from me. I think you described it correctly. Note that if you turn off UseCompressedClassPointers, there will not be a separate space for the class metadata. Jon > > Kind Regards, Thomas > > On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev > > wrote: > > Hi, > > I'm trying to figure out which parts of memory Hotspot can > allocate in HugePages on Linux (appears to be heap and code cache > by default, plus Metaspace with -XX:+UseLargePagesInMetaspace) and > noticed something I can't explain. > > There's 1G allocation on the OS level in 4K pages that is > apparently attributed to Class metadata, although Metaspace is > maxed at a value much lower than 1G. Question: why? > > [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' > /proc/18906/smaps | grep '^Size:' | head > Size: 4 kB > Size: 4 kB > Size: 2764 kB > Size: 12288 kB > Size: 1036288 kB > Size: 128 kB > Size: 4 kB > Size: 4 kB > Size: 4 kB > Size: 8 kB > > [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 > VM.native_memory summary > 18906: > > Native Memory Tracking: > > Total: reserved=2069703KB, committed=1032559KB > - Java Heap (reserved=786432KB, committed=786432KB) > (mmap: reserved=786432KB, committed=786432KB) > - Class (reserved=1142808KB, committed=106520KB) > (classes #13624) > (malloc=2072KB #22966) > (mmap: reserved=1140736KB, committed=104448KB) > - Thread (reserved=56009KB, committed=56009KB) > (thread #55) > (stack: reserved=55512KB, committed=55512KB) > (malloc=178KB #272) > (arena=319KB #108) > - Code (reserved=55407KB, committed=55407KB) > (malloc=5487KB #7167) > (mmap: reserved=49920KB, committed=49920KB) > - GC (reserved=2600KB, committed=1748KB) > (malloc=32KB #355) > (mmap: reserved=2568KB, committed=1716KB) > - Compiler (reserved=213KB, committed=213KB) > (malloc=82KB #331) > (arena=131KB #3) > - Internal (reserved=4921KB, committed=4917KB) > (malloc=4885KB #16137) > (mmap: reserved=36KB, committed=32KB) > - Symbol (reserved=17518KB, committed=17518KB) > (malloc=14377KB #172159) > (arena=3141KB #1) > - Native Memory Tracking (reserved=3617KB, committed=3617KB) > (malloc=149KB #2329) > (tracking overhead=3468KB) > - Arena Chunk (reserved=178KB, committed=178KB) > (malloc=178KB) > > [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 > VM.native_memory detail | grep -A 10 'for Class' > [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for > Class from > [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, > unsigned long, bool, char*, unsigned long, bool)+0xc2 > [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, > unsigned long, bool, char*, unsigned long)+0x1b > [0x00007f7194ec8300] > Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, > unsigned char*)+0x40 > [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf > > [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from > [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, > bool)+0x199 > [0x00007f7194ec6766] > VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, > unsigned long)+0x76 > [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned long, > unsigned long)+0xf0 > [0x00007f7194ec9713] VirtualSpaceList::get_new_chunk(unsigned > long, unsigned long, unsigned long)+0xb3 > ... > > [oracle at oel6u7 bin]$ head ../logs/gc.log > Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE > (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with > gcc 4.3.0 20080428 (Red Hat 4.3.0-8) > Memory: 4k page, physical 3082784k(506980k free), swap > 1257468k(1148276k free) > CommandLine flags: -XX:InitialCodeCacheSize=50331648 > -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 > -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 > -XX:NativeMemoryTracking=detail -XX:+PrintGC > -XX:+PrintGCTimeStamps -XX:ReservedCodeCacheSize=50331648 > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops > -XX:+UseLargePages -XX:+UseLargePagesInMetaspace > > -- > Regards > Timur Akhmadeev > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From timur.akhmadeev at gmail.com Fri Feb 5 06:53:57 2016 From: timur.akhmadeev at gmail.com (Timur Akhmadeev) Date: Fri, 5 Feb 2016 09:53:57 +0300 Subject: Metaspace - memory allocation In-Reply-To: <56B3A031.30605@oracle.com> References: <56B3A031.30605@oracle.com> Message-ID: Thanks Thomas and Jon! I think I've opened the page Thomas has referenced more than once before asking my question but somehow I missed this paragraph which is the answer :) If UseCompressedOops is turned on and UseCompressedClassesPointers is used, > then two logically different areas of native memory are used for class > metadata. UseCompressedClassPointers uses a 32-bit offset to represent > the class pointer in a 64-bit process as does UseCompressedOops for Java > object references. A region is allocated for these compressed class > pointers (the 32-bit offsets). The size of the region can be set with > CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The space for > the compressed class pointers is reserved as space allocated by mmap at > initialization and committed as needed. TheMaxMetaspaceSize applies to > the sum of the committed compressed class space and the space for the other > class metadata. It's a bit unnatural to call Compressed Class pointers part of the Metaspace and at the same time allow its part to be set bigger than the whole space while not allowing Compressed Class pointers (hopefully) to grow beyond MaxMetaspaceSize. On Thu, Feb 4, 2016 at 10:02 PM, Jon Masamitsu wrote: > > > On 02/04/2016 06:41 AM, Thomas St?fe wrote: > > Hi Timur, > > I believe you see the space for the compressed class data (Compressed > class space), which is by default 1GB and controlled by > -XX:CompressedClassSpaceSize . I think this area is semantically one > subarea of the metaspace, the other being non-class data. > > I am unsure about the difference between MaxMetaSpaceSize and > CompressedClassSpaceSize, but what I believe is that: > > - MaxMetaSpaceSize limits the amount of total committed (not reserved) > space > - CompressedClassSpaceSize sets the amount of space to be reserved (not > committed) for compressed classes only. > > This would fit your NMT output, which says: > > "Class (reserved=1142808KB, committed=106520KB)" > > so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you specified > - you would not be able to commit the whole 1GB, but hit the limit far > earlier. > > See also: > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm > (the section about UseCompressedClassesPointers) > > Please take this with a grain of salt. I am not sure I understand this > correctly and would value input from other developers. > > > Thomas, > > No disagreement from me. I think you described it > correctly. Note that if you turn off UseCompressedClassPointers, > there will not be a separate space for the class metadata. > > Jon > > > > Kind Regards, Thomas > > On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev > wrote: > >> Hi, >> >> I'm trying to figure out which parts of memory Hotspot can allocate in >> HugePages on Linux (appears to be heap and code cache by default, plus >> Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't >> explain. >> >> There's 1G allocation on the OS level in 4K pages that is apparently >> attributed to Class metadata, although Metaspace is maxed at a value much >> lower than 1G. Question: why? >> >> [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' >> /proc/18906/smaps | grep '^Size:' | head >> Size: 4 kB >> Size: 4 kB >> Size: 2764 kB >> Size: 12288 kB >> Size: 1036288 kB >> Size: 128 kB >> Size: 4 kB >> Size: 4 kB >> Size: 4 kB >> Size: 8 kB >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >> VM.native_memory summary >> 18906: >> >> Native Memory Tracking: >> >> Total: reserved=2069703KB, committed=1032559KB >> - Java Heap (reserved=786432KB, committed=786432KB) >> (mmap: reserved=786432KB, committed=786432KB) >> >> - Class (reserved=1142808KB, committed=106520KB) >> (classes #13624) >> (malloc=2072KB #22966) >> (mmap: reserved=1140736KB, >> committed=104448KB) >> >> - Thread (reserved=56009KB, committed=56009KB) >> (thread #55) >> (stack: reserved=55512KB, committed=55512KB) >> (malloc=178KB #272) >> (arena=319KB #108) >> >> - Code (reserved=55407KB, committed=55407KB) >> (malloc=5487KB #7167) >> (mmap: reserved=49920KB, committed=49920KB) >> >> - GC (reserved=2600KB, committed=1748KB) >> (malloc=32KB #355) >> (mmap: reserved=2568KB, committed=1716KB) >> >> - Compiler (reserved=213KB, committed=213KB) >> (malloc=82KB #331) >> (arena=131KB #3) >> >> - Internal (reserved=4921KB, committed=4917KB) >> (malloc=4885KB #16137) >> (mmap: reserved=36KB, committed=32KB) >> >> - Symbol (reserved=17518KB, committed=17518KB) >> (malloc=14377KB #172159) >> (arena=3141KB #1) >> >> - Native Memory Tracking (reserved=3617KB, committed=3617KB) >> (malloc=149KB #2329) >> (tracking overhead=3468KB) >> >> - Arena Chunk (reserved=178KB, committed=178KB) >> (malloc=178KB) >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >> VM.native_memory detail | grep -A 10 'for Class' >> [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class >> from >> [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, >> unsigned long, bool, char*, unsigned long, bool)+0xc2 >> [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, >> unsigned long, bool, char*, unsigned long)+0x1b >> [0x00007f7194ec8300] >> Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned >> char*)+0x40 >> [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf >> >> [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from >> [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, >> bool)+0x199 >> [0x00007f7194ec6766] >> VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned >> long)+0x76 >> [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned >> long, unsigned long)+0xf0 >> [0x00007f7194ec9713] VirtualSpaceList::get_new_chunk(unsigned >> long, unsigned long, unsigned long)+0xb3 >> ... >> >> [oracle at oel6u7 bin]$ head ../logs/gc.log >> Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE >> (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 >> 20080428 (Red Hat 4.3.0-8) >> Memory: 4k page, physical 3082784k(506980k free), swap 1257468k(1148276k >> free) >> CommandLine flags: -XX:InitialCodeCacheSize=50331648 >> -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 >> -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 >> -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps >> -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers >> -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace >> >> -- >> Regards >> Timur Akhmadeev >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -- Regards Timur Akhmadeev -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengtao04 at gmail.com Fri Feb 5 07:02:24 2016 From: fengtao04 at gmail.com (Tao Feng) Date: Thu, 4 Feb 2016 23:02:24 -0800 Subject: gc logging basic question Message-ID: Hi, I have a basic question out of curiosity: is GC logging part of GC cycle? If yes, is it a blocking IO or non-blocking IO? And it would be good if someone could help point out the code base about the GC logging part as I have difficult to identify the code location. Thanks, -Tao -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Fri Feb 5 08:09:34 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 5 Feb 2016 09:09:34 +0100 Subject: Metaspace - memory allocation In-Reply-To: <56B3A031.30605@oracle.com> References: <56B3A031.30605@oracle.com> Message-ID: On Thu, Feb 4, 2016 at 8:02 PM, Jon Masamitsu wrote: > > > On 02/04/2016 06:41 AM, Thomas St?fe wrote: > > Hi Timur, > > I believe you see the space for the compressed class data (Compressed > class space), which is by default 1GB and controlled by > -XX:CompressedClassSpaceSize . I think this area is semantically one > subarea of the metaspace, the other being non-class data. > > I am unsure about the difference between MaxMetaSpaceSize and > CompressedClassSpaceSize, but what I believe is that: > > - MaxMetaSpaceSize limits the amount of total committed (not reserved) > space > - CompressedClassSpaceSize sets the amount of space to be reserved (not > committed) for compressed classes only. > > This would fit your NMT output, which says: > > "Class (reserved=1142808KB, committed=106520KB)" > > so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you specified > - you would not be able to commit the whole 1GB, but hit the limit far > earlier. > > See also: > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm > (the section about UseCompressedClassesPointers) > > Please take this with a grain of salt. I am not sure I understand this > correctly and would value input from other developers. > > > Thomas, > > No disagreement from me. I think you described it > correctly. Note that if you turn off UseCompressedClassPointers, > there will not be a separate space for the class metadata. > > Jon > > Jon, thanks for confirming! ..Thomas > > > Kind Regards, Thomas > > On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev > wrote: > >> Hi, >> >> I'm trying to figure out which parts of memory Hotspot can allocate in >> HugePages on Linux (appears to be heap and code cache by default, plus >> Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't >> explain. >> >> There's 1G allocation on the OS level in 4K pages that is apparently >> attributed to Class metadata, although Metaspace is maxed at a value much >> lower than 1G. Question: why? >> >> [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' >> /proc/18906/smaps | grep '^Size:' | head >> Size: 4 kB >> Size: 4 kB >> Size: 2764 kB >> Size: 12288 kB >> Size: 1036288 kB >> Size: 128 kB >> Size: 4 kB >> Size: 4 kB >> Size: 4 kB >> Size: 8 kB >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >> VM.native_memory summary >> 18906: >> >> Native Memory Tracking: >> >> Total: reserved=2069703KB, committed=1032559KB >> - Java Heap (reserved=786432KB, committed=786432KB) >> (mmap: reserved=786432KB, committed=786432KB) >> >> - Class (reserved=1142808KB, committed=106520KB) >> (classes #13624) >> (malloc=2072KB #22966) >> (mmap: reserved=1140736KB, >> committed=104448KB) >> >> - Thread (reserved=56009KB, committed=56009KB) >> (thread #55) >> (stack: reserved=55512KB, committed=55512KB) >> (malloc=178KB #272) >> (arena=319KB #108) >> >> - Code (reserved=55407KB, committed=55407KB) >> (malloc=5487KB #7167) >> (mmap: reserved=49920KB, committed=49920KB) >> >> - GC (reserved=2600KB, committed=1748KB) >> (malloc=32KB #355) >> (mmap: reserved=2568KB, committed=1716KB) >> >> - Compiler (reserved=213KB, committed=213KB) >> (malloc=82KB #331) >> (arena=131KB #3) >> >> - Internal (reserved=4921KB, committed=4917KB) >> (malloc=4885KB #16137) >> (mmap: reserved=36KB, committed=32KB) >> >> - Symbol (reserved=17518KB, committed=17518KB) >> (malloc=14377KB #172159) >> (arena=3141KB #1) >> >> - Native Memory Tracking (reserved=3617KB, committed=3617KB) >> (malloc=149KB #2329) >> (tracking overhead=3468KB) >> >> - Arena Chunk (reserved=178KB, committed=178KB) >> (malloc=178KB) >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >> VM.native_memory detail | grep -A 10 'for Class' >> [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class >> from >> [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, >> unsigned long, bool, char*, unsigned long, bool)+0xc2 >> [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, >> unsigned long, bool, char*, unsigned long)+0x1b >> [0x00007f7194ec8300] >> Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned >> char*)+0x40 >> [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf >> >> [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from >> [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, >> bool)+0x199 >> [0x00007f7194ec6766] >> VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned >> long)+0x76 >> [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned >> long, unsigned long)+0xf0 >> [0x00007f7194ec9713] VirtualSpaceList::get_new_chunk(unsigned >> long, unsigned long, unsigned long)+0xb3 >> ... >> >> [oracle at oel6u7 bin]$ head ../logs/gc.log >> Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE >> (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 >> 20080428 (Red Hat 4.3.0-8) >> Memory: 4k page, physical 3082784k(506980k free), swap 1257468k(1148276k >> free) >> CommandLine flags: -XX:InitialCodeCacheSize=50331648 >> -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 >> -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 >> -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps >> -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers >> -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace >> >> -- >> Regards >> Timur Akhmadeev >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Fri Feb 5 08:24:48 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 5 Feb 2016 09:24:48 +0100 Subject: Metaspace - memory allocation In-Reply-To: References: <56B3A031.30605@oracle.com> Message-ID: Hi Timur, On Fri, Feb 5, 2016 at 7:53 AM, Timur Akhmadeev wrote: > Thanks Thomas and Jon! > I think I've opened the page Thomas has referenced more than once before > asking my question but somehow I missed this paragraph which is the answer > :) > > If UseCompressedOops is turned on and UseCompressedClassesPointers is >> used, then two logically different areas of native memory are used for >> class metadata. UseCompressedClassPointers uses a 32-bit offset to >> represent the class pointer in a 64-bit process as does UseCompressedOops for >> Java object references. A region is allocated for these compressed class >> pointers (the 32-bit offsets). The size of the region can be set with >> CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The space >> for the compressed class pointers is reserved as space allocated by mmap at >> initialization and committed as needed. TheMaxMetaspaceSize applies to >> the sum of the committed compressed class space and the space for the other >> class metadata. > > > It's a bit unnatural to call Compressed Class pointers part of the > Metaspace and at the same time allow its part to be set bigger than the > whole space while not allowing Compressed Class pointers (hopefully) to > grow beyond MaxMetaspaceSize. > > I agree. Maybe it is just me, but I also have a hard time understanding the relationship between those two parameters. See also the mail thread here: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013797.html . I do not know the history, so I only can guess that the permgen removal came first, and that at first it was just a "simple" metaspace. But with moving out class data from the heap we lost the ability to address them with compressed oops (32bit pointers) and this feature was reintroduced in the form of compressed class pointers, which made the implementation more complicated and now we have those two switches. But this is only a guess from my side. Whatever the reasons, I think the parameters are confusing, especially as the sizes have two different meanings - reserved size in case of CompressedClassSpaceSize, committed size in case of MaxMetaSpaceSize. I also do not know if CompressedClassSpaceSize is a hard limit - what happens when we fill compressed class space with more than that? Will it expand or throw an OOM? Kind Regards, Thomas > On Thu, Feb 4, 2016 at 10:02 PM, Jon Masamitsu > wrote: > >> >> >> On 02/04/2016 06:41 AM, Thomas St?fe wrote: >> >> Hi Timur, >> >> I believe you see the space for the compressed class data (Compressed >> class space), which is by default 1GB and controlled by >> -XX:CompressedClassSpaceSize . I think this area is semantically one >> subarea of the metaspace, the other being non-class data. >> >> I am unsure about the difference between MaxMetaSpaceSize and >> CompressedClassSpaceSize, but what I believe is that: >> >> - MaxMetaSpaceSize limits the amount of total committed (not reserved) >> space >> - CompressedClassSpaceSize sets the amount of space to be reserved (not >> committed) for compressed classes only. >> >> This would fit your NMT output, which says: >> >> "Class (reserved=1142808KB, committed=106520KB)" >> >> so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you >> specified - you would not be able to commit the whole 1GB, but hit the >> limit far earlier. >> >> See also: >> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm >> (the section about UseCompressedClassesPointers) >> >> Please take this with a grain of salt. I am not sure I understand this >> correctly and would value input from other developers. >> >> >> Thomas, >> >> No disagreement from me. I think you described it >> correctly. Note that if you turn off UseCompressedClassPointers, >> there will not be a separate space for the class metadata. >> >> Jon >> >> >> >> Kind Regards, Thomas >> >> On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev < >> timur.akhmadeev at gmail.com> wrote: >> >>> Hi, >>> >>> I'm trying to figure out which parts of memory Hotspot can allocate in >>> HugePages on Linux (appears to be heap and code cache by default, plus >>> Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't >>> explain. >>> >>> There's 1G allocation on the OS level in 4K pages that is apparently >>> attributed to Class metadata, although Metaspace is maxed at a value much >>> lower than 1G. Question: why? >>> >>> [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' >>> /proc/18906/smaps | grep '^Size:' | head >>> Size: 4 kB >>> Size: 4 kB >>> Size: 2764 kB >>> Size: 12288 kB >>> Size: 1036288 kB >>> Size: 128 kB >>> Size: 4 kB >>> Size: 4 kB >>> Size: 4 kB >>> Size: 8 kB >>> >>> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >>> VM.native_memory summary >>> 18906: >>> >>> Native Memory Tracking: >>> >>> Total: reserved=2069703KB, committed=1032559KB >>> - Java Heap (reserved=786432KB, committed=786432KB) >>> (mmap: reserved=786432KB, >>> committed=786432KB) >>> >>> - Class (reserved=1142808KB, committed=106520KB) >>> (classes #13624) >>> (malloc=2072KB #22966) >>> (mmap: reserved=1140736KB, >>> committed=104448KB) >>> >>> - Thread (reserved=56009KB, committed=56009KB) >>> (thread #55) >>> (stack: reserved=55512KB, committed=55512KB) >>> (malloc=178KB #272) >>> (arena=319KB #108) >>> >>> - Code (reserved=55407KB, committed=55407KB) >>> (malloc=5487KB #7167) >>> (mmap: reserved=49920KB, committed=49920KB) >>> >>> - GC (reserved=2600KB, committed=1748KB) >>> (malloc=32KB #355) >>> (mmap: reserved=2568KB, committed=1716KB) >>> >>> - Compiler (reserved=213KB, committed=213KB) >>> (malloc=82KB #331) >>> (arena=131KB #3) >>> >>> - Internal (reserved=4921KB, committed=4917KB) >>> (malloc=4885KB #16137) >>> (mmap: reserved=36KB, committed=32KB) >>> >>> - Symbol (reserved=17518KB, committed=17518KB) >>> (malloc=14377KB #172159) >>> (arena=3141KB #1) >>> >>> - Native Memory Tracking (reserved=3617KB, committed=3617KB) >>> (malloc=149KB #2329) >>> (tracking overhead=3468KB) >>> >>> - Arena Chunk (reserved=178KB, committed=178KB) >>> (malloc=178KB) >>> >>> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >>> VM.native_memory detail | grep -A 10 'for Class' >>> [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class >>> from >>> [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, >>> unsigned long, bool, char*, unsigned long, bool)+0xc2 >>> [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, >>> unsigned long, bool, char*, unsigned long)+0x1b >>> [0x00007f7194ec8300] >>> Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned >>> char*)+0x40 >>> [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf >>> >>> [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from >>> [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, >>> bool)+0x199 >>> [0x00007f7194ec6766] >>> VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned >>> long)+0x76 >>> [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned >>> long, unsigned long)+0xf0 >>> [0x00007f7194ec9713] >>> VirtualSpaceList::get_new_chunk(unsigned long, unsigned long, unsigned >>> long)+0xb3 >>> ... >>> >>> [oracle at oel6u7 bin]$ head ../logs/gc.log >>> Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE >>> (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 >>> 20080428 (Red Hat 4.3.0-8) >>> Memory: 4k page, physical 3082784k(506980k free), swap 1257468k(1148276k >>> free) >>> CommandLine flags: -XX:InitialCodeCacheSize=50331648 >>> -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 >>> -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 >>> -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps >>> -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers >>> -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace >>> >>> -- >>> Regards >>> Timur Akhmadeev >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > > -- > Regards > Timur Akhmadeev > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Fri Feb 5 16:42:47 2016 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 5 Feb 2016 08:42:47 -0800 Subject: Metaspace - memory allocation In-Reply-To: References: <56B3A031.30605@oracle.com> Message-ID: <56B4D107.6000008@oracle.com> On 2/5/2016 12:24 AM, Thomas St?fe wrote: > Hi Timur, > > On Fri, Feb 5, 2016 at 7:53 AM, Timur Akhmadeev > > wrote: > > Thanks Thomas and Jon! > I think I've opened the page Thomas has referenced more than once > before asking my question but somehow I missed this paragraph > which is the answer :) > > If |UseCompressedOops| is turned on and > |UseCompressedClassesPointers| is used, then two logically > different areas of native memory are used for class metadata. > |UseCompressedClassPointers| uses a 32-bit offset to represent > the class pointer in a 64-bit process as does > |UseCompressedOops| for Java object references. A region is > allocated for these compressed class pointers (the 32-bit > offsets). The size of the region can be set with > |CompressedClassSpaceSize| and is 1 gigabyte (GB) by default. > The space for the compressed class pointers is reserved as > space allocated by |mmap| at initialization and committed as > needed. The|MaxMetaspaceSize| applies to the sum of the > committed compressed class space and the space for the other > class metadata. > > It's a bit unnatural to call Compressed Class pointers part of the > Metaspace and at the same time allow its part to be set bigger > than the whole space while not allowing Compressed Class pointers > (hopefully) to grow beyond MaxMetaspaceSize. > > > I agree. Maybe it is just me, but I also have a hard time > understanding the relationship between those two parameters. > > See also the mail thread here: > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013797.html > . > > I do not know the history, so I only can guess that the permgen > removal came first, and that at first it was just a "simple" > metaspace. But with moving out class data from the heap we lost the > ability to address them with compressed oops (32bit pointers) and this > feature was reintroduced in the form of compressed class pointers, > which made the implementation more complicated and now we have those > two switches. When there was a perm gen, the perm gen and the rest of the heap were a contiguous block of addressed with one base address (heap-base). With compressed oops (COOPS) object pointers (including pointers to metadata in the perm gen) were 32bit offsets from heap-base. When metadata was moved to native memory, the address space for the metadata was reserved and committed as it was needed (so there are multiple chunks of addresses for metadata and so not a single metadata base address (meta-base)). For compressed class pointers we needed a single base address (class-base) again so we could use offsets from class-base for the class pointers. So we reserve a single contiguous address space for the compressed class pointers (separate from the other metadata) and commit from that reserved space as it is needed. > > But this is only a guess from my side. Whatever the reasons, I think > the parameters are confusing, especially as the sizes have two > different meanings - reserved size in case of > CompressedClassSpaceSize, committed size in case of MaxMetaSpaceSize. We're trying to manage data that can be accessed differently (full 64bit address and 32bit offset) so need to have different spaces. CompressedClassSpaceSize is the size we reserve at initialization. The part of that which is committed plus the amount of committed space for the other metadata is limited by MaxMetaSpaceSize. > I also do not know if CompressedClassSpaceSize is a hard limit - what > happens when we fill compressed class space with more than that? Will > it expand or throw an OOM? CompressedClassSpaceSize is a hard limit and will limit the number of classes that can be loaded. There are plans to fix that. Jon > > Kind Regards, Thomas > > > On Thu, Feb 4, 2016 at 10:02 PM, Jon Masamitsu > > wrote: > > > > On 02/04/2016 06:41 AM, Thomas St?fe wrote: >> Hi Timur, >> >> I believe you see the space for the compressed class data >> (Compressed class space), which is by default 1GB and >> controlled by -XX:CompressedClassSpaceSize . I think this >> area is semantically one subarea of the metaspace, the other >> being non-class data. >> >> I am unsure about the difference between MaxMetaSpaceSize and >> CompressedClassSpaceSize, but what I believe is that: >> >> - MaxMetaSpaceSize limits the amount of total committed (not >> reserved) space >> - CompressedClassSpaceSize sets the amount of space to be >> reserved (not committed) for compressed classes only. >> >> This would fit your NMT output, which says: >> >> "Class (reserved=1142808KB, committed=106520KB)" >> >> so, ~100M are committed. With a MaxMetaSpaceSize=256M - what >> you specified - you would not be able to commit the whole >> 1GB, but hit the limit far earlier. >> >> See also: >> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm >> (the section about UseCompressedClassesPointers) >> >> Please take this with a grain of salt. I am not sure I >> understand this correctly and would value input from other >> developers. > > Thomas, > > No disagreement from me. I think you described it > correctly. Note that if you turn off UseCompressedClassPointers, > there will not be a separate space for the class metadata. > > Jon > > >> >> Kind Regards, Thomas >> >> On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev >> > > wrote: >> >> Hi, >> >> I'm trying to figure out which parts of memory Hotspot >> can allocate in HugePages on Linux (appears to be heap >> and code cache by default, plus Metaspace with >> -XX:+UseLargePagesInMetaspace) and noticed something I >> can't explain. >> >> There's 1G allocation on the OS level in 4K pages that is >> apparently attributed to Class metadata, although >> Metaspace is maxed at a value much lower than 1G. >> Question: why? >> >> [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' >> /proc/18906/smaps | grep '^Size:' | head >> Size: 4 kB >> Size: 4 kB >> Size: 2764 kB >> Size: 12288 kB >> Size: 1036288 kB >> Size: 128 kB >> Size: 4 kB >> Size: 4 kB >> Size: 4 kB >> Size: 8 kB >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd >> 18906 VM.native_memory summary >> 18906: >> >> Native Memory Tracking: >> >> Total: reserved=2069703KB, committed=1032559KB >> - Java Heap (reserved=786432KB, committed=786432KB) >> (mmap: reserved=786432KB, >> committed=786432KB) >> - Class (reserved=1142808KB, committed=106520KB) >> (classes #13624) >> (malloc=2072KB #22966) >> (mmap: reserved=1140736KB, >> committed=104448KB) >> - Thread (reserved=56009KB, committed=56009KB) >> (thread #55) >> (stack: reserved=55512KB, committed=55512KB) >> (malloc=178KB #272) >> (arena=319KB #108) >> - Code (reserved=55407KB, committed=55407KB) >> (malloc=5487KB #7167) >> (mmap: reserved=49920KB, committed=49920KB) >> - GC (reserved=2600KB, committed=1748KB) >> (malloc=32KB #355) >> (mmap: reserved=2568KB, committed=1716KB) >> - Compiler (reserved=213KB, committed=213KB) >> (malloc=82KB #331) >> (arena=131KB #3) >> - Internal (reserved=4921KB, committed=4917KB) >> (malloc=4885KB #16137) >> (mmap: reserved=36KB, committed=32KB) >> - Symbol (reserved=17518KB, committed=17518KB) >> (malloc=14377KB #172159) >> (arena=3141KB #1) >> - Native Memory Tracking (reserved=3617KB, >> committed=3617KB) >> (malloc=149KB #2329) >> (tracking overhead=3468KB) >> - Arena Chunk (reserved=178KB, committed=178KB) >> (malloc=178KB) >> >> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd >> 18906 VM.native_memory detail | grep -A 10 'for Class' >> [0x0000000100000000 - 0x0000000140000000] reserved >> 1048576KB for Class from >> [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned >> long, unsigned long, bool, char*, unsigned long, bool)+0xc2 >> [0x00007f71950e2d0b] >> ReservedSpace::ReservedSpace(unsigned long, unsigned >> long, bool, char*, unsigned long)+0x1b >> [0x00007f7194ec8300] >> Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned >> char*)+0x40 >> [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf >> >> [0x0000000100000000 - 0x0000000100c00000] committed >> 12288KB from >> [0x00007f71950e2549] VirtualSpace::expand_by(unsigned >> long, bool)+0x199 >> [0x00007f7194ec6766] >> VirtualSpaceList::expand_node_by(VirtualSpaceNode*, >> unsigned long, unsigned long)+0x76 >> [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned >> long, unsigned long)+0xf0 >> [0x00007f7194ec9713] >> VirtualSpaceList::get_new_chunk(unsigned long, unsigned >> long, unsigned long)+0xb3 >> ... >> >> [oracle at oel6u7 bin]$ head ../logs/gc.log >> Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for >> linux-amd64 JRE (1.8.0_72-b15), built on Dec 22 2015 >> 22:00:07 by "java_re" with gcc 4.3.0 20080428 (Red Hat >> 4.3.0-8) >> Memory: 4k page, physical 3082784k(506980k free), swap >> 1257468k(1148276k free) >> CommandLine flags: -XX:InitialCodeCacheSize=50331648 >> -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 >> -XX:MaxMetaspaceSize=268435456 >> -XX:MetaspaceSize=268435456 >> -XX:NativeMemoryTracking=detail -XX:+PrintGC >> -XX:+PrintGCTimeStamps -XX:ReservedCodeCacheSize=50331648 >> -XX:+UseCompressedClassPointers -XX:+UseCompressedOops >> -XX:+UseLargePages -XX:+UseLargePagesInMetaspace >> >> -- >> Regards >> Timur Akhmadeev >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -- > Regards > Timur Akhmadeev > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Tue Feb 9 06:35:15 2016 From: yu.zhang at oracle.com (Yu Zhang) Date: Mon, 8 Feb 2016 22:35:15 -0800 Subject: gc logging basic question In-Reply-To: References: Message-ID: <56B988A3.2090505@oracle.com> Tao, GC logging is not part of GC STW pause. Though we may see application being stopped due to gc logging, if there are a lot of I/O activities on the system. The gc log file is opened by the following _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); Thanks, Jenny On 2/4/2016 11:02 PM, Tao Feng wrote: > Hi, > > I have a basic question out of curiosity: is GC logging part of GC > cycle? If yes, is it a blocking IO or non-blocking IO? And it would be > good if someone could help point out the code base about the GC > logging part as I have difficult to identify the code location. > > Thanks, > -Tao > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Feb 9 10:11:58 2016 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 09 Feb 2016 11:11:58 +0100 Subject: gc logging basic question In-Reply-To: <56B988A3.2090505@oracle.com> References: <56B988A3.2090505@oracle.com> Message-ID: <1455012718.2210.31.camel@oracle.com> Hi, On Mon, 2016-02-08 at 22:35 -0800, Yu Zhang wrote: > Tao, > > GC logging is not part of GC STW pause. Though we may see application > being stopped due to gc logging, if there are a lot of I/O activities > on the system. > The gc log file is opened by the following > _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); > > Thanks, > Jenny > On 2/4/2016 11:02 PM, Tao Feng wrote: > > Hi, > > > > I have a basic question out of curiosity: is GC logging part of GC > > cycle? If yes, is it a blocking IO or non-blocking IO? And it would > > be good if someone could help point out the code base about the GC > > logging part as I have difficult to identify the code location. I think any logging enabled by -Xlog opens the file in asynchronous write mode, output is buffered. There is no deliberate fsync'ing going on (again, afaik). However, lower levels (OS, file system implementations) might impose additional synchronization. If a different process saturates I/O to the disk you want to write to at the same time, the behavior might not be optimal. :) See e.g. http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-Apr il/001835.html . If you noticed a problem because of that, there are the following options: 1) use a separate disk for the logging (throwing the "tmpfs" keyword into this bucket) 2) use JFR. Afaik its events are written asynchronously using separate threads. That might or might not be a problem if you saturate I/O bandwidth of your target disk for a long time (go to option 1 in this case) 3) experiment with different FS implementations that do less blocking. 4) provide a patch to improve this situation :-) Thanks, Thomas From fengtao04 at gmail.com Tue Feb 9 17:46:03 2016 From: fengtao04 at gmail.com (Tao Feng) Date: Tue, 9 Feb 2016 09:46:03 -0800 Subject: gc logging basic question In-Reply-To: <1455012718.2210.31.camel@oracle.com> References: <56B988A3.2090505@oracle.com> <1455012718.2210.31.camel@oracle.com> Message-ID: Hi Jenny, Thomas, Thanks a lot for your reply. Indeed we observe high real time in gc which we correlate with high write syscall time. This makes us doubt whether GC logging is part of cycle. Thanks, -Tao On Tue, Feb 9, 2016 at 2:11 AM, Thomas Schatzl wrote: > Hi, > > On Mon, 2016-02-08 at 22:35 -0800, Yu Zhang wrote: > > Tao, > > > > GC logging is not part of GC STW pause. Though we may see application > > being stopped due to gc logging, if there are a lot of I/O activities > > on the system. > > The gc log file is opened by the following > > _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); > > > > Thanks, > > Jenny > > On 2/4/2016 11:02 PM, Tao Feng wrote: > > > Hi, > > > > > > I have a basic question out of curiosity: is GC logging part of GC > > > cycle? If yes, is it a blocking IO or non-blocking IO? And it would > > > be good if someone could help point out the code base about the GC > > > logging part as I have difficult to identify the code location. > > I think any logging enabled by -Xlog opens the file in asynchronous > write mode, output is buffered. There is no deliberate fsync'ing going > on (again, afaik). > > However, lower levels (OS, file system implementations) might impose > additional synchronization. If a different process saturates I/O to the > disk you want to write to at the same time, the behavior might not be > optimal. :) > > See e.g. http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-Apr > il/001835.html . > > If you noticed a problem because of that, there are the following > options: > > 1) use a separate disk for the logging (throwing the "tmpfs" keyword > into this bucket) > > 2) use JFR. Afaik its events are written asynchronously using separate > threads. That might or might not be a problem if you saturate I/O > bandwidth of your target disk for a long time (go to option 1 in this > case) > > 3) experiment with different FS implementations that do less blocking. > > 4) provide a patch to improve this situation :-) > > Thanks, > Thomas > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed Feb 10 10:07:58 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 10 Feb 2016 11:07:58 +0100 Subject: Metaspace - memory allocation In-Reply-To: <56B4D107.6000008@oracle.com> References: <56B3A031.30605@oracle.com> <56B4D107.6000008@oracle.com> Message-ID: Hi Jon, thanks again for the clarification! Kind Regards, Thomas On Fri, Feb 5, 2016 at 5:42 PM, Jon Masamitsu wrote: > > > On 2/5/2016 12:24 AM, Thomas St?fe wrote: > > Hi Timur, > > On Fri, Feb 5, 2016 at 7:53 AM, Timur Akhmadeev > wrote: > >> Thanks Thomas and Jon! >> I think I've opened the page Thomas has referenced more than once before >> asking my question but somehow I missed this paragraph which is the answer >> :) >> >> If UseCompressedOops is turned on and UseCompressedClassesPointers is >>> used, then two logically different areas of native memory are used for >>> class metadata. UseCompressedClassPointers uses a 32-bit offset to >>> represent the class pointer in a 64-bit process as does >>> UseCompressedOops for Java object references. A region is allocated for >>> these compressed class pointers (the 32-bit offsets). The size of the >>> region can be set with CompressedClassSpaceSize and is 1 gigabyte (GB) >>> by default. The space for the compressed class pointers is reserved as >>> space allocated by mmap at initialization and committed as needed. The >>> MaxMetaspaceSize applies to the sum of the committed compressed class >>> space and the space for the other class metadata. >> >> >> It's a bit unnatural to call Compressed Class pointers part of the >> Metaspace and at the same time allow its part to be set bigger than the >> whole space while not allowing Compressed Class pointers (hopefully) to >> grow beyond MaxMetaspaceSize. >> >> > I agree. Maybe it is just me, but I also have a hard time understanding > the relationship between those two parameters. > > > > See also the mail thread here: > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013797.html > . > > I do not know the history, so I only can guess that the permgen removal > came first, and that at first it was just a "simple" metaspace. But with > moving out class data from the heap we lost the ability to address them > with compressed oops (32bit pointers) and this feature was reintroduced in > the form of compressed class pointers, which made the implementation more > complicated and now we have those two switches. > > > > When there was a perm gen, the perm gen and the rest of the heap were a > contiguous > block of addressed with one base address (heap-base). With compressed > oops (COOPS) > object pointers (including pointers to metadata in the perm gen) were > 32bit offsets from > heap-base. When metadata was moved to native memory, the address space > for the > metadata was reserved and committed as it was needed (so there are > multiple chunks > of addresses for metadata and so not a single metadata base address > (meta-base)). > For compressed class pointers we needed a single base address (class-base) > again > so we could use offsets from class-base for the class pointers. So we > reserve a > single contiguous address space for the compressed class pointers (separate > from the other metadata) and commit from that reserved space as it is > needed. > > > But this is only a guess from my side. Whatever the reasons, I think the > parameters are confusing, especially as the sizes have two different > meanings - reserved size in case of CompressedClassSpaceSize, committed > size in case of MaxMetaSpaceSize. > > > We're trying to manage data that can be accessed differently (full 64bit > address and 32bit offset) > so need to have different spaces. CompressedClassSpaceSize is the size we > reserve at > initialization. The part of that which is committed plus the amount of > committed space for > the other metadata is limited by MaxMetaSpaceSize. > > I also do not know if CompressedClassSpaceSize is a hard limit - what > happens when we fill compressed class space with more than that? Will it > expand or throw an OOM? > > > CompressedClassSpaceSize is a hard limit and will limit the number of > classes that can be > loaded. There are plans to fix that. > > Jon > > > Kind Regards, Thomas > > > >> On Thu, Feb 4, 2016 at 10:02 PM, Jon Masamitsu < >> jon.masamitsu at oracle.com> wrote: >> >>> >>> >>> On 02/04/2016 06:41 AM, Thomas St?fe wrote: >>> >>> Hi Timur, >>> >>> I believe you see the space for the compressed class data (Compressed >>> class space), which is by default 1GB and controlled by >>> -XX:CompressedClassSpaceSize . I think this area is semantically one >>> subarea of the metaspace, the other being non-class data. >>> >>> I am unsure about the difference between MaxMetaSpaceSize and >>> CompressedClassSpaceSize, but what I believe is that: >>> >>> - MaxMetaSpaceSize limits the amount of total committed (not reserved) >>> space >>> - CompressedClassSpaceSize sets the amount of space to be reserved (not >>> committed) for compressed classes only. >>> >>> This would fit your NMT output, which says: >>> >>> "Class (reserved=1142808KB, committed=106520KB)" >>> >>> so, ~100M are committed. With a MaxMetaSpaceSize=256M - what you >>> specified - you would not be able to commit the whole 1GB, but hit the >>> limit far earlier. >>> >>> See also: >>> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.htm >>> (the section about UseCompressedClassesPointers) >>> >>> Please take this with a grain of salt. I am not sure I understand this >>> correctly and would value input from other developers. >>> >>> >>> Thomas, >>> >>> No disagreement from me. I think you described it >>> correctly. Note that if you turn off UseCompressedClassPointers, >>> there will not be a separate space for the class metadata. >>> >>> Jon >>> >>> >>> >>> Kind Regards, Thomas >>> >>> On Thu, Feb 4, 2016 at 2:13 PM, Timur Akhmadeev < >>> timur.akhmadeev at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I'm trying to figure out which parts of memory Hotspot can allocate in >>>> HugePages on Linux (appears to be heap and code cache by default, plus >>>> Metaspace with -XX:+UseLargePagesInMetaspace) and noticed something I can't >>>> explain. >>>> >>>> There's 1G allocation on the OS level in 4K pages that is apparently >>>> attributed to Class metadata, although Metaspace is maxed at a value much >>>> lower than 1G. Question: why? >>>> >>>> [oracle at oel6u7 bin]$ grep -B 11 'KernelPageSize: 4 kB' >>>> /proc/18906/smaps | grep '^Size:' | head >>>> Size: 4 kB >>>> Size: 4 kB >>>> Size: 2764 kB >>>> Size: 12288 kB >>>> Size: 1036288 kB >>>> Size: 128 kB >>>> Size: 4 kB >>>> Size: 4 kB >>>> Size: 4 kB >>>> Size: 8 kB >>>> >>>> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >>>> VM.native_memory summary >>>> 18906: >>>> >>>> Native Memory Tracking: >>>> >>>> Total: reserved=2069703KB, committed=1032559KB >>>> - Java Heap (reserved=786432KB, committed=786432KB) >>>> (mmap: reserved=786432KB, >>>> committed=786432KB) >>>> >>>> - Class (reserved=1142808KB, committed=106520KB) >>>> (classes #13624) >>>> (malloc=2072KB #22966) >>>> (mmap: reserved=1140736KB, >>>> committed=104448KB) >>>> >>>> - Thread (reserved=56009KB, committed=56009KB) >>>> (thread #55) >>>> (stack: reserved=55512KB, committed=55512KB) >>>> (malloc=178KB #272) >>>> (arena=319KB #108) >>>> >>>> - Code (reserved=55407KB, committed=55407KB) >>>> (malloc=5487KB #7167) >>>> (mmap: reserved=49920KB, committed=49920KB) >>>> >>>> - GC (reserved=2600KB, committed=1748KB) >>>> (malloc=32KB #355) >>>> (mmap: reserved=2568KB, committed=1716KB) >>>> >>>> - Compiler (reserved=213KB, committed=213KB) >>>> (malloc=82KB #331) >>>> (arena=131KB #3) >>>> >>>> - Internal (reserved=4921KB, committed=4917KB) >>>> (malloc=4885KB #16137) >>>> (mmap: reserved=36KB, committed=32KB) >>>> >>>> - Symbol (reserved=17518KB, committed=17518KB) >>>> (malloc=14377KB #172159) >>>> (arena=3141KB #1) >>>> >>>> - Native Memory Tracking (reserved=3617KB, committed=3617KB) >>>> (malloc=149KB #2329) >>>> (tracking overhead=3468KB) >>>> >>>> - Arena Chunk (reserved=178KB, committed=178KB) >>>> (malloc=178KB) >>>> >>>> [oracle at oel6u7 bin]$ /home/oracle/jdk1.8.0_72/bin/jcmd 18906 >>>> VM.native_memory detail | grep -A 10 'for Class' >>>> [0x0000000100000000 - 0x0000000140000000] reserved 1048576KB for Class >>>> from >>>> [0x00007f71950e2ad2] ReservedSpace::initialize(unsigned long, >>>> unsigned long, bool, char*, unsigned long, bool)+0xc2 >>>> [0x00007f71950e2d0b] ReservedSpace::ReservedSpace(unsigned long, >>>> unsigned long, bool, char*, unsigned long)+0x1b >>>> [0x00007f7194ec8300] >>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned >>>> char*)+0x40 >>>> [0x00007f7194eca7df] Metaspace::global_initialize()+0x4cf >>>> >>>> [0x0000000100000000 - 0x0000000100c00000] committed 12288KB from >>>> [0x00007f71950e2549] VirtualSpace::expand_by(unsigned long, >>>> bool)+0x199 >>>> [0x00007f7194ec6766] >>>> VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned >>>> long)+0x76 >>>> [0x00007f7194ec9580] VirtualSpaceList::expand_by(unsigned >>>> long, unsigned long)+0xf0 >>>> [0x00007f7194ec9713] >>>> VirtualSpaceList::get_new_chunk(unsigned long, unsigned long, unsigned >>>> long)+0xb3 >>>> ... >>>> >>>> [oracle at oel6u7 bin]$ head ../logs/gc.log >>>> Java HotSpot(TM) 64-Bit Server VM (25.72-b15) for linux-amd64 JRE >>>> (1.8.0_72-b15), built on Dec 22 2015 22:00:07 by "java_re" with gcc 4.3.0 >>>> 20080428 (Red Hat 4.3.0-8) >>>> Memory: 4k page, physical 3082784k(506980k free), swap >>>> 1257468k(1148276k free) >>>> CommandLine flags: -XX:InitialCodeCacheSize=50331648 >>>> -XX:InitialHeapSize=49324544 -XX:MaxHeapSize=805306368 >>>> -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=268435456 >>>> -XX:NativeMemoryTracking=detail -XX:+PrintGC -XX:+PrintGCTimeStamps >>>> -XX:ReservedCodeCacheSize=50331648 -XX:+UseCompressedClassPointers >>>> -XX:+UseCompressedOops -XX:+UseLargePages -XX:+UseLargePagesInMetaspace >>>> >>>> -- >>>> Regards >>>> Timur Akhmadeev >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> >> >> -- >> Regards >> Timur Akhmadeev >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.r.akesson at gmail.com Wed Feb 10 17:12:49 2016 From: gustav.r.akesson at gmail.com (=?UTF-8?Q?Gustav_=C3=85kesson?=) Date: Wed, 10 Feb 2016 18:12:49 +0100 Subject: gc logging basic question In-Reply-To: References: <56B988A3.2090505@oracle.com> <1455012718.2210.31.camel@oracle.com> Message-ID: Hi, Well, AFAIK the loggning IS part of STW cycle. I had troubles with this before, which results in GCs with high systime. Since then I always print the logs to RAM disk, which solves the problem. Best Regards, Gustav ?kesson Den 9 feb 2016 18:46 skrev "Tao Feng" : > Hi Jenny, Thomas, > > Thanks a lot for your reply. Indeed we observe high real time in gc which > we correlate with high write syscall time. This makes us doubt whether GC > logging is part of cycle. > > Thanks, > -Tao > > On Tue, Feb 9, 2016 at 2:11 AM, Thomas Schatzl > wrote: > >> Hi, >> >> On Mon, 2016-02-08 at 22:35 -0800, Yu Zhang wrote: >> > Tao, >> > >> > GC logging is not part of GC STW pause. Though we may see application >> > being stopped due to gc logging, if there are a lot of I/O activities >> > on the system. >> > The gc log file is opened by the following >> > _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); >> > >> > Thanks, >> > Jenny >> > On 2/4/2016 11:02 PM, Tao Feng wrote: >> > > Hi, >> > > >> > > I have a basic question out of curiosity: is GC logging part of GC >> > > cycle? If yes, is it a blocking IO or non-blocking IO? And it would >> > > be good if someone could help point out the code base about the GC >> > > logging part as I have difficult to identify the code location. >> >> I think any logging enabled by -Xlog opens the file in asynchronous >> write mode, output is buffered. There is no deliberate fsync'ing going >> on (again, afaik). >> >> However, lower levels (OS, file system implementations) might impose >> additional synchronization. If a different process saturates I/O to the >> disk you want to write to at the same time, the behavior might not be >> optimal. :) >> >> See e.g. http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-Apr >> il/001835.html >> >> . >> >> If you noticed a problem because of that, there are the following >> options: >> >> 1) use a separate disk for the logging (throwing the "tmpfs" keyword >> into this bucket) >> >> 2) use JFR. Afaik its events are written asynchronously using separate >> threads. That might or might not be a problem if you saturate I/O >> bandwidth of your target disk for a long time (go to option 1 in this >> case) >> >> 3) experiment with different FS implementations that do less blocking. >> >> 4) provide a patch to improve this situation :-) >> >> Thanks, >> Thomas >> >> >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysr1729 at gmail.com Wed Feb 10 21:04:13 2016 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Wed, 10 Feb 2016 13:04:13 -0800 Subject: gc logging basic question In-Reply-To: References: <56B988A3.2090505@oracle.com> <1455012718.2210.31.camel@oracle.com> Message-ID: We've run into similar issues at Twitter, and are readying a patch (re # (4) of Thomas' mail) for buffering in-memory and writing to disk/file out-of-line wrt safepoints or loggers. The patch is under test and review internally at this time, and we hope to publish it for review on Open JDK once internal test and review are complete. Thomas, does unified logging include GC and will it be done outside of safepoints? In that case, we can use the same code rather than spinning code specifically for gc logging (as I am doing in our patch for this). I must confess I haven't looked at the unified logging jep or its state in the JDK 9 repo. thanks. -- ramki On Wed, Feb 10, 2016 at 9:12 AM, Gustav ?kesson wrote: > Hi, > > Well, AFAIK the loggning IS part of STW cycle. I had troubles with this > before, which results in GCs with high systime. Since then I always print > the logs to RAM disk, which solves the problem. > > Best Regards, > Gustav ?kesson > Den 9 feb 2016 18:46 skrev "Tao Feng" : > >> Hi Jenny, Thomas, >> >> Thanks a lot for your reply. Indeed we observe high real time in gc which >> we correlate with high write syscall time. This makes us doubt whether GC >> logging is part of cycle. >> >> Thanks, >> -Tao >> >> On Tue, Feb 9, 2016 at 2:11 AM, Thomas Schatzl > > wrote: >> >>> Hi, >>> >>> On Mon, 2016-02-08 at 22:35 -0800, Yu Zhang wrote: >>> > Tao, >>> > >>> > GC logging is not part of GC STW pause. Though we may see application >>> > being stopped due to gc logging, if there are a lot of I/O activities >>> > on the system. >>> > The gc log file is opened by the following >>> > _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); >>> > >>> > Thanks, >>> > Jenny >>> > On 2/4/2016 11:02 PM, Tao Feng wrote: >>> > > Hi, >>> > > >>> > > I have a basic question out of curiosity: is GC logging part of GC >>> > > cycle? If yes, is it a blocking IO or non-blocking IO? And it would >>> > > be good if someone could help point out the code base about the GC >>> > > logging part as I have difficult to identify the code location. >>> >>> I think any logging enabled by -Xlog opens the file in asynchronous >>> write mode, output is buffered. There is no deliberate fsync'ing going >>> on (again, afaik). >>> >>> However, lower levels (OS, file system implementations) might impose >>> additional synchronization. If a different process saturates I/O to the >>> disk you want to write to at the same time, the behavior might not be >>> optimal. :) >>> >>> See e.g. http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-Apr >>> il/001835.html >>> >>> . >>> >>> If you noticed a problem because of that, there are the following >>> options: >>> >>> 1) use a separate disk for the logging (throwing the "tmpfs" keyword >>> into this bucket) >>> >>> 2) use JFR. Afaik its events are written asynchronously using separate >>> threads. That might or might not be a problem if you saturate I/O >>> bandwidth of your target disk for a long time (go to option 1 in this >>> case) >>> >>> 3) experiment with different FS implementations that do less blocking. >>> >>> 4) provide a patch to improve this situation :-) >>> >>> Thanks, >>> Thomas >>> >>> >>> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengtao04 at gmail.com Fri Feb 12 00:52:32 2016 From: fengtao04 at gmail.com (Tao Feng) Date: Thu, 11 Feb 2016 16:52:32 -0800 Subject: gc logging basic question In-Reply-To: References: <56B988A3.2090505@oracle.com> <1455012718.2210.31.camel@oracle.com> Message-ID: Just FYI, at linkedin we encountered this issue many times. And one of our coworker recently wrote a blog posted about this issue: https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic . Thanks, -Tao On Wed, Feb 10, 2016 at 1:04 PM, Srinivas Ramakrishna wrote: > We've run into similar issues at Twitter, and are readying a patch (re # > (4) of Thomas' mail) for buffering in-memory > and writing to disk/file out-of-line wrt safepoints or loggers. The patch > is under test and review internally at this time, > and we hope to publish it for review on Open JDK once internal test and > review are complete. > > Thomas, does unified logging include GC and will it be done outside of > safepoints? In that case, we can use the same > code rather than spinning code specifically for gc logging (as I am doing > in our patch for this). I must confess I haven't > looked at the unified logging jep or its state in the JDK 9 repo. > > thanks. > -- ramki > > On Wed, Feb 10, 2016 at 9:12 AM, Gustav ?kesson < > gustav.r.akesson at gmail.com> wrote: > >> Hi, >> >> Well, AFAIK the loggning IS part of STW cycle. I had troubles with this >> before, which results in GCs with high systime. Since then I always print >> the logs to RAM disk, which solves the problem. >> >> Best Regards, >> Gustav ?kesson >> Den 9 feb 2016 18:46 skrev "Tao Feng" : >> >>> Hi Jenny, Thomas, >>> >>> Thanks a lot for your reply. Indeed we observe high real time in gc >>> which we correlate with high write syscall time. This makes us doubt >>> whether GC logging is part of cycle. >>> >>> Thanks, >>> -Tao >>> >>> On Tue, Feb 9, 2016 at 2:11 AM, Thomas Schatzl < >>> thomas.schatzl at oracle.com> wrote: >>> >>>> Hi, >>>> >>>> On Mon, 2016-02-08 at 22:35 -0800, Yu Zhang wrote: >>>> > Tao, >>>> > >>>> > GC logging is not part of GC STW pause. Though we may see application >>>> > being stopped due to gc logging, if there are a lot of I/O activities >>>> > on the system. >>>> > The gc log file is opened by the following >>>> > _fd = open(file_name, O_WRONLY | O_CREAT | O_TRUNC, 0666); >>>> > >>>> > Thanks, >>>> > Jenny >>>> > On 2/4/2016 11:02 PM, Tao Feng wrote: >>>> > > Hi, >>>> > > >>>> > > I have a basic question out of curiosity: is GC logging part of GC >>>> > > cycle? If yes, is it a blocking IO or non-blocking IO? And it would >>>> > > be good if someone could help point out the code base about the GC >>>> > > logging part as I have difficult to identify the code location. >>>> >>>> I think any logging enabled by -Xlog opens the file in asynchronous >>>> write mode, output is buffered. There is no deliberate fsync'ing going >>>> on (again, afaik). >>>> >>>> However, lower levels (OS, file system implementations) might impose >>>> additional synchronization. If a different process saturates I/O to the >>>> disk you want to write to at the same time, the behavior might not be >>>> optimal. :) >>>> >>>> See e.g. http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-Apr >>>> il/001835.html >>>> >>>> . >>>> >>>> If you noticed a problem because of that, there are the following >>>> options: >>>> >>>> 1) use a separate disk for the logging (throwing the "tmpfs" keyword >>>> into this bucket) >>>> >>>> 2) use JFR. Afaik its events are written asynchronously using separate >>>> threads. That might or might not be a problem if you saturate I/O >>>> bandwidth of your target disk for a long time (go to option 1 in this >>>> case) >>>> >>>> 3) experiment with different FS implementations that do less blocking. >>>> >>>> 4) provide a patch to improve this situation :-) >>>> >>>> Thanks, >>>> Thomas >>>> >>>> >>>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jun.zhuang at hobsons.com Sat Feb 20 04:15:32 2016 From: jun.zhuang at hobsons.com (Jun Zhuang) Date: Sat, 20 Feb 2016 04:15:32 +0000 Subject: Questions regarding GC Message-ID: Hi, I tried to find answers to a couple of questions I have regarding GC but found nothing on the Internet, I wonder if the experts here can help me out? * How are objects stored in the old generation when promoted from the young gen or survivor space? Does the JVM use something similar to the TLAB? * The time for a young GC pause consists stack scan, card scan and copying, are there command line options for me to get a time breakdown for those times in the GC log? Thanks, Jun Jun Zhuang Sr. Performance QA Engineer | Hobsons T: +1 513 746 2288 | jun.zhuang at hobsons.com 50 E-Business Way, Suite 300 | Cincinnati, OH 45241 | USA Hobsons This message contains confidential information and is intended only for the intended recipients. If you are not an intended recipient you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image370ea1.gif at 780cb1d1.gif Type: image/png Size: 1672 bytes Desc: image370ea1.gif at 780cb1d1.gif URL: From yiyeguhu at gmail.com Sat Feb 20 04:52:08 2016 From: yiyeguhu at gmail.com (Tao Mao) Date: Fri, 19 Feb 2016 20:52:08 -0800 Subject: Questions regarding GC In-Reply-To: References: Message-ID: Hi Jun, 1. Search for PLAB 2. The following are pretty much all printing-GC-related flags we have: PrintGC PrintGCDetails PrintGCTimeStamps PrintGCDateStamps PrintTenuringDistribution PrintGCTaskTimeStamps PrintApplicationConcurrentTime PrintApplicationStoppedTime You might want to try the last three options. Plus, "-XX:+UseG1GC +XX:+PrintGCDetails" has more verbose gc logs than other collectors'. Thanks. Tao Mao On Fri, Feb 19, 2016 at 8:15 PM, Jun Zhuang wrote: > Hi, > > > > I tried to find answers to a couple of questions I have regarding GC but > found nothing on the Internet, I wonder if the experts here can help me out? > > > > ? How are objects stored in the old generation when promoted from > the young gen or survivor space? Does the JVM use something similar to the > TLAB? > > ? The time for a young GC pause consists stack scan, card scan > and copying, are there command line options for me to get a time breakdown > for those times in the GC log? > > > > Thanks, > > Jun > > > > *Jun Zhuang* > > *Sr. Performance QA Engineer* | Hobsons > > > T: +1 513 746 2288 | jun.zhuang at hobsons.com > > 50 E-Business Way, Suite 300 | Cincinnati, OH 45241 | USA > > > > > > [image: Hobsons] > This message contains confidential information and is intended only for > the intended recipients. If you are not an intended recipient you should > not disseminate, distribute or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. E-mail transmission cannot be > guaranteed to be secure or error-free as information could be intercepted, > corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. > The sender therefore does not accept liability for any errors or omissions > in the contents of this message, which arise as a result of e-mail > transmission. If verification is required please request a hard-copy > version. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image370ea1.gif at 780cb1d1.gif Type: image/png Size: 1672 bytes Desc: not available URL: From thomas.schatzl at oracle.com Tue Feb 23 12:48:31 2016 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 23 Feb 2016 13:48:31 +0100 Subject: Questions regarding GC In-Reply-To: References: Message-ID: <1456231711.2262.57.camel@oracle.com> Hi Jun, On Sat, 2016-02-20 at 04:15 +0000, Jun Zhuang wrote: > Hi, > > I tried to find answers to a couple of questions I have regarding GC > but found nothing on the Internet, I wonder if the experts here can > help me out? > > ? How are objects stored in the old generation when promoted > from the young gen or survivor space? Does the JVM use something > similar to the TLAB? as Tao already mentioned, the equivalent is PLAB (Promotion Local Allocation Buffer). All collectors but Serial GC (which, due to being single threaded) use them. Parallel GC always uses PLABs of the same size, CMS and G1 support dynamic PLAB resizing. You could play around with that using the -XX:+/ -ResizePLAB and -XX:Old/YoungPLABSize options. However, generally, the collector's automatic PLAB sizing is pretty good. > ? The time for a young GC pause consists stack scan, card > scan and copying, are there command line options for me to get a time > breakdown for those times in the GC log? Only G1 provides a detailed breakdown of the phases, most detail with -XX:+EnableExperimentalVMOptions -XX:G1LogLevel=finest (on 8u60). Tao already mentioned a few of the more useful options. Beginning with JDK9, due to changes with the logging system, something like -Xlog:gc*=trace will get you all available information (iirc). There are some aliases (-XX:+PrintGCDetails -XX:+PrintGC) that map to the new logging options, but some of the logging switches are/will not be usable any more. Thanks, Thomas