From aph at redhat.com Tue Sep 19 10:44:27 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 19 Sep 2017 11:44:27 +0100 Subject: OpenJDK OOM issue - In-Reply-To: References: Message-ID: On 19/09/17 09:50, Yu, Tim (NSB - CN/Chengdu) wrote: > We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. Hard to say. If I had something I could reproduce and debug, I could tell you the answer in a few minutes. Otherwise it's going to be hard. strace might help. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From david.holmes at oracle.com Tue Sep 19 10:46:08 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Sep 2017 20:46:08 +1000 Subject: OpenJDK OOM issue - In-Reply-To: References: Message-ID: <8f7f43fd-fbee-fd54-21ab-d43e24e8a694@oracle.com> Hi Tim, On 19/09/2017 6:50 PM, Yu, Tim (NSB - CN/Chengdu) wrote: > Hi OpenJDK dev group This is better discussed on hotspot-dev, so redirecting there. > We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. Unfortunately there is no way to tell. As you indicate there appears to be enough memory, and there appear to be enough threads/processes available for creation. Does this happen regularly or was this a one-of failure? You really need to see the exact state of the machine at the time this happened. David > " > on Sep 18 11:05:04 EEST 2017 2 or first INFO log missing: Error occurred during initialization of VM > java.lang.OutOfMemoryError: unable to create new native thread > Error occurred during initialization of VM > java.lang.OutOfMemoryError: unable to create new native thread > > 1. Memory Usage > MemFree: 898332 kB > From below core file generated during OMM, it can be seen about 900M physical memory available during that time. > > > 2 Thread number > > sh-4.1$ ps -eLf|wc -l > 5326 > > > sh-4.1$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 43497 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 43497 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > Br, > Tim > > > > From poonam.bajaj at oracle.com Tue Sep 19 13:08:57 2017 From: poonam.bajaj at oracle.com (Poonam Parhar) Date: Tue, 19 Sep 2017 06:08:57 -0700 (PDT) Subject: OpenJDK OOM issue - In-Reply-To: <8f7f43fd-fbee-fd54-21ab-d43e24e8a694@oracle.com> References: <8f7f43fd-fbee-fd54-21ab-d43e24e8a694@oracle.com> Message-ID: Hello Tim, With CompressedOops enabled (which is enabled by default with 64-bit JVM), the Java heap may get placed in the lower virtual address space leaving very little space for the native heap allocations, and that can cause these kinds of failures even when there is lot of native memory available. Please read details here: https://blogs.oracle.com/poonam/running-on-a-64bit-platform-and-still-running-out-of-memory You can check the output collected with -XX:+PrintGCDetails that would show where your Java Heap is being based at. Thanks, Poonam > -----Original Message----- > From: David Holmes > Sent: Tuesday, September 19, 2017 3:46 AM > To: Yu, Tim (NSB - CN/Chengdu) > Cc: jdk8-dev at openjdk.java.net; jdk8u-dev at openjdk.java.net; hotspot-dev > developers; Shen, David (NSB - CN/Chengdu) > Subject: Re: OpenJDK OOM issue - > > Hi Tim, > > On 19/09/2017 6:50 PM, Yu, Tim (NSB - CN/Chengdu) wrote: > > Hi OpenJDK dev group > > This is better discussed on hotspot-dev, so redirecting there. > > > We meet one issue that the VM failed to initialize. The error log is > as below. We checked both memory usage and thread number. They do not > hit the limit. So could you please help to confirm why > "java.lang.OutOfMemoryError: unable to create new native thread" error > occurs? Many thanks. > > Unfortunately there is no way to tell. As you indicate there appears to > be enough memory, and there appear to be enough threads/processes > available for creation. > > Does this happen regularly or was this a one-of failure? You really > need to see the exact state of the machine at the time this happened. > > David > > > " > > on Sep 18 11:05:04 EEST 2017 2 or first INFO log missing: Error > > occurred during initialization of VM > > java.lang.OutOfMemoryError: unable to create new native thread Error > > occurred during initialization of VM > > java.lang.OutOfMemoryError: unable to create new native thread > > > > 1. Memory Usage > > MemFree: 898332 kB > > From below core file generated during OMM, it can be seen about 900M > physical memory available during that time. > > > > > > 2 Thread number > > > > sh-4.1$ ps -eLf|wc -l > > 5326 > > > > > > sh-4.1$ ulimit -a > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 43497 > > max locked memory (kbytes, -l) 64 > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 10240 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 43497 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > Br, > > Tim > > > > > > > > From aph at redhat.com Tue Sep 19 13:13:10 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 19 Sep 2017 14:13:10 +0100 Subject: OpenJDK OOM issue - In-Reply-To: References: Message-ID: <99906d91-dea4-dd5f-2d35-f50f9f5264de@redhat.com> On 19/09/17 09:50, Yu, Tim (NSB - CN/Chengdu) wrote: > Hi OpenJDK dev group > > We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. What OS is this? -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From tim.yu at nokia-sbell.com Tue Sep 19 08:50:01 2017 From: tim.yu at nokia-sbell.com (Yu, Tim (NSB - CN/Chengdu)) Date: Tue, 19 Sep 2017 08:50:01 +0000 Subject: OpenJDK OOM issue - Message-ID: Hi OpenJDK dev group We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. " on Sep 18 11:05:04 EEST 2017 2 or first INFO log missing: Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create new native thread Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create new native thread 1. Memory Usage MemFree: 898332 kB >From below core file generated during OMM, it can be seen about 900M physical memory available during that time. 2 Thread number sh-4.1$ ps -eLf|wc -l 5326 sh-4.1$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 43497 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 43497 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Br, Tim From tim.yu at nokia-sbell.com Wed Sep 20 08:44:23 2017 From: tim.yu at nokia-sbell.com (Yu, Tim (NSB - CN/Chengdu)) Date: Wed, 20 Sep 2017 08:44:23 +0000 Subject: OpenJDK OOM issue - In-Reply-To: <99906d91-dea4-dd5f-2d35-f50f9f5264de@redhat.com> References: <99906d91-dea4-dd5f-2d35-f50f9f5264de@redhat.com> Message-ID: Hi All Thank you all for the quick response. The environment information is listed as below, could you please help to further check? 1. What OS is this? # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.9 (Santiago) # uname -a Linux cloudyvm16 2.6.32-696.6.3.el6.x86_64 #1 SMP Fri Jun 30 13:24:18 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux 2.GC log is listed as below. The heap information cannot be printed out in gc-2017_09_20-09_21_15.log when OOM happens. In gc-2017_09_20-09_21_17.log, you can see the heap begins with 0x0000000787380000 and it should be not the first 4G virtual memory address. -rw-r--r-- 1 19477 Sep 20 09:21 hs_err_pid12678.log -rw-r--r-- 1 570 Sep 20 09:21 gc-2017_09_20-09_21_15.log -rw-r--r-- 1 17741 Sep 20 09:21 hs_err_pid12706.log -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_17.log -rw-r--r-- 1 1722 Sep 20 09:21 gc-2017_09_20-09_21_18.log -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_19.log -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_20.log 3. This issue happens occasionally but frequently. We periodically launch a JAVA program to use JMX to monitor service status of another JAVA service. Br, Tim -----Original Message----- From: Andrew Haley [mailto:aph at redhat.com] Sent: Tuesday, September 19, 2017 9:13 PM To: Yu, Tim (NSB - CN/Chengdu) ; jdk8-dev at openjdk.java.net; jdk8u-dev at openjdk.java.net Cc: Shen, David (NSB - CN/Chengdu) Subject: Re: OpenJDK OOM issue - On 19/09/17 09:50, Yu, Tim (NSB - CN/Chengdu) wrote: > Hi OpenJDK dev group > > We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. What OS is this? -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From david.holmes at oracle.com Wed Sep 20 11:43:02 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 20 Sep 2017 21:43:02 +1000 Subject: OpenJDK OOM issue - In-Reply-To: References: <99906d91-dea4-dd5f-2d35-f50f9f5264de@redhat.com> Message-ID: Tim, Please note attachments get stripped from the mailing lists. All - please drop the jdk8-dev and jdk8u-dev mailing lists from this and leave it just on hotspot-dev. I've tried to bcc those lists. Thank you. David On 20/09/2017 6:44 PM, Yu, Tim (NSB - CN/Chengdu) wrote: > Hi All > > Thank you all for the quick response. > The environment information is listed as below, could you please help to further check? > > 1. What OS is this? > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 6.9 (Santiago) > # uname -a > Linux cloudyvm16 2.6.32-696.6.3.el6.x86_64 #1 SMP Fri Jun 30 13:24:18 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux > > 2.GC log is listed as below. The heap information cannot be printed out in gc-2017_09_20-09_21_15.log when OOM happens. In gc-2017_09_20-09_21_17.log, you can see the heap begins with 0x0000000787380000 and it should be not the first 4G virtual memory address. > -rw-r--r-- 1 19477 Sep 20 09:21 hs_err_pid12678.log > -rw-r--r-- 1 570 Sep 20 09:21 gc-2017_09_20-09_21_15.log > -rw-r--r-- 1 17741 Sep 20 09:21 hs_err_pid12706.log > -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_17.log > -rw-r--r-- 1 1722 Sep 20 09:21 gc-2017_09_20-09_21_18.log > -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_19.log > -rw-r--r-- 1 1297 Sep 20 09:21 gc-2017_09_20-09_21_20.log > > 3. This issue happens occasionally but frequently. We periodically launch a JAVA program to use JMX to monitor service status of another JAVA service. > > Br, > Tim > > > > > -----Original Message----- > From: Andrew Haley [mailto:aph at redhat.com] > Sent: Tuesday, September 19, 2017 9:13 PM > To: Yu, Tim (NSB - CN/Chengdu) ; jdk8-dev at openjdk.java.net; jdk8u-dev at openjdk.java.net > Cc: Shen, David (NSB - CN/Chengdu) > Subject: Re: OpenJDK OOM issue - > > On 19/09/17 09:50, Yu, Tim (NSB - CN/Chengdu) wrote: >> Hi OpenJDK dev group >> >> We meet one issue that the VM failed to initialize. The error log is as below. We checked both memory usage and thread number. They do not hit the limit. So could you please help to confirm why "java.lang.OutOfMemoryError: unable to create new native thread" error occurs? Many thanks. > > What OS is this? >