RFR (S): 8237775: Core file generation will silently fail if /cores directory has wrong permissions set

[gerard ziemski]

hi all,

Please review this enhancement, which adds explanation to the user when 
we can not dump core file (i.e. "The core files directory does not exist 
or it has no write permissions.") because either the core folder does 
not exist or the process has no write permission in that folder (i.e. 
default behavior on recent macOS). We already warn the users when 
"ulimit" is not set, so we are just trying to be consistent and cover 
all cases that can cause the core dump not to happen.

Even though we are adding more code to the signal handler here, it 
shouldn't affect it negatively. I couldn't measure (using Xcode 
profiler) any memory impact of the new code (i.e. fopen/close/remove 
APIs) Moreover we already use "fopen()" and "fclose()" when writing the 
hs_err log, so the only new API used is "remove()".

We could conceivably move this code to the startup and cache the answer 
to be later used in signal handler (it does impact the the startup by 
-0.6% for "hello world" test case), but I don't like the idea of adding 
more code to the startup, when it's not going to be used majority of the 
time.

Please notice that I had to move the "Default location:" part of the 
message around, so that it gets printed after the new "The core files 
directory does not exist or it has no write permissions." message. 
That's because some tests expect the core location immediately after 
"Default location:".


Before this change a usual message would look like this:

# No core dump will be written. Core dumps have been disabled. To enable 
core dumping, try "ulimit -c unlimited" before starting Java again

or

# Core dump will be written. Default location: /cores/core.8201


Now we can print:

# No core dump will be written. The core files directory does not exist 
or it has no write permissions. Core dumps have been disabled. To enable 
core dumping, try "ulimit -c unlimited" before starting Java again

or

# Core dump will be requested from the system. The core files directory 
does not exist or it has no write permissions. Default location: 
/cores/core.8171


bug: https://bugs.openjdk.java.net/browse/JDK-8237775
webrev: http://cr.openjdk.java.net/~gziemski/8237775_rev1
testing: passes Mach5 hs-tier1,2,3,4,5


cheers