RFR: 8368551: Core dump warning may be confusing

[Paul Hübner]

Hi all,

The `os::check_core_dump_prerequisites` function is used to build the strings of 1) warnings if `CreateCoredumpOnCrash` is specified explicitly and some prerequisite is not fulfilled (fully), and 2) the location of the dump when the VM aborts on error.

This is fine for many cases, but there are some edgecases on POSIX systems where the warning makes little sense. For example, as reported in the issue:
> OpenJDK 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but /shared/cores/core-%e-585191 (max size 0 k). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again

This PR refactors overhauls some of these edge cases to ensure that the warning messages and core locations are more clear and user-friendly.

**Scenario 1: `CreateCoredumpOnCrash` is disabled**
> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # CreateCoredumpOnCrash turned off, no core file dumped

**Scenario 2: `get_core_path` errors**
> Java HotSpot(TM) 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but core path is unknown, trying core.1552174 (may not exist)
> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # Core dump will be written. Default location: core.1552174 (may not exist)

**Scenario 3: the user processes core dump with e.g. `systemd-coredump`**
> Java HotSpot(TM) 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %e" (or dumping to /home/opc/betterCoreMsg/core.1555400)
> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # Core dump will be written. Default location: "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %e" (or dumping to /home/foo/bar/core.1555400)

**Scenario 4: the resource limit for core dumps could not be determined**
> Java HotSpot(TM) 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but couldn't deduce rlim, trying /tmp/cores/core.%e.1560382.%h.%t (may not exist)
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #  [...]
> # Core dump will be written. Default location: /tmp/cores/core.%e.1560382.%h.%t (may not exist)

**Scenarios 5, 6, 7: ulimit of 0, 1024, and unlimited**
> Java HotSpot(TM) 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

> Java HotSpot(TM) 64-Bit Server VM warning: CreateCoredumpOnCrash specified, but core dumps are constrained (max size 1024 k). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # Core dump will be written. Default location: /tmp/cores/core.%e.1562878.%h.%t (max size 1024 k). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again

ℹ️ ^^^ this is the message reported to be misleading in the original issue.

> #
> # A fatal error has been detected by the Java Runtime Environment:
> # [...]
> # Core dump will be written. Default location: /tmp/cores/core.%e.1562952.%h.%t

Testing: tiers 1-5 on Linux (x64, AArch64), macOS (x64, AArch64), Windows (x64).

-------------

Commit messages:
 - Better posix coredump messages.

Changes: https://git.openjdk.org/jdk/pull/28174/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=28174&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8368551
  Stats: 30 lines in 2 files changed: 18 ins; 0 del; 12 mod
  Patch: https://git.openjdk.org/jdk/pull/28174.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/28174/head:pull/28174

PR: https://git.openjdk.org/jdk/pull/28174