RFR: JDK-8293313: NMT: Add "Fake OOM" mode to MallocLimit

[Thomas Stuefe]

This RFE adds the optional ability for `-XX:MallocLimit` to, instead of exiting the VM with a fatal error, fake a native OOM on this and all future malloc calls.

The user controls this by appending the option "oom" to the malloc limit argument:


-XX:MallocLimit=2g # will cause fatal error upon reaching limit
-XX:MallocLimit=2g,oom # will cause malloc to return NULL, mimicking a real malloc OOM


That makes it possible to test hotspot robustness in the face of real malloc OOMs, and is the prerequisite for removing the old `MallocMaxTestWords` machinery.

-------------

Commit messages:
 - NMT: Add "Fake OOM" mode to MallocLimit

Changes: https://git.openjdk.org/jdk/pull/10144/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10144&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8293313
  Stats: 299 lines in 9 files changed: 228 ins; 38 del; 33 mod
  Patch: https://git.openjdk.org/jdk/pull/10144.diff
  Fetch: git fetch https://git.openjdk.org/jdk pull/10144/head:pull/10144

PR: https://git.openjdk.org/jdk/pull/10144