RFR: JDK-8293313: NMT: Add "Fake OOM" mode to MallocLimit

[Thomas Stuefe]

On Fri, 2 Sep 2022 13:44:02 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> * still work in progress *
> 
> This RFE adds the optional ability for `-XX:MallocLimit` to, instead of exiting the VM with a fatal error, fake a native OOM on this and all future malloc calls.
> 
> The user controls this by appending the option "oom" to the malloc limit argument:
> 
> 
> -XX:MallocLimit=2g # will cause fatal error upon reaching limit
> -XX:MallocLimit=2g,oom # will cause malloc to return NULL, mimicking a real malloc OOM
> 
> 
> That makes it possible to test hotspot robustness in the face of real malloc OOMs, and is the prerequisite for removing the old `MallocMaxTestWords` machinery.

Not yet bot

-------------

PR: https://git.openjdk.org/jdk/pull/10144