RFR: 8306580: Propagate CDS dumping errors instead of directly exiting the VM [v3]

[Thomas Stuefe]

On Fri, 7 Jun 2024 03:01:21 GMT, Ioi Lam <iklam at openjdk.org> wrote:

>> Matias Saavedra Silva has updated the pull request incrementally with two additional commits since the last revision:
>> 
>>  - Thomas comments and cleanup
>>  - Added test for file errors
>
> src/hotspot/share/cds/metaspaceShared.cpp line 520:
> 
>> 518:   if (_builder.reserve_buffer() == nullptr) {
>> 519:     this->_failed = true;
>> 520:     return;
> 
> I think this should be removed, since reserve_buffer() will exit the JVM when OOM happens.
> 
> I did some experiments with `docket run --memory=1024m ...` and found that, at least on Linux, when a process runs over the memory limit, it just gets killed immediately by the OOM killer. There's no way to detect an OOM condition and handle it gracefully. So I think we can't (and don't need to) handle OOM errors.

Isn't this just reservation? Can fail if you run out of address space (unlikely) or if you run against VMA limit.
Even for commit, you will fail if you run against commit limit or VMA number limit. In all these cases, no real memory has been used yet, so no reason to wake the OOM killer.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19370#discussion_r1630635724