RFR: 8297138: UB leading to crash in Amalloc with optimized builds

[David Holmes]

On Wed, 23 Nov 2022 12:52:04 GMT, Afshin Zafari <duke at openjdk.org> wrote:

> ### Problem Description , from JBS:
> In Amalloc() we do this:
> 
> debug_only(if (UseMallocOnly) return malloc(x);) 
> so, if and only if DEBUG is on do we use malloc().
> 
> However, the matching free() in ResourceArea::rollback_to() does not have a matching debug_only guard:
> 
> if (UseMallocOnly) { 
>     free_malloced_objects ... 
> UseMallocOnly is a product flag. So, if PRODUCT is true, and DEBUG is true, we have a mismatched malloc() and free(). This is undefined behaviour.
> 
> I suggest we should remove the debug_only guard.
> 
> ### Patch:
> removed debug_only instances and #if ASSERT around Arena::malloc().
> 
> ### Test
> **Configure**: --with-debug-level=optimized
> **Test**: make test TEST=runtime/8007475
> **Mach5**: tier1 to tier5

I think we can close this PR in favour of a RFE that simply removes `UseMallocOnly`. It may have been useful whilst VM memory management was in its infancy (it was introduced in 1998) but now it is just a liability. I think we can simply remove it (and not worry if Arenas need better debuggability or error tracking - that would be a separate RFE).

-------------

PR: https://git.openjdk.org/jdk/pull/11320