RFR: 8297138: UB leading to crash in Amalloc with optimized builds

[Afshin Zafari]

On Wed, 23 Nov 2022 12:52:04 GMT, Afshin Zafari <duke at openjdk.org> wrote:

> 8297138: UB leading to crash in Amalloc with optimized builds

Problem Description , from JBS:
In Amalloc() we do this: 

    debug_only(if (UseMallocOnly) return malloc(x);) 

so, if and only if DEBUG is on do we use malloc(). 

However, the matching free() in ResourceArea::rollback_to() does not have a matching debug_only guard: 

    if (UseMallocOnly) { 
        free_malloced_objects ... 

UseMallocOnly is a product flag. So, if PRODUCT is true, and DEBUG is true, we have a mismatched malloc() and free(). This is undefined behaviour. 

I suggest we should remove the debug_only guard. 


Patch:
removed `debug_only` instances and `#if ASSERT` around `Arena::malloc().`

I have no idea about other possible patches. Please, let me know specifically what should I do?

-------------

PR: https://git.openjdk.org/jdk/pull/11320