RFR: 8269537: memset() is called after operator new [v2]

[Leo Korinth]

On Wed, 8 Sep 2021 11:34:21 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

>> src/hotspot/share/memory/allocation.hpp line 439:
>> 
>>> 437:   void* operator new(size_t size, const std::nothrow_t& nothrow_constant) throw() {
>>> 438:       address res = (address)resource_allocate_bytes(size, AllocFailStrategy::RETURN_NULL);
>>> 439:       DEBUG_ONLY(if (res != NULL) _thread_last_allocated = RESOURCE_AREA;)
>> 
>> Maybe we should also guard against the possibility of nested allocations, which may trash `_thread_last_allocated`?
>> 
>> 
>> #define PUSH_RESOURCE_OBJ_ALLOC_TYPE(t) \
>>   assert(_thread_last_allocated == STACK_OR_EMBEDDED, "must not be nested"); \
>>   DEBUG_ONLY(_thread_last_allocated = t); \
>> 
>> ...
>>   if (res != NULL) {
>>     PUSH_RESOURCE_OBJ_ALLOC_TYPE(RESOURCE_AREA);
>>   }
>> 
>> 
>> Similarly, the `ResourceObj` constructor should use a corresponding `POP_RESOURCE_OBJ_ALLOC_TYPE` macro.
>
> I added a `set_type` method that ensures that the `_thread_last_allocated` always transition over a `STACK_OR_EMBEDDED`. I did *not* create a PUSH/POP macro pair because i believe it would give the false impression that we are doing a stack operation. 
> 
> Other than that I also made `allocation_type` use a `u8` as backing type. I also removed the now unused `allocation_mask` and the now unimportant detail that `STACK_OR_EMBEDDED = 0`.

Hmm, u8 was not what I was thinking, I will change that to a uint8_t in the next update...

-------------

PR: https://git.openjdk.java.net/jdk/pull/5387