RFR: 8208686: [AOT] JVMTI ResourceExhausted event repeated for same allocation

[Vladimir Kozlov]

To let you know, me and Tom R. did review these changes and agreed that it is the least intrusive 
changes for Hotspot shared code.

Thanks,
Vladimir

On 9/25/18 8:11 AM, Daniel D. Daugherty wrote:
> Adding serviceability-dev at ... since this is JVM/TI...
> 
> Dan
> 
> 
> On 9/25/18 10:48 AM, Doug Simon wrote:
>> A major design point of Graal is to treat allocations as non-side effecting to give more freedom 
>> to the optimizer by reducing the number of distinct FrameStates that need to be managed. When 
>> failing an allocation, Graal will deoptimize to the last side effecting instruction before the 
>> allocation. This mean the VM code for heap allocation will potentially be executed twice, once 
>> from Graal compiled code and then again in the interpreter. While this is perfectly fine according 
>> to the JVM specification, it can cause confusing behavior for JVMTI based tools. They will receive 
>> 2 ResourceExhausted events for a single allocation. Furthermore, the first ResourceExhausted event 
>> (on the Graal allocation slow path) might denote a bytecode instruction that performs no 
>> allocation, making it hard to debug the memory failure.
>>
>> The proposed solution is to add an extra set of JVMCI VM runtime calls for allocation. These entry 
>> points will attempt the allocation and upon failure,
>> skip side-effects such as posting JVMTI events or handling -XX:OnOutOfMemoryError. The compiled 
>> code using these entry points is expected deoptmize on null.
>>
>> The path from these new entry points to where allocation can fail goes through quite a bit of VM 
>> code. One could modify all these paths by:
>> * Returning null instead of throwing an exception on failure.
>> * Adding a `bool null_on_fail` argument to all relevant methods.
>> * Adding extra null checking where necessary after each call to these methods when `null_on_fail 
>> == true`.
>> This represents a significant number of changes.
>>
>> Instead, the proposed solution introduces a new _in_retryable_allocation thread-local. This way, 
>> only the entry points and allocation routines that raise an exception need to be modified. Failure 
>> is communicated back to the new entry points by throwing a special pre-allocated OOME object 
>> (i.e., Universe::out_of_memory_error_retry()) which must not propagate back to Java code. Use of 
>> this object is not strictly necessary; it is introduced to highlight/document the special 
>> allocation mode.
>>
>> The proposed solution is at http://cr.openjdk.java.net/~dnsimon/8208686.
>> THE JBS bug is: https://bugs.openjdk.java.net/browse/JDK-8208686
>>
>> -Doug
>