[13] RFR(S): 8226536: Catch OOM from deopt that fails rematerializing objects

[Vladimir Ivanov]

test/hotspot/jtreg/vmTestbase/nsk/share/gc/gp/GarbageUtils.java:

+          * It is Important that the impl is not inlined.

What do you think about putting @DontInline on eatMemoryImpl to 
guarantee that invariant?

Best regards,
Vladimir Ivanov

On 04/07/2019 16:27, Nils Eliasson wrote:
> Hi,
> 
> This is a patch of GarbageUtils.java used by a bunch of GC tests.
> 
> This issue was diagnosed by Stefan Karlsson and passed to the compiler 
> team.
> 
> The core problem is that the test relies on successfully catching OOM 
> errors. But the OOM causes a deopt, and the deopt must rematerialize 
> some objects before passing the execution on to the interpreter to run 
> the catch-block. If the rematerialization can't allocate, it will pass 
> the OOM on, without having run the catch-block.
> 
> With this patch I fix the tests by wrapping the faulting code with an 
> extra layer of try-catch-OOM. It relies on not inlining the inner 
> method, because then it would be part of the compiled frame being 
> deoptimized.
> 
> In general this is a problem for any kind of deopt when we are out of 
> heap. Another bug will be opened to keep tracking this issue.
> 
> Webrev:http://cr.openjdk.java.net/~neliasso/8226536/webrev.02/ 
> <http://cr.openjdk.java.net/~neliasso/8226536/webrev.02/>
> 
> Bug: https://bugs.openjdk.java.net/browse/JDK-8226536
> 
> // Nils
>