RFR: JDK-8318445: More broken bailout chains in C2

[Thomas Stuefe]

On Fri, 20 Oct 2023 05:50:31 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

> @tstuefe Thanks for the fix. Yes, it seems we should catch failure after any igvn phase.
> 
> What exactly causes the crash? I guess we have somehow a messed up graph, want to bail out but then do not bail out but instead go back into some optimization, and then encounter the broken graph?

Yes. I am currently implementing a way to abort compilations if the compiler uses too much memory. https://github.com/openjdk/jdk/pull/16220 . For C2, I piggyback on the node limit check. But there, I found that C2 is often not heeding the bailout: some caller up the stack ignores it and goes on. And because Compiler::record_failure sets Compile::_root to nullptr, we get errors and crashes.

Even though this is tied to my memory limiter, this can also happen when hitting the node check. Therefore I treat these errors as separate bugs.

BTW I wrote https://github.com/openjdk/jdk/pull/16247, which helps analyzing these kinds of errors. If you like to review :)

> 
> I guess we could catch this with some sort of stress testing: randomly decide to mess up the graph and bail out. Then we will see if every place really processes the bailout correctly. @vnkozlov what do you think?

I am currently doing exactly that: let each compilation run with a random lowish memory limit. That way I shake up a lot of these problems. I plan to incorporate this into the final memory limit PR, so that can used to randomly bail out of compilations (The advantage is that this also tests C1, but I found C1 to be less of a problem than C2).

> 
> @vnkozlov should we file any RFE's?
> 
> Do you have a reproducer for this? I see none here and none on JBS.

No, and it is difficult to reproduce.

I am currently experimenting with better ways to do this in code, and maybe to semi-automate the process of bailing out. Bailing out is like an exception path, or like we do in the VM runtime need to handle exception_pending.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16248#issuecomment-1772173762