RFR: 8315074: Possible null pointer access in native glass

[Johan Vos]

On Tue, 29 Aug 2023 12:33:53 GMT, Kevin Rushforth <kcr at openjdk.org> wrote:

>> I agree a crash due to a null pointer is not desired, as that gives very little info to the developer. 
>> If that malloc fails, it is an indication that there is a major chance that we are in serious trouble. In that case, simply printing something (which could fail as well if there is that limited memory) and not informing the caller will most likely just postpone the crash.
>> Unless we can free some memory immediately, I think it might be good if we can try to exit gracefully. The drawback of this is that *if* there is a trivial way to free memory and the native code was just about to invoke free() on a big memory chunk, we are exiting without a good reason (although I think this scenario is unlikely).
>
> The idea is to avoid the crash entirely. If we actually hit this case, it is very likely that other calls will also run out of memory. Returning to Java as quickly as possible will let any pending OOME be thrown. A library should not exit, so really we have two choices here:
> 
> 1. Throw OOM and then return
> 2. Just return
> 
> While option 1 might be the better choice, it would be a more intrusive fix. Most of the native code just returns to Java, although we do have a few places where we throw. OOME. It might be better to keep this fix simple (and more in line with what other functions in glass do), and address this with a follow-up issue?

I'm not against that, especially since it's in line with what we do in other functions in glass.
However, I am worried about the consequences. In case we just return, the caller has no idea that there is a major problem. A Runnable is supplied to e.g. _invokeAndWait, but it will never get executed while the caller (and the application logic) assumes it is scheduled. This can have serious consequences and unexpected behavior in the application.
But maybe I'm missing something and it is less severe than I'm picturing it?

-------------

PR Review Comment: https://git.openjdk.org/jfx/pull/1223#discussion_r1308778972