RFR: 8315074: Possible null pointer access in native glass

[Kevin Rushforth]

On Tue, 29 Aug 2023 08:07:49 GMT, Johan Vos <jvos at openjdk.org> wrote:

>> That's a good question. Since this is a void method (thus there is no way to signal an error), the ideal thing would be to throw an `OutOfMemoryError` before returning, but if a malloc of this small size were to fail, we might not even be able to create the OOME. Not sure it's worth it in this case. What do you think?
>
> I agree a crash due to a null pointer is not desired, as that gives very little info to the developer. 
> If that malloc fails, it is an indication that there is a major chance that we are in serious trouble. In that case, simply printing something (which could fail as well if there is that limited memory) and not informing the caller will most likely just postpone the crash.
> Unless we can free some memory immediately, I think it might be good if we can try to exit gracefully. The drawback of this is that *if* there is a trivial way to free memory and the native code was just about to invoke free() on a big memory chunk, we are exiting without a good reason (although I think this scenario is unlikely).

The idea is to avoid the crash entirely. If we actually hit this case, it is very likely that other calls will also run out of memory. Returning to Java as quickly as possible will let any pending OOME be thrown. A library should not exit, so really we have two choices here:

1. Throw OOM and then return
2. Just return

While option 1 might be the better choice, it would be a more intrusive fix. Most of the native code just returns to Java, although we do have a few places where we throw. OOME. It might be better to keep this fix simple (and more in line with what other functions in glass do), and address this with a follow-up issue?

-------------

PR Review Comment: https://git.openjdk.org/jfx/pull/1223#discussion_r1308750346