RFR: 8296007: crash in runtime/DefineClass/NullClassBytesTest.java

[Dean Long]

On Thu, 27 Oct 2022 21:16:07 GMT, Daniel D. Daugherty <dcubed at openjdk.org> wrote:

> Thumbs up.

Thanks, Dan, for the review.

> So did the callers of the function just assume there was a return value?

Yes, the test creates its own ClassLoader, so the value is needed as part of class loading.

> Why didn't this result in a linkage warning or some other type of warning?

I don't think there is any way to automatically detect this.  The return type of a native method is not part of the symbol name for exported C functions, as far as I know.  We could probably check that the result is a valid oop for native methods that return objects.  That could be something extra that -Xcheck:jni does, perhaps.

How this went undetected is because the native method either throws an exception, so the result is ignored, or it calls DefineClass(), which sets the result register to the right value.  This is the last line of the function, so there is a good chance that the result register will have the right value when it returns.  However I was running a debug build and the function epilogue that gcc generated modified the result register to perform a stack corruption check.

-------------

PR: https://git.openjdk.org/jdk/pull/10892