RFR: 8365203: defineClass with direct buffer can cause use-after-free

[Jaikiran Pai]

On Mon, 11 Aug 2025 12:35:07 GMT, Per Minborg <pminborg at openjdk.org> wrote:

>> ### Description
>> This PR proposes to update the `ClassLoader` implementation to properly guard access to the provided `ByteBuffer` when defining a class using `defineClass(String, ByteBuffer, ...)`. Specifically, calls to `SharedSecrets.getJavaNioAccess().acquireSession(ByteBuffer)` and `releaseSession(ByteBuffer)` have been introduced to ensure safe and consistent buffer access throughout the native class definition process, even in the case of a `ByteBuffer` is backed by a `MemorySegment`.
>> 
>> ### Impact
>> This modification is internal to the `ClassLoader` implementation and does not affect the public API.
>> Improves the robustness and security of class loading from buffers.
>> 
>> ### Testing
>> Tier 1, 2, and 3 JDK tests pass on multiple platforms.
>
> test/jdk/java/lang/ClassLoader/defineClass/GuardByteBuffer.java line 43:
> 
>> 41: 
>> 42:     @Test
>> 43:     void guardCrash() throws InterruptedException {
> 
> I was not able to reproduce the crash using this test on a Mac. The original reproducer worked on a Windows machine.

Hello Per, I too couldn't reproduce the crash from the original reproducer (and this test) on macos and I found that a bit odd. I read up a bit about macos memory debugging tools and it turns out macos has a "Guard Malloc" implementation https://developer.apple.com/library/archive/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html which can be optionally enabled to debug issues like these. `man libgmalloc` has additional details about it.

So I ran the original reproducer again, this time with:
 ```
export DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
java ... <that-reproducer>


and that consistently reproduces the crash on macos. I will build this PR locally and give it a try soon to make sure the crash no longer reproduces.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26724#discussion_r2293850857