RFR: 8365203: defineClass with direct buffer can cause use-after-free

[Jaikiran Pai]

On Mon, 11 Aug 2025 11:33:19 GMT, Per Minborg <pminborg at openjdk.org> wrote:

> ### Description
> This PR proposes to update the `ClassLoader` implementation to properly guard access to the provided `ByteBuffer` when defining a class using `defineClass(String, ByteBuffer, ...)`. Specifically, calls to `SharedSecrets.getJavaNioAccess().acquireSession(ByteBuffer)` and `releaseSession(ByteBuffer)` have been introduced to ensure safe and consistent buffer access throughout the native class definition process, even in the case of a `ByteBuffer` is backed by a `MemorySegment`.
> 
> ### Impact
> This modification is internal to the `ClassLoader` implementation and does not affect the public API.
> Improves the robustness and security of class loading from buffers.
> 
> ### Testing
> Tier 1, 2, and 3 JDK tests pass on multiple platforms.

I think for the new test introduced, I don't think we should test for a crash. Instead maybe we can simplify it as follows to assert that a closed `Arena` doesn't allow a `Class` to be defined from the `ByteBuffer`. I think that will simplify the test and also make it more deterministic. This is what I had in mind:


/*
 * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 *
 * This code is free software; you can redistribute it and/or modify it
 * under the terms of the GNU General Public License version 2 only, as
 * published by the Free Software Foundation.
 *
 * This code is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
 * version 2 for more details (a copy is included in the LICENSE file that
 * accompanied this code).
 *
 * You should have received a copy of the GNU General Public License version
 * 2 along with this work; if not, write to the Free Software Foundation,
 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
 * or visit www.oracle.com if you need additional information or have any
 * questions.
 */

/*
 * @test
 * @bug 8365203
 * @summary Tests guarding of ByteBuffers in ClassLoader::defineClass
 * @run junit GuardByteBuffer
 */

import java.lang.foreign.Arena;
import java.util.HexFormat;

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertThrows;

final class GuardByteBuffer {
    @Test
    void guardCrash() {
        final byte[] classBytes = getClassBytes(); // get bytes of a valid class
        final var cl = new ClassLoader() {
            void tryCrash() {
                var arena = Arena.ofConfined();
                var byteBuffer = arena.allocate(classBytes.length).asByteBuffer();
                // Close the arena underneath
                arena.close();
                // expected to always fail because the arena
                // from which the ByteBuffer was constructed
                // has been closed
                assertThrows(IllegalStateException.class,
                        () -> defineClass(null, byteBuffer, null));
            }
        };
        for (int i = 0; i < 10000; i++) {
            cl.tryCrash();
        }
    }

    private static byte[] getClassBytes() {
        // unused. this is here just for reference
        final String source = """
                    public class NoOp {}
                """;
        // (externally) compiled content of the above "source", represented as hex
        final String classBytesHex = """
                cafebabe00000044000d0a000200030700040c000500060100106a6176612f
                6c616e672f4f626a6563740100063c696e69743e0100032829560700080100
                044e6f4f70010004436f646501000f4c696e654e756d6265725461626c6501
                000a536f7572636546696c650100094e6f4f702e6a61766100210007000200
                0000000001000100050006000100090000001d00010001000000052ab70001
                b100000001000a000000060001000000010001000b00000002000c
                """;

        return HexFormat.of().parseHex(classBytesHex.replaceAll("\n", ""));
    }

}

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26724#issuecomment-3214529831