RFR: 8331320: ClassFile API OutOfMemoryError with certain class files

[Paul Sandoz]

On Tue, 30 Apr 2024 15:31:02 GMT, Adam Sotona <asotona at openjdk.org> wrote:

> Class files with specifically corrupted tableswitch or lookupswitch instructions in the bytecode cause OutOfMemoryError while parsing with Class-File API.
> This patch performs additional checks to avoid OOME and adds relevant tests.
> 
> Please review.
> 
> Thank you,
> Adam

src/java.base/share/classes/jdk/internal/classfile/impl/AbstractInstruction.java line 320:

> 318:             int low = code.classReader.readInt(ap + 4);
> 319:             int high = code.classReader.readInt(ap + 8);
> 320:             if (high < low || high - low > code.codeLength >> 2) {

May be its also an opportunity to reduce duplication e.g., replace line 316 with a call to `afterPadding()`

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19024#discussion_r1585300727