RFR: 8331320: ClassFile API OutOfMemoryError with certain class files

[ExE Boss]

On Tue, 30 Apr 2024 18:18:30 GMT, Paul Sandoz <psandoz at openjdk.org> wrote:

>> Class files with specifically corrupted tableswitch or lookupswitch instructions in the bytecode cause OutOfMemoryError while parsing with Class-File API.
>> This patch performs additional checks to avoid OOME and adds relevant tests.
>> 
>> Please review.
>> 
>> Thank you,
>> Adam
>
> src/java.base/share/classes/jdk/internal/classfile/impl/AbstractInstruction.java line 320:
> 
>> 318:             int low = code.classReader.readInt(ap + 4);
>> 319:             int high = code.classReader.readInt(ap + 8);
>> 320:             if (high < low || high - low > code.codeLength >> 2) {
> 
> May be its also an opportunity to reduce duplication e.g., replace line 316 with a call to `afterPadding()`

`BoundTableSwitchInstruction​::afterPadding()` is an instance method, and `BoundTableSwitchInstruction​::size(…)` is a static method, so this would require further refactoring.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19024#discussion_r1585700103