RFR: 8364655: Loading class with nested annotations causes stack overflow in VM

[David Holmes]

On Fri, 5 Dec 2025 07:39:00 GMT, Johan Sjölen <jsjolen at openjdk.org> wrote:

> Hi,
> 
> `skip_annotation` and `skip_annotation_value` are two mutually recursive functions calling each other in order to skip over classfile annotations. If a classfile contains a highly nested annotation, then this will lead to a stack overflow and a subsequent crash of the JVM. I propose that we insert a recursion limit to prevent this from happening.
> 
> This recursion limit will make the annotation parsing to bail out on the JVM side, skipping the rest of the annotations present. This is fine! The reason that the JVM parses these annotations is to gain access to a select few JDK-internal annotations, and we trust our own code to construct classfiles without egregious nesting. All classfile annotations are also parsed by Java-code, so we're not skipping general annotation parsing.
> 
> A regression test has been added.
> 
> Thanks!

test/hotspot/jtreg/runtime/ClassFile/NestedAnnotations.java line 64:

> 62: 
> 63:         cw.visitEnd();
> 64:         // Does not matter whether the class is hidden, used for simplicity’s sake.

Check the actual character used for the apostrophe here. When I grabbed the raw file it failed to compile: `unmappable character (0x92) for encoding UTF-8`

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28674#discussion_r2625475271