RFR: 8364655: Loading class with nested annotations causes stack overflow in VM

[Chen Liang]

On Fri, 5 Dec 2025 07:39:00 GMT, Johan Sjölen <jsjolen at openjdk.org> wrote:

> Hi,
> 
> `skip_annotation` and `skip_annotation_value` are two mutually recursive functions calling each other in order to skip over classfile annotations. If a classfile contains a highly nested annotation, then this will lead to a stack overflow and a subsequent crash of the JVM. I propose that we insert a recursion limit to prevent this from happening.
> 
> This recursion limit will make the annotation parsing to bail out on the JVM side, skipping the rest of the annotations present. This is fine! The reason that the JVM parses these annotations is to gain access to a select few JDK-internal annotations, and we trust our own code to construct classfiles without egregious nesting. All classfile annotations are also parsed by Java-code, so we're not skipping general annotation parsing.
> 
> A regression test has been added.
> 
> Thanks!

For context, the JFR feature that uses this was added in https://bugs.openjdk.org/browse/JDK-8211238, which is so far also the only use. It may be possible for JFR to wait for early VM bootstrap and defer annotation parsing to Java core library.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28674#issuecomment-3662519931