RFR: 8364655: Loading class with nested annotations causes stack overflow in VM

[David Holmes]

On Fri, 5 Dec 2025 07:39:00 GMT, Johan Sjölen <jsjolen at openjdk.org> wrote:

> Hi,
> 
> `skip_annotation` and `skip_annotation_value` are two mutually recursive functions calling each other in order to skip over classfile annotations. If a classfile contains a highly nested annotation, then this will lead to a stack overflow and a subsequent crash of the JVM. I propose that we insert a recursion limit to prevent this from happening.
> 
> This recursion limit will make the annotation parsing to bail out on the JVM side, skipping the rest of the annotations present. This is fine! The reason that the JVM parses these annotations is to gain access to a select few JDK-internal annotations, and we trust our own code to construct classfiles without egregious nesting. All classfile annotations are also parsed by Java-code, so we're not skipping general annotation parsing.
> 
> A regression test has been added.
> 
> Thanks!

That seems reasonable. Do you want to add some logging to report the bail out?

EDIT: I hadn't seen @liach 's comment about JFR usage. That changes things somewhat.

-------------

Marked as reviewed by dholmes (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28674#pullrequestreview-3584763027