RFR: 8311500: StackWalker.getCallerClass() throws UOE if invoked reflectively [v3]

[Mandy Chung]

On Tue, 11 Jul 2023 15:47:35 GMT, Volker Simonis <simonis at openjdk.org> wrote:

>> As the included jtreg test demonstrates, `StackWalker.getCallerClass()` can throw an `UnsupportedOperationException` if called reflectively. Currently this only happens if we invoke `StackWalker.getCallerClass()` recursively reflectively, but this issue will become more prominent once we fix [JDK-8285447](https://bugs.openjdk.org/browse/JDK-8285447). The gory details follow below:
>> 
>> The protocol between the Java API and the JVM for `StackWalker.getCallerClass()/walk()` is as follows:
>> - On the Java side, `StackWalker` calls into `StackStreamFactory` for the real work.
>> - For `StackWalker.getCallerClass()` `StackStreamFactory` basically creates a `Class[]` which will be passed down and filled in the JVM. For `StackWalker.walk()` it will normally be a `StackFrameInfo[]` (or a `LiveStackFrameInfo[]` if the internal `ExtendedOption.LOCALS_AND_OPERANDS` option was used).
>> - The default size of this arrays is currently `StackStreamFactory.SMALL_BATCH` which is 8 (but see [JDK-8285447](https://bugs.openjdk.org/browse/JDK-8285447)).
>> - `StackStreamFactory` than calls `AbstractStackWalker.callStackWalk()` which is a natively implemented in the VM by `JVM_CallStackWalk()`.
>> -  `JVM_CallStackWalk()` calls `StackWalk::walk()` which calls `StackWalk::fetchFirstBatch()` which calls `StackWalk::fill_in_frames()` which walks the stack and fills in the available class/stackframe slots in the passed in array until the array is full or there are no more stack frames,
>> - Once  `StackWalk::fill_in_frames()` returns, `StackWalk::fetchFirstBatch()` calls back to Java by invoking `AbstractStackWalker::doStackWalk()` to consume the result.
>> - `AbstractStackWalker::doStackWalk()` calls `consumeFrames()` (which is overridden depending on whether we initially called `getCallerClass()` or `walk()`) which consumes the frames until it either finishes (e.g. finds the caller class) or until there are no more frames.
>> - In the latter case `consumeFrames()` will call into the the VM again by calling `AbstractStackWalker.fetchStackFrames()` to fetch additional frames from the stack.
>> - `AbstractStackWalker.fetchStackFrames()` is implemented by `JVM_MoreStackWalk()` which calls `StackWalk::fetchNextBatch()` which calls `StackWalk::fill_in_frames()` (the same method that already fetched the initial batch of frames).
>> 
>> Following is a stacktrace of what I've explained so far:
>> 
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>> V  [libjvm.so+0x1...
>
> Volker Simonis has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fixed case when calling getCallerClass() from a @CallerSensitive method reflectively

I expect `getCallerClass` is rarely called via reflection.  JDK-8210375 was reported 5 years ago but no customer reports this issue since then.    This bug is not critical to be fixed immediately and so we should look into a proper solution.

I do have a question if filtering in the VM is the right thing to do.   We need to consider the alternatives and understand why this solution is chosen.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14773#issuecomment-1636082537