RFR: 8311500: StackWalker.getCallerClass() throws UOE if invoked reflectively [v3]

[Volker Simonis]

On Tue, 11 Jul 2023 15:47:35 GMT, Volker Simonis <simonis at openjdk.org> wrote:

>> As the included jtreg test demonstrates, `StackWalker.getCallerClass()` can throw an `UnsupportedOperationException` if called reflectively. Currently this only happens if we invoke `StackWalker.getCallerClass()` recursively reflectively, but this issue will become more prominent once we fix [JDK-8285447](https://bugs.openjdk.org/browse/JDK-8285447). The gory details follow below:
>> 
>> The protocol between the Java API and the JVM for `StackWalker.getCallerClass()/walk()` is as follows:
>> - On the Java side, `StackWalker` calls into `StackStreamFactory` for the real work.
>> - For `StackWalker.getCallerClass()` `StackStreamFactory` basically creates a `Class[]` which will be passed down and filled in the JVM. For `StackWalker.walk()` it will normally be a `StackFrameInfo[]` (or a `LiveStackFrameInfo[]` if the internal `ExtendedOption.LOCALS_AND_OPERANDS` option was used).
>> - The default size of this arrays is currently `StackStreamFactory.SMALL_BATCH` which is 8 (but see [JDK-8285447](https://bugs.openjdk.org/browse/JDK-8285447)).
>> - `StackStreamFactory` than calls `AbstractStackWalker.callStackWalk()` which is a natively implemented in the VM by `JVM_CallStackWalk()`.
>> -  `JVM_CallStackWalk()` calls `StackWalk::walk()` which calls `StackWalk::fetchFirstBatch()` which calls `StackWalk::fill_in_frames()` which walks the stack and fills in the available class/stackframe slots in the passed in array until the array is full or there are no more stack frames,
>> - Once  `StackWalk::fill_in_frames()` returns, `StackWalk::fetchFirstBatch()` calls back to Java by invoking `AbstractStackWalker::doStackWalk()` to consume the result.
>> - `AbstractStackWalker::doStackWalk()` calls `consumeFrames()` (which is overridden depending on whether we initially called `getCallerClass()` or `walk()`) which consumes the frames until it either finishes (e.g. finds the caller class) or until there are no more frames.
>> - In the latter case `consumeFrames()` will call into the the VM again by calling `AbstractStackWalker.fetchStackFrames()` to fetch additional frames from the stack.
>> - `AbstractStackWalker.fetchStackFrames()` is implemented by `JVM_MoreStackWalk()` which calls `StackWalk::fetchNextBatch()` which calls `StackWalk::fill_in_frames()` (the same method that already fetched the initial batch of frames).
>> 
>> Following is a stacktrace of what I've explained so far:
>> 
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>> V  [libjvm.so+0x1...
>
> Volker Simonis has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fixed case when calling getCallerClass() from a @CallerSensitive method reflectively

We actually have two problems here:
1. If called refelctively, `getCallerClass()` can throw an UOE even if it was **not** called from a `@CallerSensitive` method (see inital test case).
2. If called reflectively from a  `@CallerSensitive` method, `getCallerClass()` will currently not throw a UOE as expected (see extended test in this PR).

`getCallerClass()` **is** performance sensitive and we want to improve its performance rather than slow it down (see [JDK-8285447](https://bugs.openjdk.org/browse/JDK-8285447)). I think performance-wise it would be better to do all the filtering in the VM, where we have all the required information at hand and minimize the amount of data that needs to be passed between Java and the VM.

Before starting to implement a new version of the fix which moves all the checks to Java, I'd like to hear some more opinions about whether we agree to move all the filtering and checks from the VM to Java (even at the cost of performance regressions) or if we better want to go the other way and do all the filtering/checks in the JVM. @shipilev, @dholmes-ora, @dfuch, @AlanBateman - what do you think?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14773#issuecomment-1637900903