RFR: 8334167: Test java/lang/instrument/NativeMethodPrefixApp.java timed out

[Jaikiran Pai]

Can I please get a review of this test-only change which proposes to fix the test timeout reported in https://bugs.openjdk.org/browse/JDK-8334167?

The JBS issue has comments which explains what causes the timeout. The commit in this PR addresses those issues by updating the test specific `ClassFileTransformer` to only instrument application specific class instead of all (core) classes. The test was introduced several years back to verify the feature introduced in https://bugs.openjdk.org/browse/JDK-6263319. As such, the proposed changes in this PR continue to test that feature - we now merely use application specific class' native method to verify the semantics of that feature.

Additional cleanups have been done in the test to make sure that if any exception does occur in the ClassFileTransformer then it does get recorded and that then causes the test to fail.

With this change, I have run tier1 through tier6 and the test passes. Additionally, without this change I've run the test with a test repeat of 100 with virtual threads enabled and the test hangs occasionally (as expected). With this proposed fix, I have then run the test with virtual threads, around 300 times and it hasn't failed or hung in any of those instances.

-------------

Commit messages:
 - 8334167: Test java/lang/instrument/NativeMethodPrefixApp.java timed out

Changes: https://git.openjdk.org/jdk/pull/20154/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=20154&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8334167
  Stats: 152 lines in 3 files changed: 74 ins; 37 del; 41 mod
  Patch: https://git.openjdk.org/jdk/pull/20154.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/20154/head:pull/20154

PR: https://git.openjdk.org/jdk/pull/20154