RFR: 8252657: JVMTI agent is not unloaded when Agent_OnAttach is failed [v3]

[Serguei Spitsyn]

On Thu, 14 Jan 2021 03:08:19 GMT, Yasumasa Suenaga <ysuenaga at openjdk.org> wrote:

>> If `Agent_OnAttach()` in JVMTI agent which is attempted to load via JVMTI.agent_load dcmd is failed, it would not be unloaded.  
>> We've [discussed it on serviceability-dev](https://mail.openjdk.java.net/pipermail/serviceability-dev/2020-September/032839.html). This PR is a continuation of that.
>> 
>> This PR also includes to call `Agent_OnUnload()` when `Agent_OnAttach()` failed.
>> 
>> How to reproduce:
>> 
>> 1. Build JVMTI agent for test
>> $ git clone https://github.com/YaSuenag/jvmti-examples.git
>> $ cd jvmti-examples/helloworld/out/build
>> $ cmake ../..
>> 
>> 2. Run JShell
>> 
>> 3. Load JVMTI agent via `jcmd <PID> JVMTI.agent_load` with "error" ("error" means `Agent_OnAttach()` returns JNI_ERR)
>> $ jcmd
>> 89456 jdk.jshell.execution.RemoteExecutionControl 45651
>> 89547 sun.tools.jcmd.JCmd
>> 89436 jdk.jshell/jdk.internal.jshell.tool.JShellToolProvider
>> $ jcmd 89436 JVMTI.agent_load `pwd`/libhelloworld.so error
>> 89436:
>> return code: -1
>> 
>> 4. Check loaded libraries via `jcmd <PID> VM.dynlibs`
>> $ jcmd 89436 VM.dynlibs | grep libhelloworld
>> 7f2f8b06b000-7f2f8b06c000 r--p 00000000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>> 7f2f8b06c000-7f2f8b06d000 r-xp 00001000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>> 7f2f8b06d000-7f2f8b06e000 r--p 00002000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>> 7f2f8b06e000-7f2f8b06f000 r--p 00002000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>> 7f2f8b06f000-7f2f8b070000 rw-p 00003000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>
> Yasumasa Suenaga has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits:
> 
>  - Update jvmti.xml to follow CSR
>  - Merge remote-tracking branch 'upstream/master' into JDK-8252657
>  - Update patch
>  - Merge remote-tracking branch 'upstream/master' into JDK-8252657
>  - revert
>  - JVMTI agent is not unloaded when Agent_OnAttach is failed

Hi Yasumasa,

Both the CSR and the fix look good.
Thank you for your patience with the process.

Thanks,
Serguei

-------------

Marked as reviewed by sspitsyn (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/19