RFR: 8252657: JVMTI agent is not unloaded when Agent_OnAttach is failed

[Yasumasa Suenaga]

On Sat, 5 Sep 2020 14:26:17 GMT, Yasumasa Suenaga <ysuenaga at openjdk.org> wrote:

> If `Agent_OnAttach()` in JVMTI agent which is attempted to load via JVMTI.agent_load dcmd is failed, it would not be
> unloaded.   We've [discussed it on
> serviceability-dev](https://mail.openjdk.java.net/pipermail/serviceability-dev/2020-September/032839.html). This PR is
> a continuation of that.  This PR also includes to call `Agent_OnUnload()` when `Agent_OnAttach()` failed.
> 
> How to reproduce:
> 
> 1. Build JVMTI agent for test
> $ git clone https://github.com/YaSuenag/jvmti-examples.git
> $ cd jvmti-examples/helloworld/out/build
> $ cmake ../..
> 
> 2. Run JShell
> 
> 3. Load JVMTI agent via `jcmd <PID> JVMTI.agent_load` with "error" ("error" means `Agent_OnAttach()` returns JNI_ERR)
> $ jcmd
> 89456 jdk.jshell.execution.RemoteExecutionControl 45651
> 89547 sun.tools.jcmd.JCmd
> 89436 jdk.jshell/jdk.internal.jshell.tool.JShellToolProvider
> $ jcmd 89436 JVMTI.agent_load `pwd`/libhelloworld.so error
> 89436:
> return code: -1
> 
> 4. Check loaded libraries via `jcmd <PID> VM.dynlibs`
> $ jcmd 89436 VM.dynlibs | grep libhelloworld
> 7f2f8b06b000-7f2f8b06c000 r--p 00000000 fd:00 11818202
> /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06c000-7f2f8b06d000 r-xp 00001000
> fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06d000-7f2f8b06e000
> r--p 00002000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
> 7f2f8b06e000-7f2f8b06f000 r--p 00002000 fd:00 11818202
> /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06f000-7f2f8b070000 rw-p 00003000
> fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so

@edvbld Can you approve me to run tier1 tests with /test PR command again?

-------------

PR: https://git.openjdk.java.net/jdk/pull/19