RFR: 8252657: JVMTI agent is not unloaded when Agent_OnAttach is failed

[Serguei Spitsyn]

On Tue, 8 Sep 2020 07:12:51 GMT, Yasumasa Suenaga <ysuenaga at openjdk.org> wrote:

>> If `Agent_OnAttach()` in JVMTI agent which is attempted to load via JVMTI.agent_load dcmd is failed, it would not be
>> unloaded.   We've [discussed it on
>> serviceability-dev](https://mail.openjdk.java.net/pipermail/serviceability-dev/2020-September/032839.html). This PR is
>> a continuation of that.  This PR also includes to call `Agent_OnUnload()` when `Agent_OnAttach()` failed.
>> 
>> How to reproduce:
>> 
>> 1. Build JVMTI agent for test
>> $ git clone https://github.com/YaSuenag/jvmti-examples.git
>> $ cd jvmti-examples/helloworld/out/build
>> $ cmake ../..
>> 
>> 2. Run JShell
>> 
>> 3. Load JVMTI agent via `jcmd <PID> JVMTI.agent_load` with "error" ("error" means `Agent_OnAttach()` returns JNI_ERR)
>> $ jcmd
>> 89456 jdk.jshell.execution.RemoteExecutionControl 45651
>> 89547 sun.tools.jcmd.JCmd
>> 89436 jdk.jshell/jdk.internal.jshell.tool.JShellToolProvider
>> $ jcmd 89436 JVMTI.agent_load `pwd`/libhelloworld.so error
>> 89436:
>> return code: -1
>> 
>> 4. Check loaded libraries via `jcmd <PID> VM.dynlibs`
>> $ jcmd 89436 VM.dynlibs | grep libhelloworld
>> 7f2f8b06b000-7f2f8b06c000 r--p 00000000 fd:00 11818202
>> /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06c000-7f2f8b06d000 r-xp 00001000
>> fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06d000-7f2f8b06e000
>> r--p 00002000 fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>> 7f2f8b06e000-7f2f8b06f000 r--p 00002000 fd:00 11818202
>> /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so 7f2f8b06f000-7f2f8b070000 rw-p 00003000
>> fd:00 11818202 /home/ysuenaga/github/jvmti-examples/helloworld/out/build/libhelloworld.so
>
> @edvbld Can you approve me to run tier1 tests with /test PR command again?

I was initially wrong by supporting this, and now I share David's concerns about unclear semantics of this. The
questions are:
 - Q1: Is it necessary to call the Agent_OnUnload()?
 - Q2: Would it be a JVMTI spec violation to call the Agent_OnAttach() multiple times? (It seems to be the case to me.)
 - Q3: What has to be done for statically linked agent?
 - Q4: Should the agent be correctly loadable in the first place? What were the reasons its loading to fail?

Yes, at least, a CSR is needed for this.

-------------

PR: https://git.openjdk.java.net/jdk/pull/19