RFR: 8252657: JVMTI agent is not unloaded when Agent_OnAttach is failed

[David Holmes]

Looking at the original webrev from September:

https://cr.openjdk.java.net/~ysuenaga/JDK-8252657/webrev.00/src/hotspot/share/prims/jvmtiExport.cpp.cdiff.html

The suggestion to unload the library if Agent_OnAttach fails seems quite 
reasonable - it is what happens if no Agent_OnAttach function can be 
found in the agent.

But the specification is lacking because it simply states any such error 
is "ignored" - which is an oversimplification and I think was really 
intended to contrast with the abort behaviour if Agent_OnLoad fails. In 
addition the specification indicates that Agent_OnUnload is called any 
time the agent library is unloaded, except for "uncontrolled shutdown". 
This unfortunately suggests that before unloading the library after 
OnAttach fails, we also call OnUnload. That seems wrong and in fact the 
VM does not do that - and noone seems to have complained.

Also note the specification states an agent "must export a start-up 
function ..." but doesn't say what happens if it doesn't do so.

My gut feeling for what the specification should say here is that if the 
start-up function does not exist, or the call to Agent_OnAttach reports 
an error, then the agent library is immediately unloaded with no attempt 
to call the agent shutdown function (Agent_OnUnload).

But with my "compatibility glasses" on this may be too strong to 
retrofit to the specification as other VMs may behave differently. So as 
I stated in the CSR request we probably want to allow the current 
hotspot behaviour but not mandate it (unless we check with all the other 
VM implementations that such a specification is okay with them).

I agree that the more general issue of re-loading an agent is a separate 
issue.

David
-----

On 1/12/2020 3:21 pm, David Holmes wrote:
> On 1/12/2020 3:19 pm, David Holmes wrote:
>> On 1/12/2020 2:46 pm, Yasumasa Suenaga wrote:
>>> Hi Chris, David,
>>>
>>> Currently Agent_OnUnload() is not called when Agent_OnLoad() is 
>>> failed - JVM will abort.
>>> Should we re-think this behavior?
>>
>> We should we rethink that? It is probably one of the clearest parts of 
>> the spec. If Agent_Onload fails that is considered a fatal error - end 
>> of story.
> 
> I meant, of course, "Why should we rethink that?".
> 
> David
> 
>> The issue is with Agent_onAttach and how its failure should, or should 
>> not, impact Agent_OnUnload.
>>
>> David
>> -----
>>
>>> https://github.com/YaSuenag/jvmti-examples/tree/master/helloworld
>>>
>>> ```
>>> $ java -agentpath:/path/to/libhelloworld.so=error --version
>>> Hello World from Agent_OnLoad()
>>>    options = error
>>> Error occurred during initialization of VM
>>> agent library failed to init: /path/to/libhelloworld.so
>>> ```
>>>
>>>
>>> Thanks,
>>>
>>> Yasumasa
>>>
>>>
>>> On 2020/12/01 11:44, David Holmes wrote:
>>>> On 1/12/2020 11:45 am, Chris Plummer wrote:
>>>>> On Fri, 2 Oct 2020 07:27:43 GMT, Yasumasa Suenaga 
>>>>> <ysuenaga at openjdk.org> wrote:
>>>>>
>>>>>>> * Q3: What has to be done for statically linked agent?
>>>>>>
>>>>>> JVMTI spec says "unless it is statically linked into the 
>>>>>> executable", so I think we can ignore about Agent_OnUnload_L() in 
>>>>>> this PR.
>>>>>
>>>>> I don't think that makes sense. If you call it for dynamically 
>>>>> linked then you need to call it for statically linked also.
>>>>
>>>> Agreed. Even though you can't physically unload the statically 
>>>> linked library, if it is logically unloaded by some mechanism, then 
>>>> Agent_OnUnload_L is supposed to be called.
>>>>
>>>> David
>>>> -----
>>>>
>>>>> -------------
>>>>>
>>>>> PR: https://git.openjdk.java.net/jdk/pull/19
>>>>>