RFR [XS]: 8234968: check calloc rv in libinstrument InvocationAdapter

[Alan Bateman]

On 29/11/2019 06:29, Thomas Stüfe wrote:
> Hi Matthias,
>
> I am not certain the callers are prepared to handle NULL.
>
> This is used in a chain of TRANSFORM macro calls which AFAICS do not 
> handle NULL; e.g. , at 872, we pass the returned pointer to 
> convertUft8ToPlatformString which passes it on (on Windows) to 
> MultiByteToWideChar, which does not handle NULL input.
>
> So I wonder whether a clear error message with an exit would be better 
> in this case. Otherwise we may get a crash just some instructions later.
>
Right, this needs a lot more analysis to see if it's even possible to 
continue. The main usage is VM startup where the -javaagent option 
specifies agents that have the Boot-Class-Path attribute. In that case 
it would not be unreasonable to abort the process, it's unlikely to get 
much startup in the startup if memory is exhausted. The other possible 
context is where a tool agent is loaded into a running VM, in which case 
have the attach thread return with a pending exception might be okay 
(although the VM is likely to shutdown anyway as the memory exhaustion 
will be detected/handled elsewhere).

-Alan