Question about vfprintf hook VM argument

[David Holmes]

Hi Thomas,

On 8/05/2017 7:29 PM, Thomas Stüfe wrote:
> Hi all,
>
> what exactly is the purpose of the FILE* argument in the vfprintf hook?

I see your point. :) The vfprint_hook is a replacement vfprintf function 
to be called from jio_vfprintf:

int jio_vfprintf(FILE* f, const char *fmt, va_list args) {
   if (Arguments::vfprintf_hook() != NULL) {
      return Arguments::vfprintf_hook()(f, fmt, args);
   } else {
     return vfprintf(f, fmt, args);
   }
}

so whatever gets passed to jio_vfprintf gets passed through to the hook.

But ...

> We had - actually several times already - the problem that our VM was
> embedded by a customized launcher which used the vfprintf hook to redirect
> the VM output. If the launcher uses the FILE* handed over by the VM to
> write to, it must be linked against the same C-Runtime as the VM itself.
> This is not necessarily a given, especially on Windows: the launcher may
> link against the debug C-Runtime (compiled with /MDd) wheras the JDK is
> build with "/MD" and links against the release C-Runtime. Or the launcher
> may even have been linked statically against the C-Runtime. Or...
>
> In my opinion it is not a good idea to hand over C-Runtime internals - be
> it malloced memory or FILE* pointers - to other binaries which may have
> been built with different build options. But I do not even understand the
> point of passing FILE* to the hook? If the point of the hook is to give
> embedding code the ability to write to somewhere else, why even bother
> giving it *my* file pointer?

... I confess I had no idea why this vfprint hook exists, but this 
somewhat explains it:

https://bugs.openjdk.java.net/browse/JDK-4015550

and yes it does suggest that although the FILE* is passed in the 
expectation is that the function will actually write somewhere else. 
IIUC the intent was to allow fd's 0,1 and 2 to be re-mapped by the hook 
to match whatever the embedded app had change System.out/err to. But as 
fd's were per-dll they couldn't pass through the fd so they passed 
through the FILE*. But how they expected that to be mapped to 
stdout/stderr I have no idea.

Cheers,
David

> Thanks & Kind Regards, Thomas
>