MethodHandle initialization process - problem with JVM TI early VM start event

[Alan Bateman]

On 04/01/2017 03:11, David Holmes wrote:

> We have encountered a crash in a JVM TI test, starting with b148, that 
> is caused by the attempted use of MethodRefs and invoke_dynamic within 
> class loading that occurs from the "early VM start" JVM TI event 
> callback - which happens before the VM and core classes are fully 
> initialized. The class being loaded/initialized is in java.base and so 
> this is allowed. The class, java.net.Authenticator, was changed in 
> b148 to have static initialization involving method refs and hence the 
> problem was introduced.
>
> I've included a stack dump below.
>
> It is far from clear to me where responsibility for this lies, or even 
> how to narrow that down. Is it in the code of MethodHandleNatives? Or 
> is it in the VM linking/resolution code?
>
> Thoughts appreciated. :)
Ugh, this looks like the callback for the VMStart event is triggering 
the execution of arbitrary code via its initializer. When the 
can_generate_early_vmstart capability is enabled then the VMStart event 
is posted before the system classes are initialized (before 
Threads::initialize_java_lang_classes is called). If the callbacks uses 
JNI FindClass to load classes in java.base (it's restricted to java.base 
in this phase) then it will trigger the execution of random bytecode 
with all sort of consequences. Even if the system classes are 
initialized then it can still cause all sorts of issues - e.g. 
triggering code in classes such as ResourceBundle and elsewhere that 
depend on the system class loader to be initialized. For the specific 
crash then it looks like MethodHandleNatives has asserts and so 
Class::desiredAssertionStatus will be invoked to test if assertions are 
enabled but, as I say, there are lots of other reasons that would cause 
random classes in java.base to fail when invoked before the system 
classes are initialized.

The reason this event is called so early is to allow native (not java) 
agents get other events during startup. Also it's not new that this 
event is posted before the system classes are initialized, you'll see 
the same with JDK 8 and older because VMStart was always posted very 
early. In JDK 9 then the VMStart is only posted early when the 
can_generate_early_vmstart capability is enabled. This chage.

As regards fixing this then I don't think it would be unreasonable to 
put some restrictions on what can be done in the VMStart callback when 
can_generate_early_vmstart capability is enabled. Maybe it could be just 
further re-wording of the paragraph on what can be done in this phase 
rather than trying to enforce it.

-Alan