RFR (XS): 8023037 : Race between ciEnv::register_method and nmethod::make_not_entrant_or_zombie

[Vladimir Ivanov]

http://cr.openjdk.java.net/~vlivanov/8023037/webrev.00/

There's a race between compiler thread during method registration and 
sweeper: sweeper can invalidate a nmethod which hasn't been installed yet.

Consider the following scenario:
   ciEnv::register_method:
     - new nmethod(...)

   sweeper:
     - invalidates newly allocated nmethod and patches VEP to call 
handle_wrong_method
     - tries to unlink it, but method()->from_compiled_entry() != 
verified_entry_point() since nmethod hasn't been installed yet

   ciEnv::register_method:
     - installs already invalidated nmethod

Calling corresponding Java method will lead to infinite loop: VEP of the 
compiled method calls handle_wrong_method and call site resolution 
returns the very same compiled method.

The fix is to grab a lock after nmethod is allocated in the code cache 
and check that it hasn't been already invalidated by the sweeper before 
proceeding. Sweeper already synchronizes on a nmethod before 
invalidating it.

Testing: failing test w/ diagnostic output.

Thanks!

Best regards,
Vladimir Ivanov