RFR: 8219724: ZGC: Make inline cache cleaning more robust

[Erik Österlund]

Hi,

Today, during the nmethod unlinking phase, the per-nmethod lock is held 
across first an is_unloading() call on the nmethod and then inline cache 
cleaning, which may take the nmethod locks of all nmethods referred to 
from the inline caches.
If care is not taken, an nmethod A can have an inline cache pointing at 
nmethod B, and B can have an inline cache pointing back at A. This could 
potentially cause a deadlock. Today it is subtly safe, because between 
calling is_unloading() and cleaning the inline caches, the nmethod entry 
barrier is disarmed, which causes an mfence in the patching code. This 
ensures that the racing threads do not enter a deadlock situation, 
because they will observe the is_unloading state that was published as a 
cache by the other thread in the race, causing the locks that would 
cause the deadlock to not be taken.

I would like to move the locks so that this becomes more robust, and 
does not rely on the implicit fencing between is_unloading() and 
cleaning the inline caches.

Bug:
https://bugs.openjdk.java.net/browse/JDK-8219724

Webrev:
http://cr.openjdk.java.net/~eosterlund/8219724/webrev.00/

Thanks,
/Erik