RFR: 8138889: Restructure InstanceRefKlass discovery code to increase readability

[Stefan Johansson]

Hi,

Please review this small cleanup:
https://bugs.openjdk.java.net/browse/JDK-8138889

Webrev:
http://cr.openjdk.java.net/~sjohanss/8138889/hotspot.00/

Summary:
This is a follow up to JDK-8138889 and this patch is based on the 
changes made to address that issue. When fixing 8138889 the 
InstanceRefKlass handling was slightly simplified, but the check for 
discovery is still somewhat hard to follow. This patch extracts these 
checks and re-arranges them into a separate function.

Testing:
* JPRT
* RBT and Aurora runs
* Locally run gc-test-suite

Thanks,
Stefan