RFR: Add flag to prevent multiple deopts on the same nmethod. [v4]

[Coleen Phillimore]

On Wed, 10 Nov 2021 14:23:27 GMT, Coleen Phillimore <coleenp at openjdk.org> wrote:

>> I added a flag to prevent the same nmethod from being optimized multiple times.  It might be better to have a new MarkForDeoptimizationStatus enum but wasn't sure how that would interact with the deoptimize_update state.  Advice welcome.
>> I also don't know why the sweeper is so slow to get rid of these entries.
>> This makes ctw_2 test run in an hour on my machine, just like mainline jdk/jdk (machine is slow!) in fastdebug.
>
> Coleen Phillimore has updated the pull request incrementally with one additional commit since the last revision:
> 
>   CodeCache::mark_all_nmethods_for_deoptimization does try to make the state go backwards.

I added some logging to CodeCache::make_marked_nmethods_not_entrant().
For mainline, it's called 395581 times -> max 2946 not entrant found -> 4072 max nmethods iterated
For loom,       it's called 337990 times -> max 1367 not entrant found -> 94950 max nmethods iterated

1367 is with this change.  The 94K nmethods iterated seems to be why CTW takes 7x longer for loom than mainline.

-------------

PR: https://git.openjdk.java.net/loom/pull/81