Request review: 4360113: Evict nmethods when code cache gets full

[Vladimir Kozlov]

Eric,

In compileBroker.cpp and sweeper.cpp, I think, you should use separate
new flag instead of 3*CodeCacheMinimumFreeSpace so it could
be adjusted separatly.

Also you can move new condition after the body of the original
so you will not need to check "> CodeCacheMinimumFreeSpace".

I looked on verified_code_entry() and it works since
clear_code() stores _adapter->get_c2i_entry() into
_from_compiled_entry so it is not NULL.

In handle_full_code_cache() do "if(!is_full) {} else {}"
Change print format for "is_full".
Change print format for for jint values (%ld% is for longs).
Use INT64_FORMAT instead of %ld.

I am not sure the next code is correct.
What if method was compiled again and got new nmethod?
Also you should not reset all methodOop values (see failures below):

+         if ((((nmethod*)nm)->method()->code() == nm)) {
+           // This method has not been previously considered for preemptive unloading
+           // or it was restored once already
+           ((nmethod*)nm)->method()->clear_code_hedge();
+         } else {
+           // This method was previously considered for preemptive unloading and was not called since then
+           ((nmethod*)nm)->method()->set_saved_code(NULL);
+           ((nmethod*)nm)->method()->invocation_counter()->reset();
+           ((nmethod*)nm)->method()->backedge_counter()->reset();
+           ((nmethod*)nm)->method()->set_interpreter_invocation_count(0);
+           ((nmethod*)nm)->method()->set_method_data(NULL);
+           ((nmethod*)nm)->make_not_entrant();
+         }

When I ran fastdebug VM with your patch on MagnyCours machine I got next failures.

Thanks,
Vladimir


% java -XX:ReservedCodeCacheSize=6M -XX:+PrintCodeCacheExtension -XX:+UseCodeCacheFlushing -XX:MinCodeCacheFlushingInterval=300 -XX:+PrintCompilation -jar SPECjvm2008.jar -ikv compiler.compiler

#  Internal Error (/net/irkutsk/export/home/kvn/work2/hg/4360113/src/share/vm/runtime/frame.cpp:373), pid=26211, tid=35
#  Error: assert(mdo != 0,"")

V  [libjvm.so+0x170231a] void VMError::report(outputStream*) + 0x5be
V  [libjvm.so+0x17032f6] void VMError::report_and_die() + 0x586
V  [libjvm.so+0x7bf93d] void report_assertion_failure(const char*,int,const char*) + 0x61
V  [libjvm.so+0x88f534] void frame::interpreter_frame_set_bcx(int) + 0x254
V  [libjvm.so+0x894888] void frame::gc_prologue() + 0x3c
V  [libjvm.so+0x1616292] void frame_gc_prologue(frame*,const RegisterMap*) + 0x1e
V  [libjvm.so+0x1615c28] void JavaThread::frames_do(void(*)(frame*,const RegisterMap*)) + 0xa4
V  [libjvm.so+0x16162c1] void JavaThread::gc_prologue() + 0x25
V  [libjvm.so+0x161e6ba] void Threads::gc_prologue() + 0x26
V  [libjvm.so+0x139bf91] void PSMarkSweep::invoke_no_policy(bool) + 0x46d
V  [libjvm.so+0x13c7eda] void PSScavenge::invoke() + 0x1da
V  [libjvm.so+0x1317b15] HeapWord*ParallelScavengeHeap::failed_mem_allocate(unsigned,bool) + 0x115
V  [libjvm.so+0x1704fa1] void VM_ParallelGCFailedAllocation::doit() + 0xc5
V  [libjvm.so+0x1737c89] void VM_Operation::evaluate() + 0xe1
V  [libjvm.so+0x1736457] void VMThread::evaluate_operation(VM_Operation*) + 0


% java -XX:ReservedCodeCacheSize=6M -XX:+PrintCodeCacheExtension -XX:+UseCodeCacheFlushing -XX:MinCodeCacheFlushingInterval=300 -XX:+PrintCompilation -XX:+PrintMethodFlushing -jar SPECjvm2008.jar -ikv compiler.sunflow compiler.compiler

#  Internal Error (/net/irkutsk/export/home/kvn/work2/hg/4360113/src/share/vm/runtime/compilationPolicy.cpp:124), pid=26531, tid=51
#  Error: assert(UseCompiler || CompileTheWorld,"UseCompiler should be set by now.")

Current thread (0x08c84400):  JavaThread "BenchmarkThread compiler.sunflow 6" [_thread_in_vm, id=51, stack(0xfa2c9000,0xfa319000)]

Stack: [0xfa2c9000,0xfa319000],  sp=0xfa317ad8,  free space=13afa319000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x170235a] void VMError::report(outputStream*) + 0x5be
V  [libjvm.so+0x1703336] void VMError::report_and_die() + 0x586
V  [libjvm.so+0x7bf95d] void report_assertion_failure(const char*,int,const char*) + 0x61
V  [libjvm.so+0x6ccd4e] void SimpleCompPolicy::method_invocation_event(methodHandle,Thread*) + 0x20a
V  [libjvm.so+0xa68949] nmethod*InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*,unsigned char*) + 0x1551


Eric Caspole wrote:
> 4360113: Evict nmethods when code cache gets full
> 
> http://cr.openjdk.java.net/~ecaspole/4360113/
> 
> In this change, under a flag and off by default, when compilers notice 
> the code cache is getting full, they will call a vm op that calls new 
> code that attempts to speculatively unload the oldest half of the 
> nmethods (based on the compile job id) by hiding the methodOop's ref to 
> the nmethod in the new _saved_code field. Then execution resumes. After 
> inline cache cleaning, callers will have to go back to the methodOop to 
> get the verified entry point.
> At that point, the methodOop's _code field is restored from the 
> _saved_code field and the methodOop/nmethod go back to their normal state.
> 
> If a method so marked is not called by the second sweep cycle after the 
> one where forced unloading happened, the nmethod will be marked 
> non-entrant and got rid of by normal sweeping. That gives the app a few 
> seconds to make progress and call its hottest methods.
> 
> We chose to target the oldest half of nmethods due to a customer 
> experience with a long-running app server, and despite multiple 
> redeployments of the same web app, something was preventing old 
> instances of the web app from ever getting unloaded. In that case, they 
> ran into the code cache full problem so the most recent redeployment was 
> running interpreter only. We have also observed that for many 
> applications a lot of methods get compiled and used during the startup 
> phase that are never used again.
> 
> In this change there is also a timer based backoff, default of 30 
> seconds, so that if the normal state of the app is constantly triggering 
> unloading, the unloading will stop and it will fall back to the existing 
> situation of disabling the compiler.
> 
> In my testing, this allows the program to quickly resume normal 
> operation with no noticeable performance degradation.
> Thanks for your comments,
> Eric
> 
>