Request review: 4360113: Evict nmethods when code cache gets full

[Eric Caspole]

Hi Vladimir,
Thanks for your comments and trying to run it. More comments below.
Eric

On Dec 15, 2009, at 5:25 PM, Vladimir Kozlov wrote:

> Eric,
>
> In compileBroker.cpp and sweeper.cpp, I think, you should use separate
> new flag instead of 3*CodeCacheMinimumFreeSpace so it could
> be adjusted separatly.

OK. I am running out of good names, how about  
CodeCacheFlushingTriggerFreeSpace?

>
> Also you can move new condition after the body of the original
> so you will not need to check "> CodeCacheMinimumFreeSpace".
>
> I looked on verified_code_entry() and it works since
> clear_code() stores _adapter->get_c2i_entry() into
> _from_compiled_entry so it is not NULL.
>
> In handle_full_code_cache() do "if(!is_full) {} else {}"
> Change print format for "is_full".
> Change print format for for jint values (%ld% is for longs).
> Use INT64_FORMAT instead of %ld.

Will do.

>
> I am not sure the next code is correct.
> What if method was compiled again and got new nmethod?

I am not sure what you mean about compiled again, but I think the  
method would not be selected for flushing in that case.

> Also you should not reset all methodOop values (see failures below):

See below

>
> +         if ((((nmethod*)nm)->method()->code() == nm)) {
> +           // This method has not been previously considered for  
> preemptive unloading
> +           // or it was restored once already
> +           ((nmethod*)nm)->method()->clear_code_hedge();
> +         } else {
> +           // This method was previously considered for preemptive  
> unloading and was not called since then
> +           ((nmethod*)nm)->method()->set_saved_code(NULL);
> +           ((nmethod*)nm)->method()->invocation_counter()->reset();
> +           ((nmethod*)nm)->method()->backedge_counter()->reset();
> +           ((nmethod*)nm)->method()- 
> >set_interpreter_invocation_count(0);
> +           ((nmethod*)nm)->method()->set_method_data(NULL);
> +           ((nmethod*)nm)->make_not_entrant();
> +         }
>
> When I ran fastdebug VM with your patch on MagnyCours machine I got  
> next failures.
>
> Thanks,
> Vladimir
>
>

Yes I got carried away here erasing the mdo. I saw other places can  
reallocate mdo if necessary, but not here. The effect I was going for  
was to slow the stampede of recompiles. I will change this and retest  
it.

> % java -XX:ReservedCodeCacheSize=6M -XX:+PrintCodeCacheExtension - 
> XX:+UseCodeCacheFlushing -XX:MinCodeCacheFlushingInterval=300 -XX: 
> +PrintCompilation -jar SPECjvm2008.jar -ikv compiler.compiler
>
> #  Internal Error (/net/irkutsk/export/home/kvn/work2/hg/4360113/ 
> src/share/vm/runtime/frame.cpp:373), pid=26211, tid=35
> #  Error: assert(mdo != 0,"")
>
> V  [libjvm.so+0x170231a] void VMError::report(outputStream*) + 0x5be
> V  [libjvm.so+0x17032f6] void VMError::report_and_die() + 0x586
> V  [libjvm.so+0x7bf93d] void report_assertion_failure(const  
> char*,int,const char*) + 0x61
> V  [libjvm.so+0x88f534] void frame::interpreter_frame_set_bcx(int)  
> + 0x254
> V  [libjvm.so+0x894888] void frame::gc_prologue() + 0x3c
> V  [libjvm.so+0x1616292] void frame_gc_prologue(frame*,const  
> RegisterMap*) + 0x1e
> V  [libjvm.so+0x1615c28] void JavaThread::frames_do(void(*) 
> (frame*,const RegisterMap*)) + 0xa4
> V  [libjvm.so+0x16162c1] void JavaThread::gc_prologue() + 0x25
> V  [libjvm.so+0x161e6ba] void Threads::gc_prologue() + 0x26
> V  [libjvm.so+0x139bf91] void PSMarkSweep::invoke_no_policy(bool) +  
> 0x46d
> V  [libjvm.so+0x13c7eda] void PSScavenge::invoke() + 0x1da
> V  [libjvm.so+0x1317b15]  
> HeapWord*ParallelScavengeHeap::failed_mem_allocate(unsigned,bool) +  
> 0x115
> V  [libjvm.so+0x1704fa1] void VM_ParallelGCFailedAllocation::doit()  
> + 0xc5
> V  [libjvm.so+0x1737c89] void VM_Operation::evaluate() + 0xe1
> V  [libjvm.so+0x1736457] void VMThread::evaluate_operation 
> (VM_Operation*) + 0
>
>

The one below looks like a too eager assert in  
SimpleCompPolicy::method_invocation_event(), since the next if block  
below the assert again checks "&& UseCompiler)" -- what do you think?


> % java -XX:ReservedCodeCacheSize=6M -XX:+PrintCodeCacheExtension - 
> XX:+UseCodeCacheFlushing -XX:MinCodeCacheFlushingInterval=300 -XX: 
> +PrintCompilation -XX:+PrintMethodFlushing -jar SPECjvm2008.jar - 
> ikv compiler.sunflow compiler.compiler
>
> #  Internal Error (/net/irkutsk/export/home/kvn/work2/hg/4360113/ 
> src/share/vm/runtime/compilationPolicy.cpp:124), pid=26531, tid=51
> #  Error: assert(UseCompiler || CompileTheWorld,"UseCompiler should  
> be set by now.")
>
> Current thread (0x08c84400):  JavaThread "BenchmarkThread  
> compiler.sunflow 6" [_thread_in_vm, id=51, stack 
> (0xfa2c9000,0xfa319000)]
>
> Stack: [0xfa2c9000,0xfa319000],  sp=0xfa317ad8,  free  
> space=13afa319000k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,  
> C=native code)
> V  [libjvm.so+0x170235a] void VMError::report(outputStream*) + 0x5be
> V  [libjvm.so+0x1703336] void VMError::report_and_die() + 0x586
> V  [libjvm.so+0x7bf95d] void report_assertion_failure(const  
> char*,int,const char*) + 0x61
> V  [libjvm.so+0x6ccd4e] void  
> SimpleCompPolicy::method_invocation_event(methodHandle,Thread*) +  
> 0x20a
> V  [libjvm.so+0xa68949]  
> nmethod*InterpreterRuntime::frequency_counter_overflow_inner 
> (JavaThread*,unsigned char*) + 0x1551
>
>
> Eric Caspole wrote:
>> 4360113: Evict nmethods when code cache gets full
>> http://cr.openjdk.java.net/~ecaspole/4360113/
>> In this change, under a flag and off by default, when compilers  
>> notice the code cache is getting full, they will call a vm op that  
>> calls new code that attempts to speculatively unload the oldest  
>> half of the nmethods (based on the compile job id) by hiding the  
>> methodOop's ref to the nmethod in the new _saved_code field. Then  
>> execution resumes. After inline cache cleaning, callers will have  
>> to go back to the methodOop to get the verified entry point.
>> At that point, the methodOop's _code field is restored from the  
>> _saved_code field and the methodOop/nmethod go back to their  
>> normal state.
>> If a method so marked is not called by the second sweep cycle  
>> after the one where forced unloading happened, the nmethod will be  
>> marked non-entrant and got rid of by normal sweeping. That gives  
>> the app a few seconds to make progress and call its hottest methods.
>> We chose to target the oldest half of nmethods due to a customer  
>> experience with a long-running app server, and despite multiple  
>> redeployments of the same web app, something was preventing old  
>> instances of the web app from ever getting unloaded. In that case,  
>> they ran into the code cache full problem so the most recent  
>> redeployment was running interpreter only. We have also observed  
>> that for many applications a lot of methods get compiled and used  
>> during the startup phase that are never used again.
>> In this change there is also a timer based backoff, default of 30  
>> seconds, so that if the normal state of the app is constantly  
>> triggering unloading, the unloading will stop and it will fall  
>> back to the existing situation of disabling the compiler.
>> In my testing, this allows the program to quickly resume normal  
>> operation with no noticeable performance degradation.
>> Thanks for your comments,
>> Eric
>