Request review: 4360113: Evict nmethods when code cache gets full

[Vladimir Kozlov]

Eric Caspole wrote:
> I see what you mean. If there are other already disconnected methods 
> (nm->method()->code() != nm) due to normal operation, can I safely 
> assume they are already marked not entrant and they will go away by 
> normal sweeping? Or should I hurry them along here? I  see how not 

The assert(nm->method()->code()->is_not_entrant() is enough, I think.

> having the extra check means I might reset the counters many times for 
> no reason, I'll change that.
> 

>> It seems other thread set UseCompiler to false between the check and 
>> assert.
> 
> So that means this could have happened in the existing system if the 
> broker shut off the compiler due to fullness, racing with the java 
> thread making this call. Yuck.

Yes. With your changes it could happened more frequently.
I think, it is related to what we talked about today on meeting:

# Consolidate "shut off compiler" logic into one place, and reconsider logic to turn compilers on and off.  Currently this logic is spread across three different places even without the unloading change.

May be we should use separate flag SuspendCompilation when we think
codecache could be freed and switch of UseCompiler when we give up.

Thanks,
Vladimir

> 
>>
>> Vladimir
>>
>>>> % java -XX:ReservedCodeCacheSize=6M -XX:+PrintCodeCacheExtension 
>>>> -XX:+UseCodeCacheFlushing -XX:MinCodeCacheFlushingInterval=300 
>>>> -XX:+PrintCompilation -XX:+PrintMethodFlushing -jar SPECjvm2008.jar 
>>>> -ikv compiler.sunflow compiler.compiler
>>>>
>>>> #  Internal Error 
>>>> (/net/irkutsk/export/home/kvn/work2/hg/4360113/src/share/vm/runtime/compilationPolicy.cpp:124), 
>>>> pid=26531, tid=51
>>>> #  Error: assert(UseCompiler || CompileTheWorld,"UseCompiler should 
>>>> be set by now.")
>>>>
>>>> Current thread (0x08c84400):  JavaThread "BenchmarkThread 
>>>> compiler.sunflow 6" [_thread_in_vm, id=51, 
>>>> stack(0xfa2c9000,0xfa319000)]
>>>>
>>>> Stack: [0xfa2c9000,0xfa319000],  sp=0xfa317ad8,  free 
>>>> space=13afa319000k
>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, 
>>>> C=native code)
>>>> V  [libjvm.so+0x170235a] void VMError::report(outputStream*) + 0x5be
>>>> V  [libjvm.so+0x1703336] void VMError::report_and_die() + 0x586
>>>> V  [libjvm.so+0x7bf95d] void report_assertion_failure(const 
>>>> char*,int,const char*) + 0x61
>>>> V  [libjvm.so+0x6ccd4e] void 
>>>> SimpleCompPolicy::method_invocation_event(methodHandle,Thread*) + 0x20a
>>>> V  [libjvm.so+0xa68949] 
>>>> nmethod*InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*,unsigned 
>>>> char*) + 0x1551
>>>>
>>>>
>>>> Eric Caspole wrote:
>>>>> 4360113: Evict nmethods when code cache gets full
>>>>> http://cr.openjdk.java.net/~ecaspole/4360113/
>>>>> In this change, under a flag and off by default, when compilers 
>>>>> notice the code cache is getting full, they will call a vm op that 
>>>>> calls new code that attempts to speculatively unload the oldest 
>>>>> half of the nmethods (based on the compile job id) by hiding the 
>>>>> methodOop's ref to the nmethod in the new _saved_code field. Then 
>>>>> execution resumes. After inline cache cleaning, callers will have 
>>>>> to go back to the methodOop to get the verified entry point.
>>>>> At that point, the methodOop's _code field is restored from the 
>>>>> _saved_code field and the methodOop/nmethod go back to their normal 
>>>>> state.
>>>>> If a method so marked is not called by the second sweep cycle after 
>>>>> the one where forced unloading happened, the nmethod will be marked 
>>>>> non-entrant and got rid of by normal sweeping. That gives the app a 
>>>>> few seconds to make progress and call its hottest methods.
>>>>> We chose to target the oldest half of nmethods due to a customer 
>>>>> experience with a long-running app server, and despite multiple 
>>>>> redeployments of the same web app, something was preventing old 
>>>>> instances of the web app from ever getting unloaded. In that case, 
>>>>> they ran into the code cache full problem so the most recent 
>>>>> redeployment was running interpreter only. We have also observed 
>>>>> that for many applications a lot of methods get compiled and used 
>>>>> during the startup phase that are never used again.
>>>>> In this change there is also a timer based backoff, default of 30 
>>>>> seconds, so that if the normal state of the app is constantly 
>>>>> triggering unloading, the unloading will stop and it will fall back 
>>>>> to the existing situation of disabling the compiler.
>>>>> In my testing, this allows the program to quickly resume normal 
>>>>> operation with no noticeable performance degradation.
>>>>> Thanks for your comments,
>>>>> Eric
>>>>
>>
> 
>