Stack walking performance issue

[Aleksey Shipilev]

On 3/17/19 3:43 PM, Amir Hadadi wrote:
> We've encountered the following performance issue which happens on an instance deployed in docker with ubuntu 16.04, Linux kernel 4.4.0-92-generic and OpenJDK 11.0.2.
> The issue shows up with zgc but does not show up with G1.
> 
> During a period of 10 days after deployment, CPU usage goes up steadily at a rate of ~10% per day. Eventually we have to restart all instances.
> I profiled our app after 4 days of uptime using async-profiler and found that the following stack is the most frequent stack: https://gist.github.com/amirhadadi/48b6f84e3b2412124e817a50608e6ddd

I wonder if this is due to ResolvedMethodTable (RMT) never cleaned, and this is why lookup takes a
very long time. RMT cleanups are normally done in the course of class unloading (see
SystemDictionary::do_unloading), but ZGC does only implement it starting jdk12 [1]. If so, there is
no workaround for jdk11.

-Aleksey

[1] https://bugs.openjdk.java.net/browse/JDK-8214897