Avoiding excessive creation of LambdaForm classes

[Jesse Schulman]

An update with more details, we have set MaxMetaspaceSize and have been
running for a while now.  The number of loaded classes seems to be under
control as well as the metaspace size.

We are however still seeing a slow/steady increase of the "Internal"
category as reported by Native Memory Tracking, as well as the overall RES
memory as reported by the OS.  This high memory usage is what caused us to
start investigating in the first place.

Currently the high traffic JVM in question is configured with -Xmx512m
-XX:MaxMetaspaceSize=256m and is consuming over 2G of RES memory with
Native Memory Tracking reporting over 1.2G of memory used by the "Internal"
category (and only 64M of MetaSpace reported).  It seems that internal is
growing at ~43KB/minute and doing GC does not recover any of it.

I am not sure how best to determine what is in that "Internal" category.  I
have included a standalone reproducer at
https://github.com/jesseschulman/native_memory that demonstrates the issue.

We use a single instance of NashornScriptEngine across our whole
application and for each user transaction we create what we call a
"ScriptEnvironment" (not the same as the nashorn class of the same name).
When we create a new environment we make a call to
NashornScriptEngine.createBindings() and all evaluation within that
environment uses the eval(String, Bindings) method from
AbstractScriptEngine.  After calling createBindings we eval the Class
implementation from prototypejs which also includes some polyfills for
Object/Function/Array, as seen in the attached reproducer, we also put some
of our own JSObject implementations into that bindings.  Turning off the
evaluation of Class.js seems to greatly reduce the memory growth but not
completely.

I am happy to take this to another openjdk mailing list (possibly the gc
list?) to help me figure out what is filling up that "Internal" category, I
have tried most every strategy I can find online to determine what is in
there but I have not been successful in finding anything conclusive.

Thanks,
Jesse



On Thu, Jul 27, 2017 at 5:08 PM Jesse Schulman <jesse at dreamtsoft.com> wrote:

> We have noticed on a high traffic instance of our application that the
> number of loaded classes continues to grow, this in turn increases the
> amount of Metaspace and native memory used by the JVM.  Moving forward we
> are working on a safe value for MaxMetaspaceSize which should eventually
> trigger a GC that cleans up these classes, but we would like to avoid these
> classes piling up unnecessarily if possible.
>
> During investigation we found that the large majority of these classes are
> LambdaForm that are created as a result of how we convert java Map and List
> instances into proper javascript Object and Array instances.
>
> We run with the --no-java option and expose functionality to javascript
> via our own implementations of JSObject.  The attached reproducer
> represents how we convert java Maps and Lists in order to return them from
> our JSObject implementations.
>
> My best guess from investigating is that when we convert a complex java
> Map/List that contains many nested Maps/Lists we create many new instances
> of javascript Objects/Arrays as seen in the attached reproducer.  It seems
> we go thru the Invoker.maybeCustomize every time we call getMember and
> every time we call newObject in createNewGlobalObject method of the
> reproducer.  Each time it increments the MethodHandle.customizationCount
> until that is 127, and then another LambdaForm is created.
>
> I ran the attached reproducer with -verbose:class turned on and a logging
> breakpoint in my IDE that logs the mh.customizationCount every time
> Invokers.maybeCustomize calls mh.customize().  Once the reproducer logs
> "STARTING TEST" and loads some initial classes I see the breakpoint log
> that we've hit 127 customizations followed by the loading of another
> LambdaForm class:
>
> customize 127
> customize 127
> [Loaded java.lang.invoke.LambdaForm$BMH/1049817027 from
> java.lang.invoke.LambdaForm]
> [Loaded java.lang.invoke.LambdaForm$BMH/23211803 from
> java.lang.invoke.LambdaForm]
> customize 127
> [Loaded java.lang.invoke.LambdaForm$BMH/1923598304 from
> java.lang.invoke.LambdaForm]
> customize 127
> [Loaded java.lang.invoke.LambdaForm$BMH/776700275 from
> java.lang.invoke.LambdaForm]
> customize 127
> customize 127
> [Loaded java.lang.invoke.LambdaForm$BMH/118394766 from
> java.lang.invoke.LambdaForm]
> [Loaded java.lang.invoke.LambdaForm$BMH/386163331 from
> java.lang.invoke.LambdaForm]
> customize 127
> [Loaded java.lang.invoke.LambdaForm$BMH/1540374340 from
> java.lang.invoke.LambdaForm]
>
>
> So I have 2 questions...
>
> 1. Is there a better/faster way for us to create the proper javascript
> Object/Array instances within our java code so we don't trigger these
> classes to be loaded?
>
> 2. Is this the intended behavior given:
>     a. We don't actually evaluate any javascript at all in the reproducer,
> I can see generating these classes if we were evaluating a lot of different
> javascript
>     b. Every time we call getMember against the global bindings we only
> ever pass "Array" or "Object" and when calling newObject against the
> Array/Object constructors we always do so with zero arguments, meaning we
> only call 4 different signatures yet the MethodHandle seems to be
> "customized" hundreds of times
>
> My assessment may be off or this may be exactly the expected behavior, but
> would still really appreciate any guidance on doing this type of conversion
> to avoid loading so many LambdaForm classes.
>
> Thanks!
> Jesse
>