From jluzon at riotgames.com Thu Dec 1 00:10:14 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Wed, 30 Nov 2016 16:10:14 -0800 Subject: Nashorn Memory Leak using Invocale.invokeFunction Message-ID: Hey folks, We are running into issues right now where repeated use of an Invocable is eventually causing a crash due to an OutOfMemoryError. The code snippets that are causing the crashes are: > serviceBody = > JSON.stringify(variables.summonerIds[0].split(',').filter(function(el) > {return el.length != 0}) and > serviceBody = JSON.stringify(variables.summonerNames[0].split(',')); There is some boilerplate javascript code that includes all those variables being referenced, but I have ommited it to make this e-mail shorter. A heap dump after the process crashes shows an ArrayList with an Object[] of size 15704907 filed with java.lang.Long objects. A heap dump analysis before it crashes has made me believe that it has to do with *jdk.nashorn.internal.runtime.arraysDeletedArrayFilter.java* but I might be in the wrong here. Our service runs in *jetty-9.3.5.v20151012. * JRE is *openjdk version "1.8.0_111".* Args passed are *-Xms4g -Xmx4g -XX:NewSize=1g -XX:MaxMetaspaceSize=1g -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:-OmitStackTraceInFastThrow* I also have the heap dump which is ~5.4G uncompressed and ~750M compressed. I can upload those somewhere if someone thinks they could find value in taking a look in it. Much thanks before hand to anyone taking a look at this. I've been trying to figure this out for days but looking at cecompiled Java code to solve this hasn't netted me any results yet. Thanks. Jesus Luzon From sundararajan.athijegannathan at oracle.com Thu Dec 1 05:08:51 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 10:38:51 +0530 Subject: JSObject call() is passed undefined/null for the argument 'thiz'? In-Reply-To: References: Message-ID: <583FB063.5070509@oracle.com> Thanks for reporting this issue. Filed a bug: https://bugs.openjdk.java.net/browse/JDK-8170565 Thanks, -Sundar On 01/12/16, 12:36 AM, Art Fiedler wrote: > @Sundararajan > > When calling the following code, I expected the call to myJSObject inside > testJSOCall to pass the current > scope as a ScriptObjectMirror to JSObjectImpl's call method as the thiz > argument. In this case without > modifying the call to myJSObject() should 'this' automatically be passed to > the call() method? > > Thanks, > Arthur Fiedler > > public class NashornBugTest > { > public static void main(String[] args) throws Exception { > NashornScriptEngineFactory factory = new > NashornScriptEngineFactory(); > ScriptEngine scriptEngine = factory.getScriptEngine(); > scriptEngine.put("myJSObject", new JSObjectImpl()); > scriptEngine.eval( > "var a = 42;\n" + > "function testJSOCall(b) {\n" + > " myJSObject('text purposely did not pass b');\n" + > "}\n" + > "testJSOCall(a);\n"); > } > public static class JSObjectImpl extends AbstractJSObject > { > @Override > public Object call(Object thiz, Object... args) > { > if (ScriptObjectMirror.isUndefined(thiz) || thiz == null) > throw new IllegalArgumentException("'thiz' is > undefined/null"); > // TODO: need the value of 'b' in the calling function > return super.call(thiz, args); > } > @Override > public boolean isFunction() > { > return true; > } > } > } From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:21:18 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 11:51:18 +0530 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' Message-ID: <583FC15E.4060509@oracle.com> Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 Thanks, -Sundar From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:31:39 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 12:01:39 +0530 Subject: SourceMaps In-Reply-To: References: Message-ID: <583FC3CB.4060108@oracle.com> This is a feature request. For jdk9, except for jigsaw, we're not adding any new feature at this point. It *may* be considered for a release beyond jdk9. Please do file rfe if you think this is an important feature. Thanks, -Sundar On 01/12/16, 4:35 AM, Art Fiedler wrote: > Would be nice to provide SourceMaps to the ScriptEngine... I've seen here... > > http://stackoverflow.com/questions/35137455/nashorn-use-of-source-maps > > That nashorn doesn't support source maps, and I'm sure there is a library > out there that > is catching the exception and formatting it before it would come to my > code. Also note > new Error() doesn't seem to provide the columnNumber to do my own > source map conversion. > > I would propose supporting inline source maps in the js code... or > implementing in java by starting out with > something like a SourceMapProvider interface... > > public interface SourceMapProvider { > public SourceMapConsumer getConsumer(String filename); > public SourceMapConsumer putConsumer(String filename, SourceMapConsumer > consumer); > > public SourcePosition getOriginalPosition(String filename, int line, > int column); > public SourcePosition getOriginalPosition(String filename, int offset); > public SourcePosition getGeneratedPosition(String filename, int line, > int column); > public SourcePosition getGeneratedPosition(String filename, int offset); > } > public interface SourceMapConsumer { > public String getFilename(); > public SourcePosition getOriginalPosition(int line, int column); > public SourcePosition getOriginalPosition(int offset); > public SourcePosition getGeneratedPosition(int line, int column); > public SourcePosition getGeneratedPosition(int offset); > } > public interface SourcePosition { > public String getFilename(); > public int getOriginalLine(); > public int getOriginalColumn(); > public int getOriginalOffset(); > public String getOriginalName(); // Original Identifier, I believe this > would be used for replaced variable names etc > public int getGeneratedLine(); > public int getGeneratedColumn(); > public int getGeneratedOffset(); > } > > scriptEngine.setSourceMapProvider(mySourceMaps); > scriptEngine.eval("//@sourceURL=myfile.js\n... minified or transformed code > ..."); > scriptEngine.eval("//@sourceURL=myotherfile.js\n... minified or transformed > code ..."); > > The idea would be that the CodeGenerator, etc... would query the provider > for line/column info > passed to an error if a SourceMapProvider was specified and a > SourceMapConsumer exists for that the file. > > -Arthur Fiedler From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:40:46 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 12:10:46 +0530 Subject: Confusing ScriptObjectMirror size() vs mirror.get("length") In-Reply-To: References: Message-ID: <583FC5EE.1070509@oracle.com> I think this is not a bug. "length" is not the same as number of properties in an object. For example, if you try this: var a = [3, 5, 66]; a.foo = "nar"; for (var i = 0; i < a.length; i++) print(a[i]); "length" is 3 [and not 4]. keySet size is the number of "keys" (properties). You've to get "length" property of the array. -Sundar On 01/12/16, 1:32 AM, Art Fiedler wrote: > @Sundararajan > > When looping an array in javascript I would typically do something like > this... > > for(var i = 0; i< arr.length; i++) { > console.log('Index: ' + i + ' = ' + arr[i]); > } > > When looping a javascript array using the ScriptObjectMirror I would > initially think to use the same syntax, however I would swap .length with > .size() since that method is available. > > .size() however is not the same as .length in this case. I'm not saying > that is a bug, just noting the confusion it could cause when/if someone > write's a java loop like that and in the future they run into an undefined > item in the middle of the javascript array. See this example... > > Thanks, > Arthur Fiedler > > public static void main(String[] args) throws Exception { > NashornScriptEngineFactory factory = new > NashornScriptEngineFactory(); > ScriptEngine scriptEngine = factory.getScriptEngine(); > scriptEngine.eval( > "globalArray = new Array(3);\n" + > "globalArray[0] = 'line1';\n"+ > "globalArray[2] = 'line3';\n"); // Note index 1 was skipped > ScriptObjectMirror array = > (ScriptObjectMirror)scriptEngine.get("globalArray"); > System.out.println("entrySet.size(): " + array.entrySet().size()); > System.out.println("keySet.size(): " + array.keySet().size()); > System.out.println("values.size(): " + array.values().size()); > System.out.println("length: " + array.get("length")); > System.out.println("size(): " + array.size()); > for(int i = 0; i< array.size(); i++) { > System.out.println("Index: " + i); > String iStr = String.valueOf(i); > System.out.println(" Get('"+iStr+"'): " + array.get(iStr)); > } > System.out.println("what happened to index 2?? oops!"); > } > /* > entrySet.size(): 2 > keySet.size(): 2 > values.size(): 2 > length: 3 > size(): 2 > Index: 0 > Get('0'): line1 > Index: 1 > Get('1'): null > what happened to index 2?? oops! > */ From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:44:46 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 12:14:46 +0530 Subject: Error Stack Column number In-Reply-To: References: Message-ID: <583FC6DE.4080407@oracle.com> We do not aim to provide complete compatibility with other JS implementations on the non-standard properties such as "stack". stack tries to mimic whatever is done for Java code (no column number for eg.). But, as you've noted there are enough information on Error objects via other properties like lineNumber, columnNumber, fileName. It should be possible to write a simple utility function to format stack string as desired for specific applications. Thanks, -Sundar On 01/12/16, 3:32 AM, Art Fiedler wrote: > When making an implementation of console.count([label]) part of the > Mozilla's developer docs > https://developer.mozilla.org/en-US/docs/Web/API/Console/count > mentions when label is omitted, the function will count the number of times > count() was called > on that line... however, if the javascript code has been minified all lines > of code may be on the > same line... messing up all console.count() calls... > > I was providing that implementation by using new Error().stack and parsing > the current file:line... > however I could fix the minify issue if new Error().stack output the column > also for instance... > other browser seem to be doing file:line:column in the stack trace. > Including nodejs see: > https://nodejs.org/api/errors.html#errors_error_stack > > Microsoft Edge's dev console outputs... > new Error().stack: > "Error > at eval code (eval code:1:1)" > Firefox since FF30: > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/stack > > Currently nashorn error stacks only include the line number... > public static void main(String[] args) throws Exception { > NashornScriptEngineFactory factory = new > NashornScriptEngineFactory(); > ScriptEngine scriptEngine = factory.getScriptEngine(); > scriptEngine.put(ScriptEngine.FILENAME, "myfile.js"); > scriptEngine.eval("print('fileName: ' + new Error().fileName);"); > scriptEngine.eval("print('lineNumber: ' + new > Error().lineNumber);"); > scriptEngine.eval("print('columnNumber: ' + new > Error().columnNumber);"); > scriptEngine.eval("print('stack: ' + new Error().stack);"); > } > /* > fileName: myfile.js > lineNumber: 1 > columnNumber: -1 > stack: Error > at (myfile.js:1) > */ > > Would be nice to add the column number to the stacks and error object in > nashorn. > > Thanks, > Arthur Fiedler From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:46:23 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 12:16:23 +0530 Subject: java.lang.VerifyError: Inconsistent stackmap frames at branch target In-Reply-To: <1457EC63-39E7-4AE7-BAB0-F77352D6743A@aperto.com> References: <1457EC63-39E7-4AE7-BAB0-F77352D6743A@aperto.com> Message-ID: <583FC73F.8000309@oracle.com> Is there a simple reduced test case that we can use it to reproduce the issue you're seeing? Please send us the same and we'll file a bug (or you may do that as well). Thanks, -Sundar On 29/11/16, 11:11 PM, Frantzius, J?rg wrote: > Hi, > > with JDK 1.8.0_112 (on Mac OS X) I?m running into the following error. When querying bugs.openjdk.java.net for "Current frame's stack size doesn't match stackmap?, I only found bugs dating from 2013, so this may not be known yet? > > Unfortunately I can?t see the Javascript file name or line number in the error message. The last known source location node/lib/fs.js:374 that I can step to in the Netbeans debugger is calling a constructor ?Buffer(size)?, which is likely this source: https://github.com/nodejs/node/blob/v0.12.7-release/lib/buffer.js#L48 > > Following is the error message: > > java.lang.VerifyError: Inconsistent stackmap frames at branch target 404 > Exception Details: > Location: > jdk/nashorn/internal/scripts/Script$Recompilation$414$1806AA$\^function\_.L:1#Buffer(Ljdk/nashorn/internal/runtime/ScriptFunction;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @382: goto > Reason: > Current frame's stack size doesn't match stackmap. > Current Frame: > bci: @382 > flags: { } > locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } > stack: { } > Stackmap Frame: > bci: @404 > flags: { } > locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } > stack: { 'java/lang/Object' } > Bytecode: > 0x0000000: 2ab6 0014 3a04 1904 ba00 2000 0059 ba00 > 0x0000010: 2300 005f 2bba 0027 0000 9a00 1219 04ba > 0x0000020: 002a 0000 2c2d ba00 2e00 00b0 1904 ba00 > 0x0000030: 2000 0059 ba00 3100 005f 2cba 0034 0000 > 0x0000040: 9900 102b 2cb8 003a ba00 3e00 00a7 00d5 > 0x0000050: 1904 ba00 2000 0059 ba00 4100 005f 2cba > 0x0000060: 0044 0000 9900 4d19 04ba 0020 0000 59ba > 0x0000070: 0041 0000 5f2d ba00 4400 0099 0012 2dba > 0x0000080: 0047 0000 b800 4a03 8798 9a00 0912 4cb8 > 0x0000090: 0050 4e2b 1904 ba00 5500 0059 ba00 5800 > 0x00000a0: 005f 2c2d ba00 5c00 00ba 005f 0000 a700 > 0x00000b0: 7419 04ba 0020 0000 59ba 0062 0000 5f2c > 0x00000c0: ba00 6500 0099 0043 2cba 0068 0000 126a > 0x00000d0: b800 7099 0026 1904 ba00 2000 0059 ba00 > 0x00000e0: 7300 005f 2cba 0076 0000 ba00 7900 0099 > 0x00000f0: 000a 2cba 0076 0000 4d2b 2cba 007b 0000 > 0x0000100: ba00 3e00 00a7 001d 1904 b600 80ba 0083 > 0x0000110: 0000 1285 ba00 8900 0012 8b10 4507 b800 > 0x0000120: 91bf 2bba 0047 0000 1904 ba00 9400 00b8 > 0x0000130: 0097 9900 3a19 04b6 0080 ba00 9a00 0012 > 0x0000140: 9c19 04ba 0094 0000 59ba 009f 0000 5f10 > 0x0000150: 10ba 00a3 0000 b800 a712 a9b8 00a7 ba00 > 0x0000160: ac00 0012 8b10 4907 b800 91bf 2bba 007b > 0x0000170: 0000 0e98 9c00 0d2b 03ba 00af 0000 a700 > 0x0000180: 162b 592b ba00 4700 00b8 00b3 b800 b7ba > 0x0000190: 003e 0000 2b19 04b6 0080 ba00 ba00 00ba > 0x00001a0: 00bd 0000 2bba 0047 0000 1904 ba00 5500 > 0x00001b0: 00ba 00c0 0000 047c b800 b75d 58b8 003a > 0x00001c0: 5e58 989d 00c3 2bba 007b 0000 0e97 9e00 > 0x00001d0: b82b ba00 4700 0019 04ba 00c2 0000 1904 > 0x00001e0: ba00 c500 005f b800 3a5d 58b8 003a 675d > 0x00001f0: 58b8 003a 5e58 979e 0013 1904 ba00 c800 > 0x0000200: 00b2 00cc ba00 d000 0057 2b19 04ba 00d3 > 0x0000210: 0000 b200 cc19 04ba 00d6 0000 2b19 04ba > 0x0000220: 00c5 0000 1904 ba00 c500 002b ba00 4700 > 0x0000230: 00b8 00a7 ba00 da00 00ba 00bd 0000 1904 > 0x0000240: 1904 ba00 c500 002b ba00 4700 00b8 00a7 > 0x0000250: ba00 dd00 0019 04ba 00df 0000 1007 7e99 > 0x0000260: 0024 1904 1904 ba00 df00 0010 0780 ba00 > 0x0000270: e100 0019 0419 04ba 00e3 0000 0f63 ba00 > 0x0000280: e500 00a7 001a 1904 ba00 e800 00b2 00cc > 0x0000290: 2b2b ba00 4700 00ba 00ec 0000 5719 04ba > 0x00002a0: 0020 0000 59ba 0031 0000 5f2c ba00 3400 > 0x00002b0: 0099 0007 b200 ccb0 1904 ba00 2000 0059 > 0x00002c0: ba00 4100 005f 2cba 0044 0000 9900 8b2b > 0x00002d0: 59ba 00ef 0000 5f2c 2dba 00f2 0000 3a05 > 0x00002e0: 1905 2bba 0047 0000 b800 f599 0064 2bba > 0x00002f0: 0047 0000 3a06 2b19 05ba 005f 0000 1904 > 0x0000300: ba00 f800 00b2 00cc 2b2b ba00 4700 00ba > 0x0000310: 00fb 0000 572b ba00 fe00 0019 04b6 0080 > 0x0000320: ba00 ba00 00b8 0101 9900 2419 0419 04ba > 0x0000330: 00c5 0000 1906 b800 3a19 05b8 003a 675d > 0x0000340: 58b8 003a 5e58 67ba 00e5 0000 a700 08b2 > 0x0000350: 00cc 3a06 a700 9e19 04ba 0020 0000 59ba > 0x0000360: 0023 0000 5f2c ba00 2700 0099 001d 2c59 > 0x0000370: ba01 0400 005f 2b03 032b ba00 4700 00ba > 0x0000380: 0108 0000 57a7 0063 1904 ba00 2000 0059 > 0x0000390: ba00 3100 005f 2cba 0047 0000 ba00 3400 > 0x00003a0: 009a 001a 1904 ba00 2000 0059 ba00 7300 > 0x00003b0: 005f 2cba 0079 0000 9900 3003 3607 1507 > 0x00003c0: 8739 0818 082b ba00 7b00 0098 9c00 1c2b > 0x00003d0: 1808 2c18 08ba 010e 0000 ba01 1200 0018 > 0x00003e0: 080f 6339 08a7 ffde b200 cc3a 06b2 00cc > 0x00003f0: 3a05 b200 ccb0 > Stackmap Table: > append_frame(@44,Object[#125]) > same_frame(@80) > same_frame(@141) > same_frame(@147) > same_frame(@177) > same_frame_extended(@249) > same_frame(@264) > same_frame(@290) > same_frame_extended(@364) > same_frame(@385) > same_locals_1_stack_item_frame(@404,Object[#286]) > same_locals_1_stack_item_extended(@522,Object[#286]) > same_locals_1_stack_item_extended(@643,Object[#286]) > same_locals_1_stack_item_frame(@646,Object[#286]) > same_locals_1_stack_item_frame(@669,Object[#286]) > same_locals_1_stack_item_frame(@696,Object[#286]) > full_frame(@844,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > full_frame(@847,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286]},{Object[#286]}) > full_frame(@852,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > full_frame(@855,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) > same_locals_1_stack_item_frame(@904,Object[#286]) > same_locals_1_stack_item_frame(@955,Object[#286]) > full_frame(@963,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Top,Top,Integer,Double},{Object[#286]}) > full_frame(@1000,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) > full_frame(@1010,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > > at java.lang.Class.getDeclaredFields0(Native Method) > at java.lang.Class.privateGetDeclaredFields(Class.java:2583) > at java.lang.Class.getDeclaredField(Class.java:2068) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:209) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:204) > at java.security.AccessController.doPrivileged(Native Method) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller.initialize(Context.java:204) > at jdk.nashorn.internal.codegen.CompilationPhase$InstallPhase.transform(CompilationPhase.java:508) > at jdk.nashorn.internal.codegen.CompilationPhase.apply(CompilationPhase.java:624) > at jdk.nashorn.internal.codegen.Compiler.compile(Compiler.java:655) > at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.compileTypeSpecialization(RecompilableScriptFunctionData.java:725) > at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.getBest(RecompilableScriptFunctionData.java:905) > at jdk.nashorn.internal.runtime.ScriptFunctionData.getBest(ScriptFunctionData.java:375) > at jdk.nashorn.internal.runtime.ScriptFunctionData.getBestConstructor(ScriptFunctionData.java:247) > at jdk.nashorn.internal.runtime.ScriptFunction.findNewMethod(ScriptFunction.java:758) > at jdk.nashorn.internal.runtime.ScriptObject.lookup(ScriptObject.java:1827) > at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:104) > at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:98) > at jdk.internal.dynalink.support.CompositeTypeBasedGuardingDynamicLinker.getGuardedInvocation(CompositeTypeBasedGuardingDynamicLinker.java:176) > at jdk.internal.dynalink.support.CompositeGuardingDynamicLinker.getGuardedInvocation(CompositeGuardingDynamicLinker.java:124) > at jdk.internal.dynalink.support.LinkerServicesImpl.getGuardedInvocation(LinkerServicesImpl.java:154) > at jdk.internal.dynalink.DynamicLinker.relink(DynamicLinker.java:253) > at jdk.nashorn.internal.scripts.Script$Recompilation$402$9044AA$\^function\_.L:1#readFileSync(node/lib/fs.js:374) > [..] > > Any help or hints would be very appreciated! > > Regards, > J?rg > > --- > > Dipl. Inf. J?rg von Frantzius, Technical Director > > E-Mail joerg.frantzius at aperto.com > > Phone +49 30 283921-318 > Fax +49 30 283921-29 > > Aperto GmbH ? An IBM Company > Chausseestra?e 5, D-10115 Berlin > http://www.aperto.com > http://www.facebook.com/aperto > https://www.xing.com/companies/apertoag > > HRB 77049 B, AG Berlin Charlottenburg > Vorstand: Dirk Buddensiek (Vorsitzender), Kai Gro?mann, Stephan Haagen, Daniel Simon > Aufsichtsrat: Matthew Candy (Vorsitzender) > From sundararajan.athijegannathan at oracle.com Thu Dec 1 06:54:29 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 12:24:29 +0530 Subject: Nashorn Memory Leak using Invocale.invokeFunction In-Reply-To: References: Message-ID: <583FC925.9020706@oracle.com> Will you please send us reproduceable test case? we'll file a bug with that or you could file a bug as well. Thanks -Sundar On 01/12/16, 5:40 AM, Jesus Luzon wrote: > Hey folks, > > We are running into issues right now where repeated use of an Invocable is > eventually causing a crash due to an OutOfMemoryError. > > The code snippets that are causing the crashes are: > >> serviceBody = >> JSON.stringify(variables.summonerIds[0].split(',').filter(function(el) >> {return el.length != 0}) > and > >> serviceBody = JSON.stringify(variables.summonerNames[0].split(',')); > > There is some boilerplate javascript code that includes all those variables > being referenced, but I have ommited it to make this e-mail shorter. A heap > dump after the process crashes shows an ArrayList with an Object[] of size > 15704907 filed with java.lang.Long objects. A heap dump analysis before > it crashes has made me believe that it has to do with > *jdk.nashorn.internal.runtime.arraysDeletedArrayFilter.java* but I might be > in the wrong here. > > Our service runs in *jetty-9.3.5.v20151012. * > JRE is *openjdk version "1.8.0_111".* > Args passed are *-Xms4g -Xmx4g -XX:NewSize=1g -XX:MaxMetaspaceSize=1g > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly > -XX:-OmitStackTraceInFastThrow* > > I also have the heap dump which is ~5.4G uncompressed and ~750M compressed. > I can upload those somewhere if someone thinks they could find value in > taking a look in it. > > Much thanks before hand to anyone taking a look at this. I've been trying > to figure this out for days but looking at cecompiled Java code to solve > this hasn't netted me any results yet. > > Thanks. > Jesus Luzon From hannes.wallnoefer at oracle.com Thu Dec 1 09:43:55 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 1 Dec 2016 10:43:55 +0100 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: <583FC15E.4060509@oracle.com> References: <583FC15E.4060509@oracle.com> Message-ID: <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> Hi Sundar, The problem with this approach is that it will replace any occurrence of undefined this with the global object. However, this should only occur for scope calls. For example, the following call would see undefined replaced with global: func.call(undefined) This is probably not a problem that will occur very often, but ideally I think we should do the check and replacement on the linking side, i.e. in JSObjectLinker.findCallMethod. On the other hand we can?t check for function strictness that way. Maybe do it your way but add a boolean isScope parameter and bind that at link time? Hannes > Am 01.12.2016 um 07:21 schrieb Sundararajan Athijegannathan : > > Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 > > Thanks, > -Sundar From Joerg.Frantzius at aperto.com Thu Dec 1 11:18:04 2016 From: Joerg.Frantzius at aperto.com (=?utf-8?B?RnJhbnR6aXVzLCBKw7ZyZw==?=) Date: Thu, 1 Dec 2016 11:18:04 +0000 Subject: java.lang.VerifyError: Inconsistent stackmap frames at branch target In-Reply-To: <583FC73F.8000309@oracle.com> References: <1457EC63-39E7-4AE7-BAB0-F77352D6743A@aperto.com> <583FC73F.8000309@oracle.com> Message-ID: <561D7E3A-D986-4344-B765-3EFC25F86B17@aperto.com> Hi Sunda, you can reproduce by putting the following into a test.js file and executing it using jjs. ======== snip ============= function Buffer(subject, encoding) { if (!util.isBuffer(this)) return new Buffer(subject, encoding); if (util.isNumber(subject)) { this.length = +subject; } else if (util.isString(subject)) { if (!util.isString(encoding) || encoding.length === 0) encoding = 'utf8'; this.length = Buffer.byteLength(subject, encoding); // Handle Arrays, Buffers, Uint8Arrays or JSON. } else if (util.isObject(subject)) { if (subject.type === 'Buffer' && util.isArray(subject.data)) subject = subject.data; this.length = +subject.length; } else { throw new TypeError('must start with number, buffer, array or string'); } if (this.length > kMaxLength) { throw new RangeError('Attempt to allocate Buffer larger than maximum ' + 'size: 0x' + kMaxLength.toString(16) + ' bytes'); } if (this.length < 0) this.length = 0; else this.length >>>= 0; // Coerce to uint32. this.parent = undefined; if (this.length <= (Buffer.poolSize >>> 1) && this.length > 0) { if (this.length > poolSize - poolOffset) createPool(); this.parent = sliceOnto(allocPool, this, poolOffset, poolOffset + this.length); poolOffset += this.length; // Ensure aligned slices if (poolOffset & 0x7) { poolOffset |= 0x7; poolOffset++; } } else { alloc(this, this.length); } if (util.isNumber(subject)) { return; } if (util.isString(subject)) { // In the case of base64 it's possible that the size of the buffer // allocated was slightly too large. In this case we need to rewrite // the length to the actual length written. var len = this.write(subject, encoding); // Buffer was truncated after decode, realloc internal ExternalArray if (len !== this.length) { var prevLen = this.length; this.length = len; truncate(this, this.length); // Only need to readjust the poolOffset if the allocation is a slice. if (this.parent != undefined) poolOffset -= (prevLen - len); } } else if (util.isBuffer(subject)) { subject.copy(this, 0, 0, this.length); } else if (util.isNumber(subject.length) || util.isArray(subject)) { // Really crappy way to handle Uint8Arrays, but V8 doesn't give a simple // way to access the data from the C++ API. for (var i = 0; i < this.length; i++) this[i] = subject[i]; } } new Buffer(1024); ========== snip ============== I?ll hopefully get my OCA signed and sent in soon, so I can contribute in JIRA. Regards, J?rg --- Dipl. Inf. J?rg von Frantzius, Technical Director E-Mail joerg.frantzius at aperto.com Phone +49 30 283921-318 Fax +49 30 283921-29 Aperto GmbH ? An IBM Company Chausseestra?e 5, D-10115 Berlin http://www.aperto.com http://www.facebook.com/aperto https://www.xing.com/companies/apertoag HRB 77049 B, AG Berlin Charlottenburg Gesch?ftsf?hrer: Dirk Buddensiek, Kai Gro?mann, Stephan Haagen, Daniel Simon Am 01.12.2016 um 07:46 schrieb Sundararajan Athijegannathan >: Is there a simple reduced test case that we can use it to reproduce the issue you're seeing? Please send us the same and we'll file a bug (or you may do that as well). Thanks, -Sundar On 29/11/16, 11:11 PM, Frantzius, J?rg wrote: Hi, with JDK 1.8.0_112 (on Mac OS X) I?m running into the following error. When querying bugs.openjdk.java.net for "Current frame's stack size doesn't match stackmap?, I only found bugs dating from 2013, so this may not be known yet? Unfortunately I can?t see the Javascript file name or line number in the error message. The last known source location node/lib/fs.js:374 that I can step to in the Netbeans debugger is calling a constructor ?Buffer(size)?, which is likely this source: https://github.com/nodejs/node/blob/v0.12.7-release/lib/buffer.js#L48 Following is the error message: java.lang.VerifyError: Inconsistent stackmap frames at branch target 404 Exception Details: Location: jdk/nashorn/internal/scripts/Script$Recompilation$414$1806AA$\^function\_.L:1#Buffer(Ljdk/nashorn/internal/runtime/ScriptFunction;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @382: goto Reason: Current frame's stack size doesn't match stackmap. Current Frame: bci: @382 flags: { } locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } stack: { } Stackmap Frame: bci: @404 flags: { } locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } stack: { 'java/lang/Object' } Bytecode: 0x0000000: 2ab6 0014 3a04 1904 ba00 2000 0059 ba00 0x0000010: 2300 005f 2bba 0027 0000 9a00 1219 04ba 0x0000020: 002a 0000 2c2d ba00 2e00 00b0 1904 ba00 0x0000030: 2000 0059 ba00 3100 005f 2cba 0034 0000 0x0000040: 9900 102b 2cb8 003a ba00 3e00 00a7 00d5 0x0000050: 1904 ba00 2000 0059 ba00 4100 005f 2cba 0x0000060: 0044 0000 9900 4d19 04ba 0020 0000 59ba 0x0000070: 0041 0000 5f2d ba00 4400 0099 0012 2dba 0x0000080: 0047 0000 b800 4a03 8798 9a00 0912 4cb8 0x0000090: 0050 4e2b 1904 ba00 5500 0059 ba00 5800 0x00000a0: 005f 2c2d ba00 5c00 00ba 005f 0000 a700 0x00000b0: 7419 04ba 0020 0000 59ba 0062 0000 5f2c 0x00000c0: ba00 6500 0099 0043 2cba 0068 0000 126a 0x00000d0: b800 7099 0026 1904 ba00 2000 0059 ba00 0x00000e0: 7300 005f 2cba 0076 0000 ba00 7900 0099 0x00000f0: 000a 2cba 0076 0000 4d2b 2cba 007b 0000 0x0000100: ba00 3e00 00a7 001d 1904 b600 80ba 0083 0x0000110: 0000 1285 ba00 8900 0012 8b10 4507 b800 0x0000120: 91bf 2bba 0047 0000 1904 ba00 9400 00b8 0x0000130: 0097 9900 3a19 04b6 0080 ba00 9a00 0012 0x0000140: 9c19 04ba 0094 0000 59ba 009f 0000 5f10 0x0000150: 10ba 00a3 0000 b800 a712 a9b8 00a7 ba00 0x0000160: ac00 0012 8b10 4907 b800 91bf 2bba 007b 0x0000170: 0000 0e98 9c00 0d2b 03ba 00af 0000 a700 0x0000180: 162b 592b ba00 4700 00b8 00b3 b800 b7ba 0x0000190: 003e 0000 2b19 04b6 0080 ba00 ba00 00ba 0x00001a0: 00bd 0000 2bba 0047 0000 1904 ba00 5500 0x00001b0: 00ba 00c0 0000 047c b800 b75d 58b8 003a 0x00001c0: 5e58 989d 00c3 2bba 007b 0000 0e97 9e00 0x00001d0: b82b ba00 4700 0019 04ba 00c2 0000 1904 0x00001e0: ba00 c500 005f b800 3a5d 58b8 003a 675d 0x00001f0: 58b8 003a 5e58 979e 0013 1904 ba00 c800 0x0000200: 00b2 00cc ba00 d000 0057 2b19 04ba 00d3 0x0000210: 0000 b200 cc19 04ba 00d6 0000 2b19 04ba 0x0000220: 00c5 0000 1904 ba00 c500 002b ba00 4700 0x0000230: 00b8 00a7 ba00 da00 00ba 00bd 0000 1904 0x0000240: 1904 ba00 c500 002b ba00 4700 00b8 00a7 0x0000250: ba00 dd00 0019 04ba 00df 0000 1007 7e99 0x0000260: 0024 1904 1904 ba00 df00 0010 0780 ba00 0x0000270: e100 0019 0419 04ba 00e3 0000 0f63 ba00 0x0000280: e500 00a7 001a 1904 ba00 e800 00b2 00cc 0x0000290: 2b2b ba00 4700 00ba 00ec 0000 5719 04ba 0x00002a0: 0020 0000 59ba 0031 0000 5f2c ba00 3400 0x00002b0: 0099 0007 b200 ccb0 1904 ba00 2000 0059 0x00002c0: ba00 4100 005f 2cba 0044 0000 9900 8b2b 0x00002d0: 59ba 00ef 0000 5f2c 2dba 00f2 0000 3a05 0x00002e0: 1905 2bba 0047 0000 b800 f599 0064 2bba 0x00002f0: 0047 0000 3a06 2b19 05ba 005f 0000 1904 0x0000300: ba00 f800 00b2 00cc 2b2b ba00 4700 00ba 0x0000310: 00fb 0000 572b ba00 fe00 0019 04b6 0080 0x0000320: ba00 ba00 00b8 0101 9900 2419 0419 04ba 0x0000330: 00c5 0000 1906 b800 3a19 05b8 003a 675d 0x0000340: 58b8 003a 5e58 67ba 00e5 0000 a700 08b2 0x0000350: 00cc 3a06 a700 9e19 04ba 0020 0000 59ba 0x0000360: 0023 0000 5f2c ba00 2700 0099 001d 2c59 0x0000370: ba01 0400 005f 2b03 032b ba00 4700 00ba 0x0000380: 0108 0000 57a7 0063 1904 ba00 2000 0059 0x0000390: ba00 3100 005f 2cba 0047 0000 ba00 3400 0x00003a0: 009a 001a 1904 ba00 2000 0059 ba00 7300 0x00003b0: 005f 2cba 0079 0000 9900 3003 3607 1507 0x00003c0: 8739 0818 082b ba00 7b00 0098 9c00 1c2b 0x00003d0: 1808 2c18 08ba 010e 0000 ba01 1200 0018 0x00003e0: 080f 6339 08a7 ffde b200 cc3a 06b2 00cc 0x00003f0: 3a05 b200 ccb0 Stackmap Table: append_frame(@44,Object[#125]) same_frame(@80) same_frame(@141) same_frame(@147) same_frame(@177) same_frame_extended(@249) same_frame(@264) same_frame(@290) same_frame_extended(@364) same_frame(@385) same_locals_1_stack_item_frame(@404,Object[#286]) same_locals_1_stack_item_extended(@522,Object[#286]) same_locals_1_stack_item_extended(@643,Object[#286]) same_locals_1_stack_item_frame(@646,Object[#286]) same_locals_1_stack_item_frame(@669,Object[#286]) same_locals_1_stack_item_frame(@696,Object[#286]) full_frame(@844,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) full_frame(@847,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286]},{Object[#286]}) full_frame(@852,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) full_frame(@855,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) same_locals_1_stack_item_frame(@904,Object[#286]) same_locals_1_stack_item_frame(@955,Object[#286]) full_frame(@963,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Top,Top,Integer,Double},{Object[#286]}) full_frame(@1000,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) full_frame(@1010,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2583) at java.lang.Class.getDeclaredField(Class.java:2068) at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:209) at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:204) at java.security.AccessController.doPrivileged(Native Method) at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller.initialize(Context.java:204) at jdk.nashorn.internal.codegen.CompilationPhase$InstallPhase.transform(CompilationPhase.java:508) at jdk.nashorn.internal.codegen.CompilationPhase.apply(CompilationPhase.java:624) at jdk.nashorn.internal.codegen.Compiler.compile(Compiler.java:655) at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.compileTypeSpecialization(RecompilableScriptFunctionData.java:725) at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.getBest(RecompilableScriptFunctionData.java:905) at jdk.nashorn.internal.runtime.ScriptFunctionData.getBest(ScriptFunctionData.java:375) at jdk.nashorn.internal.runtime.ScriptFunctionData.getBestConstructor(ScriptFunctionData.java:247) at jdk.nashorn.internal.runtime.ScriptFunction.findNewMethod(ScriptFunction.java:758) at jdk.nashorn.internal.runtime.ScriptObject.lookup(ScriptObject.java:1827) at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:104) at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:98) at jdk.internal.dynalink.support.CompositeTypeBasedGuardingDynamicLinker.getGuardedInvocation(CompositeTypeBasedGuardingDynamicLinker.java:176) at jdk.internal.dynalink.support.CompositeGuardingDynamicLinker.getGuardedInvocation(CompositeGuardingDynamicLinker.java:124) at jdk.internal.dynalink.support.LinkerServicesImpl.getGuardedInvocation(LinkerServicesImpl.java:154) at jdk.internal.dynalink.DynamicLinker.relink(DynamicLinker.java:253) at jdk.nashorn.internal.scripts.Script$Recompilation$402$9044AA$\^function\_.L:1#readFileSync(node/lib/fs.js:374) [..] Any help or hints would be very appreciated! Regards, J?rg --- Dipl. Inf. J?rg von Frantzius, Technical Director E-Mail joerg.frantzius at aperto.com Phone +49 30 283921-318 Fax +49 30 283921-29 Aperto GmbH ? An IBM Company Chausseestra?e 5, D-10115 Berlin http://www.aperto.com http://www.facebook.com/aperto https://www.xing.com/companies/apertoag HRB 77049 B, AG Berlin Charlottenburg Vorstand: Dirk Buddensiek (Vorsitzender), Kai Gro?mann, Stephan Haagen, Daniel Simon Aufsichtsrat: Matthew Candy (Vorsitzender) From szegedia at gmail.com Thu Dec 1 12:01:46 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 13:01:46 +0100 Subject: java.lang.VerifyError: Inconsistent stackmap frames at branch target In-Reply-To: <561D7E3A-D986-4344-B765-3EFC25F86B17@aperto.com> References: <1457EC63-39E7-4AE7-BAB0-F77352D6743A@aperto.com> <583FC73F.8000309@oracle.com> <561D7E3A-D986-4344-B765-3EFC25F86B17@aperto.com> Message-ID: Running with assertions enabled shows that the error is in ?this.length >>>=0? expression on line 31. Reducing the testcase to just: (function (p) { if (p) { this.length >>>= 0; // Coerce to uint32. } })(false) also reproduces the problem (at least, the assertion; it will cause a somewhat different issue without assertions). Attila. > On 01 Dec 2016, at 12:18, Frantzius, J?rg wrote: > > Hi Sunda, > > you can reproduce by putting the following into a test.js file and executing it using jjs. > > ======== snip ============= > > function Buffer(subject, encoding) { > if (!util.isBuffer(this)) > return new Buffer(subject, encoding); > > if (util.isNumber(subject)) { > this.length = +subject; > > } else if (util.isString(subject)) { > if (!util.isString(encoding) || encoding.length === 0) > encoding = 'utf8'; > this.length = Buffer.byteLength(subject, encoding); > > // Handle Arrays, Buffers, Uint8Arrays or JSON. > } else if (util.isObject(subject)) { > if (subject.type === 'Buffer' && util.isArray(subject.data)) > subject = subject.data; > this.length = +subject.length; > > } else { > throw new TypeError('must start with number, buffer, array or string'); > } > > if (this.length > kMaxLength) { > throw new RangeError('Attempt to allocate Buffer larger than maximum ' + > 'size: 0x' + kMaxLength.toString(16) + ' bytes'); > } > > if (this.length < 0) > this.length = 0; > else > this.length >>>= 0; // Coerce to uint32. > > this.parent = undefined; > if (this.length <= (Buffer.poolSize >>> 1) && this.length > 0) { > if (this.length > poolSize - poolOffset) > createPool(); > this.parent = sliceOnto(allocPool, > this, > poolOffset, > poolOffset + this.length); > poolOffset += this.length; > > // Ensure aligned slices > if (poolOffset & 0x7) { > poolOffset |= 0x7; > poolOffset++; > } > } else { > alloc(this, this.length); > } > > if (util.isNumber(subject)) { > return; > } > > if (util.isString(subject)) { > // In the case of base64 it's possible that the size of the buffer > // allocated was slightly too large. In this case we need to rewrite > // the length to the actual length written. > var len = this.write(subject, encoding); > // Buffer was truncated after decode, realloc internal ExternalArray > if (len !== this.length) { > var prevLen = this.length; > this.length = len; > truncate(this, this.length); > // Only need to readjust the poolOffset if the allocation is a slice. > if (this.parent != undefined) > poolOffset -= (prevLen - len); > } > > } else if (util.isBuffer(subject)) { > subject.copy(this, 0, 0, this.length); > > } else if (util.isNumber(subject.length) || util.isArray(subject)) { > // Really crappy way to handle Uint8Arrays, but V8 doesn't give a simple > // way to access the data from the C++ API. > for (var i = 0; i < this.length; i++) > this[i] = subject[i]; > } > } > > new Buffer(1024); > > ========== snip ============== > > > I?ll hopefully get my OCA signed and sent in soon, so I can contribute in JIRA. > > Regards, > J?rg > > > --- > > Dipl. Inf. J?rg von Frantzius, Technical Director > > E-Mail joerg.frantzius at aperto.com > > Phone +49 30 283921-318 > Fax +49 30 283921-29 > > Aperto GmbH ? An IBM Company > Chausseestra?e 5, D-10115 Berlin > http://www.aperto.com > http://www.facebook.com/aperto > https://www.xing.com/companies/apertoag > > HRB 77049 B, AG Berlin Charlottenburg > Gesch?ftsf?hrer: Dirk Buddensiek, Kai Gro?mann, Stephan Haagen, Daniel Simon > > Am 01.12.2016 um 07:46 schrieb Sundararajan Athijegannathan >: > > Is there a simple reduced test case that we can use it to reproduce the issue you're seeing? Please send us the same and we'll file a bug (or you may do that as well). > > Thanks, > -Sundar > > On 29/11/16, 11:11 PM, Frantzius, J?rg wrote: > Hi, > > with JDK 1.8.0_112 (on Mac OS X) I?m running into the following error. When querying bugs.openjdk.java.net for "Current frame's stack size doesn't match stackmap?, I only found bugs dating from 2013, so this may not be known yet? > > Unfortunately I can?t see the Javascript file name or line number in the error message. The last known source location node/lib/fs.js:374 that I can step to in the Netbeans debugger is calling a constructor ?Buffer(size)?, which is likely this source: https://github.com/nodejs/node/blob/v0.12.7-release/lib/buffer.js#L48 > > Following is the error message: > > java.lang.VerifyError: Inconsistent stackmap frames at branch target 404 > Exception Details: > Location: > jdk/nashorn/internal/scripts/Script$Recompilation$414$1806AA$\^function\_.L:1#Buffer(Ljdk/nashorn/internal/runtime/ScriptFunction;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @382: goto > Reason: > Current frame's stack size doesn't match stackmap. > Current Frame: > bci: @382 > flags: { } > locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } > stack: { } > Stackmap Frame: > bci: @404 > flags: { } > locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } > stack: { 'java/lang/Object' } > Bytecode: > 0x0000000: 2ab6 0014 3a04 1904 ba00 2000 0059 ba00 > 0x0000010: 2300 005f 2bba 0027 0000 9a00 1219 04ba > 0x0000020: 002a 0000 2c2d ba00 2e00 00b0 1904 ba00 > 0x0000030: 2000 0059 ba00 3100 005f 2cba 0034 0000 > 0x0000040: 9900 102b 2cb8 003a ba00 3e00 00a7 00d5 > 0x0000050: 1904 ba00 2000 0059 ba00 4100 005f 2cba > 0x0000060: 0044 0000 9900 4d19 04ba 0020 0000 59ba > 0x0000070: 0041 0000 5f2d ba00 4400 0099 0012 2dba > 0x0000080: 0047 0000 b800 4a03 8798 9a00 0912 4cb8 > 0x0000090: 0050 4e2b 1904 ba00 5500 0059 ba00 5800 > 0x00000a0: 005f 2c2d ba00 5c00 00ba 005f 0000 a700 > 0x00000b0: 7419 04ba 0020 0000 59ba 0062 0000 5f2c > 0x00000c0: ba00 6500 0099 0043 2cba 0068 0000 126a > 0x00000d0: b800 7099 0026 1904 ba00 2000 0059 ba00 > 0x00000e0: 7300 005f 2cba 0076 0000 ba00 7900 0099 > 0x00000f0: 000a 2cba 0076 0000 4d2b 2cba 007b 0000 > 0x0000100: ba00 3e00 00a7 001d 1904 b600 80ba 0083 > 0x0000110: 0000 1285 ba00 8900 0012 8b10 4507 b800 > 0x0000120: 91bf 2bba 0047 0000 1904 ba00 9400 00b8 > 0x0000130: 0097 9900 3a19 04b6 0080 ba00 9a00 0012 > 0x0000140: 9c19 04ba 0094 0000 59ba 009f 0000 5f10 > 0x0000150: 10ba 00a3 0000 b800 a712 a9b8 00a7 ba00 > 0x0000160: ac00 0012 8b10 4907 b800 91bf 2bba 007b > 0x0000170: 0000 0e98 9c00 0d2b 03ba 00af 0000 a700 > 0x0000180: 162b 592b ba00 4700 00b8 00b3 b800 b7ba > 0x0000190: 003e 0000 2b19 04b6 0080 ba00 ba00 00ba > 0x00001a0: 00bd 0000 2bba 0047 0000 1904 ba00 5500 > 0x00001b0: 00ba 00c0 0000 047c b800 b75d 58b8 003a > 0x00001c0: 5e58 989d 00c3 2bba 007b 0000 0e97 9e00 > 0x00001d0: b82b ba00 4700 0019 04ba 00c2 0000 1904 > 0x00001e0: ba00 c500 005f b800 3a5d 58b8 003a 675d > 0x00001f0: 58b8 003a 5e58 979e 0013 1904 ba00 c800 > 0x0000200: 00b2 00cc ba00 d000 0057 2b19 04ba 00d3 > 0x0000210: 0000 b200 cc19 04ba 00d6 0000 2b19 04ba > 0x0000220: 00c5 0000 1904 ba00 c500 002b ba00 4700 > 0x0000230: 00b8 00a7 ba00 da00 00ba 00bd 0000 1904 > 0x0000240: 1904 ba00 c500 002b ba00 4700 00b8 00a7 > 0x0000250: ba00 dd00 0019 04ba 00df 0000 1007 7e99 > 0x0000260: 0024 1904 1904 ba00 df00 0010 0780 ba00 > 0x0000270: e100 0019 0419 04ba 00e3 0000 0f63 ba00 > 0x0000280: e500 00a7 001a 1904 ba00 e800 00b2 00cc > 0x0000290: 2b2b ba00 4700 00ba 00ec 0000 5719 04ba > 0x00002a0: 0020 0000 59ba 0031 0000 5f2c ba00 3400 > 0x00002b0: 0099 0007 b200 ccb0 1904 ba00 2000 0059 > 0x00002c0: ba00 4100 005f 2cba 0044 0000 9900 8b2b > 0x00002d0: 59ba 00ef 0000 5f2c 2dba 00f2 0000 3a05 > 0x00002e0: 1905 2bba 0047 0000 b800 f599 0064 2bba > 0x00002f0: 0047 0000 3a06 2b19 05ba 005f 0000 1904 > 0x0000300: ba00 f800 00b2 00cc 2b2b ba00 4700 00ba > 0x0000310: 00fb 0000 572b ba00 fe00 0019 04b6 0080 > 0x0000320: ba00 ba00 00b8 0101 9900 2419 0419 04ba > 0x0000330: 00c5 0000 1906 b800 3a19 05b8 003a 675d > 0x0000340: 58b8 003a 5e58 67ba 00e5 0000 a700 08b2 > 0x0000350: 00cc 3a06 a700 9e19 04ba 0020 0000 59ba > 0x0000360: 0023 0000 5f2c ba00 2700 0099 001d 2c59 > 0x0000370: ba01 0400 005f 2b03 032b ba00 4700 00ba > 0x0000380: 0108 0000 57a7 0063 1904 ba00 2000 0059 > 0x0000390: ba00 3100 005f 2cba 0047 0000 ba00 3400 > 0x00003a0: 009a 001a 1904 ba00 2000 0059 ba00 7300 > 0x00003b0: 005f 2cba 0079 0000 9900 3003 3607 1507 > 0x00003c0: 8739 0818 082b ba00 7b00 0098 9c00 1c2b > 0x00003d0: 1808 2c18 08ba 010e 0000 ba01 1200 0018 > 0x00003e0: 080f 6339 08a7 ffde b200 cc3a 06b2 00cc > 0x00003f0: 3a05 b200 ccb0 > Stackmap Table: > append_frame(@44,Object[#125]) > same_frame(@80) > same_frame(@141) > same_frame(@147) > same_frame(@177) > same_frame_extended(@249) > same_frame(@264) > same_frame(@290) > same_frame_extended(@364) > same_frame(@385) > same_locals_1_stack_item_frame(@404,Object[#286]) > same_locals_1_stack_item_extended(@522,Object[#286]) > same_locals_1_stack_item_extended(@643,Object[#286]) > same_locals_1_stack_item_frame(@646,Object[#286]) > same_locals_1_stack_item_frame(@669,Object[#286]) > same_locals_1_stack_item_frame(@696,Object[#286]) > full_frame(@844,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > full_frame(@847,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286]},{Object[#286]}) > full_frame(@852,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > full_frame(@855,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) > same_locals_1_stack_item_frame(@904,Object[#286]) > same_locals_1_stack_item_frame(@955,Object[#286]) > full_frame(@963,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Top,Top,Integer,Double},{Object[#286]}) > full_frame(@1000,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) > full_frame(@1010,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) > > at java.lang.Class.getDeclaredFields0(Native Method) > at java.lang.Class.privateGetDeclaredFields(Class.java:2583) > at java.lang.Class.getDeclaredField(Class.java:2068) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:209) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:204) > at java.security.AccessController.doPrivileged(Native Method) > at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller.initialize(Context.java:204) > at jdk.nashorn.internal.codegen.CompilationPhase$InstallPhase.transform(CompilationPhase.java:508) > at jdk.nashorn.internal.codegen.CompilationPhase.apply(CompilationPhase.java:624) > at jdk.nashorn.internal.codegen.Compiler.compile(Compiler.java:655) > at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.compileTypeSpecialization(RecompilableScriptFunctionData.java:725) > at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.getBest(RecompilableScriptFunctionData.java:905) > at jdk.nashorn.internal.runtime.ScriptFunctionData.getBest(ScriptFunctionData.java:375) > at jdk.nashorn.internal.runtime.ScriptFunctionData.getBestConstructor(ScriptFunctionData.java:247) > at jdk.nashorn.internal.runtime.ScriptFunction.findNewMethod(ScriptFunction.java:758) > at jdk.nashorn.internal.runtime.ScriptObject.lookup(ScriptObject.java:1827) > at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:104) > at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:98) > at jdk.internal.dynalink.support.CompositeTypeBasedGuardingDynamicLinker.getGuardedInvocation(CompositeTypeBasedGuardingDynamicLinker.java:176) > at jdk.internal.dynalink.support.CompositeGuardingDynamicLinker.getGuardedInvocation(CompositeGuardingDynamicLinker.java:124) > at jdk.internal.dynalink.support.LinkerServicesImpl.getGuardedInvocation(LinkerServicesImpl.java:154) > at jdk.internal.dynalink.DynamicLinker.relink(DynamicLinker.java:253) > at jdk.nashorn.internal.scripts.Script$Recompilation$402$9044AA$\^function\_.L:1#readFileSync(node/lib/fs.js:374) > [..] > > Any help or hints would be very appreciated! > > Regards, > J?rg > > --- > > Dipl. Inf. J?rg von Frantzius, Technical Director > > E-Mail joerg.frantzius at aperto.com > > Phone +49 30 283921-318 > Fax +49 30 283921-29 > > Aperto GmbH ? An IBM Company > Chausseestra?e 5, D-10115 Berlin > http://www.aperto.com > http://www.facebook.com/aperto > https://www.xing.com/companies/apertoag > > HRB 77049 B, AG Berlin Charlottenburg > Vorstand: Dirk Buddensiek (Vorsitzender), Kai Gro?mann, Stephan Haagen, Daniel Simon > Aufsichtsrat: Matthew Candy (Vorsitzender) > > From sundararajan.athijegannathan at oracle.com Thu Dec 1 12:48:15 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 18:18:15 +0530 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> References: <583FC15E.4060509@oracle.com> <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> Message-ID: <58401C0F.1070108@oracle.com> Good catch Hannes! Please review the updated webrev : http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ PS. Had to use Function.prototype.call.call to pass undefined this explicitly (as JSObject.call can't be called from script). -Sundar On 01/12/16, 3:13 PM, Hannes Walln?fer wrote: > Hi Sundar, > > The problem with this approach is that it will replace any occurrence of undefined this with the global object. However, this should only occur for scope calls. For example, the following call would see undefined replaced with global: > > func.call(undefined) > > This is probably not a problem that will occur very often, but ideally I think we should do the check and replacement on the linking side, i.e. in JSObjectLinker.findCallMethod. > > On the other hand we can?t check for function strictness that way. Maybe do it your way but add a boolean isScope parameter and bind that at link time? > > Hannes > > >> Am 01.12.2016 um 07:21 schrieb Sundararajan Athijegannathan: >> >> Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 >> >> Thanks, >> -Sundar From hannes.wallnoefer at oracle.com Thu Dec 1 12:47:49 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 1 Dec 2016 13:47:49 +0100 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: <58401C0F.1070108@oracle.com> References: <583FC15E.4060509@oracle.com> <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> <58401C0F.1070108@oracle.com> Message-ID: +1 Nice solution, of course much simpler than what I proposed below :) Hannes > Am 01.12.2016 um 13:48 schrieb Sundararajan Athijegannathan : > > Good catch Hannes! Please review the updated webrev : http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ > > PS. Had to use Function.prototype.call.call to pass undefined this explicitly (as JSObject.call can't be called from script). > > -Sundar > > On 01/12/16, 3:13 PM, Hannes Walln?fer wrote: >> Hi Sundar, >> >> The problem with this approach is that it will replace any occurrence of undefined this with the global object. However, this should only occur for scope calls. For example, the following call would see undefined replaced with global: >> >> func.call(undefined) >> >> This is probably not a problem that will occur very often, but ideally I think we should do the check and replacement on the linking side, i.e. in JSObjectLinker.findCallMethod. >> >> On the other hand we can?t check for function strictness that way. Maybe do it your way but add a boolean isScope parameter and bind that at link time? >> >> Hannes >> >> >>> Am 01.12.2016 um 07:21 schrieb Sundararajan Athijegannathan: >>> >>> Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 >>> >>> Thanks, >>> -Sundar From james.laskey at oracle.com Thu Dec 1 12:50:57 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 1 Dec 2016 08:50:57 -0400 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: <58401C0F.1070108@oracle.com> References: <583FC15E.4060509@oracle.com> <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> <58401C0F.1070108@oracle.com> Message-ID: +1 > On Dec 1, 2016, at 8:48 AM, Sundararajan Athijegannathan wrote: > > Good catch Hannes! Please review the updated webrev : http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ > > PS. Had to use Function.prototype.call.call to pass undefined this explicitly (as JSObject.call can't be called from script). > > -Sundar > > On 01/12/16, 3:13 PM, Hannes Walln?fer wrote: >> Hi Sundar, >> >> The problem with this approach is that it will replace any occurrence of undefined this with the global object. However, this should only occur for scope calls. For example, the following call would see undefined replaced with global: >> >> func.call(undefined) >> >> This is probably not a problem that will occur very often, but ideally I think we should do the check and replacement on the linking side, i.e. in JSObjectLinker.findCallMethod. >> >> On the other hand we can?t check for function strictness that way. Maybe do it your way but add a boolean isScope parameter and bind that at link time? >> >> Hannes >> >> >>> Am 01.12.2016 um 07:21 schrieb Sundararajan Athijegannathan: >>> >>> Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 >>> >>> Thanks, >>> -Sundar From szegedia at gmail.com Thu Dec 1 12:53:17 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 13:53:17 +0100 Subject: Error Stack Column number In-Reply-To: <583FC6DE.4080407@oracle.com> References: <583FC6DE.4080407@oracle.com> Message-ID: <1EB9E6C5-65F3-4ABF-BB52-F0609C338680@gmail.com> Okay, I tracked this down to an incorrect optimization in code generator for when you use >>=0 to coerce to an uint32 :-( There is a special case for exactly this usage, where the right-hand-side operand is a literal 0, so we avoid emitting a no-op ?ICONST_0; IUSHR? bytecode sequence. It unfortunately wasn?t working correctly for when the left-hand-side operand of a compound-assignment-shift-right was either a property or element access expression (either ?obj.foo? or ?obj[x]?) Decomposing the compound assignment into an explicit shift and assignment would work around the problem for now: this.length = this.length >>> 0; // Coerce to uint32. I filed > and have a fix for it. Attila. > On 01 Dec 2016, at 07:44, Sundararajan Athijegannathan wrote: > > We do not aim to provide complete compatibility with other JS implementations on the non-standard properties such as "stack". stack tries to mimic whatever is done for Java code (no column number for eg.). But, as you've noted there are enough information on Error objects via other properties like lineNumber, columnNumber, fileName. It should be possible to write a simple utility function to format stack string as desired for specific applications. > > Thanks, > -Sundar > > On 01/12/16, 3:32 AM, Art Fiedler wrote: >> When making an implementation of console.count([label]) part of the >> Mozilla's developer docs >> https://developer.mozilla.org/en-US/docs/Web/API/Console/count >> mentions when label is omitted, the function will count the number of times >> count() was called >> on that line... however, if the javascript code has been minified all lines >> of code may be on the >> same line... messing up all console.count() calls... >> >> I was providing that implementation by using new Error().stack and parsing >> the current file:line... >> however I could fix the minify issue if new Error().stack output the column >> also for instance... >> other browser seem to be doing file:line:column in the stack trace. >> Including nodejs see: >> https://nodejs.org/api/errors.html#errors_error_stack >> >> Microsoft Edge's dev console outputs... >> new Error().stack: >> "Error >> at eval code (eval code:1:1)" >> Firefox since FF30: >> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/stack >> >> Currently nashorn error stacks only include the line number... >> public static void main(String[] args) throws Exception { >> NashornScriptEngineFactory factory = new >> NashornScriptEngineFactory(); >> ScriptEngine scriptEngine = factory.getScriptEngine(); >> scriptEngine.put(ScriptEngine.FILENAME, "myfile.js"); >> scriptEngine.eval("print('fileName: ' + new Error().fileName);"); >> scriptEngine.eval("print('lineNumber: ' + new >> Error().lineNumber);"); >> scriptEngine.eval("print('columnNumber: ' + new >> Error().columnNumber);"); >> scriptEngine.eval("print('stack: ' + new Error().stack);"); >> } >> /* >> fileName: myfile.js >> lineNumber: 1 >> columnNumber: -1 >> stack: Error >> at (myfile.js:1) >> */ >> >> Would be nice to add the column number to the stacks and error object in >> nashorn. >> >> Thanks, >> Arthur Fiedler From szegedia at gmail.com Thu Dec 1 12:55:17 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 13:55:17 +0100 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: <58401C0F.1070108@oracle.com> References: <583FC15E.4060509@oracle.com> <45D7C5A6-1763-461F-89C1-0E4178680E7F@oracle.com> <58401C0F.1070108@oracle.com> Message-ID: +1 > On 01 Dec 2016, at 13:48, Sundararajan Athijegannathan wrote: > > Good catch Hannes! Please review the updated webrev : http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ > > PS. Had to use Function.prototype.call.call to pass undefined this explicitly (as JSObject.call can't be called from script). > > -Sundar > > On 01/12/16, 3:13 PM, Hannes Walln?fer wrote: >> Hi Sundar, >> >> The problem with this approach is that it will replace any occurrence of undefined this with the global object. However, this should only occur for scope calls. For example, the following call would see undefined replaced with global: >> >> func.call(undefined) >> >> This is probably not a problem that will occur very often, but ideally I think we should do the check and replacement on the linking side, i.e. in JSObjectLinker.findCallMethod. >> >> On the other hand we can?t check for function strictness that way. Maybe do it your way but add a boolean isScope parameter and bind that at link time? >> >> Hannes >> >> >>> Am 01.12.2016 um 07:21 schrieb Sundararajan Athijegannathan: >>> >>> Please review http://cr.openjdk.java.net/~sundar/8170565/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8170565 >>> >>> Thanks, >>> -Sundar From szegedia at gmail.com Thu Dec 1 12:56:46 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 13:56:46 +0100 Subject: Error Stack Column number In-Reply-To: <1EB9E6C5-65F3-4ABF-BB52-F0609C338680@gmail.com> References: <583FC6DE.4080407@oracle.com> <1EB9E6C5-65F3-4ABF-BB52-F0609C338680@gmail.com> Message-ID: <13F5248D-A90D-4F8D-9F21-A8B993AD6BFE@gmail.com> Sorry, disregard the message below, it was meant for a different e-mail thread :-) > On 01 Dec 2016, at 13:53, Attila Szegedi wrote: > > Okay, I tracked this down to an incorrect optimization in code generator for when you use >>=0 to coerce to an uint32 :-( > > There is a special case for exactly this usage, where the right-hand-side operand is a literal 0, so we avoid emitting a no-op ?ICONST_0; IUSHR? bytecode sequence. It unfortunately wasn?t working correctly for when the left-hand-side operand of a compound-assignment-shift-right was either a property or element access expression (either ?obj.foo? or ?obj[x]?) > > Decomposing the compound assignment into an explicit shift and assignment would work around the problem for now: > > this.length = this.length >>> 0; // Coerce to uint32. > > I filed > and have a fix for it. > > Attila. > >> On 01 Dec 2016, at 07:44, Sundararajan Athijegannathan > wrote: >> >> We do not aim to provide complete compatibility with other JS implementations on the non-standard properties such as "stack". stack tries to mimic whatever is done for Java code (no column number for eg.). But, as you've noted there are enough information on Error objects via other properties like lineNumber, columnNumber, fileName. It should be possible to write a simple utility function to format stack string as desired for specific applications. >> >> Thanks, >> -Sundar >> >> On 01/12/16, 3:32 AM, Art Fiedler wrote: >>> When making an implementation of console.count([label]) part of the >>> Mozilla's developer docs >>> https://developer.mozilla.org/en-US/docs/Web/API/Console/count >>> mentions when label is omitted, the function will count the number of times >>> count() was called >>> on that line... however, if the javascript code has been minified all lines >>> of code may be on the >>> same line... messing up all console.count() calls... >>> >>> I was providing that implementation by using new Error().stack and parsing >>> the current file:line... >>> however I could fix the minify issue if new Error().stack output the column >>> also for instance... >>> other browser seem to be doing file:line:column in the stack trace. >>> Including nodejs see: >>> https://nodejs.org/api/errors.html#errors_error_stack >>> >>> Microsoft Edge's dev console outputs... >>> new Error().stack: >>> "Error >>> at eval code (eval code:1:1)" >>> Firefox since FF30: >>> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/stack >>> >>> Currently nashorn error stacks only include the line number... >>> public static void main(String[] args) throws Exception { >>> NashornScriptEngineFactory factory = new >>> NashornScriptEngineFactory(); >>> ScriptEngine scriptEngine = factory.getScriptEngine(); >>> scriptEngine.put(ScriptEngine.FILENAME, "myfile.js"); >>> scriptEngine.eval("print('fileName: ' + new Error().fileName);"); >>> scriptEngine.eval("print('lineNumber: ' + new >>> Error().lineNumber);"); >>> scriptEngine.eval("print('columnNumber: ' + new >>> Error().columnNumber);"); >>> scriptEngine.eval("print('stack: ' + new Error().stack);"); >>> } >>> /* >>> fileName: myfile.js >>> lineNumber: 1 >>> columnNumber: -1 >>> stack: Error >>> at (myfile.js:1) >>> */ >>> >>> Would be nice to add the column number to the stacks and error object in >>> nashorn. >>> >>> Thanks, >>> Arthur Fiedler > From szegedia at gmail.com Thu Dec 1 12:57:10 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 13:57:10 +0100 Subject: java.lang.VerifyError: Inconsistent stackmap frames at branch target In-Reply-To: References: <1457EC63-39E7-4AE7-BAB0-F77352D6743A@aperto.com> <583FC73F.8000309@oracle.com> <561D7E3A-D986-4344-B765-3EFC25F86B17@aperto.com> Message-ID: <1A6F811C-11C4-4985-A727-3F4E52E07DFE@gmail.com> Okay, I tracked this down to an incorrect optimization in code generator for when you use >>=0 to coerce to an uint32 :-( There is a special case for exactly this usage, where the right-hand-side operand is a literal 0, so we avoid emitting a no-op ?ICONST_0; IUSHR? bytecode sequence. It unfortunately wasn?t working correctly for when the left-hand-side operand of a compound-assignment-shift-right was either a property or element access expression (either ?obj.foo? or ?obj[x]?) Decomposing the compound assignment into an explicit shift and assignment would work around the problem for now: this.length = this.length >>> 0; // Coerce to uint32. I filed > and have a fix for it. Attila. > On 01 Dec 2016, at 13:01, Attila Szegedi wrote: > > Running with assertions enabled shows that the error is in ?this.length >>>=0? expression on line 31. Reducing the testcase to just: > > (function (p) { > if (p) { > this.length >>>= 0; // Coerce to uint32. > } > })(false) > > also reproduces the problem (at least, the assertion; it will cause a somewhat different issue without assertions). > > Attila. > >> On 01 Dec 2016, at 12:18, Frantzius, J?rg wrote: >> >> Hi Sunda, >> >> you can reproduce by putting the following into a test.js file and executing it using jjs. >> >> ======== snip ============= >> >> function Buffer(subject, encoding) { >> if (!util.isBuffer(this)) >> return new Buffer(subject, encoding); >> >> if (util.isNumber(subject)) { >> this.length = +subject; >> >> } else if (util.isString(subject)) { >> if (!util.isString(encoding) || encoding.length === 0) >> encoding = 'utf8'; >> this.length = Buffer.byteLength(subject, encoding); >> >> // Handle Arrays, Buffers, Uint8Arrays or JSON. >> } else if (util.isObject(subject)) { >> if (subject.type === 'Buffer' && util.isArray(subject.data)) >> subject = subject.data; >> this.length = +subject.length; >> >> } else { >> throw new TypeError('must start with number, buffer, array or string'); >> } >> >> if (this.length > kMaxLength) { >> throw new RangeError('Attempt to allocate Buffer larger than maximum ' + >> 'size: 0x' + kMaxLength.toString(16) + ' bytes'); >> } >> >> if (this.length < 0) >> this.length = 0; >> else >> this.length >>>= 0; // Coerce to uint32. >> >> this.parent = undefined; >> if (this.length <= (Buffer.poolSize >>> 1) && this.length > 0) { >> if (this.length > poolSize - poolOffset) >> createPool(); >> this.parent = sliceOnto(allocPool, >> this, >> poolOffset, >> poolOffset + this.length); >> poolOffset += this.length; >> >> // Ensure aligned slices >> if (poolOffset & 0x7) { >> poolOffset |= 0x7; >> poolOffset++; >> } >> } else { >> alloc(this, this.length); >> } >> >> if (util.isNumber(subject)) { >> return; >> } >> >> if (util.isString(subject)) { >> // In the case of base64 it's possible that the size of the buffer >> // allocated was slightly too large. In this case we need to rewrite >> // the length to the actual length written. >> var len = this.write(subject, encoding); >> // Buffer was truncated after decode, realloc internal ExternalArray >> if (len !== this.length) { >> var prevLen = this.length; >> this.length = len; >> truncate(this, this.length); >> // Only need to readjust the poolOffset if the allocation is a slice. >> if (this.parent != undefined) >> poolOffset -= (prevLen - len); >> } >> >> } else if (util.isBuffer(subject)) { >> subject.copy(this, 0, 0, this.length); >> >> } else if (util.isNumber(subject.length) || util.isArray(subject)) { >> // Really crappy way to handle Uint8Arrays, but V8 doesn't give a simple >> // way to access the data from the C++ API. >> for (var i = 0; i < this.length; i++) >> this[i] = subject[i]; >> } >> } >> >> new Buffer(1024); >> >> ========== snip ============== >> >> >> I?ll hopefully get my OCA signed and sent in soon, so I can contribute in JIRA. >> >> Regards, >> J?rg >> >> >> --- >> >> Dipl. Inf. J?rg von Frantzius, Technical Director >> >> E-Mail joerg.frantzius at aperto.com >> >> Phone +49 30 283921-318 >> Fax +49 30 283921-29 >> >> Aperto GmbH ? An IBM Company >> Chausseestra?e 5, D-10115 Berlin >> http://www.aperto.com >> http://www.facebook.com/aperto >> https://www.xing.com/companies/apertoag >> >> HRB 77049 B, AG Berlin Charlottenburg >> Gesch?ftsf?hrer: Dirk Buddensiek, Kai Gro?mann, Stephan Haagen, Daniel Simon >> >> Am 01.12.2016 um 07:46 schrieb Sundararajan Athijegannathan >: >> >> Is there a simple reduced test case that we can use it to reproduce the issue you're seeing? Please send us the same and we'll file a bug (or you may do that as well). >> >> Thanks, >> -Sundar >> >> On 29/11/16, 11:11 PM, Frantzius, J?rg wrote: >> Hi, >> >> with JDK 1.8.0_112 (on Mac OS X) I?m running into the following error. When querying bugs.openjdk.java.net for "Current frame's stack size doesn't match stackmap?, I only found bugs dating from 2013, so this may not be known yet? >> >> Unfortunately I can?t see the Javascript file name or line number in the error message. The last known source location node/lib/fs.js:374 that I can step to in the Netbeans debugger is calling a constructor ?Buffer(size)?, which is likely this source: https://github.com/nodejs/node/blob/v0.12.7-release/lib/buffer.js#L48 >> >> Following is the error message: >> >> java.lang.VerifyError: Inconsistent stackmap frames at branch target 404 >> Exception Details: >> Location: >> jdk/nashorn/internal/scripts/Script$Recompilation$414$1806AA$\^function\_.L:1#Buffer(Ljdk/nashorn/internal/runtime/ScriptFunction;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @382: goto >> Reason: >> Current frame's stack size doesn't match stackmap. >> Current Frame: >> bci: @382 >> flags: { } >> locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } >> stack: { } >> Stackmap Frame: >> bci: @404 >> flags: { } >> locals: { 'jdk/nashorn/internal/runtime/ScriptFunction', 'java/lang/Object', 'java/lang/Object', 'java/lang/Object', 'jdk/nashorn/internal/runtime/ScriptObject' } >> stack: { 'java/lang/Object' } >> Bytecode: >> 0x0000000: 2ab6 0014 3a04 1904 ba00 2000 0059 ba00 >> 0x0000010: 2300 005f 2bba 0027 0000 9a00 1219 04ba >> 0x0000020: 002a 0000 2c2d ba00 2e00 00b0 1904 ba00 >> 0x0000030: 2000 0059 ba00 3100 005f 2cba 0034 0000 >> 0x0000040: 9900 102b 2cb8 003a ba00 3e00 00a7 00d5 >> 0x0000050: 1904 ba00 2000 0059 ba00 4100 005f 2cba >> 0x0000060: 0044 0000 9900 4d19 04ba 0020 0000 59ba >> 0x0000070: 0041 0000 5f2d ba00 4400 0099 0012 2dba >> 0x0000080: 0047 0000 b800 4a03 8798 9a00 0912 4cb8 >> 0x0000090: 0050 4e2b 1904 ba00 5500 0059 ba00 5800 >> 0x00000a0: 005f 2c2d ba00 5c00 00ba 005f 0000 a700 >> 0x00000b0: 7419 04ba 0020 0000 59ba 0062 0000 5f2c >> 0x00000c0: ba00 6500 0099 0043 2cba 0068 0000 126a >> 0x00000d0: b800 7099 0026 1904 ba00 2000 0059 ba00 >> 0x00000e0: 7300 005f 2cba 0076 0000 ba00 7900 0099 >> 0x00000f0: 000a 2cba 0076 0000 4d2b 2cba 007b 0000 >> 0x0000100: ba00 3e00 00a7 001d 1904 b600 80ba 0083 >> 0x0000110: 0000 1285 ba00 8900 0012 8b10 4507 b800 >> 0x0000120: 91bf 2bba 0047 0000 1904 ba00 9400 00b8 >> 0x0000130: 0097 9900 3a19 04b6 0080 ba00 9a00 0012 >> 0x0000140: 9c19 04ba 0094 0000 59ba 009f 0000 5f10 >> 0x0000150: 10ba 00a3 0000 b800 a712 a9b8 00a7 ba00 >> 0x0000160: ac00 0012 8b10 4907 b800 91bf 2bba 007b >> 0x0000170: 0000 0e98 9c00 0d2b 03ba 00af 0000 a700 >> 0x0000180: 162b 592b ba00 4700 00b8 00b3 b800 b7ba >> 0x0000190: 003e 0000 2b19 04b6 0080 ba00 ba00 00ba >> 0x00001a0: 00bd 0000 2bba 0047 0000 1904 ba00 5500 >> 0x00001b0: 00ba 00c0 0000 047c b800 b75d 58b8 003a >> 0x00001c0: 5e58 989d 00c3 2bba 007b 0000 0e97 9e00 >> 0x00001d0: b82b ba00 4700 0019 04ba 00c2 0000 1904 >> 0x00001e0: ba00 c500 005f b800 3a5d 58b8 003a 675d >> 0x00001f0: 58b8 003a 5e58 979e 0013 1904 ba00 c800 >> 0x0000200: 00b2 00cc ba00 d000 0057 2b19 04ba 00d3 >> 0x0000210: 0000 b200 cc19 04ba 00d6 0000 2b19 04ba >> 0x0000220: 00c5 0000 1904 ba00 c500 002b ba00 4700 >> 0x0000230: 00b8 00a7 ba00 da00 00ba 00bd 0000 1904 >> 0x0000240: 1904 ba00 c500 002b ba00 4700 00b8 00a7 >> 0x0000250: ba00 dd00 0019 04ba 00df 0000 1007 7e99 >> 0x0000260: 0024 1904 1904 ba00 df00 0010 0780 ba00 >> 0x0000270: e100 0019 0419 04ba 00e3 0000 0f63 ba00 >> 0x0000280: e500 00a7 001a 1904 ba00 e800 00b2 00cc >> 0x0000290: 2b2b ba00 4700 00ba 00ec 0000 5719 04ba >> 0x00002a0: 0020 0000 59ba 0031 0000 5f2c ba00 3400 >> 0x00002b0: 0099 0007 b200 ccb0 1904 ba00 2000 0059 >> 0x00002c0: ba00 4100 005f 2cba 0044 0000 9900 8b2b >> 0x00002d0: 59ba 00ef 0000 5f2c 2dba 00f2 0000 3a05 >> 0x00002e0: 1905 2bba 0047 0000 b800 f599 0064 2bba >> 0x00002f0: 0047 0000 3a06 2b19 05ba 005f 0000 1904 >> 0x0000300: ba00 f800 00b2 00cc 2b2b ba00 4700 00ba >> 0x0000310: 00fb 0000 572b ba00 fe00 0019 04b6 0080 >> 0x0000320: ba00 ba00 00b8 0101 9900 2419 0419 04ba >> 0x0000330: 00c5 0000 1906 b800 3a19 05b8 003a 675d >> 0x0000340: 58b8 003a 5e58 67ba 00e5 0000 a700 08b2 >> 0x0000350: 00cc 3a06 a700 9e19 04ba 0020 0000 59ba >> 0x0000360: 0023 0000 5f2c ba00 2700 0099 001d 2c59 >> 0x0000370: ba01 0400 005f 2b03 032b ba00 4700 00ba >> 0x0000380: 0108 0000 57a7 0063 1904 ba00 2000 0059 >> 0x0000390: ba00 3100 005f 2cba 0047 0000 ba00 3400 >> 0x00003a0: 009a 001a 1904 ba00 2000 0059 ba00 7300 >> 0x00003b0: 005f 2cba 0079 0000 9900 3003 3607 1507 >> 0x00003c0: 8739 0818 082b ba00 7b00 0098 9c00 1c2b >> 0x00003d0: 1808 2c18 08ba 010e 0000 ba01 1200 0018 >> 0x00003e0: 080f 6339 08a7 ffde b200 cc3a 06b2 00cc >> 0x00003f0: 3a05 b200 ccb0 >> Stackmap Table: >> append_frame(@44,Object[#125]) >> same_frame(@80) >> same_frame(@141) >> same_frame(@147) >> same_frame(@177) >> same_frame_extended(@249) >> same_frame(@264) >> same_frame(@290) >> same_frame_extended(@364) >> same_frame(@385) >> same_locals_1_stack_item_frame(@404,Object[#286]) >> same_locals_1_stack_item_extended(@522,Object[#286]) >> same_locals_1_stack_item_extended(@643,Object[#286]) >> same_locals_1_stack_item_frame(@646,Object[#286]) >> same_locals_1_stack_item_frame(@669,Object[#286]) >> same_locals_1_stack_item_frame(@696,Object[#286]) >> full_frame(@844,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) >> full_frame(@847,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286]},{Object[#286]}) >> full_frame(@852,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) >> full_frame(@855,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) >> same_locals_1_stack_item_frame(@904,Object[#286]) >> same_locals_1_stack_item_frame(@955,Object[#286]) >> full_frame(@963,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Top,Top,Integer,Double},{Object[#286]}) >> full_frame(@1000,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125]},{Object[#286]}) >> full_frame(@1010,{Object[#16],Object[#286],Object[#286],Object[#286],Object[#125],Object[#286],Object[#286]},{Object[#286]}) >> >> at java.lang.Class.getDeclaredFields0(Native Method) >> at java.lang.Class.privateGetDeclaredFields(Class.java:2583) >> at java.lang.Class.getDeclaredField(Class.java:2068) >> at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:209) >> at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller$1.run(Context.java:204) >> at java.security.AccessController.doPrivileged(Native Method) >> at jdk.nashorn.internal.runtime.Context$ContextCodeInstaller.initialize(Context.java:204) >> at jdk.nashorn.internal.codegen.CompilationPhase$InstallPhase.transform(CompilationPhase.java:508) >> at jdk.nashorn.internal.codegen.CompilationPhase.apply(CompilationPhase.java:624) >> at jdk.nashorn.internal.codegen.Compiler.compile(Compiler.java:655) >> at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.compileTypeSpecialization(RecompilableScriptFunctionData.java:725) >> at jdk.nashorn.internal.runtime.RecompilableScriptFunctionData.getBest(RecompilableScriptFunctionData.java:905) >> at jdk.nashorn.internal.runtime.ScriptFunctionData.getBest(ScriptFunctionData.java:375) >> at jdk.nashorn.internal.runtime.ScriptFunctionData.getBestConstructor(ScriptFunctionData.java:247) >> at jdk.nashorn.internal.runtime.ScriptFunction.findNewMethod(ScriptFunction.java:758) >> at jdk.nashorn.internal.runtime.ScriptObject.lookup(ScriptObject.java:1827) >> at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:104) >> at jdk.nashorn.internal.runtime.linker.NashornLinker.getGuardedInvocation(NashornLinker.java:98) >> at jdk.internal.dynalink.support.CompositeTypeBasedGuardingDynamicLinker.getGuardedInvocation(CompositeTypeBasedGuardingDynamicLinker.java:176) >> at jdk.internal.dynalink.support.CompositeGuardingDynamicLinker.getGuardedInvocation(CompositeGuardingDynamicLinker.java:124) >> at jdk.internal.dynalink.support.LinkerServicesImpl.getGuardedInvocation(LinkerServicesImpl.java:154) >> at jdk.internal.dynalink.DynamicLinker.relink(DynamicLinker.java:253) >> at jdk.nashorn.internal.scripts.Script$Recompilation$402$9044AA$\^function\_.L:1#readFileSync(node/lib/fs.js:374) >> [..] >> >> Any help or hints would be very appreciated! >> >> Regards, >> J?rg >> >> --- >> >> Dipl. Inf. J?rg von Frantzius, Technical Director >> >> E-Mail joerg.frantzius at aperto.com >> >> Phone +49 30 283921-318 >> Fax +49 30 283921-29 >> >> Aperto GmbH ? An IBM Company >> Chausseestra?e 5, D-10115 Berlin >> http://www.aperto.com >> http://www.facebook.com/aperto >> https://www.xing.com/companies/apertoag >> >> HRB 77049 B, AG Berlin Charlottenburg >> Vorstand: Dirk Buddensiek (Vorsitzender), Kai Gro?mann, Stephan Haagen, Daniel Simon >> Aufsichtsrat: Matthew Candy (Vorsitzender) >> >> > From szegedia at gmail.com Thu Dec 1 13:19:20 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 1 Dec 2016 14:19:20 +0100 Subject: Review request for JDK-8170594: >>>=0 generates invalid bytecode for BaseNode LHS Message-ID: <74DF8C11-7856-474B-AD02-5FFABAB89A87@gmail.com> Please review JDK-8170594 ">>>=0 generates invalid bytecode for BaseNode LHS" at for Thanks, Attila. From james.laskey at oracle.com Thu Dec 1 13:27:09 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 1 Dec 2016 09:27:09 -0400 Subject: Review request for JDK-8170594: >>>=0 generates invalid bytecode for BaseNode LHS In-Reply-To: <74DF8C11-7856-474B-AD02-5FFABAB89A87@gmail.com> References: <74DF8C11-7856-474B-AD02-5FFABAB89A87@gmail.com> Message-ID: <9A315C45-6A41-4375-8D54-3BCC444E231C@oracle.com> +1 > On Dec 1, 2016, at 9:19 AM, Attila Szegedi wrote: > > Please review JDK-8170594 ">>>=0 generates invalid bytecode for BaseNode LHS" at for > > Thanks, > Attila. From sundararajan.athijegannathan at oracle.com Thu Dec 1 13:42:57 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 01 Dec 2016 19:12:57 +0530 Subject: Review request for JDK-8170594: >>>=0 generates invalid bytecode for BaseNode LHS In-Reply-To: <74DF8C11-7856-474B-AD02-5FFABAB89A87@gmail.com> References: <74DF8C11-7856-474B-AD02-5FFABAB89A87@gmail.com> Message-ID: <584028E1.8040307@oracle.com> +1 On 01/12/16, 6:49 PM, Attila Szegedi wrote: > Please review JDK-8170594 ">>>=0 generates invalid bytecode for BaseNode LHS" at for > > Thanks, > Attila. From srinivas.dama at oracle.com Fri Dec 2 06:41:02 2016 From: srinivas.dama at oracle.com (Srinivas Dama) Date: Thu, 1 Dec 2016 22:41:02 -0800 (PST) Subject: RFR 8168663: Nashorn: ant testng tests doesn't support external java options Message-ID: Hi, Please review http://cr.openjdk.java.net/~sdama/8168663/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8168663 Added run.test.jvmargs.external property so that we can pass jvm options for complete test run as below. ant -Drun.test.jvmargs.external="-XX:+UseSerialGC -XX:-TieredCompilation" test Regards, Srinivas From artfiedler at gmail.com Fri Dec 2 07:26:58 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 01:26:58 -0600 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' Message-ID: This seems to be working as expected. I tested your modifications on jdk8 nashorn-dev Good catch Hannes! Please review the updated webrev : > http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ > > PS. Had to use Function.prototype.call.call to pass undefined this > explicitly (as JSObject.call can't be called from script). > > -Sundar From sundararajan.athijegannathan at oracle.com Fri Dec 2 08:10:33 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 2 Dec 2016 13:40:33 +0530 Subject: RFR 8170565: JSObject call() is passed undefined for the argument 'thiz' In-Reply-To: References: Message-ID: Thanks. I've tested the patch on jdk8u. I've sent for approval request: http://mail.openjdk.java.net/pipermail/jdk8u-dev/2016-December/006186.html -Sundar On 12/2/2016 12:56 PM, Art Fiedler wrote: > This seems to be working as expected. I tested your modifications on jdk8 > nashorn-dev > > Good catch Hannes! Please review the updated webrev : >> http://cr.openjdk.java.net/~sundar/8170565/webrev.01/ >> >> PS. Had to use Function.prototype.call.call to pass undefined this >> explicitly (as JSObject.call can't be called from script). >> >> -Sundar From artfiedler at gmail.com Fri Dec 2 08:36:16 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 02:36:16 -0600 Subject: Error Stack Column number Message-ID: Your right it's still not standard even in the EMCAScript 2016, it just seems to be the commonality. When thinking java yes line only makes sense as typically one who writes java does not minify the code. When thinking javascript however minified code is quite normal and myfile:1 might mean 5000 other lines as well. I looked into the source some to see if it actually is feasible to build my own stacktrace with file:line:column, however in the current state it seems impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() to get the stack and that ends up being native. The only place you can get the columnNumber is for the specific location the error was thrown. Keyword "thrown". If you noticed my previous sample new Error().columnNumber === -1 since columnNumber is not populated... however if you throw & catch then... try { throw new Error(); } catch(e) { /* e.columnNumber === 6 */ } In my case with console.count(), the need for columnNumber is not the location of new Error() either, since the console.count() function would need the callers file:line:col as that is where console.count() is called from. In the end I still think if the stack element is from a script it should include the column number like node.js, ff, edge all seem to be doing. Since a line only is useless when the script is minified. However considering that nashorn is not providing the stackTrace directly, I could see why the team may not want to include the columnNumber. If you do however know of another way to get the callers file:line:col please let me know. My second thought was to check arguments.callee.caller for that info but no dice. -Arthur Fiedler We do not aim to provide complete compatibility with other JS > implementations on the non-standard properties such as "stack". stack > tries to mimic whatever is done for Java code (no column number for > eg.). But, as you've noted there are enough information on Error objects > via other properties like lineNumber, columnNumber, fileName. It should > be possible to write a simple utility function to format stack string as > desired for specific applications. > > Thanks, > -Sundar > From artfiedler at gmail.com Fri Dec 2 08:56:03 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 02:56:03 -0600 Subject: Confusing ScriptObjectMirror size() vs mirror.get("length") Message-ID: Thanks for making it more confusing! ;) The reason I used array.get(iStr) instead of array.getSlot(I) just happened to be because of my previous post about array.getSlot() returning the undefined object. I should however have used getSlot in this example to make it more clear that I knew I was dealing specifically with an array without properties. You do have a point though and I have a fantastic solution!! Add to ScriptObjectMirror something like... public int getSlotLength() { return ScriptUtils.convert(this.get("length"), Integer.class); } This should help reduce any potential bugs if someone happened to use mirror.size() to loop an accidently on a fragmented array not realizing .get("length") should be used. Having that method should bring more visibility for someone getting slots. Thanks, Arthur Fiedler > I think this is not a bug. "length" is not the same as number of > properties in an object. For example, if you try this: > > var a = [3, 5, 66]; > a.foo = "nar"; > for (var i = 0; i < a.length; i++) > print(a[i]); > > "length" is 3 [and not 4]. keySet size is the number of "keys" > (properties). You've to get "length" property of the array. > > -Sundar > From sundararajan.athijegannathan at oracle.com Fri Dec 2 08:59:35 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 2 Dec 2016 14:29:35 +0530 Subject: Error Stack Column number In-Reply-To: References: Message-ID: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> Error has columnNumber property - but as you've noted it is not always available :( If the error is thrown from script code anywhere via ECMAScript throw statement, we do populate column number - because nashorn could compile the code appropriately to put column number! But, there are places in nashorn runtime code where ECMAScript errors are thrown. For example, RangeError thrown from deep inside nashorn runtime Java code! From those places, we can only recover java stack trace & populate properties of ECMA error object. As java StackTraceElement objects do not have column number, we can't populate column number. -Sundar On 12/2/2016 2:06 PM, Art Fiedler wrote: > Your right it's still not standard even in the EMCAScript 2016, it just > seems to > be the commonality. When thinking java yes line only makes sense as > typically > one who writes java does not minify the code. When thinking javascript > however > minified code is quite normal and myfile:1 might mean 5000 other lines as > well. > > I looked into the source some to see if it actually is feasible to build my > own > stacktrace with file:line:column, however in the current state it seems > impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() to > get the > stack and that ends up being native. The only place you can get the > columnNumber > is for the specific location the error was thrown. Keyword "thrown". > > If you noticed my previous sample new Error().columnNumber === -1 since > columnNumber is not populated... however if you throw & catch then... > try { throw new Error(); } > catch(e) { /* e.columnNumber === 6 */ } > > In my case with console.count(), the need for columnNumber is not the > location > of new Error() either, since the console.count() function would need the > callers file:line:col as that is where console.count() is called from. > > In the end I still think if the stack element is from a script it should > include > the column number like node.js, ff, edge all seem to be doing. Since a line > only > is useless when the script is minified. However considering that nashorn is > not > providing the stackTrace directly, I could see why the team may not want to > include the columnNumber. If you do however know of another way to get the > callers file:line:col please let me know. My second thought was to check > arguments.callee.caller for that info but no dice. > > -Arthur Fiedler > > We do not aim to provide complete compatibility with other JS >> implementations on the non-standard properties such as "stack". stack >> tries to mimic whatever is done for Java code (no column number for >> eg.). But, as you've noted there are enough information on Error objects >> via other properties like lineNumber, columnNumber, fileName. It should >> be possible to write a simple utility function to format stack string as >> desired for specific applications. >> >> Thanks, >> -Sundar >> From artfiedler at gmail.com Fri Dec 2 09:20:55 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 03:20:55 -0600 Subject: SourceMaps Message-ID: A long time from now, in a galaxy far, far away... jdk9 came today 07/27/2017 :) I definitely do believe it's an important feature, most web tech now a days supports source maps and a portion of the open source project I'm working on uses a source transformer to inject some Interrupt checks among other transforms. I already track where I modify the original source but if I could implement a SourceMapProvider that would be a much better solution as it would provide the user accurate information in any error they may receive and also help mask the fact that their original source has been modified making it difficult to determine in their original code where a problem was. Even if I could provide an inline source map through the //#sourceURL directive I would be happy. Would be fantastic to see in both jdk8 and jdk9... when looking through the source code the number of times line numbers and columns are used seems very few so I wouldn't think it would take much effort to implement. The SourceMapProvider being the easiest as the implementation of the provider wouldn't be expected, just the interface to do so. In response to feature freeze, jcp... ?I am free, no matter what rules surround me. If I find them tolerable, I > tolerate them; if I find them too obnoxious, I break them. I am free > because I know that I alone am morally responsible for everything I do.? > ? Robert A. Heinlein > Thanks, Arthur Fiedler This is a feature request. For jdk9, except for jigsaw, we're not adding > any new feature at this point. It *may* be considered for a release > beyond jdk9. Please do file rfe if you think this is an important feature. > > Thanks, > -Sundar From magnus.ihse.bursie at oracle.com Fri Dec 2 09:24:28 2016 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Fri, 2 Dec 2016 10:24:28 +0100 Subject: RFR: JDK-8170629 Remove code duplication in test makefiles Message-ID: <94a50a91-f8d7-a957-5c13-8d1e75ec864e@oracle.com> There is a lot of common code that has been duplicated in */test/Makefile. For jdk, hotspot, jaxp and nashorn, most of the code in the corresponding files are identical. (The odd-man-out is langtools which is quite different.) These files should be unified to share a single code base, to facilitate further improvements to the test makefiles. The intent of this bug is a pure refactoring. The duplication should go, but all functionality should remain exactly the same. This will leave room for some future improvements to the code, but sets a clear limit for this first step. The consolidated code base thus contains a fair amount of if-blocks, but I hope to be able to address most of these later on, by adjusting the individual component to adhere more to the standard behavior. To minimize the amount of ifdefs in the shared code, I allowed for a few changes in behavior. I do not believe these will cause any changes in the real world, and to the extent that they do, they could be considered bug fixes. Behavior that have changed due to unification: * jaxp now defaults ABS_JDK_IMAGE to images/jdk instead of j2sdk. * jaxp now sets TEST_SELECTION to $(TESTDIRS) if it exists. * jaxp and hotspot now get additional option e:JDK8_HOME=${JT_JAVA} * hotspot now sets JAVA_VM_ARGS to $(JPRT_PRODUCT_VM_ARGS) if it exists. * hotspot now sets JAVA_ARGS to $(JPRT_PRODUCT_ARGS) if it exists. Bug: https://bugs.openjdk.java.net/browse/JDK-8170629 WebRev: http://cr.openjdk.java.net/~ihse/JDK-8170629-remove-test-makefile-duplication/webrev.01 /Magnus From artfiedler at gmail.com Fri Dec 2 09:30:05 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 03:30:05 -0600 Subject: Error Stack Column number In-Reply-To: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> References: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> Message-ID: Yeah I almost mentioned that StackTraceElement doesn't provide a column number to populate but removed that comment, thinking "well, StackTraceElement could just be extended for NashornStackTraceElement", but now looking at it I notice it's final. ugh. But it doesn't technically mean an array of StackTraceElement's cant be converted to NashornStackTraceElement's after getting the stack from the jvm, then matching up script frames and populating additional information. -Arthur Fiedler On Fri, Dec 2, 2016 at 2:59 AM, Sundararajan Athijegannathan < sundararajan.athijegannathan at oracle.com> wrote: > Error has columnNumber property - but as you've noted it is not always > available :( If the error is thrown from script code anywhere via > ECMAScript throw statement, we do populate column number - because > nashorn could compile the code appropriately to put column number! > > But, there are places in nashorn runtime code where ECMAScript errors > are thrown. For example, RangeError thrown from deep inside nashorn > runtime Java code! From those places, we can only recover java stack > trace & populate properties of ECMA error object. As java > StackTraceElement objects do not have column number, we can't populate > column number. > > -Sundar > > On 12/2/2016 2:06 PM, Art Fiedler wrote: > > Your right it's still not standard even in the EMCAScript 2016, it just > > seems to > > be the commonality. When thinking java yes line only makes sense as > > typically > > one who writes java does not minify the code. When thinking javascript > > however > > minified code is quite normal and myfile:1 might mean 5000 other lines as > > well. > > > > I looked into the source some to see if it actually is feasible to build > my > > own > > stacktrace with file:line:column, however in the current state it seems > > impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() to > > get the > > stack and that ends up being native. The only place you can get the > > columnNumber > > is for the specific location the error was thrown. Keyword "thrown". > > > > If you noticed my previous sample new Error().columnNumber === -1 since > > columnNumber is not populated... however if you throw & catch then... > > try { throw new Error(); } > > catch(e) { /* e.columnNumber === 6 */ } > > > > In my case with console.count(), the need for columnNumber is not the > > location > > of new Error() either, since the console.count() function would need the > > callers file:line:col as that is where console.count() is called from. > > > > In the end I still think if the stack element is from a script it should > > include > > the column number like node.js, ff, edge all seem to be doing. Since a > line > > only > > is useless when the script is minified. However considering that nashorn > is > > not > > providing the stackTrace directly, I could see why the team may not want > to > > include the columnNumber. If you do however know of another way to get > the > > callers file:line:col please let me know. My second thought was to check > > arguments.callee.caller for that info but no dice. > > > > -Arthur Fiedler > > > > We do not aim to provide complete compatibility with other JS > >> implementations on the non-standard properties such as "stack". stack > >> tries to mimic whatever is done for Java code (no column number for > >> eg.). But, as you've noted there are enough information on Error objects > >> via other properties like lineNumber, columnNumber, fileName. It should > >> be possible to write a simple utility function to format stack string as > >> desired for specific applications. > >> > >> Thanks, > >> -Sundar > >> > > From sundararajan.athijegannathan at oracle.com Fri Dec 2 10:09:34 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 2 Dec 2016 15:39:34 +0530 Subject: Error Stack Column number In-Reply-To: References: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> Message-ID: <7f8b5da9-76bb-a125-a683-43b7ace36dc9@oracle.com> StackTraceElement being final is not the only issue here. How do to pass around & populate column numbers for the script frames? That is the main issue here.. -Sundar On 12/2/2016 3:00 PM, Art Fiedler wrote: > Yeah I almost mentioned that StackTraceElement doesn't provide a column > number to populate but removed that comment, thinking "well, > StackTraceElement could just be extended for NashornStackTraceElement", but > now looking at it I notice it's final. ugh. But it doesn't technically mean > an array of StackTraceElement's cant be converted to > NashornStackTraceElement's after getting the stack from the jvm, then > matching up script frames and populating additional information. > > -Arthur Fiedler > > On Fri, Dec 2, 2016 at 2:59 AM, Sundararajan Athijegannathan < > sundararajan.athijegannathan at oracle.com> wrote: > >> Error has columnNumber property - but as you've noted it is not always >> available :( If the error is thrown from script code anywhere via >> ECMAScript throw statement, we do populate column number - because >> nashorn could compile the code appropriately to put column number! >> >> But, there are places in nashorn runtime code where ECMAScript errors >> are thrown. For example, RangeError thrown from deep inside nashorn >> runtime Java code! From those places, we can only recover java stack >> trace & populate properties of ECMA error object. As java >> StackTraceElement objects do not have column number, we can't populate >> column number. >> >> -Sundar >> >> On 12/2/2016 2:06 PM, Art Fiedler wrote: >>> Your right it's still not standard even in the EMCAScript 2016, it just >>> seems to >>> be the commonality. When thinking java yes line only makes sense as >>> typically >>> one who writes java does not minify the code. When thinking javascript >>> however >>> minified code is quite normal and myfile:1 might mean 5000 other lines as >>> well. >>> >>> I looked into the source some to see if it actually is feasible to build >> my >>> own >>> stacktrace with file:line:column, however in the current state it seems >>> impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() to >>> get the >>> stack and that ends up being native. The only place you can get the >>> columnNumber >>> is for the specific location the error was thrown. Keyword "thrown". >>> >>> If you noticed my previous sample new Error().columnNumber === -1 since >>> columnNumber is not populated... however if you throw & catch then... >>> try { throw new Error(); } >>> catch(e) { /* e.columnNumber === 6 */ } >>> >>> In my case with console.count(), the need for columnNumber is not the >>> location >>> of new Error() either, since the console.count() function would need the >>> callers file:line:col as that is where console.count() is called from. >>> >>> In the end I still think if the stack element is from a script it should >>> include >>> the column number like node.js, ff, edge all seem to be doing. Since a >> line >>> only >>> is useless when the script is minified. However considering that nashorn >> is >>> not >>> providing the stackTrace directly, I could see why the team may not want >> to >>> include the columnNumber. If you do however know of another way to get >> the >>> callers file:line:col please let me know. My second thought was to check >>> arguments.callee.caller for that info but no dice. >>> >>> -Arthur Fiedler >>> >>> We do not aim to provide complete compatibility with other JS >>>> implementations on the non-standard properties such as "stack". stack >>>> tries to mimic whatever is done for Java code (no column number for >>>> eg.). But, as you've noted there are enough information on Error objects >>>> via other properties like lineNumber, columnNumber, fileName. It should >>>> be possible to write a simple utility function to format stack string as >>>> desired for specific applications. >>>> >>>> Thanks, >>>> -Sundar >>>> >> From artfiedler at gmail.com Fri Dec 2 10:43:53 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 04:43:53 -0600 Subject: What about JSObject? Object.prototype.hasOwnProperty.call(obj, prop) Message-ID: Thanks for pointing out the alternative, I hadn't thought of that option. However. ;) I would still argue this as a bug or overlooked. I understand JSObject is not intended to be a real ScriptObject with a prototype etc. However let me ask you the difference between hasMember and hasOwnProperty? They appear to have the same function just different names. On to why I think its a bug or that it was overlooked, if you ask the internet, "javascript how to copy an object", the first thing you will find is... http://stackoverflow.com/questions/728360/how-do-i-correctly-clone-a-javascript-object and for the last 7 year's(855,148 views later) the answer has to do something similar to this (for objects)... for (var attr in obj) { if (obj.hasOwnProperty(attr)) copy[attr] = obj[attr]; } The static function Object.prototype.hasOwnProperty.call(obj,'property') is however just an extension of obj.hasOwnProperty(). I wasn't expecting an actual prototype on the JSObject. So if you take a look at some popular javascript libraries, like lesscss, csso, uglifyjs2, typescript, etc. they are all using obj.hasOwnProperty or Object.prototype.hasOwnProperty.call to copy basic objects, which I believe JSObject is intended to be. As a workaround for lesscss I just slapped a method on my class called hasOwnProperty and that solved that issue. The others though seem to use the static method so I hi-jacked Object.prototype.hasOwnProperty in javascript and if its a JSObject I return hasMember otherwise call the original function. This all seems a little too dirty though for just copying an object. So what I would recommend is adding a default implementation of hasOwnProperty to call hasMember in the JSObject interface and supporting the static calls for properties/keys on the Object.prototype, keys, values and properties are already part of the JSObject, why not make them more accessible to standards and get more use out of them? Current Standards: Object.assign(), Object.keys(), Object.values(), Object.prototype.hasOwnProperty.call, Object.getOwnPropertyNames() EMCAScript 2017 Object.entries() Thanks, Arthur Fiedler JSObject is a way to implement script friendly Java objects. But > JSObjects can not be substituted in all places where a real ScriptObject > is expected. In particular, JSObjects don't have any notion of prototype > and so JSObjects are not on par with ECMAScript script objects. > > But, you can bind your POJO to an empty script object and pass it to > ECMAScript APIs. > > See also: > > https://wiki.openjdk.java.net/display/Nashorn/Nashorn+extensions#Nashornextensions-Object.bindProperties > > HTH, > > -Sundar > From artfiedler at gmail.com Fri Dec 2 11:04:15 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 05:04:15 -0600 Subject: Error Stack Column number In-Reply-To: <7f8b5da9-76bb-a125-a683-43b7ace36dc9@oracle.com> References: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> <7f8b5da9-76bb-a125-a683-43b7ace36dc9@oracle.com> Message-ID: Without knowing all the internals of nashorn. I don't know if an internal calling tree exists or how complicated it would be to build, but it seems the runtime could track function calls and maintain its own stack. CallNode before code is generated has the token/linenumber where it came from, store a unique id of the callnode, with filename, line, column, in the runtime. Then as a call node is executed push to the call stack the id of the call node, and pop when call is complete... when the stack is requested capture the current threads callnode id list (maybe use a thread local?) match it up with the stored callnode id info map... merge/overwrite stackelements in the stack trace gathered from Throwable. Well, that's what just rolled off my head of how it could possibly be done. -Arthur Fiedler On Fri, Dec 2, 2016 at 4:09 AM, Sundararajan Athijegannathan < sundararajan.athijegannathan at oracle.com> wrote: > StackTraceElement being final is not the only issue here. How do to pass > around & populate column numbers for the script frames? That is the > main issue here.. > > -Sundar > > On 12/2/2016 3:00 PM, Art Fiedler wrote: > > Yeah I almost mentioned that StackTraceElement doesn't provide a column > > number to populate but removed that comment, thinking "well, > > StackTraceElement could just be extended for NashornStackTraceElement", > but > > now looking at it I notice it's final. ugh. But it doesn't technically > mean > > an array of StackTraceElement's cant be converted to > > NashornStackTraceElement's after getting the stack from the jvm, then > > matching up script frames and populating additional information. > > > > -Arthur Fiedler > > > > On Fri, Dec 2, 2016 at 2:59 AM, Sundararajan Athijegannathan < > > sundararajan.athijegannathan at oracle.com> wrote: > > > >> Error has columnNumber property - but as you've noted it is not always > >> available :( If the error is thrown from script code anywhere via > >> ECMAScript throw statement, we do populate column number - because > >> nashorn could compile the code appropriately to put column number! > >> > >> But, there are places in nashorn runtime code where ECMAScript errors > >> are thrown. For example, RangeError thrown from deep inside nashorn > >> runtime Java code! From those places, we can only recover java stack > >> trace & populate properties of ECMA error object. As java > >> StackTraceElement objects do not have column number, we can't populate > >> column number. > >> > >> -Sundar > >> > >> On 12/2/2016 2:06 PM, Art Fiedler wrote: > >>> Your right it's still not standard even in the EMCAScript 2016, it just > >>> seems to > >>> be the commonality. When thinking java yes line only makes sense as > >>> typically > >>> one who writes java does not minify the code. When thinking javascript > >>> however > >>> minified code is quite normal and myfile:1 might mean 5000 other lines > as > >>> well. > >>> > >>> I looked into the source some to see if it actually is feasible to > build > >> my > >>> own > >>> stacktrace with file:line:column, however in the current state it seems > >>> impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() > to > >>> get the > >>> stack and that ends up being native. The only place you can get the > >>> columnNumber > >>> is for the specific location the error was thrown. Keyword "thrown". > >>> > >>> If you noticed my previous sample new Error().columnNumber === -1 since > >>> columnNumber is not populated... however if you throw & catch then... > >>> try { throw new Error(); } > >>> catch(e) { /* e.columnNumber === 6 */ } > >>> > >>> In my case with console.count(), the need for columnNumber is not the > >>> location > >>> of new Error() either, since the console.count() function would need > the > >>> callers file:line:col as that is where console.count() is called from. > >>> > >>> In the end I still think if the stack element is from a script it > should > >>> include > >>> the column number like node.js, ff, edge all seem to be doing. Since a > >> line > >>> only > >>> is useless when the script is minified. However considering that > nashorn > >> is > >>> not > >>> providing the stackTrace directly, I could see why the team may not > want > >> to > >>> include the columnNumber. If you do however know of another way to get > >> the > >>> callers file:line:col please let me know. My second thought was to > check > >>> arguments.callee.caller for that info but no dice. > >>> > >>> -Arthur Fiedler > >>> > >>> We do not aim to provide complete compatibility with other JS > >>>> implementations on the non-standard properties such as "stack". stack > >>>> tries to mimic whatever is done for Java code (no column number for > >>>> eg.). But, as you've noted there are enough information on Error > objects > >>>> via other properties like lineNumber, columnNumber, fileName. It > should > >>>> be possible to write a simple utility function to format stack string > as > >>>> desired for specific applications. > >>>> > >>>> Thanks, > >>>> -Sundar > >>>> > >> > > From sundararajan.athijegannathan at oracle.com Fri Dec 2 11:25:58 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 2 Dec 2016 16:55:58 +0530 Subject: Error Stack Column number In-Reply-To: References: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> <7f8b5da9-76bb-a125-a683-43b7ace36dc9@oracle.com> Message-ID: Note that nashorn always compiles to bytecode - no parallel side data structure for JS stacks or "interpreter stack" here. That said (if this can be done without hurting performance) we'd be happy to accept contributions. -Sundar On 12/2/2016 4:34 PM, Art Fiedler wrote: > Without knowing all the internals of nashorn. I don't know if an internal > calling tree exists or how complicated it would be to build, but it seems > the runtime could track function calls and maintain its own stack. > > CallNode before code is generated has the token/linenumber where it came > from, store a unique id of the callnode, with filename, line, column, in > the runtime. Then as a call node is executed push to the call stack the id > of the call node, and pop when call is complete... when the stack is > requested capture the current threads callnode id list (maybe use a thread > local?) match it up with the stored callnode id info map... merge/overwrite > stackelements in the stack trace gathered from Throwable. > > Well, that's what just rolled off my head of how it could possibly be done. > > -Arthur Fiedler > > On Fri, Dec 2, 2016 at 4:09 AM, Sundararajan Athijegannathan < > sundararajan.athijegannathan at oracle.com> wrote: > >> StackTraceElement being final is not the only issue here. How do to pass >> around & populate column numbers for the script frames? That is the >> main issue here.. >> >> -Sundar >> >> On 12/2/2016 3:00 PM, Art Fiedler wrote: >>> Yeah I almost mentioned that StackTraceElement doesn't provide a column >>> number to populate but removed that comment, thinking "well, >>> StackTraceElement could just be extended for NashornStackTraceElement", >> but >>> now looking at it I notice it's final. ugh. But it doesn't technically >> mean >>> an array of StackTraceElement's cant be converted to >>> NashornStackTraceElement's after getting the stack from the jvm, then >>> matching up script frames and populating additional information. >>> >>> -Arthur Fiedler >>> >>> On Fri, Dec 2, 2016 at 2:59 AM, Sundararajan Athijegannathan < >>> sundararajan.athijegannathan at oracle.com> wrote: >>> >>>> Error has columnNumber property - but as you've noted it is not always >>>> available :( If the error is thrown from script code anywhere via >>>> ECMAScript throw statement, we do populate column number - because >>>> nashorn could compile the code appropriately to put column number! >>>> >>>> But, there are places in nashorn runtime code where ECMAScript errors >>>> are thrown. For example, RangeError thrown from deep inside nashorn >>>> runtime Java code! From those places, we can only recover java stack >>>> trace & populate properties of ECMA error object. As java >>>> StackTraceElement objects do not have column number, we can't populate >>>> column number. >>>> >>>> -Sundar >>>> >>>> On 12/2/2016 2:06 PM, Art Fiedler wrote: >>>>> Your right it's still not standard even in the EMCAScript 2016, it just >>>>> seems to >>>>> be the commonality. When thinking java yes line only makes sense as >>>>> typically >>>>> one who writes java does not minify the code. When thinking javascript >>>>> however >>>>> minified code is quite normal and myfile:1 might mean 5000 other lines >> as >>>>> well. >>>>> >>>>> I looked into the source some to see if it actually is feasible to >> build >>>> my >>>>> own >>>>> stacktrace with file:line:column, however in the current state it seems >>>>> impossible! Nashorn actually uses java.lang.Throwable.getStackTrace() >> to >>>>> get the >>>>> stack and that ends up being native. The only place you can get the >>>>> columnNumber >>>>> is for the specific location the error was thrown. Keyword "thrown". >>>>> >>>>> If you noticed my previous sample new Error().columnNumber === -1 since >>>>> columnNumber is not populated... however if you throw & catch then... >>>>> try { throw new Error(); } >>>>> catch(e) { /* e.columnNumber === 6 */ } >>>>> >>>>> In my case with console.count(), the need for columnNumber is not the >>>>> location >>>>> of new Error() either, since the console.count() function would need >> the >>>>> callers file:line:col as that is where console.count() is called from. >>>>> >>>>> In the end I still think if the stack element is from a script it >> should >>>>> include >>>>> the column number like node.js, ff, edge all seem to be doing. Since a >>>> line >>>>> only >>>>> is useless when the script is minified. However considering that >> nashorn >>>> is >>>>> not >>>>> providing the stackTrace directly, I could see why the team may not >> want >>>> to >>>>> include the columnNumber. If you do however know of another way to get >>>> the >>>>> callers file:line:col please let me know. My second thought was to >> check >>>>> arguments.callee.caller for that info but no dice. >>>>> >>>>> -Arthur Fiedler >>>>> >>>>> We do not aim to provide complete compatibility with other JS >>>>>> implementations on the non-standard properties such as "stack". stack >>>>>> tries to mimic whatever is done for Java code (no column number for >>>>>> eg.). But, as you've noted there are enough information on Error >> objects >>>>>> via other properties like lineNumber, columnNumber, fileName. It >> should >>>>>> be possible to write a simple utility function to format stack string >> as >>>>>> desired for specific applications. >>>>>> >>>>>> Thanks, >>>>>> -Sundar >>>>>> >> From erik.joelsson at oracle.com Fri Dec 2 11:56:58 2016 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 2 Dec 2016 12:56:58 +0100 Subject: RFR: JDK-8170629 Remove code duplication in test makefiles In-Reply-To: <94a50a91-f8d7-a957-5c13-8d1e75ec864e@oracle.com> References: <94a50a91-f8d7-a957-5c13-8d1e75ec864e@oracle.com> Message-ID: <5d130cb7-794f-9fec-51aa-6ecd432847da@oracle.com> Looks good. /Erik On 2016-12-02 10:24, Magnus Ihse Bursie wrote: > There is a lot of common code that has been duplicated in > */test/Makefile. For jdk, hotspot, jaxp and nashorn, most of the code > in the corresponding files are identical. (The odd-man-out is > langtools which is quite different.) > > These files should be unified to share a single code base, to > facilitate further improvements to the test makefiles. > > The intent of this bug is a pure refactoring. The duplication should > go, but all functionality should remain exactly the same. This will > leave room for some future improvements to the code, but sets a clear > limit for this first step. The consolidated code base thus contains a > fair amount of if-blocks, but I hope to be able to address most of > these later on, by adjusting the individual component to adhere more > to the standard behavior. > > To minimize the amount of ifdefs in the shared code, I allowed for a > few changes in behavior. I do not believe these will cause any changes > in the real world, and to the extent that they do, they could be > considered bug fixes. > > Behavior that have changed due to unification: > * jaxp now defaults ABS_JDK_IMAGE to images/jdk instead of j2sdk. > * jaxp now sets TEST_SELECTION to $(TESTDIRS) if it exists. > * jaxp and hotspot now get additional option e:JDK8_HOME=${JT_JAVA} > * hotspot now sets JAVA_VM_ARGS to $(JPRT_PRODUCT_VM_ARGS) if it exists. > * hotspot now sets JAVA_ARGS to $(JPRT_PRODUCT_ARGS) if it exists. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8170629 > WebRev: > http://cr.openjdk.java.net/~ihse/JDK-8170629-remove-test-makefile-duplication/webrev.01 > > /Magnus From hannes.wallnoefer at oracle.com Fri Dec 2 13:30:54 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Fri, 2 Dec 2016 14:30:54 +0100 Subject: RFR 8168663: Nashorn: ant testng tests doesn't support external java options In-Reply-To: References: Message-ID: <6190FBC9-119C-426B-8DAC-C0751E63F6E3@oracle.com> +1 Hannes > Am 02.12.2016 um 07:41 schrieb Srinivas Dama : > > Hi, > > Please review http://cr.openjdk.java.net/~sdama/8168663/webrev.00/ > for https://bugs.openjdk.java.net/browse/JDK-8168663 > > Added run.test.jvmargs.external property so that we can pass jvm options for complete test run as below. > ant -Drun.test.jvmargs.external="-XX:+UseSerialGC -XX:-TieredCompilation" test > > Regards, > Srinivas From sundararajan.athijegannathan at oracle.com Fri Dec 2 13:57:59 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 02 Dec 2016 19:27:59 +0530 Subject: RFR 8168663: Nashorn: ant testng tests doesn't support external java options In-Reply-To: References: Message-ID: <58417DE7.6010805@oracle.com> +1 On 02/12/16, 12:11 PM, Srinivas Dama wrote: > Hi, > > Please review http://cr.openjdk.java.net/~sdama/8168663/webrev.00/ > for https://bugs.openjdk.java.net/browse/JDK-8168663 > > Added run.test.jvmargs.external property so that we can pass jvm options for complete test run as below. > ant -Drun.test.jvmargs.external="-XX:+UseSerialGC -XX:-TieredCompilation" test > > Regards, > Srinivas From artfiedler at gmail.com Fri Dec 2 21:23:46 2016 From: artfiedler at gmail.com (Art Fiedler) Date: Fri, 2 Dec 2016 15:23:46 -0600 Subject: Error Stack Column number In-Reply-To: References: <4835bd54-5b36-797d-2340-6817cecf88b8@oracle.com> <7f8b5da9-76bb-a125-a683-43b7ace36dc9@oracle.com> Message-ID: I had a few thoughts on this but after evaluating it wouldn't work out... If it were possible to modify java.lang.Throwable & StackTraceElement I had thought to do some bit shifting to combine line & column numbers into a negative number and place that as the line number value stored in the LineNumberTable attribute, then converting the linenumber into line/column in StackTraceElement. However it seems that the datatype is 2bytes only creating a huge limitation on the size of script. Then I had thought well maybe store the offset position as a negative number and provide a map to get line & column... but again 2bytes isn't enough to store an offset, less.min.js's character count is 152,193. So even if such a modification was allowed it would not be very useful. The other idea to push/pop on a threadlocal would I think also require a try/finally on every method call... which I'm guessing would be a large performance hit. In this case I would think the column number would only be included if the error was caught in nashorn so it can populate the column numbers on the stack... that would solve my case but it wouldn't be consistent when caught externally. -Arthur Fiedler On Fri, Dec 2, 2016 at 5:25 AM, Sundararajan Athijegannathan < sundararajan.athijegannathan at oracle.com> wrote: > Note that nashorn always compiles to bytecode - no parallel side data > structure for JS stacks or "interpreter stack" here. > > That said (if this can be done without hurting performance) we'd be > happy to accept contributions. > > -Sundar > > > On 12/2/2016 4:34 PM, Art Fiedler wrote: > > Without knowing all the internals of nashorn. I don't know if an internal > > calling tree exists or how complicated it would be to build, but it seems > > the runtime could track function calls and maintain its own stack. > > > > CallNode before code is generated has the token/linenumber where it came > > from, store a unique id of the callnode, with filename, line, column, in > > the runtime. Then as a call node is executed push to the call stack the > id > > of the call node, and pop when call is complete... when the stack is > > requested capture the current threads callnode id list (maybe use a > thread > > local?) match it up with the stored callnode id info map... > merge/overwrite > > stackelements in the stack trace gathered from Throwable. > > > > Well, that's what just rolled off my head of how it could possibly be > done. > > > > -Arthur Fiedler > > > > On Fri, Dec 2, 2016 at 4:09 AM, Sundararajan Athijegannathan < > > sundararajan.athijegannathan at oracle.com> wrote: > > > >> StackTraceElement being final is not the only issue here. How do to pass > >> around & populate column numbers for the script frames? That is the > >> main issue here.. > >> > >> -Sundar > >> > >> On 12/2/2016 3:00 PM, Art Fiedler wrote: > >>> Yeah I almost mentioned that StackTraceElement doesn't provide a column > >>> number to populate but removed that comment, thinking "well, > >>> StackTraceElement could just be extended for NashornStackTraceElement", > >> but > >>> now looking at it I notice it's final. ugh. But it doesn't technically > >> mean > >>> an array of StackTraceElement's cant be converted to > >>> NashornStackTraceElement's after getting the stack from the jvm, then > >>> matching up script frames and populating additional information. > >>> > >>> -Arthur Fiedler > >>> > >>> On Fri, Dec 2, 2016 at 2:59 AM, Sundararajan Athijegannathan < > >>> sundararajan.athijegannathan at oracle.com> wrote: > >>> > >>>> Error has columnNumber property - but as you've noted it is not always > >>>> available :( If the error is thrown from script code anywhere via > >>>> ECMAScript throw statement, we do populate column number - because > >>>> nashorn could compile the code appropriately to put column number! > >>>> > >>>> But, there are places in nashorn runtime code where ECMAScript errors > >>>> are thrown. For example, RangeError thrown from deep inside nashorn > >>>> runtime Java code! From those places, we can only recover java stack > >>>> trace & populate properties of ECMA error object. As java > >>>> StackTraceElement objects do not have column number, we can't populate > >>>> column number. > >>>> > >>>> -Sundar > >>>> > >>>> On 12/2/2016 2:06 PM, Art Fiedler wrote: > >>>>> Your right it's still not standard even in the EMCAScript 2016, it > just > >>>>> seems to > >>>>> be the commonality. When thinking java yes line only makes sense as > >>>>> typically > >>>>> one who writes java does not minify the code. When thinking > javascript > >>>>> however > >>>>> minified code is quite normal and myfile:1 might mean 5000 other > lines > >> as > >>>>> well. > >>>>> > >>>>> I looked into the source some to see if it actually is feasible to > >> build > >>>> my > >>>>> own > >>>>> stacktrace with file:line:column, however in the current state it > seems > >>>>> impossible! Nashorn actually uses java.lang.Throwable. > getStackTrace() > >> to > >>>>> get the > >>>>> stack and that ends up being native. The only place you can get the > >>>>> columnNumber > >>>>> is for the specific location the error was thrown. Keyword "thrown". > >>>>> > >>>>> If you noticed my previous sample new Error().columnNumber === -1 > since > >>>>> columnNumber is not populated... however if you throw & catch then... > >>>>> try { throw new Error(); } > >>>>> catch(e) { /* e.columnNumber === 6 */ } > >>>>> > >>>>> In my case with console.count(), the need for columnNumber is not the > >>>>> location > >>>>> of new Error() either, since the console.count() function would need > >> the > >>>>> callers file:line:col as that is where console.count() is called > from. > >>>>> > >>>>> In the end I still think if the stack element is from a script it > >> should > >>>>> include > >>>>> the column number like node.js, ff, edge all seem to be doing. Since > a > >>>> line > >>>>> only > >>>>> is useless when the script is minified. However considering that > >> nashorn > >>>> is > >>>>> not > >>>>> providing the stackTrace directly, I could see why the team may not > >> want > >>>> to > >>>>> include the columnNumber. If you do however know of another way to > get > >>>> the > >>>>> callers file:line:col please let me know. My second thought was to > >> check > >>>>> arguments.callee.caller for that info but no dice. > >>>>> > >>>>> -Arthur Fiedler > >>>>> > >>>>> We do not aim to provide complete compatibility with other JS > >>>>>> implementations on the non-standard properties such as "stack". > stack > >>>>>> tries to mimic whatever is done for Java code (no column number for > >>>>>> eg.). But, as you've noted there are enough information on Error > >> objects > >>>>>> via other properties like lineNumber, columnNumber, fileName. It > >> should > >>>>>> be possible to write a simple utility function to format stack > string > >> as > >>>>>> desired for specific applications. > >>>>>> > >>>>>> Thanks, > >>>>>> -Sundar > >>>>>> > >> > > From asashour at yahoo.com Sat Dec 3 09:00:10 2016 From: asashour at yahoo.com (Ahmed Ashour) Date: Sat, 3 Dec 2016 09:00:10 +0000 (UTC) Subject: NoSuchElementException in PropertyMapIterator References: <2099099219.5680681.1480755610621.ref@mail.yahoo.com> Message-ID: <2099099219.5680681.1480755610621@mail.yahoo.com> Hi all, I believe there is a bug in next() of PropertyMap.PropertyMapIterator, where 'iter' may not have more elements. The line should be changed from "property = iter.next();" to "property = iter.hasNext() ? iter.next() : null;". This checking is done in the constructor and in skipNotEnumerable(), but not in next(). Below is a complete test case, that fails in jdk8 and jdk9 code bases. Thanks a lot,Ahmed Ashour public class PropertyMapTest { ? ? @Test? ? public ?void iterator() {? ? ? ? ScriptObject scriptObject = new ObjectConstructor();? ? ? ? for (Object o : scriptObject.getMap()) {? ? ? ? }? ? } ? ? public static final class ObjectConstructor extends ScriptObject {? ? ? ? private ScriptFunction addEventListener; ? ? ? ? public ScriptFunction G$addEventListener() {? ? ? ? ? ? return addEventListener;? ? ? ? } ? ? ? ? public void S$addEventListener(final ScriptFunction function) {? ? ? ? ? ? this.addEventListener = function;? ? ? ? } ? ? ? ? {? ? ? ? ? ? final List list = new ArrayList<>(1);? ? ? ? ? ? list.add(AccessorProperty.create("addEventListener", Property.WRITABLE_ENUMERABLE_CONFIGURABLE,?? ? ? ? ? ? ? ? ? ? virtualHandle("G$addEventListener", ScriptFunction.class),? ? ? ? ? ? ? ? ? ? virtualHandle("S$addEventListener", void.class, ScriptFunction.class)));? ? ? ? ? ? setMap(PropertyMap.newMap(list));? ? ? ? } ? ? ? ? private static MethodHandle virtualHandle(final String name, final Class rtype, final Class... ptypes) {? ? ? ? ? ? try {? ? ? ? ? ? ? ? return MethodHandles.lookup().findVirtual(ObjectConstructor.class, name,? ? ? ? ? ? ? ? ? ? ? ? MethodType.methodType(rtype, ptypes));? ? ? ? ? ? }? ? ? ? ? ? catch (final ReflectiveOperationException e) {? ? ? ? ? ? ? ? throw new IllegalStateException(e);? ? ? ? ? ? }? ? ? ? }? ? }} From sundararajan.athijegannathan at oracle.com Mon Dec 5 05:55:50 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Mon, 5 Dec 2016 11:25:50 +0530 Subject: JavaScript Array slot/member == undefined instead of null? In-Reply-To: References: Message-ID: Scripting language independent javax.script APIs like ScriptEngine.eval, Bindings.get, Invocable.invokeFunction etc. translate undefined to null (closest to what you'd expect in java world). But, nashorn specific APIs like jdk.nashorn.api.scripting.JSObject.getMember/getSlot etc. return undefined as undefined - without null translation. [as you're calling nashorn engine specific API]. -Sundar On 12/1/2016 1:13 AM, Art Fiedler wrote: > @Sundararajan, > > I was previously using .getSlot(index) when extracting items from a > ScriptObjectMirror when that mirror was an array until I ran into > a undefined element and that is not equal to null as I had for my > condition... so I swapped .getSlot(i) with .get(Integer.toString(i)) > > Are getMember(), get(), and getSlot() suppose to all return the same null > or undefined value? Or is it expected to have > getMember() and getSlot() to return Undefined object and get() to return > null? > > As a side note... If this is the intended behavior this might be a better > way to get the Undefined object instead of making a > call out of nashorn to a public static method. > > Here is an example... (jdk8 1.8.0_102) > > Thanks, > Arthur Fiedler > > > public static void main(String[] args) throws Exception { > NashornScriptEngineFactory factory = new > NashornScriptEngineFactory(); > ScriptEngine scriptEngine = factory.getScriptEngine(); > scriptEngine.eval("globalArray = ['line1',undefined,null];"); > ScriptObjectMirror array = > (ScriptObjectMirror)scriptEngine.get("globalArray"); > System.out.println("size(): " + array.size()); > for(int i = 0; i < array.size(); i++) { > System.out.println("Index: " + i); > String iStr = String.valueOf(i); > System.out.println(" GetMember('"+iStr+"'): " + > array.getMember(iStr)); > System.out.println(" Get('"+iStr+"'): " + array.get(iStr)); > System.out.println(" GetSlot("+iStr+"): " + > array.getSlot(i)); > } > } > /* > size(): 3 > Index: 0 > GetMember('0'): line1 > Get('0'): line1 > GetSlot(0): line1 > Index: 1 > GetMember('1'): undefined > Get('1'): null > GetSlot(1): undefined > Index: 2 > GetMember('2'): null > Get('2'): null > GetSlot(2): null > */ From sundararajan.athijegannathan at oracle.com Mon Dec 5 06:35:48 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Mon, 5 Dec 2016 12:05:48 +0530 Subject: Undefined should be part of the exposed nashorn API In-Reply-To: References: Message-ID: If you want to get undefined value, you can do something like this: import javax.script.*; import jdk.nashorn.api.scripting.JSObject; public class Main { public static void main(String[] args) throws Exception { ScriptEngine e = new ScriptEngineManager().getEngineByName("nashorn"); JSObject global = (JSObject)e.getBindings(ScriptContext.ENGINE_SCOPE); Object undefined = global.getMember("undefined"); System.out.println(undefined); e.put("foo", undefined); System.out.println(e.eval("typeof foo")); } } Thanks, -Sundar On 12/1/2016 2:24 AM, Art Fiedler wrote: > @Sundararajan > > I ran into an issue with a JSObject I implemented and some code in LessCSS. > The team is likely aware by now of a need to > return Undefined from JSObject's getMember() and possibly call(), and > getSlot()... You had previously asked a long time ago > why someone would want to do that and I responded here: > > http://stackoverflow.com/questions/30528083/passing-undefined-to-nashorn-javascript-in-scala-java/30533897#30533897 > > with a comment: > > Why pass undefined? Let's say your calling an existing javascript lib (like > lesscss) that > calls something like this... > var variables = options.variables; > // keep in mind typeof null === 'object' and typeof undefined === > 'undefined' > if (typeof variables === 'object') { > var keys = Object.keys(variables); //error 'null is not an Object' > } > Now to put into perspective, 'options' is an implementation of > AbstractJSObject... how else > would you specify in getMember that the member does not exist, returning > null would throw > an error when Object.keys() is called. > > I used this workaround... because in my sandbox that runs lesscss etc java > accessibility is removed and I would not be able > to make the callback... > > // Code Fragment > if (Undefined.getUndefined() == null) { > // Get the undefined object now, in its own script engine, to avoid > problems with permissions/security > > factory.getScriptEngine().eval("Java.type('io.github.artfiedler.nashorn.Undefined').setUndefined(undefined);"); > } > > /** > * Used to reference the Undefined value used in nashorn. > * > * This is a standalone class to avoid a passing a reference > * to an object like NashornSandbox to javascript that may be > * exploited in some way unexpected way. > */ > public final class Undefined > { > private static Object undefinedObject; > public static void setUndefined(Object undefined) { undefinedObject > = undefined; } > /** > * Provides and easy way to get the undefined javascript value used > by nashorn. > * This value is not populated until after a NashornSandbox has > been initialized > * > * @return the internal Nashorn object used to reference undefined > */ > public static Object getUndefined() { return undefinedObject; } > } > > Now using my Undefined class I can do the following in my > AbstractJSObject... and this will now support the existing lesscss code > without modifications. > @Override > public Object getMember(String name) > { > return spill.getOrDefault(name, Undefined.getUndefined()); > } > > I know that Undefined is defined at > jdk.nashorn.internal.runtime.ScriptRuntime.UNDEFINED && > jdk.nashorn.internal.runtime.Undefined.getUndefined() > and the team may not want to expose those specific classes to the API but I > believe at least something like this should be done... > > @Exported > public final class ScriptObjectMirror extends AbstractJSObject > implements Bindings { > // This already exists... > public static boolean isUndefined(Object obj) { > return obj == ScriptRuntime.UNDEFINED; > } > // Add... > public static Object getUndefined() { > return ScriptRuntime.UNDEFINED; > } > ... > } > > -- or -- > > @Exported > public interface JSObject > { > Object Undefined = ScriptRuntime.UNDEFINED; > ... > } > > -- or -- > > @Exported > public interface JSObject > { > Object Undefined = new Object() { @Override public String > toString() {return "undefined";} }; > ... > } > // This would require any usages of JSObject to check for > JSObject.Undefined coming out of getMember(), call(), getSlot(), eval(), > newObject() From hannes.wallnoefer at oracle.com Mon Dec 5 13:22:27 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Mon, 5 Dec 2016 14:22:27 +0100 Subject: NoSuchElementException in PropertyMapIterator In-Reply-To: <2099099219.5680681.1480755610621@mail.yahoo.com> References: <2099099219.5680681.1480755610621.ref@mail.yahoo.com> <2099099219.5680681.1480755610621@mail.yahoo.com> Message-ID: <118568AC-855B-4FB8-AB80-42656A016908@oracle.com> Hi Ahmed, yes, this is a bug, thanks for reporting it. Out of curiosity, , since it is not an exported API class: How are you using PropertyMap? Hannes > Am 03.12.2016 um 10:00 schrieb Ahmed Ashour : > > Hi all, > I believe there is a bug in next() of PropertyMap.PropertyMapIterator, where 'iter' may not have more elements. > The line should be changed from "property = iter.next();" to "property = iter.hasNext() ? iter.next() : null;". > This checking is done in the constructor and in skipNotEnumerable(), but not in next(). > Below is a complete test case, that fails in jdk8 and jdk9 code bases. > Thanks a lot,Ahmed Ashour > public class PropertyMapTest { > @Test public void iterator() { ScriptObject scriptObject = new ObjectConstructor(); for (Object o : scriptObject.getMap()) { } } > public static final class ObjectConstructor extends ScriptObject { private ScriptFunction addEventListener; > public ScriptFunction G$addEventListener() { return addEventListener; } > public void S$addEventListener(final ScriptFunction function) { this.addEventListener = function; } > { final List list = new ArrayList<>(1); list.add(AccessorProperty.create("addEventListener", Property.WRITABLE_ENUMERABLE_CONFIGURABLE, virtualHandle("G$addEventListener", ScriptFunction.class), virtualHandle("S$addEventListener", void.class, ScriptFunction.class))); setMap(PropertyMap.newMap(list)); } > private static MethodHandle virtualHandle(final String name, final Class rtype, final Class... ptypes) { try { return MethodHandles.lookup().findVirtual(ObjectConstructor.class, name, MethodType.methodType(rtype, ptypes)); } catch (final ReflectiveOperationException e) { throw new IllegalStateException(e); } } }} From asashour at yahoo.com Mon Dec 5 13:33:58 2016 From: asashour at yahoo.com (Ahmed Ashour) Date: Mon, 5 Dec 2016 13:33:58 +0000 (UTC) Subject: NoSuchElementException in PropertyMapIterator In-Reply-To: <118568AC-855B-4FB8-AB80-42656A016908@oracle.com> References: <2099099219.5680681.1480755610621.ref@mail.yahoo.com> <2099099219.5680681.1480755610621@mail.yahoo.com> <118568AC-855B-4FB8-AB80-42656A016908@oracle.com> Message-ID: <1646072191.6628773.1480944838510@mail.yahoo.com> Hi Hannes, Well, as part of HtmlUnit (headless browser simulator), we can not use Nashorn without modifications, since there are browser-specific peculiarities we must blindly follow. I would say the most difficult part, is to customize Global ('window' in browser usage) to join it with other objects. ?This is currently overcome, by having one-to-one relationship between Global and Window. Thanks a lot, and keep the good work. Ahmed From: Hannes Walln?fer To: Ahmed Ashour Cc: "nashorn-dev at openjdk.java.net" Sent: Monday, December 5, 2016 2:22 PM Subject: Re: NoSuchElementException in PropertyMapIterator Hi Ahmed, yes, this is a bug, thanks for reporting it. Out of curiosity, , since it is not an exported API class: How are you using PropertyMap? Hannes From yikesaroni at gmail.com Mon Dec 5 18:35:28 2016 From: yikesaroni at gmail.com (yikes aroni) Date: Mon, 5 Dec 2016 13:35:28 -0500 Subject: Caching and re-using bindings Message-ID: I want to cache ScriptEngine bindings for reuse. The basic algo would be 1) Build up my ScriptEngine (SE) with stuff i need. 2) Snapshot the bindings -- i.e., cache them Use engine.getBindings(ScriptContext.ENGINE_SCOPE) 3) Use the SE for stuff that might modify its state. 4) When done, replace the SE's bindings with my snapshotted bindings Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) 5) I now have a "fresh" SE to use. The problem is that this doesn't work as expected. The cached bindings appear to point to the actual SE bindings and therefore whatever gets added to the SE bindings, also gets added to the cached bindings. Here's some code to show how it's not doing what i would expect. public class TempEngineTest { static ScriptEngineManager seManager = new ScriptEngineManager(); static ScriptEngine se = seManager.getEngineByName("nashorn"); public static void printBindings(Map bindings) { for (Map.Entry entry : bindings.entrySet()) { System.out.println(entry.getKey() + " = " + entry.getValue()); } } public static void main(String[] a) throws Exception { // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings se.eval("globalA = 'testA';"); // Snapshot the bindings from the engine. Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); // print the bindings to confirm there is only one variable in them. --> TRUE printBindings(bEngine); // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings se.eval("globalB = 'testB';"); // print the bindings. Now both variables are present........... WHY??? printBindings(bEngine); } } I've seen suggestions to cache and reuse bindings in various articles, but no specific code for doing so. How do i accomplish this in actual code? thanks From ekemokai at gmail.com Mon Dec 5 20:01:56 2016 From: ekemokai at gmail.com (Edmond Kemokai) Date: Mon, 5 Dec 2016 15:01:56 -0500 Subject: Caching and re-using bindings In-Reply-To: References: Message-ID: Why don't you create instances of SimpleBinding and use those as needed? Use ScriptEngine.setBindings to reset the binding... On Dec 5, 2016 1:36 PM, "yikes aroni" wrote: > I want to cache ScriptEngine bindings for reuse. The basic algo would be > > 1) Build up my ScriptEngine (SE) with stuff i need. > 2) Snapshot the bindings -- i.e., cache them > Use engine.getBindings(ScriptContext.ENGINE_SCOPE) > 3) Use the SE for stuff that might modify its state. > 4) When done, replace the SE's bindings with my snapshotted bindings > Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) > 5) I now have a "fresh" SE to use. > > The problem is that this doesn't work as expected. The cached bindings > appear to point to the actual SE bindings and therefore whatever gets added > to the SE bindings, also gets added to the cached bindings. Here's some > code to show how it's not doing what i would expect. > > public class TempEngineTest { > > static ScriptEngineManager seManager = new ScriptEngineManager(); > static ScriptEngine se = seManager.getEngineByName("nashorn"); > public static void printBindings(Map bindings) { > for (Map.Entry entry : bindings.entrySet()) { > System.out.println(entry.getKey() + " = " + entry.getValue()); > } > } > public static void main(String[] a) throws Exception { > // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings > se.eval("globalA = 'testA';"); > // Snapshot the bindings from the engine. > Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); > // print the bindings to confirm there is only one variable in them. --> > TRUE > printBindings(bEngine); > // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings > se.eval("globalB = 'testB';"); > // print the bindings. Now both variables are present........... WHY??? > printBindings(bEngine); > } > } > > I've seen suggestions to cache and reuse bindings in various articles, but > no specific code for doing so. How do i accomplish this in actual code? > > thanks > From jluzon at riotgames.com Tue Dec 6 12:45:11 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Tue, 6 Dec 2016 04:45:11 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale Message-ID: Hey folks, I've tried many different ways of using Nashorn multithreaded based on what I've found on the internet and I still can't get a single one to scale. Even the most naive method of making many script engines with my script tends to bottleneck itself when I have more than 10 threads invoking functions. I'm using the following code to compile my script and invocable.invokeFunction("transform", input) to execute: > static Invocable generateInvocable(String script) throws > ScriptException { > ScriptEngineManager manager = new ScriptEngineManager(); > ScriptEngine engine = > manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > Compilable compilable = (Compilable) engine; > final CompiledScript compiled = compilable.compile(script); > compiled.eval(); > return (Invocable) engine; > } The script I'm compiling is: > String script = "function transform(input) {" + > "var result = JSON.parse(input);" + > "response = {};\n" + > "for (var i = 0; i < result.length; i++) {\n" + > " var summoner = {};\n" + > " summoner.id = result[i].id;\n" + > " summoner.name = result[i].name;\n" + > " summoner.profileIconId = result[i].profileIconId;\n" + > " summoner.revisionDate = result[i].revisionDate;\n" + > " summoner.summonerLevel = result[i].level;\n" + > " response[summoner.id] = summoner;\n" + > "}\n" + > "result = response;" + > "return JSON.stringify(result);" + > "};"; I've also tried other more scaleable ways to work with scripts concurrently, but given that this is the most naive method where everything is brand new and I still get slowness calling them concurrently I fear that maybe I'm overlooking something extremely basic on my code. Thanks. -Jesus Luzon From yikesaroni at gmail.com Tue Dec 6 12:59:22 2016 From: yikesaroni at gmail.com (yikes aroni) Date: Tue, 6 Dec 2016 07:59:22 -0500 Subject: Caching and re-using bindings In-Reply-To: References: Message-ID: Thanks for your response Edmond -- the problem is that a simplebindings object still gets all the changes to the original bindings.... see below: // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings se.eval("globalA = 'testA';"); // Snapshot the bindings from the engine. Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); // print the bindings to confirm there is only one variable in them. --> TRUE printBindings(bEngine); ------------------------ globalA = testA // At this point there is only 'globalA' in the bindings so instantiate simplebindings: Bindings bSimple = new SimpleBindings(bEngine); // Verify by printing the bindings printBindings(bSimple); ------------------------ globalA = testA // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings se.eval("globalB = 'testB';"); // print the bindings. Now both variables are present........... WHY??? printBindings(bEngine); ------------------------ globalA = testA globalB = testB // But look -- they are in the simple bindings obj as well! printBindings(bSimple);------------------------ globalA = testA globalB = testB That's precisely what i don't get --> Why are variables added to the engine bindings *after* instantiating the simpleBindings showing up in the simpleBindings object? And how do i avoid that? thanks On Mon, Dec 5, 2016 at 3:01 PM, Edmond Kemokai wrote: > Why don't you create instances of SimpleBinding and use those as needed? > Use ScriptEngine.setBindings to reset the binding... > > On Dec 5, 2016 1:36 PM, "yikes aroni" wrote: > >> I want to cache ScriptEngine bindings for reuse. The basic algo would be >> >> 1) Build up my ScriptEngine (SE) with stuff i need. >> 2) Snapshot the bindings -- i.e., cache them >> Use engine.getBindings(ScriptContext.ENGINE_SCOPE) >> 3) Use the SE for stuff that might modify its state. >> 4) When done, replace the SE's bindings with my snapshotted bindings >> Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) >> 5) I now have a "fresh" SE to use. >> >> The problem is that this doesn't work as expected. The cached bindings >> appear to point to the actual SE bindings and therefore whatever gets >> added >> to the SE bindings, also gets added to the cached bindings. Here's some >> code to show how it's not doing what i would expect. >> >> public class TempEngineTest { >> >> static ScriptEngineManager seManager = new ScriptEngineManager(); >> static ScriptEngine se = seManager.getEngineByName("nashorn"); >> public static void printBindings(Map bindings) { >> for (Map.Entry entry : bindings.entrySet()) { >> System.out.println(entry.getKey() + " = " + entry.getValue()); >> } >> } >> public static void main(String[] a) throws Exception { >> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalA = 'testA';"); >> // Snapshot the bindings from the engine. >> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >> // print the bindings to confirm there is only one variable in them. --> >> TRUE >> printBindings(bEngine); >> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalB = 'testB';"); >> // print the bindings. Now both variables are present........... WHY??? >> printBindings(bEngine); >> } >> } >> >> I've seen suggestions to cache and reuse bindings in various articles, but >> no specific code for doing so. How do i accomplish this in actual code? >> >> thanks >> > From yikesaroni at gmail.com Tue Dec 6 13:04:14 2016 From: yikesaroni at gmail.com (yikes aroni) Date: Tue, 6 Dec 2016 08:04:14 -0500 Subject: Caching and re-using bindings In-Reply-To: References: Message-ID: Hmm.... it works if i instantiate the SimpleBinding obj and then putAll the bindings from the original bindings obj. Bindings bSimple = new SimpleBindings(); bSimple.putAll(bEngine); Otherwise they either remain the same object or are bound in some way s.t. changes to bEngine are in bSimple. This just seems strange to me. Still curious to know what the reason for this is, despite having found a workaround. On Tue, Dec 6, 2016 at 7:59 AM, yikes aroni wrote: > Thanks for your response Edmond -- the problem is that a simplebindings > object still gets all the changes to the original bindings.... see below: > > // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings > se.eval("globalA = 'testA';"); > // Snapshot the bindings from the engine. > Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); > // print the bindings to confirm there is only one variable in them. --> > TRUE > printBindings(bEngine); > ------------------------ > globalA = testA > // At this point there is only 'globalA' in the bindings so instantiate > simplebindings: > Bindings bSimple = new SimpleBindings(bEngine); > // Verify by printing the bindings > printBindings(bSimple); > ------------------------ > globalA = testA > // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings > se.eval("globalB = 'testB';"); > // print the bindings. Now both variables are present........... WHY??? > printBindings(bEngine); > ------------------------ > globalA = testA > globalB = testB > // But look -- they are in the simple bindings obj as well! > printBindings(bSimple);------------------------ > globalA = testA > globalB = testB > > That's precisely what i don't get --> Why are variables added to the > engine bindings *after* instantiating the simpleBindings showing up in the > simpleBindings object? > > And how do i avoid that? > > thanks > > On Mon, Dec 5, 2016 at 3:01 PM, Edmond Kemokai wrote: > >> Why don't you create instances of SimpleBinding and use those as needed? >> Use ScriptEngine.setBindings to reset the binding... >> >> On Dec 5, 2016 1:36 PM, "yikes aroni" wrote: >> >>> I want to cache ScriptEngine bindings for reuse. The basic algo would be >>> >>> 1) Build up my ScriptEngine (SE) with stuff i need. >>> 2) Snapshot the bindings -- i.e., cache them >>> Use engine.getBindings(ScriptContext.ENGINE_SCOPE) >>> 3) Use the SE for stuff that might modify its state. >>> 4) When done, replace the SE's bindings with my snapshotted bindings >>> Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) >>> 5) I now have a "fresh" SE to use. >>> >>> The problem is that this doesn't work as expected. The cached bindings >>> appear to point to the actual SE bindings and therefore whatever gets >>> added >>> to the SE bindings, also gets added to the cached bindings. Here's some >>> code to show how it's not doing what i would expect. >>> >>> public class TempEngineTest { >>> >>> static ScriptEngineManager seManager = new ScriptEngineManager(); >>> static ScriptEngine se = seManager.getEngineByName("nashorn"); >>> public static void printBindings(Map bindings) { >>> for (Map.Entry entry : bindings.entrySet()) { >>> System.out.println(entry.getKey() + " = " + entry.getValue()); >>> } >>> } >>> public static void main(String[] a) throws Exception { >>> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >>> se.eval("globalA = 'testA';"); >>> // Snapshot the bindings from the engine. >>> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >>> // print the bindings to confirm there is only one variable in them. --> >>> TRUE >>> printBindings(bEngine); >>> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >>> se.eval("globalB = 'testB';"); >>> // print the bindings. Now both variables are present........... WHY??? >>> printBindings(bEngine); >>> } >>> } >>> >>> I've seen suggestions to cache and reuse bindings in various articles, >>> but >>> no specific code for doing so. How do i accomplish this in actual code? >>> >>> thanks >>> >> > From james.laskey at oracle.com Tue Dec 6 13:07:57 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 6 Dec 2016 09:07:57 -0400 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: Message-ID: Jesus, Probably the most informative information is in this blog. https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt This example uses Executors but threads would work as well. I did a talk that looked at different methods to max out multithreading performance. A new engine per thread is the worst case. A new context per thread does much better. A new global per thread is the best while remaining thread safe. Cheers, ? Jim > On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: > > Hey folks, > > I've tried many different ways of using Nashorn multithreaded based on what > I've found on the internet and I still can't get a single one to scale. > Even the most naive method of making many script engines with my script > tends to bottleneck itself when I have more than 10 threads invoking > functions. > > I'm using the following code to compile my script and > invocable.invokeFunction("transform", input) to execute: > >> static Invocable generateInvocable(String script) throws >> ScriptException { >> ScriptEngineManager manager = new ScriptEngineManager(); >> ScriptEngine engine = >> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >> Compilable compilable = (Compilable) engine; >> final CompiledScript compiled = compilable.compile(script); >> compiled.eval(); >> return (Invocable) engine; >> } > > > The script I'm compiling is: > >> String script = "function transform(input) {" + >> "var result = JSON.parse(input);" + >> "response = {};\n" + >> "for (var i = 0; i < result.length; i++) {\n" + >> " var summoner = {};\n" + >> " summoner.id = result[i].id;\n" + >> " summoner.name = result[i].name;\n" + >> " summoner.profileIconId = result[i].profileIconId;\n" + >> " summoner.revisionDate = result[i].revisionDate;\n" + >> " summoner.summonerLevel = result[i].level;\n" + >> " response[summoner.id] = summoner;\n" + >> "}\n" + >> "result = response;" + >> "return JSON.stringify(result);" + >> "};"; > > > I've also tried other more scaleable ways to work with scripts > concurrently, but given that this is the most naive method where everything > is brand new and I still get slowness calling them concurrently I fear that > maybe I'm overlooking something extremely basic on my code. > > Thanks. > -Jesus Luzon From jluzon at riotgames.com Tue Dec 6 13:19:12 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Tue, 6 Dec 2016 05:19:12 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: Message-ID: Hey Jim, I looked at it and I will look into loadWithNewGlobal to see what exactly it does since it could be relevant. As for the rest, for my use case having threads in the JS would not help. We're using Nashorn to build JSON filters in a Dynamic Proxy Service and need any of the threads processing a request to be able to execute the script to filter. Also, when you say a new engine per threads is the worst case what exactly do you mean? I would expect an initial cost of compiling the script on each thread and then each engine should be able to do its own thing, but what I'm seeing is that when running with more than 10 threads all my engines get slow at executing code, even though they are all completely separate from each other. On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) wrote: > Jesus, > > Probably the most informative information is in this blog. > > https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt > > This example uses Executors but threads would work as well. > > I did a talk that looked at different methods to max out multithreading > performance. A new engine per thread is the worst case. A new context per > thread does much better. A new global per thread is the best while > remaining thread safe. > > Cheers, > > ? Jim > > > > > > On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: > > Hey folks, > > I've tried many different ways of using Nashorn multithreaded based on what > I've found on the internet and I still can't get a single one to scale. > Even the most naive method of making many script engines with my script > tends to bottleneck itself when I have more than 10 threads invoking > functions. > > I'm using the following code to compile my script and > invocable.invokeFunction("transform", input) to execute: > > static Invocable generateInvocable(String script) throws > ScriptException { > ScriptEngineManager manager = new ScriptEngineManager(); > ScriptEngine engine = > manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > Compilable compilable = (Compilable) engine; > final CompiledScript compiled = compilable.compile(script); > compiled.eval(); > return (Invocable) engine; > } > > > > The script I'm compiling is: > > String script = "function transform(input) {" + > "var result = JSON.parse(input);" + > "response = {};\n" + > "for (var i = 0; i < result.length; i++) {\n" + > " var summoner = {};\n" + > " summoner.id = result[i].id;\n" + > " summoner.name = result[i].name;\n" + > " summoner.profileIconId = result[i].profileIconId;\n" + > " summoner.revisionDate = result[i].revisionDate;\n" + > " summoner.summonerLevel = result[i].level;\n" + > " response[summoner.id] = summoner;\n" + > "}\n" + > "result = response;" + > "return JSON.stringify(result);" + > "};"; > > > > I've also tried other more scaleable ways to work with scripts > concurrently, but given that this is the most naive method where everything > is brand new and I still get slowness calling them concurrently I fear that > maybe I'm overlooking something extremely basic on my code. > > Thanks. > -Jesus Luzon > > > From james.laskey at oracle.com Tue Dec 6 13:31:51 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 6 Dec 2016 09:31:51 -0400 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: Message-ID: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> > On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: > > Hey Jim, > > I looked at it and I will look into loadWithNewGlobal to see what exactly it does since it could be relevant. As for the rest, for my use case having threads in the JS would not help. We're using Nashorn to build JSON filters in a Dynamic Proxy Service and need any of the threads processing a request to be able to execute the script to filter. The cost of creating a new engine is significant. So share an engine across threads but use eval (String script, ScriptContext context) instead, separate context per execution. If your JavaScript code does not modify globals you can get away with using the same engine, same compiled script on each thread. > > Also, when you say a new engine per threads is the worst case what exactly do you mean? I would expect an initial cost of compiling the script on each thread and then each engine should be able to do its own thing, but what I'm seeing is that when running with more than 10 threads all my engines get slow at executing code, even though they are all completely separate from each other. Of course there are many factors involved n performance. How many cores do you have on the test machine? How much memory in the process? What transport are you using between threads? That sort of thing. Other than constructing then engine and context Nashorn performance should scale. > > On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) > wrote: > Jesus, > > Probably the most informative information is in this blog. > > https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt > > This example uses Executors but threads would work as well. > > I did a talk that looked at different methods to max out multithreading performance. A new engine per thread is the worst case. A new context per thread does much better. A new global per thread is the best while remaining thread safe. > > Cheers, > > ? Jim > > > > > >> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: >> >> Hey folks, >> >> I've tried many different ways of using Nashorn multithreaded based on what >> I've found on the internet and I still can't get a single one to scale. >> Even the most naive method of making many script engines with my script >> tends to bottleneck itself when I have more than 10 threads invoking >> functions. >> >> I'm using the following code to compile my script and >> invocable.invokeFunction("transform", input) to execute: >> >>> static Invocable generateInvocable(String script) throws >>> ScriptException { >>> ScriptEngineManager manager = new ScriptEngineManager(); >>> ScriptEngine engine = >>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>> Compilable compilable = (Compilable) engine; >>> final CompiledScript compiled = compilable.compile(script); >>> compiled.eval(); >>> return (Invocable) engine; >>> } >> >> >> The script I'm compiling is: >> >>> String script = "function transform(input) {" + >>> "var result = JSON.parse(input);" + >>> "response = {};\n" + >>> "for (var i = 0; i < result.length; i++) {\n" + >>> " var summoner = {};\n" + >>> " summoner.id = result[i].id;\n" + >>> " summoner.name = result[i].name;\n" + >>> " summoner.profileIconId = result[i].profileIconId;\n" + >>> " summoner.revisionDate = result[i].revisionDate;\n" + >>> " summoner.summonerLevel = result[i].level;\n" + >>> " response[summoner.id ] = summoner;\n" + >>> "}\n" + >>> "result = response;" + >>> "return JSON.stringify(result);" + >>> "};"; >> >> >> I've also tried other more scaleable ways to work with scripts >> concurrently, but given that this is the most naive method where everything >> is brand new and I still get slowness calling them concurrently I fear that >> maybe I'm overlooking something extremely basic on my code. >> >> Thanks. >> -Jesus Luzon From jluzon at riotgames.com Tue Dec 6 13:56:12 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Tue, 6 Dec 2016 05:56:12 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: > > The cost of creating a new engine is significant. So share an engine > across threads but use *eval > * > (String > script, ScriptContext > > context) instead, separate context per execution. If your JavaScript > code does not modify globals you can get away with using the same engine, > same compiled script on each thread. I guess there's a few things here I don't understand. One thing I'm trying to do is sharing a CompiledScript (which is why I'm using invocable). Also, what exactly does modify globals mean? All our filters do the same thing, make a function that takes a JSON String, turns it into a JSON, modifies it and then stringifies it back. No state is changed of anything else but there are temporary vars created inside the scope of the function. When we run this multithreaded, running invokeFunction slows down significantly and we get crazy memory leaks. Of course there are many factors involved n performance. How many cores do > you have on the test machine? How much memory in the process? What > transport are you using between threads? That sort of thing. Other than > constructing then engine and context Nashorn performance should scale. I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the process. Not sure what transports between threads means, but this is the code I'm benchmarking with. Increasing the number of threads actually makes it go faster until about 4 threads, then adding more threads takes the same amount to get to 1000 and and after a certain point it is just slower to get to 1000 counts. Some of our filters need to be able to run over 1000 times a second (across all threads) and the fastest time I could actually get with this was about 2.4 seconds for a 1000 counts. > ExecutorService executor = Executors.newFixedThreadPool(50); > > AtomicLong count = new AtomicLong(); > > for (int i = 0; i < 50; i++) { > > executor.submit(new Runnable() { > > @Override > > public void run() { > > >> try { > > Invocable invocable = >> generateInvocable(script); > > while(true) { > > invocable.invokeFunction("transform", >> something); > > count.incrementAndGet(); > > } > > } catch (NoSuchMethodException | ScriptException >> e) { > > e.printStackTrace(); > > } > > } > > }); > > } > > long lastTimestamp = System.currentTimeMillis(); > > while(true) { > > >> if (count.get() > 1000) { > > count.getAndAdd(-1000); > > System.out.println((System.currentTimeMillis() - >> lastTimestamp)/1000.0); > > lastTimestamp = System.currentTimeMillis(); > > } > > } > > On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) wrote: > > On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: > > Hey Jim, > > I looked at it and I will look into loadWithNewGlobal to see what exactly > it does since it could be relevant. As for the rest, for my use case having > threads in the JS would not help. We're using Nashorn to build JSON filters > in a Dynamic Proxy Service and need any of the threads processing a request > to be able to execute the script to filter. > > > The cost of creating a new engine is significant. So share an engine > across threads but use *eval > * > (String > script, ScriptContext > > context) instead, separate context per execution. If your JavaScript > code does not modify globals you can get away with using the same engine, > same compiled script on each thread. > > > Also, when you say a new engine per threads is the worst case what exactly > do you mean? I would expect an initial cost of compiling the script on each > thread and then each engine should be able to do its own thing, but what > I'm seeing is that when running with more than 10 threads all my engines > get slow at executing code, even though they are all completely separate > from each other. > > > Of course there are many factors involved n performance. How many cores > do you have on the test machine? How much memory in the process? What > transport are you using between threads? That sort of thing. Other than > constructing then engine and context Nashorn performance should scale. > > > On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) com> wrote: > >> Jesus, >> >> Probably the most informative information is in this blog. >> >> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >> >> This example uses Executors but threads would work as well. >> >> I did a talk that looked at different methods to max out multithreading >> performance. A new engine per thread is the worst case. A new context per >> thread does much better. A new global per thread is the best while >> remaining thread safe. >> >> Cheers, >> >> ? Jim >> >> >> >> >> >> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: >> >> Hey folks, >> >> I've tried many different ways of using Nashorn multithreaded based on >> what >> I've found on the internet and I still can't get a single one to scale. >> Even the most naive method of making many script engines with my script >> tends to bottleneck itself when I have more than 10 threads invoking >> functions. >> >> I'm using the following code to compile my script and >> invocable.invokeFunction("transform", input) to execute: >> >> static Invocable generateInvocable(String script) throws >> ScriptException { >> ScriptEngineManager manager = new ScriptEngineManager(); >> ScriptEngine engine = >> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >> Compilable compilable = (Compilable) engine; >> final CompiledScript compiled = compilable.compile(script); >> compiled.eval(); >> return (Invocable) engine; >> } >> >> >> >> The script I'm compiling is: >> >> String script = "function transform(input) {" + >> "var result = JSON.parse(input);" + >> "response = {};\n" + >> "for (var i = 0; i < result.length; i++) {\n" + >> " var summoner = {};\n" + >> " summoner.id = result[i].id;\n" + >> " summoner.name = result[i].name;\n" + >> " summoner.profileIconId = result[i].profileIconId;\n" + >> " summoner.revisionDate = result[i].revisionDate;\n" + >> " summoner.summonerLevel = result[i].level;\n" + >> " response[summoner.id] = summoner;\n" + >> "}\n" + >> "result = response;" + >> "return JSON.stringify(result);" + >> "};"; >> >> >> >> I've also tried other more scaleable ways to work with scripts >> concurrently, but given that this is the most naive method where >> everything >> is brand new and I still get slowness calling them concurrently I fear >> that >> maybe I'm overlooking something extremely basic on my code. >> >> Thanks. >> -Jesus Luzon >> >> > From ekemokai at gmail.com Tue Dec 6 13:56:44 2016 From: ekemokai at gmail.com (Edmond Kemokai) Date: Tue, 6 Dec 2016 08:56:44 -0500 Subject: Caching and re-using bindings In-Reply-To: References: Message-ID: I doubt there is a problem, you're probably making a minor error. I am the developer of HiveMind, it is a web app platform that relies entirely on scripting engines. It doesn't cache bindings but rather caches the engine and creates a new binding to handle every request. You can access a demo instance: http://demo.crudzilla.com:7000/ Login with: developer/developer I have created a test file in: /com/crudzilla/cloudTest/web/binding-test.ste Put your code in there and save, I'll take a look to see what might be missing. On Tue, Dec 6, 2016 at 8:04 AM, yikes aroni wrote: > Hmm.... it works if i instantiate the SimpleBinding obj and then putAll > the bindings from the original bindings obj. > > Bindings bSimple = new SimpleBindings(); > bSimple.putAll(bEngine); > > Otherwise they either remain the same object or are bound in some way s.t. > changes to bEngine are in bSimple. This just seems strange to me. Still > curious to know what the reason for this is, despite having found a > workaround. > > > > On Tue, Dec 6, 2016 at 7:59 AM, yikes aroni wrote: > >> Thanks for your response Edmond -- the problem is that a simplebindings >> object still gets all the changes to the original bindings.... see below: >> >> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalA = 'testA';"); >> // Snapshot the bindings from the engine. >> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >> // print the bindings to confirm there is only one variable in them. --> >> TRUE >> printBindings(bEngine); >> ------------------------ >> globalA = testA >> // At this point there is only 'globalA' in the bindings so instantiate >> simplebindings: >> Bindings bSimple = new SimpleBindings(bEngine); >> // Verify by printing the bindings >> printBindings(bSimple); >> ------------------------ >> globalA = testA >> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalB = 'testB';"); >> // print the bindings. Now both variables are present........... WHY??? >> printBindings(bEngine); >> ------------------------ >> globalA = testA >> globalB = testB >> // But look -- they are in the simple bindings obj as well! >> printBindings(bSimple);------------------------ >> globalA = testA >> globalB = testB >> >> That's precisely what i don't get --> Why are variables added to the >> engine bindings *after* instantiating the simpleBindings showing up in the >> simpleBindings object? >> >> And how do i avoid that? >> >> thanks >> >> On Mon, Dec 5, 2016 at 3:01 PM, Edmond Kemokai >> wrote: >> >>> Why don't you create instances of SimpleBinding and use those as needed? >>> Use ScriptEngine.setBindings to reset the binding... >>> >>> On Dec 5, 2016 1:36 PM, "yikes aroni" wrote: >>> >>>> I want to cache ScriptEngine bindings for reuse. The basic algo would be >>>> >>>> 1) Build up my ScriptEngine (SE) with stuff i need. >>>> 2) Snapshot the bindings -- i.e., cache them >>>> Use engine.getBindings(ScriptContext.ENGINE_SCOPE) >>>> 3) Use the SE for stuff that might modify its state. >>>> 4) When done, replace the SE's bindings with my snapshotted bindings >>>> Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) >>>> 5) I now have a "fresh" SE to use. >>>> >>>> The problem is that this doesn't work as expected. The cached bindings >>>> appear to point to the actual SE bindings and therefore whatever gets >>>> added >>>> to the SE bindings, also gets added to the cached bindings. Here's some >>>> code to show how it's not doing what i would expect. >>>> >>>> public class TempEngineTest { >>>> >>>> static ScriptEngineManager seManager = new ScriptEngineManager(); >>>> static ScriptEngine se = seManager.getEngineByName("nashorn"); >>>> public static void printBindings(Map bindings) { >>>> for (Map.Entry entry : bindings.entrySet()) { >>>> System.out.println(entry.getKey() + " = " + entry.getValue()); >>>> } >>>> } >>>> public static void main(String[] a) throws Exception { >>>> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >>>> se.eval("globalA = 'testA';"); >>>> // Snapshot the bindings from the engine. >>>> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >>>> // print the bindings to confirm there is only one variable in them. --> >>>> TRUE >>>> printBindings(bEngine); >>>> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >>>> se.eval("globalB = 'testB';"); >>>> // print the bindings. Now both variables are present........... WHY??? >>>> printBindings(bEngine); >>>> } >>>> } >>>> >>>> I've seen suggestions to cache and reuse bindings in various articles, >>>> but >>>> no specific code for doing so. How do i accomplish this in actual code? >>>> >>>> thanks >>>> >>> >> > From james.laskey at oracle.com Tue Dec 6 14:30:56 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 6 Dec 2016 10:30:56 -0400 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: > On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: > > The cost of creating a new engine is significant. So share an engine across threads but use eval (String script, ScriptContext context) instead, separate context per execution. If your JavaScript code does not modify globals you can get away with using the same engine, same compiled script on each thread. > > I guess there's a few things here I don't understand. One thing I'm trying to do is sharing a CompiledScript (which is why I'm using invocable). Also, what exactly does modify globals mean? All our filters do the same thing, make a function that takes a JSON String, turns it into a JSON, modifies it and then stringifies it back. No state is changed of anything else but there are temporary vars created inside the scope of the function. When we run this multithreaded, running invokeFunction slows down significantly and we get crazy memory leaks. So you are using the same invocable instance for all threads? If so, then you are probably good to go. As far as leaks are concerned, not sure how you would get leaks from Nashorn. The JSON object is written in Java, and little JavaScript involved. > > Of course there are many factors involved n performance. How many cores do you have on the test machine? How much memory in the process? What transport are you using between threads? That sort of thing. Other than constructing then engine and context Nashorn performance should scale. > I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the process. Not sure what transports between threads means, but this is the code I'm benchmarking with. Increasing the number of threads actually makes it go faster until about 4 threads, then adding more threads takes the same amount to get to 1000 and and after a certain point it is just slower to get to 1000 counts. Some of our filters need to be able to run over 1000 times a second (across all threads) and the fastest time I could actually get with this was about 2.4 seconds for a 1000 counts. > ExecutorService executor = Executors.newFixedThreadPool(50); > AtomicLong count = new AtomicLong(); > for (int i = 0; i < 50; i++) { > executor.submit(new Runnable() { > @Override > public void run() { > > try { > Invocable invocable = generateInvocable(script); > while(true) { > invocable.invokeFunction("transform", something); > count.incrementAndGet(); > } > } catch (NoSuchMethodException | ScriptException e) { > e.printStackTrace(); > } > } > }); > } > long lastTimestamp = System.currentTimeMillis(); > while(true) { > > if (count.get() > 1000) { > count.getAndAdd(-1000); > System.out.println((System.currentTimeMillis() - lastTimestamp)/1000.0); > lastTimestamp = System.currentTimeMillis(); > } > } With more threads you are impacting the same 8 cores, so it will taper off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a threshold depending on System performance. Transport: I meant if you were using sockets to provide the script. In your example, pull up Invocable invocable = generateInvocable(script); out of the loop and use the same invocable for all threads. - Jim > > On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) > wrote: > >> On Dec 6, 2016, at 9:19 AM, Jesus Luzon > wrote: >> >> Hey Jim, >> >> I looked at it and I will look into loadWithNewGlobal to see what exactly it does since it could be relevant. As for the rest, for my use case having threads in the JS would not help. We're using Nashorn to build JSON filters in a Dynamic Proxy Service and need any of the threads processing a request to be able to execute the script to filter. > > The cost of creating a new engine is significant. So share an engine across threads but use eval (String script, ScriptContext context) instead, separate context per execution. If your JavaScript code does not modify globals you can get away with using the same engine, same compiled script on each thread. > >> >> Also, when you say a new engine per threads is the worst case what exactly do you mean? I would expect an initial cost of compiling the script on each thread and then each engine should be able to do its own thing, but what I'm seeing is that when running with more than 10 threads all my engines get slow at executing code, even though they are all completely separate from each other. > > Of course there are many factors involved n performance. How many cores do you have on the test machine? How much memory in the process? What transport are you using between threads? That sort of thing. Other than constructing then engine and context Nashorn performance should scale. > >> >> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) > wrote: >> Jesus, >> >> Probably the most informative information is in this blog. >> >> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >> >> This example uses Executors but threads would work as well. >> >> I did a talk that looked at different methods to max out multithreading performance. A new engine per thread is the worst case. A new context per thread does much better. A new global per thread is the best while remaining thread safe. >> >> Cheers, >> >> ? Jim >> >> >> >> >> >>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: >>> >>> Hey folks, >>> >>> I've tried many different ways of using Nashorn multithreaded based on what >>> I've found on the internet and I still can't get a single one to scale. >>> Even the most naive method of making many script engines with my script >>> tends to bottleneck itself when I have more than 10 threads invoking >>> functions. >>> >>> I'm using the following code to compile my script and >>> invocable.invokeFunction("transform", input) to execute: >>> >>>> static Invocable generateInvocable(String script) throws >>>> ScriptException { >>>> ScriptEngineManager manager = new ScriptEngineManager(); >>>> ScriptEngine engine = >>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>>> Compilable compilable = (Compilable) engine; >>>> final CompiledScript compiled = compilable.compile(script); >>>> compiled.eval(); >>>> return (Invocable) engine; >>>> } >>> >>> >>> The script I'm compiling is: >>> >>>> String script = "function transform(input) {" + >>>> "var result = JSON.parse(input);" + >>>> "response = {};\n" + >>>> "for (var i = 0; i < result.length; i++) {\n" + >>>> " var summoner = {};\n" + >>>> " summoner.id = result[i].id;\n" + >>>> " summoner.name = result[i].name;\n" + >>>> " summoner.profileIconId = result[i].profileIconId;\n" + >>>> " summoner.revisionDate = result[i].revisionDate;\n" + >>>> " summoner.summonerLevel = result[i].level;\n" + >>>> " response[summoner.id ] = summoner;\n" + >>>> "}\n" + >>>> "result = response;" + >>>> "return JSON.stringify(result);" + >>>> "};"; >>> >>> >>> I've also tried other more scaleable ways to work with scripts >>> concurrently, but given that this is the most naive method where everything >>> is brand new and I still get slowness calling them concurrently I fear that >>> maybe I'm overlooking something extremely basic on my code. >>> >>> Thanks. >>> -Jesus Luzon > > From jluzon at riotgames.com Tue Dec 6 21:12:16 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Tue, 6 Dec 2016 13:12:16 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: > > With more threads you are impacting the same 8 cores, so it will taper off > after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > threshold depending on System performance. Transport: I meant if you were > using sockets to provide the script. This makes sense. This one's on me then. > So you are using the same invocable instance for all threads? If so, then > you are probably good to go. As far as leaks are concerned, not sure how > you would get leaks from Nashorn. The JSON object is written in Java, and > little JavaScript involved. > In your example, pull up Invocable invocable = generateInvocable(script); > out of the loop and use the same invocable for all threads. We were using one invocable across all threads and we were getting slowdowns on execution, high CPU Usage and memory leaks that led to OutOfMemory errors. I could trace the leak to jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> jdk.nashorn.internal.scripts.JO4 -> *arrayData* jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter which just keeps growing forever. On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) wrote: > > On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: > > The cost of creating a new engine is significant. So share an engine >> across threads but use *eval >> * >> (String >> script, ScriptContext >> >> context) instead, separate context per execution. If your JavaScript >> code does not modify globals you can get away with using the same engine, >> same compiled script on each thread. > > > I guess there's a few things here I don't understand. One thing I'm trying > to do is sharing a CompiledScript (which is why I'm using invocable). Also, > what exactly does modify globals mean? All our filters do the same thing, > make a function that takes a JSON String, turns it into a JSON, modifies it > and then stringifies it back. No state is changed of anything else but > there are temporary vars created inside the scope of the function. When we > run this multithreaded, running invokeFunction slows down significantly and > we get crazy memory leaks. > > > So you are using the same invocable instance for all threads? If so, then > you are probably good to go. As far as leaks are concerned, not sure how > you would get leaks from Nashorn. The JSON object is written in Java, and > little JavaScript involved. > > > Of course there are many factors involved n performance. How many cores >> do you have on the test machine? How much memory in the process? What >> transport are you using between threads? That sort of thing. Other than >> constructing then engine and context Nashorn performance should scale. > > I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the > process. Not sure what transports between threads means, but this is the > code I'm benchmarking with. Increasing the number of threads actually makes > it go faster until about 4 threads, then adding more threads takes the same > amount to get to 1000 and and after a certain point it is just slower to > get to 1000 counts. Some of our filters need to be able to run over 1000 > times a second (across all threads) and the fastest time I could actually > get with this was about 2.4 seconds for a 1000 counts. > >> ExecutorService executor = Executors.newFixedThreadPool(50); >> >> AtomicLong count = new AtomicLong(); >> >> for (int i = 0; i < 50; i++) { >> >> executor.submit(new Runnable() { >> >> @Override >> >> public void run() { >> >> >>> try { >> >> Invocable invocable = >>> generateInvocable(script); >> >> while(true) { >> >> invocable.invokeFunction("transform", >>> something); >> >> count.incrementAndGet(); >> >> } >> >> } catch (NoSuchMethodException | ScriptException >>> e) { >> >> e.printStackTrace(); >> >> } >> >> } >> >> }); >> >> } >> >> long lastTimestamp = System.currentTimeMillis(); >> >> while(true) { >> >> >>> if (count.get() > 1000) { >> >> count.getAndAdd(-1000); >> >> System.out.println((System.currentTimeMillis() - >>> lastTimestamp)/1000.0); >> >> lastTimestamp = System.currentTimeMillis(); >> >> } >> >> } >> >> > With more threads you are impacting the same 8 cores, so it will taper off > after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > threshold depending on System performance. Transport: I meant if you were > using sockets to provide the script. > > In your example, pull up Invocable invocable = generateInvocable(script); > out of the loop and use the same invocable for all threads. > > - Jim > > > > > On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > james.laskey at oracle.com> wrote: > >> >> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: >> >> Hey Jim, >> >> I looked at it and I will look into loadWithNewGlobal to see what exactly >> it does since it could be relevant. As for the rest, for my use case having >> threads in the JS would not help. We're using Nashorn to build JSON filters >> in a Dynamic Proxy Service and need any of the threads processing a request >> to be able to execute the script to filter. >> >> >> The cost of creating a new engine is significant. So share an engine >> across threads but use *eval >> * >> (String >> script, ScriptContext >> >> context) instead, separate context per execution. If your JavaScript >> code does not modify globals you can get away with using the same engine, >> same compiled script on each thread. >> >> >> Also, when you say a new engine per threads is the worst case what >> exactly do you mean? I would expect an initial cost of compiling the script >> on each thread and then each engine should be able to do its own thing, but >> what I'm seeing is that when running with more than 10 threads all my >> engines get slow at executing code, even though they are all completely >> separate from each other. >> >> >> Of course there are many factors involved n performance. How many cores >> do you have on the test machine? How much memory in the process? What >> transport are you using between threads? That sort of thing. Other than >> constructing then engine and context Nashorn performance should scale. >> >> >> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) > com> wrote: >> >>> Jesus, >>> >>> Probably the most informative information is in this blog. >>> >>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >>> >>> This example uses Executors but threads would work as well. >>> >>> I did a talk that looked at different methods to max out multithreading >>> performance. A new engine per thread is the worst case. A new context per >>> thread does much better. A new global per thread is the best while >>> remaining thread safe. >>> >>> Cheers, >>> >>> ? Jim >>> >>> >>> >>> >>> >>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: >>> >>> Hey folks, >>> >>> I've tried many different ways of using Nashorn multithreaded based on >>> what >>> I've found on the internet and I still can't get a single one to scale. >>> Even the most naive method of making many script engines with my script >>> tends to bottleneck itself when I have more than 10 threads invoking >>> functions. >>> >>> I'm using the following code to compile my script and >>> invocable.invokeFunction("transform", input) to execute: >>> >>> static Invocable generateInvocable(String script) throws >>> ScriptException { >>> ScriptEngineManager manager = new ScriptEngineManager(); >>> ScriptEngine engine = >>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>> Compilable compilable = (Compilable) engine; >>> final CompiledScript compiled = compilable.compile(script); >>> compiled.eval(); >>> return (Invocable) engine; >>> } >>> >>> >>> >>> The script I'm compiling is: >>> >>> String script = "function transform(input) {" + >>> "var result = JSON.parse(input);" + >>> "response = {};\n" + >>> "for (var i = 0; i < result.length; i++) {\n" + >>> " var summoner = {};\n" + >>> " summoner.id = result[i].id;\n" + >>> " summoner.name = result[i].name;\n" + >>> " summoner.profileIconId = result[i].profileIconId;\n" >>> + >>> " summoner.revisionDate = result[i].revisionDate;\n" + >>> " summoner.summonerLevel = result[i].level;\n" + >>> " response[summoner.id] = summoner;\n" + >>> "}\n" + >>> "result = response;" + >>> "return JSON.stringify(result);" + >>> "};"; >>> >>> >>> >>> I've also tried other more scaleable ways to work with scripts >>> concurrently, but given that this is the most naive method where >>> everything >>> is brand new and I still get slowness calling them concurrently I fear >>> that >>> maybe I'm overlooking something extremely basic on my code. >>> >>> Thanks. >>> -Jesus Luzon >>> >>> >> > > From james.laskey at oracle.com Tue Dec 6 22:59:13 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 6 Dec 2016 18:59:13 -0400 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Intersting. The example you posted demonstrates this behaviour? If so I?ll file a bug and dig in. It sounds like an object is being reused across invocations and accumulating changes to the property map. ? Jim > On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > > With more threads you are impacting the same 8 cores, so it will taper off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a threshold depending on System performance. Transport: I meant if you were using sockets to provide the script. > This makes sense. This one's on me then. > > So you are using the same invocable instance for all threads? If so, then you are probably good to go. As far as leaks are concerned, not sure how you would get leaks from Nashorn. The JSON object is written in Java, and little JavaScript involved. > > In your example, pull up Invocable invocable = generateInvocable(script); out of the loop and use the same invocable for all threads. > > We were using one invocable across all threads and we were getting slowdowns on execution, high CPU Usage and memory leaks that led to OutOfMemory errors. I could trace the leak to > jdk.nashorn.internal.objects.Global -> objectSpill Object[8] -> jdk.nashorn.internal.scripts.JO4 -> arrayData jdk.nashorn.internal.runtime.arrays.SparseArraysData -> underlying jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > which just keeps growing forever. > > On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) > wrote: > >> On Dec 6, 2016, at 9:56 AM, Jesus Luzon > wrote: >> >> The cost of creating a new engine is significant. So share an engine across threads but use eval (String script, ScriptContext context) instead, separate context per execution. If your JavaScript code does not modify globals you can get away with using the same engine, same compiled script on each thread. >> >> I guess there's a few things here I don't understand. One thing I'm trying to do is sharing a CompiledScript (which is why I'm using invocable). Also, what exactly does modify globals mean? All our filters do the same thing, make a function that takes a JSON String, turns it into a JSON, modifies it and then stringifies it back. No state is changed of anything else but there are temporary vars created inside the scope of the function. When we run this multithreaded, running invokeFunction slows down significantly and we get crazy memory leaks. > > So you are using the same invocable instance for all threads? If so, then you are probably good to go. As far as leaks are concerned, not sure how you would get leaks from Nashorn. The JSON object is written in Java, and little JavaScript involved. > >> >> Of course there are many factors involved n performance. How many cores do you have on the test machine? How much memory in the process? What transport are you using between threads? That sort of thing. Other than constructing then engine and context Nashorn performance should scale. >> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the process. Not sure what transports between threads means, but this is the code I'm benchmarking with. Increasing the number of threads actually makes it go faster until about 4 threads, then adding more threads takes the same amount to get to 1000 and and after a certain point it is just slower to get to 1000 counts. Some of our filters need to be able to run over 1000 times a second (across all threads) and the fastest time I could actually get with this was about 2.4 seconds for a 1000 counts. >> ExecutorService executor = Executors.newFixedThreadPool(50); >> AtomicLong count = new AtomicLong(); >> for (int i = 0; i < 50; i++) { >> executor.submit(new Runnable() { >> @Override >> public void run() { >> >> try { >> Invocable invocable = generateInvocable(script); >> while(true) { >> invocable.invokeFunction("transform", something); >> count.incrementAndGet(); >> } >> } catch (NoSuchMethodException | ScriptException e) { >> e.printStackTrace(); >> } >> } >> }); >> } >> long lastTimestamp = System.currentTimeMillis(); >> while(true) { >> >> if (count.get() > 1000) { >> count.getAndAdd(-1000); >> System.out.println((System.currentTimeMillis() - lastTimestamp)/1000.0); >> lastTimestamp = System.currentTimeMillis(); >> } >> } > > With more threads you are impacting the same 8 cores, so it will taper off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a threshold depending on System performance. Transport: I meant if you were using sockets to provide the script. > > In your example, pull up Invocable invocable = generateInvocable(script); out of the loop and use the same invocable for all threads. > > - Jim > > > >> >> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) > wrote: >> >>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon > wrote: >>> >>> Hey Jim, >>> >>> I looked at it and I will look into loadWithNewGlobal to see what exactly it does since it could be relevant. As for the rest, for my use case having threads in the JS would not help. We're using Nashorn to build JSON filters in a Dynamic Proxy Service and need any of the threads processing a request to be able to execute the script to filter. >> >> The cost of creating a new engine is significant. So share an engine across threads but use eval (String script, ScriptContext context) instead, separate context per execution. If your JavaScript code does not modify globals you can get away with using the same engine, same compiled script on each thread. >> >>> >>> Also, when you say a new engine per threads is the worst case what exactly do you mean? I would expect an initial cost of compiling the script on each thread and then each engine should be able to do its own thing, but what I'm seeing is that when running with more than 10 threads all my engines get slow at executing code, even though they are all completely separate from each other. >> >> Of course there are many factors involved n performance. How many cores do you have on the test machine? How much memory in the process? What transport are you using between threads? That sort of thing. Other than constructing then engine and context Nashorn performance should scale. >> >>> >>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) > wrote: >>> Jesus, >>> >>> Probably the most informative information is in this blog. >>> >>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >>> >>> This example uses Executors but threads would work as well. >>> >>> I did a talk that looked at different methods to max out multithreading performance. A new engine per thread is the worst case. A new context per thread does much better. A new global per thread is the best while remaining thread safe. >>> >>> Cheers, >>> >>> ? Jim >>> >>> >>> >>> >>> >>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: >>>> >>>> Hey folks, >>>> >>>> I've tried many different ways of using Nashorn multithreaded based on what >>>> I've found on the internet and I still can't get a single one to scale. >>>> Even the most naive method of making many script engines with my script >>>> tends to bottleneck itself when I have more than 10 threads invoking >>>> functions. >>>> >>>> I'm using the following code to compile my script and >>>> invocable.invokeFunction("transform", input) to execute: >>>> >>>>> static Invocable generateInvocable(String script) throws >>>>> ScriptException { >>>>> ScriptEngineManager manager = new ScriptEngineManager(); >>>>> ScriptEngine engine = >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>>>> Compilable compilable = (Compilable) engine; >>>>> final CompiledScript compiled = compilable.compile(script); >>>>> compiled.eval(); >>>>> return (Invocable) engine; >>>>> } >>>> >>>> >>>> The script I'm compiling is: >>>> >>>>> String script = "function transform(input) {" + >>>>> "var result = JSON.parse(input);" + >>>>> "response = {};\n" + >>>>> "for (var i = 0; i < result.length; i++) {\n" + >>>>> " var summoner = {};\n" + >>>>> " summoner.id = result[i].id;\n" + >>>>> " summoner.name = result[i].name;\n" + >>>>> " summoner.profileIconId = result[i].profileIconId;\n" + >>>>> " summoner.revisionDate = result[i].revisionDate;\n" + >>>>> " summoner.summonerLevel = result[i].level;\n" + >>>>> " response[summoner.id ] = summoner;\n" + >>>>> "}\n" + >>>>> "result = response;" + >>>>> "return JSON.stringify(result);" + >>>>> "};"; >>>> >>>> >>>> I've also tried other more scaleable ways to work with scripts >>>> concurrently, but given that this is the most naive method where everything >>>> is brand new and I still get slowness calling them concurrently I fear that >>>> maybe I'm overlooking something extremely basic on my code. >>>> >>>> Thanks. >>>> -Jesus Luzon >> >> > > From jluzon at riotgames.com Tue Dec 6 23:03:54 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Tue, 6 Dec 2016 15:03:54 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: When we share one invocable across many threads and run invokeFunction it happens, such as this: ExecutorService executor = Executors.newFixedThreadPool(50); > > Invocable invocable = generateInvocable(script); > > AtomicLong count = new AtomicLong(); > > for (int i = 0; i < 50; i++) { > > executor.submit(new Runnable() { > > @Override > > public void run() { > > try { > > while(true) { > > invocable.invokeFunction("transform", >> something); > > count.incrementAndGet(); > > } > > } catch (NoSuchMethodException | ScriptException >> e) { > > e.printStackTrace(); > > } > > } > > }); > > } > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) wrote: > Intersting. The example you posted demonstrates this behaviour? If so > I?ll file a bug and dig in. It sounds like an object is being reused > across invocations and accumulating changes to the property map. > > ? Jim > > > On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > > With more threads you are impacting the same 8 cores, so it will taper off >> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a >> threshold depending on System performance. Transport: I meant if you were >> using sockets to provide the script. > > This makes sense. This one's on me then. > > >> So you are using the same invocable instance for all threads? If so, >> then you are probably good to go. As far as leaks are concerned, not sure >> how you would get leaks from Nashorn. The JSON object is written in Java, >> and little JavaScript involved. > > > >> In your example, pull up Invocable invocable = generateInvocable(script); >> out of the loop and use the same invocable for all threads. > > > We were using one invocable across all threads and we were getting > slowdowns on execution, high CPU Usage and memory leaks that led to > OutOfMemory errors. I could trace the leak to > > jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > jdk.nashorn.internal.scripts.JO4 -> *arrayData* > jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > > which just keeps growing forever. > > On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > james.laskey at oracle.com> wrote: > >> >> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: >> >> The cost of creating a new engine is significant. So share an engine >>> across threads but use *eval >>> * >>> (String >>> >>> script, ScriptContext >>> >>> context) instead, separate context per execution. If your JavaScript >>> code does not modify globals you can get away with using the same engine, >>> same compiled script on each thread. >> >> >> I guess there's a few things here I don't understand. One thing I'm >> trying to do is sharing a CompiledScript (which is why I'm using >> invocable). Also, what exactly does modify globals mean? All our filters do >> the same thing, make a function that takes a JSON String, turns it into a >> JSON, modifies it and then stringifies it back. No state is changed of >> anything else but there are temporary vars created inside the scope of the >> function. When we run this multithreaded, running invokeFunction slows down >> significantly and we get crazy memory leaks. >> >> >> So you are using the same invocable instance for all threads? If so, >> then you are probably good to go. As far as leaks are concerned, not sure >> how you would get leaks from Nashorn. The JSON object is written in Java, >> and little JavaScript involved. >> >> >> Of course there are many factors involved n performance. How many cores >>> do you have on the test machine? How much memory in the process? What >>> transport are you using between threads? That sort of thing. Other than >>> constructing then engine and context Nashorn performance should scale. >> >> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the >> process. Not sure what transports between threads means, but this is the >> code I'm benchmarking with. Increasing the number of threads actually makes >> it go faster until about 4 threads, then adding more threads takes the same >> amount to get to 1000 and and after a certain point it is just slower to >> get to 1000 counts. Some of our filters need to be able to run over 1000 >> times a second (across all threads) and the fastest time I could actually >> get with this was about 2.4 seconds for a 1000 counts. >> >>> ExecutorService executor = Executors.newFixedThreadPool(50); >>> >>> AtomicLong count = new AtomicLong(); >>> >>> for (int i = 0; i < 50; i++) { >>> >>> executor.submit(new Runnable() { >>> >>> @Override >>> >>> public void run() { >>> >>> >>>> try { >>> >>> Invocable invocable = >>>> generateInvocable(script); >>> >>> while(true) { >>> >>> invocable.invokeFunction("transform", >>>> something); >>> >>> count.incrementAndGet(); >>> >>> } >>> >>> } catch (NoSuchMethodException | ScriptException >>>> e) { >>> >>> e.printStackTrace(); >>> >>> } >>> >>> } >>> >>> }); >>> >>> } >>> >>> long lastTimestamp = System.currentTimeMillis(); >>> >>> while(true) { >>> >>> >>>> if (count.get() > 1000) { >>> >>> count.getAndAdd(-1000); >>> >>> System.out.println((System.currentTimeMillis() - >>>> lastTimestamp)/1000.0); >>> >>> lastTimestamp = System.currentTimeMillis(); >>> >>> } >>> >>> } >>> >>> >> With more threads you are impacting the same 8 cores, so it will taper >> off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a >> threshold depending on System performance. Transport: I meant if you were >> using sockets to provide the script. >> >> In your example, pull up Invocable invocable = generateInvocable(script); >> out of the loop and use the same invocable for all threads. >> >> - Jim >> >> >> >> >> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < >> james.laskey at oracle.com> wrote: >> >>> >>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: >>> >>> Hey Jim, >>> >>> I looked at it and I will look into loadWithNewGlobal to see what >>> exactly it does since it could be relevant. As for the rest, for my use >>> case having threads in the JS would not help. We're using Nashorn to build >>> JSON filters in a Dynamic Proxy Service and need any of the threads >>> processing a request to be able to execute the script to filter. >>> >>> >>> The cost of creating a new engine is significant. So share an engine >>> across threads but use *eval >>> * >>> (String >>> >>> script, ScriptContext >>> >>> context) instead, separate context per execution. If your JavaScript >>> code does not modify globals you can get away with using the same engine, >>> same compiled script on each thread. >>> >>> >>> Also, when you say a new engine per threads is the worst case what >>> exactly do you mean? I would expect an initial cost of compiling the script >>> on each thread and then each engine should be able to do its own thing, but >>> what I'm seeing is that when running with more than 10 threads all my >>> engines get slow at executing code, even though they are all completely >>> separate from each other. >>> >>> >>> Of course there are many factors involved n performance. How many cores >>> do you have on the test machine? How much memory in the process? What >>> transport are you using between threads? That sort of thing. Other than >>> constructing then engine and context Nashorn performance should scale. >>> >>> >>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < >>> james.laskey at oracle.com> wrote: >>> >>>> Jesus, >>>> >>>> Probably the most informative information is in this blog. >>>> >>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >>>> >>>> This example uses Executors but threads would work as well. >>>> >>>> I did a talk that looked at different methods to max out multithreading >>>> performance. A new engine per thread is the worst case. A new context per >>>> thread does much better. A new global per thread is the best while >>>> remaining thread safe. >>>> >>>> Cheers, >>>> >>>> ? Jim >>>> >>>> >>>> >>>> >>>> >>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: >>>> >>>> Hey folks, >>>> >>>> I've tried many different ways of using Nashorn multithreaded based on >>>> what >>>> I've found on the internet and I still can't get a single one to scale. >>>> Even the most naive method of making many script engines with my script >>>> tends to bottleneck itself when I have more than 10 threads invoking >>>> functions. >>>> >>>> I'm using the following code to compile my script and >>>> invocable.invokeFunction("transform", input) to execute: >>>> >>>> static Invocable generateInvocable(String script) throws >>>> ScriptException { >>>> ScriptEngineManager manager = new ScriptEngineManager(); >>>> ScriptEngine engine = >>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>>> Compilable compilable = (Compilable) engine; >>>> final CompiledScript compiled = compilable.compile(script); >>>> compiled.eval(); >>>> return (Invocable) engine; >>>> } >>>> >>>> >>>> >>>> The script I'm compiling is: >>>> >>>> String script = "function transform(input) {" + >>>> "var result = JSON.parse(input);" + >>>> "response = {};\n" + >>>> "for (var i = 0; i < result.length; i++) {\n" + >>>> " var summoner = {};\n" + >>>> " summoner.id = result[i].id;\n" + >>>> " summoner.name = result[i].name;\n" + >>>> " summoner.profileIconId = >>>> result[i].profileIconId;\n" + >>>> " summoner.revisionDate = result[i].revisionDate;\n" + >>>> " summoner.summonerLevel = result[i].level;\n" + >>>> " response[summoner.id] = summoner;\n" + >>>> "}\n" + >>>> "result = response;" + >>>> "return JSON.stringify(result);" + >>>> "};"; >>>> >>>> >>>> >>>> I've also tried other more scaleable ways to work with scripts >>>> concurrently, but given that this is the most naive method where >>>> everything >>>> is brand new and I still get slowness calling them concurrently I fear >>>> that >>>> maybe I'm overlooking something extremely basic on my code. >>>> >>>> Thanks. >>>> -Jesus Luzon >>>> >>>> >>> >> >> > > From hannes.wallnoefer at oracle.com Wed Dec 7 15:01:27 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Wed, 7 Dec 2016 16:01:27 +0100 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Hi Jesus, I?m trying to reproduce the problem, and just want to make sure I get the missing pieces right. You already showed us how you?re setting up the engine and the JS code you?re running. I assume the JSON code you?re parsing is a simple array of objects? And you?re just calling Invocable.invokeFunction on the ScriptEngine from multiple threads in parallel, right? Thanks, Hannes > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > When we share one invocable across many threads and run invokeFunction it > happens, such as this: > > ExecutorService executor = Executors.newFixedThreadPool(50); >> >> Invocable invocable = generateInvocable(script); >> >> AtomicLong count = new AtomicLong(); >> >> for (int i = 0; i < 50; i++) { >> >> executor.submit(new Runnable() { >> >> @Override >> >> public void run() { >> >> try { >> >> while(true) { >> >> invocable.invokeFunction("transform", >>> something); >> >> count.incrementAndGet(); >> >> } >> >> } catch (NoSuchMethodException | ScriptException >>> e) { >> >> e.printStackTrace(); >> >> } >> >> } >> >> }); >> >> } >> >> > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) > wrote: > >> Intersting. The example you posted demonstrates this behaviour? If so >> I?ll file a bug and dig in. It sounds like an object is being reused >> across invocations and accumulating changes to the property map. >> >> ? Jim >> >> >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: >> >> With more threads you are impacting the same 8 cores, so it will taper off >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a >>> threshold depending on System performance. Transport: I meant if you were >>> using sockets to provide the script. >> >> This makes sense. This one's on me then. >> >> >>> So you are using the same invocable instance for all threads? If so, >>> then you are probably good to go. As far as leaks are concerned, not sure >>> how you would get leaks from Nashorn. The JSON object is written in Java, >>> and little JavaScript involved. >> >> >> >>> In your example, pull up Invocable invocable = generateInvocable(script); >>> out of the loop and use the same invocable for all threads. >> >> >> We were using one invocable across all threads and we were getting >> slowdowns on execution, high CPU Usage and memory leaks that led to >> OutOfMemory errors. I could trace the leak to >> >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter >> >> which just keeps growing forever. >> >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < >> james.laskey at oracle.com> wrote: >> >>> >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: >>> >>> The cost of creating a new engine is significant. So share an engine >>>> across threads but use *eval >>>> * >>>> (String >>>> >>>> script, ScriptContext >>>> >>>> context) instead, separate context per execution. If your JavaScript >>>> code does not modify globals you can get away with using the same engine, >>>> same compiled script on each thread. >>> >>> >>> I guess there's a few things here I don't understand. One thing I'm >>> trying to do is sharing a CompiledScript (which is why I'm using >>> invocable). Also, what exactly does modify globals mean? All our filters do >>> the same thing, make a function that takes a JSON String, turns it into a >>> JSON, modifies it and then stringifies it back. No state is changed of >>> anything else but there are temporary vars created inside the scope of the >>> function. When we run this multithreaded, running invokeFunction slows down >>> significantly and we get crazy memory leaks. >>> >>> >>> So you are using the same invocable instance for all threads? If so, >>> then you are probably good to go. As far as leaks are concerned, not sure >>> how you would get leaks from Nashorn. The JSON object is written in Java, >>> and little JavaScript involved. >>> >>> >>> Of course there are many factors involved n performance. How many cores >>>> do you have on the test machine? How much memory in the process? What >>>> transport are you using between threads? That sort of thing. Other than >>>> constructing then engine and context Nashorn performance should scale. >>> >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the >>> process. Not sure what transports between threads means, but this is the >>> code I'm benchmarking with. Increasing the number of threads actually makes >>> it go faster until about 4 threads, then adding more threads takes the same >>> amount to get to 1000 and and after a certain point it is just slower to >>> get to 1000 counts. Some of our filters need to be able to run over 1000 >>> times a second (across all threads) and the fastest time I could actually >>> get with this was about 2.4 seconds for a 1000 counts. >>> >>>> ExecutorService executor = Executors.newFixedThreadPool(50); >>>> >>>> AtomicLong count = new AtomicLong(); >>>> >>>> for (int i = 0; i < 50; i++) { >>>> >>>> executor.submit(new Runnable() { >>>> >>>> @Override >>>> >>>> public void run() { >>>> >>>> >>>>> try { >>>> >>>> Invocable invocable = >>>>> generateInvocable(script); >>>> >>>> while(true) { >>>> >>>> invocable.invokeFunction("transform", >>>>> something); >>>> >>>> count.incrementAndGet(); >>>> >>>> } >>>> >>>> } catch (NoSuchMethodException | ScriptException >>>>> e) { >>>> >>>> e.printStackTrace(); >>>> >>>> } >>>> >>>> } >>>> >>>> }); >>>> >>>> } >>>> >>>> long lastTimestamp = System.currentTimeMillis(); >>>> >>>> while(true) { >>>> >>>> >>>>> if (count.get() > 1000) { >>>> >>>> count.getAndAdd(-1000); >>>> >>>> System.out.println((System.currentTimeMillis() - >>>>> lastTimestamp)/1000.0); >>>> >>>> lastTimestamp = System.currentTimeMillis(); >>>> >>>> } >>>> >>>> } >>>> >>>> >>> With more threads you are impacting the same 8 cores, so it will taper >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a >>> threshold depending on System performance. Transport: I meant if you were >>> using sockets to provide the script. >>> >>> In your example, pull up Invocable invocable = generateInvocable(script); >>> out of the loop and use the same invocable for all threads. >>> >>> - Jim >>> >>> >>> >>> >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < >>> james.laskey at oracle.com> wrote: >>> >>>> >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: >>>> >>>> Hey Jim, >>>> >>>> I looked at it and I will look into loadWithNewGlobal to see what >>>> exactly it does since it could be relevant. As for the rest, for my use >>>> case having threads in the JS would not help. We're using Nashorn to build >>>> JSON filters in a Dynamic Proxy Service and need any of the threads >>>> processing a request to be able to execute the script to filter. >>>> >>>> >>>> The cost of creating a new engine is significant. So share an engine >>>> across threads but use *eval >>>> * >>>> (String >>>> >>>> script, ScriptContext >>>> >>>> context) instead, separate context per execution. If your JavaScript >>>> code does not modify globals you can get away with using the same engine, >>>> same compiled script on each thread. >>>> >>>> >>>> Also, when you say a new engine per threads is the worst case what >>>> exactly do you mean? I would expect an initial cost of compiling the script >>>> on each thread and then each engine should be able to do its own thing, but >>>> what I'm seeing is that when running with more than 10 threads all my >>>> engines get slow at executing code, even though they are all completely >>>> separate from each other. >>>> >>>> >>>> Of course there are many factors involved n performance. How many cores >>>> do you have on the test machine? How much memory in the process? What >>>> transport are you using between threads? That sort of thing. Other than >>>> constructing then engine and context Nashorn performance should scale. >>>> >>>> >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < >>>> james.laskey at oracle.com> wrote: >>>> >>>>> Jesus, >>>>> >>>>> Probably the most informative information is in this blog. >>>>> >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt >>>>> >>>>> This example uses Executors but threads would work as well. >>>>> >>>>> I did a talk that looked at different methods to max out multithreading >>>>> performance. A new engine per thread is the worst case. A new context per >>>>> thread does much better. A new global per thread is the best while >>>>> remaining thread safe. >>>>> >>>>> Cheers, >>>>> >>>>> ? Jim >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: >>>>> >>>>> Hey folks, >>>>> >>>>> I've tried many different ways of using Nashorn multithreaded based on >>>>> what >>>>> I've found on the internet and I still can't get a single one to scale. >>>>> Even the most naive method of making many script engines with my script >>>>> tends to bottleneck itself when I have more than 10 threads invoking >>>>> functions. >>>>> >>>>> I'm using the following code to compile my script and >>>>> invocable.invokeFunction("transform", input) to execute: >>>>> >>>>> static Invocable generateInvocable(String script) throws >>>>> ScriptException { >>>>> ScriptEngineManager manager = new ScriptEngineManager(); >>>>> ScriptEngine engine = >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >>>>> Compilable compilable = (Compilable) engine; >>>>> final CompiledScript compiled = compilable.compile(script); >>>>> compiled.eval(); >>>>> return (Invocable) engine; >>>>> } >>>>> >>>>> >>>>> >>>>> The script I'm compiling is: >>>>> >>>>> String script = "function transform(input) {" + >>>>> "var result = JSON.parse(input);" + >>>>> "response = {};\n" + >>>>> "for (var i = 0; i < result.length; i++) {\n" + >>>>> " var summoner = {};\n" + >>>>> " summoner.id = result[i].id;\n" + >>>>> " summoner.name = result[i].name;\n" + >>>>> " summoner.profileIconId = >>>>> result[i].profileIconId;\n" + >>>>> " summoner.revisionDate = result[i].revisionDate;\n" + >>>>> " summoner.summonerLevel = result[i].level;\n" + >>>>> " response[summoner.id] = summoner;\n" + >>>>> "}\n" + >>>>> "result = response;" + >>>>> "return JSON.stringify(result);" + >>>>> "};"; >>>>> >>>>> >>>>> >>>>> I've also tried other more scaleable ways to work with scripts >>>>> concurrently, but given that this is the most naive method where >>>>> everything >>>>> is brand new and I still get slowness calling them concurrently I fear >>>>> that >>>>> maybe I'm overlooking something extremely basic on my code. >>>>> >>>>> Thanks. >>>>> -Jesus Luzon >>>>> >>>>> >>>> >>> >>> >> >> From jluzon at riotgames.com Wed Dec 7 19:23:10 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Wed, 7 Dec 2016 11:23:10 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Yes, it's an array of objects which I'll paste. And yes, I'm just calling invokeFunction from many many different threads. I'm also going to go back and take a look at all the heap dumps we have to re-confirm what I mentioned. "[\n" + > " {\n" + > " \"id\": 6011511,\n" + > " \"accountId\": 203192481,\n" + > " \"name\": \"Adam Pro Qarda?\",\n" + > " \"profileIconId\": 25,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 1475,\n" + > " \"revisionDate\": 1406631727000\n" + > " },\n" + > " {\n" + > " \"id\": 2810674,\n" + > " \"accountId\": 200706913,\n" + > " \"name\": \"ABZ Devrim\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 982,\n" + > " \"infPoints\": 10472,\n" + > " \"revisionDate\": 1450791227000\n" + > " },\n" + > " {\n" + > " \"id\": 5411195,\n" + > " \"accountId\": 202647689,\n" + > " \"name\": \"Ace HypcronN\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 73,\n" + > " \"infPoints\": 182445,\n" + > " \"revisionDate\": 1480781650000\n" + > " },\n" + > " {\n" + > " \"id\": 1363020,\n" + > " \"accountId\": 1357837,\n" + > " \"name\": \"AdanaLee\",\n" + > " \"profileIconId\": 502,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 125,\n" + > " \"infPoints\": 719299,\n" + > " \"revisionDate\": 1480530778000\n" + > " },\n" + > " {\n" + > " \"id\": 8261198,\n" + > " \"accountId\": 205027096,\n" + > " \"name\": \"Achilehuz\",\n" + > " \"profileIconId\": 1381,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 158603,\n" + > " \"revisionDate\": 1480770307000\n" + > " },\n" + > " {\n" + > " \"id\": 12685857,\n" + > " \"accountId\": 207591166,\n" + > " \"name\": \"ac?mas?zpicc\",\n" + > " \"profileIconId\": 9,\n" + > " \"level\": 21,\n" + > " \"expPoints\": 840,\n" + > " \"infPoints\": 16659,\n" + > " \"revisionDate\": 1480515325000\n" + > " },\n" + > " {\n" + > " \"id\": 10860127,\n" + > " \"accountId\": 206507727,\n" + > " \"name\": \"AAngelFlyy\",\n" + > " \"profileIconId\": 1395,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 73111,\n" + > " \"revisionDate\": 1480787870000\n" + > " },\n" + > " {\n" + > " \"id\": 3292376,\n" + > " \"accountId\": 201048714,\n" + > " \"name\": \"ACAB1907\",\n" + > " \"profileIconId\": 20,\n" + > " \"level\": 6,\n" + > " \"expPoints\": 305,\n" + > " \"infPoints\": 2107,\n" + > " \"revisionDate\": 1402448089000\n" + > " },\n" + > " {\n" + > " \"id\": 461671,\n" + > " \"accountId\": 446571,\n" + > " \"name\": \"Acta Est Fabul?\",\n" + > " \"profileIconId\": 1435,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 47,\n" + > " \"infPoints\": 644672,\n" + > " \"revisionDate\": 1480626505000\n" + > " },\n" + > " {\n" + > " \"id\": 394183,\n" + > " \"accountId\": 379083,\n" + > " \"name\": \"acekse4\",\n" + > " \"profileIconId\": 27,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 223,\n" + > " \"infPoints\": 908,\n" + > " \"revisionDate\": 1348116544000\n" + > " },\n" + > " {\n" + > " \"id\": 5941247,\n" + > " \"accountId\": 203106300,\n" + > " \"name\": \"abdul7878\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 3,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 401,\n" + > " \"revisionDate\": 1406029148000\n" + > " },\n" + > " {\n" + > " \"id\": 2467446,\n" + > " \"accountId\": 200459837,\n" + > " \"name\": \"ActionC\",\n" + > " \"profileIconId\": 986,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 74,\n" + > " \"infPoints\": 401367,\n" + > " \"revisionDate\": 1480808608000\n" + > " },\n" + > " {\n" + > " \"id\": 9402979,\n" + > " \"accountId\": 205698832,\n" + > " \"name\": \"Ablenia \",\n" + > " \"profileIconId\": 1129,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 19,\n" + > " \"infPoints\": 163518,\n" + > " \"revisionDate\": 1480687603000\n" + > " },\n" + > " {\n" + > " \"id\": 13187505,\n" + > " \"accountId\": 207898213,\n" + > " \"name\": \"aDaMiYiM\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 116,\n" + > " \"infPoints\": 45214,\n" + > " \"revisionDate\": 1480793258000\n" + > " },\n" + > " {\n" + > " \"id\": 4141059,\n" + > " \"accountId\": 201688290,\n" + > " \"name\": \"Abimin?ar?\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 152,\n" + > " \"infPoints\": 752477,\n" + > " \"revisionDate\": 1480635961000\n" + > " },\n" + > " {\n" + > " \"id\": 5702134,\n" + > " \"accountId\": 202899395,\n" + > " \"name\": \"Above the Clouds\",\n" + > " \"profileIconId\": 684,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 110,\n" + > " \"infPoints\": 288096,\n" + > " \"revisionDate\": 1471011372000\n" + > " },\n" + > " {\n" + > " \"id\": 5810740,\n" + > " \"accountId\": 202985228,\n" + > " \"name\": \"aBimm\",\n" + > " \"profileIconId\": 11,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 1180,\n" + > " \"infPoints\": 10736,\n" + > " \"revisionDate\": 1409832684000\n" + > " },\n" + > " {\n" + > " \"id\": 5817751,\n" + > " \"accountId\": 203050678,\n" + > " \"name\": \"AD Glor?am\",\n" + > " \"profileIconId\": 982,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 111,\n" + > " \"infPoints\": 304658,\n" + > " \"revisionDate\": 1480795250000\n" + > " },\n" + > " {\n" + > " \"id\": 9851802,\n" + > " \"accountId\": 206011054,\n" + > " \"name\": \"AdarAllame\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 73763,\n" + > " \"revisionDate\": 1479422812000\n" + > " },\n" + > " {\n" + > " \"id\": 12735622,\n" + > " \"accountId\": 207587019,\n" + > " \"name\": \"absinthe666\",\n" + > " \"profileIconId\": 903,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 40302,\n" + > " \"revisionDate\": 1480782923000\n" + > " },\n" + > " {\n" + > " \"id\": 6371389,\n" + > " \"accountId\": 203416952,\n" + > " \"name\": \"adamsat?c?\",\n" + > " \"profileIconId\": 3,\n" + > " \"level\": 4,\n" + > " \"expPoints\": 17,\n" + > " \"infPoints\": 685,\n" + > " \"revisionDate\": 1409320171000\n" + > " },\n" + > " {\n" + > " \"id\": 7828139,\n" + > " \"accountId\": 204927980,\n" + > " \"name\": \"AbsoluteForce\",\n" + > " \"profileIconId\": 950,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 208789,\n" + > " \"revisionDate\": 1480804396000\n" + > " },\n" + > " {\n" + > " \"id\": 1373229,\n" + > " \"accountId\": 1358441,\n" + > " \"name\": \"AbsoluteDeath\",\n" + > " \"profileIconId\": 7,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 34,\n" + > " \"infPoints\": 223655,\n" + > " \"revisionDate\": 1471867646000\n" + > " },\n" + > " {\n" + > " \"id\": 7694972,\n" + > " \"accountId\": 204803668,\n" + > " \"name\": \"ac pnp\",\n" + > " \"profileIconId\": 937,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 161,\n" + > " \"infPoints\": 249681,\n" + > " \"revisionDate\": 1480801507000\n" + > " },\n" + > " {\n" + > " \"id\": 1373524,\n" + > " \"accountId\": 1350474,\n" + > " \"name\": \"Abd??\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 103,\n" + > " \"infPoints\": 286803,\n" + > " \"revisionDate\": 1476621827000\n" + > " },\n" + > " {\n" + > " \"id\": 1650227,\n" + > " \"accountId\": 200000503,\n" + > " \"name\": \"AD Ambrosia\",\n" + > " \"profileIconId\": 1152,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 156333,\n" + > " \"revisionDate\": 1480805320000\n" + > " },\n" + > " {\n" + > " \"id\": 8331358,\n" + > " \"accountId\": 205073925,\n" + > " \"name\": \"acarmanyust2\",\n" + > " \"profileIconId\": 0,\n" + > " \"level\": 2,\n" + > " \"expPoints\": 43,\n" + > " \"infPoints\": 318,\n" + > " \"revisionDate\": 1423915139000\n" + > " },\n" + > " {\n" + > " \"id\": 1862106,\n" + > " \"accountId\": 200139838,\n" + > " \"name\": \"aboU\",\n" + > " \"profileIconId\": 1155,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 0,\n" + > " \"infPoints\": 412616,\n" + > " \"revisionDate\": 1480771055000\n" + > " },\n" + > " {\n" + > " \"id\": 2362628,\n" + > " \"accountId\": 685649,\n" + > " \"name\": \"Ac?F?sT?k\",\n" + > " \"profileIconId\": 1074,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 233882,\n" + > " \"revisionDate\": 1480786233000\n" + > " },\n" + > " {\n" + > " \"id\": 4323909,\n" + > " \"accountId\": 201917672,\n" + > " \"name\": \"Addrenalin\",\n" + > " \"profileIconId\": 603,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 55,\n" + > " \"infPoints\": 220605,\n" + > " \"revisionDate\": 1432647338000\n" + > " },\n" + > " {\n" + > " \"id\": 377206,\n" + > " \"accountId\": 362106,\n" + > " \"name\": \"Aburame Shino\",\n" + > " \"profileIconId\": 844,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 84,\n" + > " \"infPoints\": 354087,\n" + > " \"revisionDate\": 1477666556000\n" + > " },\n" + > " {\n" + > " \"id\": 5377433,\n" + > " \"accountId\": 202697921,\n" + > " \"name\": \"AcEcolton35\",\n" + > " \"profileIconId\": 984,\n" + > " \"level\": 25,\n" + > " \"expPoints\": 751,\n" + > " \"infPoints\": 30061,\n" + > " \"revisionDate\": 1475503024000\n" + > " },\n" + > " {\n" + > " \"id\": 2381404,\n" + > " \"accountId\": 200333680,\n" + > " \"name\": \"adafakaaa\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 8,\n" + > " \"infPoints\": 534204,\n" + > " \"revisionDate\": 1480719827000\n" + > " },\n" + > " {\n" + > " \"id\": 1281203,\n" + > " \"accountId\": 1259342,\n" + > " \"name\": \"AC Klondike\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 27,\n" + > " \"infPoints\": 191429,\n" + > " \"revisionDate\": 1480294973000\n" + > " },\n" + > " {\n" + > " \"id\": 13161471,\n" + > " \"accountId\": 207847181,\n" + > " \"name\": \"adar21\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 10,\n" + > " \"expPoints\": 143,\n" + > " \"infPoints\": 3558,\n" + > " \"revisionDate\": 1476529855000\n" + > " },\n" + > " {\n" + > " \"id\": 5841915,\n" + > " \"accountId\": 202998794,\n" + > " \"name\": \"Achilles29\",\n" + > " \"profileIconId\": 666,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 41,\n" + > " \"infPoints\": 219714,\n" + > " \"revisionDate\": 1480777744000\n" + > " },\n" + > " {\n" + > " \"id\": 2853062,\n" + > " \"accountId\": 200726707,\n" + > " \"name\": \"AbIanStarBebegim\",\n" + > " \"profileIconId\": 909,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 64,\n" + > " \"infPoints\": 297580,\n" + > " \"revisionDate\": 1480556859000\n" + > " },\n" + > " {\n" + > " \"id\": 8323114,\n" + > " \"accountId\": 205093515,\n" + > " \"name\": \"Absuruk\",\n" + > " \"profileIconId\": 6,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 21,\n" + > " \"infPoints\": 121086,\n" + > " \"revisionDate\": 1480820801000\n" + > " }\n" + > "]" On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer < hannes.wallnoefer at oracle.com> wrote: > Hi Jesus, > > I?m trying to reproduce the problem, and just want to make sure I get the > missing pieces right. > > You already showed us how you?re setting up the engine and the JS code > you?re running. I assume the JSON code you?re parsing is a simple array of > objects? And you?re just calling Invocable.invokeFunction on the > ScriptEngine from multiple threads in parallel, right? > > Thanks, > Hannes > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > > > When we share one invocable across many threads and run invokeFunction it > > happens, such as this: > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > >> > >> Invocable invocable = generateInvocable(script); > >> > >> AtomicLong count = new AtomicLong(); > >> > >> for (int i = 0; i < 50; i++) { > >> > >> executor.submit(new Runnable() { > >> > >> @Override > >> > >> public void run() { > >> > >> try { > >> > >> while(true) { > >> > >> invocable.invokeFunction("transform", > >>> something); > >> > >> count.incrementAndGet(); > >> > >> } > >> > >> } catch (NoSuchMethodException | ScriptException > >>> e) { > >> > >> e.printStackTrace(); > >> > >> } > >> > >> } > >> > >> }); > >> > >> } > >> > >> > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) < > james.laskey at oracle.com > >> wrote: > > > >> Intersting. The example you posted demonstrates this behaviour? If so > >> I?ll file a bug and dig in. It sounds like an object is being reused > >> across invocations and accumulating changes to the property map. > >> > >> ? Jim > >> > >> > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > >> > >> With more threads you are impacting the same 8 cores, so it will taper > off > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > >>> threshold depending on System performance. Transport: I meant if you > were > >>> using sockets to provide the script. > >> > >> This makes sense. This one's on me then. > >> > >> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not > sure > >>> how you would get leaks from Nashorn. The JSON object is written in > Java, > >>> and little JavaScript involved. > >> > >> > >> > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >> > >> > >> We were using one invocable across all threads and we were getting > >> slowdowns on execution, high CPU Usage and memory leaks that led to > >> OutOfMemory errors. I could trace the leak to > >> > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > >> > >> which just keeps growing forever. > >> > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > >> james.laskey at oracle.com> wrote: > >> > >>> > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: > >>> > >>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > >>>> (String > >>>> > >>>> script, ScriptContext > >>>> script/ScriptContext.html> > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same > engine, > >>>> same compiled script on each thread. > >>> > >>> > >>> I guess there's a few things here I don't understand. One thing I'm > >>> trying to do is sharing a CompiledScript (which is why I'm using > >>> invocable). Also, what exactly does modify globals mean? All our > filters do > >>> the same thing, make a function that takes a JSON String, turns it > into a > >>> JSON, modifies it and then stringifies it back. No state is changed of > >>> anything else but there are temporary vars created inside the scope of > the > >>> function. When we run this multithreaded, running invokeFunction slows > down > >>> significantly and we get crazy memory leaks. > >>> > >>> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not > sure > >>> how you would get leaks from Nashorn. The JSON object is written in > Java, > >>> and little JavaScript involved. > >>> > >>> > >>> Of course there are many factors involved n performance. How many > cores > >>>> do you have on the test machine? How much memory in the process? > What > >>>> transport are you using between threads? That sort of thing. Other > than > >>>> constructing then engine and context Nashorn performance should scale. > >>> > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the > >>> process. Not sure what transports between threads means, but this is > the > >>> code I'm benchmarking with. Increasing the number of threads actually > makes > >>> it go faster until about 4 threads, then adding more threads takes the > same > >>> amount to get to 1000 and and after a certain point it is just slower > to > >>> get to 1000 counts. Some of our filters need to be able to run over > 1000 > >>> times a second (across all threads) and the fastest time I could > actually > >>> get with this was about 2.4 seconds for a 1000 counts. > >>> > >>>> ExecutorService executor = Executors.newFixedThreadPool(50); > >>>> > >>>> AtomicLong count = new AtomicLong(); > >>>> > >>>> for (int i = 0; i < 50; i++) { > >>>> > >>>> executor.submit(new Runnable() { > >>>> > >>>> @Override > >>>> > >>>> public void run() { > >>>> > >>>> > >>>>> try { > >>>> > >>>> Invocable invocable = > >>>>> generateInvocable(script); > >>>> > >>>> while(true) { > >>>> > >>>> invocable.invokeFunction("transform", > >>>>> something); > >>>> > >>>> count.incrementAndGet(); > >>>> > >>>> } > >>>> > >>>> } catch (NoSuchMethodException | > ScriptException > >>>>> e) { > >>>> > >>>> e.printStackTrace(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> }); > >>>> > >>>> } > >>>> > >>>> long lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> while(true) { > >>>> > >>>> > >>>>> if (count.get() > 1000) { > >>>> > >>>> count.getAndAdd(-1000); > >>>> > >>>> System.out.println((System.currentTimeMillis() - > >>>>> lastTimestamp)/1000.0); > >>>> > >>>> lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> > >>> With more threads you are impacting the same 8 cores, so it will taper > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 > being a > >>> threshold depending on System performance. Transport: I meant if you > were > >>> using sockets to provide the script. > >>> > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >>> > >>> - Jim > >>> > >>> > >>> > >>> > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > >>> james.laskey at oracle.com> wrote: > >>> > >>>> > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: > >>>> > >>>> Hey Jim, > >>>> > >>>> I looked at it and I will look into loadWithNewGlobal to see what > >>>> exactly it does since it could be relevant. As for the rest, for my > use > >>>> case having threads in the JS would not help. We're using Nashorn to > build > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads > >>>> processing a request to be able to execute the script to filter. > >>>> > >>>> > >>>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > >>>> (String > >>>> > >>>> script, ScriptContext > >>>> script/ScriptContext.html> > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same > engine, > >>>> same compiled script on each thread. > >>>> > >>>> > >>>> Also, when you say a new engine per threads is the worst case what > >>>> exactly do you mean? I would expect an initial cost of compiling the > script > >>>> on each thread and then each engine should be able to do its own > thing, but > >>>> what I'm seeing is that when running with more than 10 threads all my > >>>> engines get slow at executing code, even though they are all > completely > >>>> separate from each other. > >>>> > >>>> > >>>> Of course there are many factors involved n performance. How many > cores > >>>> do you have on the test machine? How much memory in the process? > What > >>>> transport are you using between threads? That sort of thing. Other > than > >>>> constructing then engine and context Nashorn performance should scale. > >>>> > >>>> > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > >>>> james.laskey at oracle.com> wrote: > >>>> > >>>>> Jesus, > >>>>> > >>>>> Probably the most informative information is in this blog. > >>>>> > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_ > threading_and_mt > >>>>> > >>>>> This example uses Executors but threads would work as well. > >>>>> > >>>>> I did a talk that looked at different methods to max out > multithreading > >>>>> performance. A new engine per thread is the worst case. A new > context per > >>>>> thread does much better. A new global per thread is the best while > >>>>> remaining thread safe. > >>>>> > >>>>> Cheers, > >>>>> > >>>>> ? Jim > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: > >>>>> > >>>>> Hey folks, > >>>>> > >>>>> I've tried many different ways of using Nashorn multithreaded based > on > >>>>> what > >>>>> I've found on the internet and I still can't get a single one to > scale. > >>>>> Even the most naive method of making many script engines with my > script > >>>>> tends to bottleneck itself when I have more than 10 threads invoking > >>>>> functions. > >>>>> > >>>>> I'm using the following code to compile my script and > >>>>> invocable.invokeFunction("transform", input) to execute: > >>>>> > >>>>> static Invocable generateInvocable(String script) throws > >>>>> ScriptException { > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > >>>>> ScriptEngine engine = > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > >>>>> Compilable compilable = (Compilable) engine; > >>>>> final CompiledScript compiled = compilable.compile(script); > >>>>> compiled.eval(); > >>>>> return (Invocable) engine; > >>>>> } > >>>>> > >>>>> > >>>>> > >>>>> The script I'm compiling is: > >>>>> > >>>>> String script = "function transform(input) {" + > >>>>> "var result = JSON.parse(input);" + > >>>>> "response = {};\n" + > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > >>>>> " var summoner = {};\n" + > >>>>> " summoner.id = result[i].id;\n" + > >>>>> " summoner.name = result[i].name;\n" + > >>>>> " summoner.profileIconId = > >>>>> result[i].profileIconId;\n" + > >>>>> " summoner.revisionDate = > result[i].revisionDate;\n" + > >>>>> " summoner.summonerLevel = result[i].level;\n" + > >>>>> " response[summoner.id] = summoner;\n" + > >>>>> "}\n" + > >>>>> "result = response;" + > >>>>> "return JSON.stringify(result);" + > >>>>> "};"; > >>>>> > >>>>> > >>>>> > >>>>> I've also tried other more scaleable ways to work with scripts > >>>>> concurrently, but given that this is the most naive method where > >>>>> everything > >>>>> is brand new and I still get slowness calling them concurrently I > fear > >>>>> that > >>>>> maybe I'm overlooking something extremely basic on my code. > >>>>> > >>>>> Thanks. > >>>>> -Jesus Luzon > >>>>> > >>>>> > >>>> > >>> > >>> > >> > >> > > From jluzon at riotgames.com Wed Dec 7 23:40:34 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Wed, 7 Dec 2016 15:40:34 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Looks like the memory leak is due to the way we wrote our code and how javascript works. I was expecting the line *response[**summoner.id **] = summoner; *to build a map but it turns out that if you use a number as the "key", javscript automatically fills the indexes in the middle with null (Undefined type?). When these IDs are very large, it is creating huge arrays that take longer to garbage collect than the code executing. I am about to start testing this on our end to make sure we see the improvements we expect. Does this idea seem like it is reasonably possible? On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon wrote: > Yes, it's an array of objects which I'll paste. And yes, I'm just calling > invokeFunction from many many different threads. I'm also going to go back > and take a look at all the heap dumps we have to re-confirm what I > mentioned. > > "[\n" + >> " {\n" + >> " \"id\": 6011511,\n" + >> " \"accountId\": 203192481,\n" + >> " \"name\": \"Adam Pro Qarda?\",\n" + >> " \"profileIconId\": 25,\n" + >> " \"level\": 5,\n" + >> " \"expPoints\": 83,\n" + >> " \"infPoints\": 1475,\n" + >> " \"revisionDate\": 1406631727000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 2810674,\n" + >> " \"accountId\": 200706913,\n" + >> " \"name\": \"ABZ Devrim\",\n" + >> " \"profileIconId\": 663,\n" + >> " \"level\": 13,\n" + >> " \"expPoints\": 982,\n" + >> " \"infPoints\": 10472,\n" + >> " \"revisionDate\": 1450791227000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5411195,\n" + >> " \"accountId\": 202647689,\n" + >> " \"name\": \"Ace HypcronN\",\n" + >> " \"profileIconId\": 911,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 73,\n" + >> " \"infPoints\": 182445,\n" + >> " \"revisionDate\": 1480781650000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1363020,\n" + >> " \"accountId\": 1357837,\n" + >> " \"name\": \"AdanaLee\",\n" + >> " \"profileIconId\": 502,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 125,\n" + >> " \"infPoints\": 719299,\n" + >> " \"revisionDate\": 1480530778000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 8261198,\n" + >> " \"accountId\": 205027096,\n" + >> " \"name\": \"Achilehuz\",\n" + >> " \"profileIconId\": 1381,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 10,\n" + >> " \"infPoints\": 158603,\n" + >> " \"revisionDate\": 1480770307000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 12685857,\n" + >> " \"accountId\": 207591166,\n" + >> " \"name\": \"ac?mas?zpicc\",\n" + >> " \"profileIconId\": 9,\n" + >> " \"level\": 21,\n" + >> " \"expPoints\": 840,\n" + >> " \"infPoints\": 16659,\n" + >> " \"revisionDate\": 1480515325000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 10860127,\n" + >> " \"accountId\": 206507727,\n" + >> " \"name\": \"AAngelFlyy\",\n" + >> " \"profileIconId\": 1395,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 10,\n" + >> " \"infPoints\": 73111,\n" + >> " \"revisionDate\": 1480787870000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 3292376,\n" + >> " \"accountId\": 201048714,\n" + >> " \"name\": \"ACAB1907\",\n" + >> " \"profileIconId\": 20,\n" + >> " \"level\": 6,\n" + >> " \"expPoints\": 305,\n" + >> " \"infPoints\": 2107,\n" + >> " \"revisionDate\": 1402448089000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 461671,\n" + >> " \"accountId\": 446571,\n" + >> " \"name\": \"Acta Est Fabul?\",\n" + >> " \"profileIconId\": 1435,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 47,\n" + >> " \"infPoints\": 644672,\n" + >> " \"revisionDate\": 1480626505000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 394183,\n" + >> " \"accountId\": 379083,\n" + >> " \"name\": \"acekse4\",\n" + >> " \"profileIconId\": 27,\n" + >> " \"level\": 5,\n" + >> " \"expPoints\": 223,\n" + >> " \"infPoints\": 908,\n" + >> " \"revisionDate\": 1348116544000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5941247,\n" + >> " \"accountId\": 203106300,\n" + >> " \"name\": \"abdul7878\",\n" + >> " \"profileIconId\": 26,\n" + >> " \"level\": 3,\n" + >> " \"expPoints\": 10,\n" + >> " \"infPoints\": 401,\n" + >> " \"revisionDate\": 1406029148000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 2467446,\n" + >> " \"accountId\": 200459837,\n" + >> " \"name\": \"ActionC\",\n" + >> " \"profileIconId\": 986,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 74,\n" + >> " \"infPoints\": 401367,\n" + >> " \"revisionDate\": 1480808608000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 9402979,\n" + >> " \"accountId\": 205698832,\n" + >> " \"name\": \"Ablenia \",\n" + >> " \"profileIconId\": 1129,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 19,\n" + >> " \"infPoints\": 163518,\n" + >> " \"revisionDate\": 1480687603000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 13187505,\n" + >> " \"accountId\": 207898213,\n" + >> " \"name\": \"aDaMiYiM\",\n" + >> " \"profileIconId\": 1301,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 116,\n" + >> " \"infPoints\": 45214,\n" + >> " \"revisionDate\": 1480793258000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 4141059,\n" + >> " \"accountId\": 201688290,\n" + >> " \"name\": \"Abimin?ar?\",\n" + >> " \"profileIconId\": 898,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 152,\n" + >> " \"infPoints\": 752477,\n" + >> " \"revisionDate\": 1480635961000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5702134,\n" + >> " \"accountId\": 202899395,\n" + >> " \"name\": \"Above the Clouds\",\n" + >> " \"profileIconId\": 684,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 110,\n" + >> " \"infPoints\": 288096,\n" + >> " \"revisionDate\": 1471011372000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5810740,\n" + >> " \"accountId\": 202985228,\n" + >> " \"name\": \"aBimm\",\n" + >> " \"profileIconId\": 11,\n" + >> " \"level\": 13,\n" + >> " \"expPoints\": 1180,\n" + >> " \"infPoints\": 10736,\n" + >> " \"revisionDate\": 1409832684000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5817751,\n" + >> " \"accountId\": 203050678,\n" + >> " \"name\": \"AD Glor?am\",\n" + >> " \"profileIconId\": 982,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 111,\n" + >> " \"infPoints\": 304658,\n" + >> " \"revisionDate\": 1480795250000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 9851802,\n" + >> " \"accountId\": 206011054,\n" + >> " \"name\": \"AdarAllame\",\n" + >> " \"profileIconId\": 911,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 48,\n" + >> " \"infPoints\": 73763,\n" + >> " \"revisionDate\": 1479422812000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 12735622,\n" + >> " \"accountId\": 207587019,\n" + >> " \"name\": \"absinthe666\",\n" + >> " \"profileIconId\": 903,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 83,\n" + >> " \"infPoints\": 40302,\n" + >> " \"revisionDate\": 1480782923000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 6371389,\n" + >> " \"accountId\": 203416952,\n" + >> " \"name\": \"adamsat?c?\",\n" + >> " \"profileIconId\": 3,\n" + >> " \"level\": 4,\n" + >> " \"expPoints\": 17,\n" + >> " \"infPoints\": 685,\n" + >> " \"revisionDate\": 1409320171000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 7828139,\n" + >> " \"accountId\": 204927980,\n" + >> " \"name\": \"AbsoluteForce\",\n" + >> " \"profileIconId\": 950,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 139,\n" + >> " \"infPoints\": 208789,\n" + >> " \"revisionDate\": 1480804396000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1373229,\n" + >> " \"accountId\": 1358441,\n" + >> " \"name\": \"AbsoluteDeath\",\n" + >> " \"profileIconId\": 7,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 34,\n" + >> " \"infPoints\": 223655,\n" + >> " \"revisionDate\": 1471867646000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 7694972,\n" + >> " \"accountId\": 204803668,\n" + >> " \"name\": \"ac pnp\",\n" + >> " \"profileIconId\": 937,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 161,\n" + >> " \"infPoints\": 249681,\n" + >> " \"revisionDate\": 1480801507000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1373524,\n" + >> " \"accountId\": 1350474,\n" + >> " \"name\": \"Abd??\",\n" + >> " \"profileIconId\": 1301,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 103,\n" + >> " \"infPoints\": 286803,\n" + >> " \"revisionDate\": 1476621827000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1650227,\n" + >> " \"accountId\": 200000503,\n" + >> " \"name\": \"AD Ambrosia\",\n" + >> " \"profileIconId\": 1152,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 139,\n" + >> " \"infPoints\": 156333,\n" + >> " \"revisionDate\": 1480805320000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 8331358,\n" + >> " \"accountId\": 205073925,\n" + >> " \"name\": \"acarmanyust2\",\n" + >> " \"profileIconId\": 0,\n" + >> " \"level\": 2,\n" + >> " \"expPoints\": 43,\n" + >> " \"infPoints\": 318,\n" + >> " \"revisionDate\": 1423915139000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1862106,\n" + >> " \"accountId\": 200139838,\n" + >> " \"name\": \"aboU\",\n" + >> " \"profileIconId\": 1155,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 0,\n" + >> " \"infPoints\": 412616,\n" + >> " \"revisionDate\": 1480771055000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 2362628,\n" + >> " \"accountId\": 685649,\n" + >> " \"name\": \"Ac?F?sT?k\",\n" + >> " \"profileIconId\": 1074,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 48,\n" + >> " \"infPoints\": 233882,\n" + >> " \"revisionDate\": 1480786233000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 4323909,\n" + >> " \"accountId\": 201917672,\n" + >> " \"name\": \"Addrenalin\",\n" + >> " \"profileIconId\": 603,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 55,\n" + >> " \"infPoints\": 220605,\n" + >> " \"revisionDate\": 1432647338000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 377206,\n" + >> " \"accountId\": 362106,\n" + >> " \"name\": \"Aburame Shino\",\n" + >> " \"profileIconId\": 844,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 84,\n" + >> " \"infPoints\": 354087,\n" + >> " \"revisionDate\": 1477666556000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5377433,\n" + >> " \"accountId\": 202697921,\n" + >> " \"name\": \"AcEcolton35\",\n" + >> " \"profileIconId\": 984,\n" + >> " \"level\": 25,\n" + >> " \"expPoints\": 751,\n" + >> " \"infPoints\": 30061,\n" + >> " \"revisionDate\": 1475503024000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 2381404,\n" + >> " \"accountId\": 200333680,\n" + >> " \"name\": \"adafakaaa\",\n" + >> " \"profileIconId\": 663,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 8,\n" + >> " \"infPoints\": 534204,\n" + >> " \"revisionDate\": 1480719827000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 1281203,\n" + >> " \"accountId\": 1259342,\n" + >> " \"name\": \"AC Klondike\",\n" + >> " \"profileIconId\": 898,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 27,\n" + >> " \"infPoints\": 191429,\n" + >> " \"revisionDate\": 1480294973000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 13161471,\n" + >> " \"accountId\": 207847181,\n" + >> " \"name\": \"adar21\",\n" + >> " \"profileIconId\": 26,\n" + >> " \"level\": 10,\n" + >> " \"expPoints\": 143,\n" + >> " \"infPoints\": 3558,\n" + >> " \"revisionDate\": 1476529855000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 5841915,\n" + >> " \"accountId\": 202998794,\n" + >> " \"name\": \"Achilles29\",\n" + >> " \"profileIconId\": 666,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 41,\n" + >> " \"infPoints\": 219714,\n" + >> " \"revisionDate\": 1480777744000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 2853062,\n" + >> " \"accountId\": 200726707,\n" + >> " \"name\": \"AbIanStarBebegim\",\n" + >> " \"profileIconId\": 909,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 64,\n" + >> " \"infPoints\": 297580,\n" + >> " \"revisionDate\": 1480556859000\n" + >> " },\n" + >> " {\n" + >> " \"id\": 8323114,\n" + >> " \"accountId\": 205093515,\n" + >> " \"name\": \"Absuruk\",\n" + >> " \"profileIconId\": 6,\n" + >> " \"level\": 30,\n" + >> " \"expPoints\": 21,\n" + >> " \"infPoints\": 121086,\n" + >> " \"revisionDate\": 1480820801000\n" + >> " }\n" + >> "]" > > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer < > hannes.wallnoefer at oracle.com> wrote: > >> Hi Jesus, >> >> I?m trying to reproduce the problem, and just want to make sure I get the >> missing pieces right. >> >> You already showed us how you?re setting up the engine and the JS code >> you?re running. I assume the JSON code you?re parsing is a simple array of >> objects? And you?re just calling Invocable.invokeFunction on the >> ScriptEngine from multiple threads in parallel, right? >> >> Thanks, >> Hannes >> >> >> > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : >> > >> > When we share one invocable across many threads and run invokeFunction >> it >> > happens, such as this: >> > >> > ExecutorService executor = Executors.newFixedThreadPool(50); >> >> >> >> Invocable invocable = generateInvocable(script); >> >> >> >> AtomicLong count = new AtomicLong(); >> >> >> >> for (int i = 0; i < 50; i++) { >> >> >> >> executor.submit(new Runnable() { >> >> >> >> @Override >> >> >> >> public void run() { >> >> >> >> try { >> >> >> >> while(true) { >> >> >> >> invocable.invokeFunction("transform", >> >>> something); >> >> >> >> count.incrementAndGet(); >> >> >> >> } >> >> >> >> } catch (NoSuchMethodException | ScriptException >> >>> e) { >> >> >> >> e.printStackTrace(); >> >> >> >> } >> >> >> >> } >> >> >> >> }); >> >> >> >> } >> >> >> >> >> > >> > >> > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) < >> james.laskey at oracle.com >> >> wrote: >> > >> >> Intersting. The example you posted demonstrates this behaviour? If so >> >> I?ll file a bug and dig in. It sounds like an object is being reused >> >> across invocations and accumulating changes to the property map. >> >> >> >> ? Jim >> >> >> >> >> >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: >> >> >> >> With more threads you are impacting the same 8 cores, so it will taper >> off >> >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a >> >>> threshold depending on System performance. Transport: I meant if you >> were >> >>> using sockets to provide the script. >> >> >> >> This makes sense. This one's on me then. >> >> >> >> >> >>> So you are using the same invocable instance for all threads? If so, >> >>> then you are probably good to go. As far as leaks are concerned, not >> sure >> >>> how you would get leaks from Nashorn. The JSON object is written in >> Java, >> >>> and little JavaScript involved. >> >> >> >> >> >> >> >>> In your example, pull up Invocable invocable = >> generateInvocable(script); >> >>> out of the loop and use the same invocable for all threads. >> >> >> >> >> >> We were using one invocable across all threads and we were getting >> >> slowdowns on execution, high CPU Usage and memory leaks that led to >> >> OutOfMemory errors. I could trace the leak to >> >> >> >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> >> >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* >> >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* >> >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter >> >> >> >> which just keeps growing forever. >> >> >> >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < >> >> james.laskey at oracle.com> wrote: >> >> >> >>> >> >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: >> >>> >> >>> The cost of creating a new engine is significant. So share an engine >> >>>> across threads but use *eval >> >>>> > ScriptEngine.html#eval(java.lang.String,%20javax.script.ScriptContext)>* >> >>>> (String >> >>>> >> >>>> script, ScriptContext >> >>>> > ScriptContext.html> >> >>>> context) instead, separate context per execution. If your JavaScript >> >>>> code does not modify globals you can get away with using the same >> engine, >> >>>> same compiled script on each thread. >> >>> >> >>> >> >>> I guess there's a few things here I don't understand. One thing I'm >> >>> trying to do is sharing a CompiledScript (which is why I'm using >> >>> invocable). Also, what exactly does modify globals mean? All our >> filters do >> >>> the same thing, make a function that takes a JSON String, turns it >> into a >> >>> JSON, modifies it and then stringifies it back. No state is changed of >> >>> anything else but there are temporary vars created inside the scope >> of the >> >>> function. When we run this multithreaded, running invokeFunction >> slows down >> >>> significantly and we get crazy memory leaks. >> >>> >> >>> >> >>> So you are using the same invocable instance for all threads? If so, >> >>> then you are probably good to go. As far as leaks are concerned, not >> sure >> >>> how you would get leaks from Nashorn. The JSON object is written in >> Java, >> >>> and little JavaScript involved. >> >>> >> >>> >> >>> Of course there are many factors involved n performance. How many >> cores >> >>>> do you have on the test machine? How much memory in the process? >> What >> >>>> transport are you using between threads? That sort of thing. Other >> than >> >>>> constructing then engine and context Nashorn performance should >> scale. >> >>> >> >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the >> >>> process. Not sure what transports between threads means, but this is >> the >> >>> code I'm benchmarking with. Increasing the number of threads actually >> makes >> >>> it go faster until about 4 threads, then adding more threads takes >> the same >> >>> amount to get to 1000 and and after a certain point it is just slower >> to >> >>> get to 1000 counts. Some of our filters need to be able to run over >> 1000 >> >>> times a second (across all threads) and the fastest time I could >> actually >> >>> get with this was about 2.4 seconds for a 1000 counts. >> >>> >> >>>> ExecutorService executor = Executors.newFixedThreadPool(50); >> >>>> >> >>>> AtomicLong count = new AtomicLong(); >> >>>> >> >>>> for (int i = 0; i < 50; i++) { >> >>>> >> >>>> executor.submit(new Runnable() { >> >>>> >> >>>> @Override >> >>>> >> >>>> public void run() { >> >>>> >> >>>> >> >>>>> try { >> >>>> >> >>>> Invocable invocable = >> >>>>> generateInvocable(script); >> >>>> >> >>>> while(true) { >> >>>> >> >>>> invocable.invokeFunction("transform", >> >>>>> something); >> >>>> >> >>>> count.incrementAndGet(); >> >>>> >> >>>> } >> >>>> >> >>>> } catch (NoSuchMethodException | >> ScriptException >> >>>>> e) { >> >>>> >> >>>> e.printStackTrace(); >> >>>> >> >>>> } >> >>>> >> >>>> } >> >>>> >> >>>> }); >> >>>> >> >>>> } >> >>>> >> >>>> long lastTimestamp = System.currentTimeMillis(); >> >>>> >> >>>> while(true) { >> >>>> >> >>>> >> >>>>> if (count.get() > 1000) { >> >>>> >> >>>> count.getAndAdd(-1000); >> >>>> >> >>>> System.out.println((System.currentTimeMillis() - >> >>>>> lastTimestamp)/1000.0); >> >>>> >> >>>> lastTimestamp = System.currentTimeMillis(); >> >>>> >> >>>> } >> >>>> >> >>>> } >> >>>> >> >>>> >> >>> With more threads you are impacting the same 8 cores, so it will taper >> >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 >> being a >> >>> threshold depending on System performance. Transport: I meant if you >> were >> >>> using sockets to provide the script. >> >>> >> >>> In your example, pull up Invocable invocable = >> generateInvocable(script); >> >>> out of the loop and use the same invocable for all threads. >> >>> >> >>> - Jim >> >>> >> >>> >> >>> >> >>> >> >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < >> >>> james.laskey at oracle.com> wrote: >> >>> >> >>>> >> >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon >> wrote: >> >>>> >> >>>> Hey Jim, >> >>>> >> >>>> I looked at it and I will look into loadWithNewGlobal to see what >> >>>> exactly it does since it could be relevant. As for the rest, for my >> use >> >>>> case having threads in the JS would not help. We're using Nashorn to >> build >> >>>> JSON filters in a Dynamic Proxy Service and need any of the threads >> >>>> processing a request to be able to execute the script to filter. >> >>>> >> >>>> >> >>>> The cost of creating a new engine is significant. So share an engine >> >>>> across threads but use *eval >> >>>> > ScriptEngine.html#eval(java.lang.String,%20javax.script.ScriptContext)>* >> >>>> (String >> >>>> >> >>>> script, ScriptContext >> >>>> > ScriptContext.html> >> >>>> context) instead, separate context per execution. If your JavaScript >> >>>> code does not modify globals you can get away with using the same >> engine, >> >>>> same compiled script on each thread. >> >>>> >> >>>> >> >>>> Also, when you say a new engine per threads is the worst case what >> >>>> exactly do you mean? I would expect an initial cost of compiling the >> script >> >>>> on each thread and then each engine should be able to do its own >> thing, but >> >>>> what I'm seeing is that when running with more than 10 threads all my >> >>>> engines get slow at executing code, even though they are all >> completely >> >>>> separate from each other. >> >>>> >> >>>> >> >>>> Of course there are many factors involved n performance. How many >> cores >> >>>> do you have on the test machine? How much memory in the process? >> What >> >>>> transport are you using between threads? That sort of thing. Other >> than >> >>>> constructing then engine and context Nashorn performance should >> scale. >> >>>> >> >>>> >> >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < >> >>>> james.laskey at oracle.com> wrote: >> >>>> >> >>>>> Jesus, >> >>>>> >> >>>>> Probably the most informative information is in this blog. >> >>>>> >> >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threadi >> ng_and_mt >> >>>>> >> >>>>> This example uses Executors but threads would work as well. >> >>>>> >> >>>>> I did a talk that looked at different methods to max out >> multithreading >> >>>>> performance. A new engine per thread is the worst case. A new >> context per >> >>>>> thread does much better. A new global per thread is the best while >> >>>>> remaining thread safe. >> >>>>> >> >>>>> Cheers, >> >>>>> >> >>>>> ? Jim >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon >> wrote: >> >>>>> >> >>>>> Hey folks, >> >>>>> >> >>>>> I've tried many different ways of using Nashorn multithreaded based >> on >> >>>>> what >> >>>>> I've found on the internet and I still can't get a single one to >> scale. >> >>>>> Even the most naive method of making many script engines with my >> script >> >>>>> tends to bottleneck itself when I have more than 10 threads invoking >> >>>>> functions. >> >>>>> >> >>>>> I'm using the following code to compile my script and >> >>>>> invocable.invokeFunction("transform", input) to execute: >> >>>>> >> >>>>> static Invocable generateInvocable(String script) throws >> >>>>> ScriptException { >> >>>>> ScriptEngineManager manager = new ScriptEngineManager(); >> >>>>> ScriptEngine engine = >> >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >> >>>>> Compilable compilable = (Compilable) engine; >> >>>>> final CompiledScript compiled = compilable.compile(script); >> >>>>> compiled.eval(); >> >>>>> return (Invocable) engine; >> >>>>> } >> >>>>> >> >>>>> >> >>>>> >> >>>>> The script I'm compiling is: >> >>>>> >> >>>>> String script = "function transform(input) {" + >> >>>>> "var result = JSON.parse(input);" + >> >>>>> "response = {};\n" + >> >>>>> "for (var i = 0; i < result.length; i++) {\n" + >> >>>>> " var summoner = {};\n" + >> >>>>> " summoner.id = result[i].id;\n" + >> >>>>> " summoner.name = result[i].name;\n" + >> >>>>> " summoner.profileIconId = >> >>>>> result[i].profileIconId;\n" + >> >>>>> " summoner.revisionDate = >> result[i].revisionDate;\n" + >> >>>>> " summoner.summonerLevel = result[i].level;\n" + >> >>>>> " response[summoner.id] = summoner;\n" + >> >>>>> "}\n" + >> >>>>> "result = response;" + >> >>>>> "return JSON.stringify(result);" + >> >>>>> "};"; >> >>>>> >> >>>>> >> >>>>> >> >>>>> I've also tried other more scaleable ways to work with scripts >> >>>>> concurrently, but given that this is the most naive method where >> >>>>> everything >> >>>>> is brand new and I still get slowness calling them concurrently I >> fear >> >>>>> that >> >>>>> maybe I'm overlooking something extremely basic on my code. >> >>>>> >> >>>>> Thanks. >> >>>>> -Jesus Luzon >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >> >> >> >> >> > From james.laskey at oracle.com Thu Dec 8 01:02:50 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Wed, 7 Dec 2016 21:02:50 -0400 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: <6FB18C5A-9BF8-49C8-B97D-2BB029F040E4@oracle.com> Yes this is part of the js spec I assume you mean undefined not null. > On Dec 7, 2016, at 7:40 PM, Jesus Luzon wrote: > > Looks like the memory leak is due to the way we wrote our code and how javascript works. I was expecting the line response[summoner.id ] = summoner; to build a map but it turns out that if you use a number as the "key", javscript automatically fills the indexes in the middle with null (Undefined type?). When these IDs are very large, it is creating huge arrays that take longer to garbage collect than the code executing. I am about to start testing this on our end to make sure we see the improvements we expect. > > Does this idea seem like it is reasonably possible? > > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon > wrote: > Yes, it's an array of objects which I'll paste. And yes, I'm just calling invokeFunction from many many different threads. I'm also going to go back and take a look at all the heap dumps we have to re-confirm what I mentioned. > > "[\n" + > " {\n" + > " \"id\": 6011511,\n" + > " \"accountId\": 203192481,\n" + > " \"name\": \"Adam Pro Qarda?\",\n" + > " \"profileIconId\": 25,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 1475,\n" + > " \"revisionDate\": 1406631727000\n" + > " },\n" + > " {\n" + > " \"id\": 2810674,\n" + > " \"accountId\": 200706913,\n" + > " \"name\": \"ABZ Devrim\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 982,\n" + > " \"infPoints\": 10472,\n" + > " \"revisionDate\": 1450791227000\n" + > " },\n" + > " {\n" + > " \"id\": 5411195,\n" + > " \"accountId\": 202647689,\n" + > " \"name\": \"Ace HypcronN\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 73,\n" + > " \"infPoints\": 182445,\n" + > " \"revisionDate\": 1480781650000\n" + > " },\n" + > " {\n" + > " \"id\": 1363020,\n" + > " \"accountId\": 1357837,\n" + > " \"name\": \"AdanaLee\",\n" + > " \"profileIconId\": 502,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 125,\n" + > " \"infPoints\": 719299,\n" + > " \"revisionDate\": 1480530778000\n" + > " },\n" + > " {\n" + > " \"id\": 8261198,\n" + > " \"accountId\": 205027096,\n" + > " \"name\": \"Achilehuz\",\n" + > " \"profileIconId\": 1381,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 158603,\n" + > " \"revisionDate\": 1480770307000\n" + > " },\n" + > " {\n" + > " \"id\": 12685857,\n" + > " \"accountId\": 207591166,\n" + > " \"name\": \"ac?mas?zpicc\",\n" + > " \"profileIconId\": 9,\n" + > " \"level\": 21,\n" + > " \"expPoints\": 840,\n" + > " \"infPoints\": 16659,\n" + > " \"revisionDate\": 1480515325000\n" + > " },\n" + > " {\n" + > " \"id\": 10860127,\n" + > " \"accountId\": 206507727,\n" + > " \"name\": \"AAngelFlyy\",\n" + > " \"profileIconId\": 1395,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 73111,\n" + > " \"revisionDate\": 1480787870000\n" + > " },\n" + > " {\n" + > " \"id\": 3292376,\n" + > " \"accountId\": 201048714,\n" + > " \"name\": \"ACAB1907\",\n" + > " \"profileIconId\": 20,\n" + > " \"level\": 6,\n" + > " \"expPoints\": 305,\n" + > " \"infPoints\": 2107,\n" + > " \"revisionDate\": 1402448089000\n" + > " },\n" + > " {\n" + > " \"id\": 461671,\n" + > " \"accountId\": 446571,\n" + > " \"name\": \"Acta Est Fabul?\",\n" + > " \"profileIconId\": 1435,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 47,\n" + > " \"infPoints\": 644672,\n" + > " \"revisionDate\": 1480626505000\n" + > " },\n" + > " {\n" + > " \"id\": 394183,\n" + > " \"accountId\": 379083,\n" + > " \"name\": \"acekse4\",\n" + > " \"profileIconId\": 27,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 223,\n" + > " \"infPoints\": 908,\n" + > " \"revisionDate\": 1348116544000\n" + > " },\n" + > " {\n" + > " \"id\": 5941247,\n" + > " \"accountId\": 203106300,\n" + > " \"name\": \"abdul7878\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 3,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 401,\n" + > " \"revisionDate\": 1406029148000\n" + > " },\n" + > " {\n" + > " \"id\": 2467446,\n" + > " \"accountId\": 200459837,\n" + > " \"name\": \"ActionC\",\n" + > " \"profileIconId\": 986,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 74,\n" + > " \"infPoints\": 401367,\n" + > " \"revisionDate\": 1480808608000\n" + > " },\n" + > " {\n" + > " \"id\": 9402979,\n" + > " \"accountId\": 205698832,\n" + > " \"name\": \"Ablenia \",\n" + > " \"profileIconId\": 1129,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 19,\n" + > " \"infPoints\": 163518,\n" + > " \"revisionDate\": 1480687603000\n" + > " },\n" + > " {\n" + > " \"id\": 13187505,\n" + > " \"accountId\": 207898213,\n" + > " \"name\": \"aDaMiYiM\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 116,\n" + > " \"infPoints\": 45214,\n" + > " \"revisionDate\": 1480793258000\n" + > " },\n" + > " {\n" + > " \"id\": 4141059,\n" + > " \"accountId\": 201688290,\n" + > " \"name\": \"Abimin?ar?\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 152,\n" + > " \"infPoints\": 752477,\n" + > " \"revisionDate\": 1480635961000\n" + > " },\n" + > " {\n" + > " \"id\": 5702134,\n" + > " \"accountId\": 202899395,\n" + > " \"name\": \"Above the Clouds\",\n" + > " \"profileIconId\": 684,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 110,\n" + > " \"infPoints\": 288096,\n" + > " \"revisionDate\": 1471011372000\n" + > " },\n" + > " {\n" + > " \"id\": 5810740,\n" + > " \"accountId\": 202985228,\n" + > " \"name\": \"aBimm\",\n" + > " \"profileIconId\": 11,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 1180,\n" + > " \"infPoints\": 10736,\n" + > " \"revisionDate\": 1409832684000\n" + > " },\n" + > " {\n" + > " \"id\": 5817751,\n" + > " \"accountId\": 203050678,\n" + > " \"name\": \"AD Glor?am\",\n" + > " \"profileIconId\": 982,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 111,\n" + > " \"infPoints\": 304658,\n" + > " \"revisionDate\": 1480795250000\n" + > " },\n" + > " {\n" + > " \"id\": 9851802,\n" + > " \"accountId\": 206011054,\n" + > " \"name\": \"AdarAllame\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 73763,\n" + > " \"revisionDate\": 1479422812000\n" + > " },\n" + > " {\n" + > " \"id\": 12735622,\n" + > " \"accountId\": 207587019,\n" + > " \"name\": \"absinthe666\",\n" + > " \"profileIconId\": 903,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 40302,\n" + > " \"revisionDate\": 1480782923000\n" + > " },\n" + > " {\n" + > " \"id\": 6371389,\n" + > " \"accountId\": 203416952,\n" + > " \"name\": \"adamsat?c?\",\n" + > " \"profileIconId\": 3,\n" + > " \"level\": 4,\n" + > " \"expPoints\": 17,\n" + > " \"infPoints\": 685,\n" + > " \"revisionDate\": 1409320171000\n" + > " },\n" + > " {\n" + > " \"id\": 7828139,\n" + > " \"accountId\": 204927980,\n" + > " \"name\": \"AbsoluteForce\",\n" + > " \"profileIconId\": 950,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 208789,\n" + > " \"revisionDate\": 1480804396000\n" + > " },\n" + > " {\n" + > " \"id\": 1373229,\n" + > " \"accountId\": 1358441,\n" + > " \"name\": \"AbsoluteDeath\",\n" + > " \"profileIconId\": 7,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 34,\n" + > " \"infPoints\": 223655,\n" + > " \"revisionDate\": 1471867646000\n" + > " },\n" + > " {\n" + > " \"id\": 7694972,\n" + > " \"accountId\": 204803668,\n" + > " \"name\": \"ac pnp\",\n" + > " \"profileIconId\": 937,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 161,\n" + > " \"infPoints\": 249681,\n" + > " \"revisionDate\": 1480801507000\n" + > " },\n" + > " {\n" + > " \"id\": 1373524,\n" + > " \"accountId\": 1350474,\n" + > " \"name\": \"Abd??\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 103,\n" + > " \"infPoints\": 286803,\n" + > " \"revisionDate\": 1476621827000\n" + > " },\n" + > " {\n" + > " \"id\": 1650227,\n" + > " \"accountId\": 200000503,\n" + > " \"name\": \"AD Ambrosia\",\n" + > " \"profileIconId\": 1152,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 156333,\n" + > " \"revisionDate\": 1480805320000\n" + > " },\n" + > " {\n" + > " \"id\": 8331358,\n" + > " \"accountId\": 205073925,\n" + > " \"name\": \"acarmanyust2\",\n" + > " \"profileIconId\": 0,\n" + > " \"level\": 2,\n" + > " \"expPoints\": 43,\n" + > " \"infPoints\": 318,\n" + > " \"revisionDate\": 1423915139000\n" + > " },\n" + > " {\n" + > " \"id\": 1862106,\n" + > " \"accountId\": 200139838,\n" + > " \"name\": \"aboU\",\n" + > " \"profileIconId\": 1155,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 0,\n" + > " \"infPoints\": 412616,\n" + > " \"revisionDate\": 1480771055000\n" + > " },\n" + > " {\n" + > " \"id\": 2362628,\n" + > " \"accountId\": 685649,\n" + > " \"name\": \"Ac?F?sT?k\",\n" + > " \"profileIconId\": 1074,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 233882,\n" + > " \"revisionDate\": 1480786233000\n" + > " },\n" + > " {\n" + > " \"id\": 4323909,\n" + > " \"accountId\": 201917672,\n" + > " \"name\": \"Addrenalin\",\n" + > " \"profileIconId\": 603,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 55,\n" + > " \"infPoints\": 220605,\n" + > " \"revisionDate\": 1432647338000\n" + > " },\n" + > " {\n" + > " \"id\": 377206,\n" + > " \"accountId\": 362106,\n" + > " \"name\": \"Aburame Shino\",\n" + > " \"profileIconId\": 844,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 84,\n" + > " \"infPoints\": 354087,\n" + > " \"revisionDate\": 1477666556000\n" + > " },\n" + > " {\n" + > " \"id\": 5377433,\n" + > " \"accountId\": 202697921,\n" + > " \"name\": \"AcEcolton35\",\n" + > " \"profileIconId\": 984,\n" + > " \"level\": 25,\n" + > " \"expPoints\": 751,\n" + > " \"infPoints\": 30061,\n" + > " \"revisionDate\": 1475503024000\n" + > " },\n" + > " {\n" + > " \"id\": 2381404,\n" + > " \"accountId\": 200333680,\n" + > " \"name\": \"adafakaaa\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 8,\n" + > " \"infPoints\": 534204,\n" + > " \"revisionDate\": 1480719827000\n" + > " },\n" + > " {\n" + > " \"id\": 1281203,\n" + > " \"accountId\": 1259342,\n" + > " \"name\": \"AC Klondike\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 27,\n" + > " \"infPoints\": 191429,\n" + > " \"revisionDate\": 1480294973000\n" + > " },\n" + > " {\n" + > " \"id\": 13161471,\n" + > " \"accountId\": 207847181,\n" + > " \"name\": \"adar21\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 10,\n" + > " \"expPoints\": 143,\n" + > " \"infPoints\": 3558,\n" + > " \"revisionDate\": 1476529855000\n" + > " },\n" + > " {\n" + > " \"id\": 5841915,\n" + > " \"accountId\": 202998794,\n" + > " \"name\": \"Achilles29\",\n" + > " \"profileIconId\": 666,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 41,\n" + > " \"infPoints\": 219714,\n" + > " \"revisionDate\": 1480777744000\n" + > " },\n" + > " {\n" + > " \"id\": 2853062,\n" + > " \"accountId\": 200726707,\n" + > " \"name\": \"AbIanStarBebegim\",\n" + > " \"profileIconId\": 909,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 64,\n" + > " \"infPoints\": 297580,\n" + > " \"revisionDate\": 1480556859000\n" + > " },\n" + > " {\n" + > " \"id\": 8323114,\n" + > " \"accountId\": 205093515,\n" + > " \"name\": \"Absuruk\",\n" + > " \"profileIconId\": 6,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 21,\n" + > " \"infPoints\": 121086,\n" + > " \"revisionDate\": 1480820801000\n" + > " }\n" + > "]" > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer > wrote: > Hi Jesus, > > I?m trying to reproduce the problem, and just want to make sure I get the missing pieces right. > > You already showed us how you?re setting up the engine and the JS code you?re running. I assume the JSON code you?re parsing is a simple array of objects? And you?re just calling Invocable.invokeFunction on the ScriptEngine from multiple threads in parallel, right? > > Thanks, > Hannes > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon >: > > > > When we share one invocable across many threads and run invokeFunction it > > happens, such as this: > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > >> > >> Invocable invocable = generateInvocable(script); > >> > >> AtomicLong count = new AtomicLong(); > >> > >> for (int i = 0; i < 50; i++) { > >> > >> executor.submit(new Runnable() { > >> > >> @Override > >> > >> public void run() { > >> > >> try { > >> > >> while(true) { > >> > >> invocable.invokeFunction("transform", > >>> something); > >> > >> count.incrementAndGet(); > >> > >> } > >> > >> } catch (NoSuchMethodException | ScriptException > >>> e) { > >> > >> e.printStackTrace(); > >> > >> } > >> > >> } > >> > >> }); > >> > >> } > >> > >> > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) > >> wrote: > > > >> Intersting. The example you posted demonstrates this behaviour? If so > >> I?ll file a bug and dig in. It sounds like an object is being reused > >> across invocations and accumulating changes to the property map. > >> > >> ? Jim > >> > >> > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon > wrote: > >> > >> With more threads you are impacting the same 8 cores, so it will taper off > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > >>> threshold depending on System performance. Transport: I meant if you were > >>> using sockets to provide the script. > >> > >> This makes sense. This one's on me then. > >> > >> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not sure > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > >>> and little JavaScript involved. > >> > >> > >> > >>> In your example, pull up Invocable invocable = generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >> > >> > >> We were using one invocable across all threads and we were getting > >> slowdowns on execution, high CPU Usage and memory leaks that led to > >> OutOfMemory errors. I could trace the leak to > >> > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > >> > >> which just keeps growing forever. > >> > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > >> james.laskey at oracle.com > wrote: > >> > >>> > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon > wrote: > >>> > >>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> >* > >>>> (String > >>>> > > >>>> script, ScriptContext > >>>> > > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same engine, > >>>> same compiled script on each thread. > >>> > >>> > >>> I guess there's a few things here I don't understand. One thing I'm > >>> trying to do is sharing a CompiledScript (which is why I'm using > >>> invocable). Also, what exactly does modify globals mean? All our filters do > >>> the same thing, make a function that takes a JSON String, turns it into a > >>> JSON, modifies it and then stringifies it back. No state is changed of > >>> anything else but there are temporary vars created inside the scope of the > >>> function. When we run this multithreaded, running invokeFunction slows down > >>> significantly and we get crazy memory leaks. > >>> > >>> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not sure > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > >>> and little JavaScript involved. > >>> > >>> > >>> Of course there are many factors involved n performance. How many cores > >>>> do you have on the test machine? How much memory in the process? What > >>>> transport are you using between threads? That sort of thing. Other than > >>>> constructing then engine and context Nashorn performance should scale. > >>> > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the > >>> process. Not sure what transports between threads means, but this is the > >>> code I'm benchmarking with. Increasing the number of threads actually makes > >>> it go faster until about 4 threads, then adding more threads takes the same > >>> amount to get to 1000 and and after a certain point it is just slower to > >>> get to 1000 counts. Some of our filters need to be able to run over 1000 > >>> times a second (across all threads) and the fastest time I could actually > >>> get with this was about 2.4 seconds for a 1000 counts. > >>> > >>>> ExecutorService executor = Executors.newFixedThreadPool(50); > >>>> > >>>> AtomicLong count = new AtomicLong(); > >>>> > >>>> for (int i = 0; i < 50; i++) { > >>>> > >>>> executor.submit(new Runnable() { > >>>> > >>>> @Override > >>>> > >>>> public void run() { > >>>> > >>>> > >>>>> try { > >>>> > >>>> Invocable invocable = > >>>>> generateInvocable(script); > >>>> > >>>> while(true) { > >>>> > >>>> invocable.invokeFunction("transform", > >>>>> something); > >>>> > >>>> count.incrementAndGet(); > >>>> > >>>> } > >>>> > >>>> } catch (NoSuchMethodException | ScriptException > >>>>> e) { > >>>> > >>>> e.printStackTrace(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> }); > >>>> > >>>> } > >>>> > >>>> long lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> while(true) { > >>>> > >>>> > >>>>> if (count.get() > 1000) { > >>>> > >>>> count.getAndAdd(-1000); > >>>> > >>>> System.out.println((System.currentTimeMillis() - > >>>>> lastTimestamp)/1000.0); > >>>> > >>>> lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> > >>> With more threads you are impacting the same 8 cores, so it will taper > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > >>> threshold depending on System performance. Transport: I meant if you were > >>> using sockets to provide the script. > >>> > >>> In your example, pull up Invocable invocable = generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >>> > >>> - Jim > >>> > >>> > >>> > >>> > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > >>> james.laskey at oracle.com > wrote: > >>> > >>>> > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon > wrote: > >>>> > >>>> Hey Jim, > >>>> > >>>> I looked at it and I will look into loadWithNewGlobal to see what > >>>> exactly it does since it could be relevant. As for the rest, for my use > >>>> case having threads in the JS would not help. We're using Nashorn to build > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads > >>>> processing a request to be able to execute the script to filter. > >>>> > >>>> > >>>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> >* > >>>> (String > >>>> > > >>>> script, ScriptContext > >>>> > > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same engine, > >>>> same compiled script on each thread. > >>>> > >>>> > >>>> Also, when you say a new engine per threads is the worst case what > >>>> exactly do you mean? I would expect an initial cost of compiling the script > >>>> on each thread and then each engine should be able to do its own thing, but > >>>> what I'm seeing is that when running with more than 10 threads all my > >>>> engines get slow at executing code, even though they are all completely > >>>> separate from each other. > >>>> > >>>> > >>>> Of course there are many factors involved n performance. How many cores > >>>> do you have on the test machine? How much memory in the process? What > >>>> transport are you using between threads? That sort of thing. Other than > >>>> constructing then engine and context Nashorn performance should scale. > >>>> > >>>> > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > >>>> james.laskey at oracle.com > wrote: > >>>> > >>>>> Jesus, > >>>>> > >>>>> Probably the most informative information is in this blog. > >>>>> > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt > >>>>> > >>>>> This example uses Executors but threads would work as well. > >>>>> > >>>>> I did a talk that looked at different methods to max out multithreading > >>>>> performance. A new engine per thread is the worst case. A new context per > >>>>> thread does much better. A new global per thread is the best while > >>>>> remaining thread safe. > >>>>> > >>>>> Cheers, > >>>>> > >>>>> ? Jim > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: > >>>>> > >>>>> Hey folks, > >>>>> > >>>>> I've tried many different ways of using Nashorn multithreaded based on > >>>>> what > >>>>> I've found on the internet and I still can't get a single one to scale. > >>>>> Even the most naive method of making many script engines with my script > >>>>> tends to bottleneck itself when I have more than 10 threads invoking > >>>>> functions. > >>>>> > >>>>> I'm using the following code to compile my script and > >>>>> invocable.invokeFunction("transform", input) to execute: > >>>>> > >>>>> static Invocable generateInvocable(String script) throws > >>>>> ScriptException { > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > >>>>> ScriptEngine engine = > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > >>>>> Compilable compilable = (Compilable) engine; > >>>>> final CompiledScript compiled = compilable.compile(script); > >>>>> compiled.eval(); > >>>>> return (Invocable) engine; > >>>>> } > >>>>> > >>>>> > >>>>> > >>>>> The script I'm compiling is: > >>>>> > >>>>> String script = "function transform(input) {" + > >>>>> "var result = JSON.parse(input);" + > >>>>> "response = {};\n" + > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > >>>>> " var summoner = {};\n" + > >>>>> " summoner.id = result[i].id;\n" + > >>>>> " summoner.name = result[i].name;\n" + > >>>>> " summoner.profileIconId = > >>>>> result[i].profileIconId;\n" + > >>>>> " summoner.revisionDate = result[i].revisionDate;\n" + > >>>>> " summoner.summonerLevel = result[i].level;\n" + > >>>>> " response[summoner.id ] = summoner;\n" + > >>>>> "}\n" + > >>>>> "result = response;" + > >>>>> "return JSON.stringify(result);" + > >>>>> "};"; > >>>>> > >>>>> > >>>>> > >>>>> I've also tried other more scaleable ways to work with scripts > >>>>> concurrently, but given that this is the most naive method where > >>>>> everything > >>>>> is brand new and I still get slowness calling them concurrently I fear > >>>>> that > >>>>> maybe I'm overlooking something extremely basic on my code. > >>>>> > >>>>> Thanks. > >>>>> -Jesus Luzon > >>>>> > >>>>> > >>>> > >>> > >>> > >> > >> > > > From hannes.wallnoefer at oracle.com Thu Dec 8 11:52:00 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 8 Dec 2016 12:52:00 +0100 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Yes, this is very likely to be the cause of the problem. However, I do think we should be able to handle sparse array data better, so it?s quite possible that you exposed some weak spot in our implementation. I?ll have a look into what?s going on. Hannes > Am 08.12.2016 um 00:40 schrieb Jesus Luzon : > > Looks like the memory leak is due to the way we wrote our code and how javascript works. I was expecting the line response[summoner.id] = summoner; to build a map but it turns out that if you use a number as the "key", javscript automatically fills the indexes in the middle with null (Undefined type?). When these IDs are very large, it is creating huge arrays that take longer to garbage collect than the code executing. I am about to start testing this on our end to make sure we see the improvements we expect. > > Does this idea seem like it is reasonably possible? > > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon wrote: > Yes, it's an array of objects which I'll paste. And yes, I'm just calling invokeFunction from many many different threads. I'm also going to go back and take a look at all the heap dumps we have to re-confirm what I mentioned. > > "[\n" + > " {\n" + > " \"id\": 6011511,\n" + > " \"accountId\": 203192481,\n" + > " \"name\": \"Adam Pro Qarda?\",\n" + > " \"profileIconId\": 25,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 1475,\n" + > " \"revisionDate\": 1406631727000\n" + > " },\n" + > " {\n" + > " \"id\": 2810674,\n" + > " \"accountId\": 200706913,\n" + > " \"name\": \"ABZ Devrim\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 982,\n" + > " \"infPoints\": 10472,\n" + > " \"revisionDate\": 1450791227000\n" + > " },\n" + > " {\n" + > " \"id\": 5411195,\n" + > " \"accountId\": 202647689,\n" + > " \"name\": \"Ace HypcronN\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 73,\n" + > " \"infPoints\": 182445,\n" + > " \"revisionDate\": 1480781650000\n" + > " },\n" + > " {\n" + > " \"id\": 1363020,\n" + > " \"accountId\": 1357837,\n" + > " \"name\": \"AdanaLee\",\n" + > " \"profileIconId\": 502,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 125,\n" + > " \"infPoints\": 719299,\n" + > " \"revisionDate\": 1480530778000\n" + > " },\n" + > " {\n" + > " \"id\": 8261198,\n" + > " \"accountId\": 205027096,\n" + > " \"name\": \"Achilehuz\",\n" + > " \"profileIconId\": 1381,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 158603,\n" + > " \"revisionDate\": 1480770307000\n" + > " },\n" + > " {\n" + > " \"id\": 12685857,\n" + > " \"accountId\": 207591166,\n" + > " \"name\": \"ac?mas?zpicc\",\n" + > " \"profileIconId\": 9,\n" + > " \"level\": 21,\n" + > " \"expPoints\": 840,\n" + > " \"infPoints\": 16659,\n" + > " \"revisionDate\": 1480515325000\n" + > " },\n" + > " {\n" + > " \"id\": 10860127,\n" + > " \"accountId\": 206507727,\n" + > " \"name\": \"AAngelFlyy\",\n" + > " \"profileIconId\": 1395,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 73111,\n" + > " \"revisionDate\": 1480787870000\n" + > " },\n" + > " {\n" + > " \"id\": 3292376,\n" + > " \"accountId\": 201048714,\n" + > " \"name\": \"ACAB1907\",\n" + > " \"profileIconId\": 20,\n" + > " \"level\": 6,\n" + > " \"expPoints\": 305,\n" + > " \"infPoints\": 2107,\n" + > " \"revisionDate\": 1402448089000\n" + > " },\n" + > " {\n" + > " \"id\": 461671,\n" + > " \"accountId\": 446571,\n" + > " \"name\": \"Acta Est Fabul?\",\n" + > " \"profileIconId\": 1435,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 47,\n" + > " \"infPoints\": 644672,\n" + > " \"revisionDate\": 1480626505000\n" + > " },\n" + > " {\n" + > " \"id\": 394183,\n" + > " \"accountId\": 379083,\n" + > " \"name\": \"acekse4\",\n" + > " \"profileIconId\": 27,\n" + > " \"level\": 5,\n" + > " \"expPoints\": 223,\n" + > " \"infPoints\": 908,\n" + > " \"revisionDate\": 1348116544000\n" + > " },\n" + > " {\n" + > " \"id\": 5941247,\n" + > " \"accountId\": 203106300,\n" + > " \"name\": \"abdul7878\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 3,\n" + > " \"expPoints\": 10,\n" + > " \"infPoints\": 401,\n" + > " \"revisionDate\": 1406029148000\n" + > " },\n" + > " {\n" + > " \"id\": 2467446,\n" + > " \"accountId\": 200459837,\n" + > " \"name\": \"ActionC\",\n" + > " \"profileIconId\": 986,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 74,\n" + > " \"infPoints\": 401367,\n" + > " \"revisionDate\": 1480808608000\n" + > " },\n" + > " {\n" + > " \"id\": 9402979,\n" + > " \"accountId\": 205698832,\n" + > " \"name\": \"Ablenia \",\n" + > " \"profileIconId\": 1129,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 19,\n" + > " \"infPoints\": 163518,\n" + > " \"revisionDate\": 1480687603000\n" + > " },\n" + > " {\n" + > " \"id\": 13187505,\n" + > " \"accountId\": 207898213,\n" + > " \"name\": \"aDaMiYiM\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 116,\n" + > " \"infPoints\": 45214,\n" + > " \"revisionDate\": 1480793258000\n" + > " },\n" + > " {\n" + > " \"id\": 4141059,\n" + > " \"accountId\": 201688290,\n" + > " \"name\": \"Abimin?ar?\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 152,\n" + > " \"infPoints\": 752477,\n" + > " \"revisionDate\": 1480635961000\n" + > " },\n" + > " {\n" + > " \"id\": 5702134,\n" + > " \"accountId\": 202899395,\n" + > " \"name\": \"Above the Clouds\",\n" + > " \"profileIconId\": 684,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 110,\n" + > " \"infPoints\": 288096,\n" + > " \"revisionDate\": 1471011372000\n" + > " },\n" + > " {\n" + > " \"id\": 5810740,\n" + > " \"accountId\": 202985228,\n" + > " \"name\": \"aBimm\",\n" + > " \"profileIconId\": 11,\n" + > " \"level\": 13,\n" + > " \"expPoints\": 1180,\n" + > " \"infPoints\": 10736,\n" + > " \"revisionDate\": 1409832684000\n" + > " },\n" + > " {\n" + > " \"id\": 5817751,\n" + > " \"accountId\": 203050678,\n" + > " \"name\": \"AD Glor?am\",\n" + > " \"profileIconId\": 982,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 111,\n" + > " \"infPoints\": 304658,\n" + > " \"revisionDate\": 1480795250000\n" + > " },\n" + > " {\n" + > " \"id\": 9851802,\n" + > " \"accountId\": 206011054,\n" + > " \"name\": \"AdarAllame\",\n" + > " \"profileIconId\": 911,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 73763,\n" + > " \"revisionDate\": 1479422812000\n" + > " },\n" + > " {\n" + > " \"id\": 12735622,\n" + > " \"accountId\": 207587019,\n" + > " \"name\": \"absinthe666\",\n" + > " \"profileIconId\": 903,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 83,\n" + > " \"infPoints\": 40302,\n" + > " \"revisionDate\": 1480782923000\n" + > " },\n" + > " {\n" + > " \"id\": 6371389,\n" + > " \"accountId\": 203416952,\n" + > " \"name\": \"adamsat?c?\",\n" + > " \"profileIconId\": 3,\n" + > " \"level\": 4,\n" + > " \"expPoints\": 17,\n" + > " \"infPoints\": 685,\n" + > " \"revisionDate\": 1409320171000\n" + > " },\n" + > " {\n" + > " \"id\": 7828139,\n" + > " \"accountId\": 204927980,\n" + > " \"name\": \"AbsoluteForce\",\n" + > " \"profileIconId\": 950,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 208789,\n" + > " \"revisionDate\": 1480804396000\n" + > " },\n" + > " {\n" + > " \"id\": 1373229,\n" + > " \"accountId\": 1358441,\n" + > " \"name\": \"AbsoluteDeath\",\n" + > " \"profileIconId\": 7,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 34,\n" + > " \"infPoints\": 223655,\n" + > " \"revisionDate\": 1471867646000\n" + > " },\n" + > " {\n" + > " \"id\": 7694972,\n" + > " \"accountId\": 204803668,\n" + > " \"name\": \"ac pnp\",\n" + > " \"profileIconId\": 937,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 161,\n" + > " \"infPoints\": 249681,\n" + > " \"revisionDate\": 1480801507000\n" + > " },\n" + > " {\n" + > " \"id\": 1373524,\n" + > " \"accountId\": 1350474,\n" + > " \"name\": \"Abd??\",\n" + > " \"profileIconId\": 1301,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 103,\n" + > " \"infPoints\": 286803,\n" + > " \"revisionDate\": 1476621827000\n" + > " },\n" + > " {\n" + > " \"id\": 1650227,\n" + > " \"accountId\": 200000503,\n" + > " \"name\": \"AD Ambrosia\",\n" + > " \"profileIconId\": 1152,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 139,\n" + > " \"infPoints\": 156333,\n" + > " \"revisionDate\": 1480805320000\n" + > " },\n" + > " {\n" + > " \"id\": 8331358,\n" + > " \"accountId\": 205073925,\n" + > " \"name\": \"acarmanyust2\",\n" + > " \"profileIconId\": 0,\n" + > " \"level\": 2,\n" + > " \"expPoints\": 43,\n" + > " \"infPoints\": 318,\n" + > " \"revisionDate\": 1423915139000\n" + > " },\n" + > " {\n" + > " \"id\": 1862106,\n" + > " \"accountId\": 200139838,\n" + > " \"name\": \"aboU\",\n" + > " \"profileIconId\": 1155,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 0,\n" + > " \"infPoints\": 412616,\n" + > " \"revisionDate\": 1480771055000\n" + > " },\n" + > " {\n" + > " \"id\": 2362628,\n" + > " \"accountId\": 685649,\n" + > " \"name\": \"Ac?F?sT?k\",\n" + > " \"profileIconId\": 1074,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 48,\n" + > " \"infPoints\": 233882,\n" + > " \"revisionDate\": 1480786233000\n" + > " },\n" + > " {\n" + > " \"id\": 4323909,\n" + > " \"accountId\": 201917672,\n" + > " \"name\": \"Addrenalin\",\n" + > " \"profileIconId\": 603,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 55,\n" + > " \"infPoints\": 220605,\n" + > " \"revisionDate\": 1432647338000\n" + > " },\n" + > " {\n" + > " \"id\": 377206,\n" + > " \"accountId\": 362106,\n" + > " \"name\": \"Aburame Shino\",\n" + > " \"profileIconId\": 844,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 84,\n" + > " \"infPoints\": 354087,\n" + > " \"revisionDate\": 1477666556000\n" + > " },\n" + > " {\n" + > " \"id\": 5377433,\n" + > " \"accountId\": 202697921,\n" + > " \"name\": \"AcEcolton35\",\n" + > " \"profileIconId\": 984,\n" + > " \"level\": 25,\n" + > " \"expPoints\": 751,\n" + > " \"infPoints\": 30061,\n" + > " \"revisionDate\": 1475503024000\n" + > " },\n" + > " {\n" + > " \"id\": 2381404,\n" + > " \"accountId\": 200333680,\n" + > " \"name\": \"adafakaaa\",\n" + > " \"profileIconId\": 663,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 8,\n" + > " \"infPoints\": 534204,\n" + > " \"revisionDate\": 1480719827000\n" + > " },\n" + > " {\n" + > " \"id\": 1281203,\n" + > " \"accountId\": 1259342,\n" + > " \"name\": \"AC Klondike\",\n" + > " \"profileIconId\": 898,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 27,\n" + > " \"infPoints\": 191429,\n" + > " \"revisionDate\": 1480294973000\n" + > " },\n" + > " {\n" + > " \"id\": 13161471,\n" + > " \"accountId\": 207847181,\n" + > " \"name\": \"adar21\",\n" + > " \"profileIconId\": 26,\n" + > " \"level\": 10,\n" + > " \"expPoints\": 143,\n" + > " \"infPoints\": 3558,\n" + > " \"revisionDate\": 1476529855000\n" + > " },\n" + > " {\n" + > " \"id\": 5841915,\n" + > " \"accountId\": 202998794,\n" + > " \"name\": \"Achilles29\",\n" + > " \"profileIconId\": 666,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 41,\n" + > " \"infPoints\": 219714,\n" + > " \"revisionDate\": 1480777744000\n" + > " },\n" + > " {\n" + > " \"id\": 2853062,\n" + > " \"accountId\": 200726707,\n" + > " \"name\": \"AbIanStarBebegim\",\n" + > " \"profileIconId\": 909,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 64,\n" + > " \"infPoints\": 297580,\n" + > " \"revisionDate\": 1480556859000\n" + > " },\n" + > " {\n" + > " \"id\": 8323114,\n" + > " \"accountId\": 205093515,\n" + > " \"name\": \"Absuruk\",\n" + > " \"profileIconId\": 6,\n" + > " \"level\": 30,\n" + > " \"expPoints\": 21,\n" + > " \"infPoints\": 121086,\n" + > " \"revisionDate\": 1480820801000\n" + > " }\n" + > "]" > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer wrote: > Hi Jesus, > > I?m trying to reproduce the problem, and just want to make sure I get the missing pieces right. > > You already showed us how you?re setting up the engine and the JS code you?re running. I assume the JSON code you?re parsing is a simple array of objects? And you?re just calling Invocable.invokeFunction on the ScriptEngine from multiple threads in parallel, right? > > Thanks, > Hannes > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > > > When we share one invocable across many threads and run invokeFunction it > > happens, such as this: > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > >> > >> Invocable invocable = generateInvocable(script); > >> > >> AtomicLong count = new AtomicLong(); > >> > >> for (int i = 0; i < 50; i++) { > >> > >> executor.submit(new Runnable() { > >> > >> @Override > >> > >> public void run() { > >> > >> try { > >> > >> while(true) { > >> > >> invocable.invokeFunction("transform", > >>> something); > >> > >> count.incrementAndGet(); > >> > >> } > >> > >> } catch (NoSuchMethodException | ScriptException > >>> e) { > >> > >> e.printStackTrace(); > >> > >> } > >> > >> } > >> > >> }); > >> > >> } > >> > >> > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) >> wrote: > > > >> Intersting. The example you posted demonstrates this behaviour? If so > >> I?ll file a bug and dig in. It sounds like an object is being reused > >> across invocations and accumulating changes to the property map. > >> > >> ? Jim > >> > >> > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > >> > >> With more threads you are impacting the same 8 cores, so it will taper off > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > >>> threshold depending on System performance. Transport: I meant if you were > >>> using sockets to provide the script. > >> > >> This makes sense. This one's on me then. > >> > >> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not sure > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > >>> and little JavaScript involved. > >> > >> > >> > >>> In your example, pull up Invocable invocable = generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >> > >> > >> We were using one invocable across all threads and we were getting > >> slowdowns on execution, high CPU Usage and memory leaks that led to > >> OutOfMemory errors. I could trace the leak to > >> > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > >> > >> which just keeps growing forever. > >> > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > >> james.laskey at oracle.com> wrote: > >> > >>> > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: > >>> > >>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> * > >>>> (String > >>>> > >>>> script, ScriptContext > >>>> > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same engine, > >>>> same compiled script on each thread. > >>> > >>> > >>> I guess there's a few things here I don't understand. One thing I'm > >>> trying to do is sharing a CompiledScript (which is why I'm using > >>> invocable). Also, what exactly does modify globals mean? All our filters do > >>> the same thing, make a function that takes a JSON String, turns it into a > >>> JSON, modifies it and then stringifies it back. No state is changed of > >>> anything else but there are temporary vars created inside the scope of the > >>> function. When we run this multithreaded, running invokeFunction slows down > >>> significantly and we get crazy memory leaks. > >>> > >>> > >>> So you are using the same invocable instance for all threads? If so, > >>> then you are probably good to go. As far as leaks are concerned, not sure > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > >>> and little JavaScript involved. > >>> > >>> > >>> Of course there are many factors involved n performance. How many cores > >>>> do you have on the test machine? How much memory in the process? What > >>>> transport are you using between threads? That sort of thing. Other than > >>>> constructing then engine and context Nashorn performance should scale. > >>> > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the > >>> process. Not sure what transports between threads means, but this is the > >>> code I'm benchmarking with. Increasing the number of threads actually makes > >>> it go faster until about 4 threads, then adding more threads takes the same > >>> amount to get to 1000 and and after a certain point it is just slower to > >>> get to 1000 counts. Some of our filters need to be able to run over 1000 > >>> times a second (across all threads) and the fastest time I could actually > >>> get with this was about 2.4 seconds for a 1000 counts. > >>> > >>>> ExecutorService executor = Executors.newFixedThreadPool(50); > >>>> > >>>> AtomicLong count = new AtomicLong(); > >>>> > >>>> for (int i = 0; i < 50; i++) { > >>>> > >>>> executor.submit(new Runnable() { > >>>> > >>>> @Override > >>>> > >>>> public void run() { > >>>> > >>>> > >>>>> try { > >>>> > >>>> Invocable invocable = > >>>>> generateInvocable(script); > >>>> > >>>> while(true) { > >>>> > >>>> invocable.invokeFunction("transform", > >>>>> something); > >>>> > >>>> count.incrementAndGet(); > >>>> > >>>> } > >>>> > >>>> } catch (NoSuchMethodException | ScriptException > >>>>> e) { > >>>> > >>>> e.printStackTrace(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> }); > >>>> > >>>> } > >>>> > >>>> long lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> while(true) { > >>>> > >>>> > >>>>> if (count.get() > 1000) { > >>>> > >>>> count.getAndAdd(-1000); > >>>> > >>>> System.out.println((System.currentTimeMillis() - > >>>>> lastTimestamp)/1000.0); > >>>> > >>>> lastTimestamp = System.currentTimeMillis(); > >>>> > >>>> } > >>>> > >>>> } > >>>> > >>>> > >>> With more threads you are impacting the same 8 cores, so it will taper > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > >>> threshold depending on System performance. Transport: I meant if you were > >>> using sockets to provide the script. > >>> > >>> In your example, pull up Invocable invocable = generateInvocable(script); > >>> out of the loop and use the same invocable for all threads. > >>> > >>> - Jim > >>> > >>> > >>> > >>> > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > >>> james.laskey at oracle.com> wrote: > >>> > >>>> > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: > >>>> > >>>> Hey Jim, > >>>> > >>>> I looked at it and I will look into loadWithNewGlobal to see what > >>>> exactly it does since it could be relevant. As for the rest, for my use > >>>> case having threads in the JS would not help. We're using Nashorn to build > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads > >>>> processing a request to be able to execute the script to filter. > >>>> > >>>> > >>>> The cost of creating a new engine is significant. So share an engine > >>>> across threads but use *eval > >>>> * > >>>> (String > >>>> > >>>> script, ScriptContext > >>>> > >>>> context) instead, separate context per execution. If your JavaScript > >>>> code does not modify globals you can get away with using the same engine, > >>>> same compiled script on each thread. > >>>> > >>>> > >>>> Also, when you say a new engine per threads is the worst case what > >>>> exactly do you mean? I would expect an initial cost of compiling the script > >>>> on each thread and then each engine should be able to do its own thing, but > >>>> what I'm seeing is that when running with more than 10 threads all my > >>>> engines get slow at executing code, even though they are all completely > >>>> separate from each other. > >>>> > >>>> > >>>> Of course there are many factors involved n performance. How many cores > >>>> do you have on the test machine? How much memory in the process? What > >>>> transport are you using between threads? That sort of thing. Other than > >>>> constructing then engine and context Nashorn performance should scale. > >>>> > >>>> > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > >>>> james.laskey at oracle.com> wrote: > >>>> > >>>>> Jesus, > >>>>> > >>>>> Probably the most informative information is in this blog. > >>>>> > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt > >>>>> > >>>>> This example uses Executors but threads would work as well. > >>>>> > >>>>> I did a talk that looked at different methods to max out multithreading > >>>>> performance. A new engine per thread is the worst case. A new context per > >>>>> thread does much better. A new global per thread is the best while > >>>>> remaining thread safe. > >>>>> > >>>>> Cheers, > >>>>> > >>>>> ? Jim > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: > >>>>> > >>>>> Hey folks, > >>>>> > >>>>> I've tried many different ways of using Nashorn multithreaded based on > >>>>> what > >>>>> I've found on the internet and I still can't get a single one to scale. > >>>>> Even the most naive method of making many script engines with my script > >>>>> tends to bottleneck itself when I have more than 10 threads invoking > >>>>> functions. > >>>>> > >>>>> I'm using the following code to compile my script and > >>>>> invocable.invokeFunction("transform", input) to execute: > >>>>> > >>>>> static Invocable generateInvocable(String script) throws > >>>>> ScriptException { > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > >>>>> ScriptEngine engine = > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > >>>>> Compilable compilable = (Compilable) engine; > >>>>> final CompiledScript compiled = compilable.compile(script); > >>>>> compiled.eval(); > >>>>> return (Invocable) engine; > >>>>> } > >>>>> > >>>>> > >>>>> > >>>>> The script I'm compiling is: > >>>>> > >>>>> String script = "function transform(input) {" + > >>>>> "var result = JSON.parse(input);" + > >>>>> "response = {};\n" + > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > >>>>> " var summoner = {};\n" + > >>>>> " summoner.id = result[i].id;\n" + > >>>>> " summoner.name = result[i].name;\n" + > >>>>> " summoner.profileIconId = > >>>>> result[i].profileIconId;\n" + > >>>>> " summoner.revisionDate = result[i].revisionDate;\n" + > >>>>> " summoner.summonerLevel = result[i].level;\n" + > >>>>> " response[summoner.id] = summoner;\n" + > >>>>> "}\n" + > >>>>> "result = response;" + > >>>>> "return JSON.stringify(result);" + > >>>>> "};"; > >>>>> > >>>>> > >>>>> > >>>>> I've also tried other more scaleable ways to work with scripts > >>>>> concurrently, but given that this is the most naive method where > >>>>> everything > >>>>> is brand new and I still get slowness calling them concurrently I fear > >>>>> that > >>>>> maybe I'm overlooking something extremely basic on my code. > >>>>> > >>>>> Thanks. > >>>>> -Jesus Luzon > >>>>> > >>>>> > >>>> > >>> > >>> > >> > >> > > > From Rony.Flatscher at wu.ac.at Thu Dec 8 15:05:38 2016 From: Rony.Flatscher at wu.ac.at (Rony G. Flatscher) Date: Thu, 8 Dec 2016 16:05:38 +0100 Subject: Nashorn's javax.script.ScriptEngineFactory produces wrong code Message-ID: Hi there, was directed to this list to report an error with Nashorn's implementation of javax.script.ScriptEngineFactory.getOutputStatement(String toDisplay). The test is simple: one supplies a string, containing double quotes, and immediately use the Nashort script engine to carry out the output statement produced by its factory, yielding a runtime error. In order for you to reproduce, I list the Java utility (it lists all available javax.script engines and tests them with the -t switch and is a little example of how one can take advantage of javax.script very easily) after my signature below to study the code and run it to double-check: F:\work\svn\bsf4oorexx\trunk\samples\Java\jsr223>java RunScript -t ---> language [ECMAScript]: ----------------------------------------> 1) output statement to process: hello world, this is "ECMAScript" speaking! --> testing getOutputStatement(String), produced the following [ECMAScript] output statement *print(hello world, this is "ECMAScript" speaking! )* ... now running "eval(code)" using the scripting engine [ECMAScript]: *javax.script.ScriptException: :1:12 Expected , but found world print(hello world, this is "ECMAScript" speaking! ) ^ in at line number 1 at column number 12 while eval() the code: print(hello world, this is "ECMAScript" speaking! )* 2) testing getMethodCallSyntax(obj, meth, arg...), produced the following [ECMAScript] method call statement: object.method(arg1,arg2) 3) testing getProgram(String...) produced the following [ECMAScript] program: print(hello world, this is "ECMAScript" speaking! );object.method(arg1,arg2); <--- end of testing language [ECMAScript]. <---------------------------------------- ---> language [ooRexx]: ----------------------------------------> ... cut ... The specification for javax.script created by the Java specification request group 223 (a.k.a. JSR-223) can be consulted. BTW, "jrunscript[.exe]" (from the JDK) seems to be broken, it only is be able to execute the Nashorn script engine, despite switches that allow other script engines to be employed. ---rony Code of "RunScript.java" (little utility to list the currently available javax.script languages, run any javax.script program, give helpful information about each javax.script engine and test each one using its own factory means): just compile it and run it with "java RunScript -t": ------------------ cut here ------------------ import java.io.File; import java.io.FileReader; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.HashMap; import java.util.SortedSet; import java.util.TreeSet; import javax.script.Bindings; import javax.script.ScriptContext; import javax.script.ScriptEngine; import javax.script.ScriptEngineFactory; import javax.script.ScriptEngineManager; import javax.script.ScriptException; import javax.script.SimpleScriptContext; /* ------------------------ Apache Version 2.0 license ------------------------- Copyright 2015-2016 Rony G. Flatscher Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ----------------------------------------------------------------------------- changed: 2016-12-06, rgf: if no argument is given, show the help after the list of currently available javax.script engines */ /** A command line Java utility to query, test and evaluate {@link javax.script} scripting engines that * gets distributed via the beta BSF4ooRexx package from Sourceforge. * *

Usage: java RunScript [{-i | -t | -e enginename filename [args...] | filename [args...]}] *
*

    *
  • ... no arguments: lists the currently available javax.script engines * *
  • -h ... 'help', lists and briefly explains the command line options * *
  • -i ... lists the available javax.script engines and informs about their properties * *
  • -t ... lists the available javax.script engines and tests evaluating their factory's "getOutputStatement(String)" method * *
  • -e enginename filename [args...] ... uses "enginename" to evaluate "filename" supplying the arguments "args..." * *
  • filename [args...] ... the "filename" extension determines the script engine to evaluate it, supplying the arguments "args..." *
* * * @author Rony G. Flatscher * @since 2015-05-20 */ public class RunScript { public static void main(String args[]) { ScriptEngineManager manager = new ScriptEngineManager(); List factories = manager.getEngineFactories(); HashMap n2sef=new HashMap (); HashMap n2se =new HashMap (); // create the Maps for (ScriptEngineFactory factory : factories) { String name=factory.getLanguageName(); // String name=factory.getEngineName(); n2sef.put(name, factory); n2se .put(name, factory.getScriptEngine()); } // create a sorted set of key names SortedSet keys = new TreeSet(n2sef.keySet()); int argsStartAt=-1; // arguments - if any - for filenmae to execute start at this position ScriptEngine scriptEngineToUse=null; String filename=null; if (args.length==0) // list all available javax.script engines { System.out.println("The following javax.script engines are currently available:\n"); for (String key : keys) { System.out.println('\t'+key); } System.out.println(); showHelp(); return; } else // process the first argument to decide what we need to do { String firstWord=args[0]; // get first value if (firstWord.startsWith("-")==true) // a switch at hand! { if (firstWord.length()!=2) // oops an error, we need excactly two chars (-i, -t, -e) { throw new IllegalArgumentException("switch '-' must be followed by one of the letters 'i', 't', or 'e'"); } String uSwitch=firstWord.substring(1).toUpperCase(); // get switch in uppercase if (uSwitch.equals("I")) // list all engines, list all their standard properties { for (String key : keys) // list all engines in order{ { showEngineProperties(key, n2sef.get(key)); } return; } else if (uSwitch.equals("T")) // list all engines, test them { for (String key : keys) // list all engines in order{ { testEngine(key, n2sef.get(key), n2se.get(key)); } return; } else if (uSwitch.equals("H")) // -h ... help text { showHelp(); return; } else if (uSwitch.equals("E")) // -e engineName fileName [arg... ] { if (args.length<3) { throw new IllegalArgumentException("too few command line arguments ("+args.length+"), minimum needed is 3: \"-e enginename filename\""); } // check whether engine is available, if so, then load it filename=args[2]; // assign filename String errMsg="no \""+args[1]+"\" javax.script engine available"; try { scriptEngineToUse=manager.getEngineByName(args[1]); // fetch script engine } catch (Exception exc) { throw new IllegalArgumentException(errMsg); } if (scriptEngineToUse==null) { throw new IllegalArgumentException(errMsg); } argsStartAt=3; // fourth argument! } else // unknown switch { throw new IllegalArgumentException("unknown switch '"+firstWord+"', must be followed by one of '-h', '-i', '-t', or '-e'"); } } else // a filename in hand, check whether we have a suitable engine available { // - check whether suitable engine is available, if not raise an exception filename=args[0]; // assign filename int lastIdx=filename.lastIndexOf('.'); if (lastIdx==0) { throw new IllegalArgumentException("filename \"filename\" does not have an extension, cannot determine scripting engine"); } String errMsg="cannot determine scripting engine from the filename extension in \""+filename+"\""; try { String extension=filename.substring(lastIdx+1); // get extension scriptEngineToUse=manager.getEngineByExtension(extension); // fetch script engine } catch (Exception exc) // if substring() causes an exception { throw new IllegalArgumentException(errMsg); } if (scriptEngineToUse==null) // no scripting engine found { throw new IllegalArgumentException(errMsg); } argsStartAt=1; // second argument! } } // now setup the remainder and eval() the "filename" with the script engine // - check whether file exists, if not raise an exception File f=null; try { f=new File (filename); } catch (Exception exc) { throw new IllegalArgumentException(exc); } if (f.exists()==false) { throw new IllegalArgumentException("filename \""+filename+"\" does not exist"); } // - supply filename ScriptContext sc=scriptEngineToUse.getContext(); // get the default context sc.setAttribute(ScriptEngine.FILENAME, filename, ScriptContext.ENGINE_SCOPE); // - if arguments, setup ARGV in ENGINE_SCOPE if (args.length>argsStartAt) // do we have command line arguments to supply? { String argArr []=new String [args.length-argsStartAt]; // determine array size int k=0; for (int i=argsStartAt; i FAILED!"); } System.out.print ("\tgetEngineVersion() : "); try { System.out.println(sef.getEngineVersion() ); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetExtensions() : "); try { System.out.println(sef.getExtensions ()); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetLanguageName() : "); try { System.out.println(sef.getLanguageName () ); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetLanguageVersion() : "); try { System.out.println(sef.getLanguageVersion()); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetMimeTypes() : "); try { System.out.println(sef.getMimeTypes ()); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetNames() : "); try { System.out.println(sef.getNames() ); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetParameter(\"THREADING\"): "); try { System.out.println(sef.getParameter("THREADING")); } catch (Exception e) { System.out.println("--> FAILED!"); } } // create an output statement and execute output statement for each available script engine static void testEngine(String name, ScriptEngineFactory sef, ScriptEngine se) { System.out.println("---> language ["+name+"]: ---------------------------------------->\n"); String text="hello world, this is \""+name+"\" speaking! "; String code=sef.getOutputStatement(text); System.out.println("\t1) output statement to process: "+text+"\n"); System.out.print("\t--> testing getOutputStatement(String)"); System.out.println(", produced the following ["+name+"] output statement \n\n"+code+"\n"); System.out.println("\t... now running \"eval(code)\" using the scripting engine ["+name+"]: \n\n"); try { se.eval(code); System.out.println("\n"); } catch (ScriptException sexc) { System.err.println(sexc+" while eval() the code: "+code+"\n"); } System.out.print("\t2) testing getMethodCallSyntax(obj, meth, arg...)"); String methCode=sef.getMethodCallSyntax("object", "method", "arg1", "arg2"); System.out.println(", produced the following ["+name+"] method call statement: \n\n"+methCode+"\n"); System.out.print("\t3) testing getProgram(String...)"); String programCode=sef.getProgram(code, methCode); System.out.println(" produced the following ["+name+"] program: \n\n"+programCode+"\n"); System.out.println("<--- end of testing language ["+name+"]. <----------------------------------------\n"); } // allow us to call this from different parts static void showHelp() { System.out.println("usage: java RunScript [{-i | -t | -e enginename filename [args...] | filename [args...]}]\n"); System.out.println(" ... no arguments lists the available javax.script engines"); System.out.println(" -h ... 'help', gives this usage text"); System.out.println(" -i ... lists the available javax.script engines and informs about their properties"); System.out.println(" -t ... lists the available javax.script engines and tests evaluating their \"getOutputStatement(String)\" method"); System.out.println(" -e enginename filename [args...] ... uses \"enginename\" to evaluate \"filename\" supplying the arguments \"args...\""); System.out.println(" filename [args...] ... the \"filename\" extension determines the script engine to evaluate it, supplying the arguments \"args...\""); return; } } ------------------ cut here ------------------ From Rony.Flatscher at wu.ac.at Thu Dec 8 15:17:26 2016 From: Rony.Flatscher at wu.ac.at (Rony G. Flatscher) Date: Thu, 8 Dec 2016 16:17:26 +0100 Subject: Nashorn's javax.script.ScriptEngineFactory produces wrong code In-Reply-To: References: Message-ID: <9bc6202f-7a5c-e9f2-7e41-da8e90bfccec@wu.ac.at> Unfortunately, the mailing list seems to behave as if it was running in the 1970's such that the formatted text (indented, partially colorized and formatted to preformat and Courier) has turned to become illegible, unfortunately (using the Thunderbird e-Mail client)! :( The meat is right in the first two paragraphs and the last paragraph before the signature. Should anyone be interested in the Java-utility, please drop me a personal e-mail and I will send that Java program as an attachment to you. ---rony On 08.12.2016 16:05, Rony G. Flatscher wrote: > Hi there, was directed to this list to report an error with Nashorn's implementation of > javax.script.ScriptEngineFactory.getOutputStatement(String toDisplay). > > The test is simple: one supplies a string, containing double quotes, and immediately use the Nashort > script engine to carry out the output statement produced by its factory, yielding a runtime error. > In order for you to reproduce, I list the Java utility (it lists all available javax.script engines > and tests them with the -t switch and is a little example of how one can take advantage of > javax.script very easily) after my signature below to study the code and run it to double-check: > > F:\work\svn\bsf4oorexx\trunk\samples\Java\jsr223>java RunScript -t ---> language [ECMAScript]: > ----------------------------------------> 1) output statement to process: hello world, this is > "ECMAScript" speaking! --> testing getOutputStatement(String), produced the following > [ECMAScript] output statement *print(hello world, this is "ECMAScript" speaking! )* ... now > running "eval(code)" using the scripting engine [ECMAScript]: *javax.script.ScriptException: > :1:12 Expected , but found world print(hello world, this is "ECMAScript" speaking! ) ^ in > at line number 1 at column number 12 while eval() the code: print(hello world, this is > "ECMAScript" speaking! )* 2) testing getMethodCallSyntax(obj, meth, arg...), produced the > following [ECMAScript] method call statement: object.method(arg1,arg2) 3) testing > getProgram(String...) produced the following [ECMAScript] program: print(hello world, this is > "ECMAScript" speaking! );object.method(arg1,arg2); <--- end of testing language [ECMAScript]. > <---------------------------------------- ---> language [ooRexx]: > ----------------------------------------> ... cut ... > > The specification for javax.script created by the Java specification request group 223 (a.k.a. > JSR-223) can be consulted. BTW, "jrunscript[.exe]" (from the JDK) seems to be broken, it only is be > able to execute the Nashorn script engine, despite switches that allow other script engines to be > employed. > > ---rony > > Code of "RunScript.java" (little utility to list the currently available javax.script languages, run > any javax.script program, give helpful information about each javax.script engine and test each one > using its own factory means): just compile it and run it with "java RunScript -t": > > ------------------ cut here ------------------ > import java.io.File; import java.io.FileReader; import java.util.ArrayList; import java.util.Arrays; > import java.util.List; import java.util.HashMap; import java.util.SortedSet; import > java.util.TreeSet; import javax.script.Bindings; import javax.script.ScriptContext; import > javax.script.ScriptEngine; import javax.script.ScriptEngineFactory; import > javax.script.ScriptEngineManager; import javax.script.ScriptException; import > javax.script.SimpleScriptContext; /* ------------------------ Apache Version 2.0 license > ------------------------- Copyright 2015-2016 Rony G. Flatscher Licensed under the Apache License, > Version 2.0 (the "License"); you may not use this file except in compliance with the License. You > may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by > applicable law or agreed to in writing, software distributed under the License is distributed on an > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the > License for the specific language governing permissions and limitations under the License. > ----------------------------------------------------------------------------- changed: 2016-12-06, > rgf: if no argument is given, show the help after the list of currently available javax.script > engines */ /** A command line Java utility to query, test and evaluate {@link javax.script} > scripting engines that * gets distributed via the beta BSF4ooRexx package from Sourceforge. * * >

Usage: java RunScript [{-i | -t | -e enginename filename [args...] | filename > [args...]}] *
*

    *
  • ... no arguments: lists the currently available > javax.script engines * *
  • -h ... 'help', lists and briefly explains > the command line options * *
  • -i ... lists the available javax.script > engines and informs about their properties * *
  • -t ... lists the available > javax.script engines and tests evaluating their factory's > "getOutputStatement(String)" method * *
  • -e enginename filename > [args...] ... uses "enginename" to evaluate > "filename" supplying the arguments "args..." * *
  • > filename [args...] ... the "filename" extension determines the > script engine to evaluate it, supplying the arguments "args..." *
* * * > @author Rony G. Flatscher * @since 2015-05-20 */ public class RunScript { public static void > main(String args[]) { ScriptEngineManager manager = new ScriptEngineManager(); > List factories = manager.getEngineFactories(); > HashMap n2sef=new HashMap (); > HashMap n2se =new HashMap (); // create the Maps for > (ScriptEngineFactory factory : factories) { String name=factory.getLanguageName(); // String > name=factory.getEngineName(); n2sef.put(name, factory); n2se .put(name, factory.getScriptEngine()); > } // create a sorted set of key names SortedSet keys = new TreeSet(n2sef.keySet()); > int argsStartAt=-1; // arguments - if any - for filenmae to execute start at this position > ScriptEngine scriptEngineToUse=null; String filename=null; if (args.length==0) // list all available > javax.script engines { System.out.println("The following javax.script engines are currently > available:\n"); for (String key : keys) { System.out.println('\t'+key); } System.out.println(); > showHelp(); return; } else // process the first argument to decide what we need to do { String > firstWord=args[0]; // get first value if (firstWord.startsWith("-")==true) // a switch at hand! { if > (firstWord.length()!=2) // oops an error, we need excactly two chars (-i, -t, -e) { throw new > IllegalArgumentException("switch '-' must be followed by one of the letters 'i', 't', or 'e'"); } > String uSwitch=firstWord.substring(1).toUpperCase(); // get switch in uppercase if > (uSwitch.equals("I")) // list all engines, list all their standard properties { for (String key : > keys) // list all engines in order{ { showEngineProperties(key, n2sef.get(key)); } return; } else if > (uSwitch.equals("T")) // list all engines, test them { for (String key : keys) // list all engines > in order{ { testEngine(key, n2sef.get(key), n2se.get(key)); } return; } else if > (uSwitch.equals("H")) // -h ... help text { showHelp(); return; } else if (uSwitch.equals("E")) // > -e engineName fileName [arg... ] { if (args.length<3) { throw new IllegalArgumentException("too few > command line arguments ("+args.length+"), minimum needed is 3: \"-e enginename filename\""); } // > check whether engine is available, if so, then load it filename=args[2]; // assign filename String > errMsg="no \""+args[1]+"\" javax.script engine available"; try { > scriptEngineToUse=manager.getEngineByName(args[1]); // fetch script engine } catch (Exception exc) { > throw new IllegalArgumentException(errMsg); } if (scriptEngineToUse==null) { throw new > IllegalArgumentException(errMsg); } argsStartAt=3; // fourth argument! } else // unknown switch { > throw new IllegalArgumentException("unknown switch '"+firstWord+"', must be followed by one of '-h', > '-i', '-t', or '-e'"); } } else // a filename in hand, check whether we have a suitable engine > available { // - check whether suitable engine is available, if not raise an exception > filename=args[0]; // assign filename int lastIdx=filename.lastIndexOf('.'); if (lastIdx==0) { throw > new IllegalArgumentException("filename \"filename\" does not have an extension, cannot determine > scripting engine"); } String errMsg="cannot determine scripting engine from the filename extension > in \""+filename+"\""; try { String extension=filename.substring(lastIdx+1); // get extension > scriptEngineToUse=manager.getEngineByExtension(extension); // fetch script engine } catch (Exception > exc) // if substring() causes an exception { throw new IllegalArgumentException(errMsg); } if > (scriptEngineToUse==null) // no scripting engine found { throw new IllegalArgumentException(errMsg); > } argsStartAt=1; // second argument! } } // now setup the remainder and eval() the "filename" with > the script engine // - check whether file exists, if not raise an exception File f=null; try { f=new > File (filename); } catch (Exception exc) { throw new IllegalArgumentException(exc); } if > (f.exists()==false) { throw new IllegalArgumentException("filename \""+filename+"\" does not > exist"); } // - supply filename ScriptContext sc=scriptEngineToUse.getContext(); // get the default > context sc.setAttribute(ScriptEngine.FILENAME, filename, ScriptContext.ENGINE_SCOPE); // - if > arguments, setup ARGV in ENGINE_SCOPE if (args.length>argsStartAt) // do we have command line > arguments to supply? { String argArr []=new String [args.length-argsStartAt]; // determine array > size int k=0; for (int i=argsStartAt; i sc.setAttribute(ScriptEngine.ARGV, argArr, ScriptContext.ENGINE_SCOPE); } // - eval the script try { > scriptEngineToUse.eval(new FileReader(f)); } catch (Exception exc) { throw new > IllegalArgumentException(exc); } } // show information about the script engine static void > showEngineProperties(String name, ScriptEngineFactory sef) { System.out.println(name); > System.out.print ("\tgetEngineName() : "); try { System.out.println(sef.getEngineName() ); } catch > (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetEngineVersion() : "); > try { System.out.println(sef.getEngineVersion() ); } catch (Exception e) { System.out.println("--> > FAILED!"); } System.out.print ("\tgetExtensions() : "); try { System.out.println(sef.getExtensions > ()); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print > ("\tgetLanguageName() : "); try { System.out.println(sef.getLanguageName () ); } catch (Exception e) > { System.out.println("--> FAILED!"); } System.out.print ("\tgetLanguageVersion() : "); try { > System.out.println(sef.getLanguageVersion()); } catch (Exception e) { System.out.println("--> > FAILED!"); } System.out.print ("\tgetMimeTypes() : "); try { System.out.println(sef.getMimeTypes > ()); } catch (Exception e) { System.out.println("--> FAILED!"); } System.out.print ("\tgetNames() : > "); try { System.out.println(sef.getNames() ); } catch (Exception e) { System.out.println("--> > FAILED!"); } System.out.print ("\tgetParameter(\"THREADING\"): "); try { > System.out.println(sef.getParameter("THREADING")); } catch (Exception e) { System.out.println("--> > FAILED!"); } } // create an output statement and execute output statement for each available script > engine static void testEngine(String name, ScriptEngineFactory sef, ScriptEngine se) { > System.out.println("---> language ["+name+"]: ---------------------------------------->\n"); String > text="hello world, this is \""+name+"\" speaking! "; String code=sef.getOutputStatement(text); > System.out.println("\t1) output statement to process: "+text+"\n"); System.out.print("\t--> testing > getOutputStatement(String)"); System.out.println(", produced the following ["+name+"] output > statement \n\n"+code+"\n"); System.out.println("\t... now running \"eval(code)\" using the scripting > engine ["+name+"]: \n\n"); try { se.eval(code); System.out.println("\n"); } catch (ScriptException > sexc) { System.err.println(sexc+" while eval() the code: "+code+"\n"); } System.out.print("\t2) > testing getMethodCallSyntax(obj, meth, arg...)"); String methCode=sef.getMethodCallSyntax("object", > "method", "arg1", "arg2"); System.out.println(", produced the following ["+name+"] method call > statement: \n\n"+methCode+"\n"); System.out.print("\t3) testing getProgram(String...)"); String > programCode=sef.getProgram(code, methCode); System.out.println(" produced the following ["+name+"] > program: \n\n"+programCode+"\n"); System.out.println("<--- end of testing language ["+name+"]. > <----------------------------------------\n"); } // allow us to call this from different parts > static void showHelp() { System.out.println("usage: java RunScript [{-i | -t | -e enginename > filename [args...] | filename [args...]}]\n"); System.out.println(" ... no arguments lists the > available javax.script engines"); System.out.println(" -h ... 'help', gives this usage text"); > System.out.println(" -i ... lists the available javax.script engines and informs about their > properties"); System.out.println(" -t ... lists the available javax.script engines and tests > evaluating their \"getOutputStatement(String)\" method"); System.out.println(" -e enginename > filename [args...] ... uses \"enginename\" to evaluate \"filename\" supplying the arguments > \"args...\""); System.out.println(" filename [args...] ... the \"filename\" extension determines the > script engine to evaluate it, supplying the arguments \"args...\""); return; } } ------------------ > cut here ------------------ > > > > From ekemokai at gmail.com Thu Dec 8 15:58:46 2016 From: ekemokai at gmail.com (Edmond Kemokai) Date: Thu, 8 Dec 2016 10:58:46 -0500 Subject: Caching and re-using bindings In-Reply-To: References: Message-ID: Not a problem at all, happy to help!! Also feel free to further explore HiveMind and share with other devs, it is a very innovative platform for using Java scripting engines to build webs. On Dec 8, 2016 9:18 AM, "yikes aroni" wrote: thanks Edmond --- VERY MUCH FOR YOUR TIME!!!!!!! So i see where i was thinking wrong. And the "workaround" that i've been using isn't really a workaround at all, but a canonical approach: namely, if you want to snapshot and cache the engine bindings, you createBindings(), then putAll of what's in the engine bindings at that point. again, thank you. People like you who help (and stick with it even when one is being thick!) are solid gold. On Wed, Dec 7, 2016 at 2:51 PM, Edmond Kemokai wrote: > Your code is working as expected, I think your understanding of how it > should work is a bit off. When I look at your code I see you retrieve the > current engine binding via se.getBindings, this simply returns the binding, > it doesn't disassociate it with the engine. So when you subsequently define > new variables they'll still show up in that binding because that binding > remains associated, ie "bound" to the engine. > > > I added a new version of your code that shows how you can use different > bindings without them stepping on one anothers toes. Just right-click on > *binding-test.ste* and choose "*Test In Browser*" to see the result. > Notices the second variable does not show up in the original binding. > > To see the difference in my code, just comment out the line with > setBindings (and save, then test again), it will behave like your code > again. > > > > On Wed, Dec 7, 2016 at 12:51 PM, yikes aroni wrote: > >> Hi edmond ... I went to your link and added my code but wasn't clear to >> me what to do from that point. The (java) code is simply this (below). It >> >> >> 1. adds a variable to ENGINE bindings >> 2. Creates a "cache" of the ENGINE bindings (variable named "bEngine") >> 3. Adds another variable to the engine bindings >> 4. --> this is what i find unexpected: the new variable appears in >> the "cached" bEngine bindings. >> >> That's really the nut of it for me: The "copy" of the ENGINE scope >> bindings appears to be a reference to the actual engine scope bindings, not >> a copy of them. If i explicitly create a new bindings >> >> var bEngine = new SimpleBindings(se.getBindings(ScriptContext.ENGINE)); >> >> the same thing happens -- any variable added to the engine scope bindings >> appears in bEngine. I assume this is by design, but i can't understand WHY >> that would be the design... >> >> The workaround i've found is to create a generic map to use as a cache >> (rather than using getBindings()) and putAll from the engine bindings. >> >> >> import java.util.Map; >> >> import javax.script.Bindings; >> import javax.script.ScriptContext; >> import javax.script.ScriptEngine; >> import javax.script.ScriptEngineManager; >> >> public class TempEngineTest2 { >> >> static ScriptEngineManager seManager = new ScriptEngineManager(); >> static ScriptEngine se = seManager.getEngineByName("nashorn"); >> public static void printBindings(Map bindings) { >> System.out.println("------------------------"); >> for (Map.Entry entry : bindings.entrySet()) { >> System.out.println(entry.getKey() + " = " + entry.getValue()); >> } >> } >> public static void main(String[] a) throws Exception { >> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalA = 'testA';"); >> // Snapshot the bindings from the engine. >> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >> // print the bindings to confirm there is only one variable in them. --> >> TRUE >> printBindings(bEngine); >> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >> se.eval("globalB = 'testB';"); >> // print the bindings. Now both variables are present........... WHY??? >> printBindings(bEngine); >> } >> } >> >> On Tue, Dec 6, 2016 at 8:56 AM, Edmond Kemokai >> wrote: >> >>> I doubt there is a problem, you're probably making a minor error. I am >>> the developer of HiveMind, it is a web app platform that relies entirely on >>> scripting engines. It doesn't cache bindings but rather caches the engine >>> and creates a new binding to handle every request. >>> >>> You can access a demo instance: http://demo.crudzilla.com:7000/ >>> Login with: developer/developer >>> >>> I have created a test file in: /com/crudzilla/cloudTest/web/b >>> inding-test.ste >>> >>> Put your code in there and save, I'll take a look to see what might be >>> missing. >>> >>> >>> >>> >>> >>> On Tue, Dec 6, 2016 at 8:04 AM, yikes aroni >>> wrote: >>> >>>> Hmm.... it works if i instantiate the SimpleBinding obj and then putAll >>>> the bindings from the original bindings obj. >>>> >>>> Bindings bSimple = new SimpleBindings(); >>>> bSimple.putAll(bEngine); >>>> >>>> Otherwise they either remain the same object or are bound in some way >>>> s.t. changes to bEngine are in bSimple. This just seems strange to me. >>>> Still curious to know what the reason for this is, despite having found a >>>> workaround. >>>> >>>> >>>> >>>> On Tue, Dec 6, 2016 at 7:59 AM, yikes aroni >>>> wrote: >>>> >>>>> Thanks for your response Edmond -- the problem is that a >>>>> simplebindings object still gets all the changes to the original >>>>> bindings.... see below: >>>>> >>>>> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >>>>> se.eval("globalA = 'testA';"); >>>>> // Snapshot the bindings from the engine. >>>>> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >>>>> // print the bindings to confirm there is only one variable in them. >>>>> --> TRUE >>>>> printBindings(bEngine); >>>>> ------------------------ >>>>> globalA = testA >>>>> // At this point there is only 'globalA' in the bindings so >>>>> instantiate simplebindings: >>>>> Bindings bSimple = new SimpleBindings(bEngine); >>>>> // Verify by printing the bindings >>>>> printBindings(bSimple); >>>>> ------------------------ >>>>> globalA = testA >>>>> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >>>>> se.eval("globalB = 'testB';"); >>>>> // print the bindings. Now both variables are present........... WHY??? >>>>> printBindings(bEngine); >>>>> ------------------------ >>>>> globalA = testA >>>>> globalB = testB >>>>> // But look -- they are in the simple bindings obj as well! >>>>> printBindings(bSimple);------------------------ >>>>> globalA = testA >>>>> globalB = testB >>>>> >>>>> That's precisely what i don't get --> Why are variables added to the >>>>> engine bindings *after* instantiating the simpleBindings showing up in the >>>>> simpleBindings object? >>>>> >>>>> And how do i avoid that? >>>>> >>>>> thanks >>>>> >>>>> On Mon, Dec 5, 2016 at 3:01 PM, Edmond Kemokai >>>>> wrote: >>>>> >>>>>> Why don't you create instances of SimpleBinding and use those as >>>>>> needed? Use ScriptEngine.setBindings to reset the binding... >>>>>> >>>>>> On Dec 5, 2016 1:36 PM, "yikes aroni" wrote: >>>>>> >>>>>>> I want to cache ScriptEngine bindings for reuse. The basic algo >>>>>>> would be >>>>>>> >>>>>>> 1) Build up my ScriptEngine (SE) with stuff i need. >>>>>>> 2) Snapshot the bindings -- i.e., cache them >>>>>>> Use engine.getBindings(ScriptContext.ENGINE_SCOPE) >>>>>>> 3) Use the SE for stuff that might modify its state. >>>>>>> 4) When done, replace the SE's bindings with my snapshotted bindings >>>>>>> Use engine.setBindings(_bindings_, ScriptContext.ENGINE_SCOPE) >>>>>>> 5) I now have a "fresh" SE to use. >>>>>>> >>>>>>> The problem is that this doesn't work as expected. The cached >>>>>>> bindings >>>>>>> appear to point to the actual SE bindings and therefore whatever >>>>>>> gets added >>>>>>> to the SE bindings, also gets added to the cached bindings. Here's >>>>>>> some >>>>>>> code to show how it's not doing what i would expect. >>>>>>> >>>>>>> public class TempEngineTest { >>>>>>> >>>>>>> static ScriptEngineManager seManager = new ScriptEngineManager(); >>>>>>> static ScriptEngine se = seManager.getEngineByName("nashorn"); >>>>>>> public static void printBindings(Map bindings) { >>>>>>> for (Map.Entry entry : bindings.entrySet()) { >>>>>>> System.out.println(entry.getKey() + " = " + entry.getValue()); >>>>>>> } >>>>>>> } >>>>>>> public static void main(String[] a) throws Exception { >>>>>>> // put variable globalA into the ScriptContext.ENGINE_SCOPE bindings >>>>>>> se.eval("globalA = 'testA';"); >>>>>>> // Snapshot the bindings from the engine. >>>>>>> Bindings bEngine = se.getBindings(ScriptContext.ENGINE_SCOPE ); >>>>>>> // print the bindings to confirm there is only one variable in them. >>>>>>> --> >>>>>>> TRUE >>>>>>> printBindings(bEngine); >>>>>>> // put variable globalB into the ScriptContext.ENGINE_SCOPE bindings >>>>>>> se.eval("globalB = 'testB';"); >>>>>>> // print the bindings. Now both variables are present........... >>>>>>> WHY??? >>>>>>> printBindings(bEngine); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> I've seen suggestions to cache and reuse bindings in various >>>>>>> articles, but >>>>>>> no specific code for doing so. How do i accomplish this in actual >>>>>>> code? >>>>>>> >>>>>>> thanks >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> > > > > From jluzon at riotgames.com Fri Dec 9 00:54:57 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Thu, 8 Dec 2016 16:54:57 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: It doesn't seem like changing to a string fixed it. We dug deeper and found a very interesting issue though. After running the invocable a few thousand times, when we put a breakpoint in ArrayData.computerIteratorKeys() and we found that it would stop at an execution from SparseArrayData with an underlying array of size 923392, stored length of 461672 (this is one of our IDs, the largest 6 digit one). Because of this we tried a run changing all the IDs in the array we pass in to number from 1-38 and this thing went blazing fast. When putting a break point in the same place, this array in SpareArrayData was now of size 39. We then changed an id in the array we take in to be 1337 and the size of the array in the SparseArrayData was 1338. I don't understand why or how to prevent but it's using this ID as an index for SpareArrayData underlying array. If someone can help me find a workaround for this I would be extremely grateful. On Thu, Dec 8, 2016 at 3:52 AM, Hannes Walln?fer < hannes.wallnoefer at oracle.com> wrote: > Yes, this is very likely to be the cause of the problem. However, I do > think we should be able to handle sparse array data better, so it?s quite > possible that you exposed some weak spot in our implementation. I?ll have a > look into what?s going on. > > Hannes > > > > Am 08.12.2016 um 00:40 schrieb Jesus Luzon : > > > > Looks like the memory leak is due to the way we wrote our code and how > javascript works. I was expecting the line response[summoner.id] = > summoner; to build a map but it turns out that if you use a number as the > "key", javscript automatically fills the indexes in the middle with null > (Undefined type?). When these IDs are very large, it is creating huge > arrays that take longer to garbage collect than the code executing. I am > about to start testing this on our end to make sure we see the improvements > we expect. > > > > Does this idea seem like it is reasonably possible? > > > > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon > wrote: > > Yes, it's an array of objects which I'll paste. And yes, I'm just > calling invokeFunction from many many different threads. I'm also going to > go back and take a look at all the heap dumps we have to re-confirm what I > mentioned. > > > > "[\n" + > > " {\n" + > > " \"id\": 6011511,\n" + > > " \"accountId\": 203192481,\n" + > > " \"name\": \"Adam Pro Qarda?\",\n" + > > " \"profileIconId\": 25,\n" + > > " \"level\": 5,\n" + > > " \"expPoints\": 83,\n" + > > " \"infPoints\": 1475,\n" + > > " \"revisionDate\": 1406631727000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2810674,\n" + > > " \"accountId\": 200706913,\n" + > > " \"name\": \"ABZ Devrim\",\n" + > > " \"profileIconId\": 663,\n" + > > " \"level\": 13,\n" + > > " \"expPoints\": 982,\n" + > > " \"infPoints\": 10472,\n" + > > " \"revisionDate\": 1450791227000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5411195,\n" + > > " \"accountId\": 202647689,\n" + > > " \"name\": \"Ace HypcronN\",\n" + > > " \"profileIconId\": 911,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 73,\n" + > > " \"infPoints\": 182445,\n" + > > " \"revisionDate\": 1480781650000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1363020,\n" + > > " \"accountId\": 1357837,\n" + > > " \"name\": \"AdanaLee\",\n" + > > " \"profileIconId\": 502,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 125,\n" + > > " \"infPoints\": 719299,\n" + > > " \"revisionDate\": 1480530778000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8261198,\n" + > > " \"accountId\": 205027096,\n" + > > " \"name\": \"Achilehuz\",\n" + > > " \"profileIconId\": 1381,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 158603,\n" + > > " \"revisionDate\": 1480770307000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 12685857,\n" + > > " \"accountId\": 207591166,\n" + > > " \"name\": \"ac?mas?zpicc\",\n" + > > " \"profileIconId\": 9,\n" + > > " \"level\": 21,\n" + > > " \"expPoints\": 840,\n" + > > " \"infPoints\": 16659,\n" + > > " \"revisionDate\": 1480515325000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 10860127,\n" + > > " \"accountId\": 206507727,\n" + > > " \"name\": \"AAngelFlyy\",\n" + > > " \"profileIconId\": 1395,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 73111,\n" + > > " \"revisionDate\": 1480787870000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 3292376,\n" + > > " \"accountId\": 201048714,\n" + > > " \"name\": \"ACAB1907\",\n" + > > " \"profileIconId\": 20,\n" + > > " \"level\": 6,\n" + > > " \"expPoints\": 305,\n" + > > " \"infPoints\": 2107,\n" + > > " \"revisionDate\": 1402448089000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 461671,\n" + > > " \"accountId\": 446571,\n" + > > " \"name\": \"Acta Est Fabul?\",\n" + > > " \"profileIconId\": 1435,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 47,\n" + > > " \"infPoints\": 644672,\n" + > > " \"revisionDate\": 1480626505000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 394183,\n" + > > " \"accountId\": 379083,\n" + > > " \"name\": \"acekse4\",\n" + > > " \"profileIconId\": 27,\n" + > > " \"level\": 5,\n" + > > " \"expPoints\": 223,\n" + > > " \"infPoints\": 908,\n" + > > " \"revisionDate\": 1348116544000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5941247,\n" + > > " \"accountId\": 203106300,\n" + > > " \"name\": \"abdul7878\",\n" + > > " \"profileIconId\": 26,\n" + > > " \"level\": 3,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 401,\n" + > > " \"revisionDate\": 1406029148000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2467446,\n" + > > " \"accountId\": 200459837,\n" + > > " \"name\": \"ActionC\",\n" + > > " \"profileIconId\": 986,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 74,\n" + > > " \"infPoints\": 401367,\n" + > > " \"revisionDate\": 1480808608000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 9402979,\n" + > > " \"accountId\": 205698832,\n" + > > " \"name\": \"Ablenia \",\n" + > > " \"profileIconId\": 1129,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 19,\n" + > > " \"infPoints\": 163518,\n" + > > " \"revisionDate\": 1480687603000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 13187505,\n" + > > " \"accountId\": 207898213,\n" + > > " \"name\": \"aDaMiYiM\",\n" + > > " \"profileIconId\": 1301,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 116,\n" + > > " \"infPoints\": 45214,\n" + > > " \"revisionDate\": 1480793258000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 4141059,\n" + > > " \"accountId\": 201688290,\n" + > > " \"name\": \"Abimin?ar?\",\n" + > > " \"profileIconId\": 898,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 152,\n" + > > " \"infPoints\": 752477,\n" + > > " \"revisionDate\": 1480635961000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5702134,\n" + > > " \"accountId\": 202899395,\n" + > > " \"name\": \"Above the Clouds\",\n" + > > " \"profileIconId\": 684,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 110,\n" + > > " \"infPoints\": 288096,\n" + > > " \"revisionDate\": 1471011372000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5810740,\n" + > > " \"accountId\": 202985228,\n" + > > " \"name\": \"aBimm\",\n" + > > " \"profileIconId\": 11,\n" + > > " \"level\": 13,\n" + > > " \"expPoints\": 1180,\n" + > > " \"infPoints\": 10736,\n" + > > " \"revisionDate\": 1409832684000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5817751,\n" + > > " \"accountId\": 203050678,\n" + > > " \"name\": \"AD Glor?am\",\n" + > > " \"profileIconId\": 982,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 111,\n" + > > " \"infPoints\": 304658,\n" + > > " \"revisionDate\": 1480795250000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 9851802,\n" + > > " \"accountId\": 206011054,\n" + > > " \"name\": \"AdarAllame\",\n" + > > " \"profileIconId\": 911,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 48,\n" + > > " \"infPoints\": 73763,\n" + > > " \"revisionDate\": 1479422812000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 12735622,\n" + > > " \"accountId\": 207587019,\n" + > > " \"name\": \"absinthe666\",\n" + > > " \"profileIconId\": 903,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 83,\n" + > > " \"infPoints\": 40302,\n" + > > " \"revisionDate\": 1480782923000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 6371389,\n" + > > " \"accountId\": 203416952,\n" + > > " \"name\": \"adamsat?c?\",\n" + > > " \"profileIconId\": 3,\n" + > > " \"level\": 4,\n" + > > " \"expPoints\": 17,\n" + > > " \"infPoints\": 685,\n" + > > " \"revisionDate\": 1409320171000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 7828139,\n" + > > " \"accountId\": 204927980,\n" + > > " \"name\": \"AbsoluteForce\",\n" + > > " \"profileIconId\": 950,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 139,\n" + > > " \"infPoints\": 208789,\n" + > > " \"revisionDate\": 1480804396000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1373229,\n" + > > " \"accountId\": 1358441,\n" + > > " \"name\": \"AbsoluteDeath\",\n" + > > " \"profileIconId\": 7,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 34,\n" + > > " \"infPoints\": 223655,\n" + > > " \"revisionDate\": 1471867646000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 7694972,\n" + > > " \"accountId\": 204803668,\n" + > > " \"name\": \"ac pnp\",\n" + > > " \"profileIconId\": 937,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 161,\n" + > > " \"infPoints\": 249681,\n" + > > " \"revisionDate\": 1480801507000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1373524,\n" + > > " \"accountId\": 1350474,\n" + > > " \"name\": \"Abd??\",\n" + > > " \"profileIconId\": 1301,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 103,\n" + > > " \"infPoints\": 286803,\n" + > > " \"revisionDate\": 1476621827000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1650227,\n" + > > " \"accountId\": 200000503,\n" + > > " \"name\": \"AD Ambrosia\",\n" + > > " \"profileIconId\": 1152,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 139,\n" + > > " \"infPoints\": 156333,\n" + > > " \"revisionDate\": 1480805320000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8331358,\n" + > > " \"accountId\": 205073925,\n" + > > " \"name\": \"acarmanyust2\",\n" + > > " \"profileIconId\": 0,\n" + > > " \"level\": 2,\n" + > > " \"expPoints\": 43,\n" + > > " \"infPoints\": 318,\n" + > > " \"revisionDate\": 1423915139000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1862106,\n" + > > " \"accountId\": 200139838,\n" + > > " \"name\": \"aboU\",\n" + > > " \"profileIconId\": 1155,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 0,\n" + > > " \"infPoints\": 412616,\n" + > > " \"revisionDate\": 1480771055000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2362628,\n" + > > " \"accountId\": 685649,\n" + > > " \"name\": \"Ac?F?sT?k\",\n" + > > " \"profileIconId\": 1074,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 48,\n" + > > " \"infPoints\": 233882,\n" + > > " \"revisionDate\": 1480786233000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 4323909,\n" + > > " \"accountId\": 201917672,\n" + > > " \"name\": \"Addrenalin\",\n" + > > " \"profileIconId\": 603,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 55,\n" + > > " \"infPoints\": 220605,\n" + > > " \"revisionDate\": 1432647338000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 377206,\n" + > > " \"accountId\": 362106,\n" + > > " \"name\": \"Aburame Shino\",\n" + > > " \"profileIconId\": 844,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 84,\n" + > > " \"infPoints\": 354087,\n" + > > " \"revisionDate\": 1477666556000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5377433,\n" + > > " \"accountId\": 202697921,\n" + > > " \"name\": \"AcEcolton35\",\n" + > > " \"profileIconId\": 984,\n" + > > " \"level\": 25,\n" + > > " \"expPoints\": 751,\n" + > > " \"infPoints\": 30061,\n" + > > " \"revisionDate\": 1475503024000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2381404,\n" + > > " \"accountId\": 200333680,\n" + > > " \"name\": \"adafakaaa\",\n" + > > " \"profileIconId\": 663,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 8,\n" + > > " \"infPoints\": 534204,\n" + > > " \"revisionDate\": 1480719827000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1281203,\n" + > > " \"accountId\": 1259342,\n" + > > " \"name\": \"AC Klondike\",\n" + > > " \"profileIconId\": 898,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 27,\n" + > > " \"infPoints\": 191429,\n" + > > " \"revisionDate\": 1480294973000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 13161471,\n" + > > " \"accountId\": 207847181,\n" + > > " \"name\": \"adar21\",\n" + > > " \"profileIconId\": 26,\n" + > > " \"level\": 10,\n" + > > " \"expPoints\": 143,\n" + > > " \"infPoints\": 3558,\n" + > > " \"revisionDate\": 1476529855000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5841915,\n" + > > " \"accountId\": 202998794,\n" + > > " \"name\": \"Achilles29\",\n" + > > " \"profileIconId\": 666,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 41,\n" + > > " \"infPoints\": 219714,\n" + > > " \"revisionDate\": 1480777744000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2853062,\n" + > > " \"accountId\": 200726707,\n" + > > " \"name\": \"AbIanStarBebegim\",\n" + > > " \"profileIconId\": 909,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 64,\n" + > > " \"infPoints\": 297580,\n" + > > " \"revisionDate\": 1480556859000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8323114,\n" + > > " \"accountId\": 205093515,\n" + > > " \"name\": \"Absuruk\",\n" + > > " \"profileIconId\": 6,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 21,\n" + > > " \"infPoints\": 121086,\n" + > > " \"revisionDate\": 1480820801000\n" + > > " }\n" + > > "]" > > > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer < > hannes.wallnoefer at oracle.com> wrote: > > Hi Jesus, > > > > I?m trying to reproduce the problem, and just want to make sure I get > the missing pieces right. > > > > You already showed us how you?re setting up the engine and the JS code > you?re running. I assume the JSON code you?re parsing is a simple array of > objects? And you?re just calling Invocable.invokeFunction on the > ScriptEngine from multiple threads in parallel, right? > > > > Thanks, > > Hannes > > > > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > > > > > When we share one invocable across many threads and run invokeFunction > it > > > happens, such as this: > > > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > > >> > > >> Invocable invocable = generateInvocable(script); > > >> > > >> AtomicLong count = new AtomicLong(); > > >> > > >> for (int i = 0; i < 50; i++) { > > >> > > >> executor.submit(new Runnable() { > > >> > > >> @Override > > >> > > >> public void run() { > > >> > > >> try { > > >> > > >> while(true) { > > >> > > >> invocable.invokeFunction("transform", > > >>> something); > > >> > > >> count.incrementAndGet(); > > >> > > >> } > > >> > > >> } catch (NoSuchMethodException | > ScriptException > > >>> e) { > > >> > > >> e.printStackTrace(); > > >> > > >> } > > >> > > >> } > > >> > > >> }); > > >> > > >> } > > >> > > >> > > > > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) < > james.laskey at oracle.com > > >> wrote: > > > > > >> Intersting. The example you posted demonstrates this behaviour? If > so > > >> I?ll file a bug and dig in. It sounds like an object is being reused > > >> across invocations and accumulating changes to the property map. > > >> > > >> ? Jim > > >> > > >> > > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > > >> > > >> With more threads you are impacting the same 8 cores, so it will > taper off > > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > > >>> threshold depending on System performance. Transport: I meant if > you were > > >>> using sockets to provide the script. > > >> > > >> This makes sense. This one's on me then. > > >> > > >> > > >>> So you are using the same invocable instance for all threads? If so, > > >>> then you are probably good to go. As far as leaks are concerned, > not sure > > >>> how you would get leaks from Nashorn. The JSON object is written in > Java, > > >>> and little JavaScript involved. > > >> > > >> > > >> > > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > > >>> out of the loop and use the same invocable for all threads. > > >> > > >> > > >> We were using one invocable across all threads and we were getting > > >> slowdowns on execution, high CPU Usage and memory leaks that led to > > >> OutOfMemory errors. I could trace the leak to > > >> > > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > > >> > > >> which just keeps growing forever. > > >> > > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > > >> james.laskey at oracle.com> wrote: > > >> > > >>> > > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon > wrote: > > >>> > > >>> The cost of creating a new engine is significant. So share an engine > > >>>> across threads but use *eval > > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > > >>>> (String > > >>>> > > >>>> script, ScriptContext > > >>>> script/ScriptContext.html> > > >>>> context) instead, separate context per execution. If your > JavaScript > > >>>> code does not modify globals you can get away with using the same > engine, > > >>>> same compiled script on each thread. > > >>> > > >>> > > >>> I guess there's a few things here I don't understand. One thing I'm > > >>> trying to do is sharing a CompiledScript (which is why I'm using > > >>> invocable). Also, what exactly does modify globals mean? All our > filters do > > >>> the same thing, make a function that takes a JSON String, turns it > into a > > >>> JSON, modifies it and then stringifies it back. No state is changed > of > > >>> anything else but there are temporary vars created inside the scope > of the > > >>> function. When we run this multithreaded, running invokeFunction > slows down > > >>> significantly and we get crazy memory leaks. > > >>> > > >>> > > >>> So you are using the same invocable instance for all threads? If so, > > >>> then you are probably good to go. As far as leaks are concerned, > not sure > > >>> how you would get leaks from Nashorn. The JSON object is written in > Java, > > >>> and little JavaScript involved. > > >>> > > >>> > > >>> Of course there are many factors involved n performance. How many > cores > > >>>> do you have on the test machine? How much memory in the process? > What > > >>>> transport are you using between threads? That sort of thing. > Other than > > >>>> constructing then engine and context Nashorn performance should > scale. > > >>> > > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to > the > > >>> process. Not sure what transports between threads means, but this is > the > > >>> code I'm benchmarking with. Increasing the number of threads > actually makes > > >>> it go faster until about 4 threads, then adding more threads takes > the same > > >>> amount to get to 1000 and and after a certain point it is just > slower to > > >>> get to 1000 counts. Some of our filters need to be able to run over > 1000 > > >>> times a second (across all threads) and the fastest time I could > actually > > >>> get with this was about 2.4 seconds for a 1000 counts. > > >>> > > >>>> ExecutorService executor = Executors.newFixedThreadPool(50); > > >>>> > > >>>> AtomicLong count = new AtomicLong(); > > >>>> > > >>>> for (int i = 0; i < 50; i++) { > > >>>> > > >>>> executor.submit(new Runnable() { > > >>>> > > >>>> @Override > > >>>> > > >>>> public void run() { > > >>>> > > >>>> > > >>>>> try { > > >>>> > > >>>> Invocable invocable = > > >>>>> generateInvocable(script); > > >>>> > > >>>> while(true) { > > >>>> > > >>>> invocable.invokeFunction(" > transform", > > >>>>> something); > > >>>> > > >>>> count.incrementAndGet(); > > >>>> > > >>>> } > > >>>> > > >>>> } catch (NoSuchMethodException | > ScriptException > > >>>>> e) { > > >>>> > > >>>> e.printStackTrace(); > > >>>> > > >>>> } > > >>>> > > >>>> } > > >>>> > > >>>> }); > > >>>> > > >>>> } > > >>>> > > >>>> long lastTimestamp = System.currentTimeMillis(); > > >>>> > > >>>> while(true) { > > >>>> > > >>>> > > >>>>> if (count.get() > 1000) { > > >>>> > > >>>> count.getAndAdd(-1000); > > >>>> > > >>>> System.out.println((System.currentTimeMillis() - > > >>>>> lastTimestamp)/1000.0); > > >>>> > > >>>> lastTimestamp = System.currentTimeMillis(); > > >>>> > > >>>> } > > >>>> > > >>>> } > > >>>> > > >>>> > > >>> With more threads you are impacting the same 8 cores, so it will > taper > > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 > being a > > >>> threshold depending on System performance. Transport: I meant if > you were > > >>> using sockets to provide the script. > > >>> > > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > > >>> out of the loop and use the same invocable for all threads. > > >>> > > >>> - Jim > > >>> > > >>> > > >>> > > >>> > > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > > >>> james.laskey at oracle.com> wrote: > > >>> > > >>>> > > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon > wrote: > > >>>> > > >>>> Hey Jim, > > >>>> > > >>>> I looked at it and I will look into loadWithNewGlobal to see what > > >>>> exactly it does since it could be relevant. As for the rest, for my > use > > >>>> case having threads in the JS would not help. We're using Nashorn > to build > > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads > > >>>> processing a request to be able to execute the script to filter. > > >>>> > > >>>> > > >>>> The cost of creating a new engine is significant. So share an > engine > > >>>> across threads but use *eval > > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > > >>>> (String > > >>>> > > >>>> script, ScriptContext > > >>>> script/ScriptContext.html> > > >>>> context) instead, separate context per execution. If your > JavaScript > > >>>> code does not modify globals you can get away with using the same > engine, > > >>>> same compiled script on each thread. > > >>>> > > >>>> > > >>>> Also, when you say a new engine per threads is the worst case what > > >>>> exactly do you mean? I would expect an initial cost of compiling > the script > > >>>> on each thread and then each engine should be able to do its own > thing, but > > >>>> what I'm seeing is that when running with more than 10 threads all > my > > >>>> engines get slow at executing code, even though they are all > completely > > >>>> separate from each other. > > >>>> > > >>>> > > >>>> Of course there are many factors involved n performance. How many > cores > > >>>> do you have on the test machine? How much memory in the process? > What > > >>>> transport are you using between threads? That sort of thing. > Other than > > >>>> constructing then engine and context Nashorn performance should > scale. > > >>>> > > >>>> > > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > > >>>> james.laskey at oracle.com> wrote: > > >>>> > > >>>>> Jesus, > > >>>>> > > >>>>> Probably the most informative information is in this blog. > > >>>>> > > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_ > threading_and_mt > > >>>>> > > >>>>> This example uses Executors but threads would work as well. > > >>>>> > > >>>>> I did a talk that looked at different methods to max out > multithreading > > >>>>> performance. A new engine per thread is the worst case. A new > context per > > >>>>> thread does much better. A new global per thread is the best while > > >>>>> remaining thread safe. > > >>>>> > > >>>>> Cheers, > > >>>>> > > >>>>> ? Jim > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: > > >>>>> > > >>>>> Hey folks, > > >>>>> > > >>>>> I've tried many different ways of using Nashorn multithreaded > based on > > >>>>> what > > >>>>> I've found on the internet and I still can't get a single one to > scale. > > >>>>> Even the most naive method of making many script engines with my > script > > >>>>> tends to bottleneck itself when I have more than 10 threads > invoking > > >>>>> functions. > > >>>>> > > >>>>> I'm using the following code to compile my script and > > >>>>> invocable.invokeFunction("transform", input) to execute: > > >>>>> > > >>>>> static Invocable generateInvocable(String script) throws > > >>>>> ScriptException { > > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > > >>>>> ScriptEngine engine = > > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > > >>>>> Compilable compilable = (Compilable) engine; > > >>>>> final CompiledScript compiled = compilable.compile(script); > > >>>>> compiled.eval(); > > >>>>> return (Invocable) engine; > > >>>>> } > > >>>>> > > >>>>> > > >>>>> > > >>>>> The script I'm compiling is: > > >>>>> > > >>>>> String script = "function transform(input) {" + > > >>>>> "var result = JSON.parse(input);" + > > >>>>> "response = {};\n" + > > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > > >>>>> " var summoner = {};\n" + > > >>>>> " summoner.id = result[i].id;\n" + > > >>>>> " summoner.name = result[i].name;\n" + > > >>>>> " summoner.profileIconId = > > >>>>> result[i].profileIconId;\n" + > > >>>>> " summoner.revisionDate = > result[i].revisionDate;\n" + > > >>>>> " summoner.summonerLevel = result[i].level;\n" + > > >>>>> " response[summoner.id] = summoner;\n" + > > >>>>> "}\n" + > > >>>>> "result = response;" + > > >>>>> "return JSON.stringify(result);" + > > >>>>> "};"; > > >>>>> > > >>>>> > > >>>>> > > >>>>> I've also tried other more scaleable ways to work with scripts > > >>>>> concurrently, but given that this is the most naive method where > > >>>>> everything > > >>>>> is brand new and I still get slowness calling them concurrently I > fear > > >>>>> that > > >>>>> maybe I'm overlooking something extremely basic on my code. > > >>>>> > > >>>>> Thanks. > > >>>>> -Jesus Luzon > > >>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >> > > >> > > > > > > > > From jluzon at riotgames.com Fri Dec 9 03:47:16 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Thu, 8 Dec 2016 19:47:16 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: I came up with a workaround which is getting us through this but it definitely seems like this is a bug. The gist is I append an arbitrary string (we made it complex on our end to make it unlikely something else has this string inside) to the key and then remove all instances of said string on the JSON String resulting from calling stringify. We are leaving this for overnight testing but it showed promising results immediately. String script = "function transform(input) {" + > > "var result = JSON.parse(input);" + > > "response = {};\n" + > > "for (var i = 0; i < result.length; i++) {\n" + > > " var summoner = {};\n" + > > " summoner.id = result[i].id;\n" + > > " summoner.name = result[i].name;\n" + > > " summoner.profileIconId = result[i].profileIconId;\n" + > > " summoner.revisionDate = result[i].revisionDate;\n" + > > " summoner.summonerLevel = result[i].level;\n" + > > " response[\"someArbitraryString\" + summoner.id] = >> summoner;\n" + > > "}\n" + > > "var stringy = JSON.stringify(response);" + > > "result = stringy.replace(/someArbitraryString/g, \"\");" + > > "return result;" + > > "};"; > > On Thu, Dec 8, 2016 at 4:54 PM, Jesus Luzon wrote: > It doesn't seem like changing to a string fixed it. We dug deeper and > found a very interesting issue though. After running the invocable a few > thousand times, when we put a breakpoint in ArrayData.computerIteratorKeys() > and we found that it would stop at an execution from SparseArrayData with > an underlying array of size 923392, stored length of 461672 (this is one of > our IDs, the largest 6 digit one). Because of this we tried a run changing > all the IDs in the array we pass in to number from 1-38 and this thing went > blazing fast. When putting a break point in the same place, this array in > SpareArrayData was now of size 39. We then changed an id in the array we > take in to be 1337 and the size of the array in the SparseArrayData was > 1338. > > I don't understand why or how to prevent but it's using this ID as an > index for SpareArrayData underlying array. If someone can help me find a > workaround for this I would be extremely grateful. > > On Thu, Dec 8, 2016 at 3:52 AM, Hannes Walln?fer < > hannes.wallnoefer at oracle.com> wrote: > >> Yes, this is very likely to be the cause of the problem. However, I do >> think we should be able to handle sparse array data better, so it?s quite >> possible that you exposed some weak spot in our implementation. I?ll have a >> look into what?s going on. >> >> Hannes >> >> >> > Am 08.12.2016 um 00:40 schrieb Jesus Luzon : >> > >> > Looks like the memory leak is due to the way we wrote our code and how >> javascript works. I was expecting the line response[summoner.id] = >> summoner; to build a map but it turns out that if you use a number as the >> "key", javscript automatically fills the indexes in the middle with null >> (Undefined type?). When these IDs are very large, it is creating huge >> arrays that take longer to garbage collect than the code executing. I am >> about to start testing this on our end to make sure we see the improvements >> we expect. >> > >> > Does this idea seem like it is reasonably possible? >> > >> > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon >> wrote: >> > Yes, it's an array of objects which I'll paste. And yes, I'm just >> calling invokeFunction from many many different threads. I'm also going to >> go back and take a look at all the heap dumps we have to re-confirm what I >> mentioned. >> > >> > "[\n" + >> > " {\n" + >> > " \"id\": 6011511,\n" + >> > " \"accountId\": 203192481,\n" + >> > " \"name\": \"Adam Pro Qarda?\",\n" + >> > " \"profileIconId\": 25,\n" + >> > " \"level\": 5,\n" + >> > " \"expPoints\": 83,\n" + >> > " \"infPoints\": 1475,\n" + >> > " \"revisionDate\": 1406631727000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 2810674,\n" + >> > " \"accountId\": 200706913,\n" + >> > " \"name\": \"ABZ Devrim\",\n" + >> > " \"profileIconId\": 663,\n" + >> > " \"level\": 13,\n" + >> > " \"expPoints\": 982,\n" + >> > " \"infPoints\": 10472,\n" + >> > " \"revisionDate\": 1450791227000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5411195,\n" + >> > " \"accountId\": 202647689,\n" + >> > " \"name\": \"Ace HypcronN\",\n" + >> > " \"profileIconId\": 911,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 73,\n" + >> > " \"infPoints\": 182445,\n" + >> > " \"revisionDate\": 1480781650000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1363020,\n" + >> > " \"accountId\": 1357837,\n" + >> > " \"name\": \"AdanaLee\",\n" + >> > " \"profileIconId\": 502,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 125,\n" + >> > " \"infPoints\": 719299,\n" + >> > " \"revisionDate\": 1480530778000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 8261198,\n" + >> > " \"accountId\": 205027096,\n" + >> > " \"name\": \"Achilehuz\",\n" + >> > " \"profileIconId\": 1381,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 10,\n" + >> > " \"infPoints\": 158603,\n" + >> > " \"revisionDate\": 1480770307000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 12685857,\n" + >> > " \"accountId\": 207591166,\n" + >> > " \"name\": \"ac?mas?zpicc\",\n" + >> > " \"profileIconId\": 9,\n" + >> > " \"level\": 21,\n" + >> > " \"expPoints\": 840,\n" + >> > " \"infPoints\": 16659,\n" + >> > " \"revisionDate\": 1480515325000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 10860127,\n" + >> > " \"accountId\": 206507727,\n" + >> > " \"name\": \"AAngelFlyy\",\n" + >> > " \"profileIconId\": 1395,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 10,\n" + >> > " \"infPoints\": 73111,\n" + >> > " \"revisionDate\": 1480787870000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 3292376,\n" + >> > " \"accountId\": 201048714,\n" + >> > " \"name\": \"ACAB1907\",\n" + >> > " \"profileIconId\": 20,\n" + >> > " \"level\": 6,\n" + >> > " \"expPoints\": 305,\n" + >> > " \"infPoints\": 2107,\n" + >> > " \"revisionDate\": 1402448089000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 461671,\n" + >> > " \"accountId\": 446571,\n" + >> > " \"name\": \"Acta Est Fabul?\",\n" + >> > " \"profileIconId\": 1435,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 47,\n" + >> > " \"infPoints\": 644672,\n" + >> > " \"revisionDate\": 1480626505000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 394183,\n" + >> > " \"accountId\": 379083,\n" + >> > " \"name\": \"acekse4\",\n" + >> > " \"profileIconId\": 27,\n" + >> > " \"level\": 5,\n" + >> > " \"expPoints\": 223,\n" + >> > " \"infPoints\": 908,\n" + >> > " \"revisionDate\": 1348116544000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5941247,\n" + >> > " \"accountId\": 203106300,\n" + >> > " \"name\": \"abdul7878\",\n" + >> > " \"profileIconId\": 26,\n" + >> > " \"level\": 3,\n" + >> > " \"expPoints\": 10,\n" + >> > " \"infPoints\": 401,\n" + >> > " \"revisionDate\": 1406029148000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 2467446,\n" + >> > " \"accountId\": 200459837,\n" + >> > " \"name\": \"ActionC\",\n" + >> > " \"profileIconId\": 986,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 74,\n" + >> > " \"infPoints\": 401367,\n" + >> > " \"revisionDate\": 1480808608000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 9402979,\n" + >> > " \"accountId\": 205698832,\n" + >> > " \"name\": \"Ablenia \",\n" + >> > " \"profileIconId\": 1129,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 19,\n" + >> > " \"infPoints\": 163518,\n" + >> > " \"revisionDate\": 1480687603000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 13187505,\n" + >> > " \"accountId\": 207898213,\n" + >> > " \"name\": \"aDaMiYiM\",\n" + >> > " \"profileIconId\": 1301,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 116,\n" + >> > " \"infPoints\": 45214,\n" + >> > " \"revisionDate\": 1480793258000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 4141059,\n" + >> > " \"accountId\": 201688290,\n" + >> > " \"name\": \"Abimin?ar?\",\n" + >> > " \"profileIconId\": 898,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 152,\n" + >> > " \"infPoints\": 752477,\n" + >> > " \"revisionDate\": 1480635961000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5702134,\n" + >> > " \"accountId\": 202899395,\n" + >> > " \"name\": \"Above the Clouds\",\n" + >> > " \"profileIconId\": 684,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 110,\n" + >> > " \"infPoints\": 288096,\n" + >> > " \"revisionDate\": 1471011372000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5810740,\n" + >> > " \"accountId\": 202985228,\n" + >> > " \"name\": \"aBimm\",\n" + >> > " \"profileIconId\": 11,\n" + >> > " \"level\": 13,\n" + >> > " \"expPoints\": 1180,\n" + >> > " \"infPoints\": 10736,\n" + >> > " \"revisionDate\": 1409832684000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5817751,\n" + >> > " \"accountId\": 203050678,\n" + >> > " \"name\": \"AD Glor?am\",\n" + >> > " \"profileIconId\": 982,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 111,\n" + >> > " \"infPoints\": 304658,\n" + >> > " \"revisionDate\": 1480795250000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 9851802,\n" + >> > " \"accountId\": 206011054,\n" + >> > " \"name\": \"AdarAllame\",\n" + >> > " \"profileIconId\": 911,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 48,\n" + >> > " \"infPoints\": 73763,\n" + >> > " \"revisionDate\": 1479422812000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 12735622,\n" + >> > " \"accountId\": 207587019,\n" + >> > " \"name\": \"absinthe666\",\n" + >> > " \"profileIconId\": 903,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 83,\n" + >> > " \"infPoints\": 40302,\n" + >> > " \"revisionDate\": 1480782923000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 6371389,\n" + >> > " \"accountId\": 203416952,\n" + >> > " \"name\": \"adamsat?c?\",\n" + >> > " \"profileIconId\": 3,\n" + >> > " \"level\": 4,\n" + >> > " \"expPoints\": 17,\n" + >> > " \"infPoints\": 685,\n" + >> > " \"revisionDate\": 1409320171000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 7828139,\n" + >> > " \"accountId\": 204927980,\n" + >> > " \"name\": \"AbsoluteForce\",\n" + >> > " \"profileIconId\": 950,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 139,\n" + >> > " \"infPoints\": 208789,\n" + >> > " \"revisionDate\": 1480804396000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1373229,\n" + >> > " \"accountId\": 1358441,\n" + >> > " \"name\": \"AbsoluteDeath\",\n" + >> > " \"profileIconId\": 7,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 34,\n" + >> > " \"infPoints\": 223655,\n" + >> > " \"revisionDate\": 1471867646000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 7694972,\n" + >> > " \"accountId\": 204803668,\n" + >> > " \"name\": \"ac pnp\",\n" + >> > " \"profileIconId\": 937,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 161,\n" + >> > " \"infPoints\": 249681,\n" + >> > " \"revisionDate\": 1480801507000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1373524,\n" + >> > " \"accountId\": 1350474,\n" + >> > " \"name\": \"Abd??\",\n" + >> > " \"profileIconId\": 1301,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 103,\n" + >> > " \"infPoints\": 286803,\n" + >> > " \"revisionDate\": 1476621827000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1650227,\n" + >> > " \"accountId\": 200000503,\n" + >> > " \"name\": \"AD Ambrosia\",\n" + >> > " \"profileIconId\": 1152,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 139,\n" + >> > " \"infPoints\": 156333,\n" + >> > " \"revisionDate\": 1480805320000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 8331358,\n" + >> > " \"accountId\": 205073925,\n" + >> > " \"name\": \"acarmanyust2\",\n" + >> > " \"profileIconId\": 0,\n" + >> > " \"level\": 2,\n" + >> > " \"expPoints\": 43,\n" + >> > " \"infPoints\": 318,\n" + >> > " \"revisionDate\": 1423915139000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1862106,\n" + >> > " \"accountId\": 200139838,\n" + >> > " \"name\": \"aboU\",\n" + >> > " \"profileIconId\": 1155,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 0,\n" + >> > " \"infPoints\": 412616,\n" + >> > " \"revisionDate\": 1480771055000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 2362628,\n" + >> > " \"accountId\": 685649,\n" + >> > " \"name\": \"Ac?F?sT?k\",\n" + >> > " \"profileIconId\": 1074,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 48,\n" + >> > " \"infPoints\": 233882,\n" + >> > " \"revisionDate\": 1480786233000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 4323909,\n" + >> > " \"accountId\": 201917672,\n" + >> > " \"name\": \"Addrenalin\",\n" + >> > " \"profileIconId\": 603,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 55,\n" + >> > " \"infPoints\": 220605,\n" + >> > " \"revisionDate\": 1432647338000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 377206,\n" + >> > " \"accountId\": 362106,\n" + >> > " \"name\": \"Aburame Shino\",\n" + >> > " \"profileIconId\": 844,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 84,\n" + >> > " \"infPoints\": 354087,\n" + >> > " \"revisionDate\": 1477666556000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5377433,\n" + >> > " \"accountId\": 202697921,\n" + >> > " \"name\": \"AcEcolton35\",\n" + >> > " \"profileIconId\": 984,\n" + >> > " \"level\": 25,\n" + >> > " \"expPoints\": 751,\n" + >> > " \"infPoints\": 30061,\n" + >> > " \"revisionDate\": 1475503024000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 2381404,\n" + >> > " \"accountId\": 200333680,\n" + >> > " \"name\": \"adafakaaa\",\n" + >> > " \"profileIconId\": 663,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 8,\n" + >> > " \"infPoints\": 534204,\n" + >> > " \"revisionDate\": 1480719827000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 1281203,\n" + >> > " \"accountId\": 1259342,\n" + >> > " \"name\": \"AC Klondike\",\n" + >> > " \"profileIconId\": 898,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 27,\n" + >> > " \"infPoints\": 191429,\n" + >> > " \"revisionDate\": 1480294973000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 13161471,\n" + >> > " \"accountId\": 207847181,\n" + >> > " \"name\": \"adar21\",\n" + >> > " \"profileIconId\": 26,\n" + >> > " \"level\": 10,\n" + >> > " \"expPoints\": 143,\n" + >> > " \"infPoints\": 3558,\n" + >> > " \"revisionDate\": 1476529855000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 5841915,\n" + >> > " \"accountId\": 202998794,\n" + >> > " \"name\": \"Achilles29\",\n" + >> > " \"profileIconId\": 666,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 41,\n" + >> > " \"infPoints\": 219714,\n" + >> > " \"revisionDate\": 1480777744000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 2853062,\n" + >> > " \"accountId\": 200726707,\n" + >> > " \"name\": \"AbIanStarBebegim\",\n" + >> > " \"profileIconId\": 909,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 64,\n" + >> > " \"infPoints\": 297580,\n" + >> > " \"revisionDate\": 1480556859000\n" + >> > " },\n" + >> > " {\n" + >> > " \"id\": 8323114,\n" + >> > " \"accountId\": 205093515,\n" + >> > " \"name\": \"Absuruk\",\n" + >> > " \"profileIconId\": 6,\n" + >> > " \"level\": 30,\n" + >> > " \"expPoints\": 21,\n" + >> > " \"infPoints\": 121086,\n" + >> > " \"revisionDate\": 1480820801000\n" + >> > " }\n" + >> > "]" >> > >> > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer < >> hannes.wallnoefer at oracle.com> wrote: >> > Hi Jesus, >> > >> > I?m trying to reproduce the problem, and just want to make sure I get >> the missing pieces right. >> > >> > You already showed us how you?re setting up the engine and the JS code >> you?re running. I assume the JSON code you?re parsing is a simple array of >> objects? And you?re just calling Invocable.invokeFunction on the >> ScriptEngine from multiple threads in parallel, right? >> > >> > Thanks, >> > Hannes >> > >> > >> > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : >> > > >> > > When we share one invocable across many threads and run >> invokeFunction it >> > > happens, such as this: >> > > >> > > ExecutorService executor = Executors.newFixedThreadPool(50); >> > >> >> > >> Invocable invocable = generateInvocable(script); >> > >> >> > >> AtomicLong count = new AtomicLong(); >> > >> >> > >> for (int i = 0; i < 50; i++) { >> > >> >> > >> executor.submit(new Runnable() { >> > >> >> > >> @Override >> > >> >> > >> public void run() { >> > >> >> > >> try { >> > >> >> > >> while(true) { >> > >> >> > >> invocable.invokeFunction("transform", >> > >>> something); >> > >> >> > >> count.incrementAndGet(); >> > >> >> > >> } >> > >> >> > >> } catch (NoSuchMethodException | >> ScriptException >> > >>> e) { >> > >> >> > >> e.printStackTrace(); >> > >> >> > >> } >> > >> >> > >> } >> > >> >> > >> }); >> > >> >> > >> } >> > >> >> > >> >> > > >> > > >> > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) < >> james.laskey at oracle.com >> > >> wrote: >> > > >> > >> Intersting. The example you posted demonstrates this behaviour? If >> so >> > >> I?ll file a bug and dig in. It sounds like an object is being reused >> > >> across invocations and accumulating changes to the property map. >> > >> >> > >> ? Jim >> > >> >> > >> >> > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon >> wrote: >> > >> >> > >> With more threads you are impacting the same 8 cores, so it will >> taper off >> > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being >> a >> > >>> threshold depending on System performance. Transport: I meant if >> you were >> > >>> using sockets to provide the script. >> > >> >> > >> This makes sense. This one's on me then. >> > >> >> > >> >> > >>> So you are using the same invocable instance for all threads? If >> so, >> > >>> then you are probably good to go. As far as leaks are concerned, >> not sure >> > >>> how you would get leaks from Nashorn. The JSON object is written >> in Java, >> > >>> and little JavaScript involved. >> > >> >> > >> >> > >> >> > >>> In your example, pull up Invocable invocable = >> generateInvocable(script); >> > >>> out of the loop and use the same invocable for all threads. >> > >> >> > >> >> > >> We were using one invocable across all threads and we were getting >> > >> slowdowns on execution, high CPU Usage and memory leaks that led to >> > >> OutOfMemory errors. I could trace the leak to >> > >> >> > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> >> > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* >> > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* >> > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter >> > >> >> > >> which just keeps growing forever. >> > >> >> > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < >> > >> james.laskey at oracle.com> wrote: >> > >> >> > >>> >> > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon >> wrote: >> > >>> >> > >>> The cost of creating a new engine is significant. So share an >> engine >> > >>>> across threads but use *eval >> > >>>> > ScriptEngine.html#eval(java.lang.String,%20javax.script.ScriptContext)>* >> > >>>> (String >> > >>>> >> > >>>> script, ScriptContext >> > >>>> > ScriptContext.html> >> > >>>> context) instead, separate context per execution. If your >> JavaScript >> > >>>> code does not modify globals you can get away with using the same >> engine, >> > >>>> same compiled script on each thread. >> > >>> >> > >>> >> > >>> I guess there's a few things here I don't understand. One thing I'm >> > >>> trying to do is sharing a CompiledScript (which is why I'm using >> > >>> invocable). Also, what exactly does modify globals mean? All our >> filters do >> > >>> the same thing, make a function that takes a JSON String, turns it >> into a >> > >>> JSON, modifies it and then stringifies it back. No state is changed >> of >> > >>> anything else but there are temporary vars created inside the scope >> of the >> > >>> function. When we run this multithreaded, running invokeFunction >> slows down >> > >>> significantly and we get crazy memory leaks. >> > >>> >> > >>> >> > >>> So you are using the same invocable instance for all threads? If >> so, >> > >>> then you are probably good to go. As far as leaks are concerned, >> not sure >> > >>> how you would get leaks from Nashorn. The JSON object is written >> in Java, >> > >>> and little JavaScript involved. >> > >>> >> > >>> >> > >>> Of course there are many factors involved n performance. How many >> cores >> > >>>> do you have on the test machine? How much memory in the process? >> What >> > >>>> transport are you using between threads? That sort of thing. >> Other than >> > >>>> constructing then engine and context Nashorn performance should >> scale. >> > >>> >> > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to >> the >> > >>> process. Not sure what transports between threads means, but this >> is the >> > >>> code I'm benchmarking with. Increasing the number of threads >> actually makes >> > >>> it go faster until about 4 threads, then adding more threads takes >> the same >> > >>> amount to get to 1000 and and after a certain point it is just >> slower to >> > >>> get to 1000 counts. Some of our filters need to be able to run over >> 1000 >> > >>> times a second (across all threads) and the fastest time I could >> actually >> > >>> get with this was about 2.4 seconds for a 1000 counts. >> > >>> >> > >>>> ExecutorService executor = Executors.newFixedThreadPool(5 >> 0); >> > >>>> >> > >>>> AtomicLong count = new AtomicLong(); >> > >>>> >> > >>>> for (int i = 0; i < 50; i++) { >> > >>>> >> > >>>> executor.submit(new Runnable() { >> > >>>> >> > >>>> @Override >> > >>>> >> > >>>> public void run() { >> > >>>> >> > >>>> >> > >>>>> try { >> > >>>> >> > >>>> Invocable invocable = >> > >>>>> generateInvocable(script); >> > >>>> >> > >>>> while(true) { >> > >>>> >> > >>>> invocable.invokeFunction("tran >> sform", >> > >>>>> something); >> > >>>> >> > >>>> count.incrementAndGet(); >> > >>>> >> > >>>> } >> > >>>> >> > >>>> } catch (NoSuchMethodException | >> ScriptException >> > >>>>> e) { >> > >>>> >> > >>>> e.printStackTrace(); >> > >>>> >> > >>>> } >> > >>>> >> > >>>> } >> > >>>> >> > >>>> }); >> > >>>> >> > >>>> } >> > >>>> >> > >>>> long lastTimestamp = System.currentTimeMillis(); >> > >>>> >> > >>>> while(true) { >> > >>>> >> > >>>> >> > >>>>> if (count.get() > 1000) { >> > >>>> >> > >>>> count.getAndAdd(-1000); >> > >>>> >> > >>>> System.out.println((System.currentTimeMillis() - >> > >>>>> lastTimestamp)/1000.0); >> > >>>> >> > >>>> lastTimestamp = System.currentTimeMillis(); >> > >>>> >> > >>>> } >> > >>>> >> > >>>> } >> > >>>> >> > >>>> >> > >>> With more threads you are impacting the same 8 cores, so it will >> taper >> > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 >> being a >> > >>> threshold depending on System performance. Transport: I meant if >> you were >> > >>> using sockets to provide the script. >> > >>> >> > >>> In your example, pull up Invocable invocable = >> generateInvocable(script); >> > >>> out of the loop and use the same invocable for all threads. >> > >>> >> > >>> - Jim >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < >> > >>> james.laskey at oracle.com> wrote: >> > >>> >> > >>>> >> > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon >> wrote: >> > >>>> >> > >>>> Hey Jim, >> > >>>> >> > >>>> I looked at it and I will look into loadWithNewGlobal to see what >> > >>>> exactly it does since it could be relevant. As for the rest, for >> my use >> > >>>> case having threads in the JS would not help. We're using Nashorn >> to build >> > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads >> > >>>> processing a request to be able to execute the script to filter. >> > >>>> >> > >>>> >> > >>>> The cost of creating a new engine is significant. So share an >> engine >> > >>>> across threads but use *eval >> > >>>> > ScriptEngine.html#eval(java.lang.String,%20javax.script.ScriptContext)>* >> > >>>> (String >> > >>>> >> > >>>> script, ScriptContext >> > >>>> > ScriptContext.html> >> > >>>> context) instead, separate context per execution. If your >> JavaScript >> > >>>> code does not modify globals you can get away with using the same >> engine, >> > >>>> same compiled script on each thread. >> > >>>> >> > >>>> >> > >>>> Also, when you say a new engine per threads is the worst case what >> > >>>> exactly do you mean? I would expect an initial cost of compiling >> the script >> > >>>> on each thread and then each engine should be able to do its own >> thing, but >> > >>>> what I'm seeing is that when running with more than 10 threads all >> my >> > >>>> engines get slow at executing code, even though they are all >> completely >> > >>>> separate from each other. >> > >>>> >> > >>>> >> > >>>> Of course there are many factors involved n performance. How many >> cores >> > >>>> do you have on the test machine? How much memory in the process? >> What >> > >>>> transport are you using between threads? That sort of thing. >> Other than >> > >>>> constructing then engine and context Nashorn performance should >> scale. >> > >>>> >> > >>>> >> > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < >> > >>>> james.laskey at oracle.com> wrote: >> > >>>> >> > >>>>> Jesus, >> > >>>>> >> > >>>>> Probably the most informative information is in this blog. >> > >>>>> >> > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threadi >> ng_and_mt >> > >>>>> >> > >>>>> This example uses Executors but threads would work as well. >> > >>>>> >> > >>>>> I did a talk that looked at different methods to max out >> multithreading >> > >>>>> performance. A new engine per thread is the worst case. A new >> context per >> > >>>>> thread does much better. A new global per thread is the best >> while >> > >>>>> remaining thread safe. >> > >>>>> >> > >>>>> Cheers, >> > >>>>> >> > >>>>> ? Jim >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon >> wrote: >> > >>>>> >> > >>>>> Hey folks, >> > >>>>> >> > >>>>> I've tried many different ways of using Nashorn multithreaded >> based on >> > >>>>> what >> > >>>>> I've found on the internet and I still can't get a single one to >> scale. >> > >>>>> Even the most naive method of making many script engines with my >> script >> > >>>>> tends to bottleneck itself when I have more than 10 threads >> invoking >> > >>>>> functions. >> > >>>>> >> > >>>>> I'm using the following code to compile my script and >> > >>>>> invocable.invokeFunction("transform", input) to execute: >> > >>>>> >> > >>>>> static Invocable generateInvocable(String script) throws >> > >>>>> ScriptException { >> > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); >> > >>>>> ScriptEngine engine = >> > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); >> > >>>>> Compilable compilable = (Compilable) engine; >> > >>>>> final CompiledScript compiled = compilable.compile(script); >> > >>>>> compiled.eval(); >> > >>>>> return (Invocable) engine; >> > >>>>> } >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> The script I'm compiling is: >> > >>>>> >> > >>>>> String script = "function transform(input) {" + >> > >>>>> "var result = JSON.parse(input);" + >> > >>>>> "response = {};\n" + >> > >>>>> "for (var i = 0; i < result.length; i++) {\n" + >> > >>>>> " var summoner = {};\n" + >> > >>>>> " summoner.id = result[i].id;\n" + >> > >>>>> " summoner.name = result[i].name;\n" + >> > >>>>> " summoner.profileIconId = >> > >>>>> result[i].profileIconId;\n" + >> > >>>>> " summoner.revisionDate = >> result[i].revisionDate;\n" + >> > >>>>> " summoner.summonerLevel = result[i].level;\n" + >> > >>>>> " response[summoner.id] = summoner;\n" + >> > >>>>> "}\n" + >> > >>>>> "result = response;" + >> > >>>>> "return JSON.stringify(result);" + >> > >>>>> "};"; >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> I've also tried other more scaleable ways to work with scripts >> > >>>>> concurrently, but given that this is the most naive method where >> > >>>>> everything >> > >>>>> is brand new and I still get slowness calling them concurrently I >> fear >> > >>>>> that >> > >>>>> maybe I'm overlooking something extremely basic on my code. >> > >>>>> >> > >>>>> Thanks. >> > >>>>> -Jesus Luzon >> > >>>>> >> > >>>>> >> > >>>> >> > >>> >> > >>> >> > >> >> > >> >> > >> > >> > >> >> > From hannes.wallnoefer at oracle.com Fri Dec 9 10:37:07 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Fri, 9 Dec 2016 11:37:07 +0100 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Jesus, I looked at this issue more in depth and there are two problems here, one on your side and one on ours: The problem with your code is that you left out the ?var? declaration of the ?response? variable in the third line of your function. This means that ?response? is allocated as a global variable outside the scope of the function. When you run this in parallel with multiple threads you actually access the same object from multiple threads, meaning your output will be corrupt. Because of things like this, it is actually a good idea to run JavaScript in strict mode (e.g. by using ?use strict? directive in your JavaScript code, or by creating the engine with -strict option). Another solution would be to use multiple engines or bindings, which should not have a big performance impact once the other problem is fixed. Which leads us to the problem on our side, which is performance of sparse arrays. As you have noticed, using something that is not a valid array index greatly improves performance, and that is because some array element assignment are implemented very inefficiently in Nashorn. I?ve filed a bug for this: https://bugs.openjdk.java.net/browse/JDK-8170977 Once this bug is fixed, using numeric keys should be roughly as fast as using non-numeric ones. Hannes > Am 09.12.2016 um 04:47 schrieb Jesus Luzon : > > I came up with a workaround which is getting us through this but it definitely seems like this is a bug. The gist is I append an arbitrary string (we made it complex on our end to make it unlikely something else has this string inside) to the key and then remove all instances of said string on the JSON String resulting from calling stringify. We are leaving this for overnight testing but it showed promising results immediately. > > String script = "function transform(input) {" + > "var result = JSON.parse(input);" + > "response = {};\n" + > "for (var i = 0; i < result.length; i++) {\n" + > " var summoner = {};\n" + > " summoner.id = result[i].id;\n" + > " summoner.name = result[i].name;\n" + > " summoner.profileIconId = result[i].profileIconId;\n" + > " summoner.revisionDate = result[i].revisionDate;\n" + > " summoner.summonerLevel = result[i].level;\n" + > " response[\"someArbitraryString\" + summoner.id] = summoner;\n" + > "}\n" + > "var stringy = JSON.stringify(response);" + > "result = stringy.replace(/someArbitraryString/g, \"\");" + > "return result;" + > "};"; > > On Thu, Dec 8, 2016 at 4:54 PM, Jesus Luzon wrote: > It doesn't seem like changing to a string fixed it. We dug deeper and found a very interesting issue though. After running the invocable a few thousand times, when we put a breakpoint in ArrayData.computerIteratorKeys() and we found that it would stop at an execution from SparseArrayData with an underlying array of size 923392, stored length of 461672 (this is one of our IDs, the largest 6 digit one). Because of this we tried a run changing all the IDs in the array we pass in to number from 1-38 and this thing went blazing fast. When putting a break point in the same place, this array in SpareArrayData was now of size 39. We then changed an id in the array we take in to be 1337 and the size of the array in the SparseArrayData was 1338. > > I don't understand why or how to prevent but it's using this ID as an index for SpareArrayData underlying array. If someone can help me find a workaround for this I would be extremely grateful. > > On Thu, Dec 8, 2016 at 3:52 AM, Hannes Walln?fer wrote: > Yes, this is very likely to be the cause of the problem. However, I do think we should be able to handle sparse array data better, so it?s quite possible that you exposed some weak spot in our implementation. I?ll have a look into what?s going on. > > Hannes > > > > Am 08.12.2016 um 00:40 schrieb Jesus Luzon : > > > > Looks like the memory leak is due to the way we wrote our code and how javascript works. I was expecting the line response[summoner.id] = summoner; to build a map but it turns out that if you use a number as the "key", javscript automatically fills the indexes in the middle with null (Undefined type?). When these IDs are very large, it is creating huge arrays that take longer to garbage collect than the code executing. I am about to start testing this on our end to make sure we see the improvements we expect. > > > > Does this idea seem like it is reasonably possible? > > > > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon wrote: > > Yes, it's an array of objects which I'll paste. And yes, I'm just calling invokeFunction from many many different threads. I'm also going to go back and take a look at all the heap dumps we have to re-confirm what I mentioned. > > > > "[\n" + > > " {\n" + > > " \"id\": 6011511,\n" + > > " \"accountId\": 203192481,\n" + > > " \"name\": \"Adam Pro Qarda?\",\n" + > > " \"profileIconId\": 25,\n" + > > " \"level\": 5,\n" + > > " \"expPoints\": 83,\n" + > > " \"infPoints\": 1475,\n" + > > " \"revisionDate\": 1406631727000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2810674,\n" + > > " \"accountId\": 200706913,\n" + > > " \"name\": \"ABZ Devrim\",\n" + > > " \"profileIconId\": 663,\n" + > > " \"level\": 13,\n" + > > " \"expPoints\": 982,\n" + > > " \"infPoints\": 10472,\n" + > > " \"revisionDate\": 1450791227000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5411195,\n" + > > " \"accountId\": 202647689,\n" + > > " \"name\": \"Ace HypcronN\",\n" + > > " \"profileIconId\": 911,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 73,\n" + > > " \"infPoints\": 182445,\n" + > > " \"revisionDate\": 1480781650000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1363020,\n" + > > " \"accountId\": 1357837,\n" + > > " \"name\": \"AdanaLee\",\n" + > > " \"profileIconId\": 502,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 125,\n" + > > " \"infPoints\": 719299,\n" + > > " \"revisionDate\": 1480530778000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8261198,\n" + > > " \"accountId\": 205027096,\n" + > > " \"name\": \"Achilehuz\",\n" + > > " \"profileIconId\": 1381,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 158603,\n" + > > " \"revisionDate\": 1480770307000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 12685857,\n" + > > " \"accountId\": 207591166,\n" + > > " \"name\": \"ac?mas?zpicc\",\n" + > > " \"profileIconId\": 9,\n" + > > " \"level\": 21,\n" + > > " \"expPoints\": 840,\n" + > > " \"infPoints\": 16659,\n" + > > " \"revisionDate\": 1480515325000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 10860127,\n" + > > " \"accountId\": 206507727,\n" + > > " \"name\": \"AAngelFlyy\",\n" + > > " \"profileIconId\": 1395,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 73111,\n" + > > " \"revisionDate\": 1480787870000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 3292376,\n" + > > " \"accountId\": 201048714,\n" + > > " \"name\": \"ACAB1907\",\n" + > > " \"profileIconId\": 20,\n" + > > " \"level\": 6,\n" + > > " \"expPoints\": 305,\n" + > > " \"infPoints\": 2107,\n" + > > " \"revisionDate\": 1402448089000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 461671,\n" + > > " \"accountId\": 446571,\n" + > > " \"name\": \"Acta Est Fabul?\",\n" + > > " \"profileIconId\": 1435,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 47,\n" + > > " \"infPoints\": 644672,\n" + > > " \"revisionDate\": 1480626505000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 394183,\n" + > > " \"accountId\": 379083,\n" + > > " \"name\": \"acekse4\",\n" + > > " \"profileIconId\": 27,\n" + > > " \"level\": 5,\n" + > > " \"expPoints\": 223,\n" + > > " \"infPoints\": 908,\n" + > > " \"revisionDate\": 1348116544000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5941247,\n" + > > " \"accountId\": 203106300,\n" + > > " \"name\": \"abdul7878\",\n" + > > " \"profileIconId\": 26,\n" + > > " \"level\": 3,\n" + > > " \"expPoints\": 10,\n" + > > " \"infPoints\": 401,\n" + > > " \"revisionDate\": 1406029148000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2467446,\n" + > > " \"accountId\": 200459837,\n" + > > " \"name\": \"ActionC\",\n" + > > " \"profileIconId\": 986,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 74,\n" + > > " \"infPoints\": 401367,\n" + > > " \"revisionDate\": 1480808608000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 9402979,\n" + > > " \"accountId\": 205698832,\n" + > > " \"name\": \"Ablenia \",\n" + > > " \"profileIconId\": 1129,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 19,\n" + > > " \"infPoints\": 163518,\n" + > > " \"revisionDate\": 1480687603000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 13187505,\n" + > > " \"accountId\": 207898213,\n" + > > " \"name\": \"aDaMiYiM\",\n" + > > " \"profileIconId\": 1301,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 116,\n" + > > " \"infPoints\": 45214,\n" + > > " \"revisionDate\": 1480793258000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 4141059,\n" + > > " \"accountId\": 201688290,\n" + > > " \"name\": \"Abimin?ar?\",\n" + > > " \"profileIconId\": 898,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 152,\n" + > > " \"infPoints\": 752477,\n" + > > " \"revisionDate\": 1480635961000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5702134,\n" + > > " \"accountId\": 202899395,\n" + > > " \"name\": \"Above the Clouds\",\n" + > > " \"profileIconId\": 684,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 110,\n" + > > " \"infPoints\": 288096,\n" + > > " \"revisionDate\": 1471011372000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5810740,\n" + > > " \"accountId\": 202985228,\n" + > > " \"name\": \"aBimm\",\n" + > > " \"profileIconId\": 11,\n" + > > " \"level\": 13,\n" + > > " \"expPoints\": 1180,\n" + > > " \"infPoints\": 10736,\n" + > > " \"revisionDate\": 1409832684000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5817751,\n" + > > " \"accountId\": 203050678,\n" + > > " \"name\": \"AD Glor?am\",\n" + > > " \"profileIconId\": 982,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 111,\n" + > > " \"infPoints\": 304658,\n" + > > " \"revisionDate\": 1480795250000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 9851802,\n" + > > " \"accountId\": 206011054,\n" + > > " \"name\": \"AdarAllame\",\n" + > > " \"profileIconId\": 911,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 48,\n" + > > " \"infPoints\": 73763,\n" + > > " \"revisionDate\": 1479422812000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 12735622,\n" + > > " \"accountId\": 207587019,\n" + > > " \"name\": \"absinthe666\",\n" + > > " \"profileIconId\": 903,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 83,\n" + > > " \"infPoints\": 40302,\n" + > > " \"revisionDate\": 1480782923000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 6371389,\n" + > > " \"accountId\": 203416952,\n" + > > " \"name\": \"adamsat?c?\",\n" + > > " \"profileIconId\": 3,\n" + > > " \"level\": 4,\n" + > > " \"expPoints\": 17,\n" + > > " \"infPoints\": 685,\n" + > > " \"revisionDate\": 1409320171000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 7828139,\n" + > > " \"accountId\": 204927980,\n" + > > " \"name\": \"AbsoluteForce\",\n" + > > " \"profileIconId\": 950,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 139,\n" + > > " \"infPoints\": 208789,\n" + > > " \"revisionDate\": 1480804396000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1373229,\n" + > > " \"accountId\": 1358441,\n" + > > " \"name\": \"AbsoluteDeath\",\n" + > > " \"profileIconId\": 7,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 34,\n" + > > " \"infPoints\": 223655,\n" + > > " \"revisionDate\": 1471867646000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 7694972,\n" + > > " \"accountId\": 204803668,\n" + > > " \"name\": \"ac pnp\",\n" + > > " \"profileIconId\": 937,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 161,\n" + > > " \"infPoints\": 249681,\n" + > > " \"revisionDate\": 1480801507000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1373524,\n" + > > " \"accountId\": 1350474,\n" + > > " \"name\": \"Abd??\",\n" + > > " \"profileIconId\": 1301,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 103,\n" + > > " \"infPoints\": 286803,\n" + > > " \"revisionDate\": 1476621827000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1650227,\n" + > > " \"accountId\": 200000503,\n" + > > " \"name\": \"AD Ambrosia\",\n" + > > " \"profileIconId\": 1152,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 139,\n" + > > " \"infPoints\": 156333,\n" + > > " \"revisionDate\": 1480805320000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8331358,\n" + > > " \"accountId\": 205073925,\n" + > > " \"name\": \"acarmanyust2\",\n" + > > " \"profileIconId\": 0,\n" + > > " \"level\": 2,\n" + > > " \"expPoints\": 43,\n" + > > " \"infPoints\": 318,\n" + > > " \"revisionDate\": 1423915139000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1862106,\n" + > > " \"accountId\": 200139838,\n" + > > " \"name\": \"aboU\",\n" + > > " \"profileIconId\": 1155,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 0,\n" + > > " \"infPoints\": 412616,\n" + > > " \"revisionDate\": 1480771055000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2362628,\n" + > > " \"accountId\": 685649,\n" + > > " \"name\": \"Ac?F?sT?k\",\n" + > > " \"profileIconId\": 1074,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 48,\n" + > > " \"infPoints\": 233882,\n" + > > " \"revisionDate\": 1480786233000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 4323909,\n" + > > " \"accountId\": 201917672,\n" + > > " \"name\": \"Addrenalin\",\n" + > > " \"profileIconId\": 603,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 55,\n" + > > " \"infPoints\": 220605,\n" + > > " \"revisionDate\": 1432647338000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 377206,\n" + > > " \"accountId\": 362106,\n" + > > " \"name\": \"Aburame Shino\",\n" + > > " \"profileIconId\": 844,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 84,\n" + > > " \"infPoints\": 354087,\n" + > > " \"revisionDate\": 1477666556000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5377433,\n" + > > " \"accountId\": 202697921,\n" + > > " \"name\": \"AcEcolton35\",\n" + > > " \"profileIconId\": 984,\n" + > > " \"level\": 25,\n" + > > " \"expPoints\": 751,\n" + > > " \"infPoints\": 30061,\n" + > > " \"revisionDate\": 1475503024000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2381404,\n" + > > " \"accountId\": 200333680,\n" + > > " \"name\": \"adafakaaa\",\n" + > > " \"profileIconId\": 663,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 8,\n" + > > " \"infPoints\": 534204,\n" + > > " \"revisionDate\": 1480719827000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 1281203,\n" + > > " \"accountId\": 1259342,\n" + > > " \"name\": \"AC Klondike\",\n" + > > " \"profileIconId\": 898,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 27,\n" + > > " \"infPoints\": 191429,\n" + > > " \"revisionDate\": 1480294973000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 13161471,\n" + > > " \"accountId\": 207847181,\n" + > > " \"name\": \"adar21\",\n" + > > " \"profileIconId\": 26,\n" + > > " \"level\": 10,\n" + > > " \"expPoints\": 143,\n" + > > " \"infPoints\": 3558,\n" + > > " \"revisionDate\": 1476529855000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 5841915,\n" + > > " \"accountId\": 202998794,\n" + > > " \"name\": \"Achilles29\",\n" + > > " \"profileIconId\": 666,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 41,\n" + > > " \"infPoints\": 219714,\n" + > > " \"revisionDate\": 1480777744000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 2853062,\n" + > > " \"accountId\": 200726707,\n" + > > " \"name\": \"AbIanStarBebegim\",\n" + > > " \"profileIconId\": 909,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 64,\n" + > > " \"infPoints\": 297580,\n" + > > " \"revisionDate\": 1480556859000\n" + > > " },\n" + > > " {\n" + > > " \"id\": 8323114,\n" + > > " \"accountId\": 205093515,\n" + > > " \"name\": \"Absuruk\",\n" + > > " \"profileIconId\": 6,\n" + > > " \"level\": 30,\n" + > > " \"expPoints\": 21,\n" + > > " \"infPoints\": 121086,\n" + > > " \"revisionDate\": 1480820801000\n" + > > " }\n" + > > "]" > > > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer wrote: > > Hi Jesus, > > > > I?m trying to reproduce the problem, and just want to make sure I get the missing pieces right. > > > > You already showed us how you?re setting up the engine and the JS code you?re running. I assume the JSON code you?re parsing is a simple array of objects? And you?re just calling Invocable.invokeFunction on the ScriptEngine from multiple threads in parallel, right? > > > > Thanks, > > Hannes > > > > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > > > > > When we share one invocable across many threads and run invokeFunction it > > > happens, such as this: > > > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > > >> > > >> Invocable invocable = generateInvocable(script); > > >> > > >> AtomicLong count = new AtomicLong(); > > >> > > >> for (int i = 0; i < 50; i++) { > > >> > > >> executor.submit(new Runnable() { > > >> > > >> @Override > > >> > > >> public void run() { > > >> > > >> try { > > >> > > >> while(true) { > > >> > > >> invocable.invokeFunction("transform", > > >>> something); > > >> > > >> count.incrementAndGet(); > > >> > > >> } > > >> > > >> } catch (NoSuchMethodException | ScriptException > > >>> e) { > > >> > > >> e.printStackTrace(); > > >> > > >> } > > >> > > >> } > > >> > > >> }); > > >> > > >> } > > >> > > >> > > > > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) > >> wrote: > > > > > >> Intersting. The example you posted demonstrates this behaviour? If so > > >> I?ll file a bug and dig in. It sounds like an object is being reused > > >> across invocations and accumulating changes to the property map. > > >> > > >> ? Jim > > >> > > >> > > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon wrote: > > >> > > >> With more threads you are impacting the same 8 cores, so it will taper off > > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > > >>> threshold depending on System performance. Transport: I meant if you were > > >>> using sockets to provide the script. > > >> > > >> This makes sense. This one's on me then. > > >> > > >> > > >>> So you are using the same invocable instance for all threads? If so, > > >>> then you are probably good to go. As far as leaks are concerned, not sure > > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > > >>> and little JavaScript involved. > > >> > > >> > > >> > > >>> In your example, pull up Invocable invocable = generateInvocable(script); > > >>> out of the loop and use the same invocable for all threads. > > >> > > >> > > >> We were using one invocable across all threads and we were getting > > >> slowdowns on execution, high CPU Usage and memory leaks that led to > > >> OutOfMemory errors. I could trace the leak to > > >> > > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> *underlying* > > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > > >> > > >> which just keeps growing forever. > > >> > > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > > >> james.laskey at oracle.com> wrote: > > >> > > >>> > > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon wrote: > > >>> > > >>> The cost of creating a new engine is significant. So share an engine > > >>>> across threads but use *eval > > >>>> * > > >>>> (String > > >>>> > > >>>> script, ScriptContext > > >>>> > > >>>> context) instead, separate context per execution. If your JavaScript > > >>>> code does not modify globals you can get away with using the same engine, > > >>>> same compiled script on each thread. > > >>> > > >>> > > >>> I guess there's a few things here I don't understand. One thing I'm > > >>> trying to do is sharing a CompiledScript (which is why I'm using > > >>> invocable). Also, what exactly does modify globals mean? All our filters do > > >>> the same thing, make a function that takes a JSON String, turns it into a > > >>> JSON, modifies it and then stringifies it back. No state is changed of > > >>> anything else but there are temporary vars created inside the scope of the > > >>> function. When we run this multithreaded, running invokeFunction slows down > > >>> significantly and we get crazy memory leaks. > > >>> > > >>> > > >>> So you are using the same invocable instance for all threads? If so, > > >>> then you are probably good to go. As far as leaks are concerned, not sure > > >>> how you would get leaks from Nashorn. The JSON object is written in Java, > > >>> and little JavaScript involved. > > >>> > > >>> > > >>> Of course there are many factors involved n performance. How many cores > > >>>> do you have on the test machine? How much memory in the process? What > > >>>> transport are you using between threads? That sort of thing. Other than > > >>>> constructing then engine and context Nashorn performance should scale. > > >>> > > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to the > > >>> process. Not sure what transports between threads means, but this is the > > >>> code I'm benchmarking with. Increasing the number of threads actually makes > > >>> it go faster until about 4 threads, then adding more threads takes the same > > >>> amount to get to 1000 and and after a certain point it is just slower to > > >>> get to 1000 counts. Some of our filters need to be able to run over 1000 > > >>> times a second (across all threads) and the fastest time I could actually > > >>> get with this was about 2.4 seconds for a 1000 counts. > > >>> > > >>>> ExecutorService executor = Executors.newFixedThreadPool(50); > > >>>> > > >>>> AtomicLong count = new AtomicLong(); > > >>>> > > >>>> for (int i = 0; i < 50; i++) { > > >>>> > > >>>> executor.submit(new Runnable() { > > >>>> > > >>>> @Override > > >>>> > > >>>> public void run() { > > >>>> > > >>>> > > >>>>> try { > > >>>> > > >>>> Invocable invocable = > > >>>>> generateInvocable(script); > > >>>> > > >>>> while(true) { > > >>>> > > >>>> invocable.invokeFunction("transform", > > >>>>> something); > > >>>> > > >>>> count.incrementAndGet(); > > >>>> > > >>>> } > > >>>> > > >>>> } catch (NoSuchMethodException | ScriptException > > >>>>> e) { > > >>>> > > >>>> e.printStackTrace(); > > >>>> > > >>>> } > > >>>> > > >>>> } > > >>>> > > >>>> }); > > >>>> > > >>>> } > > >>>> > > >>>> long lastTimestamp = System.currentTimeMillis(); > > >>>> > > >>>> while(true) { > > >>>> > > >>>> > > >>>>> if (count.get() > 1000) { > > >>>> > > >>>> count.getAndAdd(-1000); > > >>>> > > >>>> System.out.println((System.currentTimeMillis() - > > >>>>> lastTimestamp)/1000.0); > > >>>> > > >>>> lastTimestamp = System.currentTimeMillis(); > > >>>> > > >>>> } > > >>>> > > >>>> } > > >>>> > > >>>> > > >>> With more threads you are impacting the same 8 cores, so it will taper > > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 being a > > >>> threshold depending on System performance. Transport: I meant if you were > > >>> using sockets to provide the script. > > >>> > > >>> In your example, pull up Invocable invocable = generateInvocable(script); > > >>> out of the loop and use the same invocable for all threads. > > >>> > > >>> - Jim > > >>> > > >>> > > >>> > > >>> > > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > > >>> james.laskey at oracle.com> wrote: > > >>> > > >>>> > > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon wrote: > > >>>> > > >>>> Hey Jim, > > >>>> > > >>>> I looked at it and I will look into loadWithNewGlobal to see what > > >>>> exactly it does since it could be relevant. As for the rest, for my use > > >>>> case having threads in the JS would not help. We're using Nashorn to build > > >>>> JSON filters in a Dynamic Proxy Service and need any of the threads > > >>>> processing a request to be able to execute the script to filter. > > >>>> > > >>>> > > >>>> The cost of creating a new engine is significant. So share an engine > > >>>> across threads but use *eval > > >>>> * > > >>>> (String > > >>>> > > >>>> script, ScriptContext > > >>>> > > >>>> context) instead, separate context per execution. If your JavaScript > > >>>> code does not modify globals you can get away with using the same engine, > > >>>> same compiled script on each thread. > > >>>> > > >>>> > > >>>> Also, when you say a new engine per threads is the worst case what > > >>>> exactly do you mean? I would expect an initial cost of compiling the script > > >>>> on each thread and then each engine should be able to do its own thing, but > > >>>> what I'm seeing is that when running with more than 10 threads all my > > >>>> engines get slow at executing code, even though they are all completely > > >>>> separate from each other. > > >>>> > > >>>> > > >>>> Of course there are many factors involved n performance. How many cores > > >>>> do you have on the test machine? How much memory in the process? What > > >>>> transport are you using between threads? That sort of thing. Other than > > >>>> constructing then engine and context Nashorn performance should scale. > > >>>> > > >>>> > > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > > >>>> james.laskey at oracle.com> wrote: > > >>>> > > >>>>> Jesus, > > >>>>> > > >>>>> Probably the most informative information is in this blog. > > >>>>> > > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_threading_and_mt > > >>>>> > > >>>>> This example uses Executors but threads would work as well. > > >>>>> > > >>>>> I did a talk that looked at different methods to max out multithreading > > >>>>> performance. A new engine per thread is the worst case. A new context per > > >>>>> thread does much better. A new global per thread is the best while > > >>>>> remaining thread safe. > > >>>>> > > >>>>> Cheers, > > >>>>> > > >>>>> ? Jim > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon wrote: > > >>>>> > > >>>>> Hey folks, > > >>>>> > > >>>>> I've tried many different ways of using Nashorn multithreaded based on > > >>>>> what > > >>>>> I've found on the internet and I still can't get a single one to scale. > > >>>>> Even the most naive method of making many script engines with my script > > >>>>> tends to bottleneck itself when I have more than 10 threads invoking > > >>>>> functions. > > >>>>> > > >>>>> I'm using the following code to compile my script and > > >>>>> invocable.invokeFunction("transform", input) to execute: > > >>>>> > > >>>>> static Invocable generateInvocable(String script) throws > > >>>>> ScriptException { > > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > > >>>>> ScriptEngine engine = > > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > > >>>>> Compilable compilable = (Compilable) engine; > > >>>>> final CompiledScript compiled = compilable.compile(script); > > >>>>> compiled.eval(); > > >>>>> return (Invocable) engine; > > >>>>> } > > >>>>> > > >>>>> > > >>>>> > > >>>>> The script I'm compiling is: > > >>>>> > > >>>>> String script = "function transform(input) {" + > > >>>>> "var result = JSON.parse(input);" + > > >>>>> "response = {};\n" + > > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > > >>>>> " var summoner = {};\n" + > > >>>>> " summoner.id = result[i].id;\n" + > > >>>>> " summoner.name = result[i].name;\n" + > > >>>>> " summoner.profileIconId = > > >>>>> result[i].profileIconId;\n" + > > >>>>> " summoner.revisionDate = result[i].revisionDate;\n" + > > >>>>> " summoner.summonerLevel = result[i].level;\n" + > > >>>>> " response[summoner.id] = summoner;\n" + > > >>>>> "}\n" + > > >>>>> "result = response;" + > > >>>>> "return JSON.stringify(result);" + > > >>>>> "};"; > > >>>>> > > >>>>> > > >>>>> > > >>>>> I've also tried other more scaleable ways to work with scripts > > >>>>> concurrently, but given that this is the most naive method where > > >>>>> everything > > >>>>> is brand new and I still get slowness calling them concurrently I fear > > >>>>> that > > >>>>> maybe I'm overlooking something extremely basic on my code. > > >>>>> > > >>>>> Thanks. > > >>>>> -Jesus Luzon > > >>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >> > > >> > > > > > > > > > From jluzon at riotgames.com Fri Dec 9 10:49:42 2016 From: jluzon at riotgames.com (Jesus Luzon) Date: Fri, 9 Dec 2016 02:49:42 -0800 Subject: Can't get Multithreaded Nashorn uses to Scale In-Reply-To: References: <31833CD9-8263-478E-9400-87D5D561C29A@oracle.com> Message-ID: Wow I totally missed the var. Thanks for that! I actually had changed our code to use ThreadLocal ScriptObjectMirrors created from the CompiledScript with the following code: ScriptObjectMirror mirror = mirrorThreadLocal.get(); > > if (mirror == null) { > > Bindings bindings = compiledScript.getEngine().createBindings(); > > compiledScript.eval(bindings); > > mirror = (ScriptObjectMirror) bindings.get("transform"); > > mirrorThreadLocal.set(mirror); > > } > > return (String) mirror.call(null, something); > > Given that we will always run standalone functions to filter data, is there one we should be using over the other? Lastly thanks for filing the bug. I'll keep track of it so I can make sure to upgrade when a fix comes out. On Fri, Dec 9, 2016 at 2:37 AM, Hannes Walln?fer < hannes.wallnoefer at oracle.com> wrote: > Jesus, > > I looked at this issue more in depth and there are two problems here, one > on your side and one on ours: > > The problem with your code is that you left out the ?var? declaration of > the ?response? variable in the third line of your function. This means that > ?response? is allocated as a global variable outside the scope of the > function. When you run this in parallel with multiple threads you actually > access the same object from multiple threads, meaning your output will be > corrupt. > > Because of things like this, it is actually a good idea to run JavaScript > in strict mode (e.g. by using ?use strict? directive in your JavaScript > code, or by creating the engine with -strict option). Another solution > would be to use multiple engines or bindings, which should not have a big > performance impact once the other problem is fixed. > > Which leads us to the problem on our side, which is performance of sparse > arrays. As you have noticed, using something that is not a valid array > index greatly improves performance, and that is because some array element > assignment are implemented very inefficiently in Nashorn. I?ve filed a bug > for this: > > https://bugs.openjdk.java.net/browse/JDK-8170977 > > Once this bug is fixed, using numeric keys should be roughly as fast as > using non-numeric ones. > > Hannes > > > > Am 09.12.2016 um 04:47 schrieb Jesus Luzon : > > > > I came up with a workaround which is getting us through this but it > definitely seems like this is a bug. The gist is I append an arbitrary > string (we made it complex on our end to make it unlikely something else > has this string inside) to the key and then remove all instances of said > string on the JSON String resulting from calling stringify. We are leaving > this for overnight testing but it showed promising results immediately. > > > > String script = "function transform(input) {" + > > "var result = JSON.parse(input);" + > > "response = {};\n" + > > "for (var i = 0; i < result.length; i++) {\n" + > > " var summoner = {};\n" + > > " summoner.id = result[i].id;\n" + > > " summoner.name = result[i].name;\n" + > > " summoner.profileIconId = > result[i].profileIconId;\n" + > > " summoner.revisionDate = result[i].revisionDate;\n" + > > " summoner.summonerLevel = result[i].level;\n" + > > " response[\"someArbitraryString\" + summoner.id] = > summoner;\n" + > > "}\n" + > > "var stringy = JSON.stringify(response);" + > > "result = stringy.replace(/someArbitraryString/g, > \"\");" + > > "return result;" + > > "};"; > > > > On Thu, Dec 8, 2016 at 4:54 PM, Jesus Luzon > wrote: > > It doesn't seem like changing to a string fixed it. We dug deeper and > found a very interesting issue though. After running the invocable a few > thousand times, when we put a breakpoint in ArrayData.computerIteratorKeys() > and we found that it would stop at an execution from SparseArrayData with > an underlying array of size 923392, stored length of 461672 (this is one of > our IDs, the largest 6 digit one). Because of this we tried a run changing > all the IDs in the array we pass in to number from 1-38 and this thing went > blazing fast. When putting a break point in the same place, this array in > SpareArrayData was now of size 39. We then changed an id in the array we > take in to be 1337 and the size of the array in the SparseArrayData was > 1338. > > > > I don't understand why or how to prevent but it's using this ID as an > index for SpareArrayData underlying array. If someone can help me find a > workaround for this I would be extremely grateful. > > > > On Thu, Dec 8, 2016 at 3:52 AM, Hannes Walln?fer < > hannes.wallnoefer at oracle.com> wrote: > > Yes, this is very likely to be the cause of the problem. However, I do > think we should be able to handle sparse array data better, so it?s quite > possible that you exposed some weak spot in our implementation. I?ll have a > look into what?s going on. > > > > Hannes > > > > > > > Am 08.12.2016 um 00:40 schrieb Jesus Luzon : > > > > > > Looks like the memory leak is due to the way we wrote our code and how > javascript works. I was expecting the line response[summoner.id] = > summoner; to build a map but it turns out that if you use a number as the > "key", javscript automatically fills the indexes in the middle with null > (Undefined type?). When these IDs are very large, it is creating huge > arrays that take longer to garbage collect than the code executing. I am > about to start testing this on our end to make sure we see the improvements > we expect. > > > > > > Does this idea seem like it is reasonably possible? > > > > > > On Wed, Dec 7, 2016 at 11:23 AM, Jesus Luzon > wrote: > > > Yes, it's an array of objects which I'll paste. And yes, I'm just > calling invokeFunction from many many different threads. I'm also going to > go back and take a look at all the heap dumps we have to re-confirm what I > mentioned. > > > > > > "[\n" + > > > " {\n" + > > > " \"id\": 6011511,\n" + > > > " \"accountId\": 203192481,\n" + > > > " \"name\": \"Adam Pro Qarda?\",\n" + > > > " \"profileIconId\": 25,\n" + > > > " \"level\": 5,\n" + > > > " \"expPoints\": 83,\n" + > > > " \"infPoints\": 1475,\n" + > > > " \"revisionDate\": 1406631727000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 2810674,\n" + > > > " \"accountId\": 200706913,\n" + > > > " \"name\": \"ABZ Devrim\",\n" + > > > " \"profileIconId\": 663,\n" + > > > " \"level\": 13,\n" + > > > " \"expPoints\": 982,\n" + > > > " \"infPoints\": 10472,\n" + > > > " \"revisionDate\": 1450791227000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5411195,\n" + > > > " \"accountId\": 202647689,\n" + > > > " \"name\": \"Ace HypcronN\",\n" + > > > " \"profileIconId\": 911,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 73,\n" + > > > " \"infPoints\": 182445,\n" + > > > " \"revisionDate\": 1480781650000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1363020,\n" + > > > " \"accountId\": 1357837,\n" + > > > " \"name\": \"AdanaLee\",\n" + > > > " \"profileIconId\": 502,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 125,\n" + > > > " \"infPoints\": 719299,\n" + > > > " \"revisionDate\": 1480530778000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 8261198,\n" + > > > " \"accountId\": 205027096,\n" + > > > " \"name\": \"Achilehuz\",\n" + > > > " \"profileIconId\": 1381,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 10,\n" + > > > " \"infPoints\": 158603,\n" + > > > " \"revisionDate\": 1480770307000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 12685857,\n" + > > > " \"accountId\": 207591166,\n" + > > > " \"name\": \"ac?mas?zpicc\",\n" + > > > " \"profileIconId\": 9,\n" + > > > " \"level\": 21,\n" + > > > " \"expPoints\": 840,\n" + > > > " \"infPoints\": 16659,\n" + > > > " \"revisionDate\": 1480515325000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 10860127,\n" + > > > " \"accountId\": 206507727,\n" + > > > " \"name\": \"AAngelFlyy\",\n" + > > > " \"profileIconId\": 1395,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 10,\n" + > > > " \"infPoints\": 73111,\n" + > > > " \"revisionDate\": 1480787870000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 3292376,\n" + > > > " \"accountId\": 201048714,\n" + > > > " \"name\": \"ACAB1907\",\n" + > > > " \"profileIconId\": 20,\n" + > > > " \"level\": 6,\n" + > > > " \"expPoints\": 305,\n" + > > > " \"infPoints\": 2107,\n" + > > > " \"revisionDate\": 1402448089000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 461671,\n" + > > > " \"accountId\": 446571,\n" + > > > " \"name\": \"Acta Est Fabul?\",\n" + > > > " \"profileIconId\": 1435,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 47,\n" + > > > " \"infPoints\": 644672,\n" + > > > " \"revisionDate\": 1480626505000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 394183,\n" + > > > " \"accountId\": 379083,\n" + > > > " \"name\": \"acekse4\",\n" + > > > " \"profileIconId\": 27,\n" + > > > " \"level\": 5,\n" + > > > " \"expPoints\": 223,\n" + > > > " \"infPoints\": 908,\n" + > > > " \"revisionDate\": 1348116544000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5941247,\n" + > > > " \"accountId\": 203106300,\n" + > > > " \"name\": \"abdul7878\",\n" + > > > " \"profileIconId\": 26,\n" + > > > " \"level\": 3,\n" + > > > " \"expPoints\": 10,\n" + > > > " \"infPoints\": 401,\n" + > > > " \"revisionDate\": 1406029148000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 2467446,\n" + > > > " \"accountId\": 200459837,\n" + > > > " \"name\": \"ActionC\",\n" + > > > " \"profileIconId\": 986,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 74,\n" + > > > " \"infPoints\": 401367,\n" + > > > " \"revisionDate\": 1480808608000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 9402979,\n" + > > > " \"accountId\": 205698832,\n" + > > > " \"name\": \"Ablenia \",\n" + > > > " \"profileIconId\": 1129,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 19,\n" + > > > " \"infPoints\": 163518,\n" + > > > " \"revisionDate\": 1480687603000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 13187505,\n" + > > > " \"accountId\": 207898213,\n" + > > > " \"name\": \"aDaMiYiM\",\n" + > > > " \"profileIconId\": 1301,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 116,\n" + > > > " \"infPoints\": 45214,\n" + > > > " \"revisionDate\": 1480793258000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 4141059,\n" + > > > " \"accountId\": 201688290,\n" + > > > " \"name\": \"Abimin?ar?\",\n" + > > > " \"profileIconId\": 898,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 152,\n" + > > > " \"infPoints\": 752477,\n" + > > > " \"revisionDate\": 1480635961000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5702134,\n" + > > > " \"accountId\": 202899395,\n" + > > > " \"name\": \"Above the Clouds\",\n" + > > > " \"profileIconId\": 684,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 110,\n" + > > > " \"infPoints\": 288096,\n" + > > > " \"revisionDate\": 1471011372000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5810740,\n" + > > > " \"accountId\": 202985228,\n" + > > > " \"name\": \"aBimm\",\n" + > > > " \"profileIconId\": 11,\n" + > > > " \"level\": 13,\n" + > > > " \"expPoints\": 1180,\n" + > > > " \"infPoints\": 10736,\n" + > > > " \"revisionDate\": 1409832684000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5817751,\n" + > > > " \"accountId\": 203050678,\n" + > > > " \"name\": \"AD Glor?am\",\n" + > > > " \"profileIconId\": 982,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 111,\n" + > > > " \"infPoints\": 304658,\n" + > > > " \"revisionDate\": 1480795250000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 9851802,\n" + > > > " \"accountId\": 206011054,\n" + > > > " \"name\": \"AdarAllame\",\n" + > > > " \"profileIconId\": 911,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 48,\n" + > > > " \"infPoints\": 73763,\n" + > > > " \"revisionDate\": 1479422812000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 12735622,\n" + > > > " \"accountId\": 207587019,\n" + > > > " \"name\": \"absinthe666\",\n" + > > > " \"profileIconId\": 903,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 83,\n" + > > > " \"infPoints\": 40302,\n" + > > > " \"revisionDate\": 1480782923000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 6371389,\n" + > > > " \"accountId\": 203416952,\n" + > > > " \"name\": \"adamsat?c?\",\n" + > > > " \"profileIconId\": 3,\n" + > > > " \"level\": 4,\n" + > > > " \"expPoints\": 17,\n" + > > > " \"infPoints\": 685,\n" + > > > " \"revisionDate\": 1409320171000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 7828139,\n" + > > > " \"accountId\": 204927980,\n" + > > > " \"name\": \"AbsoluteForce\",\n" + > > > " \"profileIconId\": 950,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 139,\n" + > > > " \"infPoints\": 208789,\n" + > > > " \"revisionDate\": 1480804396000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1373229,\n" + > > > " \"accountId\": 1358441,\n" + > > > " \"name\": \"AbsoluteDeath\",\n" + > > > " \"profileIconId\": 7,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 34,\n" + > > > " \"infPoints\": 223655,\n" + > > > " \"revisionDate\": 1471867646000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 7694972,\n" + > > > " \"accountId\": 204803668,\n" + > > > " \"name\": \"ac pnp\",\n" + > > > " \"profileIconId\": 937,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 161,\n" + > > > " \"infPoints\": 249681,\n" + > > > " \"revisionDate\": 1480801507000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1373524,\n" + > > > " \"accountId\": 1350474,\n" + > > > " \"name\": \"Abd??\",\n" + > > > " \"profileIconId\": 1301,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 103,\n" + > > > " \"infPoints\": 286803,\n" + > > > " \"revisionDate\": 1476621827000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1650227,\n" + > > > " \"accountId\": 200000503,\n" + > > > " \"name\": \"AD Ambrosia\",\n" + > > > " \"profileIconId\": 1152,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 139,\n" + > > > " \"infPoints\": 156333,\n" + > > > " \"revisionDate\": 1480805320000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 8331358,\n" + > > > " \"accountId\": 205073925,\n" + > > > " \"name\": \"acarmanyust2\",\n" + > > > " \"profileIconId\": 0,\n" + > > > " \"level\": 2,\n" + > > > " \"expPoints\": 43,\n" + > > > " \"infPoints\": 318,\n" + > > > " \"revisionDate\": 1423915139000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1862106,\n" + > > > " \"accountId\": 200139838,\n" + > > > " \"name\": \"aboU\",\n" + > > > " \"profileIconId\": 1155,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 0,\n" + > > > " \"infPoints\": 412616,\n" + > > > " \"revisionDate\": 1480771055000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 2362628,\n" + > > > " \"accountId\": 685649,\n" + > > > " \"name\": \"Ac?F?sT?k\",\n" + > > > " \"profileIconId\": 1074,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 48,\n" + > > > " \"infPoints\": 233882,\n" + > > > " \"revisionDate\": 1480786233000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 4323909,\n" + > > > " \"accountId\": 201917672,\n" + > > > " \"name\": \"Addrenalin\",\n" + > > > " \"profileIconId\": 603,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 55,\n" + > > > " \"infPoints\": 220605,\n" + > > > " \"revisionDate\": 1432647338000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 377206,\n" + > > > " \"accountId\": 362106,\n" + > > > " \"name\": \"Aburame Shino\",\n" + > > > " \"profileIconId\": 844,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 84,\n" + > > > " \"infPoints\": 354087,\n" + > > > " \"revisionDate\": 1477666556000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5377433,\n" + > > > " \"accountId\": 202697921,\n" + > > > " \"name\": \"AcEcolton35\",\n" + > > > " \"profileIconId\": 984,\n" + > > > " \"level\": 25,\n" + > > > " \"expPoints\": 751,\n" + > > > " \"infPoints\": 30061,\n" + > > > " \"revisionDate\": 1475503024000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 2381404,\n" + > > > " \"accountId\": 200333680,\n" + > > > " \"name\": \"adafakaaa\",\n" + > > > " \"profileIconId\": 663,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 8,\n" + > > > " \"infPoints\": 534204,\n" + > > > " \"revisionDate\": 1480719827000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 1281203,\n" + > > > " \"accountId\": 1259342,\n" + > > > " \"name\": \"AC Klondike\",\n" + > > > " \"profileIconId\": 898,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 27,\n" + > > > " \"infPoints\": 191429,\n" + > > > " \"revisionDate\": 1480294973000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 13161471,\n" + > > > " \"accountId\": 207847181,\n" + > > > " \"name\": \"adar21\",\n" + > > > " \"profileIconId\": 26,\n" + > > > " \"level\": 10,\n" + > > > " \"expPoints\": 143,\n" + > > > " \"infPoints\": 3558,\n" + > > > " \"revisionDate\": 1476529855000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 5841915,\n" + > > > " \"accountId\": 202998794,\n" + > > > " \"name\": \"Achilles29\",\n" + > > > " \"profileIconId\": 666,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 41,\n" + > > > " \"infPoints\": 219714,\n" + > > > " \"revisionDate\": 1480777744000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 2853062,\n" + > > > " \"accountId\": 200726707,\n" + > > > " \"name\": \"AbIanStarBebegim\",\n" + > > > " \"profileIconId\": 909,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 64,\n" + > > > " \"infPoints\": 297580,\n" + > > > " \"revisionDate\": 1480556859000\n" + > > > " },\n" + > > > " {\n" + > > > " \"id\": 8323114,\n" + > > > " \"accountId\": 205093515,\n" + > > > " \"name\": \"Absuruk\",\n" + > > > " \"profileIconId\": 6,\n" + > > > " \"level\": 30,\n" + > > > " \"expPoints\": 21,\n" + > > > " \"infPoints\": 121086,\n" + > > > " \"revisionDate\": 1480820801000\n" + > > > " }\n" + > > > "]" > > > > > > On Wed, Dec 7, 2016 at 7:01 AM, Hannes Walln?fer < > hannes.wallnoefer at oracle.com> wrote: > > > Hi Jesus, > > > > > > I?m trying to reproduce the problem, and just want to make sure I get > the missing pieces right. > > > > > > You already showed us how you?re setting up the engine and the JS code > you?re running. I assume the JSON code you?re parsing is a simple array of > objects? And you?re just calling Invocable.invokeFunction on the > ScriptEngine from multiple threads in parallel, right? > > > > > > Thanks, > > > Hannes > > > > > > > > > > Am 07.12.2016 um 00:03 schrieb Jesus Luzon : > > > > > > > > When we share one invocable across many threads and run > invokeFunction it > > > > happens, such as this: > > > > > > > > ExecutorService executor = Executors.newFixedThreadPool(50); > > > >> > > > >> Invocable invocable = generateInvocable(script); > > > >> > > > >> AtomicLong count = new AtomicLong(); > > > >> > > > >> for (int i = 0; i < 50; i++) { > > > >> > > > >> executor.submit(new Runnable() { > > > >> > > > >> @Override > > > >> > > > >> public void run() { > > > >> > > > >> try { > > > >> > > > >> while(true) { > > > >> > > > >> invocable.invokeFunction(" > transform", > > > >>> something); > > > >> > > > >> count.incrementAndGet(); > > > >> > > > >> } > > > >> > > > >> } catch (NoSuchMethodException | > ScriptException > > > >>> e) { > > > >> > > > >> e.printStackTrace(); > > > >> > > > >> } > > > >> > > > >> } > > > >> > > > >> }); > > > >> > > > >> } > > > >> > > > >> > > > > > > > > > > > > On Tue, Dec 6, 2016 at 2:59 PM, Jim Laskey (Oracle) < > james.laskey at oracle.com > > > >> wrote: > > > > > > > >> Intersting. The example you posted demonstrates this behaviour? > If so > > > >> I?ll file a bug and dig in. It sounds like an object is being > reused > > > >> across invocations and accumulating changes to the property map. > > > >> > > > >> ? Jim > > > >> > > > >> > > > >> On Dec 6, 2016, at 5:12 PM, Jesus Luzon > wrote: > > > >> > > > >> With more threads you are impacting the same 8 cores, so it will > taper off > > > >>> after 8 threads. If it?s a 2x4 core machine then I can see 4 > being a > > > >>> threshold depending on System performance. Transport: I meant if > you were > > > >>> using sockets to provide the script. > > > >> > > > >> This makes sense. This one's on me then. > > > >> > > > >> > > > >>> So you are using the same invocable instance for all threads? If > so, > > > >>> then you are probably good to go. As far as leaks are concerned, > not sure > > > >>> how you would get leaks from Nashorn. The JSON object is written > in Java, > > > >>> and little JavaScript involved. > > > >> > > > >> > > > >> > > > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > > > >>> out of the loop and use the same invocable for all threads. > > > >> > > > >> > > > >> We were using one invocable across all threads and we were getting > > > >> slowdowns on execution, high CPU Usage and memory leaks that led to > > > >> OutOfMemory errors. I could trace the leak to > > > >> > > > >> jdk.nashorn.internal.objects.Global -> *objectSpill* Object[8] -> > > > >> jdk.nashorn.internal.scripts.JO4 -> *arrayData* > > > >> jdk.nashorn.internal.runtime.arrays.SparseArraysData -> > *underlying* > > > >> jdk.nashorn.internal.runtime.arrays.DeletedArrayFilter > > > >> > > > >> which just keeps growing forever. > > > >> > > > >> On Tue, Dec 6, 2016 at 6:30 AM, Jim Laskey (Oracle) < > > > >> james.laskey at oracle.com> wrote: > > > >> > > > >>> > > > >>> On Dec 6, 2016, at 9:56 AM, Jesus Luzon > wrote: > > > >>> > > > >>> The cost of creating a new engine is significant. So share an > engine > > > >>>> across threads but use *eval > > > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > > > >>>> (String > > > >>>> > > > >>>> script, ScriptContext > > > >>>> script/ScriptContext.html> > > > >>>> context) instead, separate context per execution. If your > JavaScript > > > >>>> code does not modify globals you can get away with using the same > engine, > > > >>>> same compiled script on each thread. > > > >>> > > > >>> > > > >>> I guess there's a few things here I don't understand. One thing I'm > > > >>> trying to do is sharing a CompiledScript (which is why I'm using > > > >>> invocable). Also, what exactly does modify globals mean? All our > filters do > > > >>> the same thing, make a function that takes a JSON String, turns it > into a > > > >>> JSON, modifies it and then stringifies it back. No state is > changed of > > > >>> anything else but there are temporary vars created inside the > scope of the > > > >>> function. When we run this multithreaded, running invokeFunction > slows down > > > >>> significantly and we get crazy memory leaks. > > > >>> > > > >>> > > > >>> So you are using the same invocable instance for all threads? If > so, > > > >>> then you are probably good to go. As far as leaks are concerned, > not sure > > > >>> how you would get leaks from Nashorn. The JSON object is written > in Java, > > > >>> and little JavaScript involved. > > > >>> > > > >>> > > > >>> Of course there are many factors involved n performance. How many > cores > > > >>>> do you have on the test machine? How much memory in the > process? What > > > >>>> transport are you using between threads? That sort of thing. > Other than > > > >>>> constructing then engine and context Nashorn performance should > scale. > > > >>> > > > >>> I'm using an 8 core machine to test with 2.5Gs of RAM allocated to > the > > > >>> process. Not sure what transports between threads means, but this > is the > > > >>> code I'm benchmarking with. Increasing the number of threads > actually makes > > > >>> it go faster until about 4 threads, then adding more threads takes > the same > > > >>> amount to get to 1000 and and after a certain point it is just > slower to > > > >>> get to 1000 counts. Some of our filters need to be able to run > over 1000 > > > >>> times a second (across all threads) and the fastest time I could > actually > > > >>> get with this was about 2.4 seconds for a 1000 counts. > > > >>> > > > >>>> ExecutorService executor = Executors.newFixedThreadPool( > 50); > > > >>>> > > > >>>> AtomicLong count = new AtomicLong(); > > > >>>> > > > >>>> for (int i = 0; i < 50; i++) { > > > >>>> > > > >>>> executor.submit(new Runnable() { > > > >>>> > > > >>>> @Override > > > >>>> > > > >>>> public void run() { > > > >>>> > > > >>>> > > > >>>>> try { > > > >>>> > > > >>>> Invocable invocable = > > > >>>>> generateInvocable(script); > > > >>>> > > > >>>> while(true) { > > > >>>> > > > >>>> invocable.invokeFunction(" > transform", > > > >>>>> something); > > > >>>> > > > >>>> count.incrementAndGet(); > > > >>>> > > > >>>> } > > > >>>> > > > >>>> } catch (NoSuchMethodException | > ScriptException > > > >>>>> e) { > > > >>>> > > > >>>> e.printStackTrace(); > > > >>>> > > > >>>> } > > > >>>> > > > >>>> } > > > >>>> > > > >>>> }); > > > >>>> > > > >>>> } > > > >>>> > > > >>>> long lastTimestamp = System.currentTimeMillis(); > > > >>>> > > > >>>> while(true) { > > > >>>> > > > >>>> > > > >>>>> if (count.get() > 1000) { > > > >>>> > > > >>>> count.getAndAdd(-1000); > > > >>>> > > > >>>> System.out.println((System.currentTimeMillis() - > > > >>>>> lastTimestamp)/1000.0); > > > >>>> > > > >>>> lastTimestamp = System.currentTimeMillis(); > > > >>>> > > > >>>> } > > > >>>> > > > >>>> } > > > >>>> > > > >>>> > > > >>> With more threads you are impacting the same 8 cores, so it will > taper > > > >>> off after 8 threads. If it?s a 2x4 core machine then I can see 4 > being a > > > >>> threshold depending on System performance. Transport: I meant if > you were > > > >>> using sockets to provide the script. > > > >>> > > > >>> In your example, pull up Invocable invocable = > generateInvocable(script); > > > >>> out of the loop and use the same invocable for all threads. > > > >>> > > > >>> - Jim > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On Tue, Dec 6, 2016 at 5:31 AM, Jim Laskey (Oracle) < > > > >>> james.laskey at oracle.com> wrote: > > > >>> > > > >>>> > > > >>>> On Dec 6, 2016, at 9:19 AM, Jesus Luzon > wrote: > > > >>>> > > > >>>> Hey Jim, > > > >>>> > > > >>>> I looked at it and I will look into loadWithNewGlobal to see what > > > >>>> exactly it does since it could be relevant. As for the rest, for > my use > > > >>>> case having threads in the JS would not help. We're using Nashorn > to build > > > >>>> JSON filters in a Dynamic Proxy Service and need any of the > threads > > > >>>> processing a request to be able to execute the script to filter. > > > >>>> > > > >>>> > > > >>>> The cost of creating a new engine is significant. So share an > engine > > > >>>> across threads but use *eval > > > >>>> script/ScriptEngine.html#eval(java.lang.String,%20javax. > script.ScriptContext)>* > > > >>>> (String > > > >>>> > > > >>>> script, ScriptContext > > > >>>> script/ScriptContext.html> > > > >>>> context) instead, separate context per execution. If your > JavaScript > > > >>>> code does not modify globals you can get away with using the same > engine, > > > >>>> same compiled script on each thread. > > > >>>> > > > >>>> > > > >>>> Also, when you say a new engine per threads is the worst case what > > > >>>> exactly do you mean? I would expect an initial cost of compiling > the script > > > >>>> on each thread and then each engine should be able to do its own > thing, but > > > >>>> what I'm seeing is that when running with more than 10 threads > all my > > > >>>> engines get slow at executing code, even though they are all > completely > > > >>>> separate from each other. > > > >>>> > > > >>>> > > > >>>> Of course there are many factors involved n performance. How > many cores > > > >>>> do you have on the test machine? How much memory in the > process? What > > > >>>> transport are you using between threads? That sort of thing. > Other than > > > >>>> constructing then engine and context Nashorn performance should > scale. > > > >>>> > > > >>>> > > > >>>> On Tue, Dec 6, 2016 at 5:07 AM, Jim Laskey (Oracle) < > > > >>>> james.laskey at oracle.com> wrote: > > > >>>> > > > >>>>> Jesus, > > > >>>>> > > > >>>>> Probably the most informative information is in this blog. > > > >>>>> > > > >>>>> https://blogs.oracle.com/nashorn/entry/nashorn_multi_ > threading_and_mt > > > >>>>> > > > >>>>> This example uses Executors but threads would work as well. > > > >>>>> > > > >>>>> I did a talk that looked at different methods to max out > multithreading > > > >>>>> performance. A new engine per thread is the worst case. A new > context per > > > >>>>> thread does much better. A new global per thread is the best > while > > > >>>>> remaining thread safe. > > > >>>>> > > > >>>>> Cheers, > > > >>>>> > > > >>>>> ? Jim > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> On Dec 6, 2016, at 8:45 AM, Jesus Luzon > wrote: > > > >>>>> > > > >>>>> Hey folks, > > > >>>>> > > > >>>>> I've tried many different ways of using Nashorn multithreaded > based on > > > >>>>> what > > > >>>>> I've found on the internet and I still can't get a single one to > scale. > > > >>>>> Even the most naive method of making many script engines with my > script > > > >>>>> tends to bottleneck itself when I have more than 10 threads > invoking > > > >>>>> functions. > > > >>>>> > > > >>>>> I'm using the following code to compile my script and > > > >>>>> invocable.invokeFunction("transform", input) to execute: > > > >>>>> > > > >>>>> static Invocable generateInvocable(String script) throws > > > >>>>> ScriptException { > > > >>>>> ScriptEngineManager manager = new ScriptEngineManager(); > > > >>>>> ScriptEngine engine = > > > >>>>> manager.getEngineByName(JAVASCRIPT_ENGINE_NAME); > > > >>>>> Compilable compilable = (Compilable) engine; > > > >>>>> final CompiledScript compiled = compilable.compile(script); > > > >>>>> compiled.eval(); > > > >>>>> return (Invocable) engine; > > > >>>>> } > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> The script I'm compiling is: > > > >>>>> > > > >>>>> String script = "function transform(input) {" + > > > >>>>> "var result = JSON.parse(input);" + > > > >>>>> "response = {};\n" + > > > >>>>> "for (var i = 0; i < result.length; i++) {\n" + > > > >>>>> " var summoner = {};\n" + > > > >>>>> " summoner.id = result[i].id;\n" + > > > >>>>> " summoner.name = result[i].name;\n" + > > > >>>>> " summoner.profileIconId = > > > >>>>> result[i].profileIconId;\n" + > > > >>>>> " summoner.revisionDate = > result[i].revisionDate;\n" + > > > >>>>> " summoner.summonerLevel = result[i].level;\n" + > > > >>>>> " response[summoner.id] = summoner;\n" + > > > >>>>> "}\n" + > > > >>>>> "result = response;" + > > > >>>>> "return JSON.stringify(result);" + > > > >>>>> "};"; > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> I've also tried other more scaleable ways to work with scripts > > > >>>>> concurrently, but given that this is the most naive method where > > > >>>>> everything > > > >>>>> is brand new and I still get slowness calling them concurrently > I fear > > > >>>>> that > > > >>>>> maybe I'm overlooking something extremely basic on my code. > > > >>>>> > > > >>>>> Thanks. > > > >>>>> -Jesus Luzon > > > >>>>> > > > >>>>> > > > >>>> > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > > > > > > > > > > From hannes.wallnoefer at oracle.com Thu Dec 15 10:48:05 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 15 Dec 2016 11:48:05 +0100 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation Message-ID: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) - Dense array data did not check their size before trying System.arraycopy in shiftLeft - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. Thanks, Hannes From james.laskey at oracle.com Thu Dec 15 12:13:58 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 15 Dec 2016 08:13:58 -0400 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation In-Reply-To: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> References: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Message-ID: +1 > On Dec 15, 2016, at 6:48 AM, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 > Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ > > This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: > > - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) > - Dense array data did not check their size before trying System.arraycopy in shiftLeft > - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots > - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft > - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft > - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots > - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. > > I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. > > Thanks, > Hannes From sundararajan.athijegannathan at oracle.com Thu Dec 15 12:56:56 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Thu, 15 Dec 2016 18:26:56 +0530 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation In-Reply-To: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> References: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Message-ID: <58529318.90800@oracle.com> +1 Wow! so many issues... -Sundar On 15/12/16, 4:18 PM, Hannes Walln?fer wrote: > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 > Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ > > This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: > > - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) > - Dense array data did not check their size before trying System.arraycopy in shiftLeft > - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots > - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft > - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft > - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots > - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. > > I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. > > Thanks, > Hannes From szegedia at gmail.com Thu Dec 15 12:47:05 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 15 Dec 2016 13:47:05 +0100 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation In-Reply-To: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> References: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Message-ID: +1 Wow, this is lots of fixes. Some of these bugs are so glaring I?m astonished they we did not spot them earlier, like failing to set a new length in shiftLeft. It?s great you fixed these. Attila. > On 15 Dec 2016, at 11:48, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 > Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ > > This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: > > - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) > - Dense array data did not check their size before trying System.arraycopy in shiftLeft > - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots > - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft > - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft > - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots > - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. > > I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. > > Thanks, > Hannes From hannes.wallnoefer at oracle.com Thu Dec 15 13:13:15 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 15 Dec 2016 14:13:15 +0100 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation In-Reply-To: References: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Message-ID: <19C89047-62CE-4FA6-B19C-AB74349E2157@oracle.com> I guess shift is just not a very common operation on sparse arrays. It?s almost like mutually exclusive use cases. Still good to have them fixed. Thanks for the reviews! Hannes > Am 15.12.2016 um 13:47 schrieb Attila Szegedi : > > +1 > > Wow, this is lots of fixes. Some of these bugs are so glaring I?m astonished they we did not spot them earlier, like failing to set a new length in shiftLeft. It?s great you fixed these. > > Attila. > >> On 15 Dec 2016, at 11:48, Hannes Walln?fer wrote: >> >> Please review: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 >> Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ >> >> This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: >> >> - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) >> - Dense array data did not check their size before trying System.arraycopy in shiftLeft >> - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots >> - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft >> - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft >> - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots >> - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. >> >> I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. >> >> Thanks, >> Hannes > From marcus at lagergren.net Fri Dec 16 15:44:20 2016 From: marcus at lagergren.net (Marcus Lagergren) Date: Fri, 16 Dec 2016 16:44:20 +0100 Subject: RFR: 8171219: Missing checks in sparse array shift() implementation In-Reply-To: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> References: <41F4230B-F281-4A1E-A938-225E2D714ECD@oracle.com> Message-ID: <85FBF0AA-94D4-42B7-84CA-FF12D0F48FA6@lagergren.net> This is awesome Hannes! Thanks for getting to the bottom of this. +1 /M > On 15 Dec 2016, at 11:48, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8171219 > Webrev: http://cr.openjdk.java.net/~hannesw/8171219/webrev/ > > This fixes a number of bugs around the Array.prototype.shift on sparse array data (and its underlying dense data). Unlikely as it may seem, all these issues where exposed by the short test script included in this webrev: > > - UntouchedArrayData did not convert to a real ArrayData instance when ensuring place for first element (index 0) > - Dense array data did not check their size before trying System.arraycopy in shiftLeft > - Dense array data did not set the new length in shiftLeft, thereby leaving empty slots > - Sparse array data did not ensure space in the underlying dense array when moving elements from sparse to dense in shiftLeft > - Sparse array data did not delete new in-between elements when moving elements from sparse to dense in shiftLeft > - Sparse array data did not set the length after shrinking underlying dense data in shiftRight, thereby creating new empty slots > - DeletedRangeArrayData could not return the underlying data if it turned empty in shiftLeft, causing problems later on. > > I also moved the invocation of ensure() on the underlying dense array from SparseArrayData.ensure() to where it is actually required, which is cleaner and safe since underlying is a private field. > > Thanks, > Hannes From tongbao.ztb at alibaba-inc.com Tue Dec 20 12:22:44 2016 From: tongbao.ztb at alibaba-inc.com (=?UTF-8?B?5byg5ZCM5a6dKOS6leahkCk=?=) Date: Tue, 20 Dec 2016 20:22:44 +0800 Subject: =?UTF-8?B?RXhjZXB0aW9uSW5Jbml0aWFsaXplckVycm9yIHdoZW4gdXNpbmcgamRrLm5hc2hvcm4uaW50?= =?UTF-8?B?ZXJuYWwuaXIuZGVidWcuT2JqZWN0U2l6ZUNhbGN1bGF0b3IuZ2V0T2JqZWN0U2l6ZShvYmpl?= =?UTF-8?B?Y3QpIG9uIE9wZW5KREsgVk0=?= Message-ID: <56c26ed9-017e-43eb-9a32-3aa811df8f32.tongbao.ztb@alibaba-inc.com> Hi all, We get the?ExceptionInInitializerError when using the method jdk.nashorn.internal.ir.debug.ObjectSizeCalculator.getObjectSize on our openjdk build. the exception message says "ObjectSizeCalculator?only?supported?on?HotSpot?VM". After check the source code, we find the exception was throwed by?method getEffectiveMemoryLayoutSpecification:the code is: ? ?final?String?vmName?=?System.getProperty("java.vm.name");? ? ? ? ? ? ? ? ? ? ? ? if?(vmName?==?null?||?!vmName.startsWith("OpenJDK")) ? throw?new?UnsupportedOperationException()?here is the source code:http://hg.openjdk.java.net/jdk8u/jdk8u/nashorn/file/91d33aea2714/src/jdk/nashorn/internal/ir/debug/ObjectSizeCalculator.java#l397?, and the current jdk9 nashorn source code remains the same:?http://hg.openjdk.java.net/jdk9/jdk9/nashorn/file/a9c819c3c7f6/src/jdk.scripting.nashorn/share/classes/jdk/nashorn/internal/ir/debug/ObjectSizeCalculator.java#l397? Since we are building jdk in the openjdk repo and the?build's system property "java.vm.name" starts with "OpenJDK" (e.g., "OpenJDK?64-Bit?Server?VM"), we are considering?maybe this code can be changed to !vmName.startsWith("OpenJDK"), and the exception message should also be changed to "ObjectSizeCalculator only supported?on?OpenJDK VM" Thanks,Tongbao Zhang from Alibaba ? From sundararajan.athijegannathan at oracle.com Tue Dec 20 13:36:18 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Tue, 20 Dec 2016 19:06:18 +0530 Subject: RFR 8171503: Nashorn build, test failures with the latest jdk9-dev forest - javadoc target and test target fail Message-ID: <585933D2.1080104@oracle.com> Please review http://cr.openjdk.java.net/~sundar/8171503/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8171503 Thanks, -Sundar From james.laskey at oracle.com Tue Dec 20 13:48:33 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Tue, 20 Dec 2016 09:48:33 -0400 Subject: RFR 8171503: Nashorn build, test failures with the latest jdk9-dev forest - javadoc target and test target fail In-Reply-To: <585933D2.1080104@oracle.com> References: <585933D2.1080104@oracle.com> Message-ID: <20B361C4-510B-4E95-B2D2-9E2D0C6F11F2@oracle.com> +1 > On Dec 20, 2016, at 9:36 AM, Sundararajan Athijegannathan wrote: > > Please review http://cr.openjdk.java.net/~sundar/8171503/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8171503 > > Thanks, > -Sundar From srinivas.dama at oracle.com Tue Dec 20 15:01:06 2016 From: srinivas.dama at oracle.com (Srinivas Dama) Date: Tue, 20 Dec 2016 07:01:06 -0800 (PST) Subject: RFR: 8151994 (test/script/basic/JDK-8141209.js fails) Message-ID: Hi, Please review http://cr.openjdk.java.net/~sdama/8151994/webrev.00/ for https://bugs.openjdk.java.net/browse/JDK-8151994 Added @fork tag in test case since test runner without fork uses customized output stream to compare output, whereas test case uses System.out. Regards, Srinivas From esben at esbena.dk Wed Dec 21 10:09:22 2016 From: esben at esbena.dk (Esben Andreasen) Date: Wed, 21 Dec 2016 10:09:22 +0000 Subject: Surprising difference between /usr/bin/jjs and .getEngineByName("JavaScript") Message-ID: Hi I am not sure if this is a bug or a problem with my setup. I observe a surprising difference between the "external" (jjs) and the "internal" (new ScriptEngineManager().getEngineByName("JavaScript")) Nashorn when evaluating a regular expression method. The two Nashorns seems to be the same, but I am not sure how to check. Consider the following two interactions with Nashorn (output manually transformed to have the same form): ``` Evaluating in internal Nashorn (new ScriptEngineManager().getEngineByName("JavaScript")): > java.lang.System.getProperty('java.version') 1.8.0_111 > /(x)(y)?/.exec('x')[2] null Evaluating in external Nashorn (/usr/bin/jjs): > java.lang.System.getProperty('java.version') 1.8.0_111 > /(x)(y)?/.exec('x')[2] undefined ``` Observations: 1. the Java version for both interactions is the same. I *think* this means that the version of Nashorn is the same as well, but I am not sure. 2. the two interactions disagree on `null/undefined` as the second entry of the result array. The `undefined` value is the correct one according to other JavaScript engines. Questions: 1. Are the two Nashorns of the same version? 2. Is the null/undefined discrepancy a bug? --------------------------------------- Additional content: The interactions with the Nashorns where performed with the following Java class: ``` import javax.script.ScriptEngine; import javax.script.ScriptEngineManager; import javax.script.ScriptException; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.function.Function; public class NashornTest { public static void main(String[] args) throws ScriptException, IOException, InterruptedException { String javaVersionScript = "java.lang.System.getProperty('java.version')"; String execScript = "/(x)(y)?/.exec('x')[2]"; showEvaluation("Evaluating in internal Nashorn (new ScriptEngineManager().getEngineByName(\"JavaScript\")):", NashornTest::evaluateInternally, javaVersionScript, execScript); System.out.println(); showEvaluation("Evaluating in external Nashorn (/usr/bin/jjs):", NashornTest::evaluateExternally, javaVersionScript, execScript); } private static void showEvaluation(String title, Function evaluator, String... scripts) { System.out.println(title); for (String script : scripts) { System.out.println("\t> " + script); System.out.println("\t" + evaluator.apply(script)); } } private static String evaluateInternally(String script) { ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript"); try { return engine.eval(script) + ""; } catch (ScriptException e) { throw new RuntimeException(e); } } private static String evaluateExternally(String script) { Path scriptFile = null; try { scriptFile = Files.createTempFile("script", ".js"); Files.write(scriptFile, Arrays.asList(String.format("print(%s);", script))); Process process = Runtime.getRuntime().exec(new String[]{"/usr/bin/jjs", scriptFile.toAbsolutePath().toString()}); BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream())); List lines = new ArrayList<>(); String line; while ((line = stdInput.readLine()) != null) { lines.add(line); } return String.join("\n", lines); } catch (IOException e) { throw new RuntimeException(e); } } } ``` From hannes.wallnoefer at oracle.com Wed Dec 21 10:40:20 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Wed, 21 Dec 2016 11:40:20 +0100 Subject: Surprising difference between /usr/bin/jjs and .getEngineByName("JavaScript") In-Reply-To: References: Message-ID: Hi Esben, The reason you get null instead of undefined with the embedded script engine is that undefined is a language specific artefact that can?t be represented in Java. The script API translates undefined to null when it is returned by a script. Hannes > Am 21.12.2016 um 11:09 schrieb Esben Andreasen : > > Hi > > I am not sure if this is a bug or a problem with my setup. > I observe a surprising difference between the "external" (jjs) and the > "internal" (new ScriptEngineManager().getEngineByName("JavaScript")) > Nashorn when evaluating a regular expression method. The two Nashorns seems > to be the same, but I am not sure how to check. > > Consider the following two interactions with Nashorn (output manually > transformed to have the same form): > > ``` > Evaluating in internal Nashorn (new > ScriptEngineManager().getEngineByName("JavaScript")): >> java.lang.System.getProperty('java.version') > 1.8.0_111 >> /(x)(y)?/.exec('x')[2] > null > > Evaluating in external Nashorn (/usr/bin/jjs): >> java.lang.System.getProperty('java.version') > 1.8.0_111 >> /(x)(y)?/.exec('x')[2] > undefined > ``` > > Observations: > 1. the Java version for both interactions is the same. I *think* this means > that the version of Nashorn is the same as well, but I am not sure. > 2. the two interactions disagree on `null/undefined` as the second entry of > the result array. The `undefined` value is the correct one according to > other JavaScript engines. > > Questions: > 1. Are the two Nashorns of the same version? > 2. Is the null/undefined discrepancy a bug? > > --------------------------------------- > > Additional content: > > The interactions with the Nashorns where performed with the following Java > class: > > ``` > import javax.script.ScriptEngine; > import javax.script.ScriptEngineManager; > import javax.script.ScriptException; > import java.io.BufferedReader; > import java.io.IOException; > import java.io.InputStreamReader; > import java.nio.file.Files; > import java.nio.file.Path; > import java.util.ArrayList; > import java.util.Arrays; > import java.util.List; > import java.util.function.Function; > > public class NashornTest { > > public static void main(String[] args) throws ScriptException, IOException, > InterruptedException { > String javaVersionScript = "java.lang.System.getProperty('java.version')"; > String execScript = "/(x)(y)?/.exec('x')[2]"; > > showEvaluation("Evaluating in internal Nashorn (new > ScriptEngineManager().getEngineByName(\"JavaScript\")):", > NashornTest::evaluateInternally, javaVersionScript, execScript); > System.out.println(); > showEvaluation("Evaluating in external Nashorn (/usr/bin/jjs):", > NashornTest::evaluateExternally, javaVersionScript, execScript); > } > > private static void showEvaluation(String title, Function > evaluator, String... scripts) { > System.out.println(title); > for (String script : scripts) { > System.out.println("\t> " + script); > System.out.println("\t" + evaluator.apply(script)); > } > } > > private static String evaluateInternally(String script) { > ScriptEngine engine = new > ScriptEngineManager().getEngineByName("JavaScript"); > try { > return engine.eval(script) + ""; > } catch (ScriptException e) { > throw new RuntimeException(e); > } > } > > private static String evaluateExternally(String script) { > Path scriptFile = null; > try { > scriptFile = Files.createTempFile("script", ".js"); > Files.write(scriptFile, Arrays.asList(String.format("print(%s);", script))); > > Process process = Runtime.getRuntime().exec(new String[]{"/usr/bin/jjs", > scriptFile.toAbsolutePath().toString()}); > BufferedReader stdInput = new BufferedReader(new > InputStreamReader(process.getInputStream())); > List lines = new ArrayList<>(); > String line; > while ((line = stdInput.readLine()) != null) { > lines.add(line); > } > return String.join("\n", lines); > } catch (IOException e) { > throw new RuntimeException(e); > } > } > } > > ``` From hannes.wallnoefer at oracle.com Wed Dec 21 11:45:20 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Wed, 21 Dec 2016 12:45:20 +0100 Subject: RFR: 8151994 (test/script/basic/JDK-8141209.js fails) In-Reply-To: References: Message-ID: <6C02ACD4-9BCF-49E6-8A2C-559E4D587AAB@oracle.com> +1 Hannes > Am 20.12.2016 um 16:01 schrieb Srinivas Dama : > > Hi, > > Please review http://cr.openjdk.java.net/~sdama/8151994/webrev.00/ > for https://bugs.openjdk.java.net/browse/JDK-8151994 > > Added @fork tag in test case since test runner without fork uses customized output stream to compare output, whereas test case uses System.out. > > Regards, > Srinivas From esben at esbena.dk Wed Dec 21 12:19:38 2016 From: esben at esbena.dk (Esben Andreasen) Date: Wed, 21 Dec 2016 12:19:38 +0000 Subject: Surprising difference between /usr/bin/jjs and .getEngineByName("JavaScript") In-Reply-To: References: Message-ID: Hi Hannes Thank you for that explanation. A follow-up, if allowed: I understand the motivation for eliminating `undefined`, but I would really like a way that I could avoid that elimination. Is that a discouraged use of the API? Implementation-wise: I see that `ScriptObjectMirror.translateUndefined` does the conversion from JavaScript `undefined` to Java `null`. I assume I could make a lightly edited copy of NashornScriptEngine.java that omits the call to `translateUndefined` to get the effect I want. Or is there a more elegant solution somewhere? On Wed, Dec 21, 2016 at 11:40 AM Hannes Walln?fer < hannes.wallnoefer at oracle.com> wrote: > Hi Esben, > > The reason you get null instead of undefined with the embedded script > engine is that undefined is a language specific artefact that can?t be > represented in Java. The script API translates undefined to null when it is > returned by a script. > > Hannes > > > > Am 21.12.2016 <21%2012%2020%2016> um 11:09 schrieb Esben Andreasen < > esben at esbena.dk>: > > > > Hi > > > > I am not sure if this is a bug or a problem with my setup. > > I observe a surprising difference between the "external" (jjs) and the > > "internal" (new ScriptEngineManager().getEngineByName("JavaScript")) > > Nashorn when evaluating a regular expression method. The two Nashorns > seems > > to be the same, but I am not sure how to check. > > > > Consider the following two interactions with Nashorn (output manually > > transformed to have the same form): > > > > ``` > > Evaluating in internal Nashorn (new > > ScriptEngineManager().getEngineByName("JavaScript")): > >> java.lang.System.getProperty('java.version') > > 1.8.0_111 > >> /(x)(y)?/.exec('x')[2] > > null > > > > Evaluating in external Nashorn (/usr/bin/jjs): > >> java.lang.System.getProperty('java.version') > > 1.8.0_111 > >> /(x)(y)?/.exec('x')[2] > > undefined > > ``` > > > > Observations: > > 1. the Java version for both interactions is the same. I *think* this > means > > that the version of Nashorn is the same as well, but I am not sure. > > 2. the two interactions disagree on `null/undefined` as the second entry > of > > the result array. The `undefined` value is the correct one according to > > other JavaScript engines. > > > > Questions: > > 1. Are the two Nashorns of the same version? > > 2. Is the null/undefined discrepancy a bug? > > > > --------------------------------------- > > > > Additional content: > > > > The interactions with the Nashorns where performed with the following > Java > > class: > > > > ``` > > import javax.script.ScriptEngine; > > import javax.script.ScriptEngineManager; > > import javax.script.ScriptException; > > import java.io.BufferedReader; > > import java.io.IOException; > > import java.io.InputStreamReader; > > import java.nio.file.Files; > > import java.nio.file.Path; > > import java.util.ArrayList; > > import java.util.Arrays; > > import java.util.List; > > import java.util.function.Function; > > > > public class NashornTest { > > > > public static void main(String[] args) throws ScriptException, > IOException, > > InterruptedException { > > String javaVersionScript = > "java.lang.System.getProperty('java.version')"; > > String execScript = "/(x)(y)?/.exec('x')[2]"; > > > > showEvaluation("Evaluating in internal Nashorn (new > > ScriptEngineManager().getEngineByName(\"JavaScript\")):", > > NashornTest::evaluateInternally, javaVersionScript, execScript); > > System.out.println(); > > showEvaluation("Evaluating in external Nashorn (/usr/bin/jjs):", > > NashornTest::evaluateExternally, javaVersionScript, execScript); > > } > > > > private static void showEvaluation(String title, Function > > evaluator, String... scripts) { > > System.out.println(title); > > for (String script : scripts) { > > System.out.println("\t> " + script); > > System.out.println("\t" + evaluator.apply(script)); > > } > > } > > > > private static String evaluateInternally(String script) { > > ScriptEngine engine = new > > ScriptEngineManager().getEngineByName("JavaScript"); > > try { > > return engine.eval(script) + ""; > > } catch (ScriptException e) { > > throw new RuntimeException(e); > > } > > } > > > > private static String evaluateExternally(String script) { > > Path scriptFile = null; > > try { > > scriptFile = Files.createTempFile("script", ".js"); > > Files.write(scriptFile, Arrays.asList(String.format("print(%s);", > script))); > > > > Process process = Runtime.getRuntime().exec(new String[]{"/usr/bin/jjs", > > scriptFile.toAbsolutePath().toString()}); > > BufferedReader stdInput = new BufferedReader(new > > InputStreamReader(process.getInputStream())); > > List lines = new ArrayList<>(); > > String line; > > while ((line = stdInput.readLine()) != null) { > > lines.add(line); > > } > > return String.join("\n", lines); > > } catch (IOException e) { > > throw new RuntimeException(e); > > } > > } > > } > > > > ``` > > From sundararajan.athijegannathan at oracle.com Wed Dec 21 12:43:24 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Wed, 21 Dec 2016 18:13:24 +0530 Subject: Surprising difference between /usr/bin/jjs and .getEngineByName("JavaScript") In-Reply-To: References: Message-ID: <585A78EC.6050701@oracle.com> No. We do not discourage javax.script API. javax.script is the main API. Nashorn specific extension API is in jdk.nashorn.api.* packages. This undefined to null translation was already discussed -> http://mail.openjdk.java.net/pipermail/nashorn-dev/2016-December/006707.html javax.script API - which is scripting language independent API - translates undefined to null. Nashorn specific API jdk.nashorn.api.scripting.* does not. jjs, which is a nashorn specific shell, does not translate undefined. -Sundar On 21/12/16, 5:49 PM, Esben Andreasen wrote: > Hi Hannes > > Thank you for that explanation. > > A follow-up, if allowed: > I understand the motivation for eliminating `undefined`, but I would really > like a way that I could avoid that elimination. Is that a discouraged use > of the API? > > Implementation-wise: > I see that `ScriptObjectMirror.translateUndefined` does the conversion > from JavaScript `undefined` to Java `null`. > I assume I could make a lightly edited copy of NashornScriptEngine.java > that omits the call to `translateUndefined` to get the effect I want. Or is > there a more elegant solution somewhere? > > > On Wed, Dec 21, 2016 at 11:40 AM Hannes Walln?fer< > hannes.wallnoefer at oracle.com> wrote: > >> Hi Esben, >> >> The reason you get null instead of undefined with the embedded script >> engine is that undefined is a language specific artefact that can?t be >> represented in Java. The script API translates undefined to null when it is >> returned by a script. >> >> Hannes >> >> >>> Am 21.12.2016<21%2012%2020%2016> um 11:09 schrieb Esben Andreasen< >> esben at esbena.dk>: >>> Hi >>> >>> I am not sure if this is a bug or a problem with my setup. >>> I observe a surprising difference between the "external" (jjs) and the >>> "internal" (new ScriptEngineManager().getEngineByName("JavaScript")) >>> Nashorn when evaluating a regular expression method. The two Nashorns >> seems >>> to be the same, but I am not sure how to check. >>> >>> Consider the following two interactions with Nashorn (output manually >>> transformed to have the same form): >>> >>> ``` >>> Evaluating in internal Nashorn (new >>> ScriptEngineManager().getEngineByName("JavaScript")): >>>> java.lang.System.getProperty('java.version') >>> 1.8.0_111 >>>> /(x)(y)?/.exec('x')[2] >>> null >>> >>> Evaluating in external Nashorn (/usr/bin/jjs): >>>> java.lang.System.getProperty('java.version') >>> 1.8.0_111 >>>> /(x)(y)?/.exec('x')[2] >>> undefined >>> ``` >>> >>> Observations: >>> 1. the Java version for both interactions is the same. I *think* this >> means >>> that the version of Nashorn is the same as well, but I am not sure. >>> 2. the two interactions disagree on `null/undefined` as the second entry >> of >>> the result array. The `undefined` value is the correct one according to >>> other JavaScript engines. >>> >>> Questions: >>> 1. Are the two Nashorns of the same version? >>> 2. Is the null/undefined discrepancy a bug? >>> >>> --------------------------------------- >>> >>> Additional content: >>> >>> The interactions with the Nashorns where performed with the following >> Java >>> class: >>> >>> ``` >>> import javax.script.ScriptEngine; >>> import javax.script.ScriptEngineManager; >>> import javax.script.ScriptException; >>> import java.io.BufferedReader; >>> import java.io.IOException; >>> import java.io.InputStreamReader; >>> import java.nio.file.Files; >>> import java.nio.file.Path; >>> import java.util.ArrayList; >>> import java.util.Arrays; >>> import java.util.List; >>> import java.util.function.Function; >>> >>> public class NashornTest { >>> >>> public static void main(String[] args) throws ScriptException, >> IOException, >>> InterruptedException { >>> String javaVersionScript = >> "java.lang.System.getProperty('java.version')"; >>> String execScript = "/(x)(y)?/.exec('x')[2]"; >>> >>> showEvaluation("Evaluating in internal Nashorn (new >>> ScriptEngineManager().getEngineByName(\"JavaScript\")):", >>> NashornTest::evaluateInternally, javaVersionScript, execScript); >>> System.out.println(); >>> showEvaluation("Evaluating in external Nashorn (/usr/bin/jjs):", >>> NashornTest::evaluateExternally, javaVersionScript, execScript); >>> } >>> >>> private static void showEvaluation(String title, Function >>> evaluator, String... scripts) { >>> System.out.println(title); >>> for (String script : scripts) { >>> System.out.println("\t> " + script); >>> System.out.println("\t" + evaluator.apply(script)); >>> } >>> } >>> >>> private static String evaluateInternally(String script) { >>> ScriptEngine engine = new >>> ScriptEngineManager().getEngineByName("JavaScript"); >>> try { >>> return engine.eval(script) + ""; >>> } catch (ScriptException e) { >>> throw new RuntimeException(e); >>> } >>> } >>> >>> private static String evaluateExternally(String script) { >>> Path scriptFile = null; >>> try { >>> scriptFile = Files.createTempFile("script", ".js"); >>> Files.write(scriptFile, Arrays.asList(String.format("print(%s);", >> script))); >>> Process process = Runtime.getRuntime().exec(new String[]{"/usr/bin/jjs", >>> scriptFile.toAbsolutePath().toString()}); >>> BufferedReader stdInput = new BufferedReader(new >>> InputStreamReader(process.getInputStream())); >>> List lines = new ArrayList<>(); >>> String line; >>> while ((line = stdInput.readLine()) != null) { >>> lines.add(line); >>> } >>> return String.join("\n", lines); >>> } catch (IOException e) { >>> throw new RuntimeException(e); >>> } >>> } >>> } >>> >>> ``` >> From hannes.wallnoefer at oracle.com Thu Dec 22 15:15:00 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 22 Dec 2016 16:15:00 +0100 Subject: RFR: 8170977: SparseArrayData should not grow its underlying dense array data Message-ID: <508DB2AC-3116-41A3-A9ED-77BCFEE93CF0@oracle.com> Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8170977 Webrev: http://cr.openjdk.java.net/~hannesw/8170977/webrev/ With this we keep whatever is already allocated as dense array but do not grow it up to the dense array length limit internally. I also reduced the max dense array length as with 1 million dense elements we still risk to be very wasteful with memory. Thanks, Hannes From hannes.wallnoefer at oracle.com Thu Dec 22 15:18:37 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 22 Dec 2016 16:18:37 +0100 Subject: RFR: 8166187: Regression: NPE during reparse when using persistent code cache and optimistic types Message-ID: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8166187 Webrev: http://cr.openjdk.java.net/~hannesw/8166187/webrev/ It was actually the combination of having a non-serialisable AST reference and not initialising the transient fields of nested functions that caused this error. Thanks, Hannes From james.laskey at oracle.com Thu Dec 22 15:24:56 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 22 Dec 2016 11:24:56 -0400 Subject: RFR: 8170977: SparseArrayData should not grow its underlying dense array data In-Reply-To: <508DB2AC-3116-41A3-A9ED-77BCFEE93CF0@oracle.com> References: <508DB2AC-3116-41A3-A9ED-77BCFEE93CF0@oracle.com> Message-ID: +1 > On Dec 22, 2016, at 11:15 AM, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8170977 > Webrev: http://cr.openjdk.java.net/~hannesw/8170977/webrev/ > > With this we keep whatever is already allocated as dense array but do not grow it up to the dense array length limit internally. I also reduced the max dense array length as with 1 million dense elements we still risk to be very wasteful with memory. > > Thanks, > Hannes From james.laskey at oracle.com Thu Dec 22 15:25:27 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 22 Dec 2016 11:25:27 -0400 Subject: RFR: 8166187: Regression: NPE during reparse when using persistent code cache and optimistic types In-Reply-To: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> References: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> Message-ID: +1 > On Dec 22, 2016, at 11:18 AM, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8166187 > Webrev: http://cr.openjdk.java.net/~hannesw/8166187/webrev/ > > It was actually the combination of having a non-serialisable AST reference and not initialising the transient fields of nested functions that caused this error. > > Thanks, > Hannes From szegedia at gmail.com Thu Dec 22 15:30:41 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 22 Dec 2016 16:30:41 +0100 Subject: RFR: 8170977: SparseArrayData should not grow its underlying dense array data In-Reply-To: <508DB2AC-3116-41A3-A9ED-77BCFEE93CF0@oracle.com> References: <508DB2AC-3116-41A3-A9ED-77BCFEE93CF0@oracle.com> Message-ID: +1 > On 22 Dec 2016, at 16:15, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8170977 > Webrev: http://cr.openjdk.java.net/~hannesw/8170977/webrev/ > > With this we keep whatever is already allocated as dense array but do not grow it up to the dense array length limit internally. I also reduced the max dense array length as with 1 million dense elements we still risk to be very wasteful with memory. > > Thanks, > Hannes From szegedia at gmail.com Thu Dec 22 15:48:32 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 22 Dec 2016 16:48:32 +0100 Subject: RFR: 8166187: Regression: NPE during reparse when using persistent code cache and optimistic types In-Reply-To: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> References: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> Message-ID: Hm? cachedAst is essential for split functions; specifically if it contains a SerializedAst, then its "byte[] serializedAst" field is essential. If cachedAst is lost, we can reparse an unsplit function from source, but we can?t reparse fragments of split functions. I?d suggest instead of making cachedAst transient, we should: 1. make SerializedAst Serializable, and have SerializedAst.cachedAst within it transient (reference objects aren?t serializable, and we can afford to lose it anyway). 2. introduce RecompilableScriptFunctionData.writeObject and make sure that if serializedAst contains a reference (instead of a SerializedAst object) then we don?t attempt to serialize it ? write null instead (this can be accomplished by just setting serializedAst = null, and maybe re-setting it back to what it was after defaultWriteObject) 3. make sure there?s a null check on SerializedAst.cachedAst read in getCachedAst() (as now it can actually be null on deserialization, it was an invariant that it was never null before) Attila. > On 22 Dec 2016, at 16:18, Hannes Walln?fer wrote: > > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8166187 > Webrev: http://cr.openjdk.java.net/~hannesw/8166187/webrev/ > > It was actually the combination of having a non-serialisable AST reference and not initialising the transient fields of nested functions that caused this error. > > Thanks, > Hannes From szegedia at gmail.com Thu Dec 22 16:32:34 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Thu, 22 Dec 2016 17:32:34 +0100 Subject: Review request for JDK-8171849: Collection and Queue conversions not prioritized for Arrays Message-ID: Please review JDK-8171849 "Collection and Queue conversions not prioritized for Arrays" at for Thanks, Attila. From james.laskey at oracle.com Thu Dec 22 16:44:19 2016 From: james.laskey at oracle.com (Jim Laskey (Oracle)) Date: Thu, 22 Dec 2016 12:44:19 -0400 Subject: Review request for JDK-8171849: Collection and Queue conversions not prioritized for Arrays In-Reply-To: References: Message-ID: +1 > On Dec 22, 2016, at 12:32 PM, Attila Szegedi wrote: > > Please review JDK-8171849 "Collection and Queue conversions not prioritized for Arrays" at for > > Thanks, > Attila. From hannes.wallnoefer at oracle.com Thu Dec 22 17:12:43 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Thu, 22 Dec 2016 18:12:43 +0100 Subject: Review request for JDK-8171849: Collection and Queue conversions not prioritized for Arrays In-Reply-To: References: Message-ID: <955A0C30-D257-4FF6-9388-9398FBABA0D2@oracle.com> +1 Thanks! Hannes > Am 22.12.2016 um 17:44 schrieb Jim Laskey (Oracle) : > > +1 > >> On Dec 22, 2016, at 12:32 PM, Attila Szegedi wrote: >> >> Please review JDK-8171849 "Collection and Queue conversions not prioritized for Arrays" at for >> >> Thanks, >> Attila. > From srinivas.dama at oracle.com Thu Dec 22 18:23:15 2016 From: srinivas.dama at oracle.com (Srinivas Dama) Date: Thu, 22 Dec 2016 10:23:15 -0800 (PST) Subject: RFR: 8151994 (test/script/basic/JDK-8141209.js fails) In-Reply-To: <6C02ACD4-9BCF-49E6-8A2C-559E4D587AAB@oracle.com> References: <6C02ACD4-9BCF-49E6-8A2C-559E4D587AAB@oracle.com> Message-ID: Hi, Please review http://cr.openjdk.java.net/~sdama/8151994/webrev.01/ for https://bugs.openjdk.java.net/browse/JDK-8151994 Moved earlier fix to test/script/trusted directory. Regards, Srinivas -----Original Message----- From: Hannes Walln?fer Sent: Wednesday, December 21, 2016 5:15 PM To: Srinivas Dama Cc: Nashorn-dev Subject: Re: RFR: 8151994 (test/script/basic/JDK-8141209.js fails) +1 Hannes > Am 20.12.2016 um 16:01 schrieb Srinivas Dama : > > Hi, > > Please review http://cr.openjdk.java.net/~sdama/8151994/webrev.00/ > for https://bugs.openjdk.java.net/browse/JDK-8151994 > > Added @fork tag in test case since test runner without fork uses customized output stream to compare output, whereas test case uses System.out. > > Regards, > Srinivas From hannes.wallnoefer at oracle.com Fri Dec 23 11:20:05 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Fri, 23 Dec 2016 12:20:05 +0100 Subject: RFR: 8166187: Regression: NPE during reparse when using persistent code cache and optimistic types In-Reply-To: References: <9B0F648B-71AE-432F-87BA-EB8647780427@oracle.com> Message-ID: <6EADC1F8-634B-44F8-84CD-013A1170F298@oracle.com> Thanks Attila, I had forgotten cached AST is not just a performance feature. I uploaded a new webrev. It pretty much follows your suggestions, except I used a slightly different approach to conditional serialization of cachedAst - I left the field transient and serialize it explicitly if it is an instance of SerializedAst, otherwise write out null. I also added a test case for split stored functions. http://cr.openjdk.java.net/~hannesw/8170977/webrev.01/ Hannes > Am 22.12.2016 um 16:48 schrieb Attila Szegedi : > > Hm? cachedAst is essential for split functions; specifically if it contains a SerializedAst, then its "byte[] serializedAst" field is essential. If cachedAst is lost, we can reparse an unsplit function from source, but we can?t reparse fragments of split functions. > > I?d suggest instead of making cachedAst transient, we should: > 1. make SerializedAst Serializable, and have SerializedAst.cachedAst within it transient (reference objects aren?t serializable, and we can afford to lose it anyway). > 2. introduce RecompilableScriptFunctionData.writeObject and make sure that if serializedAst contains a reference (instead of a SerializedAst object) then we don?t attempt to serialize it ? write null instead (this can be accomplished by just setting serializedAst = null, and maybe re-setting it back to what it was after defaultWriteObject) > 3. make sure there?s a null check on SerializedAst.cachedAst read in getCachedAst() (as now it can actually be null on deserialization, it was an invariant that it was never null before) > > Attila. > >> On 22 Dec 2016, at 16:18, Hannes Walln?fer wrote: >> >> Please review: >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8166187 >> Webrev: http://cr.openjdk.java.net/~hannesw/8166187/webrev/ >> >> It was actually the combination of having a non-serialisable AST reference and not initialising the transient fields of nested functions that caused this error. >> >> Thanks, >> Hannes > From hannes.wallnoefer at oracle.com Fri Dec 23 13:36:44 2016 From: hannes.wallnoefer at oracle.com (=?utf-8?Q?Hannes_Walln=C3=B6fer?=) Date: Fri, 23 Dec 2016 14:36:44 +0100 Subject: RFR: 8170781: PropertyMapIterator throws NoSuchElementException on last element Message-ID: <256BCB40-5FA4-4ED5-BA6A-EA831315E002@oracle.com> Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8170781 Webrev: http://cr.openjdk.java.net/~hannesw/8170781/webrev/ Thanks, Hannes From sundararajan.athijegannathan at oracle.com Fri Dec 23 15:08:36 2016 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Fri, 23 Dec 2016 20:38:36 +0530 Subject: RFR: 8170781: PropertyMapIterator throws NoSuchElementException on last element In-Reply-To: <256BCB40-5FA4-4ED5-BA6A-EA831315E002@oracle.com> References: <256BCB40-5FA4-4ED5-BA6A-EA831315E002@oracle.com> Message-ID: <585D3DF4.9070505@oracle.com> Because this is a java test, you can add jtreg declarations (you may want to check other tests with @bug, @test comments under test/src) to enable this test for jtreg mode. Other than that, +1 -Sundar On 23/12/16, 7:06 PM, Hannes Walln?fer wrote: > Please review: > > Bug: https://bugs.openjdk.java.net/browse/JDK-8170781 > Webrev: http://cr.openjdk.java.net/~hannesw/8170781/webrev/ > > Thanks, > Hannes From szegedia at gmail.com Mon Dec 26 23:52:16 2016 From: szegedia at gmail.com (Attila Szegedi) Date: Mon, 26 Dec 2016 23:52:16 +0000 Subject: RFR: 8170781: PropertyMapIterator throws NoSuchElementException on last element In-Reply-To: <585D3DF4.9070505@oracle.com> References: <256BCB40-5FA4-4ED5-BA6A-EA831315E002@oracle.com> <585D3DF4.9070505@oracle.com> Message-ID: +1 from me. Attila. On Fri, Dec 23, 2016 at 15:57 Sundararajan Athijegannathan < sundararajan.athijegannathan at oracle.com> wrote: > Because this is a java test, you can add jtreg declarations (you may > > want to check other tests with @bug, @test comments under test/src) to > > enable this test for jtreg mode. > > > > Other than that, +1 > > > > -Sundar > > > > On 23/12/16, 7:06 PM, Hannes Walln?fer wrote: > > > Please review: > > > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8170781 > > > Webrev: http://cr.openjdk.java.net/~hannesw/8170781/webrev/ > > > > > > Thanks, > > > Hannes > >