From martijnverburg at gmail.com Tue May 3 09:01:09 2016 From: martijnverburg at gmail.com (Martijn Verburg) Date: Tue, 3 May 2016 10:01:09 +0100 Subject: Monthly notice for new signups to Project Valhalla - How to Contribute Message-ID: Hi all, Valhalla is one of OpenJDK's more complex areas to get involved in. A guide to getting started with Valhalla and levels of suggested contribution can be found on the home page: http://openjdk.java.net/projects/valhalla/ Cheers, Martijn From maurizio.cimadamore at oracle.com Fri May 27 13:18:31 2016 From: maurizio.cimadamore at oracle.com (maurizio.cimadamore at oracle.com) Date: Fri, 27 May 2016 13:18:31 +0000 Subject: hg: valhalla/valhalla/jdk: Add support for species static access. Message-ID: <201605271318.u4RDIVaW000393@aojmv0008.oracle.com> Changeset: 836efde938c1 Author: mcimadamore Date: 2016-05-27 14:18 +0100 URL: http://hg.openjdk.java.net/valhalla/valhalla/jdk/rev/836efde938c1 Add support for species static access. * add support for species static in Model3Converter * add species static support in private copy of classfile library * optimize bootstrap method for dispatching specializable static/species static calls ! src/java.base/share/classes/java/lang/invoke/GenericStaticDispatch.java ! src/java.base/share/classes/valhalla/model3/Model3Converter.java ! src/java.base/share/classes/valhalla/model3/classfile/AccessFlags.java From maurizio.cimadamore at oracle.com Fri May 27 13:19:04 2016 From: maurizio.cimadamore at oracle.com (maurizio.cimadamore at oracle.com) Date: Fri, 27 May 2016 13:19:04 +0000 Subject: hg: valhalla/valhalla/langtools: Add support for species static access. Message-ID: <201605271319.u4RDJ4Pn000583@aojmv0008.oracle.com> Changeset: 6949c3d06e8f Author: mcimadamore Date: 2016-05-27 14:18 +0100 URL: http://hg.openjdk.java.net/valhalla/valhalla/langtools/rev/6949c3d06e8f Add support for species static access. * add parser support for new '__species' modifier * add resolution and type-checking support for species access members * add code generation support for species static * optimize translation of generic specializable methods (static and species static) to use species static * cleanup accessor bridge generation using species static * cleanup BootstrapMethodSymbol * remove dead code in ClassWriter * add javap support for species statics * add tests ! src/java.compiler/share/classes/javax/lang/model/element/Modifier.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/code/Flags.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/code/Symbol.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/code/SymbolMetadata.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/code/Types.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Attr.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/AttrContext.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Check.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Enter.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/IndifierTranslator.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/LambdaToMethod.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Lower.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/MemberEnter.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Resolve.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/SpecializeTypes.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/comp/TransTypes.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassReader.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassWriter.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Gen.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Items.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/PoolWriter.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/StringConcat.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/parser/JavacParser.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/parser/Tokens.java ! src/jdk.compiler/share/classes/com/sun/tools/javac/resources/compiler.properties ! src/jdk.compiler/share/classes/com/sun/tools/javac/util/Names.java ! src/jdk.jdeps/share/classes/com/sun/tools/classfile/AccessFlags.java ! src/jdk.jshell/share/classes/jdk/jshell/CompletenessAnalyzer.java - src/jdk.jshell/share/classes/jdk/jshell/ReplResolve.java ! src/jdk.jshell/share/classes/jdk/jshell/SourceCodeAnalysisImpl.java + test/tools/javac/annotations/typeAnnotations/classfile/TypeInitializer.java + test/tools/javac/diags/examples/TypeStaticCantBeRef.java + test/tools/javac/diags/examples/UncheckedGenericSpeciesAccess.java ! test/tools/javac/lambda/TestBootstrapMethodsCount.java ! test/tools/javac/lambda/TestInvokeDynamic.java + test/tools/javac/valhalla/speciesstatic/SpeciesAccessParserTest.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic01.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic01.out + test/tools/javac/valhalla/speciesstatic/SpeciesStatic02.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic03.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic03.out + test/tools/javac/valhalla/speciesstatic/SpeciesStatic04.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic04.out + test/tools/javac/valhalla/speciesstatic/SpeciesStatic05.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic05.out + test/tools/javac/valhalla/speciesstatic/SpeciesStatic06.java + test/tools/javac/valhalla/speciesstatic/SpeciesStatic06.out + test/tools/javac/valhalla/speciesstatic/separate/Foo.java + test/tools/javac/valhalla/speciesstatic/separate/Separate01.java ! test/tools/javac/valhalla/typespec/items/m3/Opcodes.java ! test/tools/javac/valhalla/typespec/items/m3/tests/TestArrayLoadAndStore.java ! test/tools/javac/valhalla/typespec/items/m3/tests/TestDup.java ! test/tools/javac/valhalla/typespec/items/m3/tests/TestGeneric2GenericCall.java ! test/tools/javac/valhalla/typespec/items/m3/tests/TestLoadAndStore.java ! test/tools/javac/valhalla/typespec/items/m3/tests/TestNonSpecializedGenericCall.java + test/tools/javac/valhalla/typespec/items/m3/tests/TestSpeciesStatic.java From maurizio.cimadamore at oracle.com Fri May 27 20:59:25 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 27 May 2016 21:59:25 +0100 Subject: species static - first prototype Message-ID: <5748B52D.8040409@oracle.com> Hi, I've pushed compiler and runtime support for the species static feature. More details on the prototype are available in the writeup sent to the valhalla-spec-experts mailing list [1]. This is a very early implementation effort and it will likely contain many issues. Please share your experience on the usage of such feature. Cheers Maurizio [1] - http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2016-May/000148.html From david.simms at oracle.com Mon May 30 13:10:53 2016 From: david.simms at oracle.com (david.simms at oracle.com) Date: Mon, 30 May 2016 13:10:53 +0000 Subject: hg: valhalla/valhalla/hotspot: Re-enable Valhalla tests Message-ID: <201605301310.u4UDAr5U015261@aojmv0008.oracle.com> Changeset: 430f437aface Author: dsimms Date: 2016-05-30 14:52 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/hotspot/rev/430f437aface Re-enable Valhalla tests ! test/TEST.groups From david.simms at oracle.com Mon May 30 13:11:04 2016 From: david.simms at oracle.com (david.simms at oracle.com) Date: Mon, 30 May 2016 13:11:04 +0000 Subject: hg: valhalla/valhalla/jdk: Arrayish related test fixes Message-ID: <201605301311.u4UDB4pH015419@aojmv0008.oracle.com> Changeset: 4aa612ffd864 Author: dsimms Date: 2016-05-30 15:09 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/jdk/rev/4aa612ffd864 Arrayish related test fixes ! src/java.base/share/classes/java/lang/Arrayish.java ! test/java/lang/Class/ArrayMethods.java ! test/java/lang/Class/getDeclaredField/FieldSetAccessibleTest.java From rsmogura at gmail.com Mon May 30 18:48:10 2016 From: rsmogura at gmail.com (Radek) Date: Mon, 30 May 2016 20:48:10 +0200 Subject: Java shared memory Message-ID: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> Dear all, Please don?t think I?m so crazy. Recently I was working on JVM changes to add shared memory / memory mapping support. I?m able to map part of files as primitive java arrays. First performance results are very optimistic (40% boost in searching min-max long in 1GB file) with mapped long[] as a replacement to LongBuffer. The work is not finished. I have adopted G1, c1 and c2, but I have got mainly stable solution (or I think so). I see three possible benefits of shared memory: 1. Shared memory and interprocess communication. 2. Mapping large files. 3. Aggressive inter-JVM string de-duplication. I?m looking for sponsorship for this change, with some guidelines how I can move forward with possible adoption into JDK. If solution would be viable, I would like to open JSR for JDK API modification. I use https://bitbucket.org/radoslaw_smogura/java-shared-memory/wiki/Home as umbrella repo, but I don?t update it quite often. Thanks in advance, Radek From aph at redhat.com Mon May 30 20:10:41 2016 From: aph at redhat.com (Andrew Haley) Date: Mon, 30 May 2016 21:10:41 +0100 Subject: Java shared memory In-Reply-To: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> Message-ID: <574C9E41.30302@redhat.com> On 30/05/16 19:48, Radek wrote: > I?m able to map part of files as primitive java arrays. First > performance results are very optimistic (40% boost in searching > min-max long in 1GB file) with mapped long[] as a replacement to > LongBuffer. It shouldn't be significantly different in performance. Perhaps you can share your benchmark with us. Maybe we're missing vectorization opportunities. Andrew. From rsmogura at gmail.com Mon May 30 22:57:26 2016 From: rsmogura at gmail.com (Radek) Date: Tue, 31 May 2016 00:57:26 +0200 Subject: Java shared memory In-Reply-To: <574C9E41.30302@redhat.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <574C9E41.30302@redhat.com> Message-ID: Dear Andrew, I?ve just modified test case to convert bytes to long in code. The results are comparable. Attached please find out files. Those are named as follow: * memorymap - from MemoryMapMinMax, file map from FileMapMinMax * memorymap2, filemap2 form *Bytes version respectively I?m attaching text files with results, and test classes (those are my test classes, so not nicely formatted). I could do something wrong with C2, but I think results are expected as NIO in back copies mapped array to Java array, does it? Best regards, Radek -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: filemap2_100000_jdkmod_oopcomp1_feature0.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: filemap2_100000_jdkmod_oopcomp1_feature0.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: memorymap_134217728_jdk9mod_oopcomp1_feature1.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: memorymap2_134217728_jdk9mod_oopcomp1_feature1.txt URL: -------------- next part -------------- -------------- next part -------------- > On 30 May 2016, at 22:10, Andrew Haley wrote: > > On 30/05/16 19:48, Radek wrote: > >> I?m able to map part of files as primitive java arrays. First >> performance results are very optimistic (40% boost in searching >> min-max long in 1GB file) with mapped long[] as a replacement to >> LongBuffer. > > It shouldn't be significantly different in performance. Perhaps you > can share your benchmark with us. Maybe we're missing vectorization > opportunities. > > Andrew. From martijnverburg at gmail.com Tue May 31 08:06:26 2016 From: martijnverburg at gmail.com (Martijn Verburg) Date: Tue, 31 May 2016 09:06:26 +0100 Subject: Monthly notice for new signups to Valhalla Message-ID: Hi all, Valhalla is one of OpenJDK's more complex areas to get involved in. A guide to getting started with Valhalla and levels of suggested contribution can be found on the home page: http://openjdk.java.net/projects/valhalla/ Cheers, Martijn From paul.sandoz at oracle.com Tue May 31 08:27:21 2016 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 31 May 2016 10:27:21 +0200 Subject: Java shared memory In-Reply-To: References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <574C9E41.30302@redhat.com> Message-ID: Hi Radek, > On 31 May 2016, at 00:57, Radek wrote: > > Dear Andrew, > > I?ve just modified test case to convert bytes to long in code. The results are comparable. > > Attached please find out files. Those are named as follow: > * memorymap - from MemoryMapMinMax, file map from FileMapMinMax > * memorymap2, filemap2 form *Bytes version respectively > > I?m attaching text files with results, and test classes (those are my test classes, so not nicely formatted). > > I could do something wrong with C2, but I think results are expected as NIO in back copies mapped array to Java array, does it? > Like Andrew my expectation was that a long view over a mapped byte buffer should not be significantly different. (Note: it?s hard to evaluate these kind of things without using JMH and also looking at the generated code). Here is your main loop for LongBuffer: while (lbuff.hasRemaining()) { lbuff.get(ll); for (long l : ll) { if (l < min) min = l; if (l > max) max = l; lastNumber = l; } } You are bulk copying into an array, that alone could explain an ~2x performance difference. Instead, use the indexed accessors LongBuffer.get(int ) e.g.something like: for (int i = 0; i < lbuffer.limit(); i++) { l = lbuff.get(i); if (l < min) min = l; if (l > max) max = l; lastNumber = l; ... } If you take a closer look at the buffer source you will notice long views over byte buffers are optimized in certain cases to direct calls to Unsafe.get/setLong. While lbuff.get(i) is not ll[i], it?s not far off :-) Relatedly, see the recent work by Mr. (David) Simms on this list, so that int[] implements Arrayish. I dunno if this can apply to classes, such as making LongBuffer implement Arrayish, and (thinking out loud) javac can support the array syntax for all arrayish things. Paul. > Best regards, > Radek > > > >> On 30 May 2016, at 22:10, Andrew Haley wrote: >> >> On 30/05/16 19:48, Radek wrote: >> >>> I?m able to map part of files as primitive java arrays. First >>> performance results are very optimistic (40% boost in searching >>> min-max long in 1GB file) with mapped long[] as a replacement to >>> LongBuffer. >> >> It shouldn't be significantly different in performance. Perhaps you >> can share your benchmark with us. Maybe we're missing vectorization >> opportunities. >> >> Andrew. > From david.simms at oracle.com Tue May 31 08:49:53 2016 From: david.simms at oracle.com (david.simms at oracle.com) Date: Tue, 31 May 2016 08:49:53 +0000 Subject: hg: valhalla/valhalla/hotspot: Array defining class Object for invokevirtual Message-ID: <201605310849.u4V8nrDY015742@aojmv0008.oracle.com> Changeset: a093d07cc82a Author: dsimms Date: 2016-05-31 10:48 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/hotspot/rev/a093d07cc82a Array defining class Object for invokevirtual ! src/share/vm/prims/methodHandles.cpp From rsmogura at gmail.com Tue May 31 10:44:16 2016 From: rsmogura at gmail.com (Radek) Date: Tue, 31 May 2016 12:44:16 +0200 Subject: Java shared memory In-Reply-To: References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <574C9E41.30302@redhat.com> Message-ID: <808B4C05-4F65-4DD4-9E12-49423A84CC0A@gmail.com> Dear Paul and Adrew, Indeed I?ve messed up with a coping long. I don?t know why I have done it, as my first approach actually involved buff.getLong(). I?ve took, as well, closer look at my tests and right now I use JDK9 build 120, as a reference points. There is a performance gain (build 120 vs custom slow debug) but right now it?s 12%, not so huge. Maybe after code polishing and additional optimisation I could get 15-20%. I don?t know if in such a case my work could be interesting. Best regards, Radek PS Attached please find latest results, the MappedByteBuffer case uses for (int i=0; i < mbuff.limit(); i+=8) loop. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: JSM_Mapping_results.txt URL: -------------- next part -------------- > On 31 May 2016, at 10:27, Paul Sandoz wrote: > > Hi Radek, > >> On 31 May 2016, at 00:57, Radek wrote: >> >> Dear Andrew, >> >> I?ve just modified test case to convert bytes to long in code. The results are comparable. >> >> Attached please find out files. Those are named as follow: >> * memorymap - from MemoryMapMinMax, file map from FileMapMinMax >> * memorymap2, filemap2 form *Bytes version respectively >> >> I?m attaching text files with results, and test classes (those are my test classes, so not nicely formatted). >> >> I could do something wrong with C2, but I think results are expected as NIO in back copies mapped array to Java array, does it? >> > > Like Andrew my expectation was that a long view over a mapped byte buffer should not be significantly different. > > (Note: it?s hard to evaluate these kind of things without using JMH and also looking at the generated code). > > Here is your main loop for LongBuffer: > > while (lbuff.hasRemaining()) { > lbuff.get(ll); > for (long l : ll) { > if (l < min) min = l; > if (l > max) max = l; > lastNumber = l; > } > } > > You are bulk copying into an array, that alone could explain an ~2x performance difference. > > Instead, use the indexed accessors LongBuffer.get(int ) e.g.something like: > > for (int i = 0; i < lbuffer.limit(); i++) { > l = lbuff.get(i); > if (l < min) min = l; > if (l > max) max = l; > lastNumber = l; ... > } > > If you take a closer look at the buffer source you will notice long views over byte buffers are optimized in certain cases to direct calls to Unsafe.get/setLong. > > While lbuff.get(i) is not ll[i], it?s not far off :-) > > > Relatedly, see the recent work by Mr. (David) Simms on this list, so that int[] implements Arrayish. I dunno if this can apply to classes, such as making LongBuffer implement Arrayish, and (thinking out loud) javac can support the array syntax for all arrayish things. > > Paul. > > >> Best regards, >> Radek >> >> >> >>> On 30 May 2016, at 22:10, Andrew Haley wrote: >>> >>> On 30/05/16 19:48, Radek wrote: >>> >>>> I?m able to map part of files as primitive java arrays. First >>>> performance results are very optimistic (40% boost in searching >>>> min-max long in 1GB file) with mapped long[] as a replacement to >>>> LongBuffer. >>> >>> It shouldn't be significantly different in performance. Perhaps you >>> can share your benchmark with us. Maybe we're missing vectorization >>> opportunities. >>> >>> Andrew. >> > From vladimir.x.ivanov at oracle.com Tue May 31 11:31:27 2016 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Tue, 31 May 2016 14:31:27 +0300 Subject: Java shared memory In-Reply-To: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> Message-ID: <6cdfd335-8165-e6f8-3f58-26a43395c7d2@oracle.com> Radek, Unfortunately, I don't see any source file changes in the project you linked. It seems your project can be split into 2 independent parts: (1) accessing off-heap memory; (2) representing off-heap region as primitive array. As I understand, it should cover all use cases you mentioned. Off-heap memory access is already supported by NIO (MappedByteBuffer & DirectByteBuffer). In Java 9 VarHandles will provide int[]/long[]/... views over ByteBuffers (see MethodHandles.byteBufferViewVarHandle [1]) with impressive performance characteristics. Forward compatibility aspect (represent off-heap region as byte[]/int[]/... instance) fits perfectly into Arrays 2.0 project (see John's presentation at JVMLS 2012 for more details [2]). As Paul already mentioned, David Simms has been working on j.l.Arrayish [2] support and made impressive progress (the idea is the JVM can link *aload/*astore bytecode instructions to methods on Arrayish interface). Have you had a chance to look into ByteBuffers/Varhandles & j.l.Arrayish? It would be very interesting to hear about your experience. Best regards, Vladimir Ivanov [1] http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/classes/java/lang/invoke/MethodHandles.java#l2405 [2] http://www.oracle.com/technetwork/java/javase/community/jvmls2012-1840099.html http://cr.openjdk.java.net/~jrose/pres/201207-Arrays-2.pdf [3] http://mail.openjdk.java.net/pipermail/valhalla-dev/2016-April/001892.html http://hg.openjdk.java.net/valhalla/valhalla/jdk/file/tip/src/java.base/share/classes/java/lang/Arrayish.java On 5/30/16 9:48 PM, Radek wrote: > Dear all, > > Please don?t think I?m so crazy. Recently I was working on JVM changes to add shared memory / memory mapping support. > > I?m able to map part of files as primitive java arrays. First performance results are very optimistic (40% boost in searching min-max long in 1GB file) with mapped long[] as a replacement to LongBuffer. The work is not finished. I have adopted G1, c1 and c2, but I have got mainly stable solution (or I think so). > > I see three possible benefits of shared memory: > 1. Shared memory and interprocess communication. > 2. Mapping large files. > 3. Aggressive inter-JVM string de-duplication. > > I?m looking for sponsorship for this change, with some guidelines how I can move forward with possible adoption into JDK. If solution would be viable, I would like to open JSR for JDK API modification. > > I use https://bitbucket.org/radoslaw_smogura/java-shared-memory/wiki/Home as umbrella repo, but I don?t update it quite often. > > Thanks in advance, > Radek > > From aph at redhat.com Tue May 31 13:13:25 2016 From: aph at redhat.com (Andrew Haley) Date: Tue, 31 May 2016 14:13:25 +0100 Subject: Java shared memory In-Reply-To: <808B4C05-4F65-4DD4-9E12-49423A84CC0A@gmail.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <574C9E41.30302@redhat.com> <808B4C05-4F65-4DD4-9E12-49423A84CC0A@gmail.com> Message-ID: <574D8DF5.8010301@redhat.com> On 05/31/2016 11:44 AM, Radek wrote: > There is a performance gain (build 120 vs custom slow debug) but right now it?s 12%, not so huge. Maybe after code polishing and additional optimisation I could get 15-20%. I would not expect to see any difference at all with mapped long buffers. I don't think this is worth doing, and I could not find the source for your changes to check for myself. Andrew. From rsmogura at gmail.com Tue May 31 13:30:49 2016 From: rsmogura at gmail.com (Radek) Date: Tue, 31 May 2016 15:30:49 +0200 Subject: Java shared memory In-Reply-To: <6cdfd335-8165-e6f8-3f58-26a43395c7d2@oracle.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <6cdfd335-8165-e6f8-3f58-26a43395c7d2@oracle.com> Message-ID: <9D0BEFF0-B512-4A6C-BAB6-554931732B0F@gmail.com> Dear all, Thank you for your effort, I understand that the proposal was rejected. I maintain code here https://bitbucket.org/radoslaw_smogura/jshm-jdk9-hotspot/ (please don?t laugh I started with this more as prototype, but remarks are always welcome). The approach I used is (or actually was) to store the pointer to actuall elements just after a length field (for primitive arrays). This pointer for ordinal arrays would point to heap space just after header, for attached arrays it can point anywhere. I exposed just attachMemory0 native call (which could be hidden in other ways from unrestricted access) to native library which did all the mappings. Someone could use this to create shared POSIX or other region to store char / bytes of String and then reduplicate Strings among JVMs. As pointer can be arbitrary I?m not limited to page boundaries for C mapping. With this approach for small cost of loading pointers, and slightly size increase I could maintain arrays as normal and keep assembly code still effective. The thing except changes to compiler were to properly handle SIGBUS. And yes it?s faster... Thanks for your time and have a nice day Radek > On 31 May 2016, at 13:31, Vladimir Ivanov wrote: > > Radek, > > Unfortunately, I don't see any source file changes in the project you linked. > > It seems your project can be split into 2 independent parts: > (1) accessing off-heap memory; > (2) representing off-heap region as primitive array. > > As I understand, it should cover all use cases you mentioned. > > Off-heap memory access is already supported by NIO (MappedByteBuffer & DirectByteBuffer). In Java 9 VarHandles will provide int[]/long[]/... views over ByteBuffers (see MethodHandles.byteBufferViewVarHandle [1]) with impressive performance characteristics. > > Forward compatibility aspect (represent off-heap region as byte[]/int[]/... instance) fits perfectly into Arrays 2.0 project (see John's presentation at JVMLS 2012 for more details [2]). As Paul already mentioned, David Simms has been working on j.l.Arrayish [2] support and made impressive progress (the idea is the JVM can link *aload/*astore bytecode instructions to methods on Arrayish interface). > > Have you had a chance to look into ByteBuffers/Varhandles & j.l.Arrayish? It would be very interesting to hear about your experience. > > Best regards, > Vladimir Ivanov > > [1] http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/classes/java/lang/invoke/MethodHandles.java#l2405 > > [2] http://www.oracle.com/technetwork/java/javase/community/jvmls2012-1840099.html > http://cr.openjdk.java.net/~jrose/pres/201207-Arrays-2.pdf > > [3] http://mail.openjdk.java.net/pipermail/valhalla-dev/2016-April/001892.html > http://hg.openjdk.java.net/valhalla/valhalla/jdk/file/tip/src/java.base/share/classes/java/lang/Arrayish.java > > On 5/30/16 9:48 PM, Radek wrote: >> Dear all, >> >> Please don?t think I?m so crazy. Recently I was working on JVM changes to add shared memory / memory mapping support. >> >> I?m able to map part of files as primitive java arrays. First performance results are very optimistic (40% boost in searching min-max long in 1GB file) with mapped long[] as a replacement to LongBuffer. The work is not finished. I have adopted G1, c1 and c2, but I have got mainly stable solution (or I think so). >> >> I see three possible benefits of shared memory: >> 1. Shared memory and interprocess communication. >> 2. Mapping large files. >> 3. Aggressive inter-JVM string de-duplication. >> >> I?m looking for sponsorship for this change, with some guidelines how I can move forward with possible adoption into JDK. If solution would be viable, I would like to open JSR for JDK API modification. >> >> I use https://bitbucket.org/radoslaw_smogura/java-shared-memory/wiki/Home as umbrella repo, but I don?t update it quite often. >> >> Thanks in advance, >> Radek >> >> From vladimir.x.ivanov at oracle.com Tue May 31 13:48:55 2016 From: vladimir.x.ivanov at oracle.com (Vladimir Ivanov) Date: Tue, 31 May 2016 16:48:55 +0300 Subject: Java shared memory In-Reply-To: <9D0BEFF0-B512-4A6C-BAB6-554931732B0F@gmail.com> References: <6B8F88FE-24AF-40E6-96DE-D7443667DA47@gmail.com> <6cdfd335-8165-e6f8-3f58-26a43395c7d2@oracle.com> <9D0BEFF0-B512-4A6C-BAB6-554931732B0F@gmail.com> Message-ID: <5a82c73b-ac09-8015-e8e7-e21bd3a2d3c3@oracle.com> Radek, > Thank you for your effort, I understand that the proposal was rejected. > > I maintain code here https://bitbucket.org/radoslaw_smogura/jshm-jdk9-hotspot/ (please don?t laugh I started with this more as prototype, but remarks are always welcome). Thanks for sharing it! > The approach I used is (or actually was) to store the pointer to actuall elements just after a length field (for primitive arrays). This pointer for ordinal arrays would point to heap space just after header, for attached arrays it can point anywhere. It sounds very similar to Packed Objects by IBM [1], which generalize the idea from primitive arrays to arbitrary Java objects ( having (target,offset) pair as part of Java object header). Best regards, Vladimir Ivanov [1] http://www.oracle.com/technetwork/java/jvmls2013sciam-2013525.pdf > > I exposed just attachMemory0 native call (which could be hidden in other ways from unrestricted access) to native library which did all the mappings. Someone could use this to create shared POSIX or other region to store char / bytes of String and then reduplicate Strings among JVMs. As pointer can be arbitrary I?m not limited to page boundaries for C mapping. > > With this approach for small cost of loading pointers, and slightly size increase I could maintain arrays as normal and keep assembly code still effective. > > The thing except changes to compiler were to properly handle SIGBUS. > > And yes it?s faster... > > Thanks for your time and have a nice day > Radek > >> On 31 May 2016, at 13:31, Vladimir Ivanov wrote: >> >> Radek, >> >> Unfortunately, I don't see any source file changes in the project you linked. >> >> It seems your project can be split into 2 independent parts: >> (1) accessing off-heap memory; >> (2) representing off-heap region as primitive array. >> >> As I understand, it should cover all use cases you mentioned. >> >> Off-heap memory access is already supported by NIO (MappedByteBuffer & DirectByteBuffer). In Java 9 VarHandles will provide int[]/long[]/... views over ByteBuffers (see MethodHandles.byteBufferViewVarHandle [1]) with impressive performance characteristics. >> >> Forward compatibility aspect (represent off-heap region as byte[]/int[]/... instance) fits perfectly into Arrays 2.0 project (see John's presentation at JVMLS 2012 for more details [2]). As Paul already mentioned, David Simms has been working on j.l.Arrayish [2] support and made impressive progress (the idea is the JVM can link *aload/*astore bytecode instructions to methods on Arrayish interface). >> >> Have you had a chance to look into ByteBuffers/Varhandles & j.l.Arrayish? It would be very interesting to hear about your experience. >> >> Best regards, >> Vladimir Ivanov >> >> [1] http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/classes/java/lang/invoke/MethodHandles.java#l2405 >> >> [2] http://www.oracle.com/technetwork/java/javase/community/jvmls2012-1840099.html >> http://cr.openjdk.java.net/~jrose/pres/201207-Arrays-2.pdf >> >> [3] http://mail.openjdk.java.net/pipermail/valhalla-dev/2016-April/001892.html >> http://hg.openjdk.java.net/valhalla/valhalla/jdk/file/tip/src/java.base/share/classes/java/lang/Arrayish.java >> >> On 5/30/16 9:48 PM, Radek wrote: >>> Dear all, >>> >>> Please don?t think I?m so crazy. Recently I was working on JVM changes to add shared memory / memory mapping support. >>> >>> I?m able to map part of files as primitive java arrays. First performance results are very optimistic (40% boost in searching min-max long in 1GB file) with mapped long[] as a replacement to LongBuffer. The work is not finished. I have adopted G1, c1 and c2, but I have got mainly stable solution (or I think so). >>> >>> I see three possible benefits of shared memory: >>> 1. Shared memory and interprocess communication. >>> 2. Mapping large files. >>> 3. Aggressive inter-JVM string de-duplication. >>> >>> I?m looking for sponsorship for this change, with some guidelines how I can move forward with possible adoption into JDK. If solution would be viable, I would like to open JSR for JDK API modification. >>> >>> I use https://bitbucket.org/radoslaw_smogura/java-shared-memory/wiki/Home as umbrella repo, but I don?t update it quite often. >>> >>> Thanks in advance, >>> Radek >>> >>> >