From andreas.andreakis at googlemail.com Wed Dec 12 09:16:45 2007 From: andreas.andreakis at googlemail.com (Andreas Andreakis) Date: Wed, 12 Dec 2007 10:16:45 +0100 Subject: Java should provide exact real arithmetics Message-ID: <85e6ec170712120116k3793fc3dy54b52f536d39340c@mail.gmail.com> Hi, my default platform of choice usually is the jvm. But right now, I?m working for a project, where I can?t use it. The project requires high precision, error free and fast arithmetics with real numbers. And it turns out that java does not fit this requirements. BigDecimal of course is a major improvement against primitive low precision types such as float or double, but BigDecimal does not scale and does not implement mathematical functions such as exp, sqrt, sin, ...... Especially if you need to multiply numbers from the open interval (0,1) such as 0.000234 * 0.4432001234 , you will reach BigDecimals performance limits very soon. for a demonstration execute the following loop: int loops = 100; BigDecimal number = new BigDecimal("0.123"); for(int i = 0; i From roman at kennke.org Wed Dec 12 10:01:13 2007 From: roman at kennke.org (Roman Kennke) Date: Wed, 12 Dec 2007 11:01:13 +0100 Subject: Java should provide exact real arithmetics In-Reply-To: <85e6ec170712120116k3793fc3dy54b52f536d39340c@mail.gmail.com> References: <85e6ec170712120116k3793fc3dy54b52f536d39340c@mail.gmail.com> Message-ID: <1197453673.6123.20.camel@mercury> Hi there, > Currently, there exist much better solutions than BigDecimal, > unfortunately not from the Java space. For instance GMP > ( http://gmplib.org/) is considered as one of the state of the art > libraries in the area of precision arithmetics. And several libraries > extent GMP such as the popular iRRAM project > (http://www.informatik.uni-trier.de/iRRAM/). > > Check out the following page in order to get an idea how performable > GMP is: http://gmplib.org/pi-with-gmp.html > > it demonstrates the calculation of pi with many digits. > > > I?m asking myself why there is nothing comparable for the java > platfom. Have you tried: http://www.apfloat.org/ Not everything has to be in the core platform IMO (I think it's already much too bloated). Or maybe it could be considered to write a JNI-Layer on top of GMP? Cheers, Roman -- http://kennke.org/blog/ From andreas.andreakis at googlemail.com Wed Dec 12 11:00:21 2007 From: andreas.andreakis at googlemail.com (Andreas Andreakis) Date: Wed, 12 Dec 2007 12:00:21 +0100 Subject: Java should provide exact real arithmetics In-Reply-To: <1197453673.6123.20.camel@mercury> References: <85e6ec170712120116k3793fc3dy54b52f536d39340c@mail.gmail.com> <1197453673.6123.20.camel@mercury> Message-ID: <85e6ec170712120300k5ecb3549iab6e397c09a18fda@mail.gmail.com> Hi, sorry everybody, but I copy-pasted a wrong code snipped in my original post. Because most of the time is spend in string concatination and printing the output of each calculation loop. this is a correct version: int loops = 100; for(int i = 0; i wrote: > Hi there, > > > > > Currently, there exist much better solutions than BigDecimal, > > unfortunately not from the Java space. For instance GMP > > ( http://gmplib.org/) is considered as one of the state of the art > > libraries in the area of precision arithmetics. And several libraries > > extent GMP such as the popular iRRAM project > > (http://www.informatik.uni-trier.de/iRRAM/). > > > > Check out the following page in order to get an idea how performable > > GMP is: http://gmplib.org/pi-with-gmp.html > > > > it demonstrates the calculation of pi with many digits. > > > > > > I?m asking myself why there is nothing comparable for the java > > platfom. > > Have you tried: > http://www.apfloat.org/ > > Not everything has to be in the core platform IMO (I think it's already > much too bloated). Or maybe it could be considered to write a JNI-Layer > on top of GMP? > > Cheers, Roman > -- > http://kennke.org/blog/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roman.kennke at aicas.com Tue Dec 18 14:32:06 2007 From: roman.kennke at aicas.com (Roman Kennke) Date: Tue, 18 Dec 2007 15:32:06 +0100 Subject: Fix compiler problem Message-ID: <1197988326.7440.7.camel@mercury> When trying to compile OpenJDK with the Eclipse compiler, I noticed two compiler errors related to generics. It turned out that the code there is invalid and only javac (incorrectly) accepts it. See the following bug reports for details: https://bugs.eclipse.org/bugs/show_bug.cgi?id=212147 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6400189 The attached changeset fixes the problem. Could this be included? /Roman -- Dipl.-Inform. (FH) Roman Kennke, Software Engineer, http://kennke.org aicas Allerton Interworks Computer Automated Systems GmbH Haid-und-Neu-Stra?e 18 * D-76131 Karlsruhe * Germany http://www.aicas.com * Tel: +49-721-663 968-0 USt-Id: DE216375633, Handelsregister HRB 109481, AG Karlsruhe Gesch?ftsf?hrer: Dr. James J. Hunt -------------- next part -------------- A non-text attachment was scrubbed... Name: openjdk-compile.patch Type: text/x-patch Size: 1613 bytes Desc: not available URL: From Alan.Bateman at Sun.COM Tue Dec 18 14:37:01 2007 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 18 Dec 2007 14:37:01 +0000 Subject: [Fwd: Fix compiler problem] Message-ID: <4767DB0D.7010309@sun.com> I don't know if the JMX team or the JavaBeans maintainers are on the core-libs-dev mailing list. -------------- next part -------------- An embedded message was scrubbed... From: Roman Kennke Subject: Fix compiler problem Date: Tue, 18 Dec 2007 15:32:06 +0100 Size: 8249 URL: From Eamonn.McManus at Sun.COM Tue Dec 18 18:09:16 2007 From: Eamonn.McManus at Sun.COM (Eamonn McManus) Date: Tue, 18 Dec 2007 19:09:16 +0100 Subject: jmx-dev [Fwd: Fix compiler problem] In-Reply-To: <4767DB0D.7010309@sun.com> References: <4767DB0D.7010309@sun.com> Message-ID: <47680CCC.2030206@sun.com> An HTML attachment was scrubbed... URL: From Sergey.Malenkov at Sun.COM Wed Dec 19 17:34:16 2007 From: Sergey.Malenkov at Sun.COM (Sergey Malenkov) Date: Wed, 19 Dec 2007 20:34:16 +0300 Subject: jmx-dev [Fwd: Fix compiler problem] In-Reply-To: <47680CCC.2030206@sun.com> References: <4767DB0D.7010309@sun.com> <47680CCC.2030206@sun.com> Message-ID: <47695618.9090300@sun.com> Hi, Thank you for explanation. I'll fix it soon. Thanks, SAM Eamonn McManus wrote: > Thanks for bringing this to our attention, Alan. Roman is right - the > code in question should not compile and we should change it as he > suggests so that it is correct. > > (For people interested in the gory details, the method > T getAnnotation > (Class > annotationClass) > returns T only if it is called on a properly generic variable, such as a > Constructor or Constructor or Constructor. If it is > called on a plain Constructor with no type parameter, then that is a > "raw type", and the return type is "erased" to Annotation. So something like > ConstructorProperties annotation = > constructor.getAnnotation(ConstructorProperties.class) > should not compile if constructor is declared as Constructor rather than > Constructor or whatever.) > > ?amonn McManus JMX Spec Lead http://weblogs.java.net/blog/emcmanus/ > > > > Alan Bateman wrote: >> >> I don't know if the JMX team or the JavaBeans maintainers are on the >> core-libs-dev mailing list. >> >> >> ------------------------------------------------------------------------ >> >> Subject: >> Fix compiler problem >> From: >> Roman Kennke >> Date: >> Tue, 18 Dec 2007 15:32:06 +0100 >> To: >> Core-Libs-Dev >> >> To: >> Core-Libs-Dev >> >> >> When trying to compile OpenJDK with the Eclipse compiler, I noticed two >> compiler errors related to generics. It turned out that the code there >> is invalid and only javac (incorrectly) accepts it. See the following >> bug reports for details: >> >> https://bugs.eclipse.org/bugs/show_bug.cgi?id=212147 >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6400189 >> >> The attached changeset fixes the problem. Could this be included? >> >> /Roman >> >> From linuxhippy at gmail.com Thu Dec 20 19:20:30 2007 From: linuxhippy at gmail.com (Clemens Eisserer) Date: Thu, 20 Dec 2007 20:20:30 +0100 Subject: Performance regression in java.util.zip.Deflater Message-ID: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> Hello, Sombody posted at http://forums.java.net/jive/thread.jspa?messageID=251006 that he has problems with the performance of java.util.zip.Deflater starting with version 1.5.0_07. I did a very dumb micro-benchmark and it seems to confirm it, with small buffers (the original author used a 1000 byte buffer), 1.4.2 took ~1000ms whereas 6.0/7.0b23 take 11000ms. Even when using a 32kb buffer 1.4.2 is still twice as fast. I played a bit with oprofile and it clearly shows up that memcopy eats all the memory. The problem is that every time the whole input-buffer is copied to the native side, assuming that every call 2000bytes (ratio 50%) of input data are compressed "away" from the input, the method copies every call to deflateBytes 5000k, 4998k, 4996k , .... This can't be solved easily because we don't know how many bytes zlib may consume from the input-data. I would have a few ideas how this issue could be solved: 1.) Using DirectByteBuffers for data-transfer. pros: Array-Like access from the native side, no negative inpact on GC. cons: Data has to be copied, wasted RAM (because we have two copies, one in the byte[] supplied by the user, and one outside the heap in the DirectByteBuffer, possible OOMs because out-of-native memory. 2.) Use GetPrimitiveArrayCritical: pros: no copying involved at all, no redundant copies of data arround. cons: quite harsh to the GC (blocked until compression is finished) - maybe even scalability limiter. I've modified Deflate.c to use GetPrimitiveArrayCritical, and it now compresses in 100ms instead of 11000, even twice as fast as 1.4.2. Although this solution looks quite cool, I doubt its behaviour does comply with Sun's quality expectations. 3.) Limit the amount of byted trasfereed to the native side: pros: no redundant copies of input-data cons: still a lot of copying (however not n^2), maybe more JNI calls to get same work done. I would be happy about suggestions and thoughts in general. Maybe somebody knows why the old JVMs performed so much better here? Thank you in advance, lg Clemens Test-Case: public class DeflaterTest { public static byte[] compresserZlib(byte[] donnees) { ByteArrayOutputStream resultat = new ByteArrayOutputStream(); byte[] buffer = new byte[1000]; int nbEcrits; Deflater deflater = new Deflater(); deflater.setInput(donnees); deflater.setLevel(0); deflater.finish(); while (!deflater.finished()) { nbEcrits = deflater.deflate(buffer); resultat.write(buffer, 0, nbEcrits); } return resultat.toByteArray(); } public static void main(String[] args) { Random r = new Random(); byte[] buffer = new byte[5000000]; for(int i=0; i < buffer.length; i++) { buffer[i] = (byte) (r.nextInt()%127); } for(int i=0; i < 100; i++) { long start = System.currentTimeMillis(); byte[] result = compresserZlib(buffer); long end = System.currentTimeMillis(); System.out.println("Run took: "+(end-start)+" "+result[Math.abs(buffer[0])]); } } } From forax at univ-mlv.fr Thu Dec 20 20:00:11 2007 From: forax at univ-mlv.fr (=?UTF-8?B?UsOpbWkgRm9yYXg=?=) Date: Thu, 20 Dec 2007 21:00:11 +0100 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> Message-ID: <476AC9CB.8040600@univ-mlv.fr> Clemens Eisserer a ?crit : > Hello, > > Sombody posted at > http://forums.java.net/jive/thread.jspa?messageID=251006 that he has > problems with the performance of java.util.zip.Deflater starting with > version 1.5.0_07. > Hi Clements, This bug has a long history, http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6399199 but i don't know what is the current state of this bug. > I did a very dumb micro-benchmark and it seems to confirm it, with > small buffers (the original author used a 1000 byte buffer), 1.4.2 > took ~1000ms whereas 6.0/7.0b23 take 11000ms. Even when using a 32kb > buffer 1.4.2 is still twice as fast. > I played a bit with oprofile and it clearly shows up that memcopy eats > all the memory. > > The problem is that every time the whole input-buffer is copied to the > native side, assuming that every call 2000bytes (ratio 50%) of input > data are compressed "away" from the input, the method copies every > call to deflateBytes 5000k, 4998k, 4996k , .... > This can't be solved easily because we don't know how many bytes zlib > may consume from the input-data. > > I would have a few ideas how this issue could be solved: > > 1.) Using DirectByteBuffers for data-transfer. > pros: Array-Like access from the native side, no negative inpact on GC. > cons: Data has to be copied, wasted RAM (because we have two copies, > one in the byte[] supplied by the user, and one outside the heap in > the DirectByteBuffer, possible OOMs because out-of-native memory. > > 2.) Use GetPrimitiveArrayCritical: > pros: no copying involved at all, no redundant copies of data arround. > cons: quite harsh to the GC (blocked until compression is finished) - > maybe even scalability limiter. > I've modified Deflate.c to use GetPrimitiveArrayCritical, and it now > compresses in 100ms instead of 11000, even twice as fast as 1.4.2. > Although this solution looks quite cool, I doubt its behaviour does > comply with Sun's quality expectations. > > 3.) Limit the amount of byted trasfereed to the native side: > pros: no redundant copies of input-data > cons: still a lot of copying (however not n^2), maybe more JNI calls > to get same work done. > > I would be happy about suggestions and thoughts in general. Maybe > somebody knows why the old JVMs performed so much better here? > > Thank you in advance, lg Clemens > > > > Test-Case: > public class DeflaterTest > { > > public static byte[] compresserZlib(byte[] donnees) > { > ByteArrayOutputStream resultat = new ByteArrayOutputStream(); > byte[] buffer = new byte[1000]; > int nbEcrits; > > Deflater deflater = new Deflater(); > deflater.setInput(donnees); > deflater.setLevel(0); > deflater.finish(); > > while (!deflater.finished()) > { > nbEcrits = deflater.deflate(buffer); > resultat.write(buffer, 0, nbEcrits); > } > > return resultat.toByteArray(); > } > > public static void main(String[] args) > { > Random r = new Random(); > byte[] buffer = new byte[5000000]; > for(int i=0; i < buffer.length; i++) > { > buffer[i] = (byte) (r.nextInt()%127); > } > > for(int i=0; i < 100; i++) > { > long start = System.currentTimeMillis(); > byte[] result = compresserZlib(buffer); > long end = System.currentTimeMillis(); > > System.out.println("Run took: "+(end-start)+" "+result[Math.abs(buffer[0])]); > } > > } > } > R?mi From Martin.Buchholz at Sun.COM Thu Dec 20 21:25:03 2007 From: Martin.Buchholz at Sun.COM (Martin Buchholz) Date: Thu, 20 Dec 2007 13:25:03 -0800 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> Message-ID: <476ADDAF.2070409@sun.com> A thorny problem. Here is a list of related bugs "fixed" in 5.0 updates: 6348045: REGRESSION: serious performance degradation as GZIPInputStream is slower http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6348045 6206933: GZipOutputStream/InputStream goes critical(calls JNI_Get*Critical) and causes slowness http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6206933 6364346: GZIPOutputStream is slower on 1.4.2_11-b02 than on 1.4.2_09 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6364346 Sun engineers have tried to get reasonable performance without using JNI_Get*Critical, since that introduces other serious performance problems. It was my belief that any pathological n^2 performance problems had been truly fixed. Make sure you are running at least 5.0u8 to get all of the above. Martin Clemens Eisserer wrote: > Hello, > > Sombody posted at > http://forums.java.net/jive/thread.jspa?messageID=251006 that he has > problems with the performance of java.util.zip.Deflater starting with > version 1.5.0_07. > I did a very dumb micro-benchmark and it seems to confirm it, with > small buffers (the original author used a 1000 byte buffer), 1.4.2 > took ~1000ms whereas 6.0/7.0b23 take 11000ms. Even when using a 32kb > buffer 1.4.2 is still twice as fast. > I played a bit with oprofile and it clearly shows up that memcopy eats > all the memory. > > The problem is that every time the whole input-buffer is copied to the > native side, assuming that every call 2000bytes (ratio 50%) of input > data are compressed "away" from the input, the method copies every > call to deflateBytes 5000k, 4998k, 4996k , .... > This can't be solved easily because we don't know how many bytes zlib > may consume from the input-data. > > I would have a few ideas how this issue could be solved: > > 1.) Using DirectByteBuffers for data-transfer. > pros: Array-Like access from the native side, no negative inpact on GC. > cons: Data has to be copied, wasted RAM (because we have two copies, > one in the byte[] supplied by the user, and one outside the heap in > the DirectByteBuffer, possible OOMs because out-of-native memory. > > 2.) Use GetPrimitiveArrayCritical: > pros: no copying involved at all, no redundant copies of data arround. > cons: quite harsh to the GC (blocked until compression is finished) - > maybe even scalability limiter. > I've modified Deflate.c to use GetPrimitiveArrayCritical, and it now > compresses in 100ms instead of 11000, even twice as fast as 1.4.2. > Although this solution looks quite cool, I doubt its behaviour does > comply with Sun's quality expectations. > > 3.) Limit the amount of byted trasfereed to the native side: > pros: no redundant copies of input-data > cons: still a lot of copying (however not n^2), maybe more JNI calls > to get same work done. > > I would be happy about suggestions and thoughts in general. Maybe > somebody knows why the old JVMs performed so much better here? > > Thank you in advance, lg Clemens > > > > Test-Case: > public class DeflaterTest > { > > public static byte[] compresserZlib(byte[] donnees) > { > ByteArrayOutputStream resultat = new ByteArrayOutputStream(); > byte[] buffer = new byte[1000]; > int nbEcrits; > > Deflater deflater = new Deflater(); > deflater.setInput(donnees); > deflater.setLevel(0); > deflater.finish(); > > while (!deflater.finished()) > { > nbEcrits = deflater.deflate(buffer); > resultat.write(buffer, 0, nbEcrits); > } > > return resultat.toByteArray(); > } > > public static void main(String[] args) > { > Random r = new Random(); > byte[] buffer = new byte[5000000]; > for(int i=0; i < buffer.length; i++) > { > buffer[i] = (byte) (r.nextInt()%127); > } > > for(int i=0; i < 100; i++) > { > long start = System.currentTimeMillis(); > byte[] result = compresserZlib(buffer); > long end = System.currentTimeMillis(); > > System.out.println("Run took: "+(end-start)+" "+result[Math.abs(buffer[0])]); > } > > } > } From linuxhippy at gmail.com Thu Dec 20 21:36:39 2007 From: linuxhippy at gmail.com (Clemens Eisserer) Date: Thu, 20 Dec 2007 22:36:39 +0100 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <476ADDAF.2070409@sun.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> Message-ID: <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> Hi again, > Sun engineers have tried to get reasonable performance > without using JNI_Get*Critical, since that introduces other > serious performance problems. It was my belief that any > pathological n^2 performance problems had been truly fixed. At least the code in JDK7u23 looks like (n^2)/2 or something like that, it copies every time the whole bytes which are left, including malloc/free. > Sun engineers have tried to get reasonable performance > without using JNI_Get*Critical, since that introduces other > serious performance problems. Could please tell me when and why. As far as I understood the problem with the *Critical*-Functions is that they hinder the JVM in doing some operations (GC, ...) which limits scalability. If this is the only reson, using them may not be that bad if the Get*ArrayRegion also has some GC-atomic behaviour. Copying 50mb data atomically also blocks the GC, doesn't it? I am working on a fix which processes the data in "strides", therefor the lock is only held a short time. Is this really a bad idea, except for the additional JNI overhead? Thanks, lg Clemens From Martin.Buchholz at Sun.COM Thu Dec 20 22:13:22 2007 From: Martin.Buchholz at Sun.COM (Martin Buchholz) Date: Thu, 20 Dec 2007 14:13:22 -0800 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> Message-ID: <476AE902.2090909@sun.com> Hi Clemens, (Thank you very much for working on this problem) Clemens Eisserer wrote: > Hi again, > >>Sun engineers have tried to get reasonable performance >>without using JNI_Get*Critical, since that introduces other >>serious performance problems. It was my belief that any >>pathological n^2 performance problems had been truly fixed. > At least the code in JDK7u23 looks like (n^2)/2 or something like > that, it copies every time the whole bytes which are left, including > malloc/free. > >>Sun engineers have tried to get reasonable performance >>without using JNI_Get*Critical, since that introduces other >>serious performance problems. > Could please tell me when and why. As far as I understood the problem > with the *Critical*-Functions is that they hinder the JVM in doing > some operations (GC, ...) which limits scalability. > > If this is the only reson, using them may not be that bad if the > Get*ArrayRegion also has some GC-atomic behaviour. Copying 50mb data > atomically also blocks the GC, doesn't it? I'm not the expert on this, but I believe the GC may be completely blocked from making progress while data is held "Critical"-ly, while this is not true of the regular JNI functions. If multiple threads are busy deflating furiously, the entire JVM may be blocked from making progress. > I am working on a fix which processes the data in "strides", therefor > the lock is only held a short time. Is this really a bad idea, except > for the additional JNI overhead? Offhand, dividing the data into chunks sounds like a good idea. But you're not likely to be able to persuade Sun to reintroduce the use of JNI_Get*Critical, which has already been removed at great cost. Martin > Thanks, lg Clemens From David.Bristor at Sun.COM Thu Dec 20 22:33:49 2007 From: David.Bristor at Sun.COM (Dave Bristor) Date: Thu, 20 Dec 2007 14:33:49 -0800 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> Message-ID: <476AEDCD.6080504@sun.com> Hi Ig, Thanks for the suggestions. Here's some more history. Clemens Eisserer wrote: > Hi again, > >> Sun engineers have tried to get reasonable performance >> without using JNI_Get*Critical, since that introduces other >> serious performance problems. It was my belief that any >> pathological n^2 performance problems had been truly fixed. > At least the code in JDK7u23 looks like (n^2)/2 or something like > that, it copies every time the whole bytes which are left, including > malloc/free. Successive attempts to address this performance / scalability problem have focused on minimizing the amount of data copied. As noted, the fix is in DeflaterOutputStream.write, where stride bytes at a time are deflated, *not* the entire user-provided data buffer. This results in more JNI calls (and consequently more malloc-copy-deflate-free) but does not stall GC. >> Sun engineers have tried to get reasonable performance >> without using JNI_Get*Critical, since that introduces other >> serious performance problems. > Could please tell me when and why. As far as I understood the problem > with the *Critical*-Functions is that they hinder the JVM in doing > some operations (GC, ...) which limits scalability. Prior to 1.5.0_u7 the *Critical* function were used, but for the sake of 6206933, their use was replaced with data copying. > If this is the only reson, using them may not be that bad if the > Get*ArrayRegion also has some GC-atomic behaviour. Copying 50mb data > atomically also blocks the GC, doesn't it? > > I am working on a fix which processes the data in "strides", therefor > the lock is only held a short time. Is this really a bad idea, except > for the additional JNI overhead? The observations I've made show that the use of strides results in 2x slower performance as compared with the *Critical*. Certainly not ideal, but certainly much better than the ~10x worse performance than early attempts at resolving the issue. FWIW, we looked into using DirectByteBuffer but did not like the idea of keeping 2 copies of data around. Moving the striding from DeflaterOutputStream to Deflater (and possibly providing similar functionality in the Inflater side) seems like a Good Idea. IIRC, we put the striding into DeflaterOutputStream because that has the bufer whose size is known (and optionally provided by the user when the instance is created). Thanks, Dave > > Thanks, lg Clemens From linuxhippy at gmail.com Thu Dec 20 23:20:28 2007 From: linuxhippy at gmail.com (Clemens Eisserer) Date: Fri, 21 Dec 2007 00:20:28 +0100 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <476AEDCD.6080504@sun.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> <476AEDCD.6080504@sun.com> Message-ID: <194f62550712201520p30d7b15wa8f2005749a77243@mail.gmail.com> Hi Dave, > Prior to 1.5.0_u7 the *Critical* function were used, but for the sake of > 6206933, their use was replaced with data copying. Thanks a lot for all the feedback, it really helps to understand the reasons why the code is as it is. I also thought about the DirectByteBuffers and I also did not like idea of having the data twice (expecially when keeping in mind that some may compress hundreds of mbs). I still believe that a *Critical*-Solution still could be the best solution - if the only drawback is that it can block other threads and the JVM while running. Does a benchmark exist for this situation? I still have some questions: 1.) Does GetByteArrayRegion also block the JVM/GC? (I assume it has to) 2.) Do the *Critical*-functions have other negative side-impacts except blocking the GC. 3.) If many native threads try to call a *Critical*-function, although the JVM would like to call a GC - does the JVM wait until no threads hold a Critical Refrence - or will it block all threads that plan to enter a Critical-Function, do its GC, and later resume? Because if 1.) does block, 2.) no other side impacts, and 3.) is also true I will try to benchmark and simulate a bit - I can't imagine that it would block any more than GetByteArrayRegion. Thanks a lot and sorry for beeing such a bullhead, lg Clemens From David.Bristor at Sun.COM Fri Dec 21 00:37:14 2007 From: David.Bristor at Sun.COM (Dave Bristor) Date: Thu, 20 Dec 2007 16:37:14 -0800 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201520p30d7b15wa8f2005749a77243@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> <476AEDCD.6080504@sun.com> <194f62550712201520p30d7b15wa8f2005749a77243@mail.gmail.com> Message-ID: <476B0ABA.6030102@sun.com> Clemens Eisserer wrote: > Hi Dave, > >> Prior to 1.5.0_u7 the *Critical* function were used, but for the sake of >> 6206933, their use was replaced with data copying. > > Thanks a lot for all the feedback, it really helps to understand the > reasons why the code is as it is. I also thought about the > DirectByteBuffers and I also did not like idea of having the data > twice (expecially when keeping in mind that some may compress hundreds > of mbs). > > I still believe that a *Critical*-Solution still could be the best > solution - if the only drawback is that it can block other threads and > the JVM while running. Does a benchmark exist for this situation? jdk/test/java/util/zip/FlaterTest.java can be used to compare different JVMs; I'm sorry but I don't think we have a better test. See also the code in 6364346 and/or 6507183. > I still have some questions: > 1.) Does GetByteArrayRegion also block the JVM/GC? (I assume it has to) > 2.) Do the *Critical*-functions have other negative side-impacts > except blocking the GC. I don't see a block in the ArrayRegion functions (a little surprisingly): hotspot/src/share/vm/prims/jni.cpp (Search for just "arrayregion"). Compare the array region function body with the primitivearraycritical function body; the latter obviously interacts with GC. > 3.) If many native threads try to call a *Critical*-function, although > the JVM would like to call a GC - does the JVM wait until no threads > hold a Critical Refrence - or will it block all threads that plan to > enter a Critical-Function, do its GC, and later resume? I'm no expert here! See 6186200's discussion of GC & critical sections. > > Because if 1.) does block, 2.) no other side impacts, and 3.) is also > true I will try to benchmark and simulate a bit - I can't imagine that > it would block any more than GetByteArrayRegion. > > Thanks a lot and sorry for beeing such a bullhead, lg Clemens You're welcome, and thank you for your interest! Dave From linuxhippy at gmail.com Fri Dec 21 01:02:52 2007 From: linuxhippy at gmail.com (Clemens Eisserer) Date: Fri, 21 Dec 2007 02:02:52 +0100 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <476B0ABA.6030102@sun.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> <476AEDCD.6080504@sun.com> <194f62550712201520p30d7b15wa8f2005749a77243@mail.gmail.com> <476B0ABA.6030102@sun.com> Message-ID: <194f62550712201702n6f44efd5hda27c397e8d1ce96@mail.gmail.com> Hello again, > jdk/test/java/util/zip/FlaterTest.java can be used to compare different JVMs; > I'm sorry but I don't think we have a better test. See also the code in > 6364346 and/or 6507183. Sorry if that sounds a bit dumb, how can I access jdk/test/java/util/zip/FlaterTest.java? I rewrote the native method a bit (because I thought it was messy), so I am afraid I destoyed some cases - are there also correctness testcases? > I don't see a block in the ArrayRegion functions (a little surprisingly): > hotspot/src/share/vm/prims/jni.cpp > (Search for just "arrayregion"). Compare the array region function body with > the primitivearraycritical function body; the latter obviously interacts with GC. I am no expert, but doesn't each jni-function block the jvm, like a synchronized modifier in java? I thought for this are JNI_ENTRY/JNI_END macros? I can be wrong of course, its just a guess. > I'm no expert here! See 6186200's discussion of GC & critical sections. Thanks a lot, very interesting :) lg Clemens Btw. No need to stress or hurry, its just a "holiday-fun-bugfix" From David.Bristor at Sun.COM Fri Dec 21 01:38:30 2007 From: David.Bristor at Sun.COM (Dave Bristor) Date: Thu, 20 Dec 2007 17:38:30 -0800 Subject: Performance regression in java.util.zip.Deflater In-Reply-To: <194f62550712201702n6f44efd5hda27c397e8d1ce96@mail.gmail.com> References: <194f62550712201120p1d10ac45xf86eb9cacd2eee87@mail.gmail.com> <476ADDAF.2070409@sun.com> <194f62550712201336y3380808bv3726d891873be277@mail.gmail.com> <476AEDCD.6080504@sun.com> <194f62550712201520p30d7b15wa8f2005749a77243@mail.gmail.com> <476B0ABA.6030102@sun.com> <194f62550712201702n6f44efd5hda27c397e8d1ce96@mail.gmail.com> Message-ID: <476B1916.2060502@sun.com> Clemens Eisserer wrote: > Hello again, > >> jdk/test/java/util/zip/FlaterTest.java can be used to compare different JVMs; >> I'm sorry but I don't think we have a better test. See also the code in >> 6364346 and/or 6507183. > Sorry if that sounds a bit dumb, how can I access > jdk/test/java/util/zip/FlaterTest.java? > I rewrote the native method a bit (because I thought it was messy), so > I am afraid I destoyed some cases - are there also correctness > testcases? Sorry: presuming you got the code via hg.openjdk.java.net, from e.g. jdk/tl/jdk, the test (and other zip tests) are in test/java/util/zip They're jtreg tests; you can run them using jtreg by hand, or by using the "jarzip" NetBeans project: openjdk.java.net/jtreg openjdk.java.net/groups/nb-projects >> I don't see a block in the ArrayRegion functions (a little surprisingly): >> hotspot/src/share/vm/prims/jni.cpp >> (Search for just "arrayregion"). Compare the array region function body with >> the primitivearraycritical function body; the latter obviously interacts with GC. > I am no expert, but doesn't each jni-function block the jvm, like a > synchronized modifier in java? I thought for this are > JNI_ENTRY/JNI_END macros? > I can be wrong of course, its just a guess. I suppose, but didn't find the definitions of those (yet). Dave >> I'm no expert here! See 6186200's discussion of GC & critical sections. > Thanks a lot, very interesting :) > > lg Clemens > > Btw. No need to stress or hurry, its just a "holiday-fun-bugfix" From jackieict at gmail.com Fri Dec 21 02:22:31 2007 From: jackieict at gmail.com (zhang Jackie) Date: Fri, 21 Dec 2007 10:22:31 +0800 Subject: two directbuffer have same address? Message-ID: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> hi I am using directbuffer and JNI to build a high speed network library. But When I use the directbuffer, I found some directbuffers have the same memory address(got by GetDirectBufferAddress method). I think the memory allocation of directbuffer is not reasonable enough. Can someoen give me some advice, Thanks Best Regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at mernst.org Fri Dec 21 08:28:23 2007 From: matthias at mernst.org (Matthias Ernst) Date: Fri, 21 Dec 2007 09:28:23 +0100 Subject: Fwd: two directbuffer have same address? In-Reply-To: <22ec15240712202359t1bc7a51fv7abe56954980ad4e@mail.gmail.com> References: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> <22ec15240712202359t1bc7a51fv7abe56954980ad4e@mail.gmail.com> Message-ID: <22ec15240712210028g28c2a66exe42b603644b054e6@mail.gmail.com> In Java, DirectByteBuffer#duplicate() will give you two buffers that share the same address. Also, from C, you can call NewDirectByteBuffer twice with the same address. Thinking about it, both functions kind of look like they might allocate memory for you but they don't! Did you maybe fall into that trap? Matthias On Dec 21, 2007 3:22 AM, zhang Jackie wrote: > hi > I am using directbuffer and JNI to build a high speed network library. But > When I use the directbuffer, I found some directbuffers have the same memory > address(got by GetDirectBufferAddress method). I think the memory > allocation of directbuffer is not reasonable enough. > Can someoen give me some advice, Thanks From forax at univ-mlv.fr Fri Dec 21 08:40:47 2007 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 21 Dec 2007 09:40:47 +0100 Subject: two directbuffer have same address? In-Reply-To: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> References: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> Message-ID: <476B7C0F.5030400@univ-mlv.fr> zhang Jackie a ?crit : > hi > I am using directbuffer and JNI to build a high speed network > library. But When I use the directbuffer, I found some directbuffers > have the same memory address(got by GetDirectBufferAddress method). I > think the memory allocation of directbuffer is not reasonable enough. > Can someoen give me some advice, Thanks > > Best Regards! I don't understand the whole question, but buffers can share the same content. Take a look to: http://download.java.net/jdk7/docs/api/java/nio/ByteBuffer.html#duplicate() R?mi From Alan.Bateman at Sun.COM Fri Dec 21 08:41:35 2007 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 21 Dec 2007 08:41:35 +0000 Subject: two directbuffer have same address? In-Reply-To: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> References: <13432ab00712201822w1ec9c630gbf48d6988531524f@mail.gmail.com> Message-ID: <476B7C3F.3000802@sun.com> zhang Jackie wrote: > hi > I am using directbuffer and JNI to build a high speed network > library. But When I use the directbuffer, I found some directbuffers > have the same memory address(got by GetDirectBufferAddress method). I > think the memory allocation of directbuffer is not reasonable enough. > Can someoen give me some advice, Thanks This has happen when you duplicate a byte buffer. Another possibility is JNI code that allocates more than one direct byte buffer referring to the same memory region. If you can elaborate further then this might become clear (although the forums might be a better place to seek help). -Alan. From jackieict at gmail.com Sat Dec 22 03:37:06 2007 From: jackieict at gmail.com (zhang Jackie) Date: Sat, 22 Dec 2007 11:37:06 +0800 Subject: Tools used for check whether an object is accessed or shared by other threads? Message-ID: <13432ab00712211937x4a9d1dd1o2b9fc5d1dabefaae@mail.gmail.com> Hi Is there any such tools? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gkorland at gmail.com Sat Dec 22 22:47:42 2007 From: gkorland at gmail.com (Guy Korland) Date: Sun, 23 Dec 2007 00:47:42 +0200 Subject: logging package improvments Message-ID: <79be5fa30712221447r564ad49dqd1e5b025c3458960@mail.gmail.com> Hi, It seems like there are few trivial improvement but critical for the logger performance. I think it's about time to refactor the module. e.g.: ** *1. LogRecord (line 147-148):* *synchronized* (LogRecord.*class*) { sequenceNumber = globalSequenceNumber++; *[Guy]**This is a real concurrency killer, isn't AtomicLong a much better solution?* *2. LogRecord (line 149-154):* Integer id = threadIds.get(); // This is a ThreadLocal? *if* (id == *null*) { id = *new* Integer(nextThreadId++); threadIds.set(id); } threadID = id.intValue(); *[Guy] **Isn't Thread.currentThread().getId() a better solution?* *3. LogRecord (line 143-144):* // Make sure level isn't null, by calling random method. level.getClass(); *[Guy] Is it really the best way to write a fast, readable and maintainable code???* Guy -------------- next part -------------- An HTML attachment was scrubbed... URL: