performance updates to jar and zip

Fri Oct 21 08:08:19 UTC 2011

Hi Sherman,
yes the patch is from a very old base in Java 7

I enclose the ZipCoder and a test driver that I used to obtain the results

Regards

Mike


>________________________________
>From: Xueming Shen <xueming.shen at oracle.com>
>To: core-libs-dev at openjdk.java.net
>Sent: Friday, 21 October 2011, 0:50
>Subject: Re: performance updates to jar and zip
>
>Hi Mike,
>
>It appears the patch failed to patch the ZipCoder for my JDK8 workspace (I did not look into
>details, guess the patch is against and "old" version, if you can just send me the updated
>ZipCoder file, I can add it into the workspace directly)  I pull the rest change into a webrev
>at
>
>http://cr.openjdk.java.net/~sherman/mtjar/webrev/ <http://cr.openjdk.java.net/%7Esherman/mtjar/webrev/>
>
>I have not looked into the change detail yet. And probably also need find a real 4+ cores
>machine to try on:-)
>
>-Sherman
>
>On 10/20/2011 03:55 PM, Mike Skells wrote:
>> Hi All,
>> I have some performance updates for the jar tool and for the Zip/Jar writing components, including some code to allow parallel writing of Jar and ZIP files (in java.util) 
>> This work is not finished as yet but I am looking to see if anyone has any views as to the shape this should move in
>> Currently it is a testbed for comparing different techniques, but largely based on the Jar utility
>> 
>> The changes allow the work to be spread across multiple CPUs and optimise the some of the code and I/O paths
>> 
>> This comparative figures do not include the effect of the nio changes that I proposed in earlier emails
>> 
>> Command line changes
>> 0--9 - I have added support for specifying different compression levels (the existing jar command just allows default compression or '0' for no compression
>> D This allows the files to all be written with the date of now, lather than the file date  (the conversion of the date to zip format is a CPU hog, and not needed in some use-cases)
>> Z0-5 - these are the different mechanisms to allow different parallel execution models - I would not expect this to be a production qualifier
>> 
>> The test environment is a 4 core Intel core2 pc running windows  vista 64, the test case is jaring up the content of rt.jar to a jar file. Each test is repeated 6 times and the last 5 are averaged to produce the answers. Each test is run in a fresh VM
>> 
>> The performance figures are below as a CSV. The last column is the duration of the task in ms.
>> 
>> In summary the existing jar utility takes (for uncompressed, compressed) 8.4 , 9.4 seconds to complete and this can be reduced to 1.6, 2.3 seconds  
>> The different parallel algorithms are 0 - none all in one thread as before
>> 1 - file scanning in one core, 10 threads loading and buffering files, zip writing in a single thread using the existing ZipOuputStream
>> 2. - file scanning in one core, 10 threads loading and buffering files, zip writing mostly mutithreaded (e.g. parallel compression, single write to the output stream)
>> 3 - as 2 but writes to a file rather than a stream
>> 4. as 2 but uses channels to be to write with direct buffers
>> 5 as 4 but using heap buffers
>> 
>> 3-5 have the zip capability in the code to seek and update headers that are incomplete, but this is not much tested
>>  
>> 
>> 
>> C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program Files\Java\jdk1.6.0_24\lib\tools.jar, -cf0, java 1.6 rt -cf0, 8482
>> C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program Files\Java\jdk1.6.0_24\lib\tools.jar, -cf, java 1.6 rt -cf, 9318
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program Files\Java\jdk1.7.0\lib\tools.jar, -cf0, java 1.7 rt -cf0, 8497
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program Files\Java\jdk1.7.0\lib\tools.jar, -cf, java 1.7 rt -cf, 9518
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar, -cf0, orig 1.7 rt -cf0, 8448
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar, -cf, orig 1.7 rt -cf, 9484
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0, project 1.7 rt -cf0, 3133
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0D, project 1.7 rt -cf0D, 2824
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0Z0, project 1.7 rt -cf0 parallel 0, 3026
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ0, project 1.7 rt -cf0D parallel 0, 2961
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ1, project 1.7 rt -cf0D parallel 1, 2022
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ2, project 1.7 rt -cf0D parallel 2, 1757
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ3, project 1.7 rt -cf0D parallel 3, 1632
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ4, project 1.7 rt -cf0D parallel 4, 1994
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ5, project 1.7 rt -cf0D parallel 5, 1978
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1, project 1.7 rt -cf1, 5237
>> 
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1D, project 1.7 rt -cf1D, 5073
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1Z0, project 1.7 rt -cf1 parallel 0, 5367
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ0, project 1.7 rt -cf1D parallel 0, 5002
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ1, project 1.7 rt -cf1D parallel 1, 5125
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ2, project 1.7 rt -cf1D parallel 2, 2257
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ3, project 1.7 rt -cf1D parallel 3, 2145
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ4, project 1.7 rt -cf1D parallel 4, 2505
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ5, project 1.7 rt -cf1D parallel 5, 2549
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf2, project 1.7 rt -cf2, 5371
>> 
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf3, project 1.7 rt -cf3, 5409
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf4, project 1.7 rt -cf4, 5778
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf5, project 1.7 rt -cf5, 5906
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6, project 1.7 rt -cf6, 6082
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf7, project 1.7 rt -cf7, 6070
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf8, project 1.7 rt -cf8, 6251
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9, project 1.7 rt -cf9, 6191
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6D, project 1.7 rt -cf6D, 5843
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6Z0, project 1.7 rt -cf6 parallel 0, 6095
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ0, project 1.7 rt -cf6D parallel 0, 5907
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ1, project 1.7 rt -cf6D parallel 1, 5957
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ2, project 1.7 rt -cf6D parallel 2, 2388
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ3, project 1.7 rt -cf6D parallel 3, 2351
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ4, project 1.7 rt -cf6D parallel 4, 2694
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ5, project 1.7 rt -cf6D parallel 5, 2830
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9D, project 1.7 rt -cf9D, 6134
>> 
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9Z0, project 1.7 rt -cf9 parallel 0, 6258
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ0, project 1.7 rt -cf9D parallel 0, 6066
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ1, project 1.7 rt -cf9D parallel 1, 6203
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ2, project 1.7 rt -cf9D parallel 2, 2490
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ3, project 1.7 rt -cf9D parallel 3, 2361
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ4, project 1.7 rt -cf9D parallel 4, 2788
>> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ5, project 1.7 rt -cf9D parallel 5, 2847
>> 
>> regards
>> Mike
>
>
>
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ZipCoder.java.txt
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20111021/b40f2f7f/ZipCoder.java.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: TestJar.java.txt
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20111021/b40f2f7f/TestJar.java.txt>