performance updates to jar and zip

[Xueming Shen]

Hi Mike,

It appears the patch failed to patch the ZipCoder for my JDK8 workspace 
(I did not look into
details, guess the patch is against and "old" version, if you can just 
send me the updated
ZipCoder file, I can add it into the workspace directly)  I pull the 
rest change into a webrev
at

http://cr.openjdk.java.net/~sherman/mtjar/webrev/ 
<http://cr.openjdk.java.net/%7Esherman/mtjar/webrev/>

I have not looked into the change detail yet. And probably also need 
find a real 4+ cores
machine to try on:-)

-Sherman

On 10/20/2011 03:55 PM, Mike Skells wrote:
> Hi All,
> I have some performance updates for the jar tool and for the Zip/Jar writing components, including some code to allow parallel writing of Jar and ZIP files (in java.util) 
>
> This work is not finished as yet but I am looking to see if anyone has any views as to the shape this should move in
> Currently it is a testbed for comparing different techniques, but largely based on the Jar utility
>
> The changes allow the work to be spread across multiple CPUs and optimise the some of the code and I/O paths
>
> This comparative figures do not include the effect of the nio changes that I proposed in earlier emails
>
> Command line changes
> 0--9 - I have added support for specifying different compression levels (the existing jar command just allows default compression or '0' for no compression
> D This allows the files to all be written with the date of now, lather than the file date  (the conversion of the date to zip format is a CPU hog, and not needed in some use-cases)
> Z0-5 - these are the different mechanisms to allow different parallel execution models - I would not expect this to be a production qualifier
>
> The test environment is a 4 core Intel core2 pc running windows  vista 64, the test case is jaring up the content of rt.jar to a jar file. 
> Each test is repeated 6 times and the last 5 are averaged to produce the answers. Each test is run in a fresh VM
>
> The performance figures are below as a CSV. The last column is the duration of the task in ms.
>
> In summary the existing jar utility takes (for uncompressed, compressed) 8.4 , 9.4 seconds to complete and this can be reduced to 1.6, 2.3 seconds  
>
> The different parallel algorithms are 
> 0 - none all in one thread as before
> 1 - file scanning in one core, 10 threads loading and buffering files, zip writing in a single thread using the existing ZipOuputStream
> 2. - file scanning in one core, 10 threads loading and buffering files, zip writing mostly mutithreaded (e.g. parallel compression, single write to the output stream)
> 3 - as 2 but writes to a file rather than a stream
> 4. as 2 but uses channels to be to write with direct buffers
> 5 as 4 but using heap buffers
>
> 3-5 have the zip capability in the code to seek and update headers that are incomplete, but this is not much tested
>   
>
>
>
> C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program Files\Java\jdk1.6.0_24\lib\tools.jar, -cf0, java 1.6 rt -cf0, 8482
> C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program Files\Java\jdk1.6.0_24\lib\tools.jar, -cf, java 1.6 rt -cf, 9318
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program Files\Java\jdk1.7.0\lib\tools.jar, -cf0, java 1.7 rt -cf0, 8497
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program Files\Java\jdk1.7.0\lib\tools.jar, -cf, java 1.7 rt -cf, 9518
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar, -cf0, orig 1.7 rt -cf0, 8448
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar, -cf, orig 1.7 rt -cf, 9484
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0, project 1.7 rt -cf0, 3133
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0D, project 1.7 rt -cf0D, 2824
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0Z0, project 1.7 rt -cf0 parallel 0, 3026
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ0, project 1.7 rt -cf0D parallel 0, 2961
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ1, project 1.7 rt -cf0D parallel 1, 2022
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ2, project 1.7 rt -cf0D parallel 2, 1757
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ3, project 1.7 rt -cf0D parallel 3, 1632
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ4, project 1.7 rt -cf0D parallel 4, 1994
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ5, project 1.7 rt -cf0D parallel 5, 1978
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1, project 1.7 rt -cf1, 5237
>
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1D, project 1.7 rt -cf1D, 5073
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1Z0, project 1.7 rt -cf1 parallel 0, 5367
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ0, project 1.7 rt -cf1D parallel 0, 5002
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ1, project 1.7 rt -cf1D parallel 1, 5125
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ2, project 1.7 rt -cf1D parallel 2, 2257
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ3, project 1.7 rt -cf1D parallel 3, 2145
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ4, project 1.7 rt -cf1D parallel 4, 2505
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ5, project 1.7 rt -cf1D parallel 5, 2549
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf2, project 1.7 rt -cf2, 5371
>
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf3, project 1.7 rt -cf3, 5409
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf4, project 1.7 rt -cf4, 5778
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf5, project 1.7 rt -cf5, 5906
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6, project 1.7 rt -cf6, 6082
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf7, project 1.7 rt -cf7, 6070
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf8, project 1.7 rt -cf8, 6251
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9, project 1.7 rt -cf9, 6191
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6D, project 1.7 rt -cf6D, 5843
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6Z0, project 1.7 rt -cf6 parallel 0, 6095
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ0, project 1.7 rt -cf6D parallel 0, 5907
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ1, project 1.7 rt -cf6D parallel 1, 5957
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ2, project 1.7 rt -cf6D parallel 2, 2388
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ3, project 1.7 rt -cf6D parallel 3, 2351
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ4, project 1.7 rt -cf6D parallel 4, 2694
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ5, project 1.7 rt -cf6D parallel 5, 2830
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9D, project 1.7 rt -cf9D, 6134
>
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9Z0, project 1.7 rt -cf9 parallel 0, 6258
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ0, project 1.7 rt -cf9D parallel 0, 6066
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ1, project 1.7 rt -cf9D parallel 1, 6203
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ2, project 1.7 rt -cf9D parallel 2, 2490
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ3, project 1.7 rt -cf9D parallel 3, 2361
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ4, project 1.7 rt -cf9D parallel 4, 2788
> C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ5, project 1.7 rt -cf9D parallel 5, 2847
>
> regards
> Mike