RFR: 8156484: ZipFile retains too much native memory from caching Inflaters
Peter Levart
peter.levart at gmail.com
Mon May 23 21:28:35 UTC 2016
Hi Sherman, Martin,
On 05/23/2016 08:28 PM, Xueming Shen wrote:
> My apology. I meant to say "to have Inflater to be aware of the
> ObjectPool...".
> The proposed change switches from the finalizer to the ObjectPool to
> clean up
> the native resource for Inflater. It appears to be a bigger change.
> Which has
> a default 32/maxCacheSize and 1 secod keepAliveTime setting. It might have
> a big impact to existing applications on how to use the Inflater. It
> would be
> fine to have the ZipFile to arrange how to use the Inflater based on
> its use
> scenario, but I'm concerned if it's the correct approach to change the
> default
> behavior of Inflater, which might have nothing to do with ZipFile.
>
> sherman
It might have a big impact to existing applications, yes. A positive one!
On 05/23/2016 09:21 PM, Martin Buchholz wrote:
> ....
> I am still contending that zstream caching is not worth much. The
> best I could do with a synthetic benchmark designed to make caching
> win was 25% better performance. Martin's rule is if you can't even
> come up with a friendly benchmark that gives you a 2x win, give up on
> your optimization! So performance wise, there is no big impact here
> on any applications, except to fix excessive memory retention and
> OOME.
>
>
Here's the 1st test I did. The micro benchmark Martin created and is
attached to the issue 8156484. Just modified a bit to issue a heap dump
at the end:
public class Benchmark60kMulti {
private static void dumpHistogram() {
// dump a heap histogram
try {
new ProcessBuilder(
System.getProperty("java.home") + "/bin/jmap",
"-histo:live",
String.valueOf(ProcessHandle.current().getPid())
).redirectOutput(ProcessBuilder.Redirect.to(
new File("histo_live.txt"))
).start().waitFor();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) throws Throwable {
final int REPS = 20;
final int THREADS = 8;
final byte[] data = "123456789".getBytes("UTF-8");
final File file = new File("60k.zip");
if (!file.exists()) {
try (FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
ZipOutputStream zos = new ZipOutputStream(bos)) {
for (int i = 0; i < 60_000; i++) {
ZipEntry ze = new ZipEntry(i + ".txt");
ze.setMethod(ZipEntry.DEFLATED);
ze.setSize(0);
ze.setCrc(0);
zos.putNextEntry(ze);
zos.write(data);
}
}
}
Path target = Paths.get("60k.zip");
for (int i = 0; i < THREADS; i++) {
Path link = Paths.get("symlink-" + i + ".zip");
if (!link.toFile().exists())
Files.createSymbolicLink(link, target);
}
ExecutorService pool = Executors.newFixedThreadPool(THREADS);
ArrayList<Future<?>> futures = new ArrayList<>();
for (int i = 0; i < THREADS; i++) {
final Path link = Paths.get("symlink-" + i + ".zip");
final Runnable task = () -> {
try (ZipFile zipFile = new ZipFile(link.toFile())) {
for (int j = 0; j < REPS; j++) {
byte[] buf = new byte[100];
Enumeration<? extends ZipEntry> entries =
zipFile.entries();
while (entries.hasMoreElements()) {
ZipEntry zentry = entries.nextElement();
try (InputStream is =
zipFile.getInputStream(zentry)) {
is.read(buf);
}
}
}
} catch (Throwable t) { throw new Error(t); }
};
futures.add(pool.submit(task));
}
pool.shutdown();
for (Future future : futures) future.get();
dumpHistogram();
pool.awaitTermination(1, TimeUnit.DAYS);
}
}
"time" Linux command and a top-10 heap dump of live objects:
real 0m22.197s
user 1m34.968s
sys 0m12.577s
num #instances #bytes class name (module)
-------------------------------------------------------
1: 3621046 704046712 [B (java.base at 9-internal)
2: 2410133 96405320 java.lang.ref.Finalizer
(java.base at 9-internal)
3: 1572755 75492240
java.util.zip.ZipFile$ZipFileInputStream (java.base at 9-internal)
4: 1204735 67465160
java.util.zip.ZipFile$ZipFileInflaterInputStream (java.base at 9-internal)
5: 4597 147104 java.util.HashMap$Node
(java.base at 9-internal)
6: 5875 141000 java.lang.String (java.base at 9-internal)
7: 1140 138656 java.lang.Class (java.base at 9-internal)
8: 2681 85792
java.util.concurrent.ConcurrentHashMap$Node (java.base at 9-internal)
9: 1938 84760 [Ljava.lang.Object;
(java.base at 9-internal)
10: 636 61056 [Ljava.util.HashMap$Node;
(java.base at 9-internal)
With webrev.03 applied:
real 0m6.734s
user 0m32.937s
sys 0m13.760s
num #instances #bytes class name (module)
-------------------------------------------------------
1: 6903 483592 [B (java.base at 9-internal)
2: 4598 147136 java.util.HashMap$Node
(java.base at 9-internal)
3: 1177 143112 java.lang.Class (java.base at 9-internal)
4: 5936 142464 java.lang.String (java.base at 9-internal)
5: 2681 85792
java.util.concurrent.ConcurrentHashMap$Node (java.base at 9-internal)
6: 1943 84608 [Ljava.lang.Object;
(java.base at 9-internal)
7: 2 65568 [Ljava.util.concurrent.ForkJoinTask;
(java.base at 9-internal)
8: 637 61136 [Ljava.util.HashMap$Node;
(java.base at 9-internal)
9: 11 33696 [C (java.base at 9-internal)
10: 637 30576 java.util.HashMap
(java.base at 9-internal)
I think this is more than 2x win in speed, don't you? Not to mention memory.
Regards, Peter
More information about the core-libs-dev
mailing list