long classpaths and JDK-8162492

[Liam Miller-Cushon]

I noticed a performance regression with long classpaths and large numbers
of sources. This is related to JDK-8162492, but the benchmarks in that bug
are for long classpaths with a single source file, and my proposed fix
doesn't address that.

Under JDK 9 I'm seeing compilation time scale as the product of the
classpath length and the number of calls to getFileObject or list. The
example below takes 17s with JDK 8 and 8m40s with JDK 9.

I think the issue is with the cache strategy in ArchiveContainer.
ArchiveContainer calls Files.exists for each path and caches the result,
but if it's queried for a large number of distinct paths it doesn't get
cache hits, and ends up doing a lot of work.

I'd like to propose bringing back the cache strategy that was used in
ZipArchive:
http://hg.openjdk.java.net/jdk8/jdk8/langtools/file/1ff9d5118aae/src/share/classes/com/sun/tools/javac/file/ZipArchive.java#l73.
ZipArchive scans all entries in the zip once, and puts the directory names
in a cache. If getFileObject/list are called many times with paths that
aren't in the archive it doesn't have to do any work, and the cache stays
small.

I have a couple of question about how to proceed:

- does bringing back the ZipArchive-style cache sound reasonable, or are
there other advantages to the new approach used in ArchiveContainer?

- should I add this to JDK-8162492, or does it deserve a separate bug?


Here's the repro -

generate the test inputs (start in a fresh directory):

$ javac -cp asm.jar Test.java && java -cp asm.jar:. Test

compile:

$ $JAVAC8 -fullversion
javac full version "1.8.0_122-ea-b04"
$ time $JAVAC8 @params.txt
real 0m17.385s
user 0m40.855s
sys 0m1.305s

$ $JAVAC9 -fullversion
javac full version "9-ea+148"
$ time $JAVAC9 @params.txt
real 8m40.530s
user 32m8.614s
sys 0m57.024s

=== Test.java ===
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.objectweb.asm.Opcodes.ACC_PUBLIC;
import static org.objectweb.asm.Opcodes.ACC_SUPER;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.jar.JarEntry;
import java.util.jar.JarOutputStream;
import org.objectweb.asm.ClassWriter;

class Test {

    static final int JARS = 1000;
    static final int CLASSES = 100;
    static final int SOURCES = 1000;

    public static void main(String[] args) throws Exception {
        List<String> jars = new ArrayList<>();
        for (int i = 0; i < JARS; i++) {
            String jarName = "lib" + i + ".jar";
            jars.add(jarName);
            try (JarOutputStream jos =
                    new
JarOutputStream(Files.newOutputStream(Paths.get(jarName)))) {
                for (int j = 0; j < CLASSES; j++) {
                    String name = String.format("b%d%d/B%d%d", i, j, i, j);
                    jos.putNextEntry(new JarEntry(name + ".class"));
                    ClassWriter cw = new ClassWriter(0);
                    cw.visit(52, ACC_PUBLIC | ACC_SUPER, name, null,
"java/lang/Object", null);
                    jos.write(cw.toByteArray());
                }
            }
        }
        List<String> sources = new ArrayList<>();
        for (int i = 0; i < SOURCES; i++) {
            StringBuilder sb = new StringBuilder();
            sb.append(String.format("package a%d;\n", i));
            sb.append(String.format("class A%d {\n", i));
            for (int j = 0; j < CLASSES; j++) {
                String name = String.valueOf((i + j) % JARS) + j;
                sb.append(String.format("b%s.B%s x%d;\n", name, name, j));
            }
            sb.append(" }\n");
            String file = String.format("A%d.java", i, i);
            sources.add(file);
            Files.write(Paths.get(file), sb.toString().getBytes(UTF_8));
        }
        List<String> params = new ArrayList<>();
        params.addAll(sources);
        params.add("-cp");
        params.add(String.join(File.pathSeparator, jars));
        Files.write(Paths.get("params.txt"), params, UTF_8);
    }
}
===
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/compiler-dev/attachments/20161212/48943a24/attachment-0001.html>