RFR: 8308184: Launching java with large number of jars in classpath with java.protocol.handler.pkgs system property set can lead to StackOverflowError

[Jaikiran Pai]

Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8308184?

When an application is launched, the `app` classloader internally uses a `jdk.internal.loader.URLClassPath` to find and load the main class being launched. The `URLClassPath` uses a list of classpath entries to find resources. Depending on the classpath entry, the `URLClassPath` will use a relevant loader. A couple of such loaders are `URLClassPath$FileLoader` (for loading resources from directories) and `URLClassPath$JarLoader` (for loading resources from jar files).

`JarLoader` creates instances of `java.net.URL` to represent the jar file being loaded. `java.net.URL` uses protocol specific `java.net.URLStreamHandler` instance to handle connections to those URLs. When constructing an instance of `URL`, callers can pass a protocol handler. If it is not passed then the `URL` class looks for protocol handlers that might have been configured by the application. The `java.protocol.handler.pkgs` system property is the one which allows overriding the protocol handlers (even for the `jar` protocol). When this property is set, the `URL` class triggers lookup and classloading of the protocol handler classes.

The issue that is reported is triggered when the `java.protocol.handler.pkgs` system property is set and the classpath has too many jar files. `app` classloader triggers lookup of the main class and the `URLClassPath` picks up the first entry in the classpath and uses a `JarLoader` (in this example our classpath entries have a jar file at the beginning of the list). The `JarLoader` instantiates a `java.net.URL`, which notices that the `java.protocol.handler.pkgs` is set, so it now triggers lookup of a (different) class using the same classloader and thus the same `URLClassPath`. The `URLClassPath` picks the next classpath entry and then calls into the `URL` again through the `JarLoader`. This sequence ends up being re-entrant calls and given the large number of classpath entries, these re-entrant calls end up with a `StackOverflowError` as shown in the linked JBS issue.

The commit in this PR fixes this issue by using the system provided protocol handler implementation of the `jar` protocol in the `app` classloader. This results in the `URL` instances created through the `JarLoader` to use this specific handler instance. This allows the `app` classloader which is responsible for loading the application's main class to reliably use the system provided protocol handler.

Do note that this doesn't in any way impact the ability of applications to override the protocol handler for `jar` protocol. Applications can still continue to do it by setting the `java.protocol.handler.pkgs` system property to configure their application specific implementation for the `jar` protocol (or other protocols). `URL` instances created by the application code will continue to use their overridden `jar` protocol handler. The commit in this PR merely fixes the jar protocol handler that is used by the `app` classloader.


A new jtreg test has been added which reproduces this issue and verifies the fix. tier testing is in progress.

-------------

Commit messages:
 - 8308184: Launching java with large number of jars in classpath with java.protocol.handler.pkgs system property set can lead to StackOverflowError

Changes: https://git.openjdk.org/jdk/pull/14395/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14395&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8308184
  Stats: 148 lines in 3 files changed: 143 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/14395.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14395/head:pull/14395

PR: https://git.openjdk.org/jdk/pull/14395