RFR: 8308184: Launching java with large number of jars in classpath with java.protocol.handler.pkgs system property set can lead to StackOverflowError [v3]

[Jaikiran Pai]

> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8308184?
> 
> When an application is launched, the `app` classloader internally uses a `jdk.internal.loader.URLClassPath` to find and load the main class being launched. The `URLClassPath` uses a list of classpath entries to find resources. Depending on the classpath entry, the `URLClassPath` will use a relevant loader. A couple of such loaders are `URLClassPath$FileLoader` (for loading resources from directories) and `URLClassPath$JarLoader` (for loading resources from jar files).
> 
> `JarLoader` creates instances of `java.net.URL` to represent the jar file being loaded. `java.net.URL` uses protocol specific `java.net.URLStreamHandler` instance to handle connections to those URLs. When constructing an instance of `URL`, callers can pass a protocol handler. If it is not passed then the `URL` class looks for protocol handlers that might have been configured by the application. The `java.protocol.handler.pkgs` system property is the one which allows overriding the protocol handlers (even for the `jar` protocol). When this property is set, the `URL` class triggers lookup and classloading of the protocol handler classes.
> 
> The issue that is reported is triggered when the `java.protocol.handler.pkgs` system property is set and the classpath has too many jar files. `app` classloader triggers lookup of the main class and the `URLClassPath` picks up the first entry in the classpath and uses a `JarLoader` (in this example our classpath entries have a jar file at the beginning of the list). The `JarLoader` instantiates a `java.net.URL`, which notices that the `java.protocol.handler.pkgs` is set, so it now triggers lookup of a (different) class using the same classloader and thus the same `URLClassPath`. The `URLClassPath` picks the next classpath entry and then calls into the `URL` again through the `JarLoader`. This sequence ends up being re-entrant calls and given the large number of classpath entries, these re-entrant calls end up with a `StackOverflowError` as shown in the linked JBS issue.
> 
> The commit in this PR fixes this issue by using the system provided protocol handler implementation of the `jar` protocol in the `app` classloader. This results in the `URL` instances created through the `JarLoader` to use this specific handler instance. This allows the `app` classloader which is responsible for loading the application's main class to reliably use th...

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

  newline at end of test file

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/14395/files
  - new: https://git.openjdk.org/jdk/pull/14395/files/2fdb5679..b1260557

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=14395&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=14395&range=01-02

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/14395.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14395/head:pull/14395

PR: https://git.openjdk.org/jdk/pull/14395