IllegalStateException in CharsetDecoder when inspecting JarFile contents on Java 15

[Claes Redestad]

Filed: https://bugs.openjdk.java.net/browse/JDK-8260010

I have a potential fix ready, just trying to distill a minimal test case.

/Claes

On 2021-01-19 21:46, Claes Redestad wrote:
> Hi,
>
> yes, this seems like a bug introduced by JDK-8243469 where the 
> previously thread-safe UTF8ZipCoder is now using a thread-unsafe 
> decoder to calculate the normalized hash. I'll file a bug and take a 
> look.
>
> /Claes
>
> On 2021-01-19 20:19, Vitaly Davidovich wrote:
>> Hi all,
>>
>> I observed the following stacktrace when inspecting a JarFile's contents
>> using a parallel stream (i.e. FJ pool):
>>
>> Exception in thread "main" java.lang.IllegalStateException:
>> java.lang.IllegalStateException: Current state = CODING_END, new state =
>> FLUSHED
>>
>>                at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
>>
>> Method)
>>
>>                at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) 
>>
>>
>>                at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
>>
>>
>>                at
>> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) 
>>
>>
>>                at
>> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737) 
>>
>>
>>                at
>> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) 
>>
>>
>>                at
>> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) 
>>
>>
>>                at
>> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) 
>>
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) 
>>
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:575) 
>>
>>
>>                at 
>> <stripped>.ClassVersions.conflicts(ClassVersions.java:703)
>>
>>                at <stripped>.ClassVersions.check(ClassVersions.java:743)
>>
>>                at <stripped>.ClassVersions.main(ClassVersions.java:839)
>>
>> Caused by: java.lang.IllegalStateException: Current state = 
>> CODING_END, new
>> state = FLUSHED
>>
>>                at
>> java.base/java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:998) 
>>
>>
>>                at
>> java.base/java.nio.charset.CharsetDecoder.flush(CharsetDecoder.java:681)
>>
>>                at
>> java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810) 
>>
>>
>>                at
>> java.base/java.util.zip.ZipCoder.normalizedHashDecode(ZipCoder.java:136)
>>
>>                at
>> java.base/java.util.zip.ZipCoder$UTF8ZipCoder.normalizedHash(ZipCoder.java:228) 
>>
>>
>>                at
>> java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1527)
>>
>>                at
>> java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1249)
>>
>>                at
>> java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1211)
>>
>>                at
>> java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:701) 
>>
>>
>>                at 
>> java.base/java.util.zip.ZipFile.<init>(ZipFile.java:240)
>>
>>                at 
>> java.base/java.util.zip.ZipFile.<init>(ZipFile.java:171)
>>
>>                at 
>> java.base/java.util.jar.JarFile.<init>(JarFile.java:347)
>>
>>                at 
>> java.base/java.util.jar.JarFile.<init>(JarFile.java:318)
>>
>>                at 
>> java.base/java.util.jar.JarFile.<init>(JarFile.java:284)
>>
>>                at 
>> <stripped>.ClassVersions.lookupJar(ClassVersions.java:665)
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) 
>>
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) 
>>
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) 
>>
>>
>>                at
>> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) 
>>
>>
>>                at
>> java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) 
>>
>>
>>                at
>> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) 
>>
>>
>>                at
>> java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) 
>>
>>
>>                at
>> java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598) 
>>
>>
>>                at
>> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) 
>>
>>
>>
>> (Note: I stripped off some package names to protect the innocent).
>>
>>
>> While investigating this a bit, I noticed some code that looks a bit
>> suspect.  Namely:
>>
>>     - ZipCoder has a singleton UTF8ZipCoder
>> <https://github.com/openjdk/jdk/blob/05bcd67e6501f3824020edd1ab036c87888fa229/src/java.base/share/classes/java/util/zip/ZipCoder.java#L48>
>>     instance it uses for UTF8 decoding.
>>     - Source.get() creates a new Source
>> <https://github.com/openjdk/jdk/blob/05bcd67e6501f3824020edd1ab036c87888fa229/src/java.base/share/classes/java/util/zip/ZipFile.java#L1225>
>>     instance outside of any synchronization
>>
>> The links above are to tip, but I think this code is the same in Java 
>> 15.
>>
>> It seems possible that multiple threads can end up in Source.get(), 
>> new up
>> a Source, and proceed to use the decoder of UTF8ZipCoder racily.  Per my
>> understanding (and CharsetDecoder javadoc), CharsetDecoders are not
>> threadsafe.
>>
>> Any thoughts? Did I miss anything that would suggest the stacktrace 
>> above
>> is due to something else entirely?
>>
>> Thanks