ClassLoader.loadLibrary NullPointerException Regression in 8u242

[Jaikiran Pai]

This seemed like an interesting issue, so I looked into this a bit. Like
Anton, I too was able to reproduce this issue, thanks to the
instructions provided.

The exception:

Caused by: java.lang.NullPointerException
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1847)
    at java.lang.Runtime.loadLibrary0(Runtime.java:871)
    at java.lang.System.loadLibrary(System.java:1124)
...

Looking at the code in java.lang.ClassLoader.java (of jdk8u), line 1847
is this:

for (int i = 0 ; i < sys_paths.length ; i++) {

so the only possibility is sys_paths is null. This is a static member of
the ClassLoader class and gets initialized through the
"initLibraryPaths()" method which is expected to be called from
java.lang.System class (very early on during bootstrap). Once
initialized, this sys_paths member is never written to and the code in
initLibraryPaths() is such that the initialization of this field is
always a non-null array.

After some basic checks to make sure initLibraryPaths() was indeed being
called, I decided to check if there were any race conditions involved. I
used Byteman[1] to instrument (various locations of) the
java.lang.ClassLoader. Based on the script[2], I could see that the
sys_paths is all fine and has a valid non-null value up until the point
where teamcity webapp page is accessed and asked to proceed with
installation. At that point, for no apparent reason the sys_path goes
null when the loadLibrary(...) call happens on ClassLoader class. After
an hour or so of fiddling around with the script and figuring out which
class to instrument, I decided to see if anyone was messing up the
ClassLoader class instance with any reflection. Adding a rule[3] for
tracing calls to java.lang.reflect.Field#set method, showed the culprit:

*** Field class java.lang.ClassLoader#sys_paths is being written to on
instance null to value null java.lang.reflect.Field.set(Field.java:-1)
jetbrains.buildServer.NativeLibraryPathHacker.<init>(NativeLibraryPathHacker.java:44)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-2)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
jetbrains.buildServer.spring.InstantiationStrategySelector$1.instantiate(InstantiationStrategySelector.java:77)

So jetbrains.buildServer.NativeLibraryPathHacker seems to be messing up
the classloader instance. I'm not sure why it's doing what it is doing
and I couldn't find its sources online (probably not open source).

So this looks more like an issue in the JetBrains Teamcity product and
perhaps needs to be raised there.

P.S: Chris, I updated your Dockerfile to use the non-corretto OpenJDK
binary, just to rule out any vendor issues and instead used the one
hosted at AdoptOpenJDK, which too reproduced this issue.

[1] https://byteman.jboss.org/

[2]
https://github.com/jaikiran/teamcity-docker-server/commit/2d8f72a6cb094c7272f38f605e6d0d2d8563faf6#diff-8b25b6ce5ce04dc13336cf72d144ecd0

[3]
https://github.com/jaikiran/teamcity-docker-server/commit/2d8f72a6cb094c7272f38f605e6d0d2d8563faf6#diff-8b25b6ce5ce04dc13336cf72d144ecd0R28

-Jaikiran



On 27/01/20 10:45 pm, Anton Kozlov wrote:

>
> On 27.01.2020 17:27, Chris Smith wrote:
>> There is another way to reproduce the bug but it's more complicated
>> than just starting the server.
> Thank you for such detailed instructions. I was able to reproduced the issue.
>
> I tried inserting ad-hoc synchronization but it didn't help. Looking for root cause.
>
> Thanks,
> Anton
>