Fix for dealock issue in JDK's SPNEGO resulted in another deadlock

Peter Hansson peterhansson_se at yahoo.com
Sun Aug 27 13:59:50 UTC 2017




Hi

I'm trying to fix a deadlock in NetBeans IDE and NetBeans Platform. The deadlock is a result of the fix for JDK-8032832 which introduced a lock on the classloader. Unfortunately this fix just introduces another bug (reported as JDK-8068184). I'm now soliciting advice from this list.

The bug essentially causes NetBeans IDE (or applications build on NetBeans Platform) to freeze on startup if the site is using a proxy which uses 'Negotiate' auth scheme.


The reason for the deadlock is straightforward:  

When doing SPNEGO exchange there's now (as a result of the fix) a lock on the classloader in NegotiateAuthentication.java:


   synchronized (loader) { 

       return isSupportedImpl(hci); 

   }


Now, on Windows it is very likely that the TGT cannot be obtained and for this reason the Authenticator gets invoked. In fact it is almost certain that a Java SE client application on Windows engaging in SPNEGO will have to invoke the Authenticator and for this reason the freeze affects A LOT of people. 


The Authenticator will get invoked WHILE there's a lock on the classloader. NetBeans' Authenticator will prompt the user for credentials using a dialog. Since NetBeans is Swing based this will happen Swing's EDT. Bringing up a dialog will almost inevitably require classloading (especially in the beginning of an application's life) and the EDT will then get stuck because the classloader is locked and because NetBeans' classloader also uses locking. This is what causes the freeze. The network task thread has a lock on the classloader but at the same time it is waiting for the user to respond to a dialog but this will never happen because the UI thread is stuck on class loading.


I see at least two avenues that can be pursued:


1. Fix in JDK: The fix introduced in JDK-8032832, putting a lock on the classloader, seems to me to have lots of unwanted side-effects. Was this really the only way to fix the reported issue?  Could this fix be revisited?

2. Fix in NetBeans: I'm not an expert on NetBeans' classloader. It seems to be me to be implemented at a time when locking on the classloader itself (using syncronized loadClass() method) was the general recommendation. But with the changes done in Java 7 to the default classloader the recommendations changed. 

http://docs.oracle.com/javase/7/docs/technotes/guides/lang/cl-mt.htmlSo a possible solution may be to change the NetBeans classloader to be more granular on syncronization along the lines of what the standard classloader does as of Java 7. I don't know if this is feasible at all. The NetBeans classloader(s) is really the magic sauce behind NetBeans' much acclaimed module system which underpins *everything* on the platform. So, I guess, less the original authors, there just isn't many people who would dare touch that piece of code. 


and there may be more solutions that I have overlooked.  :-)



Pls advice.

/Peter



More information about the security-dev mailing list