Another virtual threads migration story: ReentrantReadWriteLock

[robert engels]

But tbh, blocking that many threads seems doesn’t seem efficient or performant. It is isn’t cheap. I would think that a copy on write for the configuration change would be a better solution.

> On Jan 29, 2025, at 11:23 AM, robert engels <rengels at ix.netcom.com> wrote:
> 
> Nice catch! I am not sure you are going to get a resolution on this other than using your own implementation. 
> 
> The AbstractQueuedSynchronizer needs to be changed to use a long to hold the state - which will break subclasses - so it probably won’t happen.
> 
>> On Jan 29, 2025, at 10:58 AM, Matthew Swift <matthew.swift at gmail.com> wrote:
>> 
>> Hi folks,
>> 
>> As you may remember from a few months ago, we converted our LDAP Directory server/proxy[1] over to using virtual threads. It's been going pretty well and we're lucky enough to be able to leverage JDK21 as we have full control over most (not all) of the code-base, which puts us in the enviable position where we can convert code to avoid thread pinning issues. That being said, we regularly test using the latest JDK24 EA builds as well.
>> 
>> We recently hit what I feel is quite a major limitation in ReentrantReadWriteLock, which was somewhat hidden before in the old world of large-but-not-super-large platform thread pools:
>> 
>>    Error: Maximum lock count exceeded at ReentrantReadWriteLock.java:535,494 AbstractQueuedSynchronizer.java:1078 ReentrantReadWriteLock.java:738 ...
>> 
>> I'm sure that we're not alone in making extensive use of RW locks for synchronizing configuration changes to runtime components: the write lock ensures that regular processing is paused while the configuration change is applied. The component in this case could be something that talks to a remote microservice over HTTP, a logging backend, etc. In this case, there is no configuration change - just a few 100s millisecond latency in the remote service for some reason (e.g. GC pause?), which has caused many virtual threads to get blocked inside the component while holding the read lock. The RW lock then fails with the above error once there are 64K concurrent threads holding the read lock.
>> 
>> Given that scaling IO to millions of concurrent IO bound tasks was one of the key motivations for vthreads, it seems a bit surprising to me that a basic concurrency building block of many applications is constrained to 64K concurrent accesses. Are you aware of this limitation and its implications? A workaround now is to go hunting for RW locks in our application and using alternative approaches OR, where the lock is in a third party library (e.g. logging / telemetry), wrapping the library calls in a Semaphore limited to <64K permits. It seems a bit unsatisfactory to me. What do you think? Are there plans to implement a RW lock based on AbstractQueuedLongSynchronizer?
>> 
>> Kind regards,
>> Matt
>> 
>> [1] those unfamiliar with the tech, think of it is a distributed database for storing identities
>