RFR: Ensure compensate when ForkJoinWorkerThread is pinned

[Ron Pressler]

Hi.

The problem here isn’t that all of the scheduler’s workers are pinned, but that
the pinned threads can only be unblocked by another, unpinned, virtual thread. 
You might reasonably have 100K threads waiting, and no amount of compensation 
would help. As it is, compensation would only help up until the upper FJPool limit, 
which is quite low.

It makes sense to try and compensate when there’s a good chance the pinned thread 
is waiting for some other pinned thread, a platform thread, or an OS operation, and so 
we can smooth over minor and temporary disturbances. But cases like yours are 
fundamentally unbounded.

In light of that, I think the current behaviour is the right one. With compensation, 
the problem would manifest when you have ~300 threads rather than ~10, but given
that virtual threads are intended to be used when there’s need for many thousands
of them, compensation for arbitrary pinning is unlikely to significantly help in 
real-world instances. Worse — my fear is that it might make problems that could 
manifest in production harder to spot in testing.

The solution is to rely on the available diagnostics to identify the problem in the 
program — that a virtual thread can wait for another virtual thread while pinned
— and fix it.

— Ron

> On 29 Oct 2021, at 09:26, Miao Zheng <duke at openjdk.java.net> wrote:
> 
> Hello,
> We have a test case about Loom, it can pass before;
> The test case can be seen on this PR;
> 
> But it cannot pass after this patch:
> 
> commit de90a43c7a5f0b28bd0af9271de210475808ca0c
> Author: Alan Bateman <alan.bateman at oracle.com>
> Date:   Wed Sep 1 09:22:23 2021 +0100
> 
>    Replace lock/condition objects to simplify pinned park
> 
> 
> Actually, this modify cause the test fail, because getCondition.await() will call tryCompensate() of ForkJoinPool eventually:
> 
> +    private void parkOnCarrierThread(long nanos) {
> 
> -                if (!parkPermit) {
> -                    // wait to be signalled or interrupted
> -                    getCondition().await();
> 
> +            if (!parkPermit) {
> +                U.park(false, nanos);
>             }
> ``` 
> If ForkJoinWorkerThread are all pinned, how to make vt work correct?
> 
> Thanks，
> 
> Miao
> 
> -------------
> 
> Commit messages:
> - Ensure compensate when ForkJoinWorkerThread is pinned
> 
> Changes: https://git.openjdk.java.net/loom/pull/80/files
> Webrev: https://webrevs.openjdk.java.net/?repo=loom&pr=80&range=00
>  Stats: 87 lines in 1 file changed: 87 ins; 0 del; 0 mod
>  Patch: https://git.openjdk.java.net/loom/pull/80.diff
>  Fetch: git fetch https://git.openjdk.java.net/loom pull/80/head:pull/80
> 
> PR: https://git.openjdk.java.net/loom/pull/80