: Re: effectiveness of jdk.virtualThreadScheduler.maxPoolSize

[Arnaud Masson]

The average latency of different request types doesn’t matter much, I think.
What matters (for “interactive” web apps at least) is rather the request latency relatively to its own intrinsic duration.
Example:
Adding 1s to a 10s request is not much worse than adding 10ms to a 100ms request.
Adding 1s to a 100ms request is a more serious problem.

Not sure to understand the last part “some fixed set of background processing operations”, is it about using non-Loom/classic executor when time sharing is more needed? (It was considered as useless when it was suggested earlier as a workaround, so I’m a bit confused).
Sounds like a good workaround but it adds some complexity for the developer that must distinguish in advance which request must go to Loom Executor and which request must go to classic Executor. (Something Go developers don’t have to worry about now.)

Thanks
Arnaud



On 9 Jan 2023, at 18:34, Robert Engels <rengels at ix.netcom.com<mailto:rengels at ix.netcom.com>> wrote:

I think what is not given enough weight in the analysis is that long running tasks are usually deprioritized by the scheduler - not equally time slices - reducing the latency for short requests - increasing the latency for long/batch requests.

Actually, this is covered as a special case. Even assuming perfect time-sharing, its effectiveness for virtual thread use cases is unclear until we obtain more data from the field.



This is expected today based on the Linux (and many other) schedulers. The vthread scheduler is breaking from this - which is fine if it has good reasons to do so.

An OS scheduler must be a reasonable compromise for many kinds of threads. Virtual threads are optimised for transaction-processing workloads. I assume it will take some years to gather sufficient information from the field so that we can tweak our decisions based on data, but after spending years considering various hypotheses, I don’t see a reason to change what we have now without obtaining more data.



You can read the Go rationale for adding time slicing into the scheduler here https://github.com/golang/go/issues/10958<https://github.com/golang/go/issues/10958>

There were multiple issues it addressed - I’m not sure all of them apply to Java.

When we studied those motivations some years ago, it appeared they do not apply to Java. Once again, we must only solve problems faced by Java developers in the field.

— Ron




On Jan 9, 2023, at 12:19 PM, Ron Pressler <ron.pressler at oracle.com<mailto:ron.pressler at oracle.com>> wrote:

I think it would be interesting to explain in more detail the effects of scheduling, and why the question of time-sharing is not obvious and so crucially depends on real-world data.

Suppose you are given 10 tasks, 5 of them have a processing duration of 0ms, and 5 of them have a duration of 100ms. For simplicity, let’s assume we have no parallelism. Both a shortest-task-first and a longest-task-first will complete all tasks in 500ms, but their average task latency will be quite different. That of the shortest-task-first scheduler will be 150ms (= (5*0 + 100 + 200 + 300 + 400 + 500)/10), while that of the longest-task-first scheduler will be 400ms (= 100 + 200 + 300 + 400 + 500 + 5*500)/10). A perfect time-sharing scheduler (with zero overhead and infinitesimal granularity) would yield an average latency of 250ms (= 0*5 + 500*5). Those are big differences!

But now let’s consider a server where an infinite stream of requests arrive from the outside, half of them with a processing duration of 0ms and half with a duration of 100ms. Regardless of how we schedule the requests in the queue that may form, because the average request duration is 50ms, as long as the request rate is less than or equal to 20 req/s the server will be stable. If the rate is higher than that, the server will become unstable with requests piling up in an ever-growing queue and the latency will climb to infinity — again, regardless of scheduling policy.

What about latency? The average latency will depend on the distribution of requests. Without time-sharing, it can range between 50ms and 100ms; with perfect time-sharing it may be much higher (i.e. worse). Perfect time-sharing will decrease the latency of the short tasks (to 0ms!) at the expense of increasing the latency of the long tasks.

But say we conclude that reducing latencies of short tasks at the expense of long tasks is what everyone always wants; that’s not entirely obvious, but not completely unreasonable, either. Suppose that at the same request rate of 20 per second, the probability for a long task weren't 0.5 but 0.55, or that instead of 100ms it takes 110 ms. In that situation time sharing can no longer help — the server will destabilise. Alternatively, suppose that the probability of a long task is 0.05 or that its duration is 50 ms; time sharing is no longer effective, either. So at 20 req/s, within the band of 50-100ms or 0.05-0.5 probability time sharing can help; above it or below it — it doesn’t.

Keeping in mind that time-sharing can’t actually be perfect and that no request can actually have a duration of zero, I hope it is now clearer why we’re so curious to find real-world cases and why simulations provide little insight. It’s easy to construct an artificial simulation at the operational band where time-sharing is effective, but it’s precisely because, in practice, it is most effective when the server is on the edge of stability and becomes gradually less effective the further away we are from that tipping-point that the most important questions become: how often do servers operate within that operational band, where exactly along that band do they commonly find themselves, and how does that situation arise in real-world scenarios? Only when we get real-wold data can we answer those questions and can consider the pros and cons, and only then can we either conclude that the work isn’t worth it or be able to satisfactorily justify it.

(Note that for the classic case where time sharing helps— some fixed set of background processing operations — there is no need to add time-sharing to the virtual thread scheduler, as a better solution is already available.)

— Ron


Unless otherwise stated above:

Compagnie IBM France
Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 664 069 390,60 €
SIRET : 552 118 465 03644 - Code NAF 6203Z
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20230109/4d0d3b8d/attachment-0001.htm>