[External] : Re: jstack, profilers and other tools

Tue Aug 2 07:34:23 UTC 2022

Hi Ron,

The "maximum capacity for concurrent requests" statement uses "concurrent"
in the sense of Java threads. Little's law counts queued requests as
"concurrent", too. Then assuming queue can be arbitrarily large, we can't
have a finite max capacity for concurrent requests.

Alex

On Sun, 31 Jul 2022, 17:38 Ron Pressler, <ron.pressler at oracle.com> wrote:

> A thread-per-request server under an average load 100req/s and an average
> request processing duration of 10ms will not destabilise if it can have no
> more than 10 threads, where “destabilise” means that the queue grows
> indefinitely. I.e. in a a stable system is defined as one where queues do
> not grow indefinitely.
>
> While it is true that for any queue size there is some shrinking though
> non-zero probability that it might be reached, it is absolutely unrelated
> to the server being thread-per-request. However it is that you choose to
> represent a request, if your maximum capacity for concurrent requests is at
> or above L, the system will be stable.
>
> — Ron
>
> On 31 Jul 2022, at 13:27, Alex Otenko <oleksandr.otenko at gmail.com> wrote:
>
> Hi Ron,
>
> I am glad you mentioned this visualisation experiment. That's the sort of
> experiment I proposed (and done) what seems like a week ago. You may find
> some unintuitive things coming from this experiment, since you called them
> untrue.
>
>
> Alex
>
> On Fri, 29 Jul 2022, 14:53 Ron Pressler, <ron.pressler at oracle.com> wrote:
>
>>
>> BTW, the easiest way to visualise the temporary discrepancies between the
>> averages in Little’s law and instantaneous behaviour is to think of the
>> queue forming at the entry to the system.
>>
>> If the system is unstable (i.e. the equation doesn’t hold), the queue
>> will grow without bounds. If it is stable, it can momentarily grow but will
>> then shrink as we regress to the mean. So suppose λ = 100 and W = 1/20, and
>> therefore the average concurrency 5. If the *maximum* capacity for
>> concurrent operations is also 5 (e.g we have a thread-per-request server
>> that uses just one thread per request and the maximum number of threads we
>> can support is 5), then if the rate of requests momentarily rises above 100
>> then the queue will grow, but will eventually shrink when the rate drops
>> below 100 (as it must).
>>
>> So if our throughput/rate-of-requests is expected to not exceed 100, we
>> should be fine supporting just 5 threads. To get a feel that this actually
>> works, consider that the world’s most popular thread-per-request servers
>> already work in exactly this way. Rather than spawning a brand new thread
>> for every request, they borrow one from a pool. The pool is normally fixed
>> and set to something like a few hundred threads by default. They work fine
>> as long as their throughput doesn’t exceed the maximum expected throughput,
>> where the threads in the pool (or perhaps some other resource) is
>> exhausted. I.e. they do *not* work by allowing for an unbounded number of
>> threads, but a bounded one; they are very much thread-per-request, yet
>> their threads are capped. This does place an upper limit on their
>> throughput, but they work fine until they reach it.
>>
>> Of course, if other resources are exhausted before the pool is depleted,
>> then (fanout aside) lightweight threads aren’t needed. But because there
>> are so many systems where the threads are the resource that’s exhausted
>> first, people invented non-thread-per-request (i.e. asynchronous) servers
>> as well as lightweight threads.
>>
>> — Ron
>>
>> On 29 Jul 2022, at 00:46, Ron Pressler <ron.pressler at oracle.com> wrote:
>>
>>
>>
>> On 28 Jul 2022, at 21:31, Alex Otenko <oleksandr.otenko at gmail.com> wrote:
>>
>> Hi Ron,
>>
>> The claim in JEP is the same as in this email thread, so that is not much
>> help. But now I don't need help anymore, because I found the explaination
>> how the thread count, response time, request rate and thread-per-request
>> are connected.
>>
>> Now what makes me bothered about the claims.
>>
>> Little's law connects throughput to concurrency. We agreed it has no
>> connection to thread count. That's a disconnect between the claim about
>> threads and Little's law dictating it.
>>
>>
>> Not really, no. In thread-per-request systems, the number of threads is
>> equal to (or perhaps greater than) the concurrency because that’s the
>> definition of thread-per-request. That’s why we agreed that in
>> thread-per-request systems, Little’s law tells us the (average) number of
>> threads.
>>
>>
>> There's also the assumption that response time remains constant, but
>> that's a mighty assumption - response time changes with thread count.
>>
>>
>> There is absolutely no such assumption. Unless the a rising request rate
>> causes the latency to significantly *drop*, the number of threads will
>> grow. If the latency happens to rise, the number of threads will grow even
>> faster.
>>
>>
>> There's also the claim of needing more threads. That's also not something
>> that follows from thread-per-request. Essentially, thread-per-request is a
>> constructive proof of having an infinite number of threads. How can one
>> want more? Also, from a different angle - the number of threads in the
>> thread-per-request needed does not depend on throughput at all.
>>
>>
>> The average number of requests being processed concurrently is equal to
>> the rate of requests (i.e. throughput) times the average latency. Again,
>> because a thread-per-request system is defined as one that assigns (at
>> least) one thread for every request, the number of threads is therefore
>> proportional to the throughput (as long as the system is stable).
>>
>>
>> Just consider what the request rate means. It means that if you choose
>> however small time frame, and however large request count, there is a
>> nonzero probability that it will happen. Consequently the number of threads
>> needed is arbitrarily large for any throughput.
>>
>>
>> Little’s theorem is about *long-term average* concurrency, latency and
>> throughput, and it is interesting precisely because it holds regardless of
>> the distribution of the requests.
>>
>> Which is just another way to say that the number of threads is
>> effectively infinite and there is no point trying to connect it to Little's
>> law.
>>
>>
>> I don’t understand what that means. A mathematical theorem about some
>> quantity (again the theorem is about concurrency, but we *define* a
>> thread-per-request system to be one where the number of threads is equal to
>> (or greater than) the concurrency) is true whether you think there’s a
>> point to it or not. The (average) number of threads is obviously not
>> infinite, but equal to the throughput times latency (assuming just one
>> thread per request).
>>
>> Since Little’s law has been effectively used to size sever systems for
>> decades, and so obviously there’s also a very practical point to
>> understanding it.
>>
>> Request rate doesn't change the number of threads that can exist at any
>> given time
>>
>>
>> The (average) number of threads in a thread-per-request system rises
>> proportionately with the (average) throughput. We’re not talking about the
>> number of threads that *can* exist, but the number of threads that *do*
>> exist (on average, of course). The number of threads that *can* exist puts
>> a bound on the number of threads that *do* exist, and so on maximum
>> throughput.
>>
>> it only changes the probability of observing any particular number of
>> them in a fixed period of time.
>>
>>
>> It changes their (average) number. You can start any thread-per-request
>> server increase the load and see for yourself (if the server uses a pool,
>> you’ll see an increase not in live threads but in non-idle threads, but
>> it’s the same thing).
>>
>>
>> All this is only criticism of the formal mathematical claims made here
>> and in the JEP. Nothing needs doing, if no one is interested in formal
>> claims being perfect.
>>
>>
>> The claims, however, were written with care for precision, while your
>> reading of them is not only imprecise and at times incorrect, but may lead
>> people to misunderstand how concurrent systems behave.
>>
>>
>>
>> Alex
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20220802/2f8208e5/attachment.htm>