Thread pool configuration

[Dave Cheney]

Hi Avinash,

Without AIO support from the kernel it'll always be a crapshot. Some  
rough rules of thumb (pulling them from somewhere other than my thumb)  
would be no less than a page up to slightly less than your OS's read  
ahead buffering will grab in a single request.

But this is highly dependent on load, the number of readers, the  
availability of the data in the OS's cache, the IO load on the lun and  
the number of outstanding IO requests.

Again without any benchmarks to back this up I would suggest a small  
buffer, 1 page, would give the best throughput. That is, least chance  
of a worker blocking on disk IO. What you trade off with this approach  
is system time and context switches. You probably start approaching  
some bottlenecks in your filesystem or OS at the extreme of this  
approach.

You could go for an IO worker pool (as InnoDB does) where worker  
threads sleep waiting on a fixed number of IO threads that actually  
service requests. Some process, either manual or auto magic could  
adjust the size of the IO worker pool and/or the IO buffer size based  
on the average service times of the lun you are pulling from.

Just some ideas.

Cheers

Dave

On 21/11/2008, at 4:23 PM, Avinash Lakshman wrote:

> Hi All
>
> Does anyone any pointers to any literature which talks about what  
> should be the ideal amount of data to read from disk for maximizing  
> I/O throughput? Another naive way of posing this question is what  
> should be the ideal buffer size for reads for maximizing I/O  
> throughput?
>
> A