AIO and some advice needed.
Avinash Lakshman
avinash.lakshman at gmail.com
Thu Nov 27 20:15:34 PST 2008
Thanks Alan. Do you have any performance benchmark numbers that you can
share comparing the performance of synchronous and asynchronous approaches
on Linux?
Thanks
Avinash
On Thu, Nov 27, 2008 at 10:32 AM, Alan Bateman <Alan.Bateman at sun.com> wrote:
> Avinash Lakshman wrote:
>
>> Here is the test I have been trying to run in order to understand why AIO
>> is claimed to be better.
>>
>> I have a file about 44 GB in size on disk. My Linux machine has 8 cores
>> and 16 GB RAM. I first ran my program where I seek to random offsets within
>> the file and perform a synchronous buffered read of 64K chunks. I collected
>> some statistics from iostat which is shown below:
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 0.13 0.00 0.38 12.37 87.12
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
>> 20.00 0.00 0.17 0.17 0.10
>> sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24 0.00
>> 116.37 1.02 9.27 9.18 101.12
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 0.00 0.00 0.00 12.50 87.50
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97
>> 19.20 0.00 0.00 0.00 0.00
>> sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82 12.12
>> 111.38 1.01 9.78 9.61 100.00
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 0.00 0.00 0.25 12.41 87.34
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00 56.57
>> 22.40 0.01 1.40 0.80 0.40
>> sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80 0.00
>> 116.96 1.03 9.41 9.25 100.91
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 0.13 0.00 0.25 12.47 87.15
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00 44.44
>> 29.33 0.00 0.33 0.33 0.10
>> sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94 0.00
>> 123.88 1.01 10.04 10.05 100.51
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 0.00 0.00 0.25 12.39 87.36
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
>> 20.00 0.00 0.17 0.17 0.10
>> sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84 0.00
>> 120.16 1.02 9.83 9.79 100.92
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> Then I do the same thing using AIO where I just keep submitting requests
>> and then monitor the activity using iostat and the numbers are as follows:
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.21 0.00 0.76 39.06 52.97
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00 32.65
>> 10.67 0.00 0.17 0.17 0.10
>> sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55 0.00
>> 119.15 6.68 53.69 7.99 100.31
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.20 0.00 0.51 39.90 52.40
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97
>> 19.20 0.00 0.20 0.20 0.20
>> sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70 0.00
>> 116.37 6.64 46.76 6.96 97.68
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.22 0.00 0.51 38.35 53.92
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00 48.48
>> 19.20 0.00 0.80 0.80 0.40
>> sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81 0.00
>> 115.11 6.58 46.24 7.01 98.48
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.19 0.00 0.50 36.82 55.49
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00 53.06
>> 34.67 0.00 0.00 0.00 0.00
>> sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18 0.00
>> 120.70 6.68 52.11 7.87 101.22
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.09 0.00 0.51 35.06 57.34
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
>> 20.00 0.00 0.33 0.33 0.20
>> sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96 0.00
>> 123.69 6.44 44.59 6.63 95.41
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %sys %iowait %idle
>> 7.20 0.00 0.63 41.16 51.01
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await svctm %util
>> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09 12.12
>> 120.38 6.63 51.24 7.69 98.59
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00
>>
>> Now is the million dollar question - which is better and why? Could
>> someone please help me understand these numbers? I am willing to get more
>> information out if need be.
>>
> The iostat(1) page has descriptions for each of the columns. From a brief
> scan it looks like the synchronous case is doing 100-110 read requests/sec
> and reading 5.7-6.4mB/sec (do I have this right?). In the asynchronous case
> it looks like you are sending 125-140 read requests/sec and reading
> 7.4-8.8mB/sec. CPU usage can be important and the second test is clearly
> putting more load on the system.
>
> -Alan.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081127/8709f500/attachment.html
More information about the nio-discuss
mailing list