AIO and some advice needed.
Alan Bateman
Alan.Bateman at Sun.COM
Thu Nov 27 10:32:50 PST 2008
Avinash Lakshman wrote:
> Here is the test I have been trying to run in order to understand why
> AIO is claimed to be better.
>
> I have a file about 44 GB in size on disk. My Linux machine has 8
> cores and 16 GB RAM. I first ran my program where I seek to random
> offsets within the file and perform a synchronous buffered read of 64K
> chunks. I collected some statistics from iostat which is shown below:
>
> avg-cpu: %user %nice %sys %iowait %idle
> 0.13 0.00 0.38 12.37 87.12
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00
> 61.22 20.00 0.00 0.17 0.17 0.10
> sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24
> 0.00 116.37 1.02 9.27 9.18 101.12
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 0.00 0.00 0.00 12.50 87.50
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00
> 96.97 19.20 0.00 0.00 0.00 0.00
> sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82
> 12.12 111.38 1.01 9.78 9.61 100.00
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 0.00 0.00 0.25 12.41 87.34
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00
> 56.57 22.40 0.01 1.40 0.80 0.40
> sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80
> 0.00 116.96 1.03 9.41 9.25 100.91
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 0.13 0.00 0.25 12.47 87.15
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00
> 44.44 29.33 0.00 0.33 0.33 0.10
> sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94
> 0.00 123.88 1.01 10.04 10.05 100.51
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 0.00 0.00 0.25 12.39 87.36
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00
> 61.22 20.00 0.00 0.17 0.17 0.10
> sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84
> 0.00 120.16 1.02 9.83 9.79 100.92
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> Then I do the same thing using AIO where I just keep submitting
> requests and then monitor the activity using iostat and the numbers
> are as follows:
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.21 0.00 0.76 39.06 52.97
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00
> 32.65 10.67 0.00 0.17 0.17 0.10
> sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55
> 0.00 119.15 6.68 53.69 7.99 100.31
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.20 0.00 0.51 39.90 52.40
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00
> 96.97 19.20 0.00 0.20 0.20 0.20
> sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70
> 0.00 116.37 6.64 46.76 6.96 97.68
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.22 0.00 0.51 38.35 53.92
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00
> 48.48 19.20 0.00 0.80 0.80 0.40
> sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81
> 0.00 115.11 6.58 46.24 7.01 98.48
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.19 0.00 0.50 36.82 55.49
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00
> 53.06 34.67 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18
> 0.00 120.70 6.68 52.11 7.87 101.22
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.09 0.00 0.51 35.06 57.34
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00
> 61.22 20.00 0.00 0.33 0.33 0.20
> sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96
> 0.00 123.69 6.44 44.59 6.63 95.41
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.20 0.00 0.63 41.16 51.01
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09
> 12.12 120.38 6.63 51.24 7.69 98.59
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> Now is the million dollar question - which is better and why? Could
> someone please help me understand these numbers? I am willing to get
> more information out if need be.
The iostat(1) page has descriptions for each of the columns. From a
brief scan it looks like the synchronous case is doing 100-110 read
requests/sec and reading 5.7-6.4mB/sec (do I have this right?). In the
asynchronous case it looks like you are sending 125-140 read
requests/sec and reading 7.4-8.8mB/sec. CPU usage can be important and
the second test is clearly putting more load on the system.
-Alan.
More information about the nio-discuss
mailing list