AIO and some advice needed.
Avinash Lakshman
avinash.lakshman at gmail.com
Thu Nov 27 07:45:08 PST 2008
Here is the test I have been trying to run in order to understand why AIO is
claimed to be better.
I have a file about 44 GB in size on disk. My Linux machine has 8 cores and
16 GB RAM. I first ran my program where I seek to random offsets within the
file and perform a synchronous buffered read of 64K chunks. I collected some
statistics from iostat which is shown below:
avg-cpu: %user %nice %sys %iowait %idle
0.13 0.00 0.38 12.37 87.12
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
20.00 0.00 0.17 0.17 0.10
sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24 0.00
116.37 1.02 9.27 9.18 101.12
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.00 12.50 87.50
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97
19.20 0.00 0.00 0.00 0.00
sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82 12.12
111.38 1.01 9.78 9.61 100.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.25 12.41 87.34
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00 56.57
22.40 0.01 1.40 0.80 0.40
sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80 0.00
116.96 1.03 9.41 9.25 100.91
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
0.13 0.00 0.25 12.47 87.15
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00 44.44
29.33 0.00 0.33 0.33 0.10
sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94 0.00
123.88 1.01 10.04 10.05 100.51
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.25 12.39 87.36
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
20.00 0.00 0.17 0.17 0.10
sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84 0.00
120.16 1.02 9.83 9.79 100.92
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
Then I do the same thing using AIO where I just keep submitting requests and
then monitor the activity using iostat and the numbers are as follows:
avg-cpu: %user %nice %sys %iowait %idle
7.21 0.00 0.76 39.06 52.97
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00 32.65
10.67 0.00 0.17 0.17 0.10
sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55 0.00
119.15 6.68 53.69 7.99 100.31
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
7.20 0.00 0.51 39.90 52.40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97
19.20 0.00 0.20 0.20 0.20
sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70 0.00
116.37 6.64 46.76 6.96 97.68
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
7.22 0.00 0.51 38.35 53.92
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00 48.48
19.20 0.00 0.80 0.80 0.40
sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81 0.00
115.11 6.58 46.24 7.01 98.48
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
7.19 0.00 0.50 36.82 55.49
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00 53.06
34.67 0.00 0.00 0.00 0.00
sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18 0.00
120.70 6.68 52.11 7.87 101.22
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
7.09 0.00 0.51 35.06 57.34
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22
20.00 0.00 0.33 0.33 0.20
sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96 0.00
123.69 6.44 44.59 6.63 95.41
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %sys %iowait %idle
7.20 0.00 0.63 41.16 51.01
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09 12.12
120.38 6.63 51.24 7.69 98.59
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
Now is the million dollar question - which is better and why? Could someone
please help me understand these numbers? I am willing to get more
information out if need be.
Thanks
Avinash
On Thu, Nov 27, 2008 at 2:32 AM, Alan Bateman <Alan.Bateman at sun.com> wrote:
> Avinash Lakshman wrote:
>
>> iostat on Linux doesn't seem to show me correct statistics when I use AIO.
>> I know I am performing reads correctly but the r/s is always at zero. What
>> am I doing wrong? I run iostat -x 1.
>>
> Hard to say. Are you saying that iostat doesn't work or just that the read
> stats don't change? If the latter then maybe the file is already cached. I
> see in proc(5) that you can drop the cache after sync'ing (I've never used
> it but maybe useful for this test).
>
> -Alan.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081127/57197267/attachment.html
More information about the nio-discuss
mailing list