[RndTbl] RAID5 rebuild performance

Kevin McGregor kevin.a.mcgregor at gmail.com
Tue May 24 13:15:07 CDT 2011


Okay, here's (maybe) the last word on my RAID issues at work, if anyone's
still reading these. :)

My RAID5 array is 8x 300 GB 15K RPM U320 drives, 7 active with 1 hot spare
md_d1 : active raid5 sdu1[6] sdv1[7](S) sdt1[5] sds1[4] sdr1[3] sdq1[2]
sdp1[1] sdo1[0]
      1757804544 blocks level 5, 256k chunk, algorithm 2 [7/7] [UUUUUUU]

Just one drive from this array shows:
/dev/sdr:
 Timing cached reads:   1438 MB in  2.00 seconds = 718.54 MB/sec
 Timing buffered disk reads:  384 MB in  3.00 seconds = 127.97 MB/sec

/srv/d1# dd if=/dev/zero of=bigtestfile bs=2M count=16384
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 414.181 s, 83.0 MB/s [writing]

/srv/d1# dd of=/dev/null if=bigtestfile bs=2M
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 126.176 s, 272 MB/s [reading]


My RAID10 array is 14x 300 GB 10K RPM U320 drives, 12 active with two hot
spares
md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9]
sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]

Just one drive from this array shows:
/dev/sdl:
 Timing cached reads:   1440 MB in  2.00 seconds = 719.94 MB/sec
 Timing buffered disk reads:  254 MB in  3.00 seconds =  84.63 MB/sec

/srv/d0# dd if=/dev/zero of=bigtestfile bs=2M count=16384
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 280.52 s, 122 MB/s [writing]

/srv/d0# dd of=/dev/null if=bigtestfile bs=2M
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 126.134 s, 272 MB/s [reading]

Or one could say that reading is the same speed from either array, and
pretty much at the maximum practical for an Ultra-320 bus; writing is faster
(for my configuration) by 50% on the RAID10 array as compared to the RAID5
array, despite the RAID5 array sporting drives which are ~50% faster (15K
RPM vs. 10K RPM).

Curiously, copying the 'bigtestfile' from d0 to d1 or d1 to d0 results in
~80 MB/s either way. I can't think of an explanation off the top of my head.

There is a lot more testing which could be done, and I'm not saying one
configuration is better than the other. I think I may leave it as is for
now.

Kevin


On Mon, May 23, 2011 at 5:33 PM, Kevin McGregor
<kevin.a.mcgregor at gmail.com>wrote:

> For yet another comparison, my RAID10 6x 750 GB SATA XFS gives:
> # dd if=/dev/zero of=bigfile bs=1M count=16384
> 16384+0 records in
> 16384+0 records out
> 17179869184 bytes (17 GB) copied, 79.5334 s, 216 MB/s
> # dd of=/dev/null if=bigfile bs=1M
> 16384+0 records in
> 16384+0 records out
> 17179869184 bytes (17 GB) copied, 31.4869 s, 546 MB/s
>
> Maybe I should switch to RAID6. ;-)
>
> Kevin
>
> On Fri, May 20, 2011 at 4:54 PM, Trevor Cordes <trevor at tecnopolis.ca>wrote:
>
>> My RAID6 8x 2TB-drive SATA XFS gives:
>>
>> #dd if=/dev/zero of=/new/test bs=1M count=32768
>> 32768+0 records in
>> 32768+0 records out
>> 34359738368 bytes (34 GB) copied, 152.032 s, 226 MB/s
>>
>> #dd of=/dev/null if=/new/test bs=1M
>> 32768+0 records in
>> 32768+0 records out
>> 34359738368 bytes (34 GB) copied, 57.7027 s, 595 MB/s
>> (wow!!)
>>
>> During the whole write time both CPU cores were 90-100%, mostly 95-99%!
>> Glad to see RAID/XFS code is multi-core aware.  For reading it was 1
>> core at 100% and the other around 20%.  The write limiting factor
>> appears to be my piddly Pentium D on my file server. Still, this is
>> 3-4X the speed my old (1TB drives, crappy PCI SATA cards) array was
>> giving me.  The read is quite interesting in that the 100% CPU
>> indicates it is probably doing parity checks on every read.
>>
>> I think a big part of the good speed is my new 8-port SATA card, an
>> Intel PCI-Express x 8 in a x8 slot.  If your SCSI card is just PCI,
>> then the PCI MB/s speed limit is what's killing you.  Even PCI-X may be
>> limiting.  And the Intel card was pretty cheap, under $200.
>>
>> BTW, I got stuck with two spare SATA card expander cables (1 card
>> port to 4 SATA drives) if anyone wants some cheap.  I can get in the
>> Intel cards too, if anyone wants a complete package.
>> _______________________________________________
>> Roundtable mailing list
>> Roundtable at muug.mb.ca
>> http://www.muug.mb.ca/mailman/listinfo/roundtable
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.muug.mb.ca/pipermail/roundtable/attachments/20110524/c87d5811/attachment.html 


More information about the Roundtable mailing list