[RndTbl] RAID5 rebuild performance

Kevin McGregor kevin.a.mcgregor at gmail.com
Fri May 20 12:01:41 CDT 2011


So far I made this
md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9]
sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
and got this
# dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s
# dd of=/dev/null if=/srv/d0/bigzerofile bs=1M
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s

I'm wondering if 12 drives would over-saturate one Ultra-320 channel.
Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of 320
MB/s? I could try setting up a stripe set/RAID0 of varying numbers of drives
and compare. What do you think?

I don't think the external enclosure (HP MSA30?) allows for splitting the
drives into two groups. Only one cable can be connected to it, although
there may have been an option for a second at purchase.

On Thu, May 19, 2011 at 9:27 PM, Adam Thompson <athompso at athompso.net>wrote:

> Well, I did say there was more than one variable involved!  25MB/sec is a
> bit slow, but I don’t know how efficiently that LSI chip is at managing bus
> contention, or how efficient the kernel driver for that chip is.  I do know
> from experience that 8-disk arrays are well into the land of diminishing
> returns from a speed perspective: RAID-1 on two disks or RAID-10 on four
> seems to be the sweet spot for speed.
>
>
>
> Once your rebuild has finished, I would recommend doing some throughput
> tests (both reading and writing) on the array; something perhaps like:
>
> # sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M
> count=1024;sync’
>
> followed by
>
> # time dd if=/dev/md0 of=/dev/null bs=1M count=1024
>
> Those are both very naïve approaches, but should give you a feel for the
> maximum read and write speeds of your array.  I strongly suspect those
> numbers will be much higher than the raid re-sync rate, again, mostly due to
> write-flush barriers in the md code.
>
>
>
> I’m interested in knowing how this ends, personally… please let us know.
>
>
>
> -Adam
>
>
>
> (P.S. does anyone know how to avoid top-posting in Outlook 2010?)
>
>
>
>
>
> *From:* roundtable-bounces at muug.mb.ca [mailto:
> roundtable-bounces at muug.mb.ca] *On Behalf Of *Kevin McGregor
> *Sent:* Thursday, May 19, 2011 14:52
> *To:* Continuation of Round Table discussion
> *Subject:* Re: [RndTbl] RAID5 rebuild performance
>
>
>
> raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable?
> :-) Oh well, I guess I can wait until tomorrow for the rebuild to finish.
>
> On Thu, May 19, 2011 at 2:12 PM, Adam Thompson <athompso at athompso.net>
> wrote:
>
> Look back in dmesg output for the RAID module speed tests, notice which one
> was selected, and divide that number by 2.  There's your theoretical
> bottleneck on the CPU.
> Take the minimum sustained disk/channel/controller throughput, factor in
> interrupt latency, device driver efficiency, etc. and make a rough guess as
> to the overall throughput.
> Consider that md code seems to have a lot of write barriers for safety -
>  so even a rebuild will aspens much of its time waiting for the disk to
> sync().
> All in all, I think your numbers are probably reasonable.
> -Adam
>
>
>
> Kevin McGregor <kevin.a.mcgregor at gmail.com> wrote:
>
> >I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3 (4 x
> >dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the on-board
> >SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software
> RAID
> >5 set up as a 7-drive array with 1 hot-spare drive. All drives are the
> exact
> >same model with the same firmware version.
> >
> >It's currently rebuilding the array (because I just created the array) and
> >/proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that
> >sound "right" in the sense that it's the right order of magnitude? I
> though
> >it should be higher, but I haven't set up such an array before, so I don't
> >have anything to compare it to.
> >
> >If it's slow, does anyone have a suggestion for speeding it up?
> >
> >Kevin
> >
> >_______________________________________________
> >Roundtable mailing list
> >Roundtable at muug.mb.ca
> >http://www.muug.mb.ca/mailman/listinfo/roundtable
>
> _______________________________________________
> Roundtable mailing list
> Roundtable at muug.mb.ca
> http://www.muug.mb.ca/mailman/listinfo/roundtable
>
>
>
> _______________________________________________
> Roundtable mailing list
> Roundtable at muug.mb.ca
> http://www.muug.mb.ca/mailman/listinfo/roundtable
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.muug.mb.ca/pipermail/roundtable/attachments/20110520/c339cc35/attachment.html 


More information about the Roundtable mailing list