[DRBD-user] slow sync speed of 12TB over 10GE

Igor Neves igor at 3gnt.net
Mon Jan 12 13:57:13 CET 2009

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Can you please tell me, why are you doing software raid?

Of all my experience drbd goes really bad with software raid.
I have done tests with Lsi, 3ware, Areca and software raid with and
without lvm, with ext3 and with xfs and drbd on top of it.

I have clients with all the cards i mention, and none with software
raid. When you get high load (drbd on top of it) and software raid
working over it, it will get very low transfers and so much I/O wait...
try to put virtual machines on top of that and you will be surprised.

This is my opinion, if you have a big storage backend (your case) and
depending on what you will want to get from the solution, use hardware
raid with areca cards. Forget LSI, forget 3ware, they are garbage. Try
to use pcie cards, they will perform a lot better then pci-x and with
less problems.

Maybe you have some problem there, and lsi's are not very fast, but so
you can get the idea, i can get ~170MB/sec over drbd with one low end
areca card with 4 stupid sata disks (7.5k rpm) in raid5. With your
system, it should fly with one areca card. By the way i have a system
with your LSI card with 6 SAS disks, and the best i can get of it was
about ~150MB/sec.

This examples it's just so you can get one idea of the other side of the
world! :)

Good luck for your problem, i dont know if i can help you more.
Keep in mind, avoid software raid with drbd on top, they dont work well
together for perfomance.

Lars Täuber wrote:
> Hi Igor,
> Igor Neves <igor at 3gnt.net> schrieb:
>> Hi,
>> Are you using hardware raid under drbd device or software raid?
>> Cheers
> it is software raid6 over 15 disks with one spare. The disks are in an SAS cage connected through
> LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS controller.
> # hdparm -t /dev/md4
> /dev/md4:
>  Timing buffered disk reads:  1102 MB in  3.01 seconds = 366.57 MB/sec
>  Timing buffered disk reads:  1040 MB in  3.00 seconds = 346.63 MB/sec
>  Timing buffered disk reads:  1042 MB in  3.01 seconds = 346.62 MB/sec
>  Timing buffered disk reads:  1070 MB in  3.00 seconds = 356.16 MB/sec
>  Timing buffered disk reads:  1078 MB in  3.00 seconds = 358.83 MB/sec
>  Timing buffered disk reads:  1120 MB in  3.00 seconds = 373.25 MB/sec
> # cat /proc/mdstat 
> Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] 
> md4 : active raid6 sdc[0] sdr[15](S) sdq[14] sdp[13] sdo[12] sdn[11] sdm[10] sdl[9] sdk[8] sdj[7] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdd[1]
>       12697912448 blocks level 6, 64k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
> Regards
> Lars
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20090112/dba157a7/attachment.htm>

More information about the drbd-user mailing list