[DRBD-user] slow sync speed of 12TB over 10GE

Lars Ellenberg lars.ellenberg at linbit.com
Mon Jan 12 16:21:31 CET 2009


On Mon, Jan 12, 2009 at 03:58:17PM +0100, Lars Täuber wrote:
> Hallo you!
> 
> Igor Neves <igor at 3gnt.net> schrieb:
> > Hello,
> > 
> > Yes this is true. The metadata can be the problem, i would just use a
> > simple raid setup, everything over hardware raid.
> 
> The metadata doesn't seem to be the problem. I hope this config puts the meta data onto a ramdisk:
> # drbdadm dump
> # /etc/drbd.conf
> common {
>     protocol               C;
>     syncer {
>         rate             1150M;
>     }
>     handlers {
>         split-brain      "/usr/lib/drbd/notify-split-brain.sh root";
>     }
> }
> 
> # resource RAID61 on monosan: not ignored, not stacked
> resource RAID61 {
>     on monosan {
>         device           /dev/drbd0;
>         disk             /dev/md4;
>         address          ipv4 10.9.8.7:7788;
>         flexible-meta-disk /dev/ram0;
>     }
>     on duosan {
>         device           /dev/drbd0;
>         disk             /dev/md4;
>         address          ipv4 10.9.8.6:7788;
>         flexible-meta-disk /dev/ram0;

are you _sure_ that drbd uses that?
(drbdsetup /dev/drbd0 show)
you are aware that you'll get a full sync again?

>     }
>     disk {

try add (introduced more recently:)
	no-disk-barrier;
to see if it changes something.
if it does: use at your own risk.

>         no-disk-flushes;
>         no-md-flushes;
>     }
> }
> 
> But the speed hasn't increased so far:
> # cat /proc/drbd
> 
> version: 8.3.0 (api:88/proto:86-89)
> GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root at monosan, 2009-01-09 15:15:29
>  0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:22204216 dw:22201608 dr:0 al:0 bm:1352 lo:654 pe:9889 ua:652 ap:0 ep:1 wo:b oos:6530218744  
>         [>....................] sync'ed:  0.4% (6377166/6398848)M
>         finish: 59:40:09 speed: 30,364 (88,804) K/sec
> 
> 
> What are the other possible bottlenecks?

md resyncing at the same time?  ;)
misaligned device offset?
secondary device performance?
device latencies?

max-buffers (and max-epoch);
al-size;

your sync-rate is set _much_ too high.  unit is _bytes_ not bits, we are
storage guys.  and it is meant to be a _throttle_. increasing it way
beyond physical limits does hurt, not help.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed



More information about the drbd-user mailing list