[DRBD-user] FW: Slow sync speed over gigabit

Lavender, Ben ben.lavender at mcdean-europe.com
Wed Aug 15 11:16:08 CEST 2007


I appreciate all your help on this.  I'll spare you the details, but
after running a pile of benchmarks to test the settings you suggested,
the bottleneck appears to be the write speed of logical volume
management (the bare disks write much faster, and drbd takes advantage
of it).  So it seems I have to decide if I really want LVM.

One quick note, however: while in the pause-sync mode, if I mess with
the mounted partition, drbd is still throwing out a lot of traffic on
the network.  This is pretty counter-intuitive to me; I would expect it
not to be talking with the remote disk at all during pause sync.

Thanks,
Ben

> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-
> bounces at lists.linbit.com] On Behalf Of Lars Ellenberg
> Sent: Tuesday, August 14, 2007 9:05 PM
> To: drbd-user at lists.linbit.com
> Subject: Re: [DRBD-user] FW: Slow sync speed over gigabit
> 
> On Tue, Aug 14, 2007 at 05:02:54PM +0200, Lavender, Ben wrote:
> > The sysctl settings here gave a small boost, but they still aren't
even
> > approaching what gigabit is capable of.
> >
> > iperf does indeed give me 100% of the link, 990Mb/sec (the tcp
settings
> > given here boosted it from about 940).
> >
> > If I cat /proc/drbd quite a bit, I occasionally get interesting
results
> > like the following, perhaps 1 out of 20 times.  Note the drastically
> > reduced speed reading:
> >
> > [root at mail-2 ~]# cat /proc/drbd
> > version: 8.0.5 (api:86/proto:86)
> > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com,
> > 2007-08-13 15:12:01
> >  0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C
r---
> >     ns:0 nr:3844992 dw:3836800 dr:0 al:0 bm:231 lo:258 pe:6550
ua:256
> > ap:0
> >         [=>..................] sync'ed:  5.5% (65498/69245)M
> >         finish: 15:31:32 speed: 928 (17,600) K/sec
> >         resync: used:14/31 hits:246358 misses:248 starving:0 dirty:0
> > changed:248
> >         act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0
changed:0
> >
> >
> > But usually, it's more like this:
> >
> > [root at mail-2 ~]# cat /proc/drbd
> > version: 8.0.5 (api:86/proto:86)
> > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com,
> > 2007-08-13 15:12:01
> >  0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C
r---
> >     ns:0 nr:5081600 dw:5073408 dr:0 al:0 bm:308 lo:258 pe:4891
ua:256
> > ap:0
> >         [=>..................] sync'ed:  7.2% (64291/69245)M
> >         finish: 1:14:08 speed: 14,740 (15,752) K/sec
> >         resync: used:11/31 hits:321915 misses:320 starving:0 dirty:0
> > changed:320
> >         act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0
changed:0
> >
> >
> > I'm not accusing drbd of limiting me, but everything else works as
> > expected and my ethernet interfaces don't have any errors.  Any
> > thoughts?
> 
> first, tune your storage subsystem:
> play with the nr_requests and max_sectors_kb in /sys/block/*/queue/
> on your storage. use "cfq" io-scheduler. also try with "deadline".
> cfq feels more responsive while delivering about the same throughput.
> I don't recommend "anticipatory" for servers.
> 
> make sure that both storages can sustain a write rate you expect.
> measure _including fsync_ (or using direct io).
> compare with what you expect they should be capable of.
> 
> tune your drbd sync rate setting (unit is bytes, not bits!) to be just
> slightly over the physical capabilities of storage and network.  if
you
> expect a rate of max 100M (typical for 1GBit ethernet), configure a
> resync rate of 110M or so;  "650M" will be contraproductive.
> (actually, in production you should rather throttle the resync,
>  to limit the impact on the active services during resync).
> 
> up sndbuf-size. benchmark.
> up max-buffers. benchmark.
> up max-epoch size. benchmark.
> (I've also seen that tuning things down did help;
>  when hardware meets software, strange things will happen).
> 
> maybe up or down unplug watermark.
> 
> --
> : Lars Ellenberg                            Tel +43-1-8178292-0  :
> : LINBIT Information Technologies GmbH      Fax +43-1-8178292-82 :
> : Vivenotgasse 48, A-1120 Vienna/Europe    http://www.linbit.com :
> __
> please use the "List-Reply" function of your email client.
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user



More information about the drbd-user mailing list