[DRBD-user] FW: Slow sync speed over gigabit

[James Miller]

> -----Original Message-----
> From: drbd-user-bounces at linbit.com 
> [mailto:drbd-user-bounces at linbit.com] On Behalf Of Lars Ellenberg
> Sent: Tuesday, August 14, 2007 2:05 PM
> To: drbd-user at linbit.com
> Subject: Re: [DRBD-user] FW: Slow sync speed over gigabit
> 
> On Tue, Aug 14, 2007 at 05:02:54PM +0200, Lavender, Ben wrote:
> > The sysctl settings here gave a small boost, but they still aren't 
> > even approaching what gigabit is capable of.
> > 
> > iperf does indeed give me 100% of the link, 990Mb/sec (the tcp 
> > settings given here boosted it from about 940).
> > 
> > If I cat /proc/drbd quite a bit, I occasionally get interesting 
> > results like the following, perhaps 1 out of 20 times.  Note the 
> > drastically reduced speed reading:
> > 
> > [root at mail-2 ~]# cat /proc/drbd
> > version: 8.0.5 (api:86/proto:86)
> > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com,
> > 2007-08-13 15:12:01
> >  0: cs:SyncTarget st:Secondary/Primary 
> ds:Inconsistent/UpToDate C r---
> >     ns:0 nr:3844992 dw:3836800 dr:0 al:0 bm:231 lo:258 
> pe:6550 ua:256 
> > ap:0
> >         [=>..................] sync'ed:  5.5% (65498/69245)M
> >         finish: 15:31:32 speed: 928 (17,600) K/sec
> >         resync: used:14/31 hits:246358 misses:248 starving:0 dirty:0
> > changed:248
> >         act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 
> > changed:0
> > 
> > 
> > But usually, it's more like this:
> > 
> > [root at mail-2 ~]# cat /proc/drbd
> > version: 8.0.5 (api:86/proto:86)
> > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com,
> > 2007-08-13 15:12:01
> >  0: cs:SyncTarget st:Secondary/Primary 
> ds:Inconsistent/UpToDate C r---
> >     ns:0 nr:5081600 dw:5073408 dr:0 al:0 bm:308 lo:258 
> pe:4891 ua:256 
> > ap:0
> >         [=>..................] sync'ed:  7.2% (64291/69245)M
> >         finish: 1:14:08 speed: 14,740 (15,752) K/sec
> >         resync: used:11/31 hits:321915 misses:320 
> starving:0 dirty:0 
> > changed:320
> >         act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 
> > changed:0
> > 
> > 
> > I'm not accusing drbd of limiting me, but everything else works as 
> > expected and my ethernet interfaces don't have any errors.  Any 
> > thoughts?
> 
> first, tune your storage subsystem:
> play with the nr_requests and max_sectors_kb in 
> /sys/block/*/queue/ on your storage. use "cfq" io-scheduler. 
> also try with "deadline".
> cfq feels more responsive while delivering about the same throughput.
> I don't recommend "anticipatory" for servers.
> 
> make sure that both storages can sustain a write rate you expect.
> measure _including fsync_ (or using direct io).
> compare with what you expect they should be capable of.
> 
> tune your drbd sync rate setting (unit is bytes, not bits!) 
> to be just slightly over the physical capabilities of storage 
> and network.  if you expect a rate of max 100M (typical for 
> 1GBit ethernet), configure a resync rate of 110M or so;  
> "650M" will be contraproductive.
> (actually, in production you should rather throttle the 
> resync,  to limit the impact on the active services during resync).
> 
> up sndbuf-size. benchmark.
> up max-buffers. benchmark.
> up max-epoch size. benchmark.
> (I've also seen that tuning things down did help;  when 
> hardware meets software, strange things will happen).
> 
> maybe up or down unplug watermark.
> 
> 

Lars that's some GREAT information.  Could you point to where it's
documented for us who are new and sturggling with 8.0.x DRBD.


--Jim