Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I appreciate all your help on this. I'll spare you the details, but after running a pile of benchmarks to test the settings you suggested, the bottleneck appears to be the write speed of logical volume management (the bare disks write much faster, and drbd takes advantage of it). So it seems I have to decide if I really want LVM. One quick note, however: while in the pause-sync mode, if I mess with the mounted partition, drbd is still throwing out a lot of traffic on the network. This is pretty counter-intuitive to me; I would expect it not to be talking with the remote disk at all during pause sync. Thanks, Ben > -----Original Message----- > From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user- > bounces at lists.linbit.com] On Behalf Of Lars Ellenberg > Sent: Tuesday, August 14, 2007 9:05 PM > To: drbd-user at lists.linbit.com > Subject: Re: [DRBD-user] FW: Slow sync speed over gigabit > > On Tue, Aug 14, 2007 at 05:02:54PM +0200, Lavender, Ben wrote: > > The sysctl settings here gave a small boost, but they still aren't even > > approaching what gigabit is capable of. > > > > iperf does indeed give me 100% of the link, 990Mb/sec (the tcp settings > > given here boosted it from about 940). > > > > If I cat /proc/drbd quite a bit, I occasionally get interesting results > > like the following, perhaps 1 out of 20 times. Note the drastically > > reduced speed reading: > > > > [root at mail-2 ~]# cat /proc/drbd > > version: 8.0.5 (api:86/proto:86) > > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com, > > 2007-08-13 15:12:01 > > 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > > ns:0 nr:3844992 dw:3836800 dr:0 al:0 bm:231 lo:258 pe:6550 ua:256 > > ap:0 > > [=>..................] sync'ed: 5.5% (65498/69245)M > > finish: 15:31:32 speed: 928 (17,600) K/sec > > resync: used:14/31 hits:246358 misses:248 starving:0 dirty:0 > > changed:248 > > act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 changed:0 > > > > > > But usually, it's more like this: > > > > [root at mail-2 ~]# cat /proc/drbd > > version: 8.0.5 (api:86/proto:86) > > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com, > > 2007-08-13 15:12:01 > > 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > > ns:0 nr:5081600 dw:5073408 dr:0 al:0 bm:308 lo:258 pe:4891 ua:256 > > ap:0 > > [=>..................] sync'ed: 7.2% (64291/69245)M > > finish: 1:14:08 speed: 14,740 (15,752) K/sec > > resync: used:11/31 hits:321915 misses:320 starving:0 dirty:0 > > changed:320 > > act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 changed:0 > > > > > > I'm not accusing drbd of limiting me, but everything else works as > > expected and my ethernet interfaces don't have any errors. Any > > thoughts? > > first, tune your storage subsystem: > play with the nr_requests and max_sectors_kb in /sys/block/*/queue/ > on your storage. use "cfq" io-scheduler. also try with "deadline". > cfq feels more responsive while delivering about the same throughput. > I don't recommend "anticipatory" for servers. > > make sure that both storages can sustain a write rate you expect. > measure _including fsync_ (or using direct io). > compare with what you expect they should be capable of. > > tune your drbd sync rate setting (unit is bytes, not bits!) to be just > slightly over the physical capabilities of storage and network. if you > expect a rate of max 100M (typical for 1GBit ethernet), configure a > resync rate of 110M or so; "650M" will be contraproductive. > (actually, in production you should rather throttle the resync, > to limit the impact on the active services during resync). > > up sndbuf-size. benchmark. > up max-buffers. benchmark. > up max-epoch size. benchmark. > (I've also seen that tuning things down did help; > when hardware meets software, strange things will happen). > > maybe up or down unplug watermark. > > -- > : Lars Ellenberg Tel +43-1-8178292-0 : > : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : > : Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com : > __ > please use the "List-Reply" function of your email client. > _______________________________________________ > drbd-user mailing list > drbd-user at lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user