Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Tue, Aug 14, 2007 at 05:02:54PM +0200, Lavender, Ben wrote: > The sysctl settings here gave a small boost, but they still aren't even > approaching what gigabit is capable of. > > iperf does indeed give me 100% of the link, 990Mb/sec (the tcp settings > given here boosted it from about 940). > > If I cat /proc/drbd quite a bit, I occasionally get interesting results > like the following, perhaps 1 out of 20 times. Note the drastically > reduced speed reading: > > [root at mail-2 ~]# cat /proc/drbd > version: 8.0.5 (api:86/proto:86) > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com, > 2007-08-13 15:12:01 > 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:3844992 dw:3836800 dr:0 al:0 bm:231 lo:258 pe:6550 ua:256 > ap:0 > [=>..................] sync'ed: 5.5% (65498/69245)M > finish: 15:31:32 speed: 928 (17,600) K/sec > resync: used:14/31 hits:246358 misses:248 starving:0 dirty:0 > changed:248 > act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 changed:0 > > > But usually, it's more like this: > > [root at mail-2 ~]# cat /proc/drbd > version: 8.0.5 (api:86/proto:86) > SVN Revision: 3011 build by ben at m-2w9juvlmyfmj7.mcdean-europe.com, > 2007-08-13 15:12:01 > 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:5081600 dw:5073408 dr:0 al:0 bm:308 lo:258 pe:4891 ua:256 > ap:0 > [=>..................] sync'ed: 7.2% (64291/69245)M > finish: 1:14:08 speed: 14,740 (15,752) K/sec > resync: used:11/31 hits:321915 misses:320 starving:0 dirty:0 > changed:320 > act_log: used:0/907 hits:0 misses:0 starving:0 dirty:0 changed:0 > > > I'm not accusing drbd of limiting me, but everything else works as > expected and my ethernet interfaces don't have any errors. Any > thoughts? first, tune your storage subsystem: play with the nr_requests and max_sectors_kb in /sys/block/*/queue/ on your storage. use "cfq" io-scheduler. also try with "deadline". cfq feels more responsive while delivering about the same throughput. I don't recommend "anticipatory" for servers. make sure that both storages can sustain a write rate you expect. measure _including fsync_ (or using direct io). compare with what you expect they should be capable of. tune your drbd sync rate setting (unit is bytes, not bits!) to be just slightly over the physical capabilities of storage and network. if you expect a rate of max 100M (typical for 1GBit ethernet), configure a resync rate of 110M or so; "650M" will be contraproductive. (actually, in production you should rather throttle the resync, to limit the impact on the active services during resync). up sndbuf-size. benchmark. up max-buffers. benchmark. up max-epoch size. benchmark. (I've also seen that tuning things down did help; when hardware meets software, strange things will happen). maybe up or down unplug watermark. -- : Lars Ellenberg Tel +43-1-8178292-0 : : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : : Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com : __ please use the "List-Reply" function of your email client.