Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Sun, May 31, 2009 at 11:39:25AM +0800, Kelvin Leong wrote: > Hi guys, > > a) 4TB Soft Limit when using External Meta Disk > > I've noticed that there's a 4TB soft limit in place when using an External > Meta Disk. > (i.e, "meta-disk /dev/sdx1[0]" inside your drbd.conf file in the "net" > section) > > Pretty sure this isn't anything to do with weird math or stuff, because it's > limited at *exactly* 4,000,000MB if you use a disk larger than 4TB - (I > tried with a 6TB array and a 5TB array to) rather than our usual > power-of-two-number. if that is the case, then that is an oversight (read bug) and should be fixed. > After pouring through loads of forum posts, I sincerely doubt this has > anything to do with the size of my external meta disk - for 4TB you only > need somewhere around ~180MB - and I've tried with 2GB and 8GB meta disks. there are two flavours of external meta data: the indexed one, where you say meta-disk /dev/sdx1[0], and [0] is the index number into an "on disk array of meta data store with 128 MiB per slot". drbd meta data: 8 sectors "super block" + 64 sectors "activity log", + rest bitmap. bitmap of 4 TiB: almost 128 MiB. and the "flexible-external" one, where you say "flexible-meta-disk /dev/sdx1" where there is no index, and the whole disk as specified can be used for meta data. > You should be able to replicate this as of "version: 8.3.1 > (api:88/proto:86-89)". actually I cannot. do you get a "truncating very big lower level device" log line? > To draw a contrast, you can try those very same arrays and use the default > internal meta-disk (i.e, "meta-disk internal" inside your drbd.conf file). > > --- > > b) Increased Performance with External Meta Disks > > Apart from that, after running loads of dd-write's, I've personally found > that using an External Meta Disk greatly boosts performance, especially in > sync'ing - my results were on the average: > > 60,544 (62,132) K/sec - External meta disk > 46,072 (46,204) K/sec - Internal meta disk > > You can see a clear 15 to 25MB/s increase, even with initial sync. > > Thought I'd just share this. did you try different disk schedulers? cfq (before kernel 2.6.30, in any case) is very bad for internal meta data, and many other usage patterns as well. grep . /sys/block/*/queue/scheduler for d in /sys/block/*/queue/scheduler; do echo deadline > $d done -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed