Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Many thanks! I put the metadata on mirrored SSD-disks and it made a dramatic change in performance - the drop in performace for write size > 510MB disappeared completely (almost x5 raise in speed). The peak sync rate increased from ~25,000 K/s to ~119,488 K/s (yes, I understand that you dont want that high sync rate on a working cluster, but now I just want to get them in sync as fast as possible). I also found this <http://fghaas.wordpress.com/2009/08/20/internal-metadata-and-why-we-recommend-it/> when googleing for external metadata, so I suppose that connecting a battery backup unit to the controller should result in a similar performance boost - and I have just ordered two such devices to upgrade my controllers and will have the answer in a couple of weeks. Thanks again! Cheers, Tor On 09:45 Wed 07 Nov , Felix Frank wrote: > Hi, > > On 11/06/2012 05:18 PM, Tor wrote: > > the drbd-device is > > on a HW-raid5 with eight drives (total of 6.4 TiB) > > I'm not sure this is a very sound idea, performance-wise. The RAID5 in > and of itself should be fine, but you should really consider external > metadata IMHO. > > > It should be close to the sync speed that I have set to 110 > > MB/Sec (my NIC is 1Gbit/s). > > As an aside - you should limit the sync speed to 30M. You do *not* want > the background sync to saturate your link and slow down live replication. > > > The results puzzles me somewhat. There is a large drop in performance > > when the size exceeds 510 MB (I have ran tests up to 2GB). The speed > > increases when the second node is not connected, but the relative > > performance drop becomes even larger. On the "raw" device there is no > > such drop in performance. So my conclusion is that this has something > > to do with DRBD that I can not understand... > > Another remark: What you're benchmarking is replication speed, not sync > speed. Sync speed is what's applied when you reconnect your nodes and > one of them (typically the one secondary) has to catch up to the other. > > Replication is always done as fast as possible. There is no upper bound > you can configure (that would be silly). > > > Any ideas? > > I think you're missing the metadata interactions. The 500MB barrier > seems suggestive to me, although I have no clear idea at all what the > details might be here. > > I suggest you first try and create a test device with external metadata. > The quicker the metadata disk the better (yes, for testing you might > even go for ramdisk, but don't tell anyone I wrote that). See if this > changes things. > > Cheers, > Felix