Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Thu, Apr 30, 2009 at 05:58:42PM -0400, Gennadiy Nerubayev wrote: > On Thu, Apr 30, 2009 at 5:18 PM, Tom Brown <wc-linbit.com@ > vmail.baremetal.com> wrote: > > > > > I'd expect that if he's seeing sync's running faster than application > > level writes, then there is an issue with write-barriers. DRBD is taking > > steps to provide write-ordering protection, which aren't relevent to > > bulk/block copying of re-syncing. > > > > e.g. http://www.drbd.org/users-guide/re-drbdconf.html > > no-disk-barrier , no-disk-flushes , no-disk-drain > > > I've generally established that for best performance on my hardware, I need > no-disk-flushes, no-md-flushes, and no-disk-barrier, and they've all been in > place already. no-disk-drain does not improve anything, and was mentioned by > Lars as a bad idea if the other three are present. > > The connection is capable of at least ~900MB/s (Infiniband), and the top > observed DRBD sync speed is 400MB/s as mentioned. For testing purposes, the > metadata is on a ramdisk. what is your micro benchmark? for sequential write throughput micro benchmark, I suggest dd if=/dev/zero of=/dev/drbdX bs=4M count=1000 oflag=direct do variations in bs= and count= (to reveal possibly issues with cpu cache sizes). also do variations of oflag=direct no page cache/buffer cache involved, oflag=dsync completely through buffer cache/page cache, but does the equivalent of "fsync" for every "bs" no oflag, but conv=fsync completely through buffer cache/page cache, and does a real fsync only once all count * bs blocks are written smalish bs (< the size of your cpu cache), say bs=32k, high count, and oflag=direct is what is most like what the resync is doing. you can also start pinning your "dd" to a single cpu, preferably the same your DRBD kernel threads are running on. or allow only the first two cores, or whatever. my guess is that you hit some sort of cpu cache size/ emory bandwidth/cache coherence/cpu synchronization problem. once you established the throughput of of=/dev/drbdX, you can start experimenting with of=file-in-filesystem, and different file systems, file system tuning options and so on. -- : Lars Ellenberg : LINBIT HA-Solutions GmbH : DRBD®/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed