[DRBD-user] Why is write performance so low?

[Lars Ellenberg]

On Thu, May 07, 2009 at 02:10:04PM +0000, Holger Kiehl wrote:
> Hello
> 
> While setting up a new system I noticed that write performance is
> very low, only about 35MB/s. This is with 2.6.29.1 and DRBD 8.3.1.
> The disk device is a md device (software raid) and when I run the
> following test:
> 
>     dd if=/dev/zero of=/dev/md4 bs=64k count=100000
> 
> it will give me approx. 160 MB/s. If I do this against /dev/drbd0 the
> maximum I get is 70 MB/s when I disable DRBD on the secondary. The 70 MB/s
> with the secodary enabled I do manage also when I play with sndbuf-size,
> max-buffers, unplug-watermark and al-extents. But why is it limited
> at 70 MB/s when secondary is disabled? The drbd.conf looks as follows:


For large (exceeding the activity log) streaming writes,
the activity log can be considered to always be cold.

[the activity log being our implementation of the "write intent" logic.]

an AL-transaction is a synchonous single sector write.
that usually does not take too long, but may take quite a while if there
are any scheduler timeouts, or read/modify/write cycles involved...

If we assume that we have to do one activity log transaction per
one AL-extent-size worth of data (which is currently hardcoded to 4 MB),
and further assuming that the activity log transaction overhead would
be the sole cause for the slow down, we can calculate:

#!/bin/bash
DRBD_throughputs_to_estimated_transaction_latency()
{
        local ll_MBps=${1?} drbd_MBps=${2?}

        echo "local: $ll_MBps MB/s"
        echo "drbd disconnected: $drbd_MBps MB/s"
        echo "AL-transaction latency appears to be: " \
                $[1000000 * 4 / drbd_MBps - 1000000 * 4/ll_MBps] \
                "micro seconds"
        echo "you may want to verify this using e.g."
        echo "dd if=/dev/zero of=ll_dev bs=512 count=1000 oflag=direct"
}

DRBD_throughputs_to_estimated_transaction_latency 160 70

local: 160 MB/s
drbd disconnected: 70 MB/s
AL-transaction latency appears to be:  32142 micro seconds
you may want to verify this using e.g.
dd if=/dev/zero of=ll_dev bs=512 count=1000 oflag=direct

there.  now, how long does a synchonous single sector write take on you
lower level device?

does the dd throughput increase "dramatically", when reducing the bs *
count size to fit into the activity log, (bs * count < al-extents * 4MB)
and running "AL hot", i.e. on the second dd pass?

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed