Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Thu, 7 May 2009, Lars Ellenberg wrote:
> On Thu, May 07, 2009 at 02:10:04PM +0000, Holger Kiehl wrote:
>> Hello
>>
>> While setting up a new system I noticed that write performance is
>> very low, only about 35MB/s. This is with 2.6.29.1 and DRBD 8.3.1.
>> The disk device is a md device (software raid) and when I run the
>> following test:
>>
>> dd if=/dev/zero of=/dev/md4 bs=64k count=100000
>>
>> it will give me approx. 160 MB/s. If I do this against /dev/drbd0 the
>> maximum I get is 70 MB/s when I disable DRBD on the secondary. The 70 MB/s
>> with the secodary enabled I do manage also when I play with sndbuf-size,
>> max-buffers, unplug-watermark and al-extents. But why is it limited
>> at 70 MB/s when secondary is disabled? The drbd.conf looks as follows:
>
>
> For large (exceeding the activity log) streaming writes,
> the activity log can be considered to always be cold.
>
> [the activity log being our implementation of the "write intent" logic.]
>
> an AL-transaction is a synchonous single sector write.
> that usually does not take too long, but may take quite a while if there
> are any scheduler timeouts, or read/modify/write cycles involved...
>
> If we assume that we have to do one activity log transaction per
> one AL-extent-size worth of data (which is currently hardcoded to 4 MB),
> and further assuming that the activity log transaction overhead would
> be the sole cause for the slow down, we can calculate:
>
> #!/bin/bash
> DRBD_throughputs_to_estimated_transaction_latency()
> {
> local ll_MBps=${1?} drbd_MBps=${2?}
>
> echo "local: $ll_MBps MB/s"
> echo "drbd disconnected: $drbd_MBps MB/s"
> echo "AL-transaction latency appears to be: " \
> $[1000000 * 4 / drbd_MBps - 1000000 * 4/ll_MBps] \
> "micro seconds"
> echo "you may want to verify this using e.g."
> echo "dd if=/dev/zero of=ll_dev bs=512 count=1000 oflag=direct"
> }
>
> DRBD_throughputs_to_estimated_transaction_latency 160 70
>
> local: 160 MB/s
> drbd disconnected: 70 MB/s
> AL-transaction latency appears to be: 32142 micro seconds
> you may want to verify this using e.g.
> dd if=/dev/zero of=ll_dev bs=512 count=1000 oflag=direct
>
> there. now, how long does a synchonous single sector write take on you
> lower level device?
>
dd if=/dev/zero of=/dev/md4 bs=512 count=1000 oflag=direct
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 3.38987 s, 151 kB/s
> does the dd throughput increase "dramatically", when reducing the bs *
> count size to fit into the activity log, (bs * count < al-extents * 4MB)
> and running "AL hot", i.e. on the second dd pass?
>
I would say no (al-extents is not set so it's default 127):
dd if=/dev/zero of=/dev/drbd0 bs=64k count=8127
8127+0 records in
8127+0 records out
532611072 bytes (533 MB) copied, 6.38975 s, 83.4 MB/s
dd if=/dev/zero of=/dev/drbd0 bs=64k count=8127
8127+0 records in
8127+0 records out
532611072 bytes (533 MB) copied, 5.94156 s, 89.6 MB/s
Holger