[DRBD-user] Protocol A vs Protocol C performance issue?

Vladislav Bogdanov bubble at hoster-ok.com
Wed Oct 21 10:42:52 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


20.10.2015 21:52, Joeri Casteels wrote:
>
>
>> On 20 Oct 2015, at 20:47, Lars Ellenberg <lars.ellenberg at linbit.com> wrote:
>>
>> On Tue, Oct 20, 2015 at 01:07:48PM +0200, Joeri Casteels wrote:
>>> Dear All,
>>>
>>>
>>> Doing a new set-up of DRBD on Ubuntu LTS 14.04 with stock DRBD
>>> When Setting to protocol A i get double the bandwidth as protocol C anybody an idea? not changing anything else of the settings except the protocol
>>>
>>> PROTOCOL C:
>>> d if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
>>> 20000+0 records in
>>> 20000+0 records out
>>> 20971520000 bytes (21 GB) copied, 46.3583 s, 452 MB/s
>>>
>>> PROTOCOL A:
>>> dd if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
>>> 20000+0 records in
>>> 20000+0 records out
>>> 20971520000 bytes (21 GB) copied, 23.8356 s, 880 MB/s
>>
>> dd does:
>>   write one block size # which takes some time
>>   write next bs # which again takes some time
>>   ...
>>
>> Without oflag=direct,
>> you write into the linux page cache, the write itself takes microseconds
>> (at most), and you get many such writes per second, resulting usually in
>> a throughput of several gigabytes per second (until you hit the
>> boundaries of the page cache and are throttled down to the rate of the
>> background write-out).
>>
>> With oflag=direct,
>> it bypasses the linux page cache,
>> the single write call takes longer,
>> and you do no longer get that many writes per second.
>>
>> With protocol B and C, the single write latency even includes
>> the network latency and remote disk latency, possibly flush latencies,
>> and you get even less writes per second.
>>
>> Throughput is concurrency times request size by request completion latency.
>>
>> dd has no concurrency, in the given example, request size is 1M,
>> if you get 450 MB/s, your latency apparently is in the order of 2.3 ms.
>>
>> If you want more throughput,
>> you need to decrease latency, or increase concurrency.
>
> Any suggestions on how to improve that then?

There was advise on some of related lists (may be even this) to tune 
following parameters:
net/max-buffers=80k
net/max-epoch-size=20000

I additionally use
disk/c-max-rate=1500m
disk/c-fill-target=0
and get >800MB/s writes with protocol C over 10Gbps Ethernet (ConnectX-3 
in ethernet or IPoIB mode) with IO depth >=32 and IO size 512k (I use 
disktest from LTP project). That is what 6 Intel DC S3610 can provide in 
HW RAID-10 with disk cache turned on.

And, of course, Lars is absolutely correct that it is merely impossible 
to get high IO speeds with just one IO thread.

Best, Vladislav





More information about the drbd-user mailing list