[DRBD-user] Protocol A vs Protocol C performance issue?

Vladislav Bogdanov bubble at hoster-ok.com
Thu Oct 22 21:20:19 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Hi Lars,

22.10.2015 17:57, Lars Ellenberg wrote:
> On Wed, Oct 21, 2015 at 11:42:52AM +0300, Vladislav Bogdanov wrote:
>>>> Throughput is concurrency times request size by request completion latency.
>>>> dd has no concurrency, in the given example, request size is 1M,
>>>> if you get 450 MB/s, your latency apparently is in the order of 2.3 ms.
>>>> If you want more throughput,
>>>> you need to decrease latency, or increase concurrency.
>>> Any suggestions on how to improve that then?
>> There was advise on some of related lists (may be even this) to tune
>> following parameters:
>> net/max-buffers=80k
>> net/max-epoch-size=20000
> Tuning max-buffers is certainly needed if you have a capable backend and
> replication link.
> Epoch size as only a very minor effect, typical epoch sizes will be
> rather in the small hundreds at most, anyways, so the
> new-epoch-because-we-exceeded-the-configured-maximum will very
> likely never be hit.

I can't recall details right now, but I tried several settings and this 
one was chosen.
I'll do more testing (on a capable HW) in next few months.

>> I additionally use
>> disk/c-max-rate=1500m
>> disk/c-fill-target=0
> I strongly recommend using c-fill-target, and not c-delay target,
> but you may need to tune it to a value suitable for your setup.

The point was to provide/restore redundancy as fast as possible after 
second node is powered on in the price of app speed.

I played several days with different settings, but this one was optimal 
for my task. I got more than 1GB/s on initial sync - more than disks can 
provide on writes (because of read-read verification with SSDs?).

> c-max-rate is by default at 100M, so yes, if you want more than that,
> you need to change it.
> To clarify,
> both are meaningful only in the context of a *resync*,
> so are meaningless for *replication*, *cannot* have any effect
> on the appliaction throughput during normal operation.


>> and get >800MB/s writes with protocol C over 10Gbps Ethernet
>> (ConnectX-3 in ethernet or IPoIB mode) with IO depth >=32 and IO
>> size 512k (I use disktest from LTP project). That is what 6 Intel DC
>> S3610 can provide in HW RAID-10 with disk cache turned on.
>> And, of course, Lars is absolutely correct that it is merely
>> impossible to get high IO speeds with just one IO thread.


More information about the drbd-user mailing list