[DRBD-user] Protocol A vs Protocol C performance issue?

Lars Ellenberg lars.ellenberg at linbit.com
Thu Oct 22 16:57:08 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Wed, Oct 21, 2015 at 11:42:52AM +0300, Vladislav Bogdanov wrote:
> >>Throughput is concurrency times request size by request completion latency.
> >>
> >>dd has no concurrency, in the given example, request size is 1M,
> >>if you get 450 MB/s, your latency apparently is in the order of 2.3 ms.
> >>
> >>If you want more throughput,
> >>you need to decrease latency, or increase concurrency.
> >
> >Any suggestions on how to improve that then?
> There was advise on some of related lists (may be even this) to tune
> following parameters:
> net/max-buffers=80k
> net/max-epoch-size=20000

Tuning max-buffers is certainly needed if you have a capable backend and
replication link.

Epoch size as only a very minor effect, typical epoch sizes will be
rather in the small hundreds at most, anyways, so the
new-epoch-because-we-exceeded-the-configured-maximum will very
likely never be hit.

> I additionally use
> disk/c-max-rate=1500m
> disk/c-fill-target=0

I strongly recommend using c-fill-target, and not c-delay target,
but you may need to tune it to a value suitable for your setup.

c-max-rate is by default at 100M, so yes, if you want more than that,
you need to change it.

To clarify,
both are meaningful only in the context of a *resync*,
so are meaningless for *replication*, *cannot* have any effect
on the appliaction throughput during normal operation.

> and get >800MB/s writes with protocol C over 10Gbps Ethernet
> (ConnectX-3 in ethernet or IPoIB mode) with IO depth >=32 and IO
> size 512k (I use disktest from LTP project). That is what 6 Intel DC
> S3610 can provide in HW RAID-10 with disk cache turned on.
> And, of course, Lars is absolutely correct that it is merely
> impossible to get high IO speeds with just one IO thread.

: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
please don't Cc me, but send to list   --   I'm subscribed

More information about the drbd-user mailing list