[DRBD-user] DRBD terrible sync performance on 10GigE

Lars Ellenberg lars.ellenberg at linbit.com
Fri Dec 4 17:04:26 CET 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Fri, Dec 04, 2015 at 10:15:02AM +0900, Hiroshi Fujishima wrote:
> Hi,
> Emmanuel Florac <eflorac at intellique.com> writes:
> > However, no matter the way I configure DRBD, the throughput is limited
> > to 100MB/s. It really looks like some hardcoded limit. I can reliably
> > lower performance by tweaking the settings, but it never goes over
> > 1Gbit (122MB/s are reached for a couple of seconds at a time). I'm
> > really pulling my hair on this one.
> >
> >     plain vanilla kernel 3.18.24 amd64
> >     drbd 8.9.2~rc1-1~bpo70+1
> the following configuration may solve the problem.
>     net {
>         max-buffers      131072;
>         max-epoch-size   20000;
>     }
>     disk {
>         c-plan-ahead       0;
>         resync-rate      200M;

You are not supposed to disable the resync controller,
you are supposed to correctly use it.


... you should:

    * set c-plan-ahead to 20 (default with 8.4),
	or more if there’s a lot of latency on the connection (WAN
	link with protocol A); or less, if you want to have it
	react faster to changes

    * leave the fixed resync rate (the initial guess for the controller)
      at about 30% or less of what your hardware can handle;

    * set c-max-rate to 100% (or slightly more) of
      what your hardware can handle;
      (the default is 100M, which was the effective limitation in this case)

    * set c-fill-target to the minimum (just as high as necessary)
      that gets your hardware saturated, if the system is otherwise idle.

Respectively, figure out the maximum possible resync rate in your
setup while the system is idle, then set c-fill-target to the
minimum setting that still reaches that rate.

And finally, while checking application request
latency/responsiveness, tune c-min-rate to the maximum that still
allows for acceptable responsiveness.

You may need to adjust max-buffers and/or tcp send/receive buffer
sizes as well.

: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
please don't Cc me, but send to list   --   I'm subscribed

More information about the drbd-user mailing list