[DRBD-user] resync-rate not applied

Lars Ellenberg lars.ellenberg at linbit.com
Thu Aug 16 17:44:13 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Wed, Aug 15, 2012 at 02:21:33AM +0200, Dennis Jacobfeuerborn wrote:
> On 08/14/2012 02:43 AM, Dennis Jacobfeuerborn wrote:
> > On 08/03/2012 03:27 AM, Dennis Jacobfeuerborn wrote:
> >> Hi,
> >> I just set up a traditional drbd primary/secondary system between two hosts
> >> that are connected with gigabit ports. The problem is that the resync rate
> >> is only 5M even though in the config file I set it to 40M.
> >>
> >> I also tried:
> >> drbdadm disk-options --resync-rate=40M drbd1
> >>
> >> which doesn't result in an error but apparently doesn't get applied either.
> >> I also tried 1M but that gets ignored too.
> >>
> >> Any idea what might be going on here?
> >> (This is drbd 8.4.1 on a Centos 6.3 system)
> > 
> > Does nobody have any idea?
> > I just had to re-sync and the "want" value jumped around wildly between 0
> > K/sec and 80.000 K/sec and the speed it self is terrible:
> > 
> > 	finish: 0:05:07 speed: 608 (628) want: 7,160 K/sec
> > 	finish: 0:04:33 speed: 684 (628) want: 43,760 K/sec
> > 	finish: 0:04:38 speed: 680 (628) want: 9,600 K/sec
> > 	finish: 0:05:04 speed: 624 (628) want: 0 K/sec
> > 	finish: 0:04:51 speed: 648 (628) want: 10,760 K/sec
> > 
> > It looks like the variable sync mechanism does more harm than good and I
> > don't seem to be able to turn it off properly.
> Downgrading to DRBD 8.3 fixed this problem so this seems to be an issue
> with 8.4.1.

The dynamically adaptive resync rate controller is enabled by default
with 8.4, and disabled by default with 8.3.

To explicitly enable or disable,
set the "c-plan-ahead" to 20 (enable) or 0 (disable).

Note that, while enabled, the setting for the old fixed sync rate
is used only as initial guess for the controller. After that,
only the c-* settings are used, so changing the fixed sync rate
while the controller is enabled won't have much effect.

The controller does the following:

It tries to use up as much network and disk bandwidth as it can get,
but no more than c-max-rate,
and throttles if

 * more resync requests are in flight than what amounts to c-fill-target 
   (or, if c-fill-target is set to 0, if the current estimated response
   delay from the peer is more than c-delay-target)

 * it detects application IO (read or write), AND the current estimated
   resync rate is above c-min-rate (unless c-min-rate is 0).

The default c-min-rate with 8.4 is 250 kB/s (the old default of the
fixed sync-rate), with 8.3 it was 4M/s.

This "throttle if application IO is detected" is active even if the
fixed sync rate is used. You can (but IMO should not, see below)
disable this specific throttling by setting c-min-rate to 0.

You should

 * set c-plan-ahead to 20 (default with 8.4),

 * leave the fixed resync rate (the initial guess for the controller)
   at about 30% or less of what your hardware can handle,

 * set c-max-rate to 100% (or slightly more) of what your hardware can handle,

 * set c-fill-target to the minimum (just as high as necessary) that gets
   your hardware saturated, if the system is otherwise idle.

   Respectively, figure out the maximum possible resync rate in your
   setup while the system is idle, then set c-fill-target to the minimum
   setting that still reaches that rate.

 * while checking application request latency/responsiveness,
   tune c-min-rate to the maximum that still allows for acceptable

I'll see if we can make this into some blog post.


: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

More information about the drbd-user mailing list