[DRBD-user] very [very] low speed in DRBD synchronization

like of nobody like.of.nobody at gmail.com
Sun Jul 3 19:13:46 CEST 2011

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


i have two node drbd in primary/secondary mode with this config:
Connection between nodes produced by back-to-back 1Gigabit Ethernet  (cat 5)

DRBD Config:
resource "res" {
  protocol C;

  handlers {
    pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
    outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";

  startup {
    wfc-timeout  20;
    degr-wfc-timeout 20;    # 2 minutes.

  disk {
    on-io-error   detach;

  net {
    timeout       60;    #  6 seconds  (unit = 0.1 seconds)
    connect-int   10;    # 10 seconds  (unit = 1 second)
    ping-int      10;    # 10 seconds  (unit = 1 second)
    ping-timeout   5;    # 500 ms (unit = 0.1 seconds)
    max-buffers 8000;
    unplug-watermark   128;
    max-epoch-size 8000;
    ko-count 4;
    cram-hmac-alg "sha1";
    shared-secret "jecCOMirrorSystemWithPDCO&KVM";
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;

  syncer {
    rate 50M;
    al-extents 3389;

  on primary {
    device     /dev/drbd0;
    disk       /dev/sdb1;
    meta-disk  /dev/VolGroup00/p[0];

  on secondary {
    device    /dev/drbd0;
    disk       /dev/cciss/c0d0p7;
    meta-disk /dev/cciss/c0d0p8[0];

OS: Sience Linux 6 (x86_64)
Kernel: 2.6.32-71.29.1.el6   IO Scheduler: DeadLine

DRBD: version: 8.3.10
Compiled From Source with this commands:
	# tar xf drbd-8.3.10.tar.gz
	# cd drbd-8.3.10
	# make rpm
	# make km-rpm KDIR=/usr/src/kernels/2.6.32-71.29.1.el6.x86_64

primary: HP G7, with 24Core CPU
secondary: HP G6, with 8Core CPU

by this environment, synchronization speed is very very slow, about 5Mbit/s ...
by a simple scp command, i can achieve about 35Mbit/s  ...

# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by
rpmbuild at P2-SL6, 2011-06-07 11:05:12
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:214112 nr:0 dw:212963132 dr:986726277 al:417500 bm:75663 lo:1
pe:320 ua:745 ap:28 ep:1 wo:b oos:1638452
	[=>..................] sync'ed: 11.2% (1638452/1839516)K
	finish: 0:04:39 speed: 5,844 (6,484) K/sec

when we connect primary to secondary throw "drbdadm connect", load of
primary grows ... at initial sync, load of primary been about 100%,

On drbd0 at primary side, we have six simple virtual machine run throw qemu+kvm.
After drbd connection, performance really goes down ...

Output of sar command before/after drbd connection is:
      DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await
   svctm     %util
 dev147-0      0.00      0.00      0.00      0.00     15.84      0.00
    0.00     99.01

The question is Why?
Why we should experience this degradation in performance?

One note may help: at drbd startup (service drbd start) and after
connection of secondary to primary, we saw this error:
block drbd0: meta connection shut down by peer

on secondary:
block drbd0: Barriers not supported on meta data device - disabling

Best regards ...

More information about the drbd-user mailing list