[DRBD-user] Why is write performance so low?

[Holger Kiehl]

Hello

While setting up a new system I noticed that write performance is
very low, only about 35MB/s. This is with 2.6.29.1 and DRBD 8.3.1.
The disk device is a md device (software raid) and when I run the
following test:

    dd if=/dev/zero of=/dev/md4 bs=64k count=100000

it will give me approx. 160 MB/s. If I do this against /dev/drbd0 the
maximum I get is 70 MB/s when I disable DRBD on the secondary. The 70 MB/s
with the secodary enabled I do manage also when I play with sndbuf-size,
max-buffers, unplug-watermark and al-extents. But why is it limited
at 70 MB/s when secondary is disabled? The drbd.conf looks as follows:

    global {
        usage-count no;
    }


    common {
      syncer { rate 100M; }
    }

    resource r0 {
      protocol C;
      startup {
        degr-wfc-timeout 120;    # 2 minutes.
        outdated-wfc-timeout 2;  # 2 seconds.
      }

      disk {
        on-io-error   detach;
        no-disk-flushes;
        no-md-flushes;
        no-disk-barrier;
      }

    #  net {
    #    sndbuf-size 1M;
    #    max-buffers     65536;
    #    unplug-watermark   16384;
    #    after-sb-0pri disconnect;
    #    after-sb-1pri disconnect;
    #    after-sb-2pri disconnect;
    #    rr-conflict disconnect;
    #  }

    #  syncer {
    #    rate 100M;
    #    al-extents 577;
    #  }

      on asterix {
        device     /dev/drbd0;
        disk       /dev/md4;
        address    10.0.0.1:7788;
        meta-disk  internal;
      }

      on obelix {
        device    /dev/drbd0;
        disk      /dev/md4;
        address   10.0.0.2:7788;
        meta-disk internal;
      }
    }

I have the feeling that I do something very obviously wrong. Or is my
expectation of getting nearly the same performance on the drbd device
as on the md device wrong?

What else can I do to get better write performance?

Thanks,
Holger

PS: Network for DRBD between the two systems is a direct 10Gbit connection
     (actually 2 10Gbit bonded together via bonding).