[DRBD-user] write performance issues

[Mark Dokter]

Hello!

I have a rather complex setup and experience severe performance
degradation, especially on small writes.

------------------------------------
The scenario:
- Two xen servers with:
  - kernel 2.6.32 (xen-stable git)
  - xen 4.1.0
  - ubuntu 11.04
  - backported drbd 8.3.7
  - domUs using pv-on-hvm drivers

- DRBD volumes for the domUs:
  HW RAID6 -> LVM -> DRBD

- Sync Network:
  - dedicated 1Gbit/s ethernet (switched)

- DRBD volume config:

resource pegasus {
  device    /dev/drbd8;
  disk      /dev/data/pegasus;
  meta-disk /dev/data/drbd-metadata[8];

  net {
    allow-two-primaries;
    max-buffers 8000;
    max-epoch-size 8000;
    sndbuf-size 0;
    unplug-watermark 16;
  }

  syncer {
    al-extents 3389;
    rate 110M;
  }

  on xen01 {
    address   172.16.128.1:7797;
  }

  on xen02 {
    address   172.16.128.2:7797;
  }
}

------------------------------------

I have several domUs set up like this. The worst slowdowns are with
- mailserver (postfix/cyrus)
- mysql
- copying a lot of small files

A small test I ran(disk/net i/o checked with dstat):
- copied the mailman directory (110 lists, 6.5 GB of data)
  cp -a mailman mailman-backup
- after about 15 minutes I had a look how far the copying got:
  about 1 GB copied.
- in the dom0 there was about 1 MB/s traffic on the sync network.
  Same speeds as the disk write performance in the domU
- I disconnected the DRBD volume on the second xen dom0
  (the one the virtual machine was not running on)
  drbdadm down pegasus
- the disk write performance in the domU went up to around 30 - 40 MB/s

The 30-40 MB/s aren't really fast either, but that'd be bearable. When I
reconnected the volume after the copying was done, the resync went quite
fast (also with those 30 - 40 MB/s).
I know, the numbers aren't that exact, but it's enough to see the big
difference when the DRBD volumes are in connected/disconnected state.

I already did quite some research on that issue (that's how I came up
with that DRBD config), but didn't succeed in fixing it. Maybe someone
on this list has suggestions?

Thanks,
Mark