[DRBD-user] Very poor performance

[Adam Goryachev]

I have a pair of DRBD machines, and I'm getting very poor performance
from the DRBD when the second server is connected. I've been working on
resolving the performance issues for a few months, with not much luck.
Here is some info on the current configuration:

Both machines are identical (apart from drives):
CPU Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
RAM 8G

One machine (the primary) has Intel 480G SSD drives (currently just 2)
in RAID5 (one drive missing, will be added in another few days)
The second machine (backup) has 2 x WD Caviar Black 2TB HDD (in RAID1,
limited to 960GB)

Actually, I created the partitions on the SSD a little smaller than max
capacity because I read this can assist with performance, and matched
the HDD's size to the SSD RAID block size.

I am using all software from Debian Stable with all updates/security
updates installed:
drbd8-utils 2:8.3.7-2.1
Linux san1 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64
GNU/Linux

My config file for DRBD is:
resource storage2 {
    protocol A;
    device /dev/drbd2 minor 2;
    disk /dev/md1;
    meta-disk internal;
    on san1 {
        address 172.17.56.1:7802;
    }
    on san2 {
        address 172.17.56.2:7802;
    }

    net {
        after-sb-0pri discard-younger-primary;
        after-sb-1pri discard-secondary;
        after-sb-2pri call-pri-lost-after-sb;
        max-buffers 8000;
        max-epoch-size 8000;
        unplug-watermark 4096;
        sndbuf-size 512k;
    }
    startup {
        wfc-timeout 10;
        degr-wfc-timeout 20;
    }
    syncer {
        rate 100M;
    }
}

root at san1:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757

 2: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r----
    ns:30035906 nr:0 dw:29914667 dr:42547522 al:49271 bm:380 lo:0 pe:0
ua:0 ap:0 ep:1 wo:b oos:0

root at san1:/etc/drbd.d# dd if=/dev/zero of=/dev/mapper/vg0-testdisk
oflag=direct bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 2.48851 s, 42.1 MB/s

However, if I stop DRBD on the secondary:
root at san1:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757

 2: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown A r----
    ns:30797617 nr:0 dw:30898985 dr:45041554 al:51325 bm:847 lo:0 pe:0
ua:0 ap:0 ep:1 wo:b oos:118756

Then I get good performance:
root at san1:/etc/drbd.d# dd if=/dev/zero of=/dev/mapper/vg0-testdisk
oflag=direct bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.468905 s, 224 MB/s

Can anyone suggest how I might improve performance while the secondary
is connected? In the worst case scenario, I would expect a write speed
to these drives between 100 and 150M/s. I can't test writing at the
moment, since they are used by DRBD, but read performance:
root at san2:/etc/drbd.d# dd if=/dev/md1 of=/dev/null oflag=dsync bs=1M
count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.706124 s, 148 MB/s

I am using a single 1G crossover ethernet for DRBD sync

I was hoping that the SSD's would be used for reads, since they are
great at large number of small random reads, and the smaller number of
writes that occur will be sent to the SSD's also very quickly, and more
slowly to the backup san. Since the backup san only has to deal with
seeks for writes, it should be able to keep up, but it looks like
performance is being dragged down a lot.

Should I increase the connection between the servers for sync to 2 x 1G
which would exceed max write speed of the disks on san2?
Is there some other config option to say "I don't mind if san2 is behind
san1, just cache the changes and make them when you can", similar to
running DRBD over a slow remote connection?

Any other suggestions or comments?

Thank you for reading this, and any assistance you can provide.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au