[DRBD-user] slow write performance (1/3rd of disk speed)

Bram Matthys syzop at vulnscan.org
Sun Nov 3 11:26:02 CET 2013

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi,

Using a RAID10 array now and (following Arnolds advice) the metadata is now
on another (SSD) disk.
Write speeds are still bad, but are ONLY low when both nodes are connected.

root at st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd
if=/dev/zero of=5G bs=1M count=5000; rm -f 5G
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 57.1802 s, 91.7 MB/s

root at st2:/data# drbdadm disconnect data
root at st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd
if=/dev/zero of=5G bs=1M count=5000; rm -f 5G
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 21.4724 s, 244 MB/s

When I (re)connect data again, the resync speed is fine:
st1# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9

 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:5121168 dw:5121168 dr:2010340 al:0 bm:133 lo:0 pe:206 ua:0 ap:0
ep:1 wo:d oos:3130232
        [======>.............] sync'ed: 38.9% (3056/4996)Mfinish: 0:00:14
speed: 220,760 (220,760) want: 256,000 K/sec

And, as mentioned earlier, netperf achieves near-10gE speed:
# netperf -H 192.168.220.1
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.220.1
(192.168.220.1) port 0 AF_INET : demo
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.00    9897.79
(same in both directions)

It doesn't matter whether I make server st1 or st2 primary (so either
server) and do the tests... with both they achieve around 240 MB/s write
speed in disconnected mode and around 80-90 MB/s when connected, so ~ 1/3rd.

Any ideas?

# drbdadm dump
# /etc/drbd.conf
common {
    protocol               C;
    startup {
        degr-wfc-timeout 120;
        wfc-timeout      120;
    }
}

# resource data on st2: not ignored, not stacked
resource data {
    on st1 {
        device           /dev/drbd1 minor 1;
        disk             /dev/md4;
        address          ipv4 192.168.220.1:7789;
        meta-disk        /dev/md3 [0];
    }
    on st2 {
        device           /dev/drbd1 minor 1;
        disk             /dev/md4;
        address          ipv4 192.168.220.2:7789;
        meta-disk        /dev/md3 [0];
    }
    net {
        data-integrity-alg sha256;
        max-buffers      8000;
        max-epoch-size   8000;
        sndbuf-size      512k;
    }
    disk {
        no-disk-barrier;
        no-disk-flushes;
    }
    syncer {
        csums-alg        sha256;
        rate             250M;
    }
}

NOTE: I was using protocol A without no-disk-* before, just using
C/no-disk-* to see if it made any significant difference: answer is no.

root at st2:/data# uname -a
Linux st2 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux
root at st2:/data# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:5121332 nr:0 dw:15358180 dr:5130289 al:3773 bm:377 lo:0 pe:0 ua:0
ap:0 ep:1 wo:d oos:0

root at st2:/data# dpkg --list|grep drbd
ii  drbd8-utils                        2:8.3.13-2                    amd64
      RAID 1 over tcp/ip for Linux utilities

If you need anything else, just let me know.

Thanks again,

Bram.

Arnold Krille wrote, on 7-10-2013 1:37:
> Hi,
> 
> On Sun, 06 Oct 2013 18:54:12 +0200 Bram Matthys <syzop at vulnscan.org>
> wrote:
>> I'm currently testing DRBD and am having write performance problems.
>> On the local raid array I achieve 124MB/s, but with DRBD I get only
>> 41MB/s out of it, or if the secondary node is down (to rule out
>> network issues) then 53MB/s at best.
>> Tried protocol A / B / C, with and without no-disk-barriers and
>> no-disk-flushes, but this didn't change much (only +/- 2MB/s
>> difference).
> <snip>
> 
> Yep, internal meta-disk: write block, seek to end, write log, seek to
> front, write block, seek to end, write log...
> 
> Put the meta-disk on a different hd and watch your write rate go up.
> 
> - Arnold
> 
> 
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
> 


-- 
Bram Matthys
Software developer/IT consultant        syzop at vulnscan.org
Website:                                  www.vulnscan.org
PGP key:                       www.vulnscan.org/pubkey.asc
PGP fp: EBCA 8977 FCA6 0AB0 6EDB  04A7 6E67 6D45 7FE1 99A6





More information about the drbd-user mailing list