[DRBD-user] Sgisdela! Help! Horrible DRBD performance on CentOS 5. Where do I start looking?

Michael Joyner mjoyner at ewc.edu
Thu Jun 10 02:32:18 CEST 2010


Two node setup for serving out NFS to vSphere.

timing test is : sync; time iozone -a -e -g 4096

/data/nfs is on LVM striped across 3 DRBD devices. EXT4. data=journal.
/var/tmp is local filesystem.

both are on same set of disk platters and controller. Raid 6. 64K blocksize.

FYI, the initial sync (5 TB) used a full gigabit of bandwidth without
hesitation.

=== DRBD TIMES (both nodes up) ==============================
real    0m10.028s
user    0m0.116s
sys    0m1.538s

real    0m10.045s
user    0m0.085s
sys    0m1.541s

real    0m9.990s
user    0m0.091s
sys    0m1.578s

real    0m9.970s
user    0m0.099s
sys    0m1.557s

real    0m9.960s
user    0m0.091s
sys    0m1.499s

=== DRBD TIMES (2nd node down) ==============================
real    0m3.754s
user    0m0.093s
sys    0m1.070s

real    0m3.855s
user    0m0.079s
sys    0m1.064s

real    0m3.938s
user    0m0.094s
sys    0m1.044s

real    0m3.809s
user    0m0.066s
sys    0m1.067s

real    0m3.863s
user    0m0.069s
sys    0m1.072s

=== LOCAL TIMES ==============================
real    0m1.770s
user    0m0.070s
sys    0m0.977s

real    0m1.809s
user    0m0.085s
sys    0m0.974s

real    0m1.737s
user    0m0.067s
sys    0m0.942s

real    0m2.007s
user    0m0.058s
sys    0m0.955s

real    0m1.808s
user    0m0.072s
sys    0m0.956s

=== After re-enabling 2nd node ==============================

root at san-node-2 ~]# cat /proc/drbd
version: 8.3.2 (api:88/proto:86-90)
GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:07:55
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:5180 dw:5180 dr:0 al:0 bm:24 lo:1 pe:1389 ua:0 ap:0 ep:1 wo:b
oos:43964
    [==>.................] sync'ed: 16.7% (43964/49144)K
    finish: 0:00:08 speed: 5,180 (5,180) K/sec
 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
    ns:0 nr:49044 dw:49044 dr:0 al:0 bm:44 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0
 2: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:6896 dw:6896 dr:0 al:0 bm:28 lo:1 pe:1346 ua:0 ap:0 ep:1 wo:b
oos:42436
    [===>................] sync'ed: 23.1% (42436/49332)K
    finish: 0:00:05 speed: 6,896 (6,896) K/sec

=== ifconfig ==============================

bond0:0   Link encap:Ethernet  HWaddr 00:1B:21:26:B8:18
          inet addr:192.168.XXX.XX0  Bcast:192.168.XXX.255
Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:9000  Metric:1


=== rpm -qa|grep drbd ==============================
kmod-drbd83-8.3.2-6.el5_3
drbd83-8.3.2-6.el5_3

=== uname -a ==============================
Linux san-node-1.ewc.edu 2.6.18-194.3.1.el5 #1 SMP Thu May 13 13:08:30 EDT
2010 x86_64 x86_64 x86_64 GNU/Linux

=== cat /etc/issue ==============================
CentOS release 5.5 (Final)
Kernel \r on an \m

=== lspci ==============================
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078
(rev 04)

=== box to box copy test ==============================
dd if=/dev/sda of=testfile bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.94192 seconds, 553 MB/s

scp testfile root at san-node-2:/var/tmp/testfile
testfile                                                        100% 1024MB
46.6MB/s   00:22

=== /etc/drbd.conf ==============================

global {
  usage-count yes;
}
common {
    protocol C;
    syncer { rate 700000K; }
    net {
        sndbuf-size 0;
        after-sb-0pri discard-least-changes;
        after-sb-1pri discard-secondary;
    }
    startup {
        degr-wfc-timeout 30000;
    }
    disk {
        no-disk-flushes;
        no-md-flushes;
    }

}
resource sdb {
  on san-node-1.ewc.edu {
    device    /dev/drbd0;
    disk      /dev/sdb;
    address   192.168.XXX.XX1:7789;
    meta-disk internal;
  }
  on san-node-2.ewc.edu {
    device    /dev/drbd0;
    disk      /dev/sdb;
    address   192.168.XXX.XX2:7789;
    meta-disk internal;
  }
}
resource sdc {
  on san-node-1.ewc.edu {
    device    /dev/drbd1;
    disk      /dev/sdc;
    address   192.168.XXX.XX1:7790;
    meta-disk internal;
  }
  on san-node-2.ewc.edu {
    device    /dev/drbd1;
    disk      /dev/sdc;
    address   192.168.XXX.XX2:7790;
    meta-disk internal;
  }
}
resource sdd {
  on san-node-1.ewc.edu {
    device    /dev/drbd2;
    disk      /dev/sdd;
    address   192.168.XXX.XX1:7791;
    meta-disk internal;
  }
  on san-node-2.ewc.edu {
    device    /dev/drbd2;
    disk      /dev/sdd;
    address   192.168.XXX.XX2:7791;
    meta-disk internal;
  }
}

-- 
Michael Joyner/System Administrator
Edward Waters College
1658 Kings Road
Jacksonville, FL 32209
(V) 904-470-8170/(F) 904-470-8170
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20100609/3829fd51/attachment.htm>


More information about the drbd-user mailing list