Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all,
I've been trying to do some tuning with DRBD. The network interface
sustains 1.225 GB/sec tested with iperf (multiple tests, hf and fd). The
disks themselves sustain 1.2 GB/sec sequential writes (975 GB write via
'dd if=/dev/zero of=/dev/sda5 bs=4M), tested repeatedly and on both
nodes. The storage array is an 8-disk (146 GB 15krpm 2.5" SAS disks on
an LSI RAID controller with a 2208 ASIC).
When I started tweaking with DRBD 8.3.16, I was able to get a
somewhat consistent >750~838 MB/sec with the same 'dd' command to
/dev/drbd0. Once I tried enabling no-disk-flushes and no-disk-barriers
though, the performance fell to ~500 MB/sec. Since I disabled it, it has
not returned to a much higher speed.
I reverted all my changes except for 'sndbuf-size 512k;', which is
where I saw the biggest jump in performance, and I got up to the
~650MB/sec, but nothing faster.
So I just decided to wipe out the 8.3 install (and rm'ed
/dev/drbd.d), did a fresh install of 8.4.4 and started it sync'ing.
Previously, with 8.3.16, I'd do an initial 'dd' to quickly push the
other node to UpToDate, which went at 473MB/sec across the same 976 GB
of sequential writing.
This time, having freshly installed and configured 8.4.4, I tried to
push along the initial sync and I'm only getting 212 MB/sec...
I'm not sure what I am doing wrong that has caused such a draw down
on performance. I could really use some help/guidance here. I thought
8.4.3+ was supposed to be a lot faster, so I have to assume I am doing
something wrong. :(
Current setup:
[root at an-c07n01 ~]# uname -a
Linux an-c07n01.alteeve.ca 2.6.32-431.1.2.el6.x86_64 #1 SMP Sun Nov 24
09:37:37 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
[root at an-c07n01 ~]# rpm -q drbd84-utils
drbd84-utils-8.4.4-2.el6.elrepo.x86_64
[root at an-c07n01 ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by
phil at Build64R6, 2013-10-14 15:33:06
0: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r---d-
ns:172939348 nr:0 dw:172791788 dr:156040 al:42172 bm:10545 lo:0
pe:2049 ua:0 ap:2049 ep:1 wo:d oos:780375744
[==>.................] sync'ed: 18.2% (762084/930816)M
finish: 1:05:12 speed: 199,468 (189,036) K/sec
[root at an-c07n01 ~]# cat /etc/drbd.d/global_common.conf
global {
usage-count yes;
}
common {
handlers {
fence-peer "/sbin/rhcs_fence";
}
startup {
}
options {
}
disk {
fencing resource-and-stonith;
}
net {
protocol C;
allow-two-primaries yes;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}
[root at an-c07n01 ~]# cat /etc/drbd.d/r0.res
resource r0 {
device /dev/drbd0;
disk /dev/sda5;
meta-disk internal;
startup {
become-primary-on both;
}
net {
verify-alg md5;
data-integrity-alg md5;
}
disk {
disk-flushes no;
md-flushes no;
}
on an-c07n01.alteeve.ca {
address 10.10.70.1:7788;
}
on an-c07n02.alteeve.ca {
address 10.10.70.2:7788;
}
}
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?