[DRBD-user] Having trouble with performance on hardware RAID + 10gbps

Digimer lists at alteeve.ca
Thu Jan 9 07:39:14 CET 2014

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Hi all,

   I've been trying to do some tuning with DRBD. The network interface 
sustains 1.225 GB/sec tested with iperf (multiple tests, hf and fd). The 
disks themselves sustain 1.2 GB/sec sequential writes (975 GB write via 
'dd if=/dev/zero of=/dev/sda5 bs=4M), tested repeatedly and on both 
nodes. The storage array is an 8-disk (146 GB 15krpm 2.5" SAS disks on 
an LSI RAID controller with a 2208 ASIC).

   When I started tweaking with DRBD 8.3.16, I was able to get a 
somewhat consistent >750~838 MB/sec with the same 'dd' command to 
/dev/drbd0. Once I tried enabling no-disk-flushes and no-disk-barriers 
though, the performance fell to ~500 MB/sec. Since I disabled it, it has 
not returned to a much higher speed.

   I reverted all my changes except for 'sndbuf-size 512k;', which is 
where I saw the biggest jump in performance, and I got up to the 
~650MB/sec, but nothing faster.

   So I just decided to wipe out the 8.3 install (and rm'ed 
/dev/drbd.d), did a fresh install of 8.4.4 and started it sync'ing. 
Previously, with 8.3.16, I'd do an initial 'dd' to quickly push the 
other node to UpToDate, which went at 473MB/sec across the same 976 GB 
of sequential writing.

   This time, having freshly installed and configured 8.4.4, I tried to 
push along the initial sync and I'm only getting 212 MB/sec...

   I'm not sure what I am doing wrong that has caused such a draw down 
on performance. I could really use some help/guidance here. I thought 
8.4.3+ was supposed to be a lot faster, so I have to assume I am doing 
something wrong. :(

Current setup:

[root at an-c07n01 ~]# uname -a
Linux an-c07n01.alteeve.ca 2.6.32-431.1.2.el6.x86_64 #1 SMP Sun Nov 24 
09:37:37 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

[root at an-c07n01 ~]# rpm -q drbd84-utils

[root at an-c07n01 ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by 
phil at Build64R6, 2013-10-14 15:33:06
  0: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r---d-
     ns:172939348 nr:0 dw:172791788 dr:156040 al:42172 bm:10545 lo:0 
pe:2049 ua:0 ap:2049 ep:1 wo:d oos:780375744
	[==>.................] sync'ed: 18.2% (762084/930816)M
	finish: 1:05:12 speed: 199,468 (189,036) K/sec

[root at an-c07n01 ~]# cat /etc/drbd.d/global_common.conf
global {
	usage-count yes;

common {
	handlers {
		fence-peer "/sbin/rhcs_fence";

	startup {

	options {

	disk {
		fencing resource-and-stonith;

	net {
		protocol C;
		allow-two-primaries yes;
		after-sb-0pri discard-zero-changes;
		after-sb-1pri discard-secondary;
		after-sb-2pri disconnect;

[root at an-c07n01 ~]# cat /etc/drbd.d/r0.res
resource r0 {
	device /dev/drbd0;
	disk /dev/sda5;
	meta-disk internal;

	startup {
		become-primary-on both;

	net {
		verify-alg md5;
		data-integrity-alg md5;

	disk {
		disk-flushes no;
		md-flushes no;

	on an-c07n01.alteeve.ca {
	on an-c07n02.alteeve.ca {

Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?

More information about the drbd-user mailing list