[DRBD-user] Slower disk throughput on DRBD partition

Frederic DeMarcy fred.demarcy.ml at gmail.com
Tue Jan 31 18:03:51 CET 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi

I've configured DRBD with a view to use it with MySQL (and later on Pacemaker + Corosync) in a 2 nodes primary/secondary
(master/slave) setup.

Once both nodes have been configured and everything has been synch'ed. I can umount /var/lib/mysql on the primary then
move the primary to secondary, bring up the secondary as primary, mount /var/lib/mysql and it all looks good.
I've repeated the operation a number of times, writing files and checking they appear on the secondary after i swap
the nodes around.
Please note that MySQL is not running and therefore unless I manually create/update a file in /var/lib/mysql there's
really nothing happening.

I'm trying to figure out why there's a difference in the disk throughput between DRBD and non-DRBD when only the primary
node is up and running.

Scientific Linux is installed on /dev/sda (all Ext4) and DRBD is on /dev/sdb (Ext4 too).

In /var/lib/mysql (/dev/drbd0 on /dev/sdb)
[root at server1 ~]# dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 20.9517 s, 205 MB/s

In /home/userxxx (/dev/sda)
[root at server1 ~]# dd if=/dev/zero of=/home/userxxx/disk-test.xxx bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 10.2525 s, 419 MB/s

No replication over the 1Gb/s crossover cable is taking place since the secondary node is down yet there's x2 lower
disk performance.

I've tried to add:
  disk {
    disk-barrier no;
    disk-flushes no;
    md-flushes no;
  }
to the config but it didn't seem to change anything.

Am I missing something here?
On another note is 8.4.1 the right version to use?





Scientific Linux 6.1 (64 bits)
==============================
Kernel 2.6.32-220.4.1.el6.x86_64 SMP Thu Dec 22 11:15:52 CST 2011 x86_64

DRBD 8.4.1
==========
# cd /usr/src
# wget http://oss.linbit.com/drbd/8.4/drbd-8.4.1.tar.gz
# tar -zxvf drbd-8.4.1.tar.gz
# cd drbd-8.4.1/
# ./configure
# make rpm
# make km-rpm

Resulting in:
/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-km-2.6.32_220.2.1.el6.x86_64-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.1-1.el6.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.1-1.el6.x86_64.rpm

Installed all packages:
# rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm


DRBD config files:
==================
# cat /etc/drbd.conf 
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";


# cat /etc/drbd.d/global_common.conf 
global {
	usage-count yes;
	# minor-count dialog-refresh disable-ip-verification
}

common {
	handlers {
		pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
	}

	startup {
		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
	}

	options {
		# cpu-mask on-no-data-accessible
	}

	disk {
		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
		# disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
	}

	net {
		# protocol timeout max-epoch-size max-buffers unplug-watermark
		# connect-int ping-int sndbuf-size rcvbuf-size ko-count
		# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
		# after-sb-1pri after-sb-2pri always-asbp rr-conflict
		# ping-timeout data-integrity-alg tcp-cork on-congestion
		# congestion-fill congestion-extents csums-alg verify-alg
		# use-rle
	}
}


# cat /etc/drbd.d/mysql.res 
resource mysql {
  startup {
    wfc-timeout 3;
    degr-wfc-timeout 2;
    outdated-wfc-timeout 1;
  }
  net {
    protocol C;
    verify-alg sha1;
    csums-alg sha1;
    data-integrity-alg sha1;
    cram-hmac-alg sha1;
    shared-secret "mysecret123";
  }
  on server1 {
    device    /dev/drbd0;
    disk      /dev/sdb;
    address   192.168.111.10:7789;
    meta-disk internal;
  }
  on server2 {
    device    /dev/drbd0;
    disk      /dev/sdb;
    address   192.168.111.11:7789;
    meta-disk internal;
  }
}

192.168.111.10 and 192.168.111.11 on both ends of a CAT6 crossover cable.


Filesystem on DRBD:
===================
Added this line to /etc/fstab:
/dev/drbd0              /var/lib/mysql                   ext4    rw,suid,dev,exec,noauto,nouser,async        0 0
 
# mkfs.ext4 /dev/drbd0
# echo "DRBD Device" >/var/lib/mysql/samplefile.txt


Thx for any help.
Fred





More information about the drbd-user mailing list