<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Greetings, <br>
<br>
I'm currently working on a new server that I plan to use for an
iSCSI SAN. Because of a hardware availability issue I have to
initially configure this as a single node DRBD system until I
complete the migration of data from another server configured with
similar hardware.<br>
<br>
In a nutshell, my problem is that I'm seeing horrible write
performance when writing to the DRBD device versus writing to the
underlying Raid array.<br>
<font color="black" face="Verdana,Arial,Helvetica" size="2"><font
color="black" face="Verdana,Arial,Helvetica" size="2"></font></font><br>
root@san01-a:~# dd if=/dev/zero of=/dev/drbd1 bs=1M count=10000
oflag=direct<br>
10485760000 bytes (10 GB) copied, 25.5897 s, 410 MB/s<br>
<br>
root@san01-a:~# dd if=/dev/zero of=/dev/sda2 bs=1M count=10000
oflag=direct<br>
10485760000 bytes (10 GB) copied, 7.07579 s, 1.5 GB/s<br>
<br>
Quick Hardware description:<br>
SuperMicro Mainboard w/ Xeon E5-2603<br>
16GB Ram<br>
Dual 120G SSD drives in a Linux Raid 1 for OS - Debian Wheezy<br>
Adaptec ASR7805 w/BBU - latest firmware (32033) w/ BBU<br>
22 Toshiba 1TB SAS drives (MG03SCA100) in a Raid 60 array w/ 128k
stripe<br>
Dual 10G links (Intel card) to the second server when it is ready<br>
<br>
drbdsetup show:<br>
root@san01-a:~# drbdsetup show<br>
resource meta {<br>
options {<br>
cpu-mask "ff";<br>
}<br>
net {<br>
max-epoch-size 20000;<br>
max-buffers 131072;<br>
unplug-watermark 131072;<br>
after-sb-0pri discard-least-changes;<br>
after-sb-1pri consensus;<br>
}<br>
_remote_host {<br>
address ipv4 1.1.200.2:7788;<br>
}<br>
_this_host {<br>
address ipv4 1.1.200.1:7788;<br>
volume 0 {<br>
device minor 0;<br>
disk "/dev/sda1";<br>
meta-disk internal;<br>
disk {<br>
disk-flushes no;<br>
md-flushes no;<br>
al-extents 3389;<br>
}<br>
}<br>
}<br>
}<br>
<br>
resource data {<br>
options {<br>
cpu-mask "ff";<br>
}<br>
net {<br>
max-epoch-size 20000;<br>
max-buffers 131072;<br>
unplug-watermark 131072;<br>
after-sb-0pri discard-least-changes;<br>
after-sb-1pri consensus;<br>
}<br>
_remote_host {<br>
address ipv4 1.1.200.2:7789;<br>
}<br>
_this_host {<br>
address ipv4 1.1.200.1:7789;<br>
volume 0 {<br>
device minor 1;<br>
disk "/dev/sda2";<br>
meta-disk internal;<br>
disk {<br>
disk-flushes no;<br>
md-flushes no;<br>
al-extents 3389;<br>
}<br>
}<br>
}<br>
}<br>
=====================<br>
<br>
I've tried this with various settings and about the only setting
that really makes a difference is cpu-mask ff; The others
(md-flushes, al-extents, disk-flushes, max-buffers) can all be
removed with no real change in performance.<br>
<br>
DRBD and drbd-utils were built from the git code<br>
version: 8.4.5 (api:1/proto:86-101)<br>
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by
root@san01, 2015-02-05 11:09:59<br>
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s<br>
ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d
oos:511948<br>
1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s<br>
ns:0 nr:0 dw:10240000 dr:996 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
wo:d oos:17570968856<br>
<br>
<br>
root@san01-a:~# drbdadm -v<br>
Version: 8.9.2rc2 (api:1)<br>
GIT-hash: faeb645ecbf334347e0512b4fa2d7549543b5b50 build by
root@san01, 2015-02-10 13:41:16<br>
<br>
root@san01-a:~# uname -a<br>
Linux san01 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64
GNU/Linux<br>
<br>
<br>
The strangest thing is that I built a system with all the same
hardware 8 months ago and I was seeing 700MB/s writes. The
difference was that I had both nodes to work on while creating the
DRBD. This time I only have 1 node to start with.<br>
<br>
<div class="moz-signature">-- <br>
Brian<br>
</div>
</body>
</html>