<div dir="ltr">Hello,<div><br></div><div>I started looking at DRBD as a HA ISCSI target. I am experiencing very poor performance and decided to run some tests. My current setup is as follows:</div><div><br></div><div>Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GH</div>
<div>CentoS 6.5 - 2.6.32-431.17.1.el6.x86_64</div><div>drbd version: 8.3.16 (api:88/proto:86-97)<br></div><div>md RAID10 using 7200rpm drives</div><div><br></div><div>The 2 drbd nodes are synced using an intel 82579LM Gigabit card</div>
<div><br></div><div>I have created an logical drive using LVM and configured a couple drbd resources on top of that. drbd0 is my iscsi configuration file, which is shared between the 2 nodes and drbd1 is a 1.75TB ISCSI target.</div>
<div><br></div><div>I run heartbeat on the two nodes and expose a virtual IP to the ISCSI initiators.</div><div><br></div><div>Originally I was running ISCSI with write-cache off (for data integrity reasons) but have recently switched to write-cache on during testing (with little to no gain).</div>
<div><br></div><div>My major concern is the extremely high latency test results I got when when dd against drbd0 mounted on the primary node.</div><div><br></div><div>dd if=/dev/zero of=./testbin bs=512 count=1000 oflag=direct<br>
512000 bytes (512 kB) copied, 32.3254 s, 15.8 kB/s<br></div><div><br></div><div>I have pinged the second node as a very basic network latency test and get 0.209ms response time. I have also run the same test on both nodes with drbd disconnected (or on partitions not associated with drbd) and get typical results:</div>
<div><br></div><div>node1</div><div>dd if=/dev/zero of=./testbin bs=512 count=1000 oflag=direct<br></div><div>12000 bytes (512 kB) copied, 0.153541 s, 3.3 MB/s<br></div><div><br></div><div>node2</div><div>dd if=/dev/zero of=~/testbin bs=512 count=1000 oflag=direct<br>
</div><div>512000 bytes (512 kB) copied, 0.864994 s, 592 kB/s<br></div><div>512000 bytes (512 kB) copied, 0.328994 s, 1.6 MB/s<br></div><div><br></div><div>node2's latency (without drbd connected) is inconsistent but always falls between those two ranges.</div>
<div><br></div><div>These tests were run with no ISCSI targets exposed, no initiators connected, essentially on an idle system.</div><div><br></div><div>My question is why are my drbd connected latency tests showing results 35 to 100 times slower than my results when dbrd is not connected (or against partitions not backed by drbd)?</div>
<div><br></div><div>This seems to be the source of my horrible performance on the ISCSI targs (300~900 K/sec dd writes on the initiators) and very high iowait (35-75%) on mildly busy initiators.</div><div><br></div><div><br>
</div><div>Any advice pointers, etc. would be highly appreciated. I have already tried numerous performance tuning settings (suggested by the drbd manual). But I am open to any suggestion and will try anything again if it might solve my problem.</div>
<div><br></div><div>Here are the important bits of my current drbd.conf</div><div><br></div><div><div> net {</div><div> cram-hmac-alg sha1;</div><div> shared-secret "password";</div><div> after-sb-0pri disconnect;</div>
<div> after-sb-1pri disconnect;</div><div> after-sb-2pri disconnect;</div><div> rr-conflict disconnect;</div><div> max-buffers 8000;</div><div> max-epoch-size 8000;</div><div> sndbuf-size 0;</div>
<div> }</div><div><br></div><div> syncer {</div><div> rate 100M;</div><div> verify-alg sha1;</div><div> al-extents 3389;</div><div> }</div><div><br></div></div><div>I've played with the watermark setting and a few others and latency only seems to get worse or stay where it's at.</div>
<div><br></div><div><br></div><div>Thank you,<br>Bret</div>
</div>