<div dir="ltr">Hi guys,<div><br></div><div>
<table cellspacing="0" border="0"><colgroup span="6" width="85"></colgroup><tbody></tbody></table></div><div>I am testing the performance of DRBD transport added in DRBD9. The test result seems problematic because</div><div>it's even worse than ISCSI transport. </div><div><br></div><div>Here's the test result of 4k random write on SSD disk with fio. </div><div><br></div><div> Raw disk DRBD8 Local DRBD9 Local DRBD Transport ISCSI Transport with DRBD9<br></div><div>IOPS 79756 <span style="text-align:-webkit-right">55549 </span><span style="text-align:-webkit-right">28624 </span><span style="text-align:-webkit-right">16343 </span><span style="text-align:-webkit-right">33435</span></div>Latency(us) 199.65 286.36 138.43 977.73 463.08<div><br></div><div>Please note the "DRBD8 Local" is tested with a tuned configuration while all DRBD9 tests are tested with the configurations generated by drbdmanage.</div><div><br></div><div>It's assumed that DRBD transport should give better performance than ISCSI transport. I don't know why it's even worse. Any suggestion to tune the DRBD transport? Thanks!</div><div><br></div><div><br></div><div>The following is my test setup and configurations:</div><div><br></div><div>Disk: SSD Samsung SSD 845DC EVO 480GB</div><div>NIC: 10Gb</div><div>OS Kernel: 3.10.0-327.10.1.el7</div><div>DRBD9 kernel module: 9.0.1-1 (GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e)</div><div>drbdmanage-0.92<br></div><div>drbd-utils-8.9.6<br></div><div><br></div><div>DRBD8 tuned configurations are copied from <a href="http://www.linbit.com/en/resources/technical-publications/101-performance/545-intel-sata-ssd-testing-with-drbd-2">http://www.linbit.com/en/resources/technical-publications/101-performance/545-intel-sata-ssd-testing-with-drbd-2</a></div><div><br></div><div>DRBD9 volume configuration (The nodes drbd1 and drbd2 provide the storage of volume vol2 and the drbd3 is the diskless client)</div><div><br></div><div><div>[root@drbd1 ~]# drbdsetup show vol2</div><div>resource vol2 {</div><div> _this_host {</div><div> node-id<span class="" style="white-space:pre"> </span>0;</div><div> volume 0 {</div><div> device<span class="" style="white-space:pre"> </span>minor 102;</div><div> disk<span class="" style="white-space:pre"> </span>"/dev/drbdpool/vol2_00";</div><div> meta-disk<span class="" style="white-space:pre"> </span>internal;</div><div> disk {</div><div> size <span class="" style="white-space:pre"> </span>39062500s; # bytes</div><div> disk-flushes <span class="" style="white-space:pre"> </span>no;</div><div> md-flushes <span class="" style="white-space:pre"> </span>no;</div><div> al-extents <span class="" style="white-space:pre"> </span>6433;</div><div> al-updates <span class="" style="white-space:pre"> </span>no;</div><div> read-balancing <span class="" style="white-space:pre"> </span>least-pending;</div><div> }</div><div> }</div><div> }</div><div> connection {</div><div> _peer_node_id 1;</div><div> path {</div><div> _this_host ipv4 <a href="http://192.168.253.131:7001">192.168.253.131:7001</a>;</div><div> _remote_host ipv4 <a href="http://192.168.253.132:7001">192.168.253.132:7001</a>;</div><div> }</div><div> net {</div><div> max-epoch-size <span class="" style="white-space:pre"> </span>20000;</div><div> sndbuf-size <span class="" style="white-space:pre"> </span>524288; # bytes</div><div> cram-hmac-alg <span class="" style="white-space:pre"> </span>"sha1";</div><div> shared-secret <span class="" style="white-space:pre"> </span>"q9Ku9G/Z/fhG1b3aemcD";</div><div> verify-alg <span class="" style="white-space:pre"> </span>"sha1";</div><div> max-buffers <span class="" style="white-space:pre"> </span>81920;</div><div> _name <span class="" style="white-space:pre"> </span>"drbd2";</div><div> }</div><div> volume 0 {</div><div> disk {</div><div> resync-rate <span class="" style="white-space:pre"> </span>409600k; # bytes/second</div><div> c-plan-ahead <span class="" style="white-space:pre"> </span>10; # 1/10 seconds</div><div> c-fill-target <span class="" style="white-space:pre"> </span>88s; # bytes</div><div> c-max-rate <span class="" style="white-space:pre"> </span>614400k; # bytes/second</div><div> c-min-rate <span class="" style="white-space:pre"> </span>10240k; # bytes/second</div><div> }</div><div> }</div><div> }</div><div>}</div></div><div><br></div><div><br></div><div><div>[root@drbd3 ~]# drbdsetup show vol2</div><div>resource vol2 {</div><div> _this_host {</div><div> node-id<span class="" style="white-space:pre"> </span>2;</div><div> volume 0 {</div><div> device<span class="" style="white-space:pre"> </span>minor 102;</div><div> }</div><div> }</div><div> connection {</div><div> _peer_node_id 0;</div><div> path {</div><div> _this_host ipv4 <a href="http://192.168.253.133:7001">192.168.253.133:7001</a>;</div><div> _remote_host ipv4 <a href="http://192.168.253.131:7001">192.168.253.131:7001</a>;</div><div> }</div><div> net {</div><div> cram-hmac-alg <span class="" style="white-space:pre"> </span>"sha1";</div><div> shared-secret <span class="" style="white-space:pre"> </span>"q9Ku9G/Z/fhG1b3aemcD";</div><div> _name <span class="" style="white-space:pre"> </span>"drbd1";</div><div> }</div><div> }</div><div> connection {</div><div> _peer_node_id 1;</div><div> path {</div><div> _this_host ipv4 <a href="http://192.168.253.133:7001">192.168.253.133:7001</a>;</div><div> _remote_host ipv4 <a href="http://192.168.253.132:7001">192.168.253.132:7001</a>;</div><div> }</div><div> net {</div><div> cram-hmac-alg <span class="" style="white-space:pre"> </span>"sha1";</div><div> shared-secret <span class="" style="white-space:pre"> </span>"q9Ku9G/Z/fhG1b3aemcD";</div><div> _name <span class="" style="white-space:pre"> </span>"drbd2";</div><div> }</div><div> }</div><div>}</div></div><div><br></div><div>The iscsi target is exported on drbd2 via LIO. I switched the primary from drbd3 to drbd2 when testing the iscsi transport. </div><div><br></div><div><br></div><div><br></div><div>FIO test configuration</div><div><div>[global]</div><div>bs=4k</div><div>ioengine=libaio</div><div>iodepth=16</div><div>size=10g</div><div>direct=1</div><div>runtime=300</div><div>directory=/mnt/ssd1</div><div>filename=ssd.test.file</div><div><br></div><div>[rand-write]</div><div>rw=randwrite</div></div><div><br></div></div>