<div dir="ltr">Hi guys,<div><br></div><div>
        
        
        
        



<table cellspacing="0" border="0"><colgroup span="6" width="85"></colgroup><tbody></tbody></table></div><div>I am testing the performance of DRBD transport added in DRBD9.  The test result seems problematic because</div><div>it&#39;s even worse than ISCSI transport. </div><div><br></div><div>Here&#39;s the test result of 4k random write on SSD disk with fio. </div><div><br></div><div>                   Raw disk    DRBD8 Local    DRBD9 Local    DRBD Transport    ISCSI Transport with DRBD9<br></div><div>IOPS              79756           <span style="text-align:-webkit-right">55549              </span><span style="text-align:-webkit-right">28624               </span><span style="text-align:-webkit-right">16343                    </span><span style="text-align:-webkit-right">33435</span></div>Latency(us)     199.65          286.36            138.43               977.73                   463.08<div><br></div><div>Please note the &quot;DRBD8 Local&quot; is tested with a tuned configuration while all DRBD9 tests are tested with the configurations generated by drbdmanage.</div><div><br></div><div>It&#39;s assumed that DRBD transport should give better performance than ISCSI transport.  I don&#39;t know why it&#39;s even worse. Any suggestion to tune the DRBD transport?   Thanks!</div><div><br></div><div><br></div><div>The following is my test setup and configurations:</div><div><br></div><div>Disk: SSD Samsung SSD 845DC EVO 480GB</div><div>NIC:  10Gb</div><div>OS Kernel: 3.10.0-327.10.1.el7</div><div>DRBD9 kernel module: 9.0.1-1 (GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e)</div><div>drbdmanage-0.92<br></div><div>drbd-utils-8.9.6<br></div><div><br></div><div>DRBD8 tuned configurations are copied from <a href="http://www.linbit.com/en/resources/technical-publications/101-performance/545-intel-sata-ssd-testing-with-drbd-2">http://www.linbit.com/en/resources/technical-publications/101-performance/545-intel-sata-ssd-testing-with-drbd-2</a></div><div><br></div><div>DRBD9 volume configuration (The nodes drbd1 and drbd2 provide the storage of volume vol2 and the drbd3 is the diskless client)</div><div><br></div><div><div>[root@drbd1 ~]# drbdsetup show vol2</div><div>resource vol2 {</div><div>    _this_host {</div><div>        node-id<span class="" style="white-space:pre">                        </span>0;</div><div>        volume 0 {</div><div>            device<span class="" style="white-space:pre">                        </span>minor 102;</div><div>            disk<span class="" style="white-space:pre">                        </span>&quot;/dev/drbdpool/vol2_00&quot;;</div><div>            meta-disk<span class="" style="white-space:pre">                        </span>internal;</div><div>            disk {</div><div>                size            <span class="" style="white-space:pre">        </span>39062500s; # bytes</div><div>                disk-flushes    <span class="" style="white-space:pre">        </span>no;</div><div>                md-flushes      <span class="" style="white-space:pre">        </span>no;</div><div>                al-extents      <span class="" style="white-space:pre">        </span>6433;</div><div>                al-updates      <span class="" style="white-space:pre">        </span>no;</div><div>                read-balancing  <span class="" style="white-space:pre">        </span>least-pending;</div><div>            }</div><div>        }</div><div>    }</div><div>    connection {</div><div>        _peer_node_id 1;</div><div>        path {</div><div>            _this_host ipv4 <a href="http://192.168.253.131:7001">192.168.253.131:7001</a>;</div><div>            _remote_host ipv4 <a href="http://192.168.253.132:7001">192.168.253.132:7001</a>;</div><div>        }</div><div>        net {</div><div>            max-epoch-size  <span class="" style="white-space:pre">        </span>20000;</div><div>            sndbuf-size     <span class="" style="white-space:pre">        </span>524288; # bytes</div><div>            cram-hmac-alg   <span class="" style="white-space:pre">        </span>&quot;sha1&quot;;</div><div>            shared-secret   <span class="" style="white-space:pre">        </span>&quot;q9Ku9G/Z/fhG1b3aemcD&quot;;</div><div>            verify-alg      <span class="" style="white-space:pre">        </span>&quot;sha1&quot;;</div><div>            max-buffers     <span class="" style="white-space:pre">        </span>81920;</div><div>            _name           <span class="" style="white-space:pre">        </span>&quot;drbd2&quot;;</div><div>        }</div><div>        volume 0 {</div><div>            disk {</div><div>                resync-rate     <span class="" style="white-space:pre">        </span>409600k; # bytes/second</div><div>                c-plan-ahead    <span class="" style="white-space:pre">        </span>10; # 1/10 seconds</div><div>                c-fill-target   <span class="" style="white-space:pre">        </span>88s; # bytes</div><div>                c-max-rate      <span class="" style="white-space:pre">        </span>614400k; # bytes/second</div><div>                c-min-rate      <span class="" style="white-space:pre">        </span>10240k; # bytes/second</div><div>            }</div><div>        }</div><div>    }</div><div>}</div></div><div><br></div><div><br></div><div><div>[root@drbd3 ~]# drbdsetup show vol2</div><div>resource vol2 {</div><div>    _this_host {</div><div>        node-id<span class="" style="white-space:pre">                        </span>2;</div><div>        volume 0 {</div><div>            device<span class="" style="white-space:pre">                        </span>minor 102;</div><div>        }</div><div>    }</div><div>    connection {</div><div>        _peer_node_id 0;</div><div>        path {</div><div>            _this_host ipv4 <a href="http://192.168.253.133:7001">192.168.253.133:7001</a>;</div><div>            _remote_host ipv4 <a href="http://192.168.253.131:7001">192.168.253.131:7001</a>;</div><div>        }</div><div>        net {</div><div>            cram-hmac-alg   <span class="" style="white-space:pre">        </span>&quot;sha1&quot;;</div><div>            shared-secret   <span class="" style="white-space:pre">        </span>&quot;q9Ku9G/Z/fhG1b3aemcD&quot;;</div><div>            _name           <span class="" style="white-space:pre">        </span>&quot;drbd1&quot;;</div><div>        }</div><div>    }</div><div>    connection {</div><div>        _peer_node_id 1;</div><div>        path {</div><div>            _this_host ipv4 <a href="http://192.168.253.133:7001">192.168.253.133:7001</a>;</div><div>            _remote_host ipv4 <a href="http://192.168.253.132:7001">192.168.253.132:7001</a>;</div><div>        }</div><div>        net {</div><div>            cram-hmac-alg   <span class="" style="white-space:pre">        </span>&quot;sha1&quot;;</div><div>            shared-secret   <span class="" style="white-space:pre">        </span>&quot;q9Ku9G/Z/fhG1b3aemcD&quot;;</div><div>            _name           <span class="" style="white-space:pre">        </span>&quot;drbd2&quot;;</div><div>        }</div><div>    }</div><div>}</div></div><div><br></div><div>The iscsi target is exported on drbd2 via LIO.  I switched the primary from drbd3 to drbd2 when testing the iscsi transport. </div><div><br></div><div><br></div><div><br></div><div>FIO test configuration</div><div><div>[global]</div><div>bs=4k</div><div>ioengine=libaio</div><div>iodepth=16</div><div>size=10g</div><div>direct=1</div><div>runtime=300</div><div>directory=/mnt/ssd1</div><div>filename=ssd.test.file</div><div><br></div><div>[rand-write]</div><div>rw=randwrite</div></div><div><br></div></div>