Hi all,<br><br>I have encountered a DRBD write performance bottleneck issue. <br><br>According to DRBD specification "DRBD then reduces that throughput maximum by
its additional throughput overhead, which can be expected to be<span style="background-color: rgb(255, 255, 0);">
less than 3 percent</span>."<br><br>My current test environment is:<br><br><span style="background-color: rgb(255, 0, 0);">(1)</span> Hard-drive: 300G SSD <span style="background-color: rgb(255, 255, 51);">with 8 partitions on it, each of which has a DRBD device create on top it.</span> I use dd utility to test its performance: <span style="background-color: rgb(255, 255, 0);">97 MB/s with 4k block size.</span><br>
<br><br>
<span style="color: rgb(255, 0, 0);"><span style="color: rgb(0, 0, 0); background-color: rgb(255, 0, 0);">(2)</span> </span>netowork: dedicated <span style="background-color: rgb(255, 255, 0);">10G ethernet card</span> for data replication:<br>
ethtool eth2<br>Settings for eth2:<br>...<br> Speed: 10000Mb/s<br>...<br><br><span style="color: rgb(51, 51, 51);"><span style="background-color: rgb(0, 0, 0);"><span style="background-color: rgb(255, 0, 0);">(3</span>)</span> </span>DRBD configuration: (Here is one of them). <br>
<br> on Server1 {<br>
device /dev/drbd3 minor 3;<br> disk /dev/fioa3;<br> address ipv4 <a href="http://192.168.202.107:7793" target="_blank">192.168.202.107:7793</a>;<br> meta-disk internal;<br>
}<br> on NSS_108 {<br> device /dev/drbd3 minor 3;<br> disk /dev/fioa3;<br> address ipv4 <a href="http://192.168.202.108:7793" target="_blank">192.168.202.108:7793</a>;<br>
meta-disk internal;<br>
}<br> net {<br> allow-two-primaries;<br> after-sb-0pri discard-zero-changes;<br> after-sb-1pri consensus;<br> after-sb-2pri call-pri-lost-after-sb;<br> rr-conflict disconnect;<br>
max-buffers 4000;<br> max-epoch-size 16000;<br> unplug-watermark 4000;<br> sndbuf-size 2M;<br> data-integrity-alg crc32c;<br> }<br> syncer {<br> rate 300M;<br>
csums-alg md5;<br> verify-alg crc32c;<br> al-extents 3800;<br> cpu-mask 2;<br> }<br>}<br><br><span style="color: rgb(255, 0, 0);"><span style="background-color: rgb(255, 0, 0); color: rgb(51, 51, 51);">(4)</span> Test result:</span><br style="color: rgb(255, 0, 0);">
<br style="color: rgb(255, 0, 0);"><span style="color: rgb(255, 0, 0);">I have a simple script which use multiple instance of dd to their corresponding DRBD device</span><br style="color: rgb(255, 0, 0);"><span style="color: rgb(255, 0, 0);">
<br>dd if=/dev/zero of=/dev/drbd1 bs=4k count=10000 oflag=direct &</span><br style="color: rgb(255, 0, 0);"><span style="color: rgb(255, 0, 0);">....</span><br style="color: rgb(255, 0, 0);"><br style="color: rgb(255, 0, 0);">
<span style="color: rgb(255, 0, 0);">For one device, I got roughly 8M/s. As the test goes, I increase the number of device to see if it helps the performance. Unfortunately, as the number of device grows, performance seems to be distributed on each of the device with the total add up to 10M/s.</span><br>
<br>Can somebody give me a hint on what was going wrong?<br><br>Many Thanks.<br>Ben <br><br><br clear="all">Commit yourself to constant self-improvement<br>