<span style="color: rgb(0, 0, 153);">Hi all,</span><br style="color: rgb(0, 0, 153);"><br style="color: rgb(0, 0, 153);"><span style="color: rgb(0, 0, 153);">I have a two node DRBD cluster with Primary/Primary configuration. The underling hard-drive that maps to drbd device is a home-made RAM disk that present itself as SCSI device(not the Linux /dev/ram# device). I tested the read throughput via dd command, it gets at around 1.6G/s with 512k block size which is faster than SSD. In terms of write, I first test it through a 1G Ethernet. Surprisingly, it gives me 110M/s which really exceed my expectation. Later on, I try to boost the write throughput up by adding 10G Ethernet card dedication connection. However, no matter how I fine tune those parameters in drbd.conf, the best write throughput I can get stays at around 320M/s. Then I dumped out TCP package to see if there is too much overhead in protocol handshaking, in fact, I don`t see too many small package ticking around. Has anybody countered the similar issue?</span> <br style="color: rgb(0, 0, 153);">
<br style="color: rgb(0, 0, 153);"><br style="color: rgb(0, 0, 153);"><span style="color: rgb(0, 0, 153);">The following is my configuration.</span><br><br><span style="background-color: rgb(255, 255, 51);">[root@NSS-SM-34 etc]# drbdadm dump</span><br>
# /etc/drbd.conf<br>common {<br> protocol C;<br>}<br><br># resource mirror on NSS-SM-34: not ignored, not stacked<br>resource mirror {<br> on NSS-SM-33 {<br> device /dev/drbd1 minor 1;<br>
disk /dev/sdp;<br> address ipv4 <a href="http://192.168.3.33:7790">192.168.3.33:7790</a>;<br> meta-disk internal;<br> }<br> on NSS-SM-34 {<br> device /dev/drbd1 minor 1;<br>
disk /dev/sdp;<br> address ipv4 <a href="http://192.168.3.34:7790">192.168.3.34:7790</a>;<br> meta-disk internal;<br> }<br> net {<br> allow-two-primaries;<br> after-sb-0pri discard-zero-changes;<br>
after-sb-1pri consensus;<br> after-sb-2pri call-pri-lost-after-sb;<br> max-buffers 8192;<br> sndbuf-size 0;<br> }<br> syncer {<br> rate 300M;<br> csums-alg md5;<br>
al-extents 3800;<br> }<br> startup {<br> become-primary-on both;<br> }<br>}<br><br><span style="background-color: rgb(255, 255, 0);">[root@NSS-SM-34 etc]# drbdsetup /dev/drbd1 show</span><br>
disk {<br> size 0s _is_default; # bytes<br> on-io-error pass_on _is_default;<br> fencing dont-care _is_default;<br> max-bio-bvecs 0 _is_default;<br>
}<br>net {<br> timeout 60 _is_default; # 1/10 seconds<br> max-epoch-size 2048 _is_default;<br> max-buffers 8192;<br> unplug-watermark 128 _is_default;<br>
connect-int 10 _is_default; # seconds<br> ping-int 10 _is_default; # seconds<br> sndbuf-size 0; # bytes<br> rcvbuf-size 131070 _is_default; # bytes<br>
ko-count 0 _is_default;<br> allow-two-primaries;<br> after-sb-0pri discard-zero-changes;<br> after-sb-1pri consensus;<br> after-sb-2pri call-pri-lost-after-sb;<br>
rr-conflict disconnect _is_default;<br> ping-timeout 5 _is_default; # 1/10 seconds<br>}<br>syncer {<br> rate 307200k; # bytes/second<br> after -1 _is_default;<br>
al-extents 3800;<br> csums-alg "md5";<br>}<br>protocol C;<br>_this_host {<br> device minor 1;<br> disk "/dev/sdp";<br>
meta-disk internal;<br> address ipv4 <a href="http://192.168.3.34:7790">192.168.3.34:7790</a>;<br>}<br>_remote_host {<br> address ipv4 <a href="http://192.168.3.33:7790">192.168.3.33:7790</a>;<br>
}<br><br><br><span style="background-color: rgb(0, 204, 204);">10G Ethernet test:</span><br><br><span style="background-color: rgb(255, 255, 0);">[root@NSS-SM-33 etc]# isttcp -t -l 65536 -n 20480 192.168.3.34</span><br>isttcp-t: buflen=65536, nbuf=20480, align=16384/0, port=5001 tcp -> 192.168.3.34<br>
isttcp-t: socket<br>isttcp-t: nodelay<br>isttcp-t: connect<br>isttcp-t: 1342177280 bytes in 1.19 real seconds = 1105582.49 KB/sec +++<br>isttcp-t: 20480 I/O calls, msec/call = 0.06, calls/sec = 17274.73<br>isttcp-t: 0.0user 0.5sys 0:01real 43% 0i+0d 0maxrss 0+16pf 681+2csw<br>
<br><span style="color: rgb(0, 0, 0); background-color: rgb(51, 204, 255);">sync-write throughput:</span><br><span style="background-color: rgb(255, 255, 0);">[root@NSS-SM-34 etc]# dd if=/dev/zero of=/dev/drbd1 bs=512k oflag=direct</span><br>
dd: writing `/dev/drbd1': No space left on device<br>4000+0 records in<br>3999+0 records out<br>2097049600 bytes (2.1 GB) copied, 7.36803 seconds, 285 MB/s<br><br style="background-color: rgb(0, 204, 204);"><span style="background-color: rgb(0, 204, 204);">Turn off Ethernet interface on the counterpart machine and test write WITHOUT going though network.</span><br>
<span style="background-color: rgb(255, 255, 0);">[root@NSS-SM-34 etc]# dd if=/dev/zero of=/dev/drbd1 bs=512k oflag=direct</span><br>dd: writing `/dev/drbd1': No space left on device<br>4000+0 records in<br>3999+0 records out<br>
2097049600 bytes (2.1 GB) copied, 0.937389 seconds, 2.2 GB/s<br><br>Any comment is appreciated.<br><br>Thanks<br>Ben<br> <br clear="all">Commit yourself to constant self-improvement<br>