<font color="black" size="2"><font color="black" size="2">Hi all, <br> <br>I have a two node DRBD cluster with Primary/Primary configuration. The <br>underling hard-drive that maps to drbd device is a home-made RAM disk that <br>
present itself as SCSI device(not the Linux /dev/ram# device). I tested the <br>read throughput via dd command, it gets at around 1.6G/s with 512k block <br>size which is faster than SSD. In terms of write, I first test it through a <br>
1G Ethernet. Surprisingly, it gives me 110M/s which really exceed my <br>expectation. Later on, I try to boost the write throughput up by adding 10G <br>Ethernet card dedication connection. However, no matter how I fine tune <br>
those parameters in drbd.conf, the best write throughput I can get stays at <br>around 320M/s. Then I dumped out TCP package to see if there is too much <br>overhead in protocol handshaking, in fact, I don`t see too many small <br>
package ticking around. Has anybody countered the similar issue? <br> <br> <br>The following is my configuration. <br> <br>[root [at] NSS-SM-3 etc]# drbdadm dump <br><font color="#660066"># /etc/drbd.conf <br></font>common { <br>
protocol C; <br>} <br> <br><font color="#660066"># resource mirror on NSS-SM-34: not ignored, not stacked <br></font>resource mirror { <br> on NSS-SM-33 { <br> device /dev/drbd1 minor 1; <br>
disk /dev/sdp; <br> address ipv4 <a href="http://192.168.3.33:7790">192.168.3.33:7790</a>; <br> meta-disk internal; <br> } <br> on NSS-SM-34 { <br> device /dev/drbd1 minor 1; <br>
disk /dev/sdp; <br> address ipv4 <a href="http://192.168.3.34:7790">192.168.3.34:7790</a>; <br> meta-disk internal; <br> } <br> net { <br> allow-two-primaries; <br>
after-sb-0pri discard-zero-changes; <br> after-sb-1pri consensus; <br> after-sb-2pri call-pri-lost-after-sb; <br> max-buffers 8192; <br> sndbuf-size 0; <br> } <br>
syncer { <br> rate 300M; <br> csums-alg md5; <br> al-extents 3800; <br> } <br> startup { <br> become-primary-on both; <br> } <br>} <br> <br>[root [at] NSS-SM-3 etc]# drbdsetup /dev/drbd1 show <br>
disk { <br> size 0s _is_default; # bytes <br> on-io-error pass_on _is_default; <br> fencing dont-care _is_default; <br> max-bio-bvecs 0 _is_default; <br>
} <br>net { <br> timeout 60 _is_default; # 1/10 seconds <br> max-epoch-size 2048 _is_default; <br> max-buffers 8192; <br> unplug-watermark 128 _is_default; <br>
connect-int 10 _is_default; # seconds <br> ping-int 10 _is_default; # seconds <br> sndbuf-size 0; # bytes <br> rcvbuf-size 131070 _is_default; # bytes <br>
ko-count 0 _is_default; <br> allow-two-primaries; <br> after-sb-0pri discard-zero-changes; <br> after-sb-1pri consensus; <br> after-sb-2pri call-pri-lost-after-sb; <br>
rr-conflict disconnect _is_default; <br> ping-timeout 5 _is_default; # 1/10 seconds <br>} <br>syncer { <br> rate 307200k; # bytes/second <br> after -1 _is_default; <br>
al-extents 3800; <br> csums-alg "md5"; <br>} <br>protocol C; <br>_this_host { <br> device minor 1; <br> disk "/dev/sdp"; <br>
meta-disk internal; <br> address ipv4 <a href="http://192.168.3.34:7790">192.168.3.34:7790</a>; <br>} <br>_remote_host { <br> address ipv4 <a href="http://192.168.3.33:7790">192.168.3.33:7790</a>; <br>
} <br> <br> <br>10G Ethernet test: <br> <br>[root [at] NSS-SM-3 etc]# isttcp -t -l 65536 -n 20480 192.168.3.34 <br>isttcp-t: buflen=65536, nbuf=20480, align=16384/0, port=5001 tcp -> <br>192.168.3.34 <br>isttcp-t: socket <br>
isttcp-t: nodelay <br>isttcp-t: connect <br>isttcp-t: 1342177280 bytes in 1.19 real seconds = 1105582.49 KB/sec +++ <br>isttcp-t: 20480 I/O calls, msec/call = 0.06, calls/sec = 17274.73 <br>isttcp-t: 0.0user 0.5sys 0:01real 43% 0i+0d 0maxrss 0+16pf 681+2csw <br>
<br>sync-write throughput: <br>[root [at] NSS-SM-3 etc]# dd if=/dev/zero of=/dev/drbd1 bs=512k oflag=direct <br>dd: writing `/dev/drbd1': No space left on device <br>4000+0 records in <br>3999+0 records out <br>2097049600 bytes (2.1 GB) copied, 7.36803 seconds, 285 MB/s <br>
<br>Turn off Ethernet interface on the counterpart machine and test write <br>WITHOUT going though network. <br>[root [at] NSS-SM-3 etc]# dd if=/dev/zero of=/dev/drbd1 bs=512k oflag=direct <br>dd: writing `/dev/drbd1': No space left on device <br>
4000+0 records in <br>3999+0 records out <br>2097049600 bytes (2.1 GB) copied, 0.937389 seconds, 2.2 GB/s </font></font><br><br><br>For small sync write, the performance is even worse,<br>
<p style="color: rgb(255, 255, 255);" class="MsoNormal"><span style="background: purple none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;"><span style=""> </span>[root@NSS-SM-33
tools]# dd if=/dev/zero of=/dev/drbd1<span style="">
</span>bs=4k </span><span style="background: lime none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;">oflag=direct </span><span style="background: purple none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;">count=20000</span></p>
<p style="color: rgb(255, 255, 255);" class="MsoNormal"><span style="background: purple none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;">20000+0 records in</span></p>
<p style="color: rgb(255, 255, 255);" class="MsoNormal"><span style="background: purple none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;">20000+0 records out</span></p>
<p class="MsoNormal"><span style="background: purple none repeat scroll 0% 0%; color: rgb(255, 255, 255); -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;">81920000 bytes (82 MB) copied, 5.08456
seconds, 16.1 MB/s</span><span style="color: white;"></span></p>
<br>Thanks for any help.<br><br clear="all">Commit yourself to constant self-improvement<br>