Two node setup for serving out NFS to vSphere.<br><br>timing test is : sync; time iozone -a -e -g 4096<br><br>/data/nfs is on LVM striped across 3 DRBD devices. EXT4. data=journal.<br>/var/tmp is local filesystem.<br><br>both are on same set of disk platters and controller. Raid 6. 64K blocksize.<br>
<br>FYI, the initial sync (5 TB) used a full gigabit of bandwidth without hesitation.<br><br>=== DRBD TIMES (both nodes up) ==============================<br>real 0m10.028s<br>user 0m0.116s<br>sys 0m1.538s<br><br>
real 0m10.045s<br>user 0m0.085s<br>sys 0m1.541s<br><br>real 0m9.990s<br>user 0m0.091s<br>sys 0m1.578s<br><br>real 0m9.970s<br>user 0m0.099s<br>sys 0m1.557s<br><br>real 0m9.960s<br>user 0m0.091s<br>
sys 0m1.499s<br><br>=== DRBD TIMES (2nd node down) ==============================<br>real 0m3.754s<br>user 0m0.093s<br>sys 0m1.070s<br><br>real 0m3.855s<br>user 0m0.079s<br>sys 0m1.064s<br><br>real 0m3.938s<br>
user 0m0.094s<br>sys 0m1.044s<br><br>real 0m3.809s<br>user 0m0.066s<br>sys 0m1.067s<br><br>real 0m3.863s<br>user 0m0.069s<br>sys 0m1.072s<br><br>=== LOCAL TIMES ==============================<br>real 0m1.770s<br>
user 0m0.070s<br>sys 0m0.977s<br><br>real 0m1.809s<br>user 0m0.085s<br>sys 0m0.974s<br><br>real 0m1.737s<br>user 0m0.067s<br>sys 0m0.942s<br><br>real 0m2.007s<br>user 0m0.058s<br>sys 0m0.955s<br>
<br>real 0m1.808s<br>user 0m0.072s<br>sys 0m0.956s<br><br clear="all">=== After re-enabling 2nd node ==============================<br><br>root@san-node-2 ~]# cat /proc/drbd <br>version: 8.3.2 (api:88/proto:86-90)<br>
GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by mockbuild@v20z-x86-64.home.local, 2009-08-29 14:07:55<br> 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----<br> ns:0 nr:5180 dw:5180 dr:0 al:0 bm:24 lo:1 pe:1389 ua:0 ap:0 ep:1 wo:b oos:43964<br>
[==>.................] sync'ed: 16.7% (43964/49144)K<br> finish: 0:00:08 speed: 5,180 (5,180) K/sec<br> 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----<br> ns:0 nr:49044 dw:49044 dr:0 al:0 bm:44 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0<br>
2: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----<br> ns:0 nr:6896 dw:6896 dr:0 al:0 bm:28 lo:1 pe:1346 ua:0 ap:0 ep:1 wo:b oos:42436<br> [===>................] sync'ed: 23.1% (42436/49332)K<br>
finish: 0:00:05 speed: 6,896 (6,896) K/sec<br><br>=== ifconfig ==============================<br><br>bond0:0 Link encap:Ethernet HWaddr 00:1B:21:26:B8:18 <br> inet addr:192.168.XXX.XX0 Bcast:192.168.XXX.255 Mask:255.255.255.0<br>
UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1<br><br><br>=== rpm -qa|grep drbd ==============================<br>kmod-drbd83-8.3.2-6.el5_3<br>drbd83-8.3.2-6.el5_3<br><br>=== uname -a ==============================<br>
Linux <a href="http://san-node-1.ewc.edu">san-node-1.ewc.edu</a> 2.6.18-194.3.1.el5 #1 SMP Thu May 13 13:08:30 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux<br><br>=== cat /etc/issue ==============================<br>CentOS release 5.5 (Final)<br>
Kernel \r on an \m<br><br>=== lspci ==============================<br>01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)<br><br>=== box to box copy test ==============================<br>
dd if=/dev/sda of=testfile bs=1M count=1024<br>1024+0 records in<br>1024+0 records out<br>1073741824 bytes (1.1 GB) copied, 1.94192 seconds, 553 MB/s<br><br>scp testfile root@san-node-2:/var/tmp/testfile<br>testfile 100% 1024MB 46.6MB/s 00:22 <br>
<br>=== /etc/drbd.conf ==============================<br><br>global {<br> usage-count yes;<br>}<br>common {<br> protocol C;<br> syncer { rate 700000K; }<br> net {<br> sndbuf-size 0;<br> after-sb-0pri discard-least-changes;<br>
after-sb-1pri discard-secondary;<br> }<br> startup {<br> degr-wfc-timeout 30000;<br> }<br> disk {<br> no-disk-flushes;<br> no-md-flushes;<br> }<br> <br>}<br>resource sdb {<br>
on <a href="http://san-node-1.ewc.edu">san-node-1.ewc.edu</a> {<br> device /dev/drbd0;<br> disk /dev/sdb;<br> address 192.168.XXX.XX1:7789;<br> meta-disk internal;<br> }<br> on <a href="http://san-node-2.ewc.edu">san-node-2.ewc.edu</a> {<br>
device /dev/drbd0;<br> disk /dev/sdb;<br> address 192.168.XXX.XX2:7789;<br> meta-disk internal;<br> }<br>}<br>resource sdc {<br> on <a href="http://san-node-1.ewc.edu">san-node-1.ewc.edu</a> {<br>
device /dev/drbd1;<br> disk /dev/sdc;<br> address 192.168.XXX.XX1:7790;<br> meta-disk internal;<br> }<br> on <a href="http://san-node-2.ewc.edu">san-node-2.ewc.edu</a> {<br> device /dev/drbd1;<br>
disk /dev/sdc;<br> address 192.168.XXX.XX2:7790;<br> meta-disk internal;<br> }<br>}<br>resource sdd {<br> on <a href="http://san-node-1.ewc.edu">san-node-1.ewc.edu</a> {<br> device /dev/drbd2;<br>
disk /dev/sdd;<br> address 192.168.XXX.XX1:7791;<br> meta-disk internal;<br> }<br> on <a href="http://san-node-2.ewc.edu">san-node-2.ewc.edu</a> {<br> device /dev/drbd2;<br> disk /dev/sdd;<br>
address 192.168.XXX.XX2:7791;<br> meta-disk internal;<br> }<br>}<br><br>-- <br>Michael Joyner/System Administrator<br>Edward Waters College<br>1658 Kings Road<br>Jacksonville, FL 32209<br>(V) 904-470-8170/(F) 904-470-8170<br>