Hi.<div><br></div><div>So, I have a 2-node setup ... it functions fine, but I recently (last night/this morning) upgraded to the current Xen source, Xen patches to linux 2.6.18, and DRBD 8.3.0.</div><div><br></div><div>It *functions* fine, so far, however when I was re-syncing my data mirrors on both hosts, I noticed something really disturbing. The host that only had Secondary roles on it was re-syncing the disk at a substantially slower rate than it should have been. The host with Primary roles may have been performance impacted as well, but I didn't take metrics at the time.</div>
<div><br></div><div>When I went to pass out at 5am, both nodes were giving me an ETA of 4 hours to sync. When I checked on them ~7h later, the host with Primary roles was complete and the host with Secondary roles was not! I grabbed some data points:</div>
<div><br></div><div><div>md1 : active raid1 sdb3[2] sda3[0] </div><div> 727688192 blocks [2/1] [U_] </div><div> [===================>.] recovery = 98.9% (720040384/727688192) finish=5.1 </div><div> min speed=24888K/sec </div>
<div><br></div><div>Just for fun, I shut down the DRBD devices and module (/etc/init.d/drbd stop) and checked again:<br></div><div><br></div></div><div><div>md1 : active raid1 sdb3[2] sda3[0] </div><div> 727688192 blocks [2/1] [U_] </div>
<div> [===================>.] recovery = 98.2% (715188224/727688192) finish=4.8 </div><div> min speed=43250K/sec <br></div><div><br></div><div>That's a pretty significant difference.</div><div><br></div><div>
I turned on the DRBD module and devices and observed the sync performance drop back to the original values. So, I turned down the DRBD module and devices, let the sync complete and have restarted it (poor hard drives). Here's my starting point:</div>
<div><br></div><div><div>md1 : active raid1 sdb3[2] sda3[0] </div><div> 727688192 blocks [2/1] [U_] </div><div> [>....................] recovery = 1.2% (8791680/727688192) finish=131.6 </div><div> min speed=91042K/sec </div>
<div><br></div><div>(side note, it's interesting to see the performance difference between the inside and outside tracks of the physical disk, eh?)</div><div><br></div><div>Starting the DRBD module:</div><div><br></div>
<div><div># modprobe drbd</div><div><br></div><div><div>md1 : active raid1 sdb3[2] sda3[0]</div><div> 727688192 blocks [2/1] [U_]</div><div> [=>...................] recovery = 9.0% (65796864/727688192) finish=122.</div>
<div>8min speed=89828K/sec</div><div><br></div><div>I see no significant change to re-sync speed. As it progresses through the disk, it will slow a bit, so the <2M/s difference is not surprising.</div><div><br></div><div>
Starting 1 device (Secondary role):</div><div><br></div><div><div># drbdadm up ftp01-root</div><div><br></div><div><div># cat /proc/drbd</div><div>version: 8.3.0 (api:88/proto:86-89)</div><div>GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@build-hardy-x64, 2008-12-22 04:14:22</div>
<div><br></div><div> 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r---</div><div> ns:0 nr:2346 dw:2346 dr:0 al:0 bm:10 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</div><div><br></div></div><div><div>md1 : active raid1 sdb3[2] sda3[0]</div>
<div> 727688192 blocks [2/1] [U_]</div><div> [==>..................] recovery = 10.0% (72998592/727688192) finish=143.</div><div>3min speed=76120K/sec</div><div><br></div><div>That's a bit more significant ... loss of >12.7M/s in sync performance. In observing the re-sync speed, I see it fluxuating between 83M/s and 62M/s. </div>
<div><br></div><div>From /proc/drbd, we can see the device is not very active, however:</div><div><br></div><div><div> 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r---</div><div> ns:0 nr:2735 dw:2735 dr:0 al:0 bm:10 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</div>
<div><br></div><div>Firing up additional devices in the Secondary role just continues to beat down the re-sync performance (down into the 40M/s range). Unconfiguring all of the devices brings the speed back up to the upper 80's (near full speed).</div>
<div><br></div><div>Snippits from the drbd.conf:</div><div><br></div><div><div>common {</div><div> syncer {</div><div> rate 70M;</div><div> verify-alg md5;</div><div> }</div><div><br></div><div> protocol C;</div>
<div><br></div><div> startup {</div><div> wfc-timeout 0; ## Infinite!</div><div> degr-wfc-timeout 120; ## 2 minutes.</div><div> }</div><div><br></div><div> disk {</div><div> on-io-error detach;</div>
<div> }</div><div><br></div><div> net {</div><div> allow-two-primaries;<br></div><div> }</div><div>}</div><div><br></div><div><div>######################################################################</div><div>resource ftp01-root {</div>
<div> device /dev/drbd2;</div><div> disk /dev/datavg/ftp01-root;</div><div> flexible-meta-disk internal;</div><div><br></div><div> on xen-33-18-02 {</div><div> address <a href="http://192.168.250.12:7702">192.168.250.12:7702</a>;</div>
<div> }</div><div><br></div><div> on xen-33-18-03 {</div><div> address <a href="http://192.168.250.13:7702">192.168.250.13:7702</a>;</div><div> }</div><div>}<br></div><div><br></div><div>All of the rest of the devices are similar.</div>
<div><br></div><div>Server configuration:</div><div><br></div><div> Linux: 2.6.18.8-xen-x64-20081222 #8 SMP Mon Dec 22 05:08:39 EST 2008 x86_64 GNU/Linux</div><div> Xen: Xen 3.3.1-rc4</div><div> DRBD: 8.3.0 (build against the above kernel tree)</div>
<div> /dev/md1 is a linux software mirror of 2 SATA2 drives (/dev/sda3 and /dev/sdb3) of about 694GB in size</div><div> LVM sits on top of the md mirror presenting a single "disk" and LVM is used to chop into LVs</div>
<div><br></div><div><div> ftp01-data datavg -wi-a- 140.00G</div><div> ftp01-root datavg -wi-a- 4.00G</div><div><br></div><div>Each LV is used by DRBD to present a "disk" device to the Xen guest:</div>
<div><br></div><div><div> root = '/dev/xvda1 ro'</div><div> disk = [</div><div> 'drbd:ftp01-root,xvda1,w',</div><div> 'drbd:ftp01-data,xvda2,w',</div>
<div> ]</div><div><br></div><div>Aside from these performance issues, and a pretty disturbing issue with resizing an LV/DRBD device/Filesystem with flexible-meta-disk: internal, everything has been running OK.</div>
<div><br></div><div>I do not suspect the upgrade to the Xen, linux, or DRBD source ... I've seen these performance issues in the past, but never been annoyed enough to capture them in detail.</div><div><br></div><div>
Any ideas what I can do to pep up the performance?</div><div><br></div><div>My ultimate goal is to use the new stacking feature in DRBD 8.3.0 to put a 3rd node at a remote (WAN) location ... any thing I should consider for that with my configuration as detailed above?</div>
<div><br></div><div>Thanks,</div><div>Sam</div></div></div></div></div></div></div></div></div></div></div></div>