<div dir="ltr"><div><div>Hi, <br><br></div> We have a drbd setup with two resources. Both primary and secondary systems have RAID6 devices, and are running <br><br>CentOS 6.5<br>2.6.32-431.20.3.el6.x86_64<br></div>drbd 8.3.16<br>
<div><br></div><div>The secondary is connected to the primary via a gigabit connection, we run regular iperf tests and we get speeds typically 600-900 Mbits/sec. <br><br>The secondary and primary are physically in different buildings but live on the same VLAN. <br>
<br><br></div><div> We run BackupPC on the primary devices, and use drbd to replicate the BackupPC archives to the remote site. During initial testing the machines were in the same room, and connected via a gigabit switch. Running backups against the primary were replicated, no apparent problems. <br>
<br></div><div> Problems arose when we moved the secondary to the remote site. The backups would run, but eventually some backup process would block - get stuck in an uninterruptible sleep state. <br><br></div><div> Eventually these blocks would be cleared, but sometimes it took a long time (>24 hours), even when we had stopped the backups altogether. (We were unable to stop the blocked processes, but no new backups were started during that time.). <br>
<br></div><div> We observed that the send-Q on the primary, and the recv-Q on the secondary, had a LOT of data in them:<br><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt" id="docs-internal-guid-f15604d2-f5b7-914e-6d88-7522e9e5ed7e">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[djsperka@primary ~]$ netstat -tnp</span></p>
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline" id="docs-internal-guid-f15604d2-f5ba-71fb-2705-6b60df80bf11">Proto Recv-Q Send-Q Local Address </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Foreign Address </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">State </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">PID/Program name <br>
</span></div><div><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">--- edit output --- </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">tcp </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 1669560 111.222.333.444:59203 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> 111</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">.222.333.555:7790 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">ESTABLISHED - <br>
</span></p><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span>---end edit---<br>
<br><br></div><div>The other connections on the drbd ports all have empty send-q and recv-q. Note the large send-q. <br><br></div><div>On the secondary, the situation was similar:<br><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline" id="docs-internal-guid-f15604d2-f5bc-e84a-e976-37af22fcc4ef">tcp 3773784 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 111.222.333.555:7790 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">111.222.333.444:59203 ESTABLISHED -</span><br>
</div><div><br></div><div>Note the large recv-q. <br><br><br></div><div> The blocked processes on the primary are these:<br><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt" id="docs-internal-guid-f15604d2-f5be-cf1d-602a-9300edbe8535">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[djsperka@primary ~]$ ps axl | awk '$10 ~ /^D/'</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 2261 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2 20 0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 down D</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">? </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0:01 [xfsalloc/13]</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 6952 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2 20 0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 drbd_a D</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">? </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">18:41 [xfsbufd/drbd1]</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 32155 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2 20 0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0 xfs_bm D</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">? </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> </span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">0:00 [flush-147:1]</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> And on the secondary the blocked process was (sorry, I saved the results of a slightly different command)</span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[djsperka@secondary ~]$ ps -eo pid,state,wchan:40,comm | grep -Ee " D |drbd"<br>
2309 S down_interruptible drbd0_worker<br> 2338 S down_interruptible drbd1_worker<br> 2370 S sk_wait_data drbd0_receiver<br> 2374 D blkdev_issue_flush drbd1_receiver<br>
2487 S sk_wait_data drbd0_asender<br> 2601 S sk_wait_data drbd1_asender<br></span></p><p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> And now, my question(s):</span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1. Is it reasonable to assume that the large send-Q/recv-Q are leading to the blocked processes? <br>
</span></p><p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2. If so, what drbd settings can we tweak to address the size of those queues, and possibly to prevent the block(s) from happening in the first place? <br>
</span></p><p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"> Thank you, <br>
</span></p><p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Dan<br>
</span></p><br></div><div><br clear="all"><div><div><br>-- <br>Daniel J. Sperka, Ph. D. <br>UC Davis Center for Neuroscience<br>
</div></div></div></div>