<div dir="ltr"><div>Hi, <br></div><div><div><br></div><div>We tested the asynchronous replication Ahead mode of drbd9(9.0.9) at low bandwidth (1 ~ 10 Mbps Network) and found that the I/O response time rate slowed down during replication and eventually I/O of the volume hang occurred.</div><div>Tests according to the following configuration and procedure will reproduce well.</div></div><div><br></div><div><br></div><div><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"><drbd.conf><u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">global {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> disable-ip-verification;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> usage-count no;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">}<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">common {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> startup {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> wfc-timeout 1;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> disk {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> resync-rate 100M;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> net {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> on-congestion pull-ahead;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> congestion-fill 480M;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> verify-alg md5;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> proxy {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> memlimit 500M;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">}<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">resource r0 {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> protocol A;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> disk {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> on-io-error detach;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> device /dev/drbd0;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> floating <a href="http://200.200.2.10:7788/" target="_blank">200.200.2.10:7788</a> {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> disk /dev/sdd1;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> meta-disk internal;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> proxy on pm1 {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> inside <a href="http://127.0.0.1:7789/" target="_blank">127.0.0.1:7789</a>;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> outside <a href="http://200.200.2.10:7790/" target="_blank">200.200.2.10:7790</a>;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> floating <a href="http://200.200.2.11:7788/" target="_blank">200.200.2.11:7788</a> {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> disk /dev/sdc1;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> meta-disk internal;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> proxy on pm2 {<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> inside <a href="http://127.0.0.1:7789/" target="_blank">127.0.0.1:7789</a>;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> outside <a href="http://200.200.2.11:7790/" target="_blank">200.200.2.11:7790</a>;<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US"> }<u></u><u></u></span></p><p class="MsoNormal" style="font-size:14px"><span lang="EN-US">}</span></p></div><div><br></div><div><br></div><div><div><Test procedure></div><div>1. Proxy Buffer Size: 500M, congestion-fill: 480M;</div><div>2. Replication network bandwidth limit of 10Mbps (using VMware's network bandwidth limiting feature)</div><div>3. Mount the /data on the pm1 node</div><div>4. dd if=/dev/zero of=/data/test.out bs=100M count=40</div><div>5. Enter pm1 ahead mode</div><div>6. When ls -l is executed in pm1 node /data directory, the result is not output and it is almost hang status.</div></div><div><br></div><div><br></div><div>The purpose of this test was to measure the behavior of the drbd9 asynchronous Ahead mode and the I/O response rate of the volume, regardless of the network bandwidth.<br></div><div><br></div><div>In my opinion, If the number of drbd_req in drbd9 becomes considerably large (more than tens of drbd_req accumulated in trasfer_log in this test), the execution time of drbd_sender increases. Especially, it takes much time to traverse transfer_log in the following code,finally It seems that the completion time has increased. </div><div><br></div><div>// Only the execution time of this logic is measured more than 5 ms</div><div><br></div><div><div>static struct drbd_request *__next_request_for_connection(</div><div><span style="white-space:pre"> </span>struct drbd_connection *connection, struct drbd_request *r)</div><div>{</div><div><span style="white-space:pre"> </span>r = list_prepare_entry(r, &connection->resource->transfer_log, tl_requests);</div><div><br></div><div><span style="white-space:pre"> </span>list_for_each_entry_continue(r, &connection->resource->transfer_log, tl_requests) {</div><div><span style="white-space:pre"> </span>int vnr = r->device->vnr;</div><div><span style="white-space:pre"> </span>struct drbd_peer_device *peer_device = conn_peer_device(connection, vnr);</div><div><span style="white-space:pre"> </span>unsigned s = drbd_req_state_by_peer_device(r, peer_device);</div><div><span style="white-space:pre"> </span>if (!(s & RQ_NET_QUEUED))</div><div><span style="white-space:pre"> </span>continue;</div><div><span style="white-space:pre"> </span>return r;</div><div><span style="white-space:pre"> </span>}</div><div><span style="white-space:pre"> </span>return NULL;</div><div>}</div></div><div><br></div><div><br></div><div>Please check this issue and hope that the performance of asynchronous replication of drbd9 will improve.<br></div><div><br></div><div>Thanks.</div><div><br></div><div><br></div><div>Best Regards</div><div>from SungEun Kim</div><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0cm 0cm 0pt"><img src="cid:image001.png@01D0F522.B2A586F0" alt="설명: logo"></p><p style="font-size:small;margin:0cm 0cm 0.0001pt;background-image:initial;background-repeat:initial"><font face="verdana, sans-serif"><span style="font-size:10pt;color:black">Technical Research Center<span lang="EN-US"> </span><span lang="EN-US">/ </span>Dev3 Team</span><span lang="EN-US" style="font-size:10.5pt;color:rgb(136,136,136)"></span></font></p><p style="font-size:small;margin:0cm 0cm 0.0001pt;background-image:initial;background-repeat:initial"><font face="verdana, sans-serif"><span style="font-size:10pt;color:black"><span lang="EN-US">Principal Research Engineer</span></span></font></p><p style="font-size:small;margin:0cm 0cm 0pt"></p><p style="font-size:small;margin:0cm 0cm 0.0001pt;background-image:initial;background-repeat:initial"><font face="verdana, sans-serif"><b><span style="color:black"><span lang="EN-US"><span style="font-size:14px">SungEun Kim</span><span style="font-size:10pt"> </span><a href="mailto:sekim@mantech.co.kr" style="color:rgb(17,85,204);font-size:10pt" target="_blank">sekim@mantech.co.kr</a></span></span></b><span lang="EN-US" style="font-size:10.5pt;color:rgb(136,136,136)"></span></font></p><p style="font-size:small;margin:0cm 0cm 0.0001pt;line-height:21px"><font color="#000000" face="verdana, sans-serif"><span style="font-size:8pt;line-height:16px">12F, Seoulforest Kolon digital Tower, 308-4, Seongsudong 2ga, Seongdong-gu, Seoul, Korea</span></font></p><p style="font-size:small;margin:0cm 0cm 0.0001pt;line-height:21px"><span style="font-size:8pt;line-height:16px;color:rgb(0,0,0);font-family:verdana,sans-serif">Tel : 02-2136-6913 / Fax : 02-575-4858 / Call Center : 1833-7790</span></p><p style="margin:0cm 0cm 0.0001pt;line-height:21px"><span lang="EN-US" style="color:rgb(0,0,0);font-family:verdana,sans-serif;font-size:8pt;line-height:16px"><a href="http://www.mantech.co.kr/" style="color:rgb(17,85,204)" target="_blank">http://www.mantech.co.kr</a></span><span lang="EN-US" style="color:rgb(0,0,0);font-family:verdana,sans-serif;font-size:8pt;line-height:16px"> </span><font color="#000000" face="verdana, sans-serif"><span lang="EN-US" style="font-size:8pt;line-height:16px"><br></span></font></p><p style="margin:0cm 0cm 0.0001pt;line-height:21px"><img src="cid:ii_150a68c3811acf54" alt="본문 이미지 1" width="400" height="160" style="line-height: normal;"><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>