<div dir="ltr"><div class="gmail_extra"><div class="gmail_extra"><div class="gmail_extra">Dear Philipp,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Today, "Unstable Outdated problem" does not occur yet.</div><div class="gmail_extra"><br></div><div class="gmail_extra">But this CASE-16 similar problem occurred on Windows-DRBD(patched with your latest version [86e4439])</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">1. Version</div><div class="gmail_extra"> - 86e4439</div><div class="gmail_extra"> </div><div class="gmail_extra"><br></div><div class="gmail_extra">2. Reproduce step</div><div class="gmail_extra"><br></div><div class="gmail_extra"> 1) force "Outdate" state on each node by "drbdsetup outdate" command </div><div class="gmail_extra"> 2) promote one node to primary: "drbdadm primary --force r0"</div><div class="gmail_extra"> 3) file copy</div><div class="gmail_extra"> 4) result:</div><div class="gmail_extra"> (1) file copy is pending at the beginning.</div><div class="gmail_extra"> (2) primary status does not end in WFBitMapS status.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">3. Questions</div><div class="gmail_extra">There are two set-positions of CONSIDER_RESYNC.</div><div class="gmail_extra">We think drbd_set_role() part may have some problem.</div><div class="gmail_extra">If this function sets the CONSIDER_RESYNC, the bitmap-exchange should be occurred.</div><div class="gmail_extra">After then, if this bitmap-exchange starts, this hang problem will be sometimes occurred or not.</div><div class="gmail_extra">So, we try to disable the bitmap-exchange like this;</div><div class="gmail_extra"><br></div><div class="gmail_extra">receive_bitmap()</div><div class="gmail_extra">{</div><div class="gmail_extra">...</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>} else if (peer_device->repl_state[NOW] != L_WF_BITMAP_S) {</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>/* admin may have requested C_DISCONNECTING,</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span> * other threads may have noticed network errors */</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>drbd_info(peer_device, "unexpected repl_state (%s) in receive_bitmap\n",</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span> drbd_repl_str(peer_device->repl_state[NOW]));</div><div class="gmail_extra">#ifdef _WIN32_V9</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>err = -EIO;</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>goto out;</div><div class="gmail_extra">#endif</div><div class="gmail_extra"><span class="" style="white-space:pre"> </span>}</div><div class="gmail_extra">...</div><div class="gmail_extra">}</div><div class="gmail_extra"><br></div><div class="gmail_extra"> 1) What do yo think about our workaround?</div><div class="gmail_extra"> 2) When does "outdated-outdated" status occur?</div><div class="gmail_extra"> 3) Could you explain more detail about CONSIDER_RESYNC meaning or purpose?<br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">4. Logs</div><div class="gmail_extra"> - Let me give you 2 logs, one is WindowsDRBD the other is LinuxDRBD</div><div class="gmail_extra"> - Both have very similar pattern.</div><div class="gmail_extra"> - Both maybe wait for bitmap response with WBitMapS status.</div><div class="gmail_extra"> - And finally both be ended in failure of I/O hang.</div></div><div class="gmail_extra"><br></div><div class="gmail_extra"> 1) [CASE-16] Windows DRBD log</div><div class="gmail_extra"> - <a href="http://pastebin.com/Sm7pJyTa">http://pastebin.com/Sm7pJyTa</a></div><div class="gmail_extra"><br></div><div class="gmail_extra"> 2) [CASE-16] Linux DRBD Log</div><div class="gmail_extra"> - <a href="http://pastebin.com/VhGyBAwT">http://pastebin.com/VhGyBAwT</a></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_quote">2016-02-10 14:58 GMT+09:00 김재헌 <span dir="ltr"><<a href="mailto:jhkim@mantech.co.kr" target="_blank">jhkim@mantech.co.kr</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div style="font-size:14px"><div style="margin:0px;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px">Dear Philipp,</div><div style="margin:0px;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px"><br></div></div><div style="font-size:14px;margin:0px;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal">1. Test version</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal"> - CentOS-7 Linux 3.10.0-229.7.2.el7.x86_64</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal"> - Engine: V9.0.1-1</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal"> -- GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e build by root@drbd9-02, 2016-02-09 09:46:20</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;line-height:normal"><br></div></div><span style="font-size:14px">2. Test scenario</span></div><div><br></div><div>1) status</div><div><br></div><div>[root@drbd9-01 ~]# drbdadm status r0<br>r0 role:Secondary<br> disk:Outdated<br> drbd9-02 role:Secondary<br> peer-disk:Outdated<br><br>2) try mount</div></div></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>...........</div></div></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><b><font color="#ff0000">Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: pdsk( Outdated -> Consistent ) repl( Established -> WFBitMapS )</font></b><br>Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%<br>Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: unexpected<br>Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: In UUIDs from node 1 found equal UUID (3D7DF30CF04727A4) for nodes 2 3<br>Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have C223F7CE3C9D7358 for node_id=2<br>Feb 9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have C223F7CE3C9D7358 for node_id=3</div><div><br>Feb 9 14:19:35 drbd9-01 kernel: EXT4-fs (drbd1): mounting ext3 file system using the ext4 subsystem<br>Feb 9 14:20:01 drbd9-01 systemd: Starting Session 13 of user root.<br>Feb 9 14:20:01 drbd9-01 systemd: Started Session 13 of user root.</div><div><br>Feb 9 14:21:49 drbd9-01 kernel: INFO: task mount:4758 blocked for more than 120 seconds.<br>Feb 9 14:21:49 drbd9-01 kernel: "echo 0 > /proc/sys/<b><font color="#ff0000">kernel/hung_task_timeout_secs</font></b>" disables this message.<br>Feb 9 14:21:49 drbd9-01 kernel: mount D ffff88003d613680 0 4758 4685 0x00000080<br></div></div></div></blockquote></div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt;line-height:21px"><span lang="EN-US" style="font-size:8pt;line-height:16px;font-family:'\00b9d1\00c740 \00ace0\00b515';color:rgb(127,127,127)"><br></span></p></div></div></div></div></div></div>
</div></div>