<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=GBK">
  </head>
  <body>
    <pre>Version: drbd-9.0.21-1</pre>
    <pre>Layout: drbd.res within 3 nodes -- node-1(Secondary), node-2(Primary), node-3(Secondary)</pre>
    <pre>Description: 
a.reboot node-2 when cluster is working.
b.re-up the drbd.res on node-2 after it restarted.
c.an expected resync from node-3 to node-2 happens. When the resync is done, however,
  node-1 raises an unexpected WFBitMapS repl status and can't recover to normal anymore.

Status output:
</pre>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-1:
        drbdadm status</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">drbd6
        role:Secondary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">disk:UpToDate</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">hotspare
        connection:Connecting</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-2
        role:Primary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">replication:WFBitMapS
        peer-disk:Consistent</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-3
        role:Secondary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">peer-disk:UpToDate</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1"><br>
      </font> </p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-2:
        drbdadm status</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">drbd6
        role:Primary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">disk:UpToDate</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">hotspare
        connection:Connecting</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-1
        role:Secondary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">peer-disk:UpToDate</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">node-3
        role:Secondary</font></p>
    <font size="-1"> </font>
    <p style="margin: 10px 0px 0px; padding: 0px; font-style: normal;
      font-variant-ligatures: normal; font-variant-caps: normal;
      font-weight: normal; letter-spacing: normal; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255); text-decoration-style:
      initial; text-decoration-color: initial;"><font size="-1">peer-disk:UpToDate</font><tt><br>
      </tt></p>
    <pre><tt>I assume that there is a process sequence below according to my source code version:
node-1                                           node-2                                                            node-3
                                                 restarted with CRASHED_PRIMARY               
                                                 start sync with node-3 as target                                  start sync with node-2 as source
                                                 бн                                                                бн
                                                 end sync with node-3                                              end sync with node-2
                                                 w_after_state_change
                                                      loop 1 within for loop against node-1:(a)</tt>
<tt><tt>receive_uuids10                                  </tt>send uuid with UUID_FLAG_GOT_STABLE&amp;</tt><tt><tt>CRASHED_PRIMARY</tt> to node-1</tt>
<tt><tt>receive uuid of node-2 with CRASHED_PRIMARY</tt>      loop 2 within for loop against node-3:
                                                 clear  CRASHED_PRIMARY(b)
send uuid to node-2 with UUID_FLAG_RESYNC        </tt><tt><tt>receive uuids10</tt>
</tt><tt><tt>sync_handshake</tt> to SYNC_SOURCE_IF_BOTH_FAILED     </tt><tt><tt>sync_handshake to NO_SYNC</tt>
change repl state to </tt><tt>WFBitMapS

The key problem is about the order of step(a) and step(b), that is, node-2 sends the
unexpected  </tt><tt><tt><tt>CRASHED_PRIMARY</tt> to node-1 though it's actually no longer a crashed primary
after syncing with node-3.</tt></tt><tt><tt>
So may I have the below questions:
a.If this is really a BUG or just an expected result?
b.If there's already a patch fix within the newest verion?
c.If there's some workaround method against this kind of unexcepted status, since I really
  meet so many other problems like that :( 
</tt></tt></pre>
    <pre class="moz-signature" cols="72">-- 
Sincerely Yours,
Zhang Duan</pre>
  </body>
</html>