<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hello all,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I have a question regarding an odd DRBD state that our resources got into shortly after bootstrap. Despite the resource claiming it is "Connected", it is actually out of sync and the disk state shows as "UpToDate/Inconsistent", but the resync never happens. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
This is a Linux server running a Xen hypervisor where we use DRBD 9.0.30 to sync VM disks across. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
We've seen this issue appear twice in the past few weeks across maybe a 100 or so of deployments, so it seems relatively rare, but we'd like to get to the bottom of it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">This is the post-bootstrap state (rendered in the format of now deprecated
<i>drbd-overview</i>, all data is retrieved from <i>drbdsetup status --json</i>).</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif;"><b>Side 7c:</b></span></div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><i>1005:resource-5/0 Connected Primary/Secondary UpToDate/Inconsistent</i></span></div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif;"><b>Side 7d:</b></span></div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><i>1005:</i></span><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><i><i style="background-color: rgb(255, 255, 255);">resource-5</i></i></span><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><i>/0
Connected Secondary/Primary Inconsistent/UpToDate</i></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif;">A </span><span style="font-family: Calibri, Helvetica, sans-serif;"><i>drbdadm disconnect</i></span><span style="font-family: Calibri, Helvetica, sans-serif;"> and subsequent
</span><span style="font-family: Calibri, Helvetica, sans-serif;"><i>drbdadm connect</i></span><span style="font-family: Calibri, Helvetica, sans-serif;"> resolves the issue and the resource starts syncing.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><b>Side 7c:</b></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif">
<i><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">1005:</span><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><i style="background-color: rgb(255, 255, 255);">resource-5</i></span><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">/0
SyncSource Primary/Secondary UpToDate/Inconsistent</span></i></div>
<div style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif">
<i>
<div style="margin:0px;font-style:normal;background-color:rgb(255, 255, 255)"><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> [>.................................................] sync'ed: 0.4%</span></div>
</i></div>
<div style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><b>Side 7d:</b></span></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif"><i><span style="margin: 0px; background-color: rgb(255, 255, 255); font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">1005:</span><span style="margin: 0px; background-color: rgb(255, 255, 255); font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><i style="background-color: rgb(255, 255, 255);">resource-5</i></span><span style="margin: 0px; background-color: rgb(255, 255, 255); font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">/0
SyncTarget Secondary/Primary Inconsistent/UpToDate</span></i></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif"><i><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">
[>.................................................] sync'ed: 0.4%</span></i></span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="margin:0px;font-size:14px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif"><i><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><br>
</span></i></span></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><font face="Calibri, Helvetica, sans-serif">Looking at the
<b>kern.log</b> entries I can see that the issue likely happened during handshake when the resource was to begin resync. For some reason, for `resource-5` there are two handshake attempt entries one after another (only 1 for any other resource):</font></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><font face="Calibri, Helvetica, sans-serif"><b style="background-color:rgb(255, 255, 255)"><br class="Apple-interchange-newline">
</b><span style="font-size: 12pt; background-color: rgba(0, 0, 0, 0); font-family: Calibri, Helvetica, sans-serif;"><b style="background-color:rgba(0, 0, 0, 0)">Side 7c:</b></span><br>
</font></div>
<div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.305463] drbd resource-5/0 drbd1005 host-7d: drbd_sync_handshake:</i></span>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.305466] drbd resource-5/0 drbd1005 host-7d: self E8D4DA04CB03E845:AC3365A2A7A0372F:0000000000000000:0000000000000000 bits:0 flags:20</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.305468] drbd resource-5/0 drbd1005 host-7d: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:4</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.305469] drbd resource-5/0 drbd1005 host-7d: uuid_compare()=source-set-bitmap by rule=just-created-peer</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.305471] drbd resource-5/0 drbd1005 host-7d: Setting and writing one bitmap slot, after drbd_sync_handshake</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.309659] drbd resource-5/0 drbd1005 host-7d: drbd_sync_handshake:</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.309663] drbd resource-5/0 drbd1005 host-7d: self E8D4DA04CB03E845:AC3365A2A7A0372F:0000000000000000:0000000000000000 bits:0 flags:20</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.309664] drbd resource-5/0 drbd1005 host-7d: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:4</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.309666] drbd resource-5/0 drbd1005 host-7d: uuid_compare()=source-set-bitmap by rule=just-created-peer</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.309667] drbd resource-5/0 drbd1005 host-7d: Setting and writing one bitmap slot, after drbd_sync_handshake</i></span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.310865] drbd resource-5/0 drbd1005 host-7d: repl( Established -> WFBitMapS ) resync-susp( peer -> no )</i></span></div>
<span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7c: [ 801.311814] drbd resource-5/0 drbd1005 host-7d: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%</i></span></div>
</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><font face="Calibri, Helvetica, sans-serif"><br>
</font></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><font face="Calibri, Helvetica, sans-serif"><span style="font-family: Calibri, Helvetica, sans-serif;">On the other side there is very little as a response, only the notification that
the </span><span style="font-family: Calibri, Helvetica, sans-serif;"><i>repl_state
</i></span><span style="font-family: Calibri, Helvetica, sans-serif;">it saw wasn't expected:</span></font></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><font face="Calibri, Helvetica, sans-serif"><br>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif;"><b style="background-color: rgb(255, 255, 255);">Side 7d:</b></span><br>
</div>
<div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7d: [ 954.942328] drbd resource-5/0 drbd1005 host-7c: resync-susp( peer -> no )</i></span>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7d: [ 954.947226] drbd resource-5/0 drbd1005 host-7c: receive bitmap stats [</i></span><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Bytes(packets)</i></span><span style="font-family: Calibri, Helvetica, sans-serif;"><i>]:
plain 0(0), RLE 23(1), total 23; compression: 100.0%</i></span></div>
</div>
<div class="elementToProof"><span style="font-family: Calibri, Helvetica, sans-serif;"><i>Apr 22 23:22:23 host-7d: [ 954.947227] drbd resource-5/0 drbd1005 host-7c: unexpected repl_state (Established) in receive_bitmap</i></span></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><br>
</b></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Inspecting the resource it indeed had data `out-of-sync` (as also evidenced by the resync on reconnect).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-size: 12pt;">I have tried searching for the </span><span style="font-size: 12pt;"><i style="font-family: Calibri, Helvetica, sans-serif; background-color: rgb(255, 255, 255);">unexpected repl_state (Established) in receive_bitmap
</i></span><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">and found an open LINSTOR issue on GitHub: </span><span style="font-family:Calibri, Helvetica, sans-serif;font-size:14px;background-color:rgb(255, 255, 255)"><a href="https://github.com/LINBIT/linstor-server/issues/218" id="LPNoLPOWALinkPreview" style=""><span style="font-size: 12pt;">https://github.com/LINBIT/linstor-server/issues/218</span></a></span><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"> -
seems to be a similar situation as ours, but in LINSTOR. Hasn't been active for a while though and no insight into what the cause might be.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>DRBD version info:</b></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<i>DRBDADM_API_VERSION=2</i></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div><i>DRBD_KERNEL_VERSION_CODE=0x09001e</i></div>
<div><i>DRBD_KERNEL_VERSION=9.0.30</i></div>
<div><i>DRBDADM_VERSION_CODE=0x091500</i></div>
<div><i>DRBDADM_VERSION=9.21.0-rc.1</i></div>
<div><i><br>
</i></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;">Also attaching full entries for the resource in the
</span><code><span style="font-family: Calibri, Helvetica, sans-serif;">kern.log for each side in case it's</span></code><span style="font-family: Calibri, Helvetica, sans-serif;"> useful:</span></div>
<div><br>
</div>
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)">Side 7c:</b></div>
</div>
<div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.801308] drbd resource-5: Starting worker thread (from drbdsetup [20333])</i>
<div><i>Apr 22 23:11:09 host-7c: [ 127.818388] drbd resource-5 host-7d: Starting sender thread (from drbdsetup [20343])</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.847948] drbd resource-5/0 drbd1005: meta-data IO uses: blk-bio</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.848912] drbd resource-5/0 drbd1005: disk( Diskless -> Attaching )</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.848920] drbd resource-5/0 drbd1005: Maximum number of peer devices = 1</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849210] drbd resource-5: Method to ensure write ordering: flush</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849215] drbd resource-5/0 drbd1005: Adjusting my ra_pages to backing device's (32 -> 1)</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849230] drbd resource-5/0 drbd1005: drbd_bm_resize called with capacity == 41943040</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849550] drbd resource-5/0 drbd1005: resync bitmap: bits=5242880 words=81920 pages=160</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849560] drbd1005: detected capacity change from 0 to 21474836480</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.849563] drbd resource-5/0 drbd1005: size = 20 GB (20971520 KB)</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.850493] drbd resource-5/0 drbd1005: recounting of set bits took additional 0ms</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.850499] drbd resource-5/0 drbd1005: disk( Attaching -> Inconsistent )</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.850501] drbd resource-5/0 drbd1005: attached to current UUID: 0000000000000004</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.854514] drbd resource-5 host-7d: conn( StandAlone -> Unconnected )</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.854920] drbd resource-5 host-7d: Starting receiver thread (from drbd_w_0b6d3edd [20338])</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 127.859587] drbd resource-5 host-7d: conn( Unconnected -> Connecting )</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.387629] drbd resource-5 host-7d: Handshake to peer 0 successful: Agreed network protocol version 120</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.387631] drbd resource-5 host-7d: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.387635] drbd resource-5 host-7d: Starting ack_recv thread (from drbd_r_0b6d3edd [20374])</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.426356] drbd resource-5 host-7d: Preparing remote state change 1064560840</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.439641] drbd resource-5/0 drbd1005 host-7d: drbd_sync_handshake:</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.439643] drbd resource-5/0 drbd1005 host-7d: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.439645] drbd resource-5/0 drbd1005 host-7d: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.439646] drbd resource-5/0 drbd1005 host-7d: uuid_compare()=no-sync by rule=just-created-both</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.442606] drbd resource-5 host-7d: Committing remote state change 1064560840 (primary_nodes=0)</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.442613] drbd resource-5 host-7d: conn( Connecting -> Connected ) peer( Unknown -> Secondary )</i></div>
<div><i>Apr 22 23:11:09 host-7c: [ 128.442615] drbd resource-5/0 drbd1005 host-7d: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established )</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.041168] drbd resource-5/0 drbd1005 host-7d: resync-susp( no -> peer )</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042271] drbd resource-5: Preparing cluster-wide state change 3912736869 (1->-1 7683/4609)</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042524] drbd resource-5: State change 3912736869: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042525] drbd resource-5: Committing cluster-wide state change 3912736869 (0ms)</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042530] drbd resource-5: role( Secondary -> Primary )</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042531] drbd resource-5/0 drbd1005: disk( Inconsistent -> UpToDate )</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.042560] drbd resource-5/0 drbd1005: size = 20 GB (20971520 KB)</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.043722] drbd resource-5: Forced to consider local data as UpToDate!</i></div>
<div><i>Apr 22 23:11:25 host-7c: [ 144.045194] drbd resource-5/0 drbd1005: new current UUID: E8D4DA04CB03E845 weak: FFFFFFFFFFFFFFFC</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.305463] drbd resource-5/0 drbd1005 host-7d: drbd_sync_handshake:</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.305466] drbd resource-5/0 drbd1005 host-7d: self E8D4DA04CB03E845:AC3365A2A7A0372F:0000000000000000:0000000000000000 bits:0 flags:20</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.305468] drbd resource-5/0 drbd1005 host-7d: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:4</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.305469] drbd resource-5/0 drbd1005 host-7d: uuid_compare()=source-set-bitmap by rule=just-created-peer</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.305471] drbd resource-5/0 drbd1005 host-7d: Setting and writing one bitmap slot, after drbd_sync_handshake</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.309659] drbd resource-5/0 drbd1005 host-7d: drbd_sync_handshake:</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.309663] drbd resource-5/0 drbd1005 host-7d: self E8D4DA04CB03E845:AC3365A2A7A0372F:0000000000000000:0000000000000000 bits:0 flags:20</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.309664] drbd resource-5/0 drbd1005 host-7d: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:4</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.309666] drbd resource-5/0 drbd1005 host-7d: uuid_compare()=source-set-bitmap by rule=just-created-peer</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.309667] drbd resource-5/0 drbd1005 host-7d: Setting and writing one bitmap slot, after drbd_sync_handshake</i></div>
<div><i>Apr 22 23:22:23 host-7c: [ 801.310865] drbd resource-5/0 drbd1005 host-7d: repl( Established -> WFBitMapS ) resync-susp( peer -> no )</i></div>
<i>Apr 22 23:22:23 host-7c: [ 801.311814] drbd resource-5/0 drbd1005 host-7d: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%</i></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><br>
</b></div>
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><br>
</b></div>
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><br>
</b></div>
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)">Side 7d:</b></div>
</div>
<div>
<div><i>Apr 22 23:08:44 host-7d: [ 136.465878] drbd resource-5: Starting worker thread (from drbdsetup [22498])</i>
<div><i>Apr 22 23:08:45 host-7d: [ 136.482132] drbd resource-5 host-7c: Starting sender thread (from drbdsetup [22507])</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.508338] drbd resource-5/0 drbd1005: meta-data IO uses: blk-bio</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.508639] drbd resource-5/0 drbd1005: disk( Diskless -> Attaching )</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.508645] drbd resource-5/0 drbd1005: Maximum number of peer devices = 1</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509135] drbd resource-5: Method to ensure write ordering: flush</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509138] drbd resource-5/0 drbd1005: Adjusting my ra_pages to backing device's (32 -> 1)</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509147] drbd resource-5/0 drbd1005: drbd_bm_resize called with capacity == 41943040</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509507] drbd resource-5/0 drbd1005: resync bitmap: bits=5242880 words=81920 pages=160</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509541] drbd1005: detected capacity change from 0 to 21474836480</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.509544] drbd resource-5/0 drbd1005: size = 20 GB (20971520 KB)</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.511572] drbd resource-5/0 drbd1005: recounting of set bits took additional 0ms</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.511578] drbd resource-5/0 drbd1005: disk( Attaching -> Inconsistent )</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.511580] drbd resource-5/0 drbd1005: attached to current UUID: 0000000000000004</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.515806] drbd resource-5 host-7c: conn( StandAlone -> Unconnected )</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.516244] drbd resource-5 host-7c: Starting receiver thread (from drbd_w_0b6d3edd [22501])</i></div>
<div><i>Apr 22 23:08:45 host-7d: [ 136.520296] drbd resource-5 host-7c: conn( Unconnected -> Connecting )</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.016563] drbd resource-5 host-7c: Handshake to peer 1 successful: Agreed network protocol version 120</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.016568] drbd resource-5 host-7c: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.016595] drbd resource-5 host-7c: Starting ack_recv thread (from drbd_r_0b6d3edd [22530])</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.054881] drbd resource-5: Preparing cluster-wide state change 1064560840 (0->1 499/146)</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.070806] drbd resource-5/0 drbd1005 host-7c: drbd_sync_handshake:</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.070809] drbd resource-5/0 drbd1005 host-7c: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.070811] drbd resource-5/0 drbd1005 host-7c: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.070812] drbd resource-5/0 drbd1005 host-7c: uuid_compare()=no-sync by rule=just-created-both</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.071237] drbd resource-5: State change 1064560840: primary_nodes=0, weak_nodes=0</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.071240] drbd resource-5: Committing cluster-wide state change 1064560840 (16ms)</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.071279] drbd resource-5 host-7c: conn( Connecting -> Connected ) peer( Unknown -> Secondary )</i></div>
<div><i>Apr 22 23:11:10 host-7d: [ 282.071285] drbd resource-5/0 drbd1005 host-7c: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established )</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.669646] drbd resource-5/0 drbd1005 host-7c: resync-susp( no -> after dependency )</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.671299] drbd resource-5 host-7c: Preparing remote state change 3912736869</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.672524] drbd resource-5/0 drbd1005 host-7c: pdsk( Inconsistent -> UpToDate )</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.672555] drbd resource-5/0 drbd1005: size = 20 GB (20971520 KB)</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.673190] drbd resource-5 host-7c: Committing remote state change 3912736869 (primary_nodes=2)</i></div>
<div><i>Apr 22 23:11:26 host-7d: [ 297.673195] drbd resource-5 host-7c: peer( Secondary -> Primary )</i></div>
<div><i>Apr 22 23:22:23 host-7d: [ 954.942328] drbd resource-5/0 drbd1005 host-7c: resync-susp( peer -> no )</i></div>
<div><i>Apr 22 23:22:23 host-7d: [ 954.947226] drbd resource-5/0 drbd1005 host-7c: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%</i></div>
<i>Apr 22 23:22:23 host-7d: [ 954.947227] drbd resource-5/0 drbd1005 host-7c: unexpected repl_state (Established) in receive_bitmap</i></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div><b style="font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><br>
</b></div>
<div><br>
</div>
<div>Any help is much appreciated.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Alexander</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</body>
</html>