<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thank you!</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Philipp Reisner <philipp.reisner@linbit.com><br>
<b>Sent:</b> Thursday, April 4, 2024 1:06 PM<br>
<b>To:</b> Tim Westbrook <Tim_Westbrook@selinc.com><br>
<b>Cc:</b> drbd-user@lists.linbit.com <drbd-user@lists.linbit.com><br>
<b>Subject:</b> Re: Usynced blocks if replication is interrupted during initial sync</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">[Caution - External]<br>
<br>
Hello Tim,<br>
<br>
We were able to write a reproducer test case and fix this regression<br>
with this commit:<br>
<a href="https://urldefense.com/v3/__https://github.com/LINBIT/drbd/commit/be9a404134acc3d167e8a7e60adce4f1910a4893__;!!O7uE89YCNVw!Lg3rRgojII2WxVzSLqO-h7mIpRxkiz34chmd89P-b1GDlUP3QD3-jc3gdlj5aTFp9uwgCw_5PBjXtwPtevJ0JK_oC8s8ZGg$">https://urldefense.com/v3/__https://github.com/LINBIT/drbd/commit/be9a404134acc3d167e8a7e60adce4f1910a4893__;!!O7uE89YCNVw!Lg3rRgojII2WxVzSLqO-h7mIpRxkiz34chmd89P-b1GDlUP3QD3-jc3gdlj5aTFp9uwgCw_5PBjXtwPtevJ0JK_oC8s8ZGg$</a><br>
<br>
This commit will go into the drbd-9.1.20 and drbd-9.2.9 releases.<br>
<br>
best regards,<br>
Philipp<br>
<br>
On Fri, Mar 22, 2024 at 1:49 AM Tim Westbrook <Tim_Westbrook@selinc.com> wrote:<br>
><br>
><br>
><br>
> Thank you<br>
><br>
><br>
> So if "Copying bitmap of peer node_id=0" on reconnect after interruption, indicates the issue, the issue still exists for me.<br>
><br>
> I am able to dump the metadata, but not sure it is very useful at this point...<br>
><br>
> I have not tried invalidating it after a mount/unmount, nor have I tried invalidating it after adding a node, but we were trying to avoid unmounting once configured.<br>
><br>
> Would you recommend against going back to a release version prior to this change?<br>
><br>
> Is there any other information I can provide that would help ? Could I dump the meta data at any some point to show the expected/unexpected state?<br>
><br>
> Latest flow is below<br>
><br>
> Thank you so much for your assistance,<br>
> Tim<br>
><br>
> 1. /dev/vg/persist mounted directly without drbd<br>
> 2. Enable DRBD by creating a single node configuration file<br>
> 3. Reboot<br>
> 4. Create metadata on separate disk (--max-peers=5)<br>
> 5. drdbadm up persist<br>
> 6. drbdadm invalidate persist<br>
> 7. drbdadm primary --force persist<br>
> 8. drbdadm down persist<br>
> 9. drbdadm up persist<br>
> 10. drbdadm invalidate persist*<br>
> 11. drbdadm primary --force persist<br>
> 12. mount /dev/drbd0 to /persist<br>
> 13. start using that mount point<br>
> 14. some time later<br>
> 15. Modify configuration to add new target backup node<br>
> 16. Copy config to remote node and reboot, it will restart in secondary<br>
> 17. drbdadm adjust persist (on primary)<br>
> 18. secondary comes up and initial sync starts<br>
> 19. stop at 50% by disabling network interface<br>
> 20. re-enable network interface<br>
> 21. sync completes right away - node-id 0 message here<br>
> 22. drbdadm verify persist - fails many blocks<br>
><br>
><br>
><br>
><br>
> From: Joel Colledge <joel.colledge@linbit.com><br>
> Sent: Wednesday, March 20, 2024 12:02 AM<br>
> To: Tim Westbrook <Tim_Westbrook@selinc.com><br>
> Cc: drbd-user@lists.linbit.com <drbd-user@lists.linbit.com><br>
> Subject: Re: Usynced blocks if replication is interrupted during initial sync<br>
><br>
> [Caution - External]<br>
><br>
> > We are still seeing the issue as described but perhaps I am not putting the invalidate<br>
> > at the right spot<br>
> ><br>
> > Note - I've added it at step 6 below, but I'm wondering if it should be after<br>
> > the additional node is configured and adjusted (in which case I would need to<br>
> > unmount as apparently you can't invalidate a disk in use)<br>
> ><br>
> > So do I need to invalidate after every node is added?<br>
><br>
> With my reproducer, the workaround at step 6 works.<br>
><br>
> > Also Note, the node-id in the logs from the kernel is 0 but peers are configured with 1 and 2 ,<br>
> > is this an issue or they separate ids?<br>
><br>
> I presume you are referring to the line:<br>
> "Copying bitmap of peer node_id=0"<br>
> The reason that node ID 0 appears here is that DRBD stores a bitmap of<br>
> the blocks that have changed since it was first brought up. This is<br>
> the "day0" bitmap. This is stored in all unused bitmap slots. All<br>
> unused node IDs point to one of these bitmaps. In this case, node ID 0<br>
> is unused. So this line means that it is using the day0 bitmap here.<br>
> This is unexpected, as mentioned in my previous reply.<br>
><br>
> Joel<br>
</div>
</span></font></div>
</body>
</html>