<div dir="auto">Could you provide the meta data contents of the affected resource ("drbdadm dump-md") along with resource configuration contents? This might help in giving some answers...</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 20 Sep 2021 at 07:03, Michael Labriola <<a href="mailto:michael.d.labriola@gmail.com">michael.d.labriola@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I've just had a resource come up Diskless after an accidental<br>
shutdown. It wasn't a power loss situation, I literally executed<br>
poweroff on the wrong machine via ssh. Trying to raise this one<br>
particular resource (the other 11 resources were fine) spits out a ton<br>
of this:<br>
<br>
extent 4765184 beyond end of bitmap!<br>
extent 5236736 beyond end of bitmap!<br>
extent 5246976 beyond end of bitmap!<br>
extent 5247485 beyond end of bitmap!<br>
extent 5249536 beyond end of bitmap!<br>
extent 5249664 beyond end of bitmap!<br>
extent 5249665 beyond end of bitmap!<br>
extent 5249666 beyond end of bitmap!<br>
extent 5249667 beyond end of bitmap!<br>
extent 5249668 beyond end of bitmap!<br>
extent 5249669 beyond end of bitmap!<br>
extent 5249670 beyond end of bitmap!<br>
extent 5249671 beyond end of bitmap!<br>
extent 5249672 beyond end of bitmap!<br>
extent 5249673 beyond end of bitmap!<br>
...<br>
extent 6781952 beyond end of bitmap!<br>
extent 6782464 beyond end of bitmap!<br>
extent 6782976 beyond end of bitmap!<br>
../shared/drbdmeta.c:1996:apply_al: ASSERT(bm_pos - bm_on_disk_pos <=<br>
chunk - this_extent_size) failed.<br>
<br>
The hardware is all good, the underlying block device is mountable,<br>
and I can still get all my data. I just can't attach the disk to the<br>
resource anymore.<br>
<br>
My question is twofold:<br>
<br>
1. Is there any way to recover this short of re-creating the drbd<br>
metadata and doing a full sync? That's the most obvious course of<br>
action, but it's a 50T resource and I don't want to wait 4 days for it<br>
to finish if I don't have to.<br>
<br>
2. Help me understand what happened, so I can make sure it doesn't<br>
happen again. My resource's metadata got corrupted somehow, right?<br>
Did I have data being written to my resource when it shut down that<br>
didn't make it all the way to disk, specifically in the meta data?<br>
Even though it was a "safe shutdown"?<br>
<br>
Both my servers are running Ubuntu 20.04.2 w/ DRBD9 (module 9.0.29,<br>
utils 9.18.0)<br>
<br>
Thanks!<br>
<br>
--<br>
Michael D Labriola<br>
21 Rip Van Winkle Cir<br>
Warwick, RI 02886<br>
401-316-9844 (cell)<br>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div></div>