[DRBD-user] bitmap corruption?

[Yanni M.]

Could you provide the meta data contents of the affected resource
 ("drbdadm dump-md") along with resource configuration contents? This might
help in giving some answers...

On Mon, 20 Sep 2021 at 07:03, Michael Labriola <michael.d.labriola at gmail.com>
wrote:

> I've just had a resource come up Diskless after an accidental
> shutdown.  It wasn't a power loss situation, I literally executed
> poweroff on the wrong machine via ssh.  Trying to raise this one
> particular resource (the other 11 resources were fine) spits out a ton
> of this:
>
> extent 4765184 beyond end of bitmap!
> extent 5236736 beyond end of bitmap!
> extent 5246976 beyond end of bitmap!
> extent 5247485 beyond end of bitmap!
> extent 5249536 beyond end of bitmap!
> extent 5249664 beyond end of bitmap!
> extent 5249665 beyond end of bitmap!
> extent 5249666 beyond end of bitmap!
> extent 5249667 beyond end of bitmap!
> extent 5249668 beyond end of bitmap!
> extent 5249669 beyond end of bitmap!
> extent 5249670 beyond end of bitmap!
> extent 5249671 beyond end of bitmap!
> extent 5249672 beyond end of bitmap!
> extent 5249673 beyond end of bitmap!
> ...
> extent 6781952 beyond end of bitmap!
> extent 6782464 beyond end of bitmap!
> extent 6782976 beyond end of bitmap!
> ../shared/drbdmeta.c:1996:apply_al: ASSERT(bm_pos - bm_on_disk_pos <=
> chunk - this_extent_size) failed.
>
> The hardware is all good, the underlying block device is mountable,
> and I can still get all my data.  I just can't attach the disk to the
> resource anymore.
>
> My question is twofold:
>
> 1.  Is there any way to recover this short of re-creating the drbd
> metadata and doing a full sync?  That's the most obvious course of
> action, but it's a 50T resource and I don't want to wait 4 days for it
> to finish if I don't have to.
>
> 2.  Help me understand what happened, so I can make sure it doesn't
> happen again.  My resource's metadata got corrupted somehow, right?
> Did I have data being written to my resource when it shut down that
> didn't make it all the way to disk, specifically in the meta data?
> Even though it was a "safe shutdown"?
>
> Both my servers are running Ubuntu 20.04.2 w/ DRBD9 (module 9.0.29,
> utils 9.18.0)
>
> Thanks!
>
> --
> Michael D Labriola
> 21 Rip Van Winkle Cir
> Warwick, RI 02886
> 401-316-9844 (cell)
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user at lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20210920/8fcafd14/attachment.htm>