[DRBD-user] bitmap corruption?

[Michael Labriola]

I've just had a resource come up Diskless after an accidental
shutdown.  It wasn't a power loss situation, I literally executed
poweroff on the wrong machine via ssh.  Trying to raise this one
particular resource (the other 11 resources were fine) spits out a ton
of this:

extent 4765184 beyond end of bitmap!
extent 5236736 beyond end of bitmap!
extent 5246976 beyond end of bitmap!
extent 5247485 beyond end of bitmap!
extent 5249536 beyond end of bitmap!
extent 5249664 beyond end of bitmap!
extent 5249665 beyond end of bitmap!
extent 5249666 beyond end of bitmap!
extent 5249667 beyond end of bitmap!
extent 5249668 beyond end of bitmap!
extent 5249669 beyond end of bitmap!
extent 5249670 beyond end of bitmap!
extent 5249671 beyond end of bitmap!
extent 5249672 beyond end of bitmap!
extent 5249673 beyond end of bitmap!
...
extent 6781952 beyond end of bitmap!
extent 6782464 beyond end of bitmap!
extent 6782976 beyond end of bitmap!
../shared/drbdmeta.c:1996:apply_al: ASSERT(bm_pos - bm_on_disk_pos <=
chunk - this_extent_size) failed.

The hardware is all good, the underlying block device is mountable,
and I can still get all my data.  I just can't attach the disk to the
resource anymore.

My question is twofold:

1.  Is there any way to recover this short of re-creating the drbd
metadata and doing a full sync?  That's the most obvious course of
action, but it's a 50T resource and I don't want to wait 4 days for it
to finish if I don't have to.

2.  Help me understand what happened, so I can make sure it doesn't
happen again.  My resource's metadata got corrupted somehow, right?
Did I have data being written to my resource when it shut down that
didn't make it all the way to disk, specifically in the meta data?
Even though it was a "safe shutdown"?

Both my servers are running Ubuntu 20.04.2 w/ DRBD9 (module 9.0.29,
utils 9.18.0)

Thanks!

--
Michael D Labriola
21 Rip Van Winkle Cir
Warwick, RI 02886
401-316-9844 (cell)