[DRBD-user] corrupted resource can't be fixed be rolling back to old snapshot

[Michael Labriola]

On Tue, Aug 2, 2022 at 5:04 AM Joel Colledge <joel.colledge at linbit.com> wrote:
>
> Hi Michael,
>
> Are you using the most recent version of drbd-utils? There have been a
> few fixes over the years which might be related.

I was using 9.20.2 this last time.  I'm fairly certain I've been using
the focal ppa from linbit for the entire life of this particular
system, so I've probably always been newer than the Ubuntu version.

>
> Perhaps the hardware problems affected the metadata long ago and now
> the corrupted metadata is present in all the snapshots.

Possible.  But I'm fairly certain we recreated the DRBD resources from
scratch (new meta-data, initial sync, etc) after we fixed the
problems... granted, I could still have problems.  This particular
system for whatever reason is cursed.

>
> If that is not the case, this looks to me more like a bug than a
> misunderstanding of how DRBD works. Are you able to reproduce the
> issue starting from a fresh volume? It could be that this particular
> combination of device size and bitmap slot count triggers a bug that
> no-one else has yet encountered. A reproducer would be necessary to
> work on fixing it.

Well, half of what I was looking for here was somebody else to tell me
this is odd.  I *should* be able to recover by rolling back to an old
snapshot of the backing ZVOL on both nodes.  I know I've done it for
proof of concept and to rollback to fix "human error" type problems...
This was the first time I've had to try to recover from something
actually going wrong (from DRBD's standpoint).

For the record, I did not lose any data... I could still access the
ZVOL directly (e.g., mounted EXT4) and rsync into a newly created DRBD
resource...  but this particular resource is large and takes 3-4 days
to finish the initial sync.  I'd obviously like to avoid that.

>
> Best regards,
> Joel

--
Michael D Labriola
401-316-9844 (cell)