[DRBD-user] weird behavior with metadata

Lars Ellenberg lars.ellenberg at linbit.com
Mon Jan 26 10:27:13 CET 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Sat, Jan 24, 2015 at 04:01:11PM +0200, Alex Vasilenko wrote:
> Hello,
> Having a strange behavior using drbd 8.4.5. After restart one partition of

TL;DR: for the fix scroll ahead.

You should always tell both kernel *and* userland versions.

> Jan 24 14:42:39 de kernel: block drbd0: disk( UpToDate -> Failed )

That may look scary, but is normal: during detach (down),
we move via "Failed" to "Diskless" to avoid various potential races.

> Now trying to restart backup partition:
> > $ sudo drbdadm down backup
> > $ sudo drbdadm up backup
> > strange bm_offset -65608 (expected: -65600)
> > No valid meta data found
> > Command 'drbdmeta 1 v08 /dev/sda3 internal apply-al' terminated with exit
> > code 255

> Somehow metadata is not there anymore (?!). Note warning saying "strange
> bm_offset -65608 (expected: -65600)".
> Ok, how about recreate metadata and resync full partition? Trying to wipe
> old metadata shows that metadata is not present. Ok, let's pretend it's not
> there:
> > $ sudo drbdadm create-md backup
> > strange bm_offset -65608 (expected: -65600)
> > strange bm_offset -65608 (expected: -65600)
> > md_offset 1099511623680
> > al_offset 1099511590912
> > bm_offset 1099478036480
> > Found ext3 filesystem
> >   1073709016 kB data area apparently used
> >   1073709020 kB left usable by current configuration
> > Even though it looks like this would place the new meta data into
> > unused space, you still need to confirm, as this is only a guess.
> > Do you want to proceed?
> > [need to type 'yes' to confirm] yes
> initializing activity log
> > NOT initializing bitmap
> > Writing meta data...
> > New drbd meta data block successfully created.
> > success
> Same warning twice

So what.  During "probing", some validation is done with various
potential contexts.  But it keeps being "strange", which is reported.

> possible wrong fs detection (in fact it's ext4) and

ext2, 3, 4, who cares.  for this purpose, keeping you from shooting
yourself by truncating an existing file system, they are the same.
They don't differ in their "master magic",
but only in by feature flags, which are irrelevant for us.

> incorrect data usage detection (in fact takes 533GB of 1Tb partition)

How much "payload" you stored in the file system is irrelevant,
we care about the size of the file system itself,
and that you won't truncate it.

It may well still be completely "empty", it still would be very unhappy
if you truncated the data area it thinks it owns exclusively.

> Wiping and creating metadata again fixes bm_offset warning until next drbd
> service restart.

Yup. The kernel sanity checks are less picky when attaching the first
time, and just fixes the broken offsets drbdmeta put there.

> Having latest Centos 6.5. Kernel 2.6.32-504.1.3.el6.x86_64
> Disks are based on hardware RAID 1.
> $ sudo fdisk -l

Next time use "sfdisk -d" (to get the exact sectors,
not some rounded last-millenium cylinder numbers),
or better yet, show "blockdev --getsize64 /dev/X"

> backup uses /dev/sda3 and main uses /dev/sda6

> Can someone hint me what's wrong am I doing? Will provide any additional
> info on request.

The fix:
| drbdmeta: fix regression corner cases in bitmap size calculation
| Regression was introduced with the kernel/userland split.
| Regression was caused underestimating the number of bytes by
| not aligning the number of bits to 64 before converting to bytes
| (dividing by 8).

Apparently, your sda3 just happens to be of some "odd" size,
just above 2147483648 sectors, (hex: 0x80000000).
(if it was exactly that, you would not have triggered the bug).

: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
please don't Cc me, but send to list   --   I'm subscribed

More information about the drbd-user mailing list