[DRBD-user] LCMC display (and other tools) says "up to date" but DRBD is not

Whit Blauvelt whit+drbd at transpect.com
Thu Oct 25 15:38:36 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Thu, Oct 25, 2012 at 11:33:17AM +0200, Lars Ellenberg wrote:

> One such "adminnistrative error" we've come across much too frequently,
> and which shows exactly these "symptoms", is this:
> 
> (I'm in the ascii art mood today...)
> 
> You at one point had:
> ---------------------
> 
>  VM
>   \
>    \
>  [logical volume]
> 
> Then you added DRBD, and now you have:
> --------------------------------------
> 
>  VM         [DRBD] -------- [DRBD] remote node
>   \          /
>    \        /             !! THIS IS WRONG !!
>  [logical volume]
> 
>  (DRBD does not see or know about any changes done by VM)
> 
> But what you need is actually:
> ==============================
> 
>               VM 
>               |
>             [DRBD] -------- [DRBD] remote node
>             /
>  [logical volume]
> 
>  (DRBD sees every change done by the VM, and thus
>   has a chance to mirror the changes over).

I love ASCII art! Really, any diagram is good. Helps disperse the opacity.

In the interest of understanding how an average administrator can get
confused here: DRBD on setup offers to duplicate the data on the primary to
the secondary. That works. So it would seem (to the naive) like it sees the
data at that point, and has every opportunity to insert itself under the
data, per your 3rd diagram. In the case where that data was a normal
filesystem, that's what DRBD does, right? That's the point of the offer to
duplicate the existing data from primary to secondary? So with a normal
filesystem, you can have a filesystem with data, add DRBD, and DRBD slips
itself under the filesystem. With a VM written raw to a logical volume, DRBD
can see things well enough to handle the initial duplication, but fails to
slip itself under the VM.

As Chapter 5 of the manual says, "It is not necessary for this storage area
to be empty before you create a DRBD resource from it. In fact it is a
common use case to create a two-node cluster from a previously non-redundant
single-server system using DRBD," which to be fair continues "(some caveats
apply – please refer to the section called “DRBD meta data” if you are
planning to do this)." If I'd been reading that, rather than accepting a
default LCMC option based on a hazy memory of the manual, I'd have
considered external metadata. But it wouldn't be clear I simply shouldn't do
things in this order at all, if that's what you're saying now.

Which leads to a practical set of questions:

1. If I had used external metadata would this have worked?

2. If I were to reconfigure to external metadata now, would that fix it?

3. If I create new sets of LVM volumes, empty, and configure DRBD for them
   before cloning VMs into them, is this compatible with internal metadata,
   or do I need external metadata due to the way KVM-QEMU uses whole logical
   volumes?

Also, one point of ambiguity in your ASCII art: I think I understand you
correctly, but to be sure, I don't have DRBD running from within the VMs on
the logical volumes. It's running, of course, on the servers, with the
servers being, of course, beneath the VMs. So again, it seems that DRBD can
get under a VM-on-a-logical-volume to do the initial copy to a DRBD remote
node, but then subsequently fails. Something that initially works then stops
is more surprising to an admin (at least in the *nix world) than something
that just fails from the outset. All or nothing is more the norm.

Thanks,
Whit



More information about the drbd-user mailing list