[DRBD-user] drbd and LVM -> always getting duplicate volumes (was: can't use LVM2 on drbd devices)

Tomasz Chmielewski mangoo at wpkg.org
Mon Jan 14 18:11:38 CET 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Lars Ellenberg schrieb:
> On Mon, Jan 14, 2008 at 05:29:01PM +0100, Tomasz Chmielewski wrote:
>> After a couple of days of struggling I got to the point where I have to 
>> conclude that using LVM reliably over DRBD devices is impossible.
> after a couple of seconds of contemplating I got to the point where I
> have to conclude that this would mean that a non-negligible number of
> our managed production deployments are impossible, there for non
> existent.  I wonder why I hear no customer complaints, though.
>  ;)

Did any of them use dm-crypt? :)

>> 7. pvs confirms it - LVM sits on /dev/sdb, not on /dev/drbd0 - 
>> certainly, nothing will be replicated by DRBD:
> as pointed out earlier, you have to fix your lvm conf, specifically the
> filter setting.  if you don't, it won't work.

That I know.
How I don't :|

>> As device names are not necessarily persistent (if /dev/sdb is a iSCSI 
>> drive, next time one boots it can be /dev/sdc) and a similar is for 
>> device mapper devices (think of crypted devices), setting up filters in 
>> lvm.conf will not be reliable.
> you did hear about udev and scsi id, did you?

For iSCSI, true, I could use a link like /dev/iscsi/blah -> /dev/sdX and 
put that /dev/iscsi/blah as a filter for lvm.conf.

dm-crypt devices don't have any ID or anything unique (that I know).

One way to overcome seem to do all these steps during the system startup:

1. Put a filter like /etc/drbd-disk-dont-touch-AB to lvm.conf

2. On system startup, find out which one of /dev/dm-XY devices is our 
crypted device. How to do it? If the device has unique size, just parse 
the output of fdisk -l. If there are multiple devices with the same 
size, it gets problematic.

3. Point /etc/drbd-disk-dont-touch-AB symlink to the correct /dev/dm-XY 
device - this way, it would be exluded for LVM.

All that makes the things a bit crude and error-prone, especially when 
it's possible that there are more devices with the same size, which have 
to be treated differently.

Or, is there an easier way?

>> How can I use LVM over DRBD devices *reliably*?
> you make your device nodes persistent.
> how to do that is detailed in depth elsewhere.
> this problem generic, and not specific to drbd at all.

It is, a bit.

For example, md (Linux software RAID) can store its metadata (called 
superblock) at different locations on the device (at start, or at the 
end, or 4K from the start).

This way, by choosing a correct metadata format, one can be sure of data 
integrity - with metadata at start, mounting or using an underlying 
device accidentally is impossible.

Storing DRBD metadata at the beginning of the device would effectively 
make some administration tasks easier (no need to write custom scripts 
for detecting correct dm-XY device) and/or more reliable (not possible 
to mount or use a device accidentally) - this thread wouldn't have to 
exist I guess (and some of split-brain posts as well, I guess).

Just a thought.

Tomasz Chmielewski

More information about the drbd-user mailing list