[DRBD-user] Detection of Logical Volume on nested LVM

Olivier LAMBERT lambert.olivier at gmail.com
Tue Mar 9 10:59:54 CET 2010

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

I've got your point. I know my first message is long and maybe
complicated. It seems you "jump" some lines. It's not "nested" LVM as
described in the doc (because I RTFM before try to ask help).

My problem is not located on DRBD hosts. I've run some test, and I can
really simplify the example. Please, read this, I think it's clear now
Let's say I export a simple physical volume with DRBD. So, NO LVM in
DRBD, just replicate "/dev/sdb" for example, and export it
(/dev/drbd0) on each DRBD host with iSCSI.

So, on ALL my clients (connected iSCSI), I saw with "fdisk -l" :
Disk /dev/sdb: 2998.8 GB, 2998869356544 bytes

So, my CLIENT1 connected to DRBD1, decide to create a volume group and
a logical volume.
Despite all my efforts (vgscan, pvscan, lvscan), CLIENT2 connected to
replicated DRBD2, cannot see anything ! No VG, no LV ! Nothing !

But, if I decide to do that on DRBD2 :
drbdadm disconnect ressource && drbdadm connect ressource

Just after THIS, I can see all the stuff I've created on CLIENT1.

So, it's NOT a question of LVM on DRBD, because in this case, I don't
use it anymore. I can undestand if LVM data's on client are not
replicated on each DRBD hosts, but WHY it works if I
disconnect/reconnect these two ? It doesn't have any sense to me....

Thanks for your help,


XO Project

On Sun, Mar 7, 2010 at 11:47 PM, Maros Timko <timkom at gmail.com> wrote:
> 2010/3/7 Olivier LAMBERT <lambert.olivier at gmail.com>:
>> Hello,
>> cLVM doesn't provide snapshots :(
> cLVM has nothing to do with the LVM snapshots.
>> But you say my behaviour is "normal" ? so, why lvcreate/lvremove is
>> not replicated ? it's not a "block" operation for the bottom layer ?
>> And if I export a physical volume (and not a LVM one) and then after
>> on iSCSI volume I do "lvcreate" does it solve the problem ?
> Your issue is where the LVM metadata is stored and how the OS could
> refresh the changes (if any).
> I would suggest start here, as the guys from LINBIT are really sharing
> a lot of their know-how, but it gets pretty annoying to answer the
> same questions every couple of months:
> http://www.drbd.org/users-guide/s-nested-lvm.html
> If this will not explain it exactly, try to search on DRBD, Pacemaker
> and Heartbeat user mailing lists with keyword cLVM. I am pretty sure
> you will find a definite answer.
> Or try to check what cLVM does and why was it created.
>> Regards,
>> Olivier
>> XO Project
>> http://xen-orchestra.com
>> On Sat, Mar 6, 2010 at 6:01 PM, Maros Timko <timkom at gmail.com> wrote:
>>>> Message: 1
>>>> Date: Thu, 4 Mar 2010 17:48:24 +0100
>>>> From: Olivier LAMBERT <lambert.olivier at gmail.com>
>>>> Subject: [DRBD-user] Detection of Logical Volume on nested LVM
>>>>        replication
>>>> To: drbd-user at lists.linbit.com
>>>> Message-ID:
>>>>        <f769216c1003040848m4b1ffaaatc15dc952781ce946 at mail.gmail.com>
>>>> Content-Type: text/plain; charset=ISO-8859-1
>>>> Hello,
>>>> I'll try to be clear, but it's really hard to explain..
>>>> In few words : when I create a Logical Volume on a host attached
>>>> (iSCSI) to one DRBD server, it doesn't appear on the second host,
>>>> attached to the other DRBD. But, on existing LV, it works like a charm
>>>> (all data's are replicated). And more, if I create a LV on "one side",
>>>> and fill it with data, I saw the replication link with heavy traffic,
>>>> but always no LV.
>>>> Now, to understand, here is how it happened with this infrastructure.
>>>> 1) On both DRBD hosts : Debian GNU/Linux, Dual Primary setup, version:
>>>> 8.3.7 (api:88/proto:86-91). I choose to configure a resource on a
>>>> Logical volume of 500Gb (named /dev/vg0/xen). To be clear, two hosts
>>>> are "DRBD1" & "DRBD2". a cat /proc/drbd gave this :
>>>>  0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
>>>>    ns:153716 nr:408752 dw:562408 dr:355156 al:179 bm:109 lo:0 pe:0
>>>> ua:0 ap:0 ep:1 wo:n oos:0
>>>> 2) This resource is exported with iSCSI. For the example, let's say
>>>> that CLIENT1 is connected to DRBD1, and CLIENT2 is connected to DRBD2.
>>>> It's simplifed because in real, it's multiple Xen Dom0's clients with
>>>> multipath (but we don't care of that here).
>>>> 3) CLIENT1 see the device as a block device (so far so good, iSCSI
>>>> works). I choose to use LVM on this block device. I create for example
>>>> : /dev/vg_xen/mydisk on CLIENT1. I mount it, I put some stuff in there
>>>> : I saw the replication like working : some traffic. DRBD2 and 1 says
>>>> it's all OK. If I lvscan on CLIENT1, I can see my brand new volume.
>>>> 4) CLIENT2, (so, connected on iSCSI with DRBD2), see the block device,
>>>> the volume group, but NOT the Logical Volume. If I disconnect DRBD2's
>>>> resource, and reconnect it, and reconnect iSCSI of CLIENT2, wow, I saw
>>>> the LV !
>>>> And more : if the LV exists on both side (after disconnect/reconnect
>>>> the resource), data's are correctly replicated (obviously, I do NOT
>>>> mount LV on both side, I'm aware of that !). But, if I fill on one
>>>> side (e.g CLIENT1), dismount it, then mount it on CLIENT2, data are
>>>> here, without any problem.
>>>> So, my "theory", is that LVM operations (lvcreate or lvremove) on a
>>>> volume group, which is on top of iSCSI and LVM replicated device by
>>>> DRBD, are NOT replicated, UNTIL disconnect/reconnect the ressource. I
>>>> don't know why, and that's why I ask here to understand what I miss.
>>> Right. Discussed many times here and on Heartbeat/Pacemaker list. Try
>>> to search with cLVM keyword.
>>>> Additional Informations: If my clients are connected to the SAME DRBD
>>>> (let's tell DRBD1), if CLIENT1 creates a LV, CLIENT2 is immediatly
>>>> aware of that (just inactive, not a problem, vgchange and it works).
>>>> So the "problem" is during the replication.
>>>> Thanks for your help.
>>>> Regards,
>>>> Olivier
>>>> XO Project
>>>> http://xen-orchestra.com
>>> _______________________________________________
>>> drbd-user mailing list
>>> drbd-user at lists.linbit.com
>>> http://lists.linbit.com/mailman/listinfo/drbd-user

More information about the drbd-user mailing list