[DRBD-user] Understanding "classic" 3 node set up.

Paul O'Rorke paul at tracker-software.com
Mon Feb 5 19:27:31 CET 2018

Moving ahead with the assumptions before and I have defined the stacked 
resource on node "alice" (trk-kvm-01), created the drbd block device:

    root at trk-kvm-01:~# drbdadm create-md --stacked convirt-U
    md_offset 53685411840
    al_offset 53685379072
    bm_offset 53683740672

    Found some data

      ==> This might destroy existing data! <==

    Do you want to proceed?
    [need to type 'yes' to confirm] yes

    initializing activity log
    NOT initializing bitmap
    Writing meta data...
    New drbd meta data block successfully created.

It comes up OK:

    root at trk-kvm-01:~# drbdadm up --stacked convirt-U

    root at trk-kvm-01:~# drbd-overview
      12:convirt/0                Connected  Primary/Secondary
    UpToDate/UpToDate     _convirt           vda virtio
    120:convirt-U/0         ^^12 Connected  Secondary/Secondary

I made it the primary

    root at trk-kvm-01:~# drbdadm primary --force --stacked convirt-U

It seems OK on this node:

    root at trk-kvm-01:~# drbd-overview
      12:convirt/0                Connected Primary/Secondary  
    UpToDate/UpToDate     _convirt vda virtio
    120:convirt-U/0         ^^12 Connected Primary/Secondary  

However when I try to create the drbd block device on the third node I 
get the following:

    root at trk-kvm-03:~# drbdadm create-md convirt-U
    open(inetrnal) failed: No such file or directory
    open(inetrnal) failed: No such file or directory
    Command 'drbdmeta 120 v08 inetrnal flex-external create-md'
    terminated with exit code 20

What file is it looking for that it fails to find?

The resource is defined with this:

    resource convirt {
             net {
                     protocol C;

             device    /dev/drbd12;
             meta-disk internal;

             on trk-kvm-01 {
             disk      /dev/trk-kvm-01-vg/convirt;
             on trk-kvm-02 {
             disk      /dev/trk-kvm-02-vg/convirt;

    resource convirt-U {
             net {
                     protocol A;

             stacked-on-top-of convirt {
                     device /dev/drbd120;

             on trk-kvm-03 {
                     device          /dev/drbd120;
                     disk /dev/trk-kvm-03-vg/convirt;
                     meta-disk       inetrnal;

The underlying LVM volume does exist:

    root at trk-kvm-03:~# lvs
       LV                VG            Attr       LSize Pool Origin
    Data%  Meta%  Move Log Cpy%Sync Convert
       convirt           trk-kvm-03-vg -wi-a----- 50.00g

I tried removing the volume with lvremove:

    root at trk-kvm-03:~# lvremove /dev/trk-kvm-03-vg/convirt
    Do you really want to remove and DISCARD active logical volume
    convirt? [y/n]: y
       Logical volume "convirt" successfully removed

and creating it again with lvcreate:

    root at trk-kvm-03:~# lvcreate -L50G -n convirt trk-kvm-03-vg
       Logical volume "convirt" created.

it looks OK

    --- Logical volume ---
       LV Path                /dev/trk-kvm-03-vg/convirt
       LV Name                convirt
       VG Name                trk-kvm-03-vg
       LV UUID QT7YNm-zlYi-7S0d-lF26-EEi2-7g69-0iU3aC
       LV Write Access        read/write
       LV Creation host, time trk-kvm-03, 2018-02-05 09:03:56 -0800
       LV Status              available
       # open                 0
       LV Size                50.00 GiB
       Current LE             12800
       Segments               1
       Allocation             inherit
       Read ahead sectors     auto
       - currently set to     256
       Block device           252:13

Why can I not make the DRBD block device?  I see that some required file 
is missing but I don't understand what:

    root at trk-kvm-03:~# drbd-overview
    120:convirt-U/0  . . .


    root at trk-kvm-03:~# cat /proc/drbd
    version: 8.4.5 (api:1/proto:86-101)
    srcversion: 4B3E2E2CD48CAE5280B5205

    120: cs:Unconfigured

Something simple I have missed/misunderstood?

*Paul O'Rorke*
*Tracker Software Products (Canada) Limited *
www.tracker-software.com <http://www.tracker-software.com/>
Tel: +1 (250) 324 1621
Fax: +1 (250) 324 1623


Download latest Releases

On 2018-02-02 11:31 AM, Paul O'Rorke wrote:
> Hi
> I hope this is the right place to ask what may be a silly question.  I 
> have searched the archives and still do not understand properly how to 
> set up "classic 3 node" for DR.
> I have been using a simple HA setup for some years now for KVM virtual 
> machines.  I have a DRBD resource for each VM, backed by LVM volumes, 
> each using a unique IP (aliases) via CAT5 directly connecting 
> dedicated NICs on each of "alice" and "bob" (I'm using the example 
> from the docs here for simplicity).  I run half the guests on each 
> host and manually fail over if it is ever needed.  This has been rock 
> solid for me, great performance and has on more than one occasion 
> saved my bacon.  I want to set up a classic 3 node DR with a VPN 
> connected off site 3rd stacked node.
> Referring to this 
> http://docs.linbit.com/docs/users-guide-8.4/#s-three-nodes 6.18.2. 
> Configuring a stacked resource
> I don't understand where to set up the interface for the stacked resource:
> |stacked-on-top-of r0 { device /dev/drbd10; address; }|
> What interface gets this IP?  I am assuming it goes on the primary 
> node of the lower resource.  So for the example in the docs, if alice 
> is the primary I set up on alice and if I have to 
> switch bob to primary I loose replication to charlie until such time 
> as I either restore alice as the primary or can I set up 
> on bob somehow and continue off site replication 
> while alice is down?
> As I am reading it bob has no "knowledge" of charlie and the upper 
> resource can only function while alice is primary.  Is that right?
> Sorry if this is explained somewhere and I missed it.
> regards
> -- 
> *Paul O'Rorke*
> *Tracker Software Products (Canada) Limited *
> www.tracker-software.com <http://www.tracker-software.com/>
> Tel: +1 (250) 324 1621
> Fax: +1 (250) 324 1623
> <http://www.tracker-software.com/>
> Support:
> http://www.tracker-software.com/support
> Download latest Releases
> http://www.tracker-software.com/downloads/
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20180205/002463b2/attachment.htm>

More information about the drbd-user mailing list