[DRBD-user] LVM locking backing store so DRBD comes online "Diskless"

[Will McDermott]

Hello,

I am running DRBD in dual primary with LVM2 on top (for virtual machine failover). When I power off/reboot one of the nodes DRBD comes back "Diskless"

root at forge:~# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil at fat-tyre, 2011-01-28 12:17:35
0: cs:Connected ro:Primary/Primary ds:Diskless/UpToDate C r-----
   ns:0 nr:2168 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
   ns:0 nr:0 dw:0 dr:2512 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

When I try to re-attach to the backing store I get this:

root at forge:~# drbdadm attach prod
0: Failure: (104) Can not open backing device.
Command 'drbdsetup 0 disk /dev/sdb1 /dev/sdb1 internal --set-defaults --create-device' terminated with exit code 10

I noticed that LVM is locking an LV on the disk:

root at forge:~# dmsetup ls
pve-swap        (253:2)
pve-root        (253:0)
pve-data        (253:3)
prod.drbd-vm--102--disk--1      (253:1)

If I remove it:

root at forge:~# dmsetup remove prod.drbd-vm--102--disk--1

I can re-attach

root at forge:~# drbdadm attach prod
root at forge:~#

And DRBD goes back to normal

root at forge:~# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil at fat-tyre, 2011-01-28 12:17:35
0: cs:WFBitMapT ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
   ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:376
1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
   ns:0 nr:0 dw:0 dr:3120 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0


I have edited the filters in /etc/lvm/lvm.conf to:
filter = [ "r|/dev/sdb*|", "a/.*/" ]

I'm not sure if that's right. Any Ideas how I can clean this up? I have googled for a couple days and found similar problems with no one with answers.

Here is my DRBD conf:
root at forge:~# cat /etc/drbd.d/prod.res
resource prod {
 protocol C;
 startup {
   wfc-timeout  15;
   degr-wfc-timeout 60;
   become-primary-on both;
 }
 net {
   cram-hmac-alg sha1;
   shared-secret "XXXXXXX";
   allow-two-primaries;
   after-sb-0pri discard-zero-changes;
   after-sb-1pri discard-secondary;
   after-sb-2pri disconnect;
 }
 on forge {
   device      /dev/drbd0;
   disk        /dev/sdb1;
   address     XXXXXX:7788;
   meta-disk   internal;
 }
 on ironworks {
   device      /dev/drbd0;
   disk        /dev/sdb1;
   address     XXXXXX:7788;
   meta-disk   internal;
 }
}

Thanks!

--Will