Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi ! Just posted this on the drbd-dev list, but I think it might be better for drbd-users to share with others: I just started testing the (official) drbd9 modules and tools on thee CentOS 7 VMx (running on Xen Hypervisor) in PV mode. My DRBD ‘cluster’ has 3 nodes : cluster1-a.storage.as41887.net cluster1-b.storage.as41887.net cluster1-c.storage.as41887.net No actual Cluster software (pacemaker) has been installed just yet. Trying to get the hand of ‘drbdmanage’ and do some testing on the recoverability of resources after various scenario’s. Scenario that I can’t seem to ‘fix’ (or am doing wrong … ) * Create a resource/volume through drbdmanage (puts it on node-b and node-c) * Remove data from node-c and assign node-a as a secondary data node * Make node-c a Diskless client node, create a filesystem on the drbd-volume and mount it Then to see how we can can recover from a hard error I started a fio disk-benchmark on node-c and few seconds later kill the node-a by destroying the VM. All keeps running as expected, but I can’t seem to get node-a back into normal shape without having to remove te Primary state from node-c. I followed to following steps, please advise where I went wrong : IPv7:~ chris$ cat /tmp/drbd-replay.txt [root at cluster1-a ~]# drbdmanage new-volume testvm_prolocation_net 10GB --deploy 2 [root at cluster1-a ~]# drbdmanage nodes +------------------------------------------------------------------------------------------------------------+ | Name | Pool Size | Pool Free | | State | +------------------------------------------------------------------------------------------------------------+ | cluster1-a.storage.as41887.net | 204796 | 204792 | | ok | | cluster1-b.storage.as41887.net | 204796 | 195252 | | ok | | cluster1-c.storage.as41887.net | 204796 | 195252 | | ok | +------------------------------------------------------------------------------------------------------------+ [root at cluster1-a ~]# drbdmanage unassign testvm_prolocation_net cluster1-c.storage.as41887.net [root at cluster1-a ~]# drbdmanage assign testvm_prolocation_net cluster1-a.storage.as41887.net [root at cluster1-a ~]# drbdmanage assign --client testvm_prolocation_net cluster1-c.storage.as41887.net [root at cluster1-a ~]# drbdmanage nodes +------------------------------------------------------------------------------------------------------------+ | Name | Pool Size | Pool Free | | State | +------------------------------------------------------------------------------------------------------------+ | cluster1-a.storage.as41887.net | 204796 | 195252 | | ok | | cluster1-b.storage.as41887.net | 204796 | 195252 | | ok | | cluster1-c.storage.as41887.net | 204796 | 204792 | | ok | +------------------------------------------------------------------------------------------------------------+ Affer a few minutes (node-a needs to sync data from node-b) : [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:UpToDate cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate cluster1-c.storage.as41887.net role:Secondary peer-disk:Diskless [root at cluster1-c ~]# mkfs.ext4 /dev/drbd10 [root at cluster1-c ~]# mount /dev/drbd10 /mnt [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:UpToDate cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate cluster1-c.storage.as41887.net role:Primary peer-disk:Diskless [root at cluster1-c mnt]# fio /usr/share/doc/fio-2.2.8/examples/iometer-file-access-server.fio To fake a nice hard crash of one of the data-nodes : xm destroy cluster1-a.storage.as41887.net Before booting cluster1-a : [root at cluster1-c mnt]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Primary disk:Diskless cluster1-a.storage.as41887.net connection:Connecting cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate After booting cluster1-a, now the fun starts ;) [root at cluster1-c mnt]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Primary disk:Diskless cluster1-a.storage.as41887.net connection:StandAlone cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:Outdated cluster1-b.storage.as41887.net connection:StandAlone cluster1-c.storage.as41887.net connection:Connecting To resolve our 'split brain' I discard the data on node-a [root at cluster1-a ~]# drbdadm --discard-my-data connect testvm_prolocation_net [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:Outdated cluster1-b.storage.as41887.net connection:Connecting cluster1-c.storage.as41887.net connection:Connecting [root at cluster1-b ~]# drbdadm connect testvm_prolocation_net Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Handshake successful: Agreed network protocol version 110 Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Agreed to support TRIM on protocol level Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Peer authenticated using 20 bytes HMAC Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting ack_recv thread (from drbd_r_testvm_p [2297]) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Preparing remote state change 2288137045 (primary_nodes=0, weak_nodes=0) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Committing remote state change 2288137045 Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: drbd_sync_handshake: Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: self BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:0 Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: peer 374B6E4C8ECEB4F4:2CA8DD9FBE1E313E:0000000000000000:0000000000000000 bits:225445 flags:100 Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: uuid_compare()=-100 by rule 100 Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain Jun 29 13:47:52 cluster1-a drbdadm[2320]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain exit code 0 (0x0) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: Split-Brain detected, manually solved. Sync from peer node Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 82229(21), total 82229; compression: 73.1% Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 82229(21), total 82229; compression: 73.1% Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target Jun 29 13:47:52 cluster1-a drbdadm[2322]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: disk( Outdated -> Inconsistent ) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( WFBitMapT -> SyncTarget ) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( no -> connection dependency ) Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Began resync as SyncTarget (will sync 901780 KB [225445 bits set]). [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:Inconsistent cluster1-b.storage.as41887.net role:Secondary replication:SyncTarget peer-disk:UpToDate done:91.61 cluster1-c.storage.as41887.net connection:Connecting end up in ... : Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Resync done (total 56 sec; paused 0 sec; 16100 K/sec) Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Peer was unstable during resync Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: updated UUIDs BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000 Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( SyncTarget -> Established ) Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( connection dependency -> no ) Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target Jun 29 13:48:48 cluster1-a drbdadm[2393]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0) [root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6 testvm_prolocation_net role:Secondary disk:Inconsistent cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate cluster1-c.storage.as41887.net connection:Connecting To get it going I need to unmount the mountpoint on the Diskless DRBD client node-c. [root at cluster1-c ~]# umount /mnt [root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Secondary disk:Diskless cluster1-a.storage.as41887.net connection:StandAlone cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate [root at cluster1-a ~]# drbdadm disconnect testvm_prolocation_net [root at cluster1-b ~]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Secondary disk:UpToDate cluster1-a.storage.as41887.net connection:Connecting cluster1-c.storage.as41887.net role:Secondary peer-disk:Diskless [root at cluster1-a ~]# drbdadm --discard-my-data connect testvm_prolocation_net Jun 29 13:54:37 cluster1-a kernel: drbd testvm_prolocation_net tcp:cluster1-c.storage.as41887.net: Closing unexpected connection from 94.228.142.34 to port 7700 Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( StandAlone -> Unconnected ) Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting receiver thread (from drbd_w_testvm_p [1330]) Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Unconnected -> Connecting ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Handshake successful: Agreed network protocol version 110 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Agreed to support TRIM on protocol level Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Peer authenticated using 20 bytes HMAC Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting ack_recv thread (from drbd_r_testvm_p [2481]) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Preparing remote state change 439073605 (primary_nodes=0, weak_nodes=0) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Committing remote state change 439073605 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: drbd_sync_handshake: Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: self BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:0 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: peer 374B6E4C8ECEB4F4:2CA8DD9FBE1E313E:0000000000000000:0000000000000000 bits:2 flags:120 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: uuid_compare()=-100 by rule 100 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain Jun 29 13:54:45 cluster1-a drbdadm[2485]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain exit code 0 (0x0) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: Split-Brain detected, manually solved. Sync from peer node Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0% Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0% Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target Jun 29 13:54:45 cluster1-a drbdadm[2487]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( WFBitMapT -> SyncTarget ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( no -> connection dependency ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Began resync as SyncTarget (will sync 8 KB [2 bits set]). Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Resync done (total 1 sec; paused 0 sec; 8 K/sec) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: updated UUIDs 374B6E4C8ECEB4F4:0000000000000000:BF8F65EBAABAF138:406F8167C11BEB40 Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: disk( Inconsistent -> UpToDate ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( SyncTarget -> Established ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: pdsk( DUnknown -> Outdated ) resync-susp( connection dependency -> no ) Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target Jun 29 13:54:45 cluster1-a drbdadm[2489]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones... Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0) [root at cluster1-a ~]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Secondary disk:UpToDate cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate cluster1-c.storage.as41887.net connection:Connecting Now I still need to re-connect node-c to the recovered node-a [root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Secondary disk:Diskless cluster1-a.storage.as41887.net connection:StandAlone cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate [root at cluster1-c ~]# drbdadm connect testvm_prolocation_net testvm_prolocation_net: Failure: (125) Device has a net-config (use disconnect first) Command 'drbdsetup connect testvm_prolocation_net 0' terminated with exit code 10 [root at cluster1-c ~]# drbdadm connect testvm_prolocation_net --peer cluster1-a.storage.as41887.net testvm_prolocation_net: Failure: (125) Device has a net-config (use disconnect first) Command 'drbdsetup connect testvm_prolocation_net 0' terminated with exit code 10 [root at cluster1-c ~]# drbdadm disconnect testvm_prolocation_net [root at cluster1-c ~]# drbdadm connect testvm_prolocation_net [root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6 testvm_prolocation_net role:Secondary disk:Diskless cluster1-a.storage.as41887.net role:Secondary peer-disk:UpToDate cluster1-b.storage.as41887.net role:Secondary peer-disk:UpToDate Finally, all is synced and well connected again …. Above steps where a replay I did today after having seen this behaviour last night. So it seems to be reproducible. Also wondering if the ‘recover’ steps should be done though drbdmanage as well? Can’t seem to find any complete documentation on drbdmanage besides the man-page. Yours, Chris