[DRBD-user] Recover node after hard-crash (drbd9)

Christiaan den Besten chris at prolocation.net
Mon Jun 29 21:26:30 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi !

Just posted this on the drbd-dev list, but I think it might be better for drbd-users to share with others:

I just started testing the (official) drbd9 modules and tools on thee CentOS 7 VMx (running on Xen Hypervisor) in PV mode.

My DRBD ‘cluster’ has 3 nodes :

cluster1-a.storage.as41887.net
cluster1-b.storage.as41887.net
cluster1-c.storage.as41887.net

No actual Cluster software (pacemaker) has been installed just yet. Trying to get the hand of ‘drbdmanage’ and do some testing on the recoverability of resources after various scenario’s.

Scenario that I can’t seem to ‘fix’ (or am doing wrong … )

* Create a resource/volume through drbdmanage (puts it on node-b and node-c)
* Remove data from node-c and assign node-a as a secondary data node
* Make node-c a Diskless client node, create a filesystem on the drbd-volume and mount it

Then to see how we can can recover from a hard error I started a fio disk-benchmark on node-c and few seconds later kill the node-a by destroying the VM. All keeps running as expected, but I can’t seem to get node-a back into normal shape without having to remove te Primary state from node-c.

I followed to following steps, please advise where I went wrong :

IPv7:~ chris$ cat /tmp/drbd-replay.txt 

[root at cluster1-a ~]# drbdmanage new-volume testvm_prolocation_net 10GB --deploy 2
[root at cluster1-a ~]# drbdmanage nodes
+------------------------------------------------------------------------------------------------------------+
| Name                           | Pool Size | Pool Free |                                           | State |
+------------------------------------------------------------------------------------------------------------+
| cluster1-a.storage.as41887.net |    204796 |    204792 |                                           |    ok |
| cluster1-b.storage.as41887.net |    204796 |    195252 |                                           |    ok |
| cluster1-c.storage.as41887.net |    204796 |    195252 |                                           |    ok |
+------------------------------------------------------------------------------------------------------------+

[root at cluster1-a ~]# drbdmanage unassign testvm_prolocation_net cluster1-c.storage.as41887.net
[root at cluster1-a ~]# drbdmanage assign testvm_prolocation_net cluster1-a.storage.as41887.net
[root at cluster1-a ~]# drbdmanage assign --client testvm_prolocation_net cluster1-c.storage.as41887.net

[root at cluster1-a ~]# drbdmanage nodes
+------------------------------------------------------------------------------------------------------------+
| Name                           | Pool Size | Pool Free |                                           | State |
+------------------------------------------------------------------------------------------------------------+
| cluster1-a.storage.as41887.net |    204796 |    195252 |                                           |    ok |
| cluster1-b.storage.as41887.net |    204796 |    195252 |                                           |    ok |
| cluster1-c.storage.as41887.net |    204796 |    204792 |                                           |    ok |
+------------------------------------------------------------------------------------------------------------+

Affer a few minutes (node-a needs to sync data from node-b) :

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:UpToDate
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate
 cluster1-c.storage.as41887.net role:Secondary
   peer-disk:Diskless

[root at cluster1-c ~]# mkfs.ext4 /dev/drbd10 
[root at cluster1-c ~]# mount /dev/drbd10 /mnt

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:UpToDate
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate
 cluster1-c.storage.as41887.net role:Primary
   peer-disk:Diskless

[root at cluster1-c mnt]# fio /usr/share/doc/fio-2.2.8/examples/iometer-file-access-server.fio

To fake a nice hard crash of one of the data-nodes : xm destroy cluster1-a.storage.as41887.net

Before booting cluster1-a :

[root at cluster1-c mnt]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Primary
 disk:Diskless
 cluster1-a.storage.as41887.net connection:Connecting
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate

After booting cluster1-a, now the fun starts ;)

[root at cluster1-c mnt]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Primary
 disk:Diskless
 cluster1-a.storage.as41887.net connection:StandAlone
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:Outdated
 cluster1-b.storage.as41887.net connection:StandAlone
 cluster1-c.storage.as41887.net connection:Connecting

To resolve our 'split brain' I discard the data on node-a

[root at cluster1-a ~]# drbdadm --discard-my-data connect testvm_prolocation_net

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:Outdated
 cluster1-b.storage.as41887.net connection:Connecting
 cluster1-c.storage.as41887.net connection:Connecting

[root at cluster1-b ~]# drbdadm connect testvm_prolocation_net

Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Handshake successful: Agreed network protocol version 110
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Agreed to support TRIM on protocol level
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Peer authenticated using 20 bytes HMAC
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting ack_recv thread (from drbd_r_testvm_p [2297])
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Preparing remote state change 2288137045 (primary_nodes=0, weak_nodes=0)
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Committing remote state change 2288137045
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: drbd_sync_handshake:
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: self BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:0
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: peer 374B6E4C8ECEB4F4:2CA8DD9FBE1E313E:0000000000000000:0000000000000000 bits:225445 flags:100
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: uuid_compare()=-100 by rule 100
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain
Jun 29 13:47:52 cluster1-a drbdadm[2320]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain exit code 0 (0x0)
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: Split-Brain detected, manually solved. Sync from peer node
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 82229(21), total 82229; compression: 73.1%
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 82229(21), total 82229; compression: 73.1%
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target
Jun 29 13:47:52 cluster1-a drbdadm[2322]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: disk( Outdated -> Inconsistent )
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( WFBitMapT -> SyncTarget )
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( no -> connection dependency )
Jun 29 13:47:52 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Began resync as SyncTarget (will sync 901780 KB [225445 bits set]).

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:Inconsistent
 cluster1-b.storage.as41887.net role:Secondary
   replication:SyncTarget peer-disk:UpToDate done:91.61
 cluster1-c.storage.as41887.net connection:Connecting

end up in ... :

Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Resync done (total 56 sec; paused 0 sec; 16100 K/sec)
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Peer was unstable during resync
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: updated UUIDs BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( SyncTarget -> Established )
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( connection dependency -> no )
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target
Jun 29 13:48:48 cluster1-a drbdadm[2393]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:48:48 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0)

[root at cluster1-a ~]# drbdadm status | grep 'testvm_prolocation_net' -A 6
testvm_prolocation_net role:Secondary
 disk:Inconsistent
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate
 cluster1-c.storage.as41887.net connection:Connecting

To get it going I need to unmount the mountpoint on the Diskless DRBD client node-c.

[root at cluster1-c ~]# umount /mnt

[root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Secondary
 disk:Diskless
 cluster1-a.storage.as41887.net connection:StandAlone
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate

[root at cluster1-a ~]# drbdadm disconnect testvm_prolocation_net

[root at cluster1-b ~]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Secondary
 disk:UpToDate
 cluster1-a.storage.as41887.net connection:Connecting
 cluster1-c.storage.as41887.net role:Secondary
   peer-disk:Diskless

[root at cluster1-a ~]# drbdadm --discard-my-data connect testvm_prolocation_net

Jun 29 13:54:37 cluster1-a kernel: drbd testvm_prolocation_net tcp:cluster1-c.storage.as41887.net: Closing unexpected connection from 94.228.142.34 to port 7700
Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( StandAlone -> Unconnected )
Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting receiver thread (from drbd_w_testvm_p [1330])
Jun 29 13:54:44 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Unconnected -> Connecting )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Handshake successful: Agreed network protocol version 110
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Agreed to support TRIM on protocol level
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Peer authenticated using 20 bytes HMAC
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Starting ack_recv thread (from drbd_r_testvm_p [2481])
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Preparing remote state change 439073605 (primary_nodes=0, weak_nodes=0)
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: Committing remote state change 439073605
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net cluster1-b.storage.as41887.net: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: drbd_sync_handshake:
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: self BF8F65EBAABAF138:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:0
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: peer 374B6E4C8ECEB4F4:2CA8DD9FBE1E313E:0000000000000000:0000000000000000 bits:2 flags:120
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: uuid_compare()=-100 by rule 100
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain
Jun 29 13:54:45 cluster1-a drbdadm[2485]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm initial-split-brain exit code 0 (0x0)
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: Split-Brain detected, manually solved. Sync from peer node
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target
Jun 29 13:54:45 cluster1-a drbdadm[2487]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( WFBitMapT -> SyncTarget )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: resync-susp( no -> connection dependency )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Began resync as SyncTarget (will sync 8 KB [2 bits set]).
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: Resync done (total 1 sec; paused 0 sec; 8 K/sec)
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: updated UUIDs 374B6E4C8ECEB4F4:0000000000000000:BF8F65EBAABAF138:406F8167C11BEB40
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10: disk( Inconsistent -> UpToDate )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: repl( SyncTarget -> Established )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-c.storage.as41887.net: pdsk( DUnknown -> Outdated ) resync-susp( connection dependency -> no )
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target
Jun 29 13:54:45 cluster1-a drbdadm[2489]: Don't know which config file belongs to resource testvm_prolocation_net, trying default ones...
Jun 29 13:54:45 cluster1-a kernel: drbd testvm_prolocation_net/0 drbd10 cluster1-b.storage.as41887.net: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0)

[root at cluster1-a ~]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Secondary
 disk:UpToDate
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate
 cluster1-c.storage.as41887.net connection:Connecting

Now I still need to re-connect node-c to the recovered node-a

[root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Secondary
 disk:Diskless
 cluster1-a.storage.as41887.net connection:StandAlone
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate

[root at cluster1-c ~]# drbdadm connect testvm_prolocation_net
testvm_prolocation_net: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup connect testvm_prolocation_net 0' terminated with exit code 10

[root at cluster1-c ~]# drbdadm connect testvm_prolocation_net --peer cluster1-a.storage.as41887.net
testvm_prolocation_net: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup connect testvm_prolocation_net 0' terminated with exit code 10

[root at cluster1-c ~]# drbdadm disconnect testvm_prolocation_net
[root at cluster1-c ~]# drbdadm connect testvm_prolocation_net
[root at cluster1-c ~]# drbdadm status | grep testvm_prolocation_net -A 6
testvm_prolocation_net role:Secondary
 disk:Diskless
 cluster1-a.storage.as41887.net role:Secondary
   peer-disk:UpToDate
 cluster1-b.storage.as41887.net role:Secondary
   peer-disk:UpToDate

Finally, all is synced and well connected again ….

Above steps where a replay I did today after having seen this behaviour last night. So it seems to be reproducible. Also wondering if the ‘recover’ steps should be done though drbdmanage as well? Can’t seem to find any complete documentation on drbdmanage besides the man-page.

Yours,
Chris


More information about the drbd-user mailing list