Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
We have a 5 node DRDB9 test cluster with one resource named 'monitoring'. We 'drbdmanage remove-node'd the node which was in primary role for 'monitoring' (but not primary for the DRBD9 'control volume').
We'd expected that afterwards it would be possible to simply go on with the 4 remaining nodes, meaning just mounting 'monitoring' on another node and be happy, or otherwise drbdmanage complain about this being a bad idea etc.
Now it's not possible to mount the resource on another node ('write-protected, mounting read-only', 'failed: Wrong medium type' etc).
'drbdmanage list-nodes' shows 'pending actions: adjust connections' for every node except the leader since more than an hour now.
'drbdsetup events2' shows:
exists device name:monitoring volume:0 minor:100 disk:Outdated client:no
exists peer-device name:monitoring peer-node-id:4 conn-name:d95 volume:0 replication:Established peer-disk:Outdated peer-client:no resync-suspended:no
exists peer-device name:monitoring peer-node-id:0 conn-name:d91 volume:0 replication:Established peer-disk:Outdated peer-client:no resync-suspended:no
exists peer-device name:monitoring peer-node-id:2 conn-name:d93 volume:0 replication:Established peer-disk:Outdated peer-client:no resync-suspended:no
A try to mount 'monitoring' leads to this in syslog:
drbd monitoring: State change failed: Need access to UpToDate data
drbd monitoring: Failed: role( Secondary -> Primary )
drbd monitoring: Auto-promote failed: Need access to UpToDate data
(Why doesn't it have 'UpToDate' data? Cluster situation seemed to be OK before our action...)
It doesn't seem that DRBD solves the situation on it's own, as this state didn't change for quite a while now. A 'drbdmanage remove-node -f' didn't help either.
Q1: how to recover from this situation?
Q2: what's the recommended way to remove a node, given the described situation?
Thanks in advance...