[DRBD-user] Resource Level Fencing Issue: DRBD9 with pacemaker/corosync
Takashi Sogabe
tasogabe at yahoo-corp.jp
Mon Jun 24 09:26:53 CEST 2019
Hello all,
I encountered an issue when I tested behavior of resource level fencing with
pacemaker/corosync. It seems something is wrong with handling of fencing
events in case of quiescent situation.
I am a newbie of DRBD, so I followed the procedure from LINBIT documentations[1].
I would be great if DRBD guys could let me know mis-configuration
if my configuration was wrong. Or, are there any potential issues around there?
App packages:
- OS
- Ubuntu 18.04LTS
- DRBD packages
- drbd-utils
- 9.10.0-1ppa1~bionic1
- drbd-dkms
- 9.0.19~rc1-1ppa1~bionic1
- Pacemaker / Corosync
- pacemaker
- 1.1.18-0ubuntu1.1
- corosync
- 2.4.3-0ubuntu1.1
Network topology:
+--------------+ <- (Redundancy for resource level fencing)
node1 node2
| |
+-------+------+ <- (DRBD data, NFS data)
|
node3
- node1
- DRBD Primary
- NFS Server
- node2
- DRBD Secondary
- NFS Server
- node3
- NFS Client
Description of the Issue:
1. When I disconnected connections between node1 and node2 by using iptables,
fencing was invoked as expected.
# node1 syslog output:
Jun 24 05:51:22 node1 kernel: [17025.778428] drbd r0/0 drbd0 node2: pdsk( UpToDate -> DUnknown ) repl( Established ->Off )
Jun 24 05:51:22 node1 kernel: [17025.810749] drbd r0 node2: helper command: /sbin/drbdadm fence-peer
Jun 24 05:51:23 node1 kernel: [17026.624983] drbd r0/0 drbd0 node2: pdsk( DUnknown -> Outdated )
# node2 syslog output:
Jun 24 05:51:22 node2 kernel: [17020.463472] drbd r0/0 drbd0 node1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2. After I recovered connections, corresponding unfencing was not invoked because
state of pdsk (at node1) was transitioned from 'Outdated' to 'UpToDate' directly.
[2] describes 'after-resync-target' is called on a resync target when a node state
changes from Inconsnstent to Consistent when a resync finishes.
So, it seems natural nothing is happen.
# node1 syslog output:
Jun 24 05:53:19 node1 kernel: [17143.117883] drbd r0/0 drbd0 node2: pdsk( Outdated -> UpToDate ) repl( Off -> Established )
# node2 syslog output:
Jun 24 05:53:19 node2 kernel: [17137.789972] drbd r0/0 drbd0 node1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Note that If something was written in the DRBD disk after '1.', unfencing was
invoked as follows.
1'. # node1 syslog output:
Jun 24 05:56:32 node1 kernel: [17336.114786] drbd r0/0 drbd0 node2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Jun 24 05:56:32 node1 kernel: [17336.146599] drbd r0 node2: helper command: /sbin/drbdadm fence-peer
Jun 24 05:56:32 node1 kernel: [17336.186206] drbd r0/0 drbd0 node2: pdsk( DUnknown -> Outdated )
# node2 syslog output:
Jun 24 05:56:32 node2 kernel: [17330.741001] drbd r0/0 drbd0 node1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2'. # node1 syslog output:
Jun 24 06:01:54 node1 kernel: [17658.236522] drbd r0/0 drbd0 node2: pdsk( Outdated -> Consistent ) repl( Off -> WFBitMapS )
Jun 24 06:01:54 node1 kernel: [17658.237470] drbd r0/0 drbd0 node2: pdsk( Consistent -> Outdated )
Jun 24 06:01:54 node1 kernel: [17658.239750] drbd r0/0 drbd0 node2: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource )
Jun 24 06:01:54 node1 kernel: [17658.307340] drbd r0/0 drbd0 node2: pdsk( Inconsistent -> UpToDate ) repl( SyncSource-> Established )
Jun 24 06:01:54 node1 kernel: [17658.307445] drbd r0 node2: helper command: /sbin/drbdadm unfence-peer
# node2 syslog output:
Jun 24 06:01:54 node2 kernel: [17652.870284] drbd r0/0 drbd0 node1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Jun 24 06:01:54 node2 kernel: [17652.908634] drbd r0/0 drbd0: disk( Outdated -> Inconsistent )
Jun 24 06:01:54 node2 kernel: [17652.908636] drbd r0/0 drbd0 node1: repl( WFBitMapT -> SyncTarget )
Jun 24 06:01:54 node2 kernel: [17652.918096] drbd r0/0 drbd0: disk( Inconsistent -> UpToDate )
Jun 24 06:01:54 node2 kernel: [17652.918098] drbd r0/0 drbd0 node1: repl( SyncTarget -> Established )
Jun 24 06:01:54 node2 kernel: [17652.918221] drbd r0/0 drbd0 node1: helper command: /sbin/drbdadm after-resync-target
Pacemaker Configuration (crm interactive):
----
sudo crm configure
primitive drbd_nfs ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="29s" role="Master" \
op monitor interval="31s" role="Slave"
ms ms_drbd_nfs drbd_nfs \
meta master-max="1" master-node-max="1" \
clone-max="2" clone-node-max="1" \
notify="true"
primitive fs_nfs ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/r0/0" \
directory="/mnt/drbd" fstype="xfs"
primitive ip_nfs ocf:heartbeat:IPaddr2 \
params ip="192.168.1.100" nic="enp0s8"
primitive nfsd lsb:nfs-kernel-server
group nfs fs_nfs ip_nfs nfsd
colocation nfs_on_drbd \
inf: nfs ms_drbd_nfs:Master
order nfs_after_drbd \
inf: ms_drbd_nfs:promote nfs:start
commit
exit
----
DRBD Configuration (/etc/drbd.d/r0.res):
----
resource r0 {
protocol C;
# for resource level fencing
net {
fencing resource-only;
}
handlers {
fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
}
#
device /dev/drbd0;
disk /dev/local/r0;
meta-disk internal;
on node1 {
address 192.168.1.11:7789;
}
on node2 {
address 192.168.1.12:7789;
}
}
----
Corosync Configuration (/etc/corosync/corosync.conf):
----
totem {
version: 2
cluster_name: drbd
secauth: off
transport: udpu
rrp_mode: active
}
nodelist {
node {
ring0_addr: 192.168.1.11
ring1_addr: 172.24.1.11
nodeid: 1
}
node {
ring0_addr: 192.168.1.12
ring1_addr: 172.24.1.12
nodeid: 2
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
----
[1] "14. Integrating DRBD with Pacemaker clusters",
The DRBD9 and LINSTOR User’s Guide,
https://docs.linbit.com/docs/users-guide-9.0/#ch-pacemaker
[2] Section handlers Parameters
after-resync-target cmd
https://docs.linbit.com/man/v9/drbd-conf-5/#Section_handlers_Parameters
Regards,
--
Takashi Sogabe <tasogabe at yahoo-corp.jp>
More information about the drbd-user
mailing list