Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
I am using a drbd device in an active/passive cluster setup with pacemaker. We have dedicated connections for corosync heartbeats, drbd and a 10GB data connection:
- A bonded 10GB network card for data traffic that will be accessed via a virtual ip managed by pacemaker in 192.168.120.1/24. In the cluster nodes MDA1PFP-S01 and MDA1PFP-S02 are assigned to 192.168.120.10 and 192.168.120.11.
- A dedicated back-to-back connection for corosync heartbeats in 192.168.121.1/24. MDA1PFP-PCS01 and MDA1PFP-S02 are assigned to 192.168.121.10 and 192.168.121.11. When the cluster is created, we use these as primary node names and use the 10GB device as a second backup connection for increased reliability: pcs cluster setup --name MDA1PFP MDA1PFP-PCS01,MDA1PFP-S01 MDA1PFP-PCS02,MDA1PFP-S02
- A dedicated back-to-back connection for drbd in 192.168.122.1/24. Hosts MDA1PFP-DRBD01 and MDA1PFP-DRBD02 are assigned 192.168.23.10 and 192.168.123.11.
In my tests, I force a failover by
1. Shutdown the cluster node with the master with pcs cluster stop 2. Disable the network device for the virtual ip with ifdown and wait until ping detects it
The initial state of the cluster is
MDA1PFP-S01 14:40:27 1803 0 ~ # pcs status Cluster name: MDA1PFP
Last updated: Fri Sep 16 14:41:18 2016 Last change: Fri Sep 16 14:39:49 2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS02 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
Full list of resources:
Master/Slave Set: drbd1_sync [drbd1]
Masters: [ MDA1PFP-PCS02 ]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS02
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS02
shared_fs (ocf::heartbeat:Filesystem): Started MDA1PFP-PCS02
PCSD Status:
MDA1PFP-PCS01: Online
MDA1PFP-PCS02: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
MDA1PFP-S01 14:41:19 1804 0 ~ # pcs resource --full
Master: drbd1_sync
Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
Resource: drbd1 (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=shared_fs
Operations: start interval=0s timeout=240 (drbd1-start-interval-0s)
promote interval=0s timeout=90 (drbd1-promote-interval-0s)
demote interval=0s timeout=90 (drbd1-demote-interval-0s)
stop interval=0s timeout=100 (drbd1-stop-interval-0s)
monitor interval=60s (drbd1-monitor-interval-60s)
Resource: mda-ip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.120.20 cidr_netmask=32 nic=bond0
Operations: start interval=0s timeout=20s (mda-ip-start-interval-0s)
stop interval=0s timeout=20s (mda-ip-stop-interval-0s)
monitor interval=1s (mda-ip-monitor-interval-1s)
Clone: ping-clone
Resource: ping (class=ocf provider=pacemaker type=ping)
Attributes: dampen=5s multiplier=1000 host_list=pf-pep-dev-1 timeout=1 attempts=3
Operations: start interval=0s timeout=60 (ping-start-interval-0s)
stop interval=0s timeout=20 (ping-stop-interval-0s)
monitor interval=1 (ping-monitor-interval-1)
Resource: ACTIVE (class=ocf provider=heartbeat type=Dummy)
Operations: start interval=0s timeout=20 (ACTIVE-start-interval-0s)
stop interval=0s timeout=20 (ACTIVE-stop-interval-0s)
monitor interval=10 timeout=20 (ACTIVE-monitor-interval-10)
Resource: shared_fs (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd1 directory=/shared_fs fstype=xfs
Operations: start interval=0s timeout=60 (shared_fs-start-interval-0s)
stop interval=0s timeout=60 (shared_fs-stop-interval-0s)
monitor interval=20 timeout=40 (shared_fs-monitor-interval-20)
MDA1PFP-S01 14:41:35 1805 0 ~ # pcs constraint --full Location Constraints:
Resource: mda-ip
Enabled on: MDA1PFP-PCS01 (score:50) (id:location-mda-ip-MDA1PFP-PCS01-50)
Constraint: location-mda-ip
Rule: score=-INFINITY boolean-op=or (id:location-mda-ip-rule)
Expression: pingd lt 1 (id:location-mda-ip-rule-expr)
Expression: not_defined pingd (id:location-mda-ip-rule-expr-1) Ordering Constraints:
start ping-clone then start mda-ip (kind:Optional) (id:order-ping-clone-mda-ip-Optional)
promote drbd1_sync then start shared_fs (kind:Mandatory) (id:order-drbd1_sync-shared_fs-mandatory)
Colocation Constraints:
ACTIVE with mda-ip (score:INFINITY) (id:colocation-ACTIVE-mda-ip-INFINITY)
drbd1_sync with mda-ip (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-drbd1_sync-mda-ip-INFINITY)
shared_fs with drbd1_sync (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-shared_fs-drbd1_sync-INFINITY)
The status after starting is:
Last updated: Fri Sep 16 14:39:57 2016 Last change: Fri Sep 16 14:39:49 2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS02 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
Master/Slave Set: drbd1_sync [drbd1]
Masters: [ MDA1PFP-PCS02 ]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS02
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ] ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS02
shared_fs (ocf::heartbeat:Filesystem): Started MDA1PFP-PCS02
The drbd is up as primary on MDA1PFP-PCS02 and mounted. Everything is fine.
If I do any of the two tests to force a failover the resources are moved to the other node but the drbd is not promoted to master on the new active node:
Last updated: Fri Sep 16 14:43:33 2016 Last change: Fri Sep 16 14:43:31 2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS01 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 ]
OFFLINE: [ MDA1PFP-PCS02 ]
Master/Slave Set: drbd1_sync [drbd1]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS01
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 ]
ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS01
I was able to trace this to the fencing in the drbd configuration
MDA1PFP-S01 14:41:44 1806 0 ~ # cat /etc/drbd.d/shared_fs.res resource shared_fs {
disk /dev/mapper/rhel_mdaf--pf--pep--1-drbd;
disk {
fencing resource-only;
}
handlers {
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
}
device /dev/drbd1;
meta-disk internal;
on MDA1PFP-S01 {
address 192.168.123.10:7789;
}
on MDA1PFP-S02 {
address 192.168.123.11:7789;
}
}
I am using drbd 8.4.7, drbd utils 8.9.5 and pacemaker 2.3.4-7.el7 with corosyinc 0.9.143-15.el7 from the Centos7 repositories.
MDA1PFP-S01 15:00:20 1841 0 ~ # drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 5d50d9fb2a967d21c0f5746370ccc066d3a67f7d\ build\ by\ mockbuild@\,\ 2016-01-12\ 12:46:45
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x080407
DRBDADM_VERSION_CODE=0x080905
DRBDADM_VERSION=8.9.5
If I disable the fencing scripts everything works as expected. If enabled, no node is promoted to master after failover. It seems to be a sticky modificaton because once a failover is simulated with fencing scripts activated I cannot get the cluster to work anymore. Even removing the setting from the DRBD configuration does not help.
I discussed this on the Pacemaker mailing list and from a Pacemaker point of view this should not happen. The nodes are still online so no fencing should happen, however DRBD fences the wrong node?
Best wishes,
Jens Auer
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.auer at cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.