Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Jake,
Thanks for your help!
Answers to questions:
1.(Q) Why do you have LVM defined in the configuration?
(A) I wanted to make sure the LVM volumes were started before I start DRBD (I have DRBD configured on top of LVM). I assume that this should be okay.
2.(Q) Can you clarify what you mean by "DRBD is not started"?
(A) If I do a "cat /proc/drbd", I see "unconfigured". The DRBD agent START routine is never called. I believe this problem will be fixed once I work through my other problems.
3.(Q) Colocation appears to be backwards per the documentation.
(A) Thanks! I changed it per your suggestion. However, the Filesystem agent START routine is now called before the DRBD resources enters the MASTER state.
I made the changes you suggested. (I assumed I should not have to specify the stopping/demoting sequences but it was the only way I could get it to work.)
After these changes, a timeline of the behavior I see is this sequence logged by agent entry point calls:
1.Call LVM start and before LVM start finishes
2.Call Filesystem start
This fails since DRBD volume is readonly
3.LVM start completes
4.Filesystem stop (called because Filesystem start fails)
5.DRBD start called
6.DRBD promote called
My expectation was that the Filesystem start routine would not be called until DRBD was MASTER.
My configuration is:
node cnode-1-3-5
node cnode-1-3-6
primitive glance-drbd-p ocf:linbit:drbd \
params drbd_resource="glance-repos-drbd" \
op start interval="0" timeout="240" \
op stop interval="0" timeout="100" \
op monitor interval="59s" role="Master" timeout="30s" \
op monitor interval="61s" role="Slave" timeout="30s"
primitive glance-fs-p ocf:heartbeat:Filesystem \
params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \
op start interval="0" timeout="60" \
op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \
op stop interval="0" timeout="120"
primitive glance-ip-p ocf:heartbeat:IPaddr2 \
params ip="10.4.0.25" nic="br100" \
op monitor interval="5s"
primitive glance-lvm-p ocf:heartbeat:LVM \
params volgrpname="glance-repos" exclusive="true" \
op start interval="0" timeout="30" \
op stop interval="0" timeout="30"
primitive node-stonith-5-p stonith:external/ipmi \
op monitor interval="10m" timeout="1m" target_role="Started" \
params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99" userid="ADMIN" passwd="foo" interface="lan"
primitive node-stonith-6-p stonith:external/ipmi \
op monitor interval="10m" timeout="1m" target_role="Started" \
params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100" userid="ADMIN" passwd="foo" interface="lan"
group group-glance-fs glance-fs-p glance-ip-p \
meta target-role="Started"
ms ms-glance-drbd glance-drbd-p \
meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master"
clone cloneLvm glance-lvm-p
location loc-node-stonith-5 node-stonith-5-p \
rule $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5
location loc-node-stonith-6 node-stonith-6-p \
rule $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6
colocation coloc-fs-group-and-drbd inf: group-glance-fs ms-glance-drbd:Master
order order-glance-lvm-before-drbd inf: cloneLvm:start ms-glance-drbd:start
property $id="cib-bootstrap-options" \
dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="true" \
no-quorum-policy="ignore" \
last-lrm-refresh="1313440611"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
Thanks again for your help,
Bob
________________________________
From: Jake Smith <jsmith at argotec.com>
To: Bob Schatz <bschatz at yahoo.com>
Cc: drbd-user at lists.linbit.com
Sent: Thursday, August 11, 2011 11:04 AM
Subject: Re: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint cleared?
Comments in-line. Also in-line with Pacemaker config at the bottom.
HTH
Jake
----- Original Message -----
> From: "Bob Schatz" <bschatz at yahoo.com>
> To: drbd-user at lists.linbit.com
> Sent: Thursday, August 11, 2011 1:09:56 PM
> Subject: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint
> cleared?
> Hi,
> Does anyone know the answer to the question below about DRBD STONITH
> setting Pacemaker location constraints?
> Thanks!
> Bob
> ----- Forwarded Message -----
> From: Bob Schatz <bschatz at yahoo.com>
> To: "drbd-user at lists.linbit.com" <drbd-user at lists.linbit.com>
> Sent: Tuesday, August 2, 2011 12:21 PM
> Subject: [DRBD-user] DRBD STONITH - how is Pacemaker constraint
> cleared?
> Hi,
> I setup DRBD and Pacemaker using STONITH for DRBD and for Pacemaker.
> (Configs at bottom of email)
> When I reboot the PRIMARY DRBD node (cnode-1-3-6), Pacemaker shows
> this location constraint:
> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \
> rule $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master"
> -inf: #uname ne cnode-1-3-5
> and transitions the SECONDARY to PRIMARY. This makes sense to me.
> However, when I restart cnode-1-3-6 (cnode-1-3-5 still up as PRIMARY)
> the location constraint is not cleared as I would have expected.
> Also, DRBD is not started (I assume because of the location
> constraint). I would expect that since cnode-1-3-5 is still up the
> constraint would be moved and DRBD would change to SECONDARY.
The location constraint would only prevent DRBD from allowing glance-drbd to be promoted to Master on cnode-1-3-6. Basicly it says that the role of ms-glance-drbd:Master can only be on node named cnode-1-3-5. It doesn't care about ms-glance-drbd:Secondary. It would not prevent DRBD from starting either (though your ordering could cause it not to start...). Could you clarify what you mean by "DRBD is not stared"?
> Am I correct that this location constraint should be cleared?
> I assumed this would be cleared by the DRBD handler
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh" script but I
> do not believe it is called.
That is the handler that would clear the location constraint. You should see it cleared after the resync is complete. If DRBD isn't running it will never resync which means it will never run the after-resync-target commands. Have you checked that cnode-1-3-6 is UpToDate (cat /proc/drbd). Here's an excerpt of how it should look in the logs as the constraint is removed (should log on cnode-1-3-6):
kernel: [ 77.131564] block drbd4: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
kernel: [ 77.131573] block drbd4: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
kernel: [ 77.131585] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4
crm-unfence-peer.sh[3024]: invoked for bind <-- drbd4
kernel: [ 77.261360] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4 exit code 0 (0x0)
> BTW, I am pretty sure I have ordering duplications in my Pacemaker
> configuration (pointed out by Andrew on the Pacemaker mailing list)
> but I am not sure if that is the problem.
> Thanks,
> Bob
> drbd.conf file:
> global {
> usage-count yes;
> }
> common {
> protocol C;
> }
> resource glance-repos-drbd {
> disk {
> fencing resource-and-stonith;
> }
> handlers {
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
> }
> on cnode-1-3-5 {
> device /dev/drbd1;
> disk /dev/glance-repos/glance-repos-vol;
> address 10.4.1.29:7789;
> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;
> }
> on cnode-1-3-6 {
> device /dev/drbd1;
> disk /dev/glance-repos/glance-repos-vol;
> address 10.4.1.30:7789;
> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;
> }
> syncer {
> rate 40M;
> }
> }
> Pacemaker configuration:
> node cnode-1-3-5
> node cnode-1-3-6
> primitive glance-drbd-p ocf:linbit:drbd \ params
> drbd_resource="glance-repos-drbd" \ op start interval="0"
> timeout="240" \ op stop interval="0" timeout="100" \ op monitor
> interval="59s" role="Master" timeout="30s" \ op monitor
> interval="61s" role="Slave" timeout="30s"
> primitive glance-fs-p ocf:heartbeat:Filesystem \ params
> device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \ op
> start interval="0" timeout="60" \ op monitor interval="60"
> timeout="60" OCF_CHECK_LEVEL="20" \ op stop interval="0"
> timeout="120"
> primitive glance-ip-p ocf:heartbeat:IPaddr2 \ params ip="10.4.0.25"
> nic="br100" \ op monitor interval="5s"
> primitive glance-lvm-p ocf:heartbeat:LVM \ params
> volgrpname="glance-repos" exclusive="true" \ op start interval="0"
> timeout="30" \ op stop interval="0" timeout="30" \ meta
> target-role="Started"
I don't understand why you have this primitive?
> primitive node-stonith-5-p stonith:external/ipmi \ op monitor
> interval="10m" timeout="1m" target_role="Started" \ params
> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99"
> userid="ADMIN" passwd="foo" interface="lan"
> primitive node-stonith-6-p stonith:external/ipmi \ op monitor
> interval="10m" timeout="1m" target_role="Started" \ params
> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100"
> userid="ADMIN" passwd="foo" interface="lan"
> group group-glance-fs glance-fs-p glance-ip-p \ meta
> target-role="Started"
> ms ms-glance-drbd glance-drbd-p \ meta master-node-max="1"
> clone-max="2" clone-node-max="1" globally-unique="false"
> notify="true" target-role="Master"
> clone cloneLvm glance-lvm-p
> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \ rule
> $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master" -inf:
> #uname ne cnode-1-3-5
> location loc-node-stonith-5 node-stonith-5-p \ rule
> $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5
> location loc-node-stonith-6 node-stonith-6-p \ rule
> $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6
> colocation coloc-drbd-and-fs-group inf: ms-glance-drbd:Master
> group-glance-fs
This is backwards I believe... group-glance-fs runs on the ms-glance-drbd:Master correct?
Colocation reads x on y so this would say that the ms-glance-drbd:Master has to run on the group-glance-fs. That means if group-glance-fs isn't running then ms-glance-drbd:Master can never run on that node.
Quote from Pacemaker Docs:
<rsc_colocation id="colocate" rsc="resource1" with-rsc="resource2" score="INFINITY"/>
Remember, because INFINITY was used, if resource2 can't run on any of the cluster nodes (for whatever reason) then resource1 will not be allowed to run.
> order order-glance-drbd-demote-before-stop-drbd inf:
> ms-glance-drbd:demote ms-glance-drbd:stop
Not needed
> order order-glance-drbd-promote-before-fs-group inf:
> ms-glance-drbd:promote group-glance-fs:start
Ordering statements are applied in reverse when stopping so the above statements handles the demote/stop also making the ordering statements with demote unneeded.
> order order-glance-drbd-start-before-drbd-promote inf:
> ms-glance-drbd:start ms-glance-drbd:promote
Not needed - start for ms resources... they should be started normally.
> order order-glance-fs-stop-before-demote-drbd inf:
> group-glance-fs:stop ms-glance-drbd:demote
Not needed
> order order-glance-lvm-before-drbd 0: cloneLvm ms-glance-drbd:start
> property $id="cib-bootstrap-options" \
> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
> cluster-infrastructure="openais" \ expected-quorum-votes="2" \
> stonith-enabled="true" \ no-quorum-policy="ignore" \
> last-lrm-refresh="1311899021"
> rsc_defaults $id="rsc-options" \ resource-stickiness="100"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110815/cc0ce8e6/attachment.htm>