<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>Jake,</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>Thanks for your help!</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>Answers to questions:</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>1.<span class="Apple-tab-span"
style="white-space:pre"> </span>(Q) Why do you have LVM defined in the configuration?</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><span class="Apple-tab-span" style="white-space:pre"> </span>(A) I wanted to make sure the LVM volumes were started before I start DRBD (I have DRBD configured on top of LVM). I assume that this should be okay.</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>2.<span class="Apple-tab-span" style="white-space:pre"> </span>(Q) Can you clarify what you mean by "DRBD is not started"?</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><span class="Apple-tab-span" style="white-space:pre"> </span>(A) If I do a "cat /proc/drbd", I see
"unconfigured". The DRBD agent START routine is never called. I believe this problem will be fixed once I work through my other problems.</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>3.<span class="Apple-tab-span" style="white-space:pre"> </span>(Q) Colocation appears to be backwards per the documentation.</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><span class="Apple-tab-span" style="white-space:pre"> </span>(A) Thanks! I changed it per your suggestion. However, the Filesystem agent START routine is now called before the DRBD resources enters the MASTER state.</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div
style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>I made the changes you suggested. (I assumed I should not have to specify the stopping/demoting sequences but it was the only way I could get it to work.)</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>After these changes, a</span> timeline of the behavior I see is this sequence logged by agent entry point calls:</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>1.<span class="Apple-tab-span" style="white-space:pre"> </span>Call LVM start and before LVM start finishes</div><div
style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>2.<span class="Apple-tab-span" style="white-space:pre"> </span>Call Filesystem start</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>This fails since DRBD volume is readonly</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>3.<span class="Apple-tab-span" style="white-space:pre"> </span>LVM start completes</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span
class="Apple-tab-span" style="white-space:pre"> </span>4.<span class="Apple-tab-span" style="white-space:pre"> </span>Filesystem stop (called because Filesystem start fails)</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>5.<span class="Apple-tab-span" style="white-space:pre"> </span>DRBD start called</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>6.<span class="Apple-tab-span" style="white-space:pre"> </span>DRBD promote called</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">My expectation was that the Filesystem start routine would not be called until DRBD was MASTER.</div><div style="font-family:
'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">My configuration is:</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div><div> node cnode-1-3-5</div><div> node cnode-1-3-6</div><div> primitive glance-drbd-p ocf:linbit:drbd \</div><div> params drbd_resource="glance-repos-drbd" \</div><div> op start interval="0" timeout="240" \</div><div> op stop interval="0" timeout="100" \</div><div> op monitor interval="59s" role="Master" timeout="30s" \</div><div>
op monitor interval="61s" role="Slave" timeout="30s"</div><div> primitive glance-fs-p ocf:heartbeat:Filesystem \</div><div> params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \</div><div> op start interval="0" timeout="60" \</div><div> op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \</div><div> op stop interval="0" timeout="120"</div><div> primitive glance-ip-p ocf:heartbeat:IPaddr2 \</div><div> params ip="10.4.0.25" nic="br100" \</div><div> op monitor interval="5s"</div><div> primitive glance-lvm-p
ocf:heartbeat:LVM \</div><div> params volgrpname="glance-repos" exclusive="true" \</div><div> op start interval="0" timeout="30" \</div><div> op stop interval="0" timeout="30"</div><div> primitive node-stonith-5-p stonith:external/ipmi \</div><div> op monitor interval="10m" timeout="1m" target_role="Started" \</div><div> params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99" userid="ADMIN" passwd="foo" interface="lan"</div><div> primitive node-stonith-6-p stonith:external/ipmi \</div><div> op monitor interval="10m" timeout="1m" target_role="Started" \</div><div>
params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100" userid="ADMIN" passwd="foo" interface="lan"</div><div> group group-glance-fs glance-fs-p glance-ip-p \</div><div> meta target-role="Started"</div><div> ms ms-glance-drbd glance-drbd-p \</div><div> meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master"</div><div><br></div><div> clone cloneLvm glance-lvm-p</div><div><br></div><div> location loc-node-stonith-5 node-stonith-5-p \</div><div> rule $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5</div><div> location loc-node-stonith-6
node-stonith-6-p \</div><div> rule $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6</div><div><br></div><div> colocation coloc-fs-group-and-drbd inf: group-glance-fs ms-glance-drbd:Master</div><div> order order-glance-lvm-before-drbd inf: cloneLvm:start ms-glance-drbd:start</div><div><br></div><div> property $id="cib-bootstrap-options" \</div><div> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \</div><div> cluster-infrastructure="openais" \</div><div> expected-quorum-votes="2" \</div><div> stonith-enabled="true" \</div><div>
no-quorum-policy="ignore" \</div><div> last-lrm-refresh="1313440611"</div><div> rsc_defaults $id="rsc-options" \</div><div> resource-stickiness="100"</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Thanks again for your help,</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Bob</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div></div><div style="font-size: 12pt; font-family: 'times new
roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Jake Smith <jsmith@argotec.com><br><b><span style="font-weight: bold;">To:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight: bold;">Cc:</span></b> drbd-user@lists.linbit.com<br><b><span style="font-weight: bold;">Sent:</span></b> Thursday, August 11, 2011 11:04 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint cleared?<br></font><br>Comments in-line. Also in-line with Pacemaker config at the bottom.<br><br>HTH<br><br>Jake<br><br>----- Original Message ----- <br><br>> From: "Bob Schatz" <<a ymailto="mailto:bschatz@yahoo.com" href="mailto:bschatz@yahoo.com">bschatz@yahoo.com</a>><br>> To: <a
ymailto="mailto:drbd-user@lists.linbit.com" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>> Sent: Thursday, August 11, 2011 1:09:56 PM<br>> Subject: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint<br>> cleared?<br><br>> Hi,<br><br>> Does anyone know the answer to the question below about DRBD STONITH<br>> setting Pacemaker location constraints?<br><br>> Thanks!<br><br>> Bob<br><br>> ----- Forwarded Message -----<br>> From: Bob Schatz <<a ymailto="mailto:bschatz@yahoo.com" href="mailto:bschatz@yahoo.com">bschatz@yahoo.com</a>><br>> To: "<a ymailto="mailto:drbd-user@lists.linbit.com" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a>" <<a ymailto="mailto:drbd-user@lists.linbit.com" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a>><br>> Sent: Tuesday, August 2, 2011 12:21 PM<br>> Subject: [DRBD-user] DRBD STONITH - how is
Pacemaker constraint<br>> cleared?<br><br>> Hi,<br><br>> I setup DRBD and Pacemaker using STONITH for DRBD and for Pacemaker.<br>> (Configs at bottom of email)<br><br>> When I reboot the PRIMARY DRBD node (cnode-1-3-6), Pacemaker shows<br>> this location constraint:<br><br>> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \<br>> rule $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master"<br>> -inf: #uname ne cnode-1-3-5<br><br>> and transitions the SECONDARY to PRIMARY. This makes sense to me.<br><br>> However, when I restart cnode-1-3-6 (cnode-1-3-5 still up as PRIMARY)<br>> the location constraint is not cleared as I would have expected.<br>> Also, DRBD is not started (I assume because of the location<br>> constraint). I would expect that since cnode-1-3-5 is still up the<br>> constraint would be moved and DRBD would change to SECONDARY.<br><br>The location constraint would only prevent
DRBD from allowing glance-drbd to be promoted to Master on cnode-1-3-6. Basicly it says that the role of ms-glance-drbd:Master can only be on node named cnode-1-3-5. It doesn't care about ms-glance-drbd:Secondary. It would not prevent DRBD from starting either (though your ordering could cause it not to start...). Could you clarify what you mean by "DRBD is not stared"?<br><br>> Am I correct that this location constraint should be cleared?<br><br>> I assumed this would be cleared by the DRBD handler<br>> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh" script but I<br>> do not believe it is called.<br><br>That is the handler that would clear the location constraint. You should see it cleared after the resync is complete. If DRBD isn't running it will never resync which means it will never run the after-resync-target commands. Have you checked that cnode-1-3-6 is UpToDate (cat /proc/drbd).
Here's an excerpt of how it should look in the logs as the constraint is removed (should log on cnode-1-3-6):<br><br>kernel: [ 77.131564] block drbd4: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<br>kernel: [ 77.131573] block drbd4: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )<br>kernel: [ 77.131585] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4<br>crm-unfence-peer.sh[3024]: invoked for bind <-- drbd4<br>kernel: [ 77.261360] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4 exit code 0 (0x0)<br><br><br>> BTW, I am pretty sure I have ordering duplications in my Pacemaker<br>> configuration (pointed out by Andrew on the Pacemaker mailing list)<br>> but I am not sure if that is the problem.<br><br>> Thanks,<br><br>> Bob<br><br>> drbd.conf file:<br><br>> global {<br>> usage-count yes;<br>> }<br><br>> common {<br>>
protocol C;<br>> }<br><br>> resource glance-repos-drbd {<br>> disk {<br>> fencing resource-and-stonith;<br>> }<br>> handlers {<br>> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";<br>> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";<br>> }<br>> on cnode-1-3-5 {<br>> device /dev/drbd1;<br>> disk /dev/glance-repos/glance-repos-vol;<br>> address 10.4.1.29:7789;<br>> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;<br>> }<br>> on cnode-1-3-6 {<br>> device /dev/drbd1;<br>> disk /dev/glance-repos/glance-repos-vol;<br>> address 10.4.1.30:7789;<br>> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;<br>> }<br>> syncer {<br>> rate 40M;<br>> }<br>> }<br><br>> Pacemaker configuration:<br><br>> node cnode-1-3-5<br>> node cnode-1-3-6<br><br>> primitive glance-drbd-p ocf:linbit:drbd \ params<br>> drbd_resource="glance-repos-drbd" \ op start
interval="0"<br>> timeout="240" \ op stop interval="0" timeout="100" \ op monitor<br>> interval="59s" role="Master" timeout="30s" \ op monitor<br>> interval="61s" role="Slave" timeout="30s"<br><br>> primitive glance-fs-p ocf:heartbeat:Filesystem \ params<br>> device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \ op<br>> start interval="0" timeout="60" \ op monitor interval="60"<br>> timeout="60" OCF_CHECK_LEVEL="20" \ op stop interval="0"<br>> timeout="120"<br><br>> primitive glance-ip-p ocf:heartbeat:IPaddr2 \ params ip="10.4.0.25"<br>> nic="br100" \ op monitor interval="5s"<br><br>> primitive glance-lvm-p ocf:heartbeat:LVM \ params<br>> volgrpname="glance-repos" exclusive="true" \ op start interval="0"<br>> timeout="30" \ op stop interval="0" timeout="30" \ meta<br>> target-role="Started"<br><br>I don't understand why you have this primitive?<br><br>> primitive node-stonith-5-p stonith:external/ipmi
\ op monitor<br>> interval="10m" timeout="1m" target_role="Started" \ params<br>> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99"<br>> userid="ADMIN" passwd="foo" interface="lan"<br><br>> primitive node-stonith-6-p stonith:external/ipmi \ op monitor<br>> interval="10m" timeout="1m" target_role="Started" \ params<br>> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100"<br>> userid="ADMIN" passwd="foo" interface="lan"<br><br>> group group-glance-fs glance-fs-p glance-ip-p \ meta<br>> target-role="Started"<br><br>> ms ms-glance-drbd glance-drbd-p \ meta master-node-max="1"<br>> clone-max="2" clone-node-max="1" globally-unique="false"<br>> notify="true" target-role="Master"<br><br>> clone cloneLvm glance-lvm-p<br><br>> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \ rule<br>> $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master" -inf:<br>> #uname ne cnode-1-3-5<br><br>>
location loc-node-stonith-5 node-stonith-5-p \ rule<br>> $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5<br><br>> location loc-node-stonith-6 node-stonith-6-p \ rule<br>> $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6<br><br>> colocation coloc-drbd-and-fs-group inf: ms-glance-drbd:Master<br>> group-glance-fs<br><br>This is backwards I believe... group-glance-fs runs on the ms-glance-drbd:Master correct?<br>Colocation reads x on y so this would say that the ms-glance-drbd:Master has to run on the group-glance-fs. That means if group-glance-fs isn't running then ms-glance-drbd:Master can never run on that node.<br><br>Quote from Pacemaker Docs:<br> <rsc_colocation id="colocate" rsc="resource1" with-rsc="resource2" score="INFINITY"/><br>Remember, because INFINITY was used, if resource2 can't run on any of the cluster nodes (for whatever reason) then resource1 will not be allowed to run.<br><br><br>> order
order-glance-drbd-demote-before-stop-drbd inf:<br>> ms-glance-drbd:demote ms-glance-drbd:stop<br><br>Not needed<br><br>> order order-glance-drbd-promote-before-fs-group inf:<br>> ms-glance-drbd:promote group-glance-fs:start<br><br>Ordering statements are applied in reverse when stopping so the above statements handles the demote/stop also making the ordering statements with demote unneeded.<br><br>> order order-glance-drbd-start-before-drbd-promote inf:<br>> ms-glance-drbd:start ms-glance-drbd:promote<br><br>Not needed - start for ms resources... they should be started normally.<br><br>> order order-glance-fs-stop-before-demote-drbd inf:<br>> group-glance-fs:stop ms-glance-drbd:demote<br><br>Not needed<br><br>> order order-glance-lvm-before-drbd 0: cloneLvm ms-glance-drbd:start<br><br>> property $id="cib-bootstrap-options" \<br>> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \<br>>
cluster-infrastructure="openais" \ expected-quorum-votes="2" \<br>> stonith-enabled="true" \ no-quorum-policy="ignore" \<br>> last-lrm-refresh="1311899021"<br><br>> rsc_defaults $id="rsc-options" \ resource-stickiness="100"<br><br><br></div></div></div></body></html>