<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>Jake,</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">This seems to work as I wanted when I broke the order line up into:</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div><span class="Apple-tab-span" style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; white-space: pre; "> </span>colocation coloc-ip-and-fs-and-drbd inf: glance-ip-p glance-fs-p ms-glance-drbd:Master</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>order order-glance-drbd-master-before-fs inf:
ms-glance-drbd:promote glance-fs-p:start</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>order order-glance-fs-before-ip inf: glance-fs-p:start glance-ip-p:start</div><div><br></div><div>I need to run more tests to make sure.</div><div><br></div><div>It would be helpful to get the definition of an IPMI STONITH configuration which works. When I was trying it I was getting nodes shooting each other at join even though they should see each other. :(</div><div><br></div><div><br>Thanks,</div><div><br></div><div>Bob</div> <div><br><div><br></div><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight: bold;">To:</span></b> Jake Smith
<jsmith@argotec.com><br><b><span style="font-weight: bold;">Cc:</span></b> "drbd-user@lists.linbit.com" <drbd-user@lists.linbit.com><br><b><span style="font-weight: bold;">Sent:</span></b> Tuesday, August 16, 2011 3:35 PM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint cleared?<br></font><br><div id="yiv677137629"><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: times, serif; "><span>Jake,</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>Answers to questions:</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(Q)
What versions?</span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(A) Ubuntu 10.04.2 LTS, Pacemaker 1.0.8, DRBD 8.3.7 and LVM 2.02.54 using Corosync</span></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">(Q) Why have LVM in configuration?</div><div style="font-size: 12pt; font-family: times, serif; "><span>(A) I removed LVM from the configuration as well as the group.</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(Q) What dependency do you want?</span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(A) IP should start after the file system is up and the file system should only mount after DRBD is MASTER. (I plan on adding MySQL on top of the IP address after
this)</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(Q) </span>(curiosity) why 59/61 for intervals?</div><div style="font-size: 12pt; font-family: times, serif; ">(A) I knew that I had to pick different intervals for the monitors and just picked them off the top of my head. What are you using?</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">(Q) STONITH lists both hostnames, why?</div><div style="font-size: 12pt; font-family: times, serif; ">(A) I removed STONITH for now to aid in my debugging. I listed both names since the STONITH seemed to fail. It appears that the STONITH would not start on the local node because
the plugin removed the local host name from the list. How did you specify your STONITH devices? Can you give me the exact configuration?</div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(Q) You specified the grouping of resources incorrectly in the configuration file.</span></div><div style="font-size: 12pt; font-family: times, serif; "><span>(A) Thanks. I changed the configuration to no longer use groups and to just use "order" and "colocation". However, I still see the IP address starting BEFORE the files system completes START (mount). I have tried playing with the order directive but it does not seem to matter.</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>Here is my updated configuration:</span></div><div
style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div><span><div> node cnode-1-3-5</div><div> node cnode-1-3-6</div><div> primitive glance-drbd-p ocf:linbit:drbd \</div><div> params drbd_resource="glance-repos-drbd" \</div><div> op start interval="0" timeout="240" \</div><div> op stop interval="0" timeout="100" \</div><div> op monitor interval="59s" role="Master" timeout="30s" \</div><div>
op monitor interval="61s" role="Slave" timeout="30s"</div><div> primitive glance-fs-p ocf:heartbeat:Filesystem \</div><div> params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \</div><div> op start interval="0" timeout="60" \</div><div> op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \</div><div> op stop interval="0" timeout="120"</div><div> primitive glance-ip-p ocf:heartbeat:IPaddr2 \</div><div> params ip="10.4.0.25" nic="br100" \</div><div> op monitor interval="5s"</div><div> ms ms-glance-drbd glance-drbd-p
\</div><div> meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Started"</div><div> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \</div><div> rule $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master" -inf: #uname ne cnode-1-3-5</div><div> colocation coloc-ip-and-fs-and-drbd inf: glance-ip-p glance-fs-p ms-glance-drbd:Master order order-glance-drbd-master-before-fs-before-ip inf: ms-glance-drbd:promote glance-fs-p:start glance-ip-p:start </div><div> property $id="cib-bootstrap-options" \</div><div> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \</div><div>
cluster-infrastructure="openais" \</div><div> expected-quorum-votes="2" \</div><div> stonith-enabled="false" \</div><div> no-quorum-policy="ignore" \</div><div> last-lrm-refresh="1313440611"</div><div> rsc_defaults $id="rsc-options" \</div><div> resource-stickiness="100"</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Thanks again,</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Bob</div><div
style="font-size: 12pt; font-family: times, serif; "><br></div></span></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><div style="font-size: 12pt; font-family: times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Jake Smith <jsmith@argotec.com><br><b><span style="font-weight:bold;">To:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight:bold;">Cc:</span></b> drbd-user@lists.linbit.com<br><b><span style="font-weight:bold;">Sent:</span></b> Tuesday, August 16, 2011 8:31 AM<br><b><span style="
font-weight:bold;">Subject:</span></b> Re: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint cleared?<br></font><br>Bob - np that's what we're all out here for.<br><br>A few baseline questions I missed earlier:<br><br>- What OS?<br>- What versions of DRBD/Pacmaker/LVM2?<br>- Not that it should matter but Heartbeat or Corosync?<br><br>I'm thinking that the LVM stuff is unnecessary so I put some notes in your config below to remove it. Also I what is missing to get the startup order working is there too. Let me know if you have more questions and how you make out.<br><br>Jake<br><br>----- Original Message ----- <br><br>> From: "Bob Schatz" <<a rel="nofollow" ymailto="mailto:bschatz@yahoo.com" target="_blank" href="mailto:bschatz@yahoo.com">bschatz@yahoo.com</a>><br>> To: "Jake Smith" <<a rel="nofollow" ymailto="mailto:jsmith@argotec.com" target="_blank" href="mailto:jsmith@argotec.com">jsmith@argotec.com</a>><br>>
Cc: <a rel="nofollow" ymailto="mailto:drbd-user@lists.linbit.com" target="_blank" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>> Sent: Monday, August 15, 2011 6:25:22 PM<br>> Subject: Re: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker<br>> constraint cleared?<br><br>> Jake,<br><br>> Thanks for your help!<br><br>> Answers to questions:<br><br>> 1. (Q) Why do you have LVM defined in the configuration?<br>> (A) I wanted to make sure the LVM volumes were started before I start<br>> DRBD (I have DRBD configured on top of LVM). I assume that this<br>> should be okay.<br><br>My config is also DRBD on top of LVM however I don't "start" LVM... I believe it starts on boot. I don't have anything in my Pacemaker config for LVM; I just start with the DRBD primitive and build out from there.<br>I believe the purpose of the LVM resource agent is for LVM on top of DRBD not the other way
around...<br><br>> 2. (Q) Can you clarify what you mean by "DRBD is not started"?<br>> (A)
If I do a "cat /proc/drbd", I see "unconfigured". The DRBD agent<br>> START routine is never called. I believe this problem will be fixed<br>> once I work through my other problems.<br><br>Got it<br><br>> 3. (Q) Colocation appears to be backwards per the documentation.<br>> (A) Thanks! I changed it per your suggestion. However, the Filesystem<br>> agent START routine is now called before the DRBD resources enters<br>> the MASTER state.<br><br>There is no order constraint between FS and DRBD - see below in config<br><br>> I made the changes you suggested. (I assumed I should not have to<br>> specify the stopping/demoting sequences but it was the only way I<br>> could get it to work.)<br><br>> After these changes, a timeline of the behavior I see is this<br>> sequence logged by agent entry point calls:<br><br>> 1. Call LVM start and before LVM start finishes<br>> 2. Call Filesystem start<br><br>At this time I expect
the IP is up? According to your grouping that would trigger FS start.<br><br>> This fails since DRBD volume is readonly<br><br>> 3. LVM start completes<br>> 4. Filesystem stop (called because Filesystem start fails)<br>> 5. DRBD start called<br>> 6. DRBD promote called<br><br>> My expectation was that the Filesystem start routine would not be<br>> called until DRBD was MASTER.<br><br>> My configuration is:<br><br>> node cnode-1-3-5<br>> node cnode-1-3-6<br>> primitive glance-drbd-p ocf:linbit:drbd \<br>> params drbd_resource="glance-repos-drbd" \<br>> op start interval="0" timeout="240" \<br>> op stop interval="0" timeout="100" \<br>> op monitor interval="59s" role="Master" timeout="30s" \<br>> op monitor interval="61s" role="Slave" timeout="30s"<br><br>(curiosity) why 59/61 for intervals?<br><br>> primitive glance-fs-p ocf:heartbeat:Filesystem \<br>> params device="/dev/drbd1"
directory="/glance-mount" fstype="ext4" \<br>> op start interval="0" timeout="60" \<br>> op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \<br>> op stop interval="0" timeout="120"<br>> primitive glance-ip-p ocf:heartbeat:IPaddr2 \<br>> params ip="10.4.0.25" nic="br100" \<br>> op monitor interval="5s"<br>> primitive glance-lvm-p ocf:heartbeat:LVM \<br>> params volgrpname="glance-repos" exclusive="true" \<br>> op start interval="0" timeout="30" \<br>> op stop interval="0" timeout="30"<br><br>Remove the primitive for LVM.<br><br>> primitive node-stonith-5-p stonith:external/ipmi \<br>> op monitor interval="10m" timeout="1m" target_role="Started" \<br>> params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99"<br>> userid="ADMIN" passwd="foo" interface="lan"<br><br>Did you mean to have both node names in the hostname param? Not relevant to the problem...<br><br>> primitive node-stonith-6-p
stonith:external/ipmi \<br>> op monitor interval="10m" timeout="1m" target_role="Started" \<br>> params hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100"<br>> userid="ADMIN" passwd="foo" interface="lan"<br>> group group-glance-fs glance-fs-p glance-ip-p \<br>> meta target-role="Started"<br><br>This group is saying that the glance-fs has to start after the glance-ip and on the same node as the glance-ip... as soon as the IP is started then you're fs will try to start irregardless of the state of your DRBD resource. You probably know already but group my_group_resx_resy resx resy is basically saying resx must colocate with and order start after resy. What does the IP have to do with the FS? Without knowing I would expect the IP to start after the FS is ready making this one backward...<br>And lastly you can't have it this way (FS starts after IP in a group) and set FS to start on DRBD master below. If you make IP rely on
FS then it's OK. If you want it this way then you need to remove the group and do a colo and order like this:<br><br>order order-glance-drbd-master-before-ip-before-fs inf: ms-glance-drbd:promote glance-ip-p:start glance-fs-p:start<br>colocation coloc-fs-and-ip-and-drbd inf: glance-fs-p glance-ip-p ms-glance-drbd:Master<br><br>Did I confuse you yet? ;-)<br><br><br>> ms ms-glance-drbd glance-drbd-p \<br>> meta master-node-max="1" clone-max="2" clone-node-max="1"<br>> globally-unique="false" notify="true" target-role="Master"<br>> clone cloneLvm glance-lvm-p<br><br>Remove this clone<br><br>> location loc-node-stonith-5 node-stonith-5-p \<br>> rule $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5<br>> location loc-node-stonith-6 node-stonith-6-p \<br>> rule $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6<br>> colocation coloc-fs-group-and-drbd inf: group-glance-fs<br>> ms-glance-drbd:Master<br><br>Need
an order contraint to match with this colo to make sure DRBD is promoted before FS starts like:<br>order order-glance-drbd-master-before-fs inf: ms-glance-drbd:promote group-glance-fs:start<br><br>> order order-glance-lvm-before-drbd inf: cloneLvm:start<br>> ms-glance-drbd:start<br><br>Remove this order<br><br>> property $id="cib-bootstrap-options" \<br>> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \<br>> cluster-infrastructure="openais" \<br>> expected-quorum-votes="2" \<br>> stonith-enabled="true" \<br>> no-quorum-policy="ignore" \<br>> last-lrm-refresh="1313440611"<br>> rsc_defaults $id="rsc-options" \<br>> resource-stickiness="100"<br><br>*snip*<br><br><br><br></div></div></div></div><br>_______________________________________________<br>drbd-user mailing list<br><a ymailto="mailto:drbd-user@lists.linbit.com" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br><a
href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br><br><br></div></div></div></div></body></html>