<div dir="ltr">> *why*<br><br>> DRBD would not do that by itself,<br>> so likely pacemaker decided to do that,<br>> and you have to figure out *why*.<br>> Pacemaker will have logged the reasons somewhere.<div><br></div><div>The crm-fence-peer.sh script could not find the status of peer node (which went down) and assumed its status was "unknown" and thus placed a constraint on DRBD with -INFINITY score which essentially demotes and stops DRBD. The demotion failed because GFS2 was already mounted. This failure was construed as error by Pacemaker and it scheduled stonith for itself when the down node was back.</div><div><br></div><div>> "crm-fence-peer.sh" assumes that the result of "uname -n"<br>> is the local nodes "pacemaker node name".<br>Yes.</div><div><br>> If "uname -n" and "crm_node -n" do not return the same thing for you,<br>> the defaults will not work for you.<br></div><div><br></div><div>For my network the replication network (and its hostname) is different from client facing network (and its hostname):</div><div><div>[root@server7]# uname -n</div><div>server7</div><div>[root@server7]# crm_node -n</div><div>server7ha</div></div><div><br></div><div>However things seems to be working with these settings.</div><div><br></div><div><br></div><div>>Then in addition to all your other trouble,<br>> you have missing dependency constraints.<br></div><div><br></div><div>The proper integration of DRBD+GFS2+DLM+CLVM resources into Pacemaker was the issue. The pacemaker ordered constraints on these resources and definition of these resources were tricky and took time to fix. Finally I made DLM, CLVM, GFS2 as cloned resources and DRBD as master (with master-max=2) for my dual-Primary setup. After this I arrived at correct ordering of these resources: </div><div>Start & Promote DRBD then start DLM then start CLVM then start GFS2</div><div><br></div><div>Now things work fine. </div><div><br></div><div><br></div><div>To help anyone with similar situation here is my cluster status:</div><div>---------------------------------------------------------------------------------------------</div><div><div><br></div><div><div>[root@server4 ~]# pcs status</div><div>Cluster name: vCluster</div><div>Stack: corosync</div><div>Current DC: server4ha (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum</div><div>Last updated: Tue May 23 15:53:20 2017 Last change: Mon May 22 22:13:08 2017 by root via cibadmin on server4ha</div><div><br></div><div>2 nodes and 11 resources configured</div><div><br></div><div>Online: [ server4ha server7ha ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> vCluster-VirtualIP-10.168.10.199 (ocf::heartbeat:IPaddr2): Started server4ha</div><div> vCluster-Stonith-server4ha (stonith:fence_ipmilan): Started server7ha</div><div> vCluster-Stonith-server7ha (stonith:fence_ipmilan): Started server4ha</div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ server4ha server7ha ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ server4ha server7ha ]</div><div> Master/Slave Set: drbd_data_clone [drbd_data]</div><div> Masters: [ server4ha server7ha ]</div><div> Clone Set: Gfs2FS-clone [Gfs2FS]</div><div> Started: [ server4ha server7ha ]</div><div><br></div><div>Daemon Status:</div><div> corosync: active/disabled</div><div> pacemaker: active/disabled</div><div> pcsd: active/enabled</div><div>[root@server4 ~]# </div><div>[root@server4 ~]# </div><div>[root@server4 ~]# </div><div><br></div><div>My cluster constraints with ordered constraints in bold:<br></div><div>-----------------------------------------------------------------------------------</div><div>[root@server4 ~]# pcs constraint show</div><div>Location Constraints:</div><div> Resource: vCluster-Stonith-server4ha</div><div> Disabled on: server4ha (score:-INFINITY)</div><div> Resource: vCluster-Stonith-server7ha</div><div> Disabled on: server7ha (score:-INFINITY)</div><div>Ordering Constraints:</div><div><b> promote drbd_data_clone then start dlm-clone (kind:Mandatory)</b></div><div><b> start dlm-clone then start clvmd-clone (kind:Mandatory)</b></div><div><b> start clvmd-clone then start Gfs2FS-clone (kind:Mandatory)</b></div><div>Colocation Constraints:</div><div> dlm-clone with drbd_data_clone (score:INFINITY)</div><div> clvmd-clone with dlm-clone (score:INFINITY)</div><div> Gfs2FS-clone with clvmd-clone (score:INFINITY)</div><div>Ticket Constraints:</div><div>[root@server4 ~]# </div></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks for all your help.</div><div class="gmail_extra"><br></div><div class="gmail_extra">-- Raman</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 8:30 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Fri, May 12, 2017 at 02:04:57AM +0530, Raman Gupta wrote:<br>
> > I don't think this has anything to do with DRBD, because:<br>
> OK.<br>
><br>
> > Apparently, something downed the NICs for corosync communication.<br>
> > Which then leads to fencing.<br>
> No problem with NICs.<br>
><br>
> > Maybe you should double check your network configuration,<br>
> > and any automagic reconfiguration of the network,<br>
> > and only start corosync once your network is "stable"?<br>
> As another manifestation of similar problem of dual-Primary DRBD integrated<br>
> with stonith enabled Pacemaker: When server7 goes down, the DRBD resource<br>
> on surviving node server4 is attempted to be demoted as secondary.<br>
<br>
</span>*why*<br>
<br>
DRBD would not do that by itself,<br>
so likely pacemaker decided to do that,<br>
and you have to figure out *why*.<br>
Pacemaker will have logged the reasons somewhere.<br>
<br>
Seeing that you have different "uname -n" and "pacemaker node names",<br>
that may well be the source of all your troubles.<br>
<br>
"crm-fence-peer.sh" assumes that the result of "uname -n"<br>
is the local nodes "pacemaker node name".<br>
<br>
If "uname -n" and "crm_node -n" do not return the same thing for you,<br>
the defaults will not work for you.<br>
<span class="gmail-"><br>
> The<br>
> demotion fails because DRBD is hosting a GFS2 volume and Pacemaker complains<br>
> of this failure as an error.<br>
<br>
</span>Then in addition to all your other trouble,<br>
you have missing dependency constraints.<br>
IF pacemaker decides it needs to "demote" DRBD,<br>
it should know that it has a file system mounted,<br>
and should know that it needs to first unmount,<br>
and that it needs to first stop services accessing that mount,<br>
and so on.<br>
<br>
If it did not attempt to do that, your pacemaker config is broken.<br>
If it did attempt to do that and failed,<br>
you will have to look into why, which, again, should be in the logs.<br>
<br>
Double check constraints, and also double check if GFS2/DLM fencing is<br>
properly integrated with pacemaker.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Keeping the Digital World Running<br>
: DRBD -- Heartbeat -- Corosync -- Pacemaker<br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT<br>
__<br>
please don't Cc me, but send to list -- I'm subscribed<br>
______________________________<wbr>_________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/<wbr>mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br></div></div></div>