<div dir="ltr"><div>Hi,</div><div><br></div><div>Thanks for looking into this issue. Here is my 'pcs status' and attached is cib.xml pacemaker file</div><div><br></div><div><div>[root@server4 cib]# pcs status</div><div>Cluster name: vCluster</div><div>Stack: corosync</div><div>Current DC: server7ha (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum</div><div>Last updated: Fri Mar 24 18:33:05 2017 Last change: Wed Mar 22 13:22:19 2017 by root via cibadmin on server7ha</div><div><br></div><div>2 nodes and 7 resources configured</div><div><br></div><div>Online: [ server4ha server7ha ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> vCluster-VirtualIP-10.168.10.<wbr>199 (ocf::heartbeat:IPaddr2): Started server7ha</div><div> vCluster-Stonith-server7ha (stonith:fence_ipmilan): Started server4ha</div><div> vCluster-Stonith-server4ha (stonith:fence_ipmilan): Started server7ha</div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ server4ha server7ha ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ server4ha server7ha ]</div><div><br></div><div>Daemon Status:</div><div> corosync: active/disabled</div><div> pacemaker: active/disabled</div><div> pcsd: active/enabled</div><div>[root@server4 cib]# </div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 24, 2017 at 1:49 PM, Raman Gupta <span dir="ltr"><<a href="mailto:ramangupta16@gmail.com" target="_blank">ramangupta16@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div><div>I am having a problem where if in GFS2 dual-Primary-DRBD Pacemaker Cluster, a node crashes then the running node hangs! The CLVM commands hang, the libvirt VM on running node hangs. </div><div><br></div><div>Env:</div><div>---------</div><div>CentOS 7.3</div><div>DRBD 8.4 </div><div>gfs2-utils-3.1.9-3.el7.x86_64<br></div><div>Pacemaker 1.1.15-11.el7_3.4<br></div><div>corosync-2.4.0-4.el7.x86_64<br></div><div><br></div><div><br></div><div>Infrastructure:</div><div>------------------------</div><div><div>1) Running A 2 node Pacemaker Cluster with proper fencing between the two. Nodes are server4 and server7.</div><div><br></div><div>2) Running DRBD dual-Primary and hosting GFS2 filesystem.</div><div><br></div><div>3) Pacemaker has DLM and cLVM resources configured among others.</div><div><br></div><div>4) A KVM/QEMU virtual machine is running on server4 which is holding the cluster resources.</div><div><br></div></div><div><br></div><div>Normal:</div><div>------------</div><div>5) In normal condition when the two nodes are completely UP then things are fine. The DRBD dual-primary works fine. The disk of VM is hosted on DRBD mount directory /backup and VM runs fine with Live Migration happily happening between the 2 nodes.</div><div><br></div><div><br></div><div>Problem:</div><div>----------------</div><div>6) Stop server7 [shutdown -h now] ---> LVM commands like pvdisplay hangs, VM runs only for 120s ---> After 120s DRBD/GFS2 panics (/var/log/messages below) in server4 and DRBD mount directory (/backup) becomes unavailable and VM hangs in server4. The DRBD though is fine on server4 and in Primary/Secondary mode in WFConnection state.<br></div><div><br></div><div>Mar 24 11:29:28 server4 crm-fence-peer.sh[54702]: invoked for vDrbd</div><div>Mar 24 11:29:28 server4 crm-fence-peer.sh[54702]: WARNING drbd-fencing could not determine the master id of drbd resource vDrbd</div><div>Mar 24 11:29:28 server4 kernel: drbd vDrbd: helper command: /sbin/drbdadm fence-peer vDrbd exit code 1 (0x100)</div><div>Mar 24 11:29:28 server4 kernel: drbd vDrbd: fence-peer helper broken, returned 1</div><div>Mar 24 11:32:01 server4 kernel: INFO: task kworker/8:1H:822 blocked for more than 120 seconds.</div><div>Mar 24 11:32:01 server4 kernel: "echo 0 > /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.</div><div>Mar 24 11:32:01 server4 kernel: kworker/8:1H D ffff880473796c18 0 822 2 0x00000080</div><div>Mar 24 11:32:01 server4 kernel: Workqueue: glock_workqueue glock_work_func [gfs2]</div><div>Mar 24 11:32:01 server4 kernel: ffff88027674bb10 0000000000000046 ffff8802736e9f60 ffff88027674bfd8</div><div>Mar 24 11:32:01 server4 kernel: ffff88027674bfd8 ffff88027674bfd8 ffff8802736e9f60 ffff8804757ef808</div><div>Mar 24 11:32:01 server4 kernel: 0000000000000000 ffff8804757efa28 ffff8804757ef800 ffff880473796c18</div><div>Mar 24 11:32:01 server4 kernel: Call Trace:</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff8168bbb9>] schedule+0x29/0x70</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffffa0714ce4>] drbd_make_request+0x2a4/0x380 [drbd]</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff812e0000>] ? aes_decrypt+0x260/0xe10</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff812ee6f9>] generic_make_request+0x109/<wbr>0x1e0</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff812ee841>] submit_bio+0x71/0x150</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffffa063ee11>] gfs2_meta_read+0x121/0x2a0 [gfs2]</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffffa063f392>] gfs2_meta_indirect_buffer+<wbr>0x62/0x150 [gfs2]</div><div>Mar 24 11:32:01 server4 kernel: [<ffffffff810d2422>] ? load_balance+0x192/0x990</div><div><br></div><div>7) After server7 is UP, Pacemaker Cluster is started, DRBD started and Logical Volume activated and only after that in server4 the DRBD mount directory (/backup) becomes available and VM resumes in server4. So after server7 is down and till it is completely UP the VM in server4 hangs.</div></div><div><br></div><div><br></div><div>Can anyone help how to avoid running node hang when other node crashes?</div><div><br></div><div><br></div><div>Attaching DRBD config file.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div>--Raman</div><div><br></div></font></span></div>
</blockquote></div><br></div>