<p dir="ltr">I remembered something like that have faced me , and I solved that by changing kernel version. </p>
<div class="gmail_quote">On Nov 9, 2014 11:06 PM, "Digimer" <<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">CentOS 6.6, DRBD 8.3.16.<br>
<br>
So this sucked:<br>
<br>
After rebooting and restoring, I retried and got the same result a second time. After moving my VMs to the other node, I tested crashing the other node and again saw the "out of mem, failed to invoke fence-peer helper" message. After that, I rebooted both nodes. I've not yet tested if that resolved the issue.<br>
<br>
Anyone seen this before?<br>
<br>
====<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: PingAck did not arrive in time.<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: asender terminated<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: Terminating drbd0_asender<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: Connection closed<br>
<br>
*** Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: out of mem, failed to invoke fence-peer helper<br>
<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: conn( NetworkFailure -> Unconnected )<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: receiver terminated<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: Restarting drbd0_receiver<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: receiver (re)started<br>
Nov 9 15:18:40 fea-c01n01 kernel: block drbd0: conn( Unconnected -> WFConnection )<br>
Nov 9 15:18:42 fea-c01n01 corosync[3256]: [TOTEM ] A processor failed, forming new configuration.<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: PingAck did not arrive in time.<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 )<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: asender terminated<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: Terminating drbd1_asender<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: Connection closed<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: out of mem, failed to invoke fence-peer helper<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: conn( NetworkFailure -> Unconnected )<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: receiver terminated<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: Restarting drbd1_receiver<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: receiver (re)started<br>
Nov 9 15:18:42 fea-c01n01 kernel: block drbd1: conn( Unconnected -> WFConnection )<br>
Nov 9 15:18:44 fea-c01n01 corosync[3256]: [QUORUM] Members[1]: 1<br>
Nov 9 15:18:44 fea-c01n01 corosync[3256]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<br>
Nov 9 15:18:44 fea-c01n01 kernel: dlm: closing connection to node 2<br>
Nov 9 15:18:44 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Trying to acquire journal lock...<br>
Nov 9 15:18:44 fea-c01n01 fenced[3323]: fencing node <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a><br>
Nov 9 15:18:44 fea-c01n01 corosync[3256]: [CPG ] chosen downlist: sender r(0) ip(10.20.10.1) ; members(old:2 left:1)<br>
Nov 9 15:18:44 fea-c01n01 corosync[3256]: [MAIN ] Completed service synchronization, ready to provide service.<br>
Nov 9 15:19:03 fea-c01n01 fenced[3323]: fence <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a> success<br>
Nov 9 15:19:05 fea-c01n01 rgmanager[3496]: Marking service:storage_n02 as stopped: Restricted domain unavailable<br>
Nov 9 15:19:05 fea-c01n01 rgmanager[3496]: Marking service:libvirtd_n02 as stopped: Restricted domain unavailable<br>
Nov 9 15:21:16 fea-c01n01 kernel: INFO: task kslowd001:6376 blocked for more than 120 seconds.<br>
Nov 9 15:21:16 fea-c01n01 kernel: Not tainted 2.6.32-504.el6.x86_64 #1<br>
Nov 9 15:21:16 fea-c01n01 kernel: "echo 0 > /proc/sys/kernel/hung_task_<u></u>timeout_secs" disables this message.<br>
Nov 9 15:21:16 fea-c01n01 kernel: kslowd001 D 0000000000000012 0 6376 2 0x00000080<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880864a93b40 0000000000000046 000000030000fbc8 0000000000000018<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880800000008 ffffffffa04a4030 ffff880864a9f538 ffffffffa04a3fe0<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880800000003 ffff880864a9f5e8 ffff880864a91af8 ffff880864a93fd8<br>
Nov 9 15:21:16 fea-c01n01 kernel: Call Trace:<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a4030>] ? gdlm_ast+0x0/0x210 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a3fe0>] ? gdlm_bast+0x0/0x50 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0483100>] ? gfs2_glock_holder_wait+0x0/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa048310e>] gfs2_glock_holder_wait+0xe/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152ac7f>] __wait_on_bit+0x5f/0x90<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0483100>] ? gfs2_glock_holder_wait+0x0/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152ad28>] out_of_line_wait_on_bit+0x78/<u></u>0x90<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04840e5>] gfs2_glock_wait+0x45/0x90 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0487619>] gfs2_glock_nq+0x2c9/0x410 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa049ae23>] gfs2_recover_work+0xc3/0x7b0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81063bf3>] ? perf_event_task_sched_out+<u></u>0x33/0x70<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa049ae1b>] ? gfs2_recover_work+0xbb/0x7b0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0487939>] ? gfs2_glock_nq_num+0x59/0xa0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8106d59b>] ? enqueue_task_fair+0xfb/0x100<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81117ef3>] slow_work_execute+0x233/0x310<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81118127>] slow_work_thread+0x157/0x360<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/<u></u>0x40<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81117fd0>] ? slow_work_thread+0x0/0x360<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: INFO: task gfs2_quotad:6384 blocked for more than 120 seconds.<br>
Nov 9 15:21:16 fea-c01n01 kernel: Not tainted 2.6.32-504.el6.x86_64 #1<br>
Nov 9 15:21:16 fea-c01n01 kernel: "echo 0 > /proc/sys/kernel/hung_task_<u></u>timeout_secs" disables this message.<br>
Nov 9 15:21:16 fea-c01n01 kernel: gfs2_quotad D 0000000000000013 0 6384 2 0x00000080<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880864abfc20 0000000000000046 0000000000000000 0000000400000018<br>
Nov 9 15:21:16 fea-c01n01 kernel: 000000000000001c ffffffffa04a4030 00000067797edaa8 ffffffffa04a3fe0<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880800000005 00000001000229a9 ffff8808648a05f8 ffff880864abffd8<br>
Nov 9 15:21:16 fea-c01n01 kernel: Call Trace:<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a4030>] ? gdlm_ast+0x0/0x210 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a3fe0>] ? gdlm_bast+0x0/0x50 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0483100>] ? gfs2_glock_holder_wait+0x0/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa048310e>] gfs2_glock_holder_wait+0xe/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152ac7f>] __wait_on_bit+0x5f/0x90<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0483100>] ? gfs2_glock_holder_wait+0x0/<u></u>0x20 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152ad28>] out_of_line_wait_on_bit+0x78/<u></u>0x90<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04840e5>] gfs2_glock_wait+0x45/0x90 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0487619>] gfs2_glock_nq+0x2c9/0x410 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/<u></u>0xe0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a0469>] gfs2_statfs_sync+0x59/0x1c0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152a84a>] ? schedule_timeout+0x19a/0x2e0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa04a0461>] ? gfs2_statfs_sync+0x51/0x1c0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0497c27>] quotad_check_timeo+0x57/0xb0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0497eb4>] gfs2_quotad+0x234/0x2b0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/<u></u>0x40<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa0497c80>] ? gfs2_quotad+0x0/0x2b0 [gfs2]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: INFO: task qemu-kvm:24410 blocked for more than 120 seconds.<br>
Nov 9 15:21:16 fea-c01n01 kernel: Not tainted 2.6.32-504.el6.x86_64 #1<br>
Nov 9 15:21:16 fea-c01n01 kernel: "echo 0 > /proc/sys/kernel/hung_task_<u></u>timeout_secs" disables this message.<br>
Nov 9 15:21:16 fea-c01n01 kernel: qemu-kvm D 0000000000000013 0 24410 1 0x00000080<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880651e439c8 0000000000000082 0000000000000000 ffffffff8126b3c4<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff8810702c9400 ffff881072d1cb00 00000066cdb4e376 ffffffffa000461c<br>
Nov 9 15:21:16 fea-c01n01 kernel: ffff880651e43988 00000001000221b5 ffff88065a893ab8 ffff880651e43fd8<br>
Nov 9 15:21:16 fea-c01n01 kernel: Call Trace:<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8126b3c4>] ? blk_unplug+0x34/0x70<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffffa000461c>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8152a1b3>] io_schedule+0x73/0xc0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811ce7bd>] __blockdev_direct_IO_newtrunc+<u></u>0xb7d/0x1270<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811ca120>] ? blkdev_get_block+0x0/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811cef27>] __blockdev_direct_IO+0x77/0xe0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811ca120>] ? blkdev_get_block+0x0/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811cb1a7>] blkdev_direct_IO+0x57/0x60<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811ca120>] ? blkdev_get_block+0x0/0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81124ef2>] generic_file_direct_write+<u></u>0xc2/0x190<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff81126811>] __generic_file_aio_write+<u></u>0x3a1/0x490<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff810b2b43>] ? futex_wake+0x93/0x150<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff811ca6fc>] blkdev_aio_write+0x3c/0xa0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8118dd5a>] do_sync_write+0xfa/0x140<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8108e41e>] ? send_signal+0x3e/0x90<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/<u></u>0x40<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8108e856>] ? group_send_sig_info+0x56/0x70<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8123a5eb>] ? selinux_file_permission+0xfb/<u></u>0x150<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8122d446>] ? security_file_permission+0x16/<u></u>0x20<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8118e058>] vfs_write+0xb8/0x1a0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8118eae2>] sys_pwrite64+0x82/0xa0<br>
Nov 9 15:21:16 fea-c01n01 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br>
^[[A^[[BNov 9 15:21:43 fea-c01n01 corosync[3256]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<br>
Nov 9 15:21:43 fea-c01n01 corosync[3256]: [QUORUM] Members[2]: 1 2<br>
Nov 9 15:21:43 fea-c01n01 corosync[3256]: [QUORUM] Members[2]: 1 2<br>
Nov 9 15:21:43 fea-c01n01 corosync[3256]: [CPG ] chosen downlist: sender r(0) ip(10.20.10.1) ; members(old:1 left:0)<br>
Nov 9 15:21:43 fea-c01n01 corosync[3256]: [MAIN ] Completed service synchronization, ready to provide service.<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: Handshake successful: Agreed network protocol version 97<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: conn( WFConnection -> WFReportParams )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: Starting asender thread (from drbd0_receiver [5969])<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: Handshake successful: Agreed network protocol version 97<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: data-integrity-alg: <not-used><br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: conn( WFConnection -> WFReportParams )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: Starting asender thread (from drbd1_receiver [5973])<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: drbd_sync_handshake:<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: self 47E6EB8E53DA07A5:<u></u>0000000000000000:<u></u>512448D1B622C51D:<u></u>512348D1B622C51D bits:0 flags:0<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: peer 47E6EB8E53DA07A4:<u></u>0000000000000000:<u></u>512448D1B622C51D:<u></u>512348D1B622C51D bits:130048 flags:2<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: uuid_compare()=-1 by rule 40<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: I shall become SyncTarget, but I am primary!<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: data-integrity-alg: <not-used><br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: drbd_sync_handshake:<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: self 18E7A58FEF0C2589:<u></u>0000000000000000:<u></u>6D79ACDE04B51A3F:<u></u>6D78ACDE04B51A3F bits:0 flags:0<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: peer 18E7A58FEF0C2588:<u></u>0000000000000000:<u></u>6D79ACDE04B51A3F:<u></u>6D78ACDE04B51A3F bits:130048 flags:2<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: uuid_compare()=-1 by rule 40<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: I shall become SyncTarget, but I am primary!<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: conn( WFReportParams -> Disconnecting )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: error receiving ReportState, l: 4!<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: conn( WFReportParams -> Disconnecting )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: error receiving ReportState, l: 4!<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: asender terminated<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: Terminating drbd0_asender<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: asender terminated<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: Terminating drbd1_asender<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: Connection closed<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: Connection closed<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: conn( Disconnecting -> StandAlone )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: conn( Disconnecting -> StandAlone )<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: receiver terminated<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd0: Terminating drbd0_receiver<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: receiver terminated<br>
Nov 9 15:22:27 fea-c01n01 kernel: block drbd1: Terminating drbd1_receiver<br>
<br>
*** Nov 9 15:22:27 fea-c01n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...<br>
<br>
Nov 9 15:22:27 fea-c01n01 rhcs_fence: Attempting to fence peer using RHCS from DRBD...<br>
Nov 9 15:22:37 fea-c01n01 corosync[3256]: [TOTEM ] A processor failed, forming new configuration.<br>
Nov 9 15:22:39 fea-c01n01 corosync[3256]: [QUORUM] Members[1]: 1<br>
Nov 9 15:22:39 fea-c01n01 corosync[3256]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<br>
Nov 9 15:22:39 fea-c01n01 corosync[3256]: [CPG ] chosen downlist: sender r(0) ip(10.20.10.1) ; members(old:2 left:1)<br>
Nov 9 15:22:39 fea-c01n01 corosync[3256]: [MAIN ] Completed service synchronization, ready to provide service.<br>
Nov 9 15:22:39 fea-c01n01 kernel: dlm: closing connection to node 2<br>
Nov 9 15:22:39 fea-c01n01 fenced[3323]: fencing node <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a><br>
Nov 9 15:22:46 fea-c01n01 fenced[3323]: fence <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a> success<br>
Nov 9 15:22:47 fea-c01n01 fence_node[1906]: fence <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a> success<br>
<br>
*** Nov 9 15:22:47 fea-c01n01 kernel: block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 7 (0x700)<br>
<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd0: fence-peer helper returned 7 (peer was stonithed)<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd0: pdsk( DUnknown -> Outdated )<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd0: new current UUID 0B0310DD5442A87D:<u></u>47E6EB8E53DA07A5:<u></u>512448D1B622C51D:<u></u>512348D1B622C51D<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd0: susp( 1 -> 0 )<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Looking at journal...<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Acquiring the transaction lock...<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Replaying journal...<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Replayed 16 of 37 blocks<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Found 10 revoke tags<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Journal replayed in 1s<br>
Nov 9 15:22:47 fea-c01n01 kernel: GFS2: fsid=fea-cluster-01:shared.1: jid=0: Done<br>
Nov 9 15:22:47 fea-c01n01 fence_node[1907]: fence <a href="http://fea-c01n02.feaind.com" target="_blank">fea-c01n02.feaind.com</a> success<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 7 (0x700)<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd1: fence-peer helper returned 7 (peer was stonithed)<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd1: pdsk( DUnknown -> Outdated )<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd1: new current UUID FA229355474F1103:<u></u>18E7A58FEF0C2589:<u></u>6D79ACDE04B51A3F:<u></u>6D78ACDE04B51A3F<br>
Nov 9 15:22:47 fea-c01n01 kernel: block drbd1: susp( 1 -> 0 )<br>
=====<br>
<br>
-- <br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person without access to education?<br>
______________________________<u></u>_________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/<u></u>mailman/listinfo/drbd-user</a><br>
</blockquote></div>