Nov 9 23:55:10 vmhost0 external/ipmi[28054]: debug: ipmitool output: Chassis Power Control: Reset Nov 9 23:55:11 vmhost0 kernel: [ 2391.883321] bnx2 0000:08:00.0: eth0: NIC Copper Link is Down Nov 9 23:55:11 vmhost0 kernel: [ 2391.902305] br1: port 1(eth0) entering forwarding state Nov 9 23:55:11 vmhost0 stonithd: [1594]: info: Succeeded to STONITH the node VMHost1: optype=RESET. whodoit: VMHost0 Nov 9 23:55:11 vmhost0 crmd: [1599]: info: tengine_stonith_callback: call=28053, optype=1, node_name=VMHost1, result=0, node_list=VMHost0, action=64:8:0:8000989f-36a9-42de-ab56-d72693318f19 Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 11 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 65 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 36 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 39 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 46 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 60 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 63 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_rsc_command: Initiating action 82: notify resDRBD:1_post_notify_demote_0 on VMHost0 (local) Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_lrm_rsc_op: Performing key=82:8:0:8000989f-36a9-42de-ab56-d72693318f19 op=resDRBD:1_notify_0 ) Nov 9 23:55:11 vmhost0 lrmd: [1596]: info: rsc:resDRBD:1:29: notify Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 57 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 61 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 53 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 50 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 54 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 43 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 47 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=st-ipmilan-vmhost0_monitor_0, magic=0:7;9:2:7:8000989f-36a9-42de-ab56-d72693318f19, cib=0.454.4) : Resource op removal Nov 9 23:55:11 vmhost0 crmd: [1599]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Nov 9 23:55:11 vmhost0 crmd: [1599]: info: update_abort_priority: Abort action done superceeded by restart Nov 9 23:55:11 vmhost0 crmd: [1599]: info: erase_xpath_callback: Deletion of "//node_state[@uname='VMHost1']/lrm": ok (rc=0) Nov 9 23:55:11 vmhost0 cib: [1595]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='VMHost1']/lrm (origin=local/crmd/102, version=0.454.4): ok (rc=0) Nov 9 23:55:11 vmhost0 crmd: [1599]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=VMHost1, magic=NA, cib=0.454.5) : Transient attribute: removal Nov 9 23:55:11 vmhost0 crmd: [1599]: info: erase_xpath_callback: Deletion of "//node_state[@uname='VMHost1']/transient_attributes": ok (rc=0) Nov 9 23:55:11 vmhost0 cib: [1595]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='VMHost1']/transient_attributes (origin=local/crmd/103, version=0.454.5): ok (rc=0) Nov 9 23:55:11 vmhost0 lrmd: [1596]: info: RA output: (resDRBD:1:notify:stdout) Nov 9 23:55:11 vmhost0 crmd: [1599]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=29, rc=0, cib-update=104, confirmed=true) ok Nov 9 23:55:11 vmhost0 crmd: [1599]: info: match_graph_event: Action resDRBD:1_post_notify_demote_0 (82) confirmed on VMHost0 (rc=0) Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_pseudo_action: Pseudo action 40 fired and confirmed Nov 9 23:55:11 vmhost0 crmd: [1599]: info: run_graph: ==================================================== Nov 9 23:55:11 vmhost0 crmd: [1599]: notice: run_graph: Transition 8 (Complete=23, Pending=0, Fired=0, Skipped=6, Incomplete=5, Source=/var/lib/pengine/pe-warn-31.bz2): Stopped Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_graph_trigger: Transition 8 is now complete Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_pe_invoke: Query 105: Requesting the current CIB: S_POLICY_ENGINE Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_pe_invoke_callback: Invoking the PE: query=105, ref=pe_calc-dc-1320904511-62, seq=876, quorate=0 Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: unpack_config: On loss of CCM Quorum: Ignore Nov 9 23:55:11 vmhost0 pengine: [1598]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Nov 9 23:55:11 vmhost0 pengine: [1598]: info: determine_online_status: Node VMHost0 is online Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: native_print: st-ipmilan-vmhost0 (stonith:external/ipmi): Stopped Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: native_print: st-ipmilan-vmhost1 (stonith:external/ipmi): Started VMHost0 Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_print: Master/Slave Set: msDRBD Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Masters: [ VMHost0 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Stopped: [ resDRBD:0 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_print: Clone Set: cloneDLM Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Started: [ VMHost0 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Stopped: [ resDLM:1 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_print: Clone Set: cloneO2CB Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Started: [ VMHost0 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Stopped: [ resO2CB:1 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_print: Clone Set: cloneFS Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Started: [ VMHost0 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: short_print: Stopped: [ resFS:1 ] Nov 9 23:55:11 vmhost0 pengine: [1598]: info: native_color: Resource st-ipmilan-vmhost0 cannot run anywhere Nov 9 23:55:11 vmhost0 pengine: [1598]: info: native_color: Resource resDRBD:0 cannot run anywhere Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: Promoting resDRBD:1 (Master VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: Promoting resDRBD:1 (Master VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: Promoting resDRBD:1 (Master VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master Nov 9 23:55:11 vmhost0 pengine: [1598]: info: native_color: Resource resDLM:1 cannot run anywhere Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:1 with instance of cloneDLM Nov 9 23:55:11 vmhost0 pengine: [1598]: info: native_color: Resource resO2CB:1 cannot run anywhere Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: clone_rsc_colocation_rh: Cannot pair resFS:1 with instance of cloneO2CB Nov 9 23:55:11 vmhost0 pengine: [1598]: info: native_color: Resource resFS:1 cannot run anywhere Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource st-ipmilan-vmhost0 (Stopped) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource st-ipmilan-vmhost1 (Started VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resDRBD:0 (Stopped) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resDRBD:1 (Master VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resDLM:0 (Started VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resDLM:1 (Stopped) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resO2CB:0 (Started VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resO2CB:1 (Stopped) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resFS:0 (Started VMHost0) Nov 9 23:55:11 vmhost0 pengine: [1598]: notice: LogActions: Leave resource resFS:1 (Stopped) Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Nov 9 23:55:11 vmhost0 crmd: [1599]: info: unpack_graph: Unpacked transition 9: 0 actions in 0 synapses Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1320904511-62) derived from /var/lib/pengine/pe-input-582.bz2 Nov 9 23:55:11 vmhost0 crmd: [1599]: info: run_graph: ==================================================== Nov 9 23:55:11 vmhost0 crmd: [1599]: notice: run_graph: Transition 9 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-582.bz2): Complete Nov 9 23:55:11 vmhost0 crmd: [1599]: info: te_graph_trigger: Transition 9 is now complete Nov 9 23:55:11 vmhost0 crmd: [1599]: info: notify_crmd: Transition 9 status: done - Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Nov 9 23:55:11 vmhost0 crmd: [1599]: info: do_state_transition: Starting PEngine Recheck Timer Nov 9 23:55:11 vmhost0 pengine: [1598]: info: process_pe_message: Transition 9: PEngine Input stored in: /var/lib/pengine/pe-input-582.bz2 Nov 9 23:55:12 vmhost0 kernel: [ 2392.433660] (ocfs2rec,28051,0):ocfs2_replay_journal:1601 Recovering node 1459045701 from slot 1 on device (147,0) Nov 9 23:55:13 vmhost0 kernel: [ 2393.910018] block drbd0: PingAck did not arrive in time. Nov 9 23:55:13 vmhost0 kernel: [ 2393.926602] block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) Nov 9 23:55:13 vmhost0 kernel: [ 2393.959468] block drbd0: Creating new current UUID Nov 9 23:55:13 vmhost0 kernel: [ 2393.959515] block drbd0: short read expecting header on sock: r=-512 Nov 9 23:55:13 vmhost0 kernel: [ 2394.015100] block drbd0: asender terminated Nov 9 23:55:14 vmhost0 kernel: [ 2394.029343] block drbd0: Terminating drbd0_asender Nov 9 23:55:14 vmhost0 kernel: [ 2394.029449] block drbd0: Connection closed Nov 9 23:55:14 vmhost0 kernel: [ 2394.029507] block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 Nov 9 23:55:14 vmhost0 kernel: [ 2394.029509] block drbd0: conn( NetworkFailure -> Unconnected ) Nov 9 23:55:14 vmhost0 kernel: [ 2394.029520] block drbd0: receiver terminated Nov 9 23:55:14 vmhost0 kernel: [ 2394.029523] block drbd0: Restarting drbd0_receiver Nov 9 23:55:14 vmhost0 kernel: [ 2394.029525] block drbd0: receiver (re)started Nov 9 23:55:14 vmhost0 kernel: [ 2394.029531] block drbd0: conn( Unconnected -> WFConnection ) Nov 9 23:55:14 vmhost0 kernel: [ 2394.030015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030019] IP: [] sock_ioctl+0x1f/0x201 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030027] PGD 215908067 PUD 21cda7067 PMD 0 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030031] Oops: 0000 [#2] SMP Nov 9 23:55:14 vmhost0 kernel: [ 2394.030033] last sysfs file: /sys/fs/ocfs2/loaded_cluster_plugins Nov 9 23:55:14 vmhost0 kernel: [ 2394.030036] CPU 2 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030037] Modules linked in: ocfs2 ocfs2_nodemanager ocfs2_stack_user ocfs2_stackglue dlm drbd lru_cache ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle iptable_filter ip_tables bnx2 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030049] Nov 9 23:55:14 vmhost0 kernel: [ 2394.030052] Pid: 28141, comm: drbdadm Tainted: G D 2.6.38-gentoo-r6STI #4 Dell Inc. PowerEdge 1950/0NK937 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030057] RIP: 0010:[] [] sock_ioctl+0x1f/0x201 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030061] RSP: 0018:ffff88021ce27e68 EFLAGS: 00010282 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030063] RAX: 0000000000000000 RBX: 0000000000005401 RCX: 00007fff27903490 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030065] RDX: 00007fff27903490 RSI: 0000000000005401 RDI: ffff88021ce69300 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030067] RBP: ffff88021ce27e98 R08: 00007fff279034d0 R09: 00007ff44488ee60 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030070] R10: 00007fff279032b0 R11: 0000000000000202 R12: 00000000ffffffe7 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030072] R13: 00007fff27903490 R14: ffff88021e404840 R15: 0000000000000000 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030075] FS: 00007ff444a91700(0000) GS:ffff8800cf900000(0000) knlGS:0000000000000000 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030080] CR2: 0000000000000030 CR3: 000000021afa6000 CR4: 00000000000006e0 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030084] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030087] Process drbdadm (pid: 28141, threadinfo ffff88021ce26000, task ffff8802139fadc0) Nov 9 23:55:14 vmhost0 kernel: [ 2394.030088] Stack: Nov 9 23:55:14 vmhost0 kernel: [ 2394.030090] ffff88021ce27e78 0000000000000246 ffff88021ce69300 00000000ffffffe7 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030094] 00007fff27903490 00007fff27903490 ffff88021ce27f28 ffffffff811286f4 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030098] ffff88021ce27f78 ffffffff8151309f 0000000000000000 0000000000000000 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030101] Call Trace: Nov 9 23:55:14 vmhost0 kernel: [ 2394.030108] [] do_vfs_ioctl+0x484/0x4c5 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030113] [] ? page_fault+0x1f/0x30 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030116] [] sys_ioctl+0x51/0x77 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030121] [] system_call_fastpath+0x16/0x1b Nov 9 23:55:14 vmhost0 kernel: [ 2394.030123] Code: 00 5b 41 5c 41 5d 41 5e 41 5f c9 c3 55 48 89 e5 41 56 41 55 49 89 d5 41 54 53 89 f3 48 83 ec 10 4c 8b b7 a0 00 00 00 49 8b 46 20 <4c> 8b 60 30 8d 83 10 76 ff ff 83 f8 0f 77 0d 4c 89 e7 e8 4e 3e Nov 9 23:55:14 vmhost0 kernel: [ 2394.030143] RIP [] sock_ioctl+0x1f/0x201 Nov 9 23:55:14 vmhost0 kernel: [ 2394.030146] RSP Nov 9 23:55:14 vmhost0 kernel: [ 2394.030148] CR2: 0000000000000030 Nov 9 23:55:14 vmhost0 kernel: [ 2394.033648] ---[ end trace 16ec925abb8aa89d ]--- Nov 9 23:55:14 vmhost0 kernel: [ 2394.033727] block drbd0: helper command: /sbin/drbdadm fence-peer minor-0 exit code 0 (0x9) Nov 9 23:55:14 vmhost0 kernel: [ 2394.033730] block drbd0: fence-peer helper broken, returned 0 Nov 9 23:55:14 vmhost0 kernel: [ 2394.250451] bnx2 0000:08:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON