Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
ups.. fixed it. Here's what happened then... 1) rcnetwork stop on xm01 @ 8:13:30 2) xm02 stonith's xm01 (so far so good) 3) the VM fails back to xm02 (nice!) 4) xm01 never goes back!!! logs: Mar 2 08:13:47 xm01 crmd: [5550]: ERROR: tengine_stonith_notify: Peer xm02 could not be terminated (off) by <anyone> for xm01 (ref=c6a7898b-d1a7-4d89-9fb7-3a7c73f0359e): Operation timed out Mar 2 08:13:47 xm01 stonith_admin-fence-peer.sh[31426]: Failed to fence peer xm02. stonith_admin returned 249. Mar 2 08:13:47 xm01 kernel: [62341.934991] block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 1 (0x100) Mar 2 08:13:47 xm01 kernel: [62341.935006] block drbd1: fence-peer helper broken, returned 1 Mar 2 08:13:47 xm01 cluster-dlm: fence_node_time: Nothing known about node 16777316 5) went to the datacenter, checked that xm01 was POWERED OFF. Manually powered on. 6) xm01 fails back as slave, synchronized all the drbd resources and became all Master/Master (excellent!) 7) BUT.. suddenly, the VM went down for some seconds, migrated to XM01 (no stickines here) and went up again. Logs: Mar 2 08:26:34 xm01 lrmd: [5576]: info: cancel_op: operation monitor[15] on vmsvn-drbd:0 for client 5579, its parameters: CRM_meta_clone=[0] drbd_resource=[vmsvn] CRM_meta_master_node_max=[1] CRM_m eta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[true] CRM_meta_master_max=[2] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] CRM_meta_name=[monitor] CRM_meta_role=[Slave] CR M_meta_interval=[30000] CRM_meta_timeout=[20000] cancelled Mar 2 08:26:34 xm01 crmd: [5579]: info: send_direct_ack: ACK'ing resource op vmsvn-drbd:0_monitor_30000 from 4:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd: lrm_invoke-lrmd-1330687594-4 Mar 2 08:26:34 xm01 lrmd: [5576]: info: cancel_op: operation monitor[16] on srvsvn1-drbd:0 for client 5579, its parameters: CRM_meta_clone=[0] drbd_resource=[srvsvn1] CRM_meta_master_node_max=[1] C RM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[true] CRM_meta_master_max=[2] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] CRM_meta_name=[monitor] CRM_meta_role=[Slave ] CRM_meta_interval=[30000] CRM_meta_timeout=[20000] cancelled Mar 2 08:26:34 xm01 crmd: [5579]: info: send_direct_ack: ACK'ing resource op srvsvn1-drbd:0_monitor_30000 from 2:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd: lrm_invoke-lrmd-1330687594-5 Mar 2 08:26:34 xm01 lrmd: [5576]: info: cancel_op: operation monitor[17] on srvsvn2-drbd:0 for client 5579, its parameters: CRM_meta_clone=[0] drbd_resource=[srvsvn2] CRM_meta_master_node_max=[1] C RM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[true] CRM_meta_master_max=[2] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] CRM_meta_name=[monitor] CRM_meta_role=[Slave ] CRM_meta_interval=[30000] CRM_meta_timeout=[20000] cancelled Mar 2 08:26:34 xm01 crmd: [5579]: info: send_direct_ack: ACK'ing resource op srvsvn2-drbd:0_monitor_30000 from 3:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd: lrm_invoke-lrmd-1330687594-6 Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=213:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmconfig:0 notify[19] (pid 5943) Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=229:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmsvn-drbd:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmsvn-drbd:0 notify[20] (pid 5944) Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=245:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn1-drbd:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:srvsvn1-drbd:0 notify[21] (pid 5945) Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=261:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn2-drbd:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:srvsvn2-drbd:0 notify[22] (pid 5946) Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmsvn-drbd:0_monitor_30000 (call=15, status=1, cib-update=0, confirmed=true) Cancelled Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_monitor_30000 (call=16, status=1, cib-update=0, confirmed=true) Cancelled Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_monitor_30000 (call=17, status=1, cib-update=0, confirmed=true) Cancelled Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[20] on vmsvn-drbd:0 for client 5579: pid 5944 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmsvn-drbd:0_notify_0 (call=20, rc=0, cib-update=27, confirmed=true) ok Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[19] on vmconfig:0 for client 5579: pid 5943 exited with return code 0 Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[22] on srvsvn2-drbd:0 for client 5579: pid 5946 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig:0_notify_0 (call=19, rc=0, cib-update=28, confirmed=true) ok Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_notify_0 (call=22, rc=0, cib-update=29, confirmed=true) ok Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[21] on srvsvn1-drbd:0 for client 5579: pid 5945 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_notify_0 (call=21, rc=0, cib-update=30, confirmed=true) ok Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=25:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig:0_promote_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmconfig:0 promote[23] (pid 6035) Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmconfig:0:promote:stdout) allow-two-primaries; Mar 2 08:26:34 xm01 kernel: [ 103.390137] block drbd3: role( Secondary -> Primary ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmconfig:0:promote:stdout) Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation promote[23] on vmconfig:0 for client 5579: pid 6035 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig:0_promote_0 (call=23, rc=0, cib-update=31, confirmed=true) ok Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=214:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmconfig:0 notify[24] (pid 6069) Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmconfig:0:notify:stdout) Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[24] on vmconfig:0 for client 5579: pid 6069 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig:0_notify_0 (call=24, rc=0, cib-update=32, confirmed=true) ok Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=26:6:8:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig:0_monitor_20000 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmconfig:0 monitor[25] (pid 6111) Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=58:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmsvn-drbd:0_promote_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmsvn-drbd:0 promote[26] (pid 6112) Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmsvn-drbd:0:promote:stdout) allow-two-primaries; Mar 2 08:26:34 xm01 kernel: [ 103.576139] block drbd0: role( Secondary -> Primary ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmsvn-drbd:0:promote:stdout) Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation promote[26] on vmsvn-drbd:0 for client 5579: pid 6112 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmsvn-drbd:0_promote_0 (call=26, rc=0, cib-update=33, confirmed=true) ok Mar 2 08:26:34 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=230:6:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmsvn-drbd:0_notify_0 ) Mar 2 08:26:34 xm01 lrmd: [5576]: info: rsc:vmsvn-drbd:0 notify[27] (pid 6174) Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation monitor[25] on vmconfig:0 for client 5579: pid 6111 exited with return code 8 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig:0_monitor_20000 (call=25, rc=8, cib-update=34, confirmed=false) master Mar 2 08:26:34 xm01 attrd: [5577]: info: attrd_trigger_update: Sending flush op to all hosts for: master-vmsvn-drbd:0 (10000) Mar 2 08:26:34 xm01 attrd: [5577]: info: attrd_perform_update: Sent update 18: master-vmsvn-drbd:0=10000 Mar 2 08:26:34 xm01 lrmd: [5576]: info: RA output: (vmsvn-drbd:0:notify:stdout) Mar 2 08:26:34 xm01 lrmd: [5576]: info: operation notify[27] on vmsvn-drbd:0 for client 5579: pid 6174 exited with return code 0 Mar 2 08:26:34 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmsvn-drbd:0_notify_0 (call=27, rc=0, cib-update=35, confirmed=true) ok Mar 2 08:26:35 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=55:7:8:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmsvn-drbd:0_monitor_20000 ) Mar 2 08:26:35 xm01 lrmd: [5576]: info: rsc:vmsvn-drbd:0 monitor[28] (pid 6213) Mar 2 08:26:35 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=241:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn1-drbd:0_notify_0 ) Mar 2 08:26:35 xm01 lrmd: [5576]: info: rsc:srvsvn1-drbd:0 notify[29] (pid 6214) Mar 2 08:26:35 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=257:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn2-drbd:0_notify_0 ) Mar 2 08:26:35 xm01 lrmd: [5576]: info: rsc:srvsvn2-drbd:0 notify[30] (pid 6215) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation notify[29] on srvsvn1-drbd:0 for client 5579: pid 6214 exited with return code 0 Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation notify[30] on srvsvn2-drbd:0 for client 5579: pid 6215 exited with return code 0 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_notify_0 (call=29, rc=0, cib-update=36, confirmed=true) ok Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_notify_0 (call=30, rc=0, cib-update=37, confirmed=true) ok Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation monitor[28] on vmsvn-drbd:0 for client 5579: pid 6213 exited with return code 8 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmsvn-drbd:0_monitor_20000 (call=28, rc=8, cib-update=38, confirmed=false) master Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=87:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn1-drbd:0_promote_0 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn1-drbd:0 promote[31] (pid 6287) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn1-drbd:0:promote:stdout) allow-two-primaries; Mar 2 08:26:36 xm01 kernel: [ 104.875334] block drbd1: role( Secondary -> Primary ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn1-drbd:0:promote:stdout) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation promote[31] on srvsvn1-drbd:0 for client 5579: pid 6287 exited with return code 0 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_promote_0 (call=31, rc=0, cib-update=39, confirmed=true) ok Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=242:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn1-drbd:0_notify_0 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn1-drbd:0 notify[32] (pid 6323) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn1-drbd:0:notify:stdout) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation notify[32] on srvsvn1-drbd:0 for client 5579: pid 6323 exited with return code 0 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_notify_0 (call=32, rc=0, cib-update=40, confirmed=true) ok Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=88:7:8:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn1-drbd:0_monitor_20000 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn1-drbd:0 monitor[33] (pid 6361) Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=120:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn2-drbd:0_promote_0 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn2-drbd:0 promote[34] (pid 6362) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn2-drbd:0:promote:stdout) allow-two-primaries; Mar 2 08:26:36 xm01 kernel: [ 105.072274] block drbd2: role( Secondary -> Primary ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation monitor[33] on srvsvn1-drbd:0 for client 5579: pid 6361 exited with return code 8 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn1-drbd:0_monitor_20000 (call=33, rc=8, cib-update=41, confirmed=false) master Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn2-drbd:0:promote:stdout) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation promote[34] on srvsvn2-drbd:0 for client 5579: pid 6362 exited with return code 0 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_promote_0 (call=34, rc=0, cib-update=42, confirmed=true) ok Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=258:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn2-drbd:0_notify_0 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn2-drbd:0 notify[35] (pid 6425) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (srvsvn2-drbd:0:notify:stdout) Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation notify[35] on srvsvn2-drbd:0 for client 5579: pid 6425 exited with return code 0 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_notify_0 (call=35, rc=0, cib-update=43, confirmed=true) ok Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=121:7:8:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=srvsvn2-drbd:0_monitor_20000 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:srvsvn2-drbd:0 monitor[36] (pid 6463) Mar 2 08:26:36 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=150:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=dlm:0_start_0 ) Mar 2 08:26:36 xm01 lrmd: [5576]: info: rsc:dlm:0 start[37] (pid 6464) Mar 2 08:26:36 xm01 lrmd: [5576]: info: RA output: (dlm:0:start:stderr) dlm_controld.pcmk: no process found Mar 2 08:26:36 xm01 lrmd: [5576]: info: operation monitor[36] on srvsvn2-drbd:0 for client 5579: pid 6463 exited with return code 8 Mar 2 08:26:36 xm01 crmd: [5579]: info: process_lrm_event: LRM operation srvsvn2-drbd:0_monitor_20000 (call=36, rc=8, cib-update=44, confirmed=false) master Mar 2 08:26:36 xm01 kernel: [ 105.315340] DLM (built Dec 2 2011 13:32:47) installed Mar 2 08:26:36 xm01 cluster-dlm[6512]: main: dlm_controld master started Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: get_cluster_type: Cluster type is: 'openais'. Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: init_ais_connection_classic: AIS connection established Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: get_ais_nodeid: Server details: id=16777316 uname=xm01 cname=pcmk Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_new_peer: Node xm01 now has id: 16777316 Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_new_peer: Node 16777316 is now known as xm01 Mar 2 08:26:36 xm01 corosync[5558]: [pcmk ] info: pcmk_notify: Enabling node notifications for child 6512 (0x6c92a0) Mar 2 08:26:36 xm01 cluster-dlm: setup_misc_devices: found /dev/misc/dlm-control minor 56 Mar 2 08:26:36 xm01 cluster-dlm: setup_misc_devices: found /dev/misc/dlm-monitor minor 55 Mar 2 08:26:36 xm01 cluster-dlm: setup_misc_devices: found /dev/misc/dlm_plock minor 54 Mar 2 08:26:36 xm01 cluster-dlm: setup_monitor: /dev/misc/dlm-monitor fd 9 Mar 2 08:26:36 xm01 cluster-dlm: update_comms_nodes: /sys/kernel/config/dlm/cluster/comms: opendir failed: 2 Mar 2 08:26:36 xm01 cluster-dlm: clear_configfs_spaces: /sys/kernel/config/dlm/cluster/spaces: opendir failed: 2 Mar 2 08:26:36 xm01 cluster-dlm: detect_protocol: totem/rrp_mode = 'none' Mar 2 08:26:36 xm01 cluster-dlm: set_configfs_protocol: set protocol 0 Mar 2 08:26:36 xm01 cluster-dlm: setup_cpg_daemon: setup_cpg_daemon 11 Mar 2 08:26:36 xm01 cluster-dlm: log_config: dlm:controld conf 2 1 0 memb 16777316 33554532 join 16777316 left Mar 2 08:26:36 xm01 cluster-dlm: receive_protocol: run protocol from nodeid 33554532 Mar 2 08:26:36 xm01 cluster-dlm: set_protocol: daemon run 1.1.1 max 1.1.1 kernel run 1.1.1 max 1.1.1 Mar 2 08:26:36 xm01 cluster-dlm: setup_plocks: plocks 13 Mar 2 08:26:36 xm01 cluster-dlm: setup_plocks: plock cpg message size: 104 bytes Mar 2 08:26:36 xm01 cluster-dlm: [6512]: notice: ais_dispatch_message: Membership 1056: quorum acquired Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_update_peer: Node xm01: id=16777316 state=member (new) addr=r(0) ip(100.0.0.1) (new) votes=1 (new) born=1056 seen=1056 proc=000000000000000000000 00000151312 (new) Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_new_peer: Node xm02 now has id: 33554532 Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_new_peer: Node 33554532 is now known as xm02 Mar 2 08:26:36 xm01 cluster-dlm: [6512]: info: crm_update_peer: Node xm02: id=33554532 state=member (new) addr=r(0) ip(100.0.0.2) votes=1 born=1048 seen=1056 proc=00000000000000000000000000151312 Mar 2 08:26:36 xm01 cluster-dlm: update_cluster: Processing membership 1056 Mar 2 08:26:36 xm01 cluster-dlm: dlm_process_node: Adding address ip(100.0.0.1) to configfs for node 16777316 Mar 2 08:26:36 xm01 cluster-dlm: add_configfs_node: set_configfs_node 16777316 100.0.0.1 local 1 Mar 2 08:26:36 xm01 cluster-dlm: dlm_process_node: Added active node 16777316: born-on=1056, last-seen=1056, this-event=1056, last-event=0 Mar 2 08:26:36 xm01 cluster-dlm: dlm_process_node: Adding address ip(100.0.0.2) to configfs for node 33554532 Mar 2 08:26:36 xm01 cluster-dlm: add_configfs_node: set_configfs_node 33554532 100.0.0.2 local 0 Mar 2 08:26:36 xm01 cluster-dlm: dlm_process_node: Added active node 33554532: born-on=1048, last-seen=1056, this-event=1056, last-event=0 Mar 2 08:26:37 xm01 lrmd: [5576]: info: operation start[37] on dlm:0 for client 5579: pid 6464 exited with return code 0 Mar 2 08:26:37 xm01 crmd: [5579]: info: process_lrm_event: LRM operation dlm:0_start_0 (call=37, rc=0, cib-update=45, confirmed=true) ok Mar 2 08:26:37 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=151:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=dlm:0_monitor_10000 ) Mar 2 08:26:37 xm01 lrmd: [5576]: info: rsc:dlm:0 monitor[38] (pid 6523) Mar 2 08:26:37 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=152:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=o2cb:0_start_0 ) Mar 2 08:26:37 xm01 lrmd: [5576]: info: rsc:o2cb:0 start[39] (pid 6524) Mar 2 08:26:37 xm01 o2cb[6524]: INFO: Stack glue driver not loaded Mar 2 08:26:37 xm01 lrmd: [5576]: info: operation monitor[38] on dlm:0 for client 5579: pid 6523 exited with return code 0 Mar 2 08:26:37 xm01 crmd: [5579]: info: process_lrm_event: LRM operation dlm:0_monitor_10000 (call=38, rc=0, cib-update=46, confirmed=false) ok Mar 2 08:26:37 xm01 o2cb[6524]: INFO: Starting o2cb:0 Mar 2 08:26:37 xm01 kernel: [ 106.488105] ocfs2: Registered cluster interface user Mar 2 08:26:37 xm01 kernel: [ 106.521037] OCFS2 Node Manager 1.5.0 Mar 2 08:26:37 xm01 kernel: [ 106.570144] OCFS2 1.5.0 Mar 2 08:26:37 xm01 ocfs2_controld.pcmk: Core dumps enabled: /var/lib/openais Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: get_cluster_type: Cluster type is: 'openais'. Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: init_ais_connection_classic: AIS connection established Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: get_ais_nodeid: Server details: id=16777316 uname=xm01 cname=pcmk Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_new_peer: Node xm01 now has id: 16777316 Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_new_peer: Node 16777316 is now known as xm01 Mar 2 08:26:37 xm01 corosync[5558]: [pcmk ] info: pcmk_notify: Enabling node notifications for child 6585 (0x6d9fa0) Mar 2 08:26:37 xm01 ocfs2_controld: Cluster connection established. Local node id: 16777316 Mar 2 08:26:37 xm01 ocfs2_controld: Added Pacemaker as client 1 with fd 7 Mar 2 08:26:37 xm01 ocfs2_controld: Initializing CKPT service (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Connected to CKPT service with handle 0x327b23c600000000 Mar 2 08:26:37 xm01 ocfs2_controld: Opening checkpoint "ocfs2:controld:01000064" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Opened checkpoint "ocfs2:controld:01000064" with handle 0x6633487300000000 Mar 2 08:26:37 xm01 ocfs2_controld: Writing to section "daemon_max_protocol" on checkpoint "ocfs2:controld:01000064" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Stored section "daemon_max_protocol" on checkpoint "ocfs2:controld:01000064" Mar 2 08:26:37 xm01 ocfs2_controld: Writing to section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:01000064" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Stored section "ocfs2_max_protocol" on checkpoint "ocfs2:controld:01000064" Mar 2 08:26:37 xm01 ocfs2_controld: Starting join for group "ocfs2:controld" Mar 2 08:26:37 xm01 ocfs2_controld: cpg_join succeeded Mar 2 08:26:37 xm01 ocfs2_controld: setup done Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: notice: ais_dispatch_message: Membership 1056: quorum acquired Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_update_peer: Node xm01: id=16777316 state=member (new) addr=r(0) ip(100.0.0.1) (new) votes=1 (new) born=1056 seen=1056 proc=000000000000000000 00000000151312 (new) Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_new_peer: Node xm02 now has id: 33554532 Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_new_peer: Node 33554532 is now known as xm02 Mar 2 08:26:37 xm01 ocfs2_controld: [6585]: info: crm_update_peer: Node xm02: id=33554532 state=member (new) addr=r(0) ip(100.0.0.2) votes=1 born=1048 seen=1056 proc=000000000000000000000000001513 12 Mar 2 08:26:37 xm01 ocfs2_controld: confchg called Mar 2 08:26:37 xm01 ocfs2_controld: ocfs2_controld (group "ocfs2:controld") confchg: members 2, left 0, joined 1 Mar 2 08:26:37 xm01 ocfs2_controld: CPG is live, we are not the first daemon Mar 2 08:26:37 xm01 ocfs2_controld: Opening checkpoint "ocfs2:controld" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Opened checkpoint "ocfs2:controld" with handle 0x2ae8944a00000001 Mar 2 08:26:37 xm01 ocfs2_controld: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Read section "daemon_protocol" from checkpoint "ocfs2:controld" Mar 2 08:26:37 xm01 ocfs2_controld: Reading from section "ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1) Mar 2 08:26:37 xm01 ocfs2_controld: Read section "ocfs2_protocol" from checkpoint "ocfs2:controld" Mar 2 08:26:37 xm01 ocfs2_controld: Daemon protocol is 1.0 Mar 2 08:26:37 xm01 ocfs2_controld: fs protocol is 1.0 Mar 2 08:26:37 xm01 ocfs2_controld: Connecting to dlm_controld Mar 2 08:26:37 xm01 ocfs2_controld: Opening control device Mar 2 08:26:37 xm01 cluster-dlm: process_listener: client connection 5 fd 14 Mar 2 08:26:37 xm01 ocfs2_controld: Starting to listen for mounters Mar 2 08:26:37 xm01 ocfs2_controld: new listening connection 4 Mar 2 08:26:39 xm01 lrmd: [5576]: info: operation start[39] on o2cb:0 for client 5579: pid 6524 exited with return code 0 Mar 2 08:26:39 xm01 crmd: [5579]: info: process_lrm_event: LRM operation o2cb:0_start_0 (call=39, rc=0, cib-update=47, confirmed=true) ok Mar 2 08:26:39 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=153:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=o2cb:0_monitor_10000 ) Mar 2 08:26:39 xm01 lrmd: [5576]: info: rsc:o2cb:0 monitor[40] (pid 6597) Mar 2 08:26:39 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=154:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=clvm:0_start_0 ) Mar 2 08:26:39 xm01 lrmd: [5576]: info: rsc:clvm:0 start[41] (pid 6598) Mar 2 08:26:39 xm01 clvmd[6598]: INFO: Starting clvm:0 Mar 2 08:26:39 xm01 lrmd: [5576]: info: operation monitor[40] on o2cb:0 for client 5579: pid 6597 exited with return code 0 Mar 2 08:26:39 xm01 crmd: [5579]: info: process_lrm_event: LRM operation o2cb:0_monitor_10000 (call=40, rc=0, cib-update=48, confirmed=false) ok Mar 2 08:26:39 xm01 cluster-dlm: process_uevent: uevent: add@/kernel/dlm/clvmd Mar 2 08:26:39 xm01 cluster-dlm: process_uevent: kernel: add@ clvmd Mar 2 08:26:39 xm01 cluster-dlm: process_uevent: uevent: online@/kernel/dlm/clvmd Mar 2 08:26:39 xm01 cluster-dlm: process_uevent: kernel: online@ clvmd Mar 2 08:26:39 xm01 kernel: [ 108.697966] dlm: Using TCP for communications Mar 2 08:26:39 xm01 cluster-dlm: log_config: dlm:ls:clvmd conf 2 1 0 memb 16777316 33554532 join 16777316 left Mar 2 08:26:39 xm01 cluster-dlm: add_change: clvmd add_change cg 1 joined nodeid 16777316 Mar 2 08:26:39 xm01 cluster-dlm: add_change: clvmd add_change cg 1 we joined Mar 2 08:26:39 xm01 cluster-dlm: add_change: clvmd add_change cg 1 counts member 2 joined 1 remove 0 failed 0 Mar 2 08:26:39 xm01 cluster-dlm: check_fencing_done: clvmd check_fencing done Mar 2 08:26:39 xm01 cluster-dlm: check_quorum_done: clvmd check_quorum disabled Mar 2 08:26:39 xm01 cluster-dlm: check_fs_done: clvmd check_fs none registered Mar 2 08:26:39 xm01 cluster-dlm: send_info: clvmd send_start cg 1 flags 1 data2 0 counts 0 2 1 0 0 Mar 2 08:26:39 xm01 cluster-dlm: receive_start: clvmd receive_start 16777316:1 len 80 Mar 2 08:26:39 xm01 cluster-dlm: match_change: clvmd match_change 16777316:1 matches cg 1 Mar 2 08:26:39 xm01 cluster-dlm: wait_messages_done: clvmd wait_messages cg 1 need 1 of 2 Mar 2 08:26:39 xm01 cluster-dlm: receive_start: clvmd receive_start 33554532:4 len 80 Mar 2 08:26:39 xm01 cluster-dlm: match_change: clvmd match_change 33554532:4 matches cg 1 Mar 2 08:26:39 xm01 cluster-dlm: wait_messages_done: clvmd wait_messages cg 1 got all 2 Mar 2 08:26:39 xm01 cluster-dlm: start_kernel: clvmd start_kernel cg 1 member_count 2 Mar 2 08:26:39 xm01 cluster-dlm: do_sysfs: write "1090842362" to "/sys/kernel/dlm/clvmd/id" Mar 2 08:26:39 xm01 cluster-dlm: set_configfs_members: set_members mkdir "/sys/kernel/config/dlm/cluster/spaces/clvmd/nodes/16777316" Mar 2 08:26:39 xm01 cluster-dlm: set_configfs_members: set_members mkdir "/sys/kernel/config/dlm/cluster/spaces/clvmd/nodes/33554532" Mar 2 08:26:39 xm01 cluster-dlm: do_sysfs: write "1" to "/sys/kernel/dlm/clvmd/control" Mar 2 08:26:39 xm01 cluster-dlm: do_sysfs: write "0" to "/sys/kernel/dlm/clvmd/event_done" Mar 2 08:26:39 xm01 cluster-dlm: set_plock_ckpt_node: clvmd set_plock_ckpt_node from 0 to 33554532 Mar 2 08:26:39 xm01 cluster-dlm: receive_plocks_stored: clvmd receive_plocks_stored 33554532:4 flags a sig 0 need_plocks 1 Mar 2 08:26:39 xm01 cluster-dlm: match_change: clvmd match_change 33554532:4 matches cg 1 Mar 2 08:26:39 xm01 cluster-dlm: retrieve_plocks: clvmd retrieve_plocks Mar 2 08:26:39 xm01 cluster-dlm: retrieve_plocks: clvmd retrieve_plocks first 0 last 0 r_count 0 p_count 0 sig 0 Mar 2 08:26:39 xm01 cluster-dlm: process_uevent: uevent: add@/devices/virtual/misc/dlm_clvmd Mar 2 08:26:40 xm01 kernel: [ 108.753367] dlm: connecting to 33554532 Mar 2 08:26:41 xm01 clvmd: Cluster LVM daemon started - connected to OpenAIS Mar 2 08:26:43 xm01 cmirrord[6655]: Starting cmirrord: Mar 2 08:26:43 xm01 cmirrord[6655]: Built: Aug 20 2010 16:04:44 Mar 2 08:26:45 xm01 lrmd: [5576]: info: operation start[41] on clvm:0 for client 5579: pid 6598 exited with return code 0 Mar 2 08:26:45 xm01 crmd: [5579]: info: process_lrm_event: LRM operation clvm:0_start_0 (call=41, rc=0, cib-update=49, confirmed=true) ok Mar 2 08:26:45 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=155:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=clvm:0_monitor_10000 ) Mar 2 08:26:45 xm01 lrmd: [5576]: info: rsc:clvm:0 monitor[42] (pid 6665) Mar 2 08:26:45 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=182:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vg_svn:0_start_0 ) Mar 2 08:26:45 xm01 lrmd: [5576]: info: rsc:vg_svn:0 start[43] (pid 6666) Mar 2 08:26:45 xm01 lrmd: [5576]: info: operation monitor[42] on clvm:0 for client 5579: pid 6665 exited with return code 0 Mar 2 08:26:45 xm01 crmd: [5579]: info: process_lrm_event: LRM operation clvm:0_monitor_10000 (call=42, rc=0, cib-update=50, confirmed=false) ok Mar 2 08:26:45 xm01 LVM[6666]: INFO: Activating volume group vg_svn Mar 2 08:26:45 xm01 LVM[6666]: INFO: Reading all physical volumes. This may take a while... Found volume group "vg_srvsvn2" using metadata type lvm2 Found volume group "vg_srvsvn1" using metadata t ype lvm2 Found volume group "vg_svn" using metadata type lvm2 Mar 2 08:26:46 xm01 LVM[6666]: INFO: 4 logical volume(s) in volume group "vg_svn" now active Mar 2 08:26:46 xm01 lrmd: [5576]: info: operation start[43] on vg_svn:0 for client 5579: pid 6666 exited with return code 0 Mar 2 08:26:46 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vg_svn:0_start_0 (call=43, rc=0, cib-update=51, confirmed=true) ok Mar 2 08:26:46 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=174:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig-pri:0_start_0 ) Mar 2 08:26:46 xm01 lrmd: [5576]: info: rsc:vmconfig-pri:0 start[44] (pid 6742) Mar 2 08:26:46 xm01 Filesystem[6742]: INFO: Running start for /dev/drbd3 on /vmconfig Mar 2 08:26:46 xm01 ocfs2_controld: new client connection 5 Mar 2 08:26:46 xm01 ocfs2_controld: client msg Mar 2 08:26:46 xm01 ocfs2_controld: client message 0 from 5: MOUNT Mar 2 08:26:46 xm01 ocfs2_controld: start_mount: uuid "A004730022F3454BA07C0747DF400E29", device "/dev/drbd3", service "ocfs2" Mar 2 08:26:46 xm01 ocfs2_controld: Adding service "ocfs2" to device "/dev/drbd3" uuid "A004730022F3454BA07C0747DF400E29" Mar 2 08:26:46 xm01 ocfs2_controld: Starting join for group "ocfs2:A004730022F3454BA07C0747DF400E29" Mar 2 08:26:46 xm01 ocfs2_controld: cpg_join succeeded Mar 2 08:26:46 xm01 ocfs2_controld: start_mount returns 0 Mar 2 08:26:46 xm01 ocfs2_controld: confchg called Mar 2 08:26:46 xm01 ocfs2_controld: group "ocfs2:A004730022F3454BA07C0747DF400E29" confchg: members 2, left 0, joined 1 Mar 2 08:26:46 xm01 ocfs2_controld: Node 16777316 joins group ocfs2:A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 ocfs2_controld: This node joins group ocfs2:A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 ocfs2_controld: Filling node 16777316 to group ocfs2:A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 ocfs2_controld: Filling node 33554532 to group ocfs2:A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 ocfs2_controld: Registering mountgroup A004730022F3454BA07C0747DF400E29 with dlm_controld Mar 2 08:26:46 xm01 ocfs2_controld: Registering "A004730022F3454BA07C0747DF400E29" with dlm_controld Mar 2 08:26:46 xm01 ocfs2_controld: message from dlmcontrol Mar 2 08:26:46 xm01 ocfs2_controld: Registration of "A004730022F3454BA07C0747DF400E29" complete Mar 2 08:26:46 xm01 ocfs2_controld: Mountgroup A004730022F3454BA07C0747DF400E29 successfully registered with dlm_controld Mar 2 08:26:46 xm01 ocfs2_controld: notify_mount_client sending 0 "OK" Mar 2 08:26:46 xm01 ocfs2_controld: Notified client: 1 Mar 2 08:26:46 xm01 cluster-dlm: process_uevent: uevent: add@/kernel/dlm/A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 cluster-dlm: process_uevent: kernel: add@ A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 cluster-dlm: process_uevent: uevent: online@/kernel/dlm/A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 cluster-dlm: process_uevent: kernel: online@ A004730022F3454BA07C0747DF400E29 Mar 2 08:26:46 xm01 cluster-dlm: log_config: dlm:ls:A004730022F3454BA07C0747DF400E29 conf 2 1 0 memb 16777316 33554532 join 16777316 left Mar 2 08:26:46 xm01 cluster-dlm: add_change: A004730022F3454BA07C0747DF400E29 add_change cg 1 joined nodeid 16777316 Mar 2 08:26:46 xm01 cluster-dlm: add_change: A004730022F3454BA07C0747DF400E29 add_change cg 1 we joined Mar 2 08:26:46 xm01 cluster-dlm: add_change: A004730022F3454BA07C0747DF400E29 add_change cg 1 counts member 2 joined 1 remove 0 failed 0 Mar 2 08:26:46 xm01 cluster-dlm: check_fencing_done: A004730022F3454BA07C0747DF400E29 check_fencing done Mar 2 08:26:46 xm01 cluster-dlm: check_quorum_done: A004730022F3454BA07C0747DF400E29 check_quorum disabled Mar 2 08:26:46 xm01 cluster-dlm: check_fs_done: A004730022F3454BA07C0747DF400E29 check_fs done Mar 2 08:26:46 xm01 cluster-dlm: send_info: A004730022F3454BA07C0747DF400E29 send_start cg 1 flags 1 data2 0 counts 0 2 1 0 0 Mar 2 08:26:46 xm01 cluster-dlm: receive_start: A004730022F3454BA07C0747DF400E29 receive_start 16777316:1 len 80 Mar 2 08:26:46 xm01 cluster-dlm: match_change: A004730022F3454BA07C0747DF400E29 match_change 16777316:1 matches cg 1 Mar 2 08:26:46 xm01 cluster-dlm: wait_messages_done: A004730022F3454BA07C0747DF400E29 wait_messages cg 1 need 1 of 2 Mar 2 08:26:46 xm01 cluster-dlm: receive_start: A004730022F3454BA07C0747DF400E29 receive_start 33554532:2 len 80 Mar 2 08:26:46 xm01 cluster-dlm: match_change: A004730022F3454BA07C0747DF400E29 match_change 33554532:2 matches cg 1 Mar 2 08:26:46 xm01 cluster-dlm: wait_messages_done: A004730022F3454BA07C0747DF400E29 wait_messages cg 1 got all 2 Mar 2 08:26:46 xm01 cluster-dlm: start_kernel: A004730022F3454BA07C0747DF400E29 start_kernel cg 1 member_count 2 Mar 2 08:26:46 xm01 cluster-dlm: do_sysfs: write "2076474242" to "/sys/kernel/dlm/A004730022F3454BA07C0747DF400E29/id" Mar 2 08:26:46 xm01 cluster-dlm: set_configfs_members: set_members mkdir "/sys/kernel/config/dlm/cluster/spaces/A004730022F3454BA07C0747DF400E29/nodes/16777316" Mar 2 08:26:46 xm01 cluster-dlm: set_configfs_members: set_members mkdir "/sys/kernel/config/dlm/cluster/spaces/A004730022F3454BA07C0747DF400E29/nodes/33554532" Mar 2 08:26:46 xm01 cluster-dlm: do_sysfs: write "1" to "/sys/kernel/dlm/A004730022F3454BA07C0747DF400E29/control" Mar 2 08:26:46 xm01 cluster-dlm: do_sysfs: write "0" to "/sys/kernel/dlm/A004730022F3454BA07C0747DF400E29/event_done" Mar 2 08:26:46 xm01 cluster-dlm: set_plock_ckpt_node: A004730022F3454BA07C0747DF400E29 set_plock_ckpt_node from 0 to 33554532 Mar 2 08:26:46 xm01 cluster-dlm: receive_plocks_stored: A004730022F3454BA07C0747DF400E29 receive_plocks_stored 33554532:2 flags a sig 0 need_plocks 1 Mar 2 08:26:46 xm01 cluster-dlm: match_change: A004730022F3454BA07C0747DF400E29 match_change 33554532:2 matches cg 1 Mar 2 08:26:46 xm01 cluster-dlm: retrieve_plocks: A004730022F3454BA07C0747DF400E29 retrieve_plocks Mar 2 08:26:46 xm01 cluster-dlm: retrieve_plocks: A004730022F3454BA07C0747DF400E29 retrieve_plocks first 0 last 0 r_count 0 p_count 0 sig 0 Mar 2 08:26:46 xm01 ocfs2_controld: client msg Mar 2 08:26:46 xm01 ocfs2_controld: client message 1 from 5: MRESULT Mar 2 08:26:46 xm01 ocfs2_controld: complete_mount: uuid "A004730022F3454BA07C0747DF400E29", errcode "0", service "ocfs2" Mar 2 08:26:46 xm01 ocfs2_controld: client msg Mar 2 08:26:46 xm01 ocfs2_controld: client 5 fd 13 dead Mar 2 08:26:46 xm01 ocfs2_controld: client 5 fd -1 dead Mar 2 08:26:46 xm01 kernel: [ 115.709627] ocfs2: Mounting device (147,3) on (node 1677731, slot 1) with ordered data mode. Mar 2 08:26:46 xm01 ocfs2_hb_ctl[6806]: ocfs2_hb_ctl /sbin/ocfs2_hb_ctl -P -d /dev/drbd3 Mar 2 08:26:46 xm01 ocfs2_controld: new client connection 5 Mar 2 08:26:46 xm01 ocfs2_controld: client msg Mar 2 08:26:46 xm01 ocfs2_controld: client message 6 from 5: LISTCLUSTERS Mar 2 08:26:46 xm01 ocfs2_controld: client msg Mar 2 08:26:46 xm01 ocfs2_controld: client 5 fd 13 dead Mar 2 08:26:46 xm01 ocfs2_controld: client 5 fd -1 dead Mar 2 08:26:46 xm01 lrmd: [5576]: info: operation start[44] on vmconfig-pri:0 for client 5579: pid 6742 exited with return code 0 Mar 2 08:26:47 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig-pri:0_start_0 (call=44, rc=0, cib-update=52, confirmed=true) ok Mar 2 08:26:47 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=175:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=vmconfig-pri:0_monitor_20000 ) Mar 2 08:26:47 xm01 lrmd: [5576]: info: rsc:vmconfig-pri:0 monitor[45] (pid 6813) Mar 2 08:26:47 xm01 crmd: [5579]: info: do_lrm_rsc_op: Performing key=197:7:0:f0bab4e2-9453-4a56-ad94-bf84fd1bfcdd op=VMSVN_start_0 ) Mar 2 08:26:47 xm01 lrmd: [5576]: info: rsc:VMSVN start[46] (pid 6814) Mar 2 08:26:47 xm01 lrmd: [5576]: info: operation monitor[45] on vmconfig-pri:0 for client 5579: pid 6813 exited with return code 0 Mar 2 08:26:47 xm01 crmd: [5579]: info: process_lrm_event: LRM operation vmconfig-pri:0_monitor_20000 (call=45, rc=0, cib-update=53, confirmed=false) ok Mar 2 08:26:47 xm01 external/ipmi[6894]: [6905]: debug: ipmitool output: Chassis Power is on Mar 2 08:26:48 xm01 stonith: [6890]: info: external/ipmi device OK. Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51712 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51728 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51744 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51760 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51776 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/51792 Mar 2 08:26:50 xm01 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/268441856 Mar 2 08:26:51 xm01 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/0 Mar 2 08:26:51 xm01 logger: /etc/xen/scripts/block: Writing backend/vbd/1/51744/physical-device fd:6 to xenstore. Mar 2 08:26:51 xm01 logger: /etc/xen/scripts/block: Writing backend/vbd/1/51744/hotplug-status connected to xenstore. Any ideas why the VM went down and up once the node was fail backed? Thanks Daniel At 04:53 a.m. 02/03/2012, Florian Haas wrote: >On Thu, Mar 1, 2012 at 6:12 PM, Daniel Grunblatt ><dgrunblatt at invap.com.ar> wrote: > Andreas, >Lars, > > Thanks much for the quick >response. > > I made the changes. > [lots of >stuff] > > And here's what happened: > [lots of >stuff] Ever heard of pastebin? > /sbin/drbdadm >fence-peer minor-3 exit code 126 (0x7e00) > Mar >Â 1 13:37:21 xm01 kernel: [ Â 97.020369] block >drbd3: fence-peer helper > broken, returned 126 > From >http://tldp.org/LDP/abs/html/exitcodes.html: >126 Command invoked cannot >execute Permission problem or command >is not an executable Did you forget to do chmod >+x on that stonith_admin-fence-peer.sh script >when you installed it? > [lots more stuff >clipped] Hope this helps. Florian -- Need help >with High Availability? >http://www.hastexo.com/now >_______________________________________________ >drbd-user mailing list >drbd-user at lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user