Jul 23 15:50:50 mail2 ipfail: [11381]: info: Link Status update: Link mail1.example.com/eth0 now has status dead Jul 23 15:50:50 mail2 heartbeat: [11353]: info: Link mail1.example.com:eth0 dead. Jul 23 15:50:50 mail2 ipfail: [11381]: info: Asking other side for ping node count. Jul 23 15:50:50 mail2 ipfail: [11381]: info: Checking remote count of ping nodes. Jul 23 15:51:00 mail2 ipfail: [11381]: info: Status update: Node mail1.example.com now has status dead Jul 23 15:51:00 mail2 heartbeat: [11353]: WARN: node mail1.example.com: is dead Jul 23 15:51:00 mail2 ipfail: [11381]: info: NS: We are still alive! Jul 23 15:51:00 mail2 heartbeat: [11353]: WARN: No STONITH device configured. Jul 23 15:51:00 mail2 heartbeat: [11353]: WARN: Shared disks are not protected. Jul 23 15:51:00 mail2 heartbeat: [11353]: info: Resources being acquired from mail1.example.com. Jul 23 15:51:00 mail2 harc[11415]: [11425]: info: Running /etc/ha.d/rc.d/status status Jul 23 15:51:00 mail2 heartbeat: [11416]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys mail2.example.com] to acquire. Jul 23 15:51:00 mail2 heartbeat: [11416]: info: Writing type [resource] message to FIFO Jul 23 15:51:00 mail2 heartbeat: [11416]: info: FIFO message [type resource] written rc=79 Jul 23 15:51:00 mail2 heartbeat: [11353]: info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1)) Jul 23 15:51:00 mail2 heartbeat: [11353]: info: Managed req_our_resources process 11416 exited with return code 0. Jul 23 15:51:00 mail2 heartbeat: [11353]: info: AnnounceTakeover(local 1, foreign 1, reason 'req_our_resources' (1)) Jul 23 15:51:00 mail2 mach_down[11440]: [11461]: info: Taking over resource group IPaddr2::10.9.9.6/28/eth0/10.9.9.15 Jul 23 15:51:00 mail2 ResourceManager[11462]: [11473]: info: Acquiring resource group: mail1.example.com IPaddr2::10.9.9.6/28/eth0/10.9.9.15 drbddisk::home Jul 23 15:51:00 mail2 IPaddr2[11485]: [11542]: INFO: Resource is stopped Jul 23 15:51:00 mail2 ResourceManager[11462]: [11556]: info: Running /etc/ha.d/resource.d/IPaddr2 10.9.9.6/28/eth0/10.9.9.15 start Jul 23 15:51:00 mail2 IPaddr2[11587]: [11622]: INFO: ip -f inet addr add 10.9.9.6/28 brd 10.9.9.15 dev eth0 Jul 23 15:51:00 mail2 IPaddr2[11587]: [11624]: INFO: ip link set eth0 up Jul 23 15:51:00 mail2 IPaddr2[11587]: [11626]: INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.9.9.6 eth0 10.9.9.6 auto not_used not_used Jul 23 15:51:00 mail2 IPaddr2[11558]: [11630]: INFO: Success Jul 23 15:51:00 mail2 ResourceManager[11462]: [11660]: info: Running /etc/ha.d/resource.d/drbddisk home start Jul 23 15:51:12 mail2 ResourceManager[11462]: [11696]: ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk Jul 23 15:51:12 mail2 ResourceManager[11462]: [11697]: CRIT: Giving up resources due to failure of drbddisk::home Jul 23 15:51:12 mail2 ResourceManager[11462]: [11698]: info: Releasing resource group: mail1.example.com IPaddr2::10.9.9.6/28/eth0/10.9.9.15 drbddisk::home Jul 23 15:51:12 mail2 ResourceManager[11462]: [11713]: info: Running /etc/ha.d/resource.d/drbddisk home stop Jul 23 15:51:12 mail2 ResourceManager[11462]: [11733]: info: Running /etc/ha.d/resource.d/IPaddr2 10.9.9.6/28/eth0/10.9.9.15 stop Jul 23 15:51:13 mail2 IPaddr2[11764]: [11793]: INFO: ip -f inet addr delete 10.9.9.6/28 dev eth0 Jul 23 15:51:13 mail2 IPaddr2[11764]: [11795]: INFO: ip -o -f inet addr show eth0 Jul 23 15:51:13 mail2 IPaddr2[11735]: [11797]: INFO: Success Jul 23 15:51:13 mail2 mach_down[11440]: [11801]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired Jul 23 15:51:13 mail2 mach_down[11440]: [11805]: info: mach_down takeover complete for node mail1.example.com. Jul 23 15:51:13 mail2 heartbeat: [11353]: info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1)) Jul 23 15:51:13 mail2 heartbeat: [11353]: info: mach_down takeover complete. Jul 23 15:51:13 mail2 heartbeat: [11353]: info: AnnounceTakeover(local 1, foreign 1, reason 'mach_down' (1)) Jul 23 15:51:13 mail2 heartbeat: [11353]: info: Managed status process 11415 exited with return code 0. Jul 23 15:51:35 mail2 kernel: drbd0: peer( Primary -> Secondary ) Jul 23 15:51:43 mail2 hb_standby[11815]: [11821]: Going standby [foreign]. Jul 23 15:51:43 mail2 heartbeat: [11353]: info: mail2.example.com wants to go standby [foreign] Jul 23 15:51:43 mail2 heartbeat: [11353]: info: i_hold_resources: 3 Jul 23 15:51:43 mail2 heartbeat: [11353]: info: New standby state: 1 Jul 23 15:51:43 mail2 heartbeat: [11353]: WARN: Standby timer has 9930 ms left Jul 23 15:51:43 mail2 heartbeat: [11353]: WARN: Shutdown delayed until current resource activity finishes. Jul 23 15:51:44 mail2 heartbeat: [11353]: WARN: Standby timer has 9500 ms left Jul 23 15:51:44 mail2 heartbeat: [11353]: WARN: Standby timer has 9030 ms left Jul 23 15:51:44 mail2 heartbeat: [11353]: WARN: Standby timer has 9020 ms left Jul 23 15:51:46 mail2 heartbeat: [11353]: WARN: Standby timer has 7030 ms left Jul 23 15:51:46 mail2 heartbeat: [11353]: WARN: Standby timer has 7030 ms left Jul 23 15:51:46 mail2 heartbeat: [11353]: WARN: Standby timer has 7020 ms left Jul 23 15:51:48 mail2 heartbeat: [11353]: WARN: Standby timer has 5030 ms left Jul 23 15:51:48 mail2 heartbeat: [11353]: WARN: Standby timer has 5020 ms left Jul 23 15:51:50 mail2 heartbeat: [11353]: WARN: Standby timer has 3030 ms left Jul 23 15:51:50 mail2 heartbeat: [11353]: WARN: Standby timer has 3020 ms left Jul 23 15:51:52 mail2 heartbeat: [11353]: WARN: Standby timer has 1030 ms left Jul 23 15:51:52 mail2 heartbeat: [11353]: WARN: Standby timer has 1020 ms left