[DRBD-user] secondary node didnt start services when the primary failed and also didnt failback

Andrea Freire solange_sun at hotmail.com
Thu Nov 25 06:30:06 CET 2010

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.



Avanzando es la unica forma de dejar atras lo que no necesitamos. Andrea 



From: solange_sun at hotmail.com
To: linux-ha at lists.linux-ha.org
Subject: secondary node didnt start services when the primary failed and also didnt failback
Date: Thu, 25 Nov 2010 00:09:24 -0500








I have a problem with the implementation of a high availability cluster, I have made the implementation as shown on this page http://www.elastix.org/es/component/kunena/26-tips-and-tricks/53819-elastix-160-clusters-with-drbd-and-heartbeat.html,follow all the steps but for tomcat and postgresql on a drbd disk, when the service heartbeat starts all the services on the primary server starts too but then I wanted to simulate the failure I shutdown the primary node ,and the secondary node after 5 minutes did not start the services, I began to stop the firewall -the iptables- because the sanity check script  sent a few errors of connection in the crm test, mount and umount drbd partition manually works, but when I let the heartbeat do this, and when I shutdown the primary nodeb is put in a secondary in wfconnection and forever attached logs and server configurations one is a dell and the other a hp proliant but I dont.
heartbeat 2.1.3heartbeat-stonith 2.1.3heartbeat-pils 2.1.3drbd83 8.3.8kmod-drbd83 838ha-log in primary nodeheartbeat[4427]: 2010/11/22_10:13:51 info: killing HBFIFO process 4431 with signal 15heartbeat[4427]: 2010/11/22_10:13:51 info: killing HBWRITE process 4432 with signal 15heartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4433 exited. 3 remainingheartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4431 exited. 2 remainingheartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4432 exited. 1 remainingheartbeat[4427]: 2010/11/22_10:13:51 info: server1.drbd Heartbeat shutdown complete.heartbeat[3592]: 2010/11/22_10:21:03 info: Version 2 support: falseheartbeat[3592]: 2010/11/22_10:21:03 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3592]: 2010/11/22_10:21:03 info: **************************heartbeat[3592]: 2010/11/22_10:21:03 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3593]: 2010/11/22_10:21:03 info: heartbeat: version 2.1.3heartbeat[3593]: 2010/11/22_10:21:03 info: Heartbeat generation: 1290373957heartbeat[3593]: 2010/11/22_10:21:03 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3593]: 2010/11/22_10:21:03 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3593]: 2010/11/22_10:21:03 info: Local status now set to: 'up'heartbeat[3593]: 2010/11/22_10:21:04 info: Link server1.drbd:eth1 up.heartbeat[3593]: 2010/11/22_10:23:04 WARN: node server2.drbd: is deadheartbeat[3593]: 2010/11/22_10:23:04 info: Comm_now_up(): updating status to activeheartbeat[3593]: 2010/11/22_10:23:04 info: Local status now set to: 'active'heartbeat[3593]: 2010/11/22_10:23:04 WARN: No STONITH device configured.heartbeat[3593]: 2010/11/22_10:23:04 WARN: Shared disks are not protected.heartbeat[3593]: 2010/11/22_10:23:04 info: Resources being acquired from server2.drbd.harc[4274]:    2010/11/22_10:23:04 info: Running /etc/ha.d/rc.d/status statusmach_down[4304]:    2010/11/22_10:23:04 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[4275]: 2010/11/22_10:23:04 info: Local Resource acquisition completed.heartbeat[3593]: 2010/11/22_10:23:04 info: mach_down takeover complete.heartbeat[3593]: 2010/11/22_10:23:04 info: Initial resource acquisition complete (mach_down)mach_down[4304]:    2010/11/22_10:23:04 info: mach_down takeover complete for node server2.drbd.heartbeat[3593]: 2010/11/22_10:23:14 info: Local Resource acquisition completed. (none)heartbeat[3593]: 2010/11/22_10:23:14 info: local resource transition completed.heartbeat[3593]: 2010/11/22_10:35:48 info: Heartbeat shutdown in progress. (3593)heartbeat[4703]: 2010/11/22_10:35:48 info: Giving up all HA resources.ResourceManager[4716]:    2010/11/22_10:35:48 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4716]:    2010/11/22_10:35:48 info: Running /etc/init.d/tomcat6  stopResourceManager[4716]:    2010/11/22_10:35:52 info: Running /etc/init.d/postgresql  stopResourceManager[4716]:    2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[4957]:    2010/11/22_10:35:53 INFO:  SuccessResourceManager[4716]:    2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5040]:    2010/11/22_10:35:53 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5040]:    2010/11/22_10:35:53 INFO: Trying to unmount /replicaFilesystem[5040]:    2010/11/22_10:35:53 INFO: unmounted /replica successfullyFilesystem[5029]:    2010/11/22_10:35:53 INFO:  SuccessResourceManager[4716]:    2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[4703]: 2010/11/22_10:35:53 info: All HA resources relinquished.heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBFIFO process 3609 with signal 15heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBWRITE process 3610 with signal 15heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBREAD process 3611 with signal 15heartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3611 exited. 3 remainingheartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3610 exited. 2 remainingheartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3609 exited. 1 remainingheartbeat[3593]: 2010/11/22_10:35:55 info: server1.drbd Heartbeat shutdown complete.heartbeat[5189]: 2010/11/22_10:36:36 info: Version 2 support: falseheartbeat[5189]: 2010/11/22_10:36:36 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[5189]: 2010/11/22_10:36:36 info: **************************heartbeat[5189]: 2010/11/22_10:36:36 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[5190]: 2010/11/22_10:36:36 info: heartbeat: version 2.1.3heartbeat[5190]: 2010/11/22_10:36:36 info: Heartbeat generation: 1290373958heartbeat[5190]: 2010/11/22_10:36:36 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[5190]: 2010/11/22_10:36:36 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[5190]: 2010/11/22_10:36:36 info: Local status now set to: 'up'heartbeat[5190]: 2010/11/22_10:36:37 info: Link server1.drbd:eth1 up.heartbeat[5190]: 2010/11/22_10:38:37 WARN: node server2.drbd: is deadheartbeat[5190]: 2010/11/22_10:38:37 info: Comm_now_up(): updating status to activeheartbeat[5190]: 2010/11/22_10:38:37 info: Local status now set to: 'active'heartbeat[5190]: 2010/11/22_10:38:37 WARN: No STONITH device configured.heartbeat[5190]: 2010/11/22_10:38:37 WARN: Shared disks are not protected.heartbeat[5190]: 2010/11/22_10:38:37 info: Resources being acquired from server2.drbd.harc[5310]:    2010/11/22_10:38:37 info: Running /etc/ha.d/rc.d/status statusmach_down[5340]:    2010/11/22_10:38:37 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[5311]: 2010/11/22_10:38:37 info: Local Resource acquisition completed.mach_down[5340]:    2010/11/22_10:38:37 info: mach_down takeover complete for node server2.drbd.heartbeat[5190]: 2010/11/22_10:38:37 info: mach_down takeover complete.heartbeat[5190]: 2010/11/22_10:38:37 info: Initial resource acquisition complete (mach_down)harc[5403]:    2010/11/22_10:38:37 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[5403]:    2010/11/22_10:38:37 received ip-request-resp drbddisk::r0 OK yesResourceManager[5424]:    2010/11/22_10:38:37 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[5424]:    2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[5495]:    2010/11/22_10:38:37 INFO:  Resource is stoppedResourceManager[5424]:    2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[5576]:    2010/11/22_10:38:37 INFO: Running start for /dev/drbd0 on /replicaFilesystem[5565]:    2010/11/22_10:38:37 INFO:  SuccessIPaddr[5645]:    2010/11/22_10:38:37 INFO:  Resource is stoppedResourceManager[5424]:    2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[5743]:    2010/11/22_10:38:37 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[5743]:    2010/11/22_10:38:38 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[5714]:    2010/11/22_10:38:38 INFO:  SuccessResourceManager[5424]:    2010/11/22_10:38:38 info: Running /etc/init.d/postgresql  startResourceManager[5424]:    2010/11/22_10:38:40 info: Running /etc/init.d/tomcat6  startheartbeat[5190]: 2010/11/22_10:38:47 info: Local Resource acquisition completed. (none)heartbeat[5190]: 2010/11/22_10:38:47 info: local resource transition completed.heartbeat[3465]: 2010/11/22_10:46:09 info: Version 2 support: falseheartbeat[3465]: 2010/11/22_10:46:09 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3465]: 2010/11/22_10:46:09 info: **************************heartbeat[3465]: 2010/11/22_10:46:09 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3466]: 2010/11/22_10:46:09 info: heartbeat: version 2.1.3heartbeat[3466]: 2010/11/22_10:46:09 info: Heartbeat generation: 1290373959heartbeat[3466]: 2010/11/22_10:46:09 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3466]: 2010/11/22_10:46:09 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3466]: 2010/11/22_10:46:09 info: Local status now set to: 'up'heartbeat[3466]: 2010/11/22_10:46:10 info: Link server1.drbd:eth1 up.heartbeat[3466]: 2010/11/22_10:48:10 WARN: node server2.drbd: is deadheartbeat[3466]: 2010/11/22_10:48:10 info: Comm_now_up(): updating status to activeheartbeat[3466]: 2010/11/22_10:48:10 info: Local status now set to: 'active'heartbeat[3466]: 2010/11/22_10:48:10 WARN: No STONITH device configured.heartbeat[3466]: 2010/11/22_10:48:10 WARN: Shared disks are not protected.heartbeat[3466]: 2010/11/22_10:48:10 info: Resources being acquired from server2.drbd.harc[4145]:    2010/11/22_10:48:10 info: Running /etc/ha.d/rc.d/status statusheartbeat[4146]: 2010/11/22_10:48:10 info: Local Resource acquisition completed.mach_down[4186]:    2010/11/22_10:48:10 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4186]:    2010/11/22_10:48:10 info: mach_down takeover complete for node server2.drbd.heartbeat[3466]: 2010/11/22_10:48:10 info: mach_down takeover complete.heartbeat[3466]: 2010/11/22_10:48:10 info: Initial resource acquisition complete (mach_down)harc[4238]:    2010/11/22_10:48:10 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4238]:    2010/11/22_10:48:10 received ip-request-resp drbddisk::r0 OK yesResourceManager[4259]:    2010/11/22_10:48:10 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4259]:    2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4330]:    2010/11/22_10:48:10 INFO:  Resource is stoppedResourceManager[4259]:    2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4411]:    2010/11/22_10:48:10 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4400]:    2010/11/22_10:48:10 INFO:  SuccessIPaddr[4479]:    2010/11/22_10:48:10 INFO:  Resource is stoppedResourceManager[4259]:    2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4577]:    2010/11/22_10:48:11 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4577]:    2010/11/22_10:48:11 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4548]:    2010/11/22_10:48:11 INFO:  SuccessResourceManager[4259]:    2010/11/22_10:48:11 info: Running /etc/init.d/postgresql  startResourceManager[4259]:    2010/11/22_10:48:13 info: Running /etc/init.d/tomcat6  startheartbeat[3466]: 2010/11/22_10:48:20 info: Local Resource acquisition completed. (none)heartbeat[3466]: 2010/11/22_10:48:20 info: local resource transition completed.heartbeat[3466]: 2010/11/22_10:49:52 info: Heartbeat shutdown in progress. (3466)heartbeat[5176]: 2010/11/22_10:49:52 info: Giving up all HA resources.ResourceManager[5189]:    2010/11/22_10:49:52 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[5189]:    2010/11/22_10:49:52 info: Running /etc/init.d/tomcat6  stopResourceManager[5189]:    2010/11/22_10:49:56 info: Running /etc/init.d/postgresql  stopResourceManager[5189]:    2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5459]:    2010/11/22_10:49:57 INFO: ifconfig eth0:0 downIPaddr[5430]:    2010/11/22_10:49:57 INFO:  SuccessResourceManager[5189]:    2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5521]:    2010/11/22_10:49:57 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5521]:    2010/11/22_10:49:57 INFO: Trying to unmount /replicaFilesystem[5521]:    2010/11/22_10:49:57 INFO: unmounted /replica successfullyFilesystem[5510]:    2010/11/22_10:49:57 INFO:  SuccessResourceManager[5189]:    2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[5176]: 2010/11/22_10:49:57 info: All HA resources relinquished.heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBREAD process 3476 with signal 15heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBFIFO process 3474 with signal 15heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBWRITE process 3475 with signal 15heartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3476 exited. 3 remainingheartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3475 exited. 2 remainingheartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3474 exited. 1 remainingheartbeat[3466]: 2010/11/22_10:49:59 info: server1.drbd Heartbeat shutdown complete.heartbeat[3401]: 2010/11/22_10:58:15 info: Version 2 support: falseheartbeat[3401]: 2010/11/22_10:58:15 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3401]: 2010/11/22_10:58:15 info: **************************heartbeat[3401]: 2010/11/22_10:58:15 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3402]: 2010/11/22_10:58:15 info: heartbeat: version 2.1.3heartbeat[3402]: 2010/11/22_10:58:15 info: Heartbeat generation: 1290373960heartbeat[3402]: 2010/11/22_10:58:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3402]: 2010/11/22_10:58:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3402]: 2010/11/22_10:58:16 info: Local status now set to: 'up'heartbeat[3402]: 2010/11/22_10:58:17 info: Link server1.drbd:eth1 up.heartbeat[3402]: 2010/11/22_11:00:16 WARN: node server2.drbd: is deadheartbeat[3402]: 2010/11/22_11:00:16 info: Comm_now_up(): updating status to activeheartbeat[3402]: 2010/11/22_11:00:16 info: Local status now set to: 'active'heartbeat[3402]: 2010/11/22_11:00:16 WARN: No STONITH device configured.heartbeat[3402]: 2010/11/22_11:00:16 WARN: Shared disks are not protected.heartbeat[3402]: 2010/11/22_11:00:16 info: Resources being acquired from server2.drbd.harc[4070]:    2010/11/22_11:00:16 info: Running /etc/ha.d/rc.d/status statusmach_down[4100]:    2010/11/22_11:00:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[4071]: 2010/11/22_11:00:17 info: Local Resource acquisition completed.heartbeat[3402]: 2010/11/22_11:00:17 info: mach_down takeover complete.heartbeat[3402]: 2010/11/22_11:00:17 info: Initial resource acquisition complete (mach_down)mach_down[4100]:    2010/11/22_11:00:17 info: mach_down takeover complete for node server2.drbd.harc[4163]:    2010/11/22_11:00:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4163]:    2010/11/22_11:00:17 received ip-request-resp drbddisk::r0 OK yesResourceManager[4184]:    2010/11/22_11:00:17 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4184]:    2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4255]:    2010/11/22_11:00:17 INFO:  Resource is stoppedResourceManager[4184]:    2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4336]:    2010/11/22_11:00:17 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4325]:    2010/11/22_11:00:17 INFO:  SuccessIPaddr[4404]:    2010/11/22_11:00:17 INFO:  Resource is stoppedResourceManager[4184]:    2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4502]:    2010/11/22_11:00:17 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4502]:    2010/11/22_11:00:17 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4473]:    2010/11/22_11:00:17 INFO:  SuccessResourceManager[4184]:    2010/11/22_11:00:17 info: Running /etc/init.d/postgresql  startResourceManager[4184]:    2010/11/22_11:00:20 info: Running /etc/init.d/tomcat6  startheartbeat[3402]: 2010/11/22_11:00:27 info: Local Resource acquisition completed. (none)heartbeat[3402]: 2010/11/22_11:00:27 info: local resource transition completed.heartbeat[3402]: 2010/11/22_11:01:23 info: Heartbeat shutdown in progress. (3402)heartbeat[5135]: 2010/11/22_11:01:23 info: Giving up all HA resources.ResourceManager[5148]:    2010/11/22_11:01:23 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[5148]:    2010/11/22_11:01:23 info: Running /etc/init.d/tomcat6  stopResourceManager[5148]:    2010/11/22_11:01:27 info: Running /etc/init.d/postgresql  stopResourceManager[5148]:    2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5418]:    2010/11/22_11:01:28 INFO: ifconfig eth0:0 downIPaddr[5389]:    2010/11/22_11:01:28 INFO:  SuccessResourceManager[5148]:    2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5480]:    2010/11/22_11:01:28 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5480]:    2010/11/22_11:01:28 INFO: Trying to unmount /replicaFilesystem[5480]:    2010/11/22_11:01:28 INFO: unmounted /replica successfullyFilesystem[5469]:    2010/11/22_11:01:28 INFO:  SuccessResourceManager[5148]:    2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[5135]: 2010/11/22_11:01:28 info: All HA resources relinquished.heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBREAD process 3421 with signal 15heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBFIFO process 3418 with signal 15heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBWRITE process 3420 with signal 15heartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3420 exited. 3 remainingheartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3421 exited. 2 remainingheartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3418 exited. 1 remainingheartbeat[3402]: 2010/11/22_11:01:30 info: server1.drbd Heartbeat shutdown complete.heartbeat[3349]: 2010/11/22_11:08:16 info: Version 2 support: falseheartbeat[3349]: 2010/11/22_11:08:16 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3349]: 2010/11/22_11:08:16 info: **************************heartbeat[3349]: 2010/11/22_11:08:16 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3350]: 2010/11/22_11:08:16 info: heartbeat: version 2.1.3heartbeat[3350]: 2010/11/22_11:08:16 info: Heartbeat generation: 1290373961heartbeat[3350]: 2010/11/22_11:08:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3350]: 2010/11/22_11:08:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3350]: 2010/11/22_11:08:16 info: Local status now set to: 'up'heartbeat[3350]: 2010/11/22_11:08:17 info: Link server1.drbd:eth1 up.heartbeat[3350]: 2010/11/22_11:10:17 WARN: node server2.drbd: is deadheartbeat[3350]: 2010/11/22_11:10:17 info: Comm_now_up(): updating status to activeheartbeat[3350]: 2010/11/22_11:10:17 info: Local status now set to: 'active'heartbeat[3350]: 2010/11/22_11:10:17 WARN: No STONITH device configured.heartbeat[3350]: 2010/11/22_11:10:17 WARN: Shared disks are not protected.heartbeat[3350]: 2010/11/22_11:10:17 info: Resources being acquired from server2.drbd.harc[3985]:    2010/11/22_11:10:17 info: Running /etc/ha.d/rc.d/status statusheartbeat[3986]: 2010/11/22_11:10:17 info: Local Resource acquisition completed.mach_down[4015]:    2010/11/22_11:10:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4015]:    2010/11/22_11:10:17 info: mach_down takeover complete for node server2.drbd.heartbeat[3350]: 2010/11/22_11:10:17 info: mach_down takeover complete.heartbeat[3350]: 2010/11/22_11:10:17 info: Initial resource acquisition complete (mach_down)harc[4078]:    2010/11/22_11:10:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4078]:    2010/11/22_11:10:17 received ip-request-resp drbddisk::r0 OK yesResourceManager[4099]:    2010/11/22_11:10:17 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4099]:    2010/11/22_11:10:17 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4170]:    2010/11/22_11:10:17 INFO:  Resource is stoppedResourceManager[4099]:    2010/11/22_11:10:17 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4251]:    2010/11/22_11:10:17 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4240]:    2010/11/22_11:10:17 INFO:  SuccessIPaddr[4319]:    2010/11/22_11:10:18 INFO:  Resource is stoppedResourceManager[4099]:    2010/11/22_11:10:18 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4417]:    2010/11/22_11:10:18 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4417]:    2010/11/22_11:10:18 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4388]:    2010/11/22_11:10:18 INFO:  SuccessResourceManager[4099]:    2010/11/22_11:10:18 info: Running /etc/init.d/postgresql  startResourceManager[4099]:    2010/11/22_11:10:20 info: Running /etc/init.d/tomcat6  startheartbeat[3350]: 2010/11/22_11:10:27 info: Local Resource acquisition completed. (none)heartbeat[3350]: 2010/11/22_11:10:27 info: local resource transition completed.heartbeat[3444]: 2010/11/22_11:31:31 info: Version 2 support: falseheartbeat[3444]: 2010/11/22_11:31:31 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3444]: 2010/11/22_11:31:31 info: **************************heartbeat[3444]: 2010/11/22_11:31:31 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3445]: 2010/11/22_11:31:31 info: heartbeat: version 2.1.3heartbeat[3445]: 2010/11/22_11:31:31 info: Heartbeat generation: 1290373962heartbeat[3445]: 2010/11/22_11:31:31 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3445]: 2010/11/22_11:31:31 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3445]: 2010/11/22_11:31:31 info: Local status now set to: 'up'heartbeat[3445]: 2010/11/22_11:31:32 info: Link server1.drbd:eth1 up.heartbeat[3445]: 2010/11/22_11:33:32 WARN: node server2.drbd: is deadheartbeat[3445]: 2010/11/22_11:33:32 info: Comm_now_up(): updating status to activeheartbeat[3445]: 2010/11/22_11:33:32 info: Local status now set to: 'active'heartbeat[3445]: 2010/11/22_11:33:32 WARN: No STONITH device configured.heartbeat[3445]: 2010/11/22_11:33:32 WARN: Shared disks are not protected.heartbeat[3445]: 2010/11/22_11:33:32 info: Resources being acquired from server2.drbd.harc[3888]:    2010/11/22_11:33:32 info: Running /etc/ha.d/rc.d/status statusmach_down[3918]:    2010/11/22_11:33:32 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[3889]: 2010/11/22_11:33:32 info: Local Resource acquisition completed.heartbeat[3445]: 2010/11/22_11:33:32 info: mach_down takeover complete.heartbeat[3445]: 2010/11/22_11:33:32 info: Initial resource acquisition complete (mach_down)mach_down[3918]:    2010/11/22_11:33:32 info: mach_down takeover complete for node server2.drbd.harc[3981]:    2010/11/22_11:33:32 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[3981]:    2010/11/22_11:33:32 received ip-request-resp drbddisk::r0 OK yesResourceManager[4002]:    2010/11/22_11:33:32 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4002]:    2010/11/22_11:33:32 info: Running /etc/ha.d/resource.d/drbddisk r0 startheartbeat[3445]: 2010/11/22_11:33:42 info: Local Resource acquisition completed. (none)heartbeat[3445]: 2010/11/22_11:33:42 info: local resource transition completed.ResourceManager[4002]:    2010/11/22_11:33:44 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddiskResourceManager[4002]:    2010/11/22_11:33:44 CRIT: Giving up resources due to failure of drbddisk::r0ResourceManager[4002]:    2010/11/22_11:33:44 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4002]:    2010/11/22_11:33:44 info: Running /etc/init.d/tomcat6  stopResourceManager[4002]:    2010/11/22_11:33:44 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:44 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:45 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:45 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:45 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:46 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:46 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:46 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:47 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:47 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:48 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:49 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:49 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:49 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:50 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:50 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:50 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:51 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:51 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:51 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:52 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:52 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:52 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:53 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:53 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:53 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:54 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:54 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:54 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:55 info: Retrying failed stop operation [postgresql]ResourceManager[4002]:    2010/11/22_11:33:55 info: Running /etc/init.d/postgresql  stopResourceManager[4002]:    2010/11/22_11:33:55 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4002]:    2010/11/22_11:33:55 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[4002]:    2010/11/22_11:33:55 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[4002]:    2010/11/22_11:33:55 WARN: Machine reboot narrowly avoided!ResourceManager[4002]:    2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[4939]:    2010/11/22_11:33:55 INFO:  SuccessResourceManager[4002]:    2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5022]:    2010/11/22_11:33:55 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5011]:    2010/11/22_11:33:55 INFO:  SuccessResourceManager[4002]:    2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/drbddisk r0 stophb_standby[5113]:    2010/11/22_11:34:25 Going standby [foreign].heartbeat[3445]: 2010/11/22_11:34:26 info: server1.drbd wants to go standby [foreign]heartbeat[3445]: 2010/11/22_11:34:37 WARN: No reply to standby request.  Standby request cancelled.heartbeat[3445]: 2010/11/22_12:50:48 info: Heartbeat shutdown in progress. (3445)heartbeat[6977]: 2010/11/22_12:50:48 info: Giving up all HA resources.ResourceManager[6990]:    2010/11/22_12:50:48 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[6990]:    2010/11/22_12:50:48 info: Running /etc/init.d/tomcat6  stopResourceManager[6990]:    2010/11/22_12:50:48 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:48 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:49 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:49 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:49 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:50 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:50 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:50 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:51 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:51 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:51 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:52 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:52 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:53 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:54 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:54 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:54 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:55 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:55 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:55 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:56 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:56 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:56 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:57 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:57 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:57 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:58 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:58 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:58 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:59 info: Retrying failed stop operation [postgresql]ResourceManager[6990]:    2010/11/22_12:50:59 info: Running /etc/init.d/postgresql  stopResourceManager[6990]:    2010/11/22_12:50:59 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6990]:    2010/11/22_12:50:59 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[6990]:    2010/11/22_12:50:59 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[6990]:    2010/11/22_12:50:59 WARN: Machine reboot narrowly avoided!ResourceManager[6990]:    2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[7848]:    2010/11/22_12:50:59 INFO:  SuccessResourceManager[6990]:    2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[7931]:    2010/11/22_12:50:59 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[7920]:    2010/11/22_12:50:59 INFO:  SuccessResourceManager[6990]:    2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[6977]: 2010/11/22_12:50:59 info: All HA resources relinquished.heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBWRITE process 3454 with signal 15heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBREAD process 3455 with signal 15heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBFIFO process 3453 with signal 15heartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3453 exited. 3 remainingheartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3455 exited. 2 remainingheartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3454 exited. 1 remainingheartbeat[3445]: 2010/11/22_12:51:01 info: server1.drbd Heartbeat shutdown complete.heartbeat[8061]: 2010/11/22_12:51:42 info: Version 2 support: falseheartbeat[8061]: 2010/11/22_12:51:42 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[8061]: 2010/11/22_12:51:42 info: **************************heartbeat[8061]: 2010/11/22_12:51:42 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[8062]: 2010/11/22_12:51:42 info: heartbeat: version 2.1.3heartbeat[8062]: 2010/11/22_12:51:42 info: Heartbeat generation: 1290373963heartbeat[8062]: 2010/11/22_12:51:42 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[8062]: 2010/11/22_12:51:42 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[8062]: 2010/11/22_12:51:42 info: Local status now set to: 'up'heartbeat[8062]: 2010/11/22_12:51:43 info: Link server1.drbd:eth1 up.heartbeat[8062]: 2010/11/22_12:53:43 WARN: node server2.drbd: is deadheartbeat[8062]: 2010/11/22_12:53:43 info: Comm_now_up(): updating status to activeheartbeat[8062]: 2010/11/22_12:53:43 info: Local status now set to: 'active'heartbeat[8062]: 2010/11/22_12:53:43 WARN: No STONITH device configured.heartbeat[8062]: 2010/11/22_12:53:43 WARN: Shared disks are not protected.heartbeat[8062]: 2010/11/22_12:53:43 info: Resources being acquired from server2.drbd.harc[8071]:    2010/11/22_12:53:43 info: Running /etc/ha.d/rc.d/status statusmach_down[8103]:    2010/11/22_12:53:43 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[8072]: 2010/11/22_12:53:43 info: Local Resource acquisition completed.mach_down[8103]:    2010/11/22_12:53:43 info: mach_down takeover complete for node server2.drbd.heartbeat[8062]: 2010/11/22_12:53:43 info: mach_down takeover complete.heartbeat[8062]: 2010/11/22_12:53:43 info: Initial resource acquisition complete (mach_down)harc[8164]:    2010/11/22_12:53:43 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[8164]:    2010/11/22_12:53:43 received ip-request-resp drbddisk::r0 OK yesResourceManager[8185]:    2010/11/22_12:53:43 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[8185]:    2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[8256]:    2010/11/22_12:53:43 INFO:  Resource is stoppedResourceManager[8185]:    2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[8337]:    2010/11/22_12:53:43 INFO: Running start for /dev/drbd0 on /replicaFilesystem[8326]:    2010/11/22_12:53:43 INFO:  SuccessIPaddr[8405]:    2010/11/22_12:53:43 INFO:  Resource is stoppedResourceManager[8185]:    2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[8503]:    2010/11/22_12:53:43 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[8503]:    2010/11/22_12:53:43 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[8474]:    2010/11/22_12:53:43 INFO:  SuccessResourceManager[8185]:    2010/11/22_12:53:43 info: Running /etc/init.d/postgresql  startResourceManager[8185]:    2010/11/22_12:53:45 info: Running /etc/init.d/tomcat6  startheartbeat[8062]: 2010/11/22_12:53:53 info: Local Resource acquisition completed. (none)heartbeat[8062]: 2010/11/22_12:53:53 info: local resource transition completed.heartbeat[8062]: 2010/11/22_12:55:17 info: Heartbeat shutdown in progress. (8062)heartbeat[8886]: 2010/11/22_12:55:17 info: Giving up all HA resources.ResourceManager[8899]:    2010/11/22_12:55:17 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[8899]:    2010/11/22_12:55:17 info: Running /etc/init.d/tomcat6  stopResourceManager[8899]:    2010/11/22_12:55:20 info: Running /etc/init.d/postgresql  stopResourceManager[8899]:    2010/11/22_12:55:21 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[9169]:    2010/11/22_12:55:22 INFO: ifconfig eth0:0 downIPaddr[9140]:    2010/11/22_12:55:22 INFO:  SuccessResourceManager[8899]:    2010/11/22_12:55:22 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[9231]:    2010/11/22_12:55:22 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[9231]:    2010/11/22_12:55:22 INFO: Trying to unmount /replicaFilesystem[9231]:    2010/11/22_12:55:22 INFO: unmounted /replica successfullyFilesystem[9220]:    2010/11/22_12:55:22 INFO:  SuccessResourceManager[8899]:    2010/11/22_12:55:22 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[8886]: 2010/11/22_12:55:22 info: All HA resources relinquished.heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBFIFO process 8064 with signal 15heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBWRITE process 8065 with signal 15heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBREAD process 8066 with signal 15heartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8065 exited. 3 remainingheartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8066 exited. 2 remainingheartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8064 exited. 1 remainingheartbeat[8062]: 2010/11/22_12:55:24 info: server1.drbd Heartbeat shutdown complete.heartbeat[9378]: 2010/11/22_12:56:04 info: Version 2 support: falseheartbeat[9378]: 2010/11/22_12:56:04 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[9378]: 2010/11/22_12:56:04 info: **************************heartbeat[9378]: 2010/11/22_12:56:04 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[9379]: 2010/11/22_12:56:04 info: heartbeat: version 2.1.3heartbeat[9379]: 2010/11/22_12:56:04 info: Heartbeat generation: 1290373964heartbeat[9379]: 2010/11/22_12:56:04 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[9379]: 2010/11/22_12:56:04 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[9379]: 2010/11/22_12:56:04 info: Local status now set to: 'up'heartbeat[9379]: 2010/11/22_12:56:05 info: Link server1.drbd:eth1 up.heartbeat[9379]: 2010/11/22_12:58:04 WARN: node server2.drbd: is deadheartbeat[9379]: 2010/11/22_12:58:04 info: Comm_now_up(): updating status to activeheartbeat[9379]: 2010/11/22_12:58:04 info: Local status now set to: 'active'heartbeat[9379]: 2010/11/22_12:58:04 WARN: No STONITH device configured.heartbeat[9379]: 2010/11/22_12:58:04 WARN: Shared disks are not protected.heartbeat[9379]: 2010/11/22_12:58:04 info: Resources being acquired from server2.drbd.harc[9398]:    2010/11/22_12:58:04 info: Running /etc/ha.d/rc.d/status statusmach_down[9428]:    2010/11/22_12:58:04 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[9428]:    2010/11/22_12:58:04 info: mach_down takeover complete for node server2.drbd.heartbeat[9399]: 2010/11/22_12:58:04 info: Local Resource acquisition completed.heartbeat[9379]: 2010/11/22_12:58:04 info: mach_down takeover complete.heartbeat[9379]: 2010/11/22_12:58:04 info: Initial resource acquisition complete (mach_down)harc[9491]:    2010/11/22_12:58:04 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[9491]:    2010/11/22_12:58:04 received ip-request-resp drbddisk::r0 OK yesResourceManager[9512]:    2010/11/22_12:58:04 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[9512]:    2010/11/22_12:58:04 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[9583]:    2010/11/22_12:58:04 INFO:  Resource is stoppedResourceManager[9512]:    2010/11/22_12:58:04 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[9664]:    2010/11/22_12:58:05 INFO: Running start for /dev/drbd0 on /replicaFilesystem[9653]:    2010/11/22_12:58:05 INFO:  SuccessIPaddr[9732]:    2010/11/22_12:58:05 INFO:  Resource is stoppedResourceManager[9512]:    2010/11/22_12:58:05 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[9830]:    2010/11/22_12:58:05 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[9830]:    2010/11/22_12:58:05 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[9801]:    2010/11/22_12:58:05 INFO:  SuccessResourceManager[9512]:    2010/11/22_12:58:05 info: Running /etc/init.d/postgresql  startResourceManager[9512]:    2010/11/22_12:58:07 info: Running /etc/init.d/tomcat6  startheartbeat[9379]: 2010/11/22_12:58:15 info: Local Resource acquisition completed. (none)heartbeat[9379]: 2010/11/22_12:58:15 info: local resource transition completed.heartbeat[3523]: 2010/11/22_15:31:56 info: Version 2 support: falseheartbeat[3523]: 2010/11/22_15:31:56 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3523]: 2010/11/22_15:31:56 info: **************************heartbeat[3523]: 2010/11/22_15:31:56 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3524]: 2010/11/22_15:31:56 info: heartbeat: version 2.1.3heartbeat[3524]: 2010/11/22_15:31:56 info: Heartbeat generation: 1290373965heartbeat[3524]: 2010/11/22_15:31:56 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3524]: 2010/11/22_15:31:56 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3524]: 2010/11/22_15:31:56 info: Local status now set to: 'up'heartbeat[3524]: 2010/11/22_15:31:57 info: Link server1.drbd:eth1 up.heartbeat[3524]: 2010/11/22_15:33:56 WARN: node server2.drbd: is deadheartbeat[3524]: 2010/11/22_15:33:56 info: Comm_now_up(): updating status to activeheartbeat[3524]: 2010/11/22_15:33:56 info: Local status now set to: 'active'heartbeat[3524]: 2010/11/22_15:33:56 WARN: No STONITH device configured.heartbeat[3524]: 2010/11/22_15:33:56 WARN: Shared disks are not protected.heartbeat[3524]: 2010/11/22_15:33:56 info: Resources being acquired from server2.drbd.harc[4301]:    2010/11/22_15:33:56 info: Running /etc/ha.d/rc.d/status statusheartbeat[4302]: 2010/11/22_15:33:56 info: Local Resource acquisition completed.mach_down[4342]:    2010/11/22_15:33:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4342]:    2010/11/22_15:33:56 info: mach_down takeover complete for node server2.drbd.heartbeat[3524]: 2010/11/22_15:33:56 info: mach_down takeover complete.heartbeat[3524]: 2010/11/22_15:33:56 info: Initial resource acquisition complete (mach_down)harc[4394]:    2010/11/22_15:33:56 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4394]:    2010/11/22_15:33:56 received ip-request-resp drbddisk::r0 OK yesResourceManager[4415]:    2010/11/22_15:33:56 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[4415]:    2010/11/22_15:33:56 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4486]:    2010/11/22_15:33:56 INFO:  Resource is stoppedResourceManager[4415]:    2010/11/22_15:33:56 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4567]:    2010/11/22_15:33:57 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4556]:    2010/11/22_15:33:57 INFO:  SuccessIPaddr[4635]:    2010/11/22_15:33:57 INFO:  Resource is stoppedResourceManager[4415]:    2010/11/22_15:33:57 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4733]:    2010/11/22_15:33:57 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4733]:    2010/11/22_15:33:57 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4704]:    2010/11/22_15:33:57 INFO:  SuccessResourceManager[4415]:    2010/11/22_15:33:57 info: Running /etc/init.d/postgresql  startResourceManager[4415]:    2010/11/22_15:34:00 info: Running /etc/init.d/tomcat6  startheartbeat[3524]: 2010/11/22_15:34:07 info: Local Resource acquisition completed. (none)heartbeat[3524]: 2010/11/22_15:34:07 info: local resource transition completed.heartbeat[3218]: 2010/11/22_16:46:57 info: Version 2 support: falseheartbeat[3218]: 2010/11/22_16:46:57 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3218]: 2010/11/22_16:46:57 info: **************************heartbeat[3218]: 2010/11/22_16:46:57 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3219]: 2010/11/22_16:46:57 info: heartbeat: version 2.1.3heartbeat[3219]: 2010/11/22_16:46:57 info: Heartbeat generation: 1290373966heartbeat[3219]: 2010/11/22_16:46:57 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3219]: 2010/11/22_16:46:57 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3219]: 2010/11/22_16:46:57 info: Local status now set to: 'up'heartbeat[3219]: 2010/11/22_16:46:58 info: Link server1.drbd:eth1 up.heartbeat[3219]: 2010/11/22_16:48:57 WARN: node server2.drbd: is deadheartbeat[3219]: 2010/11/22_16:48:57 info: Comm_now_up(): updating status to activeheartbeat[3219]: 2010/11/22_16:48:57 info: Local status now set to: 'active'heartbeat[3219]: 2010/11/22_16:48:57 WARN: No STONITH device configured.heartbeat[3219]: 2010/11/22_16:48:57 WARN: Shared disks are not protected.heartbeat[3219]: 2010/11/22_16:48:57 info: Resources being acquired from server2.drbd.harc[3615]:    2010/11/22_16:48:57 info: Running /etc/ha.d/rc.d/status statusmach_down[3645]:    2010/11/22_16:48:57 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[3645]:    2010/11/22_16:48:57 info: mach_down takeover complete for node server2.drbd.heartbeat[3616]: 2010/11/22_16:48:57 info: Local Resource acquisition completed.heartbeat[3219]: 2010/11/22_16:48:57 info: mach_down takeover complete.heartbeat[3219]: 2010/11/22_16:48:57 info: Initial resource acquisition complete (mach_down)harc[3708]:    2010/11/22_16:48:57 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[3708]:    2010/11/22_16:48:57 received ip-request-resp drbddisk::r0 OK yesResourceManager[3729]:    2010/11/22_16:48:57 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[3729]:    2010/11/22_16:48:58 info: Running /etc/ha.d/resource.d/drbddisk r0 startheartbeat[3219]: 2010/11/22_16:49:08 info: Local Resource acquisition completed. (none)heartbeat[3219]: 2010/11/22_16:49:08 info: local resource transition completed.ResourceManager[3729]:    2010/11/22_16:49:10 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddiskResourceManager[3729]:    2010/11/22_16:49:10 CRIT: Giving up resources due to failure of drbddisk::r0ResourceManager[3729]:    2010/11/22_16:49:10 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[3729]:    2010/11/22_16:49:10 info: Running /etc/init.d/tomcat6  stopResourceManager[3729]:    2010/11/22_16:49:10 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:10 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:11 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:11 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:11 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:12 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:12 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:12 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:13 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:13 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:13 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:14 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:14 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:14 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:15 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:15 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:15 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:16 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:16 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:16 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:18 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:18 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:18 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:19 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:19 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:19 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:20 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:20 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:20 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:21 info: Retrying failed stop operation [postgresql]ResourceManager[3729]:    2010/11/22_16:49:21 info: Running /etc/init.d/postgresql  stopResourceManager[3729]:    2010/11/22_16:49:21 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[3729]:    2010/11/22_16:49:21 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[3729]:    2010/11/22_16:49:21 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[3729]:    2010/11/22_16:49:21 WARN: Machine reboot narrowly avoided!ResourceManager[3729]:    2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[4665]:    2010/11/22_16:49:21 INFO:  SuccessResourceManager[3729]:    2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[4748]:    2010/11/22_16:49:21 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[4737]:    2010/11/22_16:49:21 INFO:  SuccessResourceManager[3729]:    2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/drbddisk r0 stophb_standby[4840]:    2010/11/22_16:49:51 Going standby [foreign].heartbeat[3219]: 2010/11/22_16:49:51 info: server1.drbd wants to go standby [foreign]heartbeat[3219]: 2010/11/22_16:50:03 WARN: No reply to standby request.  Standby request cancelled.heartbeat[3219]: 2010/11/23_11:18:27 info: Heartbeat shutdown in progress. (3219)heartbeat[7475]: 2010/11/23_11:18:27 info: Giving up all HA resources.ResourceManager[7488]:    2010/11/23_11:18:27 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6ResourceManager[7488]:    2010/11/23_11:18:27 info: Running /etc/init.d/tomcat6  stopResourceManager[7488]:    2010/11/23_11:18:27 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:27 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:28 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:28 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:28 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:29 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:29 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:29 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:30 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:30 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:30 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:31 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:31 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:31 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:32 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:32 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:32 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:34 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:34 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:34 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:35 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:35 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:35 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:36 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:36 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:36 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:37 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:37 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:37 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:38 info: Retrying failed stop operation [postgresql]ResourceManager[7488]:    2010/11/23_11:18:38 info: Running /etc/init.d/postgresql  stopResourceManager[7488]:    2010/11/23_11:18:38 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7488]:    2010/11/23_11:18:38 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[7488]:    2010/11/23_11:18:38 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[7488]:    2010/11/23_11:18:38 WARN: Machine reboot narrowly avoided!ResourceManager[7488]:    2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[8346]:    2010/11/23_11:18:38 INFO:  SuccessResourceManager[7488]:    2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[8429]:    2010/11/23_11:18:38 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[8418]:    2010/11/23_11:18:38 INFO:  SuccessResourceManager[7488]:    2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/drbddisk r0 stopheartbeat[7475]: 2010/11/23_11:18:38 info: All HA resources relinquished.heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBFIFO process 3227 with signal 15heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBWRITE process 3228 with signal 15heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBREAD process 3229 with signal 15heartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3227 exited. 3 remainingheartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3229 exited. 2 remainingheartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3228 exited. 1 remainingheartbeat[3219]: 2010/11/23_11:18:40 info: server1.drbd Heartbeat shutdown complete.heartbeat[9421]: 2010/11/23_11:49:15 info: Version 2 support: falseheartbeat[9421]: 2010/11/23_11:49:15 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[9421]: 2010/11/23_11:49:15 info: **************************heartbeat[9421]: 2010/11/23_11:49:15 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[9422]: 2010/11/23_11:49:15 info: heartbeat: version 2.1.3heartbeat[9422]: 2010/11/23_11:49:15 info: Heartbeat generation: 1290373967heartbeat[9422]: 2010/11/23_11:49:15 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[9422]: 2010/11/23_11:49:15 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[9422]: 2010/11/23_11:49:15 info: Local status now set to: 'up'heartbeat[9422]: 2010/11/23_11:49:16 info: Link server1.drbd:eth1 up.heartbeat[9422]: 2010/11/23_11:51:15 WARN: node server2.drbd: is deadheartbeat[9422]: 2010/11/23_11:51:15 info: Comm_now_up(): updating status to activeheartbeat[9422]: 2010/11/23_11:51:15 info: Local status now set to: 'active'heartbeat[9422]: 2010/11/23_11:51:15 WARN: No STONITH device configured.heartbeat[9422]: 2010/11/23_11:51:15 WARN: Shared disks are not protected.heartbeat[9422]: 2010/11/23_11:51:15 info: Resources being acquired from server2.drbd.harc[9434]:    2010/11/23_11:51:15 info: Running /etc/ha.d/rc.d/status statusmach_down[9464]:    2010/11/23_11:51:15 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredheartbeat[9435]: 2010/11/23_11:51:15 info: Local Resource acquisition completed.heartbeat[9425]: 2010/11/23_11:51:15 WARN: ha_msg_add_nv_depth: line doesn't contain '='heartbeat[9425]: 2010/11/23_11:51:15 info: >>>heartbeat[9425]: 2010/11/23_11:51:15 ERROR: NV failure (msgfromsteam): [>>>]mach_down[9464]:    2010/11/23_11:51:15 info: mach_down takeover complete for node server2.drbd.heartbeat[9422]: 2010/11/23_12:08:13 WARN: Shutdown delayed until current resource activity finishes.heartbeat[3252]: 2010/11/23_12:47:22 info: Version 2 support: falseheartbeat[3252]: 2010/11/23_12:47:22 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3252]: 2010/11/23_12:47:22 info: **************************heartbeat[3252]: 2010/11/23_12:47:22 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3253]: 2010/11/23_12:47:22 info: heartbeat: version 2.1.3heartbeat[3253]: 2010/11/23_12:47:22 info: Heartbeat generation: 1290373968heartbeat[3253]: 2010/11/23_12:47:22 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3253]: 2010/11/23_12:47:22 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3253]: 2010/11/23_12:47:22 info: Local status now set to: 'up'heartbeat[3253]: 2010/11/23_12:47:23 info: Link server1.drbd:eth1 up.heartbeat[3253]: 2010/11/23_12:49:22 WARN: node server2.drbd: is deadheartbeat[3253]: 2010/11/23_12:49:22 info: Comm_now_up(): updating status to activeheartbeat[3253]: 2010/11/23_12:49:22 info: Local status now set to: 'active'heartbeat[3253]: 2010/11/23_12:49:22 WARN: No STONITH device configured.heartbeat[3253]: 2010/11/23_12:49:22 WARN: Shared disks are not protected.heartbeat[3253]: 2010/11/23_12:49:22 info: Resources being acquired from server2.drbd.harc[3625]:    2010/11/23_12:49:22 info: Running /etc/ha.d/rc.d/status statusmach_down[3655]:    2010/11/23_12:49:22 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[3655]:    2010/11/23_12:49:22 info: mach_down takeover complete for node server2.drbd.heartbeat[3253]: 2010/11/23_12:49:22 info: mach_down takeover complete.heartbeat[3253]: 2010/11/23_12:49:22 info: Initial resource acquisition complete (mach_down)IPaddr[3699]:    2010/11/23_12:49:22 INFO:  Resource is stoppedheartbeat[3626]: 2010/11/23_12:49:22 info: Local Resource acquisition completed.harc[3762]:    2010/11/23_12:49:22 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[3762]:    2010/11/23_12:49:23 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[3783]:    2010/11/23_12:49:23 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[3810]:    2010/11/23_12:49:23 INFO:  Resource is stoppedResourceManager[3783]:    2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[3908]:    2010/11/23_12:49:23 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[3908]:    2010/11/23_12:49:23 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[3879]:    2010/11/23_12:49:23 INFO:  SuccessResourceManager[3783]:    2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4056]:    2010/11/23_12:49:23 INFO:  Resource is stoppedResourceManager[3783]:    2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4137]:    2010/11/23_12:49:23 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4126]:    2010/11/23_12:49:23 INFO:  SuccessResourceManager[3783]:    2010/11/23_12:49:23 info: Running /etc/init.d/postgresql  startResourceManager[3783]:    2010/11/23_12:49:26 info: Running /etc/init.d/tomcat6  startheartbeat[3253]: 2010/11/23_12:49:33 info: Local Resource acquisition completed. (none)heartbeat[3253]: 2010/11/23_12:49:33 info: local resource transition completed.heartbeat[3253]: 2010/11/23_14:11:30 info: Heartbeat shutdown in progress. (3253)heartbeat[5002]: 2010/11/23_14:11:30 info: Giving up all HA resources.ResourceManager[5015]:    2010/11/23_14:11:30 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[5015]:    2010/11/23_14:11:30 info: Running /etc/init.d/tomcat6  stopResourceManager[5015]:    2010/11/23_14:11:34 info: Running /etc/init.d/postgresql  stopResourceManager[5015]:    2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5267]:    2010/11/23_14:11:35 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5267]:    2010/11/23_14:11:35 INFO: Trying to unmount /replicaFilesystem[5267]:    2010/11/23_14:11:35 INFO: unmounted /replica successfullyFilesystem[5256]:    2010/11/23_14:11:35 INFO:  SuccessResourceManager[5015]:    2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[5015]:    2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5415]:    2010/11/23_14:11:35 INFO: ifconfig eth0:0 downIPaddr[5386]:    2010/11/23_14:11:35 INFO:  Successheartbeat[5002]: 2010/11/23_14:11:35 info: All HA resources relinquished.heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBFIFO process 3261 with signal 15heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBWRITE process 3262 with signal 15heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBREAD process 3263 with signal 15heartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3262 exited. 3 remainingheartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3263 exited. 2 remainingheartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3261 exited. 1 remainingheartbeat[3253]: 2010/11/23_14:11:37 info: server1.drbd Heartbeat shutdown complete.heartbeat[3287]: 2010/11/23_14:14:49 info: Version 2 support: falseheartbeat[3287]: 2010/11/23_14:14:49 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3287]: 2010/11/23_14:14:49 info: **************************heartbeat[3287]: 2010/11/23_14:14:49 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3288]: 2010/11/23_14:14:49 info: heartbeat: version 2.1.3heartbeat[3288]: 2010/11/23_14:14:49 info: Heartbeat generation: 1290373969heartbeat[3288]: 2010/11/23_14:14:49 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3288]: 2010/11/23_14:14:49 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3288]: 2010/11/23_14:14:49 info: Local status now set to: 'up'heartbeat[3288]: 2010/11/23_14:14:50 info: Link server1.drbd:eth1 up.heartbeat[3288]: 2010/11/23_14:16:50 WARN: node server2.drbd: is deadheartbeat[3288]: 2010/11/23_14:16:50 info: Comm_now_up(): updating status to activeheartbeat[3288]: 2010/11/23_14:16:50 info: Local status now set to: 'active'heartbeat[3288]: 2010/11/23_14:16:50 WARN: No STONITH device configured.heartbeat[3288]: 2010/11/23_14:16:50 WARN: Shared disks are not protected.heartbeat[3288]: 2010/11/23_14:16:50 info: Resources being acquired from server2.drbd.harc[3784]:    2010/11/23_14:16:50 info: Running /etc/ha.d/rc.d/status statusmach_down[3814]:    2010/11/23_14:16:50 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[3814]:    2010/11/23_14:16:50 info: mach_down takeover complete for node server2.drbd.heartbeat[3288]: 2010/11/23_14:16:50 info: mach_down takeover complete.heartbeat[3288]: 2010/11/23_14:16:50 info: Initial resource acquisition complete (mach_down)IPaddr[3851]:    2010/11/23_14:16:50 INFO:  Resource is stoppedheartbeat[3785]: 2010/11/23_14:16:50 info: Local Resource acquisition completed.harc[3921]:    2010/11/23_14:16:50 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[3921]:    2010/11/23_14:16:50 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[3942]:    2010/11/23_14:16:50 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[3969]:    2010/11/23_14:16:50 INFO:  Resource is stoppedResourceManager[3942]:    2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4067]:    2010/11/23_14:16:50 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4067]:    2010/11/23_14:16:50 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4038]:    2010/11/23_14:16:50 INFO:  SuccessResourceManager[3942]:    2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4215]:    2010/11/23_14:16:50 INFO:  Resource is stoppedResourceManager[3942]:    2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4296]:    2010/11/23_14:16:51 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4285]:    2010/11/23_14:16:51 INFO:  SuccessResourceManager[3942]:    2010/11/23_14:16:51 info: Running /etc/init.d/postgresql  startResourceManager[3942]:    2010/11/23_14:16:53 info: Running /etc/init.d/tomcat6  startheartbeat[3288]: 2010/11/23_14:17:00 info: Local Resource acquisition completed. (none)heartbeat[3288]: 2010/11/23_14:17:00 info: local resource transition completed.heartbeat[3288]: 2010/11/23_14:17:11 info: Heartbeat shutdown in progress. (3288)heartbeat[4668]: 2010/11/23_14:17:11 info: Giving up all HA resources.ResourceManager[4681]:    2010/11/23_14:17:11 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[4681]:    2010/11/23_14:17:11 info: Running /etc/init.d/tomcat6  stopResourceManager[4681]:    2010/11/23_14:17:15 info: Running /etc/init.d/postgresql  stopResourceManager[4681]:    2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[4934]:    2010/11/23_14:17:16 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[4934]:    2010/11/23_14:17:16 INFO: Trying to unmount /replicaFilesystem[4934]:    2010/11/23_14:17:16 INFO: unmounted /replica successfullyFilesystem[4923]:    2010/11/23_14:17:16 INFO:  SuccessResourceManager[4681]:    2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[4681]:    2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5082]:    2010/11/23_14:17:16 INFO: ifconfig eth0:0 downIPaddr[5053]:    2010/11/23_14:17:16 INFO:  Successheartbeat[4668]: 2010/11/23_14:17:16 info: All HA resources relinquished.heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBFIFO process 3304 with signal 15heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBWRITE process 3305 with signal 15heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBREAD process 3306 with signal 15heartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3305 exited. 3 remainingheartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3304 exited. 2 remainingheartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3306 exited. 1 remainingheartbeat[3288]: 2010/11/23_14:17:18 info: server1.drbd Heartbeat shutdown complete.heartbeat[5368]: 2010/11/23_14:20:41 info: Version 2 support: falseheartbeat[5368]: 2010/11/23_14:20:41 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[5368]: 2010/11/23_14:20:41 info: **************************heartbeat[5368]: 2010/11/23_14:20:41 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[5369]: 2010/11/23_14:20:41 info: heartbeat: version 2.1.3heartbeat[5369]: 2010/11/23_14:20:41 info: Heartbeat generation: 1290373970heartbeat[5369]: 2010/11/23_14:20:41 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[5369]: 2010/11/23_14:20:41 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[5369]: 2010/11/23_14:20:41 info: Local status now set to: 'up'heartbeat[5369]: 2010/11/23_14:20:42 info: Link server1.drbd:eth1 up.heartbeat[5369]: 2010/11/23_14:22:41 WARN: node server2.drbd: is deadheartbeat[5369]: 2010/11/23_14:22:41 info: Comm_now_up(): updating status to activeheartbeat[5369]: 2010/11/23_14:22:41 info: Local status now set to: 'active'heartbeat[5369]: 2010/11/23_14:22:41 WARN: No STONITH device configured.heartbeat[5369]: 2010/11/23_14:22:41 WARN: Shared disks are not protected.heartbeat[5369]: 2010/11/23_14:22:41 info: Resources being acquired from server2.drbd.harc[5418]:    2010/11/23_14:22:41 info: Running /etc/ha.d/rc.d/status statusmach_down[5448]:    2010/11/23_14:22:41 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[5448]:    2010/11/23_14:22:41 info: mach_down takeover complete for node server2.drbd.heartbeat[5369]: 2010/11/23_14:22:41 info: mach_down takeover complete.heartbeat[5369]: 2010/11/23_14:22:41 info: Initial resource acquisition complete (mach_down)IPaddr[5493]:    2010/11/23_14:22:41 INFO:  Resource is stoppedheartbeat[5419]: 2010/11/23_14:22:41 info: Local Resource acquisition completed.harc[5555]:    2010/11/23_14:22:41 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[5555]:    2010/11/23_14:22:41 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[5576]:    2010/11/23_14:22:41 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[5603]:    2010/11/23_14:22:41 INFO:  Resource is stoppedResourceManager[5576]:    2010/11/23_14:22:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[5701]:    2010/11/23_14:22:42 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[5701]:    2010/11/23_14:22:42 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[5672]:    2010/11/23_14:22:42 INFO:  SuccessFilesystem[5820]:    2010/11/23_14:22:42 INFO:  Running OKResourceManager[5576]:    2010/11/23_14:22:42 info: Running /etc/init.d/postgresql  startResourceManager[5576]:    2010/11/23_14:22:44 info: Running /etc/init.d/tomcat6  startheartbeat[5369]: 2010/11/23_14:22:52 info: Local Resource acquisition completed. (none)heartbeat[5369]: 2010/11/23_14:22:52 info: local resource transition completed.heartbeat[5369]: 2010/11/23_14:29:40 info: Heartbeat shutdown in progress. (5369)heartbeat[8477]: 2010/11/23_14:29:40 info: Giving up all HA resources.ResourceManager[8490]:    2010/11/23_14:29:40 info: Releasing resource group: imalwaysdead.com IPaddr::10.253.252.251/30ResourceManager[8490]:    2010/11/23_14:29:40 info: Running /etc/ha.d/resource.d/IPaddr 10.253.252.251/30 stopIPaddr[8528]:    2010/11/23_14:29:40 INFO:  Successheartbeat[8477]: 2010/11/23_14:29:40 info: All HA resources relinquished.heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBFIFO process 5372 with signal 15heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBWRITE process 5373 with signal 15heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBREAD process 5374 with signal 15heartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5373 exited. 3 remainingheartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5374 exited. 2 remainingheartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5372 exited. 1 remainingheartbeat[5369]: 2010/11/23_14:29:42 info: server1.drbd Heartbeat shutdown complete.heartbeat[30652]: 2010/11/23_14:58:09 info: Version 2 support: falseheartbeat[30652]: 2010/11/23_14:58:09 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[30652]: 2010/11/23_14:58:09 info: **************************heartbeat[30652]: 2010/11/23_14:58:09 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[30653]: 2010/11/23_14:58:09 info: heartbeat: version 2.1.3heartbeat[30653]: 2010/11/23_14:58:09 info: Heartbeat generation: 1290373985heartbeat[30653]: 2010/11/23_14:58:09 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[30653]: 2010/11/23_14:58:09 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[30653]: 2010/11/23_14:58:09 info: Local status now set to: 'up'heartbeat[30653]: 2010/11/23_14:58:11 info: Link server1.drbd:eth1 up.heartbeat[30653]: 2010/11/23_15:00:10 WARN: node server2.drbd: is deadheartbeat[30653]: 2010/11/23_15:00:10 info: Comm_now_up(): updating status to activeheartbeat[30653]: 2010/11/23_15:00:10 info: Local status now set to: 'active'heartbeat[30653]: 2010/11/23_15:00:10 WARN: No STONITH device configured.heartbeat[30653]: 2010/11/23_15:00:10 WARN: Shared disks are not protected.heartbeat[30653]: 2010/11/23_15:00:10 info: Resources being acquired from server2.drbd.harc[30701]:    2010/11/23_15:00:10 info: Running /etc/ha.d/rc.d/status statusmach_down[30731]:    2010/11/23_15:00:10 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[30731]:    2010/11/23_15:00:10 info: mach_down takeover complete for node server2.drbd.heartbeat[30653]: 2010/11/23_15:00:10 info: mach_down takeover complete.heartbeat[30653]: 2010/11/23_15:00:10 info: Initial resource acquisition complete (mach_down)IPaddr[30775]:    2010/11/23_15:00:10 INFO:  Running OKheartbeat[30702]: 2010/11/23_15:00:10 info: Local Resource acquisition completed.heartbeat[30653]: 2010/11/23_15:00:20 info: Local Resource acquisition completed. (none)heartbeat[30653]: 2010/11/23_15:00:20 info: local resource transition completed.heartbeat[30653]: 2010/11/23_15:09:34 info: Heartbeat shutdown in progress. (30653)heartbeat[31004]: 2010/11/23_15:09:34 info: Giving up all HA resources.ResourceManager[31017]:    2010/11/23_15:09:35 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[31017]:    2010/11/23_15:09:35 info: Running /etc/init.d/tomcat6  stopResourceManager[31017]:    2010/11/23_15:09:38 info: Running /etc/init.d/postgresql  stopResourceManager[31017]:    2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[31269]:    2010/11/23_15:09:39 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[31269]:    2010/11/23_15:09:39 INFO: Trying to unmount /replicaFilesystem[31269]:    2010/11/23_15:09:39 INFO: unmounted /replica successfullyFilesystem[31258]:    2010/11/23_15:09:39 INFO:  SuccessResourceManager[31017]:    2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[31017]:    2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[31417]:    2010/11/23_15:09:39 INFO: ifconfig eth0:0 downIPaddr[31388]:    2010/11/23_15:09:39 INFO:  Successheartbeat[31004]: 2010/11/23_15:09:39 info: All HA resources relinquished.heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBWRITE process 30657 with signal 15heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBREAD process 30658 with signal 15heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBFIFO process 30656 with signal 15heartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30658 exited. 3 remainingheartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30656 exited. 2 remainingheartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30657 exited. 1 remainingheartbeat[30653]: 2010/11/23_15:09:41 info: server1.drbd Heartbeat shutdown complete.heartbeat[3461]: 2010/11/23_15:16:16 info: Version 2 support: falseheartbeat[3461]: 2010/11/23_15:16:16 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3461]: 2010/11/23_15:16:16 info: **************************heartbeat[3461]: 2010/11/23_15:16:16 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3462]: 2010/11/23_15:16:16 info: heartbeat: version 2.1.3heartbeat[3462]: 2010/11/23_15:16:16 info: Heartbeat generation: 1290373986heartbeat[3462]: 2010/11/23_15:16:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3462]: 2010/11/23_15:16:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3462]: 2010/11/23_15:16:16 info: Local status now set to: 'up'heartbeat[3462]: 2010/11/23_15:16:17 info: Link server1.drbd:eth1 up.heartbeat[3462]: 2010/11/23_15:18:17 WARN: node server2.drbd: is deadheartbeat[3462]: 2010/11/23_15:18:17 info: Comm_now_up(): updating status to activeheartbeat[3462]: 2010/11/23_15:18:17 info: Local status now set to: 'active'heartbeat[3462]: 2010/11/23_15:18:17 WARN: No STONITH device configured.heartbeat[3462]: 2010/11/23_15:18:17 WARN: Shared disks are not protected.heartbeat[3462]: 2010/11/23_15:18:17 info: Resources being acquired from server2.drbd.harc[4270]:    2010/11/23_15:18:17 info: Running /etc/ha.d/rc.d/status statusmach_down[4300]:    2010/11/23_15:18:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4300]:    2010/11/23_15:18:17 info: mach_down takeover complete for node server2.drbd.heartbeat[3462]: 2010/11/23_15:18:17 info: mach_down takeover complete.heartbeat[3462]: 2010/11/23_15:18:17 info: Initial resource acquisition complete (mach_down)IPaddr[4354]:    2010/11/23_15:18:17 INFO:  Resource is stoppedheartbeat[4271]: 2010/11/23_15:18:17 info: Local Resource acquisition completed.harc[4407]:    2010/11/23_15:18:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4407]:    2010/11/23_15:18:17 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[4428]:    2010/11/23_15:18:17 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[4455]:    2010/11/23_15:18:17 INFO:  Resource is stoppedResourceManager[4428]:    2010/11/23_15:18:17 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4553]:    2010/11/23_15:18:17 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4553]:    2010/11/23_15:18:17 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4524]:    2010/11/23_15:18:17 INFO:  SuccessResourceManager[4428]:    2010/11/23_15:18:17 info: Running /etc/ha.d/resource.d/drbddisk r0 startheartbeat[3462]: 2010/11/23_15:18:27 info: Local Resource acquisition completed. (none)heartbeat[3462]: 2010/11/23_15:18:27 info: local resource transition completed.ResourceManager[4428]:    2010/11/23_15:18:29 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddiskResourceManager[4428]:    2010/11/23_15:18:29 CRIT: Giving up resources due to failure of drbddisk::r0ResourceManager[4428]:    2010/11/23_15:18:29 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[4428]:    2010/11/23_15:18:29 info: Running /etc/init.d/tomcat6  stopResourceManager[4428]:    2010/11/23_15:18:29 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:30 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:31 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:31 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:31 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:32 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:32 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:32 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:33 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:33 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:33 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:34 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:34 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:34 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:35 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:35 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:35 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:36 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:36 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:36 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:37 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:37 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:37 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:38 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:38 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:38 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:39 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:39 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:39 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:40 info: Retrying failed stop operation [postgresql]ResourceManager[4428]:    2010/11/23_15:18:40 info: Running /etc/init.d/postgresql  stopResourceManager[4428]:    2010/11/23_15:18:40 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4428]:    2010/11/23_15:18:40 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[4428]:    2010/11/23_15:18:40 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[4428]:    2010/11/23_15:18:40 WARN: Machine reboot narrowly avoided!ResourceManager[4428]:    2010/11/23_15:18:40 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5578]:    2010/11/23_15:18:40 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5567]:    2010/11/23_15:18:41 INFO:  SuccessResourceManager[4428]:    2010/11/23_15:18:41 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[4428]:    2010/11/23_15:18:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5707]:    2010/11/23_15:18:41 INFO: ifconfig eth0:0 downIPaddr[5678]:    2010/11/23_15:18:41 INFO:  Successhb_standby[5748]:    2010/11/23_15:19:11 Going standby [foreign].heartbeat[3462]: 2010/11/23_15:19:11 info: server1.drbd wants to go standby [foreign]heartbeat[3462]: 2010/11/23_15:19:22 WARN: No reply to standby request.  Standby request cancelled.heartbeat[3462]: 2010/11/23_15:44:20 info: Heartbeat shutdown in progress. (3462)heartbeat[7067]: 2010/11/23_15:44:20 info: Giving up all HA resources.ResourceManager[7080]:    2010/11/23_15:44:20 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[7080]:    2010/11/23_15:44:20 info: Running /etc/init.d/tomcat6  stopResourceManager[7080]:    2010/11/23_15:44:20 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:20 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:21 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:21 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:21 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:22 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:22 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:22 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:24 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:24 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:24 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:25 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:25 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:25 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:26 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:26 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:26 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:27 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:27 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:27 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:28 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:28 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:28 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:29 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:29 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:29 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:30 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:30 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:30 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:31 info: Retrying failed stop operation [postgresql]ResourceManager[7080]:    2010/11/23_15:44:31 info: Running /etc/init.d/postgresql  stopResourceManager[7080]:    2010/11/23_15:44:31 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[7080]:    2010/11/23_15:44:31 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[7080]:    2010/11/23_15:44:31 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[7080]:    2010/11/23_15:44:31 WARN: Machine reboot narrowly avoided!ResourceManager[7080]:    2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[7949]:    2010/11/23_15:44:31 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[7938]:    2010/11/23_15:44:31 INFO:  SuccessResourceManager[7080]:    2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[7080]:    2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[8049]:    2010/11/23_15:44:31 INFO:  Successheartbeat[7067]: 2010/11/23_15:44:31 info: All HA resources relinquished.heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBFIFO process 3476 with signal 15heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBWRITE process 3477 with signal 15heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBREAD process 3479 with signal 15heartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3477 exited. 3 remainingheartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3479 exited. 2 remainingheartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3476 exited. 1 remainingheartbeat[3462]: 2010/11/23_15:44:33 info: server1.drbd Heartbeat shutdown complete.heartbeat[3464]: 2010/11/23_15:48:34 info: Version 2 support: falseheartbeat[3464]: 2010/11/23_15:48:34 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3464]: 2010/11/23_15:48:34 info: **************************heartbeat[3464]: 2010/11/23_15:48:34 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3465]: 2010/11/23_15:48:34 info: heartbeat: version 2.1.3heartbeat[3465]: 2010/11/23_15:48:34 info: Heartbeat generation: 1290373987heartbeat[3465]: 2010/11/23_15:48:34 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3465]: 2010/11/23_15:48:34 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3465]: 2010/11/23_15:48:34 info: Local status now set to: 'up'heartbeat[3465]: 2010/11/23_15:48:35 info: Link server1.drbd:eth1 up.heartbeat[3465]: 2010/11/23_15:50:35 WARN: node server2.drbd: is deadheartbeat[3465]: 2010/11/23_15:50:35 info: Comm_now_up(): updating status to activeheartbeat[3465]: 2010/11/23_15:50:35 info: Local status now set to: 'active'heartbeat[3465]: 2010/11/23_15:50:35 WARN: No STONITH device configured.heartbeat[3465]: 2010/11/23_15:50:35 WARN: Shared disks are not protected.heartbeat[3465]: 2010/11/23_15:50:35 info: Resources being acquired from server2.drbd.harc[4316]:    2010/11/23_15:50:35 info: Running /etc/ha.d/rc.d/status statusmach_down[4346]:    2010/11/23_15:50:35 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4346]:    2010/11/23_15:50:35 info: mach_down takeover complete for node server2.drbd.heartbeat[3465]: 2010/11/23_15:50:35 info: mach_down takeover complete.heartbeat[3465]: 2010/11/23_15:50:35 info: Initial resource acquisition complete (mach_down)IPaddr[4399]:    2010/11/23_15:50:35 INFO:  Resource is stoppedheartbeat[4317]: 2010/11/23_15:50:35 info: Local Resource acquisition completed.harc[4453]:    2010/11/23_15:50:35 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4453]:    2010/11/23_15:50:35 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[4474]:    2010/11/23_15:50:35 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[4501]:    2010/11/23_15:50:35 INFO:  Resource is stoppedResourceManager[4474]:    2010/11/23_15:50:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4599]:    2010/11/23_15:50:35 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4599]:    2010/11/23_15:50:35 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4570]:    2010/11/23_15:50:35 INFO:  SuccessResourceManager[4474]:    2010/11/23_15:50:35 info: Running /etc/ha.d/resource.d/drbddisk r0 startheartbeat[3465]: 2010/11/23_15:50:45 info: Local Resource acquisition completed. (none)heartbeat[3465]: 2010/11/23_15:50:45 info: local resource transition completed.ResourceManager[4474]:    2010/11/23_15:50:47 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddiskResourceManager[4474]:    2010/11/23_15:50:47 CRIT: Giving up resources due to failure of drbddisk::r0ResourceManager[4474]:    2010/11/23_15:50:47 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[4474]:    2010/11/23_15:50:47 info: Running /etc/init.d/tomcat6  stopResourceManager[4474]:    2010/11/23_15:50:47 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:48 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:49 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:49 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:49 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:50 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:50 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:50 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:51 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:51 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:51 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:52 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:52 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:52 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:53 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:53 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:53 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:54 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:54 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:54 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:55 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:55 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:55 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:56 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:56 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:56 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:57 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:57 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:57 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:58 info: Retrying failed stop operation [postgresql]ResourceManager[4474]:    2010/11/23_15:50:58 info: Running /etc/init.d/postgresql  stopResourceManager[4474]:    2010/11/23_15:50:58 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4474]:    2010/11/23_15:50:58 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[4474]:    2010/11/23_15:50:58 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[4474]:    2010/11/23_15:50:58 WARN: Machine reboot narrowly avoided!ResourceManager[4474]:    2010/11/23_15:50:58 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5688]:    2010/11/23_15:50:59 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5677]:    2010/11/23_15:50:59 INFO:  SuccessResourceManager[4474]:    2010/11/23_15:50:59 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[4474]:    2010/11/23_15:50:59 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5817]:    2010/11/23_15:50:59 INFO: ifconfig eth0:0 downIPaddr[5788]:    2010/11/23_15:50:59 INFO:  Successhb_standby[5859]:    2010/11/23_15:51:29 Going standby [foreign].heartbeat[3465]: 2010/11/23_15:51:29 info: server1.drbd wants to go standby [foreign]heartbeat[3465]: 2010/11/23_15:51:40 WARN: No reply to standby request.  Standby request cancelled.heartbeat[3465]: 2010/11/23_15:58:22 info: Heartbeat shutdown in progress. (3465)heartbeat[6468]: 2010/11/23_15:58:22 info: Giving up all HA resources.ResourceManager[6481]:    2010/11/23_15:58:22 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[6481]:    2010/11/23_15:58:22 info: Running /etc/init.d/tomcat6  stopResourceManager[6481]:    2010/11/23_15:58:22 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:22 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:23 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:23 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:23 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:24 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:24 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:24 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:25 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:25 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:25 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:26 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:26 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:26 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:27 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:27 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:27 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:28 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:28 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:28 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:29 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:29 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:30 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:31 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:31 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:31 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:32 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:32 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:32 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:33 info: Retrying failed stop operation [postgresql]ResourceManager[6481]:    2010/11/23_15:58:33 info: Running /etc/init.d/postgresql  stopResourceManager[6481]:    2010/11/23_15:58:33 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[6481]:    2010/11/23_15:58:33 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[6481]:    2010/11/23_15:58:33 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[6481]:    2010/11/23_15:58:33 WARN: Machine reboot narrowly avoided!ResourceManager[6481]:    2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[7351]:    2010/11/23_15:58:33 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[7340]:    2010/11/23_15:58:33 INFO:  SuccessResourceManager[6481]:    2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[6481]:    2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[7451]:    2010/11/23_15:58:33 INFO:  Successheartbeat[6468]: 2010/11/23_15:58:33 info: All HA resources relinquished.heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBFIFO process 3490 with signal 15heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBWRITE process 3492 with signal 15heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBREAD process 3493 with signal 15heartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3493 exited. 3 remainingheartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3490 exited. 2 remainingheartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3492 exited. 1 remainingheartbeat[3465]: 2010/11/23_15:58:35 info: server1.drbd Heartbeat shutdown complete.heartbeat[7643]: 2010/11/23_16:02:40 info: Version 2 support: falseheartbeat[7643]: 2010/11/23_16:02:40 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[7643]: 2010/11/23_16:02:40 info: **************************heartbeat[7643]: 2010/11/23_16:02:40 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[7644]: 2010/11/23_16:02:40 info: heartbeat: version 2.1.3heartbeat[7644]: 2010/11/23_16:02:40 info: Heartbeat generation: 1290373988heartbeat[7644]: 2010/11/23_16:02:40 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[7644]: 2010/11/23_16:02:40 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[7644]: 2010/11/23_16:02:40 info: Local status now set to: 'up'heartbeat[7644]: 2010/11/23_16:02:41 info: Link server1.drbd:eth1 up.heartbeat[7644]: 2010/11/23_16:04:40 WARN: node server2.drbd: is deadheartbeat[7644]: 2010/11/23_16:04:40 info: Comm_now_up(): updating status to activeheartbeat[7644]: 2010/11/23_16:04:40 info: Local status now set to: 'active'heartbeat[7644]: 2010/11/23_16:04:40 WARN: No STONITH device configured.heartbeat[7644]: 2010/11/23_16:04:40 WARN: Shared disks are not protected.heartbeat[7644]: 2010/11/23_16:04:40 info: Resources being acquired from server2.drbd.harc[7691]:    2010/11/23_16:04:40 info: Running /etc/ha.d/rc.d/status statusmach_down[7721]:    2010/11/23_16:04:40 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[7721]:    2010/11/23_16:04:40 info: mach_down takeover complete for node server2.drbd.heartbeat[7644]: 2010/11/23_16:04:40 info: mach_down takeover complete.heartbeat[7644]: 2010/11/23_16:04:40 info: Initial resource acquisition complete (mach_down)IPaddr[7763]:    2010/11/23_16:04:40 INFO:  Resource is stoppedheartbeat[7692]: 2010/11/23_16:04:40 info: Local Resource acquisition completed.harc[7828]:    2010/11/23_16:04:40 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[7828]:    2010/11/23_16:04:40 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[7849]:    2010/11/23_16:04:40 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[7876]:    2010/11/23_16:04:41 INFO:  Resource is stoppedResourceManager[7849]:    2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[7974]:    2010/11/23_16:04:41 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[7974]:    2010/11/23_16:04:41 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[7945]:    2010/11/23_16:04:41 INFO:  SuccessResourceManager[7849]:    2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[8122]:    2010/11/23_16:04:41 INFO:  Resource is stoppedResourceManager[7849]:    2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[8203]:    2010/11/23_16:04:41 INFO: Running start for /dev/drbd0 on /replicaFilesystem[8192]:    2010/11/23_16:04:41 INFO:  SuccessResourceManager[7849]:    2010/11/23_16:04:41 info: Running /etc/init.d/postgresql  startResourceManager[7849]:    2010/11/23_16:04:43 info: Running /etc/init.d/tomcat6  startheartbeat[7644]: 2010/11/23_16:04:51 info: Local Resource acquisition completed. (none)heartbeat[7644]: 2010/11/23_16:04:51 info: local resource transition completed.heartbeat[7644]: 2010/11/23_16:14:15 info: Heartbeat shutdown in progress. (7644)heartbeat[8872]: 2010/11/23_16:14:15 info: Giving up all HA resources.ResourceManager[8885]:    2010/11/23_16:14:15 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[8885]:    2010/11/23_16:14:15 info: Running /etc/init.d/tomcat6  stopResourceManager[8885]:    2010/11/23_16:14:18 info: Running /etc/init.d/postgresql  stopResourceManager[8885]:    2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[9138]:    2010/11/23_16:14:20 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[9138]:    2010/11/23_16:14:20 INFO: Trying to unmount /replicaFilesystem[9138]:    2010/11/23_16:14:20 INFO: unmounted /replica successfullyFilesystem[9127]:    2010/11/23_16:14:20 INFO:  SuccessResourceManager[8885]:    2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[8885]:    2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[9286]:    2010/11/23_16:14:20 INFO: ifconfig eth0:0 downIPaddr[9257]:    2010/11/23_16:14:20 INFO:  Successheartbeat[8872]: 2010/11/23_16:14:20 info: All HA resources relinquished.heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBFIFO process 7647 with signal 15heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBWRITE process 7648 with signal 15heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBREAD process 7649 with signal 15heartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7649 exited. 3 remainingheartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7648 exited. 2 remainingheartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7647 exited. 1 remainingheartbeat[7644]: 2010/11/23_16:14:22 info: server1.drbd Heartbeat shutdown complete.heartbeat[3345]: 2010/11/23_16:36:04 info: Version 2 support: falseheartbeat[3345]: 2010/11/23_16:36:04 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3345]: 2010/11/23_16:36:04 info: **************************heartbeat[3345]: 2010/11/23_16:36:04 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3346]: 2010/11/23_16:36:04 info: heartbeat: version 2.1.3heartbeat[3346]: 2010/11/23_16:36:04 info: Heartbeat generation: 1290373989heartbeat[3346]: 2010/11/23_16:36:04 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3346]: 2010/11/23_16:36:04 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3346]: 2010/11/23_16:36:04 info: Local status now set to: 'up'heartbeat[3346]: 2010/11/23_16:36:05 info: Link server1.drbd:eth1 up.heartbeat[3346]: 2010/11/23_16:38:05 WARN: node server2.drbd: is deadheartbeat[3346]: 2010/11/23_16:38:05 info: Comm_now_up(): updating status to activeheartbeat[3346]: 2010/11/23_16:38:05 info: Local status now set to: 'active'heartbeat[3346]: 2010/11/23_16:38:05 WARN: No STONITH device configured.heartbeat[3346]: 2010/11/23_16:38:05 WARN: Shared disks are not protected.heartbeat[3346]: 2010/11/23_16:38:05 info: Resources being acquired from server2.drbd.harc[3903]:    2010/11/23_16:38:05 info: Running /etc/ha.d/rc.d/status statusmach_down[3933]:    2010/11/23_16:38:05 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[3933]:    2010/11/23_16:38:05 info: mach_down takeover complete for node server2.drbd.heartbeat[3346]: 2010/11/23_16:38:05 info: mach_down takeover complete.heartbeat[3346]: 2010/11/23_16:38:05 info: Initial resource acquisition complete (mach_down)IPaddr[3989]:    2010/11/23_16:38:05 INFO:  Resource is stoppedheartbeat[3904]: 2010/11/23_16:38:05 info: Local Resource acquisition completed.harc[4040]:    2010/11/23_16:38:05 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4040]:    2010/11/23_16:38:05 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[4061]:    2010/11/23_16:38:05 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[4088]:    2010/11/23_16:38:05 INFO:  Resource is stoppedResourceManager[4061]:    2010/11/23_16:38:05 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4186]:    2010/11/23_16:38:05 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4186]:    2010/11/23_16:38:05 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4157]:    2010/11/23_16:38:05 INFO:  SuccessResourceManager[4061]:    2010/11/23_16:38:06 info: Running /etc/ha.d/resource.d/drbddisk r0 startheartbeat[3346]: 2010/11/23_16:38:16 info: Local Resource acquisition completed. (none)heartbeat[3346]: 2010/11/23_16:38:16 info: local resource transition completed.ResourceManager[4061]:    2010/11/23_16:38:18 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddiskResourceManager[4061]:    2010/11/23_16:38:18 CRIT: Giving up resources due to failure of drbddisk::r0ResourceManager[4061]:    2010/11/23_16:38:18 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[4061]:    2010/11/23_16:38:18 info: Running /etc/init.d/tomcat6  stopResourceManager[4061]:    2010/11/23_16:38:18 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:18 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:19 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:19 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:19 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:20 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:20 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:20 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:21 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:21 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:21 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:22 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:22 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:22 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:23 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:23 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:23 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:24 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:24 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:25 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:26 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:26 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:26 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:27 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:27 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:27 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:28 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:28 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:28 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:29 info: Retrying failed stop operation [postgresql]ResourceManager[4061]:    2010/11/23_16:38:29 info: Running /etc/init.d/postgresql  stopResourceManager[4061]:    2010/11/23_16:38:29 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[4061]:    2010/11/23_16:38:29 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[4061]:    2010/11/23_16:38:29 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[4061]:    2010/11/23_16:38:29 WARN: Machine reboot narrowly avoided!ResourceManager[4061]:    2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[5211]:    2010/11/23_16:38:29 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[5200]:    2010/11/23_16:38:29 INFO:  SuccessResourceManager[4061]:    2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[4061]:    2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[5340]:    2010/11/23_16:38:29 INFO: ifconfig eth0:0 downIPaddr[5311]:    2010/11/23_16:38:29 INFO:  Successhb_standby[5382]:    2010/11/23_16:38:59 Going standby [foreign].heartbeat[3346]: 2010/11/23_16:39:00 info: server1.drbd wants to go standby [foreign]heartbeat[3346]: 2010/11/23_16:39:11 WARN: No reply to standby request.  Standby request cancelled.heartbeat[3346]: 2010/11/23_16:57:31 info: Heartbeat shutdown in progress. (3346)heartbeat[5591]: 2010/11/23_16:57:31 info: Giving up all HA resources.ResourceManager[5604]:    2010/11/23_16:57:31 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[5604]:    2010/11/23_16:57:31 info: Running /etc/init.d/tomcat6  stopResourceManager[5604]:    2010/11/23_16:57:31 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:31 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:33 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:33 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:33 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:34 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:34 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:34 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:35 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:35 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:35 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:36 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:36 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:36 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:37 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:37 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:37 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:38 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:38 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:38 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:39 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:39 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:39 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:40 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:40 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:40 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:41 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:41 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:41 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:42 info: Retrying failed stop operation [postgresql]ResourceManager[5604]:    2010/11/23_16:57:42 info: Running /etc/init.d/postgresql  stopResourceManager[5604]:    2010/11/23_16:57:42 ERROR: Return code 1 from /etc/init.d/postgresqlResourceManager[5604]:    2010/11/23_16:57:42 ERROR: Resource script for postgresql probably not LSB-compliant.ResourceManager[5604]:    2010/11/23_16:57:42 WARN: it (postgresql) MUST succeed on a stop when already stoppedResourceManager[5604]:    2010/11/23_16:57:42 WARN: Machine reboot narrowly avoided!ResourceManager[5604]:    2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[6473]:    2010/11/23_16:57:42 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[6462]:    2010/11/23_16:57:42 INFO:  SuccessResourceManager[5604]:    2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[5604]:    2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[6573]:    2010/11/23_16:57:42 INFO:  Successheartbeat[5591]: 2010/11/23_16:57:43 info: All HA resources relinquished.heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBFIFO process 3362 with signal 15heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBWRITE process 3363 with signal 15heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBREAD process 3364 with signal 15heartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3364 exited. 3 remainingheartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3363 exited. 2 remainingheartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3362 exited. 1 remainingheartbeat[3346]: 2010/11/23_16:57:45 info: server1.drbd Heartbeat shutdown complete.heartbeat[6833]: 2010/11/23_17:00:05 info: Version 2 support: falseheartbeat[6833]: 2010/11/23_17:00:05 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[6833]: 2010/11/23_17:00:05 info: **************************heartbeat[6833]: 2010/11/23_17:00:05 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[6834]: 2010/11/23_17:00:05 info: heartbeat: version 2.1.3heartbeat[6834]: 2010/11/23_17:00:05 info: Heartbeat generation: 1290373990heartbeat[6834]: 2010/11/23_17:00:05 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[6834]: 2010/11/23_17:00:05 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[6834]: 2010/11/23_17:00:05 info: Local status now set to: 'up'heartbeat[6834]: 2010/11/23_17:00:06 info: Link server1.drbd:eth1 up.heartbeat[6834]: 2010/11/23_17:00:56 WARN: node server2.drbd: is deadheartbeat[6834]: 2010/11/23_17:00:56 info: Comm_now_up(): updating status to activeheartbeat[6834]: 2010/11/23_17:00:56 info: Local status now set to: 'active'heartbeat[6834]: 2010/11/23_17:00:56 WARN: No STONITH device configured.heartbeat[6834]: 2010/11/23_17:00:56 WARN: Shared disks are not protected.heartbeat[6834]: 2010/11/23_17:00:56 info: Resources being acquired from server2.drbd.harc[6878]:    2010/11/23_17:00:56 info: Running /etc/ha.d/rc.d/status statusmach_down[6908]:    2010/11/23_17:00:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[6908]:    2010/11/23_17:00:56 info: mach_down takeover complete for node server2.drbd.heartbeat[6834]: 2010/11/23_17:00:56 info: mach_down takeover complete.heartbeat[6834]: 2010/11/23_17:00:56 info: Initial resource acquisition complete (mach_down)IPaddr[6952]:    2010/11/23_17:00:56 INFO:  Resource is stoppedheartbeat[6879]: 2010/11/23_17:00:56 info: Local Resource acquisition completed.harc[7015]:    2010/11/23_17:00:56 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[7015]:    2010/11/23_17:00:56 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[7036]:    2010/11/23_17:00:56 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[7063]:    2010/11/23_17:00:56 INFO:  Resource is stoppedResourceManager[7036]:    2010/11/23_17:00:56 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[7161]:    2010/11/23_17:00:56 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[7161]:    2010/11/23_17:00:56 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[7132]:    2010/11/23_17:00:56 INFO:  SuccessFilesystem[7280]:    2010/11/23_17:00:56 INFO:  Running OKResourceManager[7036]:    2010/11/23_17:00:56 info: Running /etc/init.d/postgresql  startResourceManager[7036]:    2010/11/23_17:00:58 info: Running /etc/init.d/tomcat6  startheartbeat[6834]: 2010/11/23_17:01:06 info: Local Resource acquisition completed. (none)heartbeat[6834]: 2010/11/23_17:01:06 info: local resource transition completed.heartbeat[6838]: 2010/11/23_17:01:15 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:15 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:17 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:17 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:19 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:19 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:21 ERROR: glib: Unable to send bcast [-1] packet(len=217): No such deviceheartbeat[6838]: 2010/11/23_17:01:21 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:21 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:21 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:23 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:23 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:25 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:25 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:27 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:27 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:29 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:29 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:31 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:31 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:33 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such deviceheartbeat[6838]: 2010/11/23_17:01:33 ERROR: write_child: write failure on bcast eth1.: No such deviceheartbeat[6838]: 2010/11/23_17:01:33 WARN: Temporarily Suppressing write error messagesheartbeat[6838]: 2010/11/23_17:01:33 WARN: Is a cable unplugged on bcast eth1?heartbeat[3300]: 2010/11/23_17:17:49 info: Version 2 support: falseheartbeat[3300]: 2010/11/23_17:17:49 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[3300]: 2010/11/23_17:17:49 info: **************************heartbeat[3300]: 2010/11/23_17:17:49 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[3301]: 2010/11/23_17:17:49 info: heartbeat: version 2.1.3heartbeat[3301]: 2010/11/23_17:17:49 info: Heartbeat generation: 1290373991heartbeat[3301]: 2010/11/23_17:17:49 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[3301]: 2010/11/23_17:17:49 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[3301]: 2010/11/23_17:17:49 info: Local status now set to: 'up'heartbeat[3301]: 2010/11/23_17:17:50 info: Link server1.drbd:eth1 up.heartbeat[3301]: 2010/11/23_17:18:40 WARN: node server2.drbd: is deadheartbeat[3301]: 2010/11/23_17:18:40 info: Comm_now_up(): updating status to activeheartbeat[3301]: 2010/11/23_17:18:40 info: Local status now set to: 'active'heartbeat[3301]: 2010/11/23_17:18:40 WARN: No STONITH device configured.heartbeat[3301]: 2010/11/23_17:18:40 WARN: Shared disks are not protected.heartbeat[3301]: 2010/11/23_17:18:40 info: Resources being acquired from server2.drbd.harc[3980]:    2010/11/23_17:18:40 info: Running /etc/ha.d/rc.d/status statusmach_down[4010]:    2010/11/23_17:18:40 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[4010]:    2010/11/23_17:18:40 info: mach_down takeover complete for node server2.drbd.heartbeat[3301]: 2010/11/23_17:18:40 info: mach_down takeover complete.heartbeat[3301]: 2010/11/23_17:18:40 info: Initial resource acquisition complete (mach_down)IPaddr[4056]:    2010/11/23_17:18:40 INFO:  Resource is stoppedheartbeat[3981]: 2010/11/23_17:18:40 info: Local Resource acquisition completed.harc[4117]:    2010/11/23_17:18:40 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[4117]:    2010/11/23_17:18:40 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[4138]:    2010/11/23_17:18:40 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[4165]:    2010/11/23_17:18:40 INFO:  Resource is stoppedResourceManager[4138]:    2010/11/23_17:18:40 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[4263]:    2010/11/23_17:18:40 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[4263]:    2010/11/23_17:18:40 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[4234]:    2010/11/23_17:18:40 INFO:  SuccessResourceManager[4138]:    2010/11/23_17:18:41 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[4411]:    2010/11/23_17:18:41 INFO:  Resource is stoppedResourceManager[4138]:    2010/11/23_17:18:41 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[4492]:    2010/11/23_17:18:41 INFO: Running start for /dev/drbd0 on /replicaFilesystem[4481]:    2010/11/23_17:18:41 INFO:  SuccessResourceManager[4138]:    2010/11/23_17:18:41 info: Running /etc/init.d/postgresql  startResourceManager[4138]:    2010/11/23_17:18:43 info: Running /etc/init.d/tomcat6  startheartbeat[3301]: 2010/11/23_17:18:51 info: Local Resource acquisition completed. (none)heartbeat[3301]: 2010/11/23_17:18:51 info: local resource transition completed.heartbeat[3301]: 2010/11/23_17:22:01 info: Heartbeat shutdown in progress. (3301)heartbeat[6003]: 2010/11/23_17:22:01 info: Giving up all HA resources.ResourceManager[6016]:    2010/11/23_17:22:01 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[6016]:    2010/11/23_17:22:01 info: Running /etc/init.d/tomcat6  stopResourceManager[6016]:    2010/11/23_17:22:05 info: Running /etc/init.d/postgresql  stopResourceManager[6016]:    2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[6268]:    2010/11/23_17:22:06 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[6268]:    2010/11/23_17:22:06 INFO: Trying to unmount /replicaFilesystem[6268]:    2010/11/23_17:22:06 INFO: unmounted /replica successfullyFilesystem[6257]:    2010/11/23_17:22:06 INFO:  SuccessResourceManager[6016]:    2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[6016]:    2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[6416]:    2010/11/23_17:22:06 INFO: ifconfig eth0:0 downIPaddr[6387]:    2010/11/23_17:22:06 INFO:  Successheartbeat[6003]: 2010/11/23_17:22:06 info: All HA resources relinquished.heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBFIFO process 3307 with signal 15heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBWRITE process 3308 with signal 15heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBREAD process 3309 with signal 15heartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3308 exited. 3 remainingheartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3307 exited. 2 remainingheartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3309 exited. 1 remainingheartbeat[3301]: 2010/11/23_17:22:08 info: server1.drbd Heartbeat shutdown complete.heartbeat[6810]: 2010/11/23_17:30:44 info: Version 2 support: falseheartbeat[6810]: 2010/11/23_17:30:44 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[6810]: 2010/11/23_17:30:44 info: **************************heartbeat[6810]: 2010/11/23_17:30:44 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[6811]: 2010/11/23_17:30:44 info: heartbeat: version 2.1.3heartbeat[6811]: 2010/11/23_17:30:44 info: Heartbeat generation: 1290373992heartbeat[6811]: 2010/11/23_17:30:44 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[6811]: 2010/11/23_17:30:44 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[6811]: 2010/11/23_17:30:44 info: Local status now set to: 'up'heartbeat[6811]: 2010/11/23_17:30:45 info: Link server1.drbd:eth1 up.heartbeat[6811]: 2010/11/23_17:31:34 WARN: node server2.drbd: is deadheartbeat[6811]: 2010/11/23_17:31:34 info: Comm_now_up(): updating status to activeheartbeat[6811]: 2010/11/23_17:31:34 info: Local status now set to: 'active'heartbeat[6811]: 2010/11/23_17:31:34 WARN: No STONITH device configured.heartbeat[6811]: 2010/11/23_17:31:34 WARN: Shared disks are not protected.heartbeat[6811]: 2010/11/23_17:31:34 info: Resources being acquired from server2.drbd.harc[6900]:    2010/11/23_17:31:34 info: Running /etc/ha.d/rc.d/status statusmach_down[6930]:    2010/11/23_17:31:34 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[6930]:    2010/11/23_17:31:34 info: mach_down takeover complete for node server2.drbd.heartbeat[6811]: 2010/11/23_17:31:34 info: mach_down takeover complete.heartbeat[6811]: 2010/11/23_17:31:34 info: Initial resource acquisition complete (mach_down)IPaddr[6974]:    2010/11/23_17:31:35 INFO:  Resource is stoppedheartbeat[6901]: 2010/11/23_17:31:35 info: Local Resource acquisition completed.harc[7037]:    2010/11/23_17:31:35 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[7037]:    2010/11/23_17:31:35 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[7058]:    2010/11/23_17:31:35 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[7085]:    2010/11/23_17:31:35 INFO:  Resource is stoppedResourceManager[7058]:    2010/11/23_17:31:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[7183]:    2010/11/23_17:31:35 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[7183]:    2010/11/23_17:31:35 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[7154]:    2010/11/23_17:31:35 INFO:  SuccessFilesystem[7302]:    2010/11/23_17:31:35 INFO:  Running OKResourceManager[7058]:    2010/11/23_17:31:35 info: Running /etc/init.d/postgresql  startResourceManager[7058]:    2010/11/23_17:31:37 info: Running /etc/init.d/tomcat6  startheartbeat[6811]: 2010/11/23_17:31:45 info: Local Resource acquisition completed. (none)heartbeat[6811]: 2010/11/23_17:31:45 info: local resource transition completed.heartbeat[6811]: 2010/11/23_17:46:09 info: Heartbeat shutdown in progress. (6811)heartbeat[7831]: 2010/11/23_17:46:09 info: Giving up all HA resources.ResourceManager[7844]:    2010/11/23_17:46:09 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6ResourceManager[7844]:    2010/11/23_17:46:09 info: Running /etc/init.d/tomcat6  stopResourceManager[7844]:    2010/11/23_17:46:13 info: Running /etc/init.d/postgresql  stopResourceManager[7844]:    2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stopFilesystem[8096]:    2010/11/23_17:46:14 INFO: Running stop for /dev/drbd0 on /replicaFilesystem[8096]:    2010/11/23_17:46:14 INFO: Trying to unmount /replicaFilesystem[8096]:    2010/11/23_17:46:14 INFO: unmounted /replica successfullyFilesystem[8085]:    2010/11/23_17:46:14 INFO:  SuccessResourceManager[7844]:    2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/drbddisk r0 stopResourceManager[7844]:    2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stopIPaddr[8244]:    2010/11/23_17:46:14 INFO: ifconfig eth0:0 downIPaddr[8215]:    2010/11/23_17:46:14 INFO:  Successheartbeat[7831]: 2010/11/23_17:46:14 info: All HA resources relinquished.heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBFIFO process 6814 with signal 15heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBWRITE process 6815 with signal 15heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBREAD process 6816 with signal 15heartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6815 exited. 3 remainingheartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6816 exited. 2 remainingheartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6814 exited. 1 remainingheartbeat[6811]: 2010/11/23_17:46:16 info: server1.drbd Heartbeat shutdown complete.heartbeat[8389]: 2010/11/23_17:48:37 info: Version 2 support: falseheartbeat[8389]: 2010/11/23_17:48:37 WARN: Logging daemon is disabled --enabling logging daemon is recommendedheartbeat[8389]: 2010/11/23_17:48:37 info: **************************heartbeat[8389]: 2010/11/23_17:48:37 info: Configuration validated. Starting heartbeat 2.1.3heartbeat[8390]: 2010/11/23_17:48:37 info: heartbeat: version 2.1.3heartbeat[8390]: 2010/11/23_17:48:37 info: Heartbeat generation: 1290373993heartbeat[8390]: 2010/11/23_17:48:37 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1heartbeat[8390]: 2010/11/23_17:48:37 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1heartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_TriggerHandler: Added signal manual handlerheartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_SignalHandler: Added signal handler for signal 17heartbeat[8390]: 2010/11/23_17:48:37 info: Local status now set to: 'up'heartbeat[8390]: 2010/11/23_17:48:38 info: Link server1.drbd:eth1 up.heartbeat[8390]: 2010/11/23_17:49:27 WARN: node server2.drbd: is deadheartbeat[8390]: 2010/11/23_17:49:27 info: Comm_now_up(): updating status to activeheartbeat[8390]: 2010/11/23_17:49:27 info: Local status now set to: 'active'heartbeat[8390]: 2010/11/23_17:49:27 WARN: No STONITH device configured.heartbeat[8390]: 2010/11/23_17:49:27 WARN: Shared disks are not protected.heartbeat[8390]: 2010/11/23_17:49:27 info: Resources being acquired from server2.drbd.harc[8472]:    2010/11/23_17:49:27 info: Running /etc/ha.d/rc.d/status statusmach_down[8502]:    2010/11/23_17:49:27 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down[8502]:    2010/11/23_17:49:27 info: mach_down takeover complete for node server2.drbd.heartbeat[8390]: 2010/11/23_17:49:27 info: mach_down takeover complete.heartbeat[8390]: 2010/11/23_17:49:27 info: Initial resource acquisition complete (mach_down)IPaddr[8545]:    2010/11/23_17:49:27 INFO:  Resource is stoppedheartbeat[8473]: 2010/11/23_17:49:27 info: Local Resource acquisition completed.harc[8609]:    2010/11/23_17:49:27 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-respip-request-resp[8609]:    2010/11/23_17:49:27 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yesResourceManager[8630]:    2010/11/23_17:49:27 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6IPaddr[8657]:    2010/11/23_17:49:28 INFO:  Resource is stoppedResourceManager[8630]:    2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 startIPaddr[8755]:    2010/11/23_17:49:28 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0IPaddr[8755]:    2010/11/23_17:49:28 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255IPaddr[8726]:    2010/11/23_17:49:28 INFO:  SuccessResourceManager[8630]:    2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/drbddisk r0 startFilesystem[8903]:    2010/11/23_17:49:28 INFO:  Resource is stoppedResourceManager[8630]:    2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 startFilesystem[8984]:    2010/11/23_17:49:28 INFO: Running start for /dev/drbd0 on /replicaFilesystem[8973]:    2010/11/23_17:49:28 INFO:  SuccessResourceManager[8630]:    2010/11/23_17:49:28 info: Running /etc/init.d/postgresql  startResourceManager[8630]:    2010/11/23_17:49:30 info: Running /etc/init.d/tomcat6  startheartbeat[8390]: 2010/11/23_17:49:38 info: Local Resource acquisition completed. (none)heartbeat[8390]: 2010/11/23_17:49:38 info: local resource transition completed.
secondary node show some like this part when I shutdown the primary nodehb_standby[5382]:    2010/11/23_16:38:59 Going standby [foreign].heartbeat[3346]: 2010/11/23_16:39:00 info: server1.drbd wants to go standby [foreign]heartbeat[3346]: 2010/11/23_16:39:11 WARN: No reply to standby request.  Standby request cancelled.
well always stay in that last line dont do anything and didn't start the resources in the secondary node 
The config files are the same that this link but with different ip jabali.trexma.com/files/Conf-files.rar without stonith configuration because that tutorial say that didnt need if I wrong please be patiente with me. 

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101125/b075f0b4/attachment.htm>

More information about the drbd-user mailing list