<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
<br><br><div align="center"><font style="font-style:italic;color:rgb(255, 0, 255);font-size:12pt" size="3"><span style="font-family:Courier New,Courier,Monospace"></span></font><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/pizza.gif" alt="" width="19" height="19"><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/clock.gif" alt="" width="19" height="19"><font style="" color="#e36c09">Avanzando es la unica forma de dejar atras lo que no necesitamos</font><font style="" color="#e36c09">.</font><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/clock.gif" alt="" width="19" height="19"><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/pizza.gif" alt="" width="19" height="19"></div><div align="center"><img src="http://gfx1.hotmail.com/mail/w3/emoticons/rose.gif"><font style="" color="#d99694"> </font><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/lightbulb.gif" alt="" width="19" height="19"><font style="" color="#d99694"><font style="" color="#5f497a">Andrea</font></font><font style="" color="#5f497a"> </font><img src="http://gfx2.hotmail.com/mail/w3/ltr/emoticons/lightbulb.gif" alt="" width="19" height="19"><img src="http://gfx1.hotmail.com/mail/w3/emoticons/rose.gif"></div><br><br><br><br><hr id="stopSpelling">From: solange_sun@hotmail.com<br>To: linux-ha@lists.linux-ha.org<br>Subject: secondary node didnt start services when the primary failed and also didnt failback<br>Date: Thu, 25 Nov 2010 00:09:24 -0500<br><br>
<meta http-equiv="Content-Type" content="text/html; charset=unicode">
<meta name="Generator" content="Microsoft SafeHTML">
<style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
</style>
<span class="ecxApple-style-span" style="font-family:arial, sans-serif;font-size:16px"><span title="">I have a problem with the implementation of a high availability cluster, I have made the implementation as shown on this page <a href="http://www.elastix.org/es/component/kunena/26-tips-and-tricks/53819-elastix-160-clusters-with-drbd-and-heartbeat.html" target="_blank">http://www.elastix.org/es/component/kunena/26-tips-and-tricks/53819-elastix-160-clusters-with-drbd-and-heartbeat.html</a>,follow all the steps but for tomcat and postgresql on a drbd disk, when the service heartbeat starts all the services on the primary server starts too </span><span title="">but then I wanted to simulate the failure I shutdown the</span><span title=""> primary node ,and the secondary node after 5 minutes did not start the services, I began to stop the firewall -the iptables- because the sanity check script sent a few errors of connection in the crm test, mount and umount drbd partition manually works, but when I let </span><span title="" style="background-color:rgb(230, 236, 249);color:rgb(0, 0, 0)">the heartbeat do this, and when I shutdown the primary nodeb is put in a secondary in wfconnection and forever attached logs and server configurations one is a dell and the other a hp proliant but I dont.<br></span></span><div>heartbeat 2.1.3</div><div>heartbeat-stonith 2.1.3</div><div>heartbeat-pils 2.1.3</div><div>drbd83 8.3.8</div><div>kmod-drbd83 838</div><div>ha-log in primary node</div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)">heartbeat[4427]: 2010/11/22_10:13:51 info: killing HBFIFO process 4431 with signal 15<br style="line-height:17px">heartbeat[4427]: 2010/11/22_10:13:51 info: killing HBWRITE process 4432 with signal 15<br style="line-height:17px">heartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4433 exited. 3 remaining<br style="line-height:17px">heartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4431 exited. 2 remaining<br style="line-height:17px">heartbeat[4427]: 2010/11/22_10:13:51 info: Core process 4432 exited. 1 remaining<br style="line-height:17px">heartbeat[4427]: 2010/11/22_10:13:51 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3592]: 2010/11/22_10:21:03 info: Version 2 support: false<br style="line-height:17px">heartbeat[3592]: 2010/11/22_10:21:03 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3592]: 2010/11/22_10:21:03 info: **************************<br style="line-height:17px">heartbeat[3592]: 2010/11/22_10:21:03 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: Heartbeat generation: 1290373957<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:03 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:21:04 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4274]: 2010/11/22_10:23:04 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[4304]: 2010/11/22_10:23:04 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[4275]: 2010/11/22_10:23:04 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:04 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">mach_down[4304]: 2010/11/22_10:23:04 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:14 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:23:14 info: local resource transition completed.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:48 info: Heartbeat shutdown in progress. (3593)<br style="line-height:17px">heartbeat[4703]: 2010/11/22_10:35:48 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:48 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:48 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:52 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[4957]: 2010/11/22_10:35:53 INFO: Success<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5040]: 2010/11/22_10:35:53 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5040]: 2010/11/22_10:35:53 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[5040]: 2010/11/22_10:35:53 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[5029]: 2010/11/22_10:35:53 INFO: Success<br style="line-height:17px">ResourceManager[4716]: 2010/11/22_10:35:53 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[4703]: 2010/11/22_10:35:53 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBFIFO process 3609 with signal 15<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBWRITE process 3610 with signal 15<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: killing HBREAD process 3611 with signal 15<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3611 exited. 3 remaining<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3610 exited. 2 remaining<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: Core process 3609 exited. 1 remaining<br style="line-height:17px">heartbeat[3593]: 2010/11/22_10:35:55 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[5189]: 2010/11/22_10:36:36 info: Version 2 support: false<br style="line-height:17px">heartbeat[5189]: 2010/11/22_10:36:36 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[5189]: 2010/11/22_10:36:36 info: **************************<br style="line-height:17px">heartbeat[5189]: 2010/11/22_10:36:36 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: Heartbeat generation: 1290373958<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:36 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:36:37 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[5310]: 2010/11/22_10:38:37 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[5340]: 2010/11/22_10:38:37 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[5311]: 2010/11/22_10:38:37 info: Local Resource acquisition completed.<br style="line-height:17px">mach_down[5340]: 2010/11/22_10:38:37 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:37 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[5403]: 2010/11/22_10:38:37 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[5403]: 2010/11/22_10:38:37 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:37 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[5495]: 2010/11/22_10:38:37 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[5576]: 2010/11/22_10:38:37 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5565]: 2010/11/22_10:38:37 INFO: Success<br style="line-height:17px">IPaddr[5645]: 2010/11/22_10:38:37 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:37 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[5743]: 2010/11/22_10:38:37 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[5743]: 2010/11/22_10:38:38 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[5714]: 2010/11/22_10:38:38 INFO: Success<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:38 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[5424]: 2010/11/22_10:38:40 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:47 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[5190]: 2010/11/22_10:38:47 info: local resource transition completed.<br style="line-height:17px">heartbeat[3465]: 2010/11/22_10:46:09 info: Version 2 support: false<br style="line-height:17px">heartbeat[3465]: 2010/11/22_10:46:09 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3465]: 2010/11/22_10:46:09 info: **************************<br style="line-height:17px">heartbeat[3465]: 2010/11/22_10:46:09 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: Heartbeat generation: 1290373959<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:09 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:46:10 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4145]: 2010/11/22_10:48:10 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">heartbeat[4146]: 2010/11/22_10:48:10 info: Local Resource acquisition completed.<br style="line-height:17px">mach_down[4186]: 2010/11/22_10:48:10 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4186]: 2010/11/22_10:48:10 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:10 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[4238]: 2010/11/22_10:48:10 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4238]: 2010/11/22_10:48:10 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:10 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4330]: 2010/11/22_10:48:10 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4411]: 2010/11/22_10:48:10 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4400]: 2010/11/22_10:48:10 INFO: Success<br style="line-height:17px">IPaddr[4479]: 2010/11/22_10:48:10 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:10 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4577]: 2010/11/22_10:48:11 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4577]: 2010/11/22_10:48:11 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4548]: 2010/11/22_10:48:11 INFO: Success<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:11 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[4259]: 2010/11/22_10:48:13 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:20 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:48:20 info: local resource transition completed.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:52 info: Heartbeat shutdown in progress. (3466)<br style="line-height:17px">heartbeat[5176]: 2010/11/22_10:49:52 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:52 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:52 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:56 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5459]: 2010/11/22_10:49:57 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5430]: 2010/11/22_10:49:57 INFO: Success<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5521]: 2010/11/22_10:49:57 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5521]: 2010/11/22_10:49:57 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[5521]: 2010/11/22_10:49:57 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[5510]: 2010/11/22_10:49:57 INFO: Success<br style="line-height:17px">ResourceManager[5189]: 2010/11/22_10:49:57 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[5176]: 2010/11/22_10:49:57 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBREAD process 3476 with signal 15<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBFIFO process 3474 with signal 15<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: killing HBWRITE process 3475 with signal 15<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3476 exited. 3 remaining<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3475 exited. 2 remaining<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: Core process 3474 exited. 1 remaining<br style="line-height:17px">heartbeat[3466]: 2010/11/22_10:49:59 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3401]: 2010/11/22_10:58:15 info: Version 2 support: false<br style="line-height:17px">heartbeat[3401]: 2010/11/22_10:58:15 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3401]: 2010/11/22_10:58:15 info: **************************<br style="line-height:17px">heartbeat[3401]: 2010/11/22_10:58:15 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:15 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:15 info: Heartbeat generation: 1290373960<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:16 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3402]: 2010/11/22_10:58:17 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:16 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4070]: 2010/11/22_11:00:16 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[4100]: 2010/11/22_11:00:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[4071]: 2010/11/22_11:00:17 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:17 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:17 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">mach_down[4100]: 2010/11/22_11:00:17 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">harc[4163]: 2010/11/22_11:00:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4163]: 2010/11/22_11:00:17 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:17 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4255]: 2010/11/22_11:00:17 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4336]: 2010/11/22_11:00:17 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4325]: 2010/11/22_11:00:17 INFO: Success<br style="line-height:17px">IPaddr[4404]: 2010/11/22_11:00:17 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:17 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4502]: 2010/11/22_11:00:17 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4502]: 2010/11/22_11:00:17 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4473]: 2010/11/22_11:00:17 INFO: Success<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:17 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[4184]: 2010/11/22_11:00:20 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:27 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:00:27 info: local resource transition completed.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:23 info: Heartbeat shutdown in progress. (3402)<br style="line-height:17px">heartbeat[5135]: 2010/11/22_11:01:23 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:23 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:23 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:27 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5418]: 2010/11/22_11:01:28 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5389]: 2010/11/22_11:01:28 INFO: Success<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5480]: 2010/11/22_11:01:28 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5480]: 2010/11/22_11:01:28 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[5480]: 2010/11/22_11:01:28 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[5469]: 2010/11/22_11:01:28 INFO: Success<br style="line-height:17px">ResourceManager[5148]: 2010/11/22_11:01:28 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[5135]: 2010/11/22_11:01:28 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBREAD process 3421 with signal 15<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBFIFO process 3418 with signal 15<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: killing HBWRITE process 3420 with signal 15<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3420 exited. 3 remaining<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3421 exited. 2 remaining<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: Core process 3418 exited. 1 remaining<br style="line-height:17px">heartbeat[3402]: 2010/11/22_11:01:30 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3349]: 2010/11/22_11:08:16 info: Version 2 support: false<br style="line-height:17px">heartbeat[3349]: 2010/11/22_11:08:16 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3349]: 2010/11/22_11:08:16 info: **************************<br style="line-height:17px">heartbeat[3349]: 2010/11/22_11:08:16 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: Heartbeat generation: 1290373961<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:16 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:08:17 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3985]: 2010/11/22_11:10:17 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">heartbeat[3986]: 2010/11/22_11:10:17 info: Local Resource acquisition completed.<br style="line-height:17px">mach_down[4015]: 2010/11/22_11:10:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4015]: 2010/11/22_11:10:17 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:17 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[4078]: 2010/11/22_11:10:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4078]: 2010/11/22_11:10:17 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:17 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:17 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4170]: 2010/11/22_11:10:17 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:17 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4251]: 2010/11/22_11:10:17 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4240]: 2010/11/22_11:10:17 INFO: Success<br style="line-height:17px">IPaddr[4319]: 2010/11/22_11:10:18 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:18 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4417]: 2010/11/22_11:10:18 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4417]: 2010/11/22_11:10:18 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4388]: 2010/11/22_11:10:18 INFO: Success<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:18 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[4099]: 2010/11/22_11:10:20 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:27 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3350]: 2010/11/22_11:10:27 info: local resource transition completed.<br style="line-height:17px">heartbeat[3444]: 2010/11/22_11:31:31 info: Version 2 support: false<br style="line-height:17px">heartbeat[3444]: 2010/11/22_11:31:31 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3444]: 2010/11/22_11:31:31 info: **************************<br style="line-height:17px">heartbeat[3444]: 2010/11/22_11:31:31 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: Heartbeat generation: 1290373962<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:31 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:31:32 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3888]: 2010/11/22_11:33:32 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[3918]: 2010/11/22_11:33:32 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[3889]: 2010/11/22_11:33:32 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:32 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">mach_down[3918]: 2010/11/22_11:33:32 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">harc[3981]: 2010/11/22_11:33:32 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[3981]: 2010/11/22_11:33:32 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:32 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:32 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:42 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:33:42 info: local resource transition completed.<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 CRIT: Giving up resources due to failure of drbddisk::r0<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:44 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:45 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:45 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:45 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:46 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:46 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:46 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:47 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:47 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:48 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:49 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:49 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:49 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:50 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:50 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:50 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:51 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:51 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:51 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:52 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:52 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:52 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:53 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:53 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:53 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:54 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:54 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:54 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[4939]: 2010/11/22_11:33:55 INFO: Success<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5022]: 2010/11/22_11:33:55 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5011]: 2010/11/22_11:33:55 INFO: Success<br style="line-height:17px">ResourceManager[4002]: 2010/11/22_11:33:55 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">hb_standby[5113]: 2010/11/22_11:34:25 Going standby [foreign].<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:34:26 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3445]: 2010/11/22_11:34:37 WARN: No reply to standby request. Standby request cancelled.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:50:48 info: Heartbeat shutdown in progress. (3445)<br style="line-height:17px">heartbeat[6977]: 2010/11/22_12:50:48 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:48 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:48 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:48 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:48 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:49 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:49 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:49 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:50 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:50 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:50 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:51 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:51 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:51 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:52 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:52 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:53 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:54 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:54 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:54 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:55 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:55 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:55 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:56 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:56 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:56 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:57 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:57 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:57 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:58 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:58 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:58 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[7848]: 2010/11/22_12:50:59 INFO: Success<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[7931]: 2010/11/22_12:50:59 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[7920]: 2010/11/22_12:50:59 INFO: Success<br style="line-height:17px">ResourceManager[6990]: 2010/11/22_12:50:59 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[6977]: 2010/11/22_12:50:59 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBWRITE process 3454 with signal 15<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBREAD process 3455 with signal 15<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: killing HBFIFO process 3453 with signal 15<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3453 exited. 3 remaining<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3455 exited. 2 remaining<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: Core process 3454 exited. 1 remaining<br style="line-height:17px">heartbeat[3445]: 2010/11/22_12:51:01 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[8061]: 2010/11/22_12:51:42 info: Version 2 support: false<br style="line-height:17px">heartbeat[8061]: 2010/11/22_12:51:42 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[8061]: 2010/11/22_12:51:42 info: **************************<br style="line-height:17px">heartbeat[8061]: 2010/11/22_12:51:42 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: Heartbeat generation: 1290373963<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:42 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:51:43 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[8071]: 2010/11/22_12:53:43 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[8103]: 2010/11/22_12:53:43 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[8072]: 2010/11/22_12:53:43 info: Local Resource acquisition completed.<br style="line-height:17px">mach_down[8103]: 2010/11/22_12:53:43 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:43 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[8164]: 2010/11/22_12:53:43 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[8164]: 2010/11/22_12:53:43 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:43 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[8256]: 2010/11/22_12:53:43 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[8337]: 2010/11/22_12:53:43 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[8326]: 2010/11/22_12:53:43 INFO: Success<br style="line-height:17px">IPaddr[8405]: 2010/11/22_12:53:43 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:43 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[8503]: 2010/11/22_12:53:43 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[8503]: 2010/11/22_12:53:43 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[8474]: 2010/11/22_12:53:43 INFO: Success<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:43 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[8185]: 2010/11/22_12:53:45 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:53 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:53:53 info: local resource transition completed.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:17 info: Heartbeat shutdown in progress. (8062)<br style="line-height:17px">heartbeat[8886]: 2010/11/22_12:55:17 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:17 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:17 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:20 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:21 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[9169]: 2010/11/22_12:55:22 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[9140]: 2010/11/22_12:55:22 INFO: Success<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:22 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[9231]: 2010/11/22_12:55:22 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[9231]: 2010/11/22_12:55:22 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[9231]: 2010/11/22_12:55:22 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[9220]: 2010/11/22_12:55:22 INFO: Success<br style="line-height:17px">ResourceManager[8899]: 2010/11/22_12:55:22 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[8886]: 2010/11/22_12:55:22 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBFIFO process 8064 with signal 15<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBWRITE process 8065 with signal 15<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: killing HBREAD process 8066 with signal 15<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8065 exited. 3 remaining<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8066 exited. 2 remaining<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: Core process 8064 exited. 1 remaining<br style="line-height:17px">heartbeat[8062]: 2010/11/22_12:55:24 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[9378]: 2010/11/22_12:56:04 info: Version 2 support: false<br style="line-height:17px">heartbeat[9378]: 2010/11/22_12:56:04 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[9378]: 2010/11/22_12:56:04 info: **************************<br style="line-height:17px">heartbeat[9378]: 2010/11/22_12:56:04 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: Heartbeat generation: 1290373964<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:04 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:56:05 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[9398]: 2010/11/22_12:58:04 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[9428]: 2010/11/22_12:58:04 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[9428]: 2010/11/22_12:58:04 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[9399]: 2010/11/22_12:58:04 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:04 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[9491]: 2010/11/22_12:58:04 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[9491]: 2010/11/22_12:58:04 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:04 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:04 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[9583]: 2010/11/22_12:58:04 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:04 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[9664]: 2010/11/22_12:58:05 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[9653]: 2010/11/22_12:58:05 INFO: Success<br style="line-height:17px">IPaddr[9732]: 2010/11/22_12:58:05 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:05 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[9830]: 2010/11/22_12:58:05 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[9830]: 2010/11/22_12:58:05 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[9801]: 2010/11/22_12:58:05 INFO: Success<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:05 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[9512]: 2010/11/22_12:58:07 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:15 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[9379]: 2010/11/22_12:58:15 info: local resource transition completed.<br style="line-height:17px">heartbeat[3523]: 2010/11/22_15:31:56 info: Version 2 support: false<br style="line-height:17px">heartbeat[3523]: 2010/11/22_15:31:56 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3523]: 2010/11/22_15:31:56 info: **************************<br style="line-height:17px">heartbeat[3523]: 2010/11/22_15:31:56 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: Heartbeat generation: 1290373965<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:56 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:31:57 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4301]: 2010/11/22_15:33:56 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">heartbeat[4302]: 2010/11/22_15:33:56 info: Local Resource acquisition completed.<br style="line-height:17px">mach_down[4342]: 2010/11/22_15:33:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4342]: 2010/11/22_15:33:56 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:33:56 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[4394]: 2010/11/22_15:33:56 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4394]: 2010/11/22_15:33:56 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:33:56 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:33:56 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4486]: 2010/11/22_15:33:56 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:33:56 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4567]: 2010/11/22_15:33:57 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4556]: 2010/11/22_15:33:57 INFO: Success<br style="line-height:17px">IPaddr[4635]: 2010/11/22_15:33:57 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:33:57 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4733]: 2010/11/22_15:33:57 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4733]: 2010/11/22_15:33:57 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4704]: 2010/11/22_15:33:57 INFO: Success<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:33:57 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[4415]: 2010/11/22_15:34:00 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:34:07 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3524]: 2010/11/22_15:34:07 info: local resource transition completed.<br style="line-height:17px">heartbeat[3218]: 2010/11/22_16:46:57 info: Version 2 support: false<br style="line-height:17px">heartbeat[3218]: 2010/11/22_16:46:57 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3218]: 2010/11/22_16:46:57 info: **************************<br style="line-height:17px">heartbeat[3218]: 2010/11/22_16:46:57 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: Heartbeat generation: 1290373966<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:57 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:46:58 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3615]: 2010/11/22_16:48:57 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[3645]: 2010/11/22_16:48:57 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[3645]: 2010/11/22_16:48:57 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3616]: 2010/11/22_16:48:57 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:48:57 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">harc[3708]: 2010/11/22_16:48:57 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[3708]: 2010/11/22_16:48:57 received ip-request-resp drbddisk::r0 OK yes<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:48:57 info: Acquiring resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:48:58 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:49:08 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:49:08 info: local resource transition completed.<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 CRIT: Giving up resources due to failure of drbddisk::r0<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:10 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:11 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:11 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:11 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:12 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:12 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:12 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:13 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:13 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:13 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:14 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:14 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:14 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:15 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:15 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:15 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:16 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:16 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:16 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:18 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:18 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:18 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:19 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:19 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:19 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:20 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:20 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:20 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[4665]: 2010/11/22_16:49:21 INFO: Success<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[4748]: 2010/11/22_16:49:21 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4737]: 2010/11/22_16:49:21 INFO: Success<br style="line-height:17px">ResourceManager[3729]: 2010/11/22_16:49:21 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">hb_standby[4840]: 2010/11/22_16:49:51 Going standby [foreign].<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:49:51 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3219]: 2010/11/22_16:50:03 WARN: No reply to standby request. Standby request cancelled.<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:27 info: Heartbeat shutdown in progress. (3219)<br style="line-height:17px">heartbeat[7475]: 2010/11/23_11:18:27 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:27 info: Releasing resource group: server1.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.10.3.56/24/eth0/10.10.3.255 postgresql tomcat6<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:27 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:27 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:27 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:28 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:28 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:28 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:29 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:29 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:29 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:30 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:30 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:30 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:31 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:31 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:31 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:32 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:32 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:32 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:34 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:34 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:34 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:35 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:35 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:35 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:36 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:36 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:36 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:37 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:37 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:37 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[8346]: 2010/11/23_11:18:38 INFO: Success<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[8429]: 2010/11/23_11:18:38 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[8418]: 2010/11/23_11:18:38 INFO: Success<br style="line-height:17px">ResourceManager[7488]: 2010/11/23_11:18:38 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">heartbeat[7475]: 2010/11/23_11:18:38 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBFIFO process 3227 with signal 15<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBWRITE process 3228 with signal 15<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: killing HBREAD process 3229 with signal 15<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3227 exited. 3 remaining<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3229 exited. 2 remaining<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: Core process 3228 exited. 1 remaining<br style="line-height:17px">heartbeat[3219]: 2010/11/23_11:18:40 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[9421]: 2010/11/23_11:49:15 info: Version 2 support: false<br style="line-height:17px">heartbeat[9421]: 2010/11/23_11:49:15 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[9421]: 2010/11/23_11:49:15 info: **************************<br style="line-height:17px">heartbeat[9421]: 2010/11/23_11:49:15 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: Heartbeat generation: 1290373967<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:15 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:49:16 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[9422]: 2010/11/23_11:51:15 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[9434]: 2010/11/23_11:51:15 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[9464]: 2010/11/23_11:51:15 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">heartbeat[9435]: 2010/11/23_11:51:15 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[9425]: 2010/11/23_11:51:15 WARN: ha_msg_add_nv_depth: line doesn't contain '='<br style="line-height:17px">heartbeat[9425]: 2010/11/23_11:51:15 info: >>><br style="line-height:17px"><br style="line-height:17px">heartbeat[9425]: 2010/11/23_11:51:15 ERROR: NV failure (msgfromsteam): [>>><br style="line-height:17px">]<br style="line-height:17px">mach_down[9464]: 2010/11/23_11:51:15 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[9422]: 2010/11/23_12:08:13 WARN: Shutdown delayed until current resource activity finishes.<br style="line-height:17px">heartbeat[3252]: 2010/11/23_12:47:22 info: Version 2 support: false<br style="line-height:17px">heartbeat[3252]: 2010/11/23_12:47:22 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3252]: 2010/11/23_12:47:22 info: **************************<br style="line-height:17px">heartbeat[3252]: 2010/11/23_12:47:22 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: Heartbeat generation: 1290373968<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:22 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:47:23 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3625]: 2010/11/23_12:49:22 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[3655]: 2010/11/23_12:49:22 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[3655]: 2010/11/23_12:49:22 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:22 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[3699]: 2010/11/23_12:49:22 INFO: Resource is stopped<br style="line-height:17px">heartbeat[3626]: 2010/11/23_12:49:22 info: Local Resource acquisition completed.<br style="line-height:17px">harc[3762]: 2010/11/23_12:49:22 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[3762]: 2010/11/23_12:49:23 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:23 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[3810]: 2010/11/23_12:49:23 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[3908]: 2010/11/23_12:49:23 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[3908]: 2010/11/23_12:49:23 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[3879]: 2010/11/23_12:49:23 INFO: Success<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4056]: 2010/11/23_12:49:23 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:23 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4137]: 2010/11/23_12:49:23 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4126]: 2010/11/23_12:49:23 INFO: Success<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:23 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[3783]: 2010/11/23_12:49:26 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:33 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3253]: 2010/11/23_12:49:33 info: local resource transition completed.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:30 info: Heartbeat shutdown in progress. (3253)<br style="line-height:17px">heartbeat[5002]: 2010/11/23_14:11:30 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:30 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:30 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:34 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5267]: 2010/11/23_14:11:35 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5267]: 2010/11/23_14:11:35 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[5267]: 2010/11/23_14:11:35 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[5256]: 2010/11/23_14:11:35 INFO: Success<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[5015]: 2010/11/23_14:11:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5415]: 2010/11/23_14:11:35 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5386]: 2010/11/23_14:11:35 INFO: Success<br style="line-height:17px">heartbeat[5002]: 2010/11/23_14:11:35 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBFIFO process 3261 with signal 15<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBWRITE process 3262 with signal 15<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: killing HBREAD process 3263 with signal 15<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3262 exited. 3 remaining<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3263 exited. 2 remaining<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: Core process 3261 exited. 1 remaining<br style="line-height:17px">heartbeat[3253]: 2010/11/23_14:11:37 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3287]: 2010/11/23_14:14:49 info: Version 2 support: false<br style="line-height:17px">heartbeat[3287]: 2010/11/23_14:14:49 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3287]: 2010/11/23_14:14:49 info: **************************<br style="line-height:17px">heartbeat[3287]: 2010/11/23_14:14:49 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: Heartbeat generation: 1290373969<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:49 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:14:50 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3784]: 2010/11/23_14:16:50 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[3814]: 2010/11/23_14:16:50 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[3814]: 2010/11/23_14:16:50 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:16:50 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[3851]: 2010/11/23_14:16:50 INFO: Resource is stopped<br style="line-height:17px">heartbeat[3785]: 2010/11/23_14:16:50 info: Local Resource acquisition completed.<br style="line-height:17px">harc[3921]: 2010/11/23_14:16:50 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[3921]: 2010/11/23_14:16:50 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:50 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[3969]: 2010/11/23_14:16:50 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4067]: 2010/11/23_14:16:50 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4067]: 2010/11/23_14:16:50 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4038]: 2010/11/23_14:16:50 INFO: Success<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4215]: 2010/11/23_14:16:50 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:50 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4296]: 2010/11/23_14:16:51 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4285]: 2010/11/23_14:16:51 INFO: Success<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:51 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[3942]: 2010/11/23_14:16:53 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:00 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:00 info: local resource transition completed.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:11 info: Heartbeat shutdown in progress. (3288)<br style="line-height:17px">heartbeat[4668]: 2010/11/23_14:17:11 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:11 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:11 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:15 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[4934]: 2010/11/23_14:17:16 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4934]: 2010/11/23_14:17:16 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[4934]: 2010/11/23_14:17:16 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[4923]: 2010/11/23_14:17:16 INFO: Success<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[4681]: 2010/11/23_14:17:16 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5082]: 2010/11/23_14:17:16 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5053]: 2010/11/23_14:17:16 INFO: Success<br style="line-height:17px">heartbeat[4668]: 2010/11/23_14:17:16 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBFIFO process 3304 with signal 15<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBWRITE process 3305 with signal 15<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: killing HBREAD process 3306 with signal 15<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3305 exited. 3 remaining<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3304 exited. 2 remaining<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: Core process 3306 exited. 1 remaining<br style="line-height:17px">heartbeat[3288]: 2010/11/23_14:17:18 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[5368]: 2010/11/23_14:20:41 info: Version 2 support: false<br style="line-height:17px">heartbeat[5368]: 2010/11/23_14:20:41 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[5368]: 2010/11/23_14:20:41 info: **************************<br style="line-height:17px">heartbeat[5368]: 2010/11/23_14:20:41 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: Heartbeat generation: 1290373970<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:41 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:20:42 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[5418]: 2010/11/23_14:22:41 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[5448]: 2010/11/23_14:22:41 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[5448]: 2010/11/23_14:22:41 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:41 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[5493]: 2010/11/23_14:22:41 INFO: Resource is stopped<br style="line-height:17px">heartbeat[5419]: 2010/11/23_14:22:41 info: Local Resource acquisition completed.<br style="line-height:17px">harc[5555]: 2010/11/23_14:22:41 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[5555]: 2010/11/23_14:22:41 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[5576]: 2010/11/23_14:22:41 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[5603]: 2010/11/23_14:22:41 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[5576]: 2010/11/23_14:22:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[5701]: 2010/11/23_14:22:42 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[5701]: 2010/11/23_14:22:42 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[5672]: 2010/11/23_14:22:42 INFO: Success<br style="line-height:17px">Filesystem[5820]: 2010/11/23_14:22:42 INFO: Running OK<br style="line-height:17px">ResourceManager[5576]: 2010/11/23_14:22:42 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[5576]: 2010/11/23_14:22:44 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:52 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:22:52 info: local resource transition completed.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:40 info: Heartbeat shutdown in progress. (5369)<br style="line-height:17px">heartbeat[8477]: 2010/11/23_14:29:40 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[8490]: 2010/11/23_14:29:40 info: Releasing resource group: imalwaysdead.com IPaddr::10.253.252.251/30<br style="line-height:17px">ResourceManager[8490]: 2010/11/23_14:29:40 info: Running /etc/ha.d/resource.d/IPaddr 10.253.252.251/30 stop<br style="line-height:17px">IPaddr[8528]: 2010/11/23_14:29:40 INFO: Success<br style="line-height:17px">heartbeat[8477]: 2010/11/23_14:29:40 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBFIFO process 5372 with signal 15<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBWRITE process 5373 with signal 15<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: killing HBREAD process 5374 with signal 15<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5373 exited. 3 remaining<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5374 exited. 2 remaining<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: Core process 5372 exited. 1 remaining<br style="line-height:17px">heartbeat[5369]: 2010/11/23_14:29:42 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[30652]: 2010/11/23_14:58:09 info: Version 2 support: false<br style="line-height:17px">heartbeat[30652]: 2010/11/23_14:58:09 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[30652]: 2010/11/23_14:58:09 info: **************************<br style="line-height:17px">heartbeat[30652]: 2010/11/23_14:58:09 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: Heartbeat generation: 1290373985<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:09 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[30653]: 2010/11/23_14:58:11 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[30701]: 2010/11/23_15:00:10 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[30731]: 2010/11/23_15:00:10 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[30731]: 2010/11/23_15:00:10 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:10 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[30775]: 2010/11/23_15:00:10 INFO: Running OK<br style="line-height:17px">heartbeat[30702]: 2010/11/23_15:00:10 info: Local Resource acquisition completed.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:20 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:00:20 info: local resource transition completed.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:34 info: Heartbeat shutdown in progress. (30653)<br style="line-height:17px">heartbeat[31004]: 2010/11/23_15:09:34 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:35 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:35 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:38 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[31269]: 2010/11/23_15:09:39 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[31269]: 2010/11/23_15:09:39 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[31269]: 2010/11/23_15:09:39 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[31258]: 2010/11/23_15:09:39 INFO: Success<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[31017]: 2010/11/23_15:09:39 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[31417]: 2010/11/23_15:09:39 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[31388]: 2010/11/23_15:09:39 INFO: Success<br style="line-height:17px">heartbeat[31004]: 2010/11/23_15:09:39 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBWRITE process 30657 with signal 15<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBREAD process 30658 with signal 15<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: killing HBFIFO process 30656 with signal 15<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30658 exited. 3 remaining<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30656 exited. 2 remaining<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: Core process 30657 exited. 1 remaining<br style="line-height:17px">heartbeat[30653]: 2010/11/23_15:09:41 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3461]: 2010/11/23_15:16:16 info: Version 2 support: false<br style="line-height:17px">heartbeat[3461]: 2010/11/23_15:16:16 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3461]: 2010/11/23_15:16:16 info: **************************<br style="line-height:17px">heartbeat[3461]: 2010/11/23_15:16:16 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: Heartbeat generation: 1290373986<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:16 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:16:17 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4270]: 2010/11/23_15:18:17 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[4300]: 2010/11/23_15:18:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4300]: 2010/11/23_15:18:17 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:17 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[4354]: 2010/11/23_15:18:17 INFO: Resource is stopped<br style="line-height:17px">heartbeat[4271]: 2010/11/23_15:18:17 info: Local Resource acquisition completed.<br style="line-height:17px">harc[4407]: 2010/11/23_15:18:17 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4407]: 2010/11/23_15:18:17 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:17 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[4455]: 2010/11/23_15:18:17 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:17 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4553]: 2010/11/23_15:18:17 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4553]: 2010/11/23_15:18:17 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4524]: 2010/11/23_15:18:17 INFO: Success<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:17 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:27 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:18:27 info: local resource transition completed.<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:29 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:29 CRIT: Giving up resources due to failure of drbddisk::r0<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:29 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:29 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:29 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:30 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:31 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:31 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:31 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:32 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:32 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:32 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:33 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:33 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:33 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:34 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:34 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:34 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:35 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:35 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:35 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:36 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:36 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:36 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:37 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:37 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:37 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:38 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:38 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:38 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:39 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:39 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:39 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:40 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5578]: 2010/11/23_15:18:40 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5567]: 2010/11/23_15:18:41 INFO: Success<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:41 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[4428]: 2010/11/23_15:18:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5707]: 2010/11/23_15:18:41 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5678]: 2010/11/23_15:18:41 INFO: Success<br style="line-height:17px">hb_standby[5748]: 2010/11/23_15:19:11 Going standby [foreign].<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:19:11 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:19:22 WARN: No reply to standby request. Standby request cancelled.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:20 info: Heartbeat shutdown in progress. (3462)<br style="line-height:17px">heartbeat[7067]: 2010/11/23_15:44:20 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:20 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:20 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:20 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:20 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:21 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:21 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:21 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:22 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:22 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:22 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:24 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:24 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:24 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:25 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:25 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:25 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:26 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:26 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:26 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:27 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:27 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:27 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:28 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:28 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:28 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:29 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:29 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:29 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:30 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:30 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:30 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[7949]: 2010/11/23_15:44:31 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[7938]: 2010/11/23_15:44:31 INFO: Success<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[7080]: 2010/11/23_15:44:31 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[8049]: 2010/11/23_15:44:31 INFO: Success<br style="line-height:17px">heartbeat[7067]: 2010/11/23_15:44:31 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBFIFO process 3476 with signal 15<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBWRITE process 3477 with signal 15<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: killing HBREAD process 3479 with signal 15<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3477 exited. 3 remaining<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3479 exited. 2 remaining<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: Core process 3476 exited. 1 remaining<br style="line-height:17px">heartbeat[3462]: 2010/11/23_15:44:33 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3464]: 2010/11/23_15:48:34 info: Version 2 support: false<br style="line-height:17px">heartbeat[3464]: 2010/11/23_15:48:34 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3464]: 2010/11/23_15:48:34 info: **************************<br style="line-height:17px">heartbeat[3464]: 2010/11/23_15:48:34 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: Heartbeat generation: 1290373987<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:34 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:48:35 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[4316]: 2010/11/23_15:50:35 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[4346]: 2010/11/23_15:50:35 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4346]: 2010/11/23_15:50:35 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:35 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[4399]: 2010/11/23_15:50:35 INFO: Resource is stopped<br style="line-height:17px">heartbeat[4317]: 2010/11/23_15:50:35 info: Local Resource acquisition completed.<br style="line-height:17px">harc[4453]: 2010/11/23_15:50:35 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4453]: 2010/11/23_15:50:35 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:35 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[4501]: 2010/11/23_15:50:35 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4599]: 2010/11/23_15:50:35 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4599]: 2010/11/23_15:50:35 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4570]: 2010/11/23_15:50:35 INFO: Success<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:35 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:45 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:50:45 info: local resource transition completed.<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:47 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:47 CRIT: Giving up resources due to failure of drbddisk::r0<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:47 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:47 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:47 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:48 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:49 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:49 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:49 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:50 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:50 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:50 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:51 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:51 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:51 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:52 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:52 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:52 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:53 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:53 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:53 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:54 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:54 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:54 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:55 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:55 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:55 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:56 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:56 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:56 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:57 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:57 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:57 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:58 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5688]: 2010/11/23_15:50:59 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5677]: 2010/11/23_15:50:59 INFO: Success<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:59 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[4474]: 2010/11/23_15:50:59 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5817]: 2010/11/23_15:50:59 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5788]: 2010/11/23_15:50:59 INFO: Success<br style="line-height:17px">hb_standby[5859]: 2010/11/23_15:51:29 Going standby [foreign].<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:51:29 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:51:40 WARN: No reply to standby request. Standby request cancelled.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:22 info: Heartbeat shutdown in progress. (3465)<br style="line-height:17px">heartbeat[6468]: 2010/11/23_15:58:22 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:22 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:22 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:22 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:22 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:23 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:23 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:23 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:24 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:24 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:24 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:25 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:25 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:25 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:26 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:26 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:26 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:27 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:27 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:27 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:28 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:28 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:28 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:29 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:29 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:30 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:31 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:31 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:31 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:32 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:32 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:32 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[7351]: 2010/11/23_15:58:33 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[7340]: 2010/11/23_15:58:33 INFO: Success<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[6481]: 2010/11/23_15:58:33 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[7451]: 2010/11/23_15:58:33 INFO: Success<br style="line-height:17px">heartbeat[6468]: 2010/11/23_15:58:33 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBFIFO process 3490 with signal 15<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBWRITE process 3492 with signal 15<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: killing HBREAD process 3493 with signal 15<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3493 exited. 3 remaining<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3490 exited. 2 remaining<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: Core process 3492 exited. 1 remaining<br style="line-height:17px">heartbeat[3465]: 2010/11/23_15:58:35 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[7643]: 2010/11/23_16:02:40 info: Version 2 support: false<br style="line-height:17px">heartbeat[7643]: 2010/11/23_16:02:40 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[7643]: 2010/11/23_16:02:40 info: **************************<br style="line-height:17px">heartbeat[7643]: 2010/11/23_16:02:40 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: Heartbeat generation: 1290373988<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:40 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:02:41 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[7691]: 2010/11/23_16:04:40 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[7721]: 2010/11/23_16:04:40 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[7721]: 2010/11/23_16:04:40 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:40 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[7763]: 2010/11/23_16:04:40 INFO: Resource is stopped<br style="line-height:17px">heartbeat[7692]: 2010/11/23_16:04:40 info: Local Resource acquisition completed.<br style="line-height:17px">harc[7828]: 2010/11/23_16:04:40 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[7828]: 2010/11/23_16:04:40 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:40 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[7876]: 2010/11/23_16:04:41 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[7974]: 2010/11/23_16:04:41 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[7974]: 2010/11/23_16:04:41 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[7945]: 2010/11/23_16:04:41 INFO: Success<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[8122]: 2010/11/23_16:04:41 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:41 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[8203]: 2010/11/23_16:04:41 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[8192]: 2010/11/23_16:04:41 INFO: Success<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:41 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[7849]: 2010/11/23_16:04:43 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:51 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:04:51 info: local resource transition completed.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:15 info: Heartbeat shutdown in progress. (7644)<br style="line-height:17px">heartbeat[8872]: 2010/11/23_16:14:15 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:15 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:15 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:18 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[9138]: 2010/11/23_16:14:20 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[9138]: 2010/11/23_16:14:20 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[9138]: 2010/11/23_16:14:20 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[9127]: 2010/11/23_16:14:20 INFO: Success<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[8885]: 2010/11/23_16:14:20 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[9286]: 2010/11/23_16:14:20 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[9257]: 2010/11/23_16:14:20 INFO: Success<br style="line-height:17px">heartbeat[8872]: 2010/11/23_16:14:20 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBFIFO process 7647 with signal 15<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBWRITE process 7648 with signal 15<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: killing HBREAD process 7649 with signal 15<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7649 exited. 3 remaining<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7648 exited. 2 remaining<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: Core process 7647 exited. 1 remaining<br style="line-height:17px">heartbeat[7644]: 2010/11/23_16:14:22 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[3345]: 2010/11/23_16:36:04 info: Version 2 support: false<br style="line-height:17px">heartbeat[3345]: 2010/11/23_16:36:04 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3345]: 2010/11/23_16:36:04 info: **************************<br style="line-height:17px">heartbeat[3345]: 2010/11/23_16:36:04 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: Heartbeat generation: 1290373989<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:04 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:36:05 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3903]: 2010/11/23_16:38:05 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[3933]: 2010/11/23_16:38:05 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[3933]: 2010/11/23_16:38:05 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:05 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[3989]: 2010/11/23_16:38:05 INFO: Resource is stopped<br style="line-height:17px">heartbeat[3904]: 2010/11/23_16:38:05 info: Local Resource acquisition completed.<br style="line-height:17px">harc[4040]: 2010/11/23_16:38:05 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4040]: 2010/11/23_16:38:05 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:05 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[4088]: 2010/11/23_16:38:05 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:05 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4186]: 2010/11/23_16:38:05 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4186]: 2010/11/23_16:38:05 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4157]: 2010/11/23_16:38:05 INFO: Success<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:06 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:16 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:38:16 info: local resource transition completed.<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 CRIT: Giving up resources due to failure of drbddisk::r0<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:18 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:19 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:19 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:19 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:20 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:20 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:20 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:21 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:21 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:21 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:22 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:22 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:22 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:23 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:23 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:23 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:24 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:24 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:25 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:26 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:26 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:26 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:27 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:27 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:27 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:28 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:28 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:28 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[5211]: 2010/11/23_16:38:29 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[5200]: 2010/11/23_16:38:29 INFO: Success<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[4061]: 2010/11/23_16:38:29 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[5340]: 2010/11/23_16:38:29 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[5311]: 2010/11/23_16:38:29 INFO: Success<br style="line-height:17px">hb_standby[5382]: 2010/11/23_16:38:59 Going standby [foreign].<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:39:00 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:39:11 WARN: No reply to standby request. Standby request cancelled.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:31 info: Heartbeat shutdown in progress. (3346)<br style="line-height:17px">heartbeat[5591]: 2010/11/23_16:57:31 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:31 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:31 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:31 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:31 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:33 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:33 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:33 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:34 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:34 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:34 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:35 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:35 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:35 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:36 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:36 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:36 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:37 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:37 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:37 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:38 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:38 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:38 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:39 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:39 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:39 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:40 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:40 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:40 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:41 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:41 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:41 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 info: Retrying failed stop operation [postgresql]<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 ERROR: Return code 1 from /etc/init.d/postgresql<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 ERROR: Resource script for postgresql probably not LSB-compliant.<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 WARN: it (postgresql) MUST succeed on a stop when already stopped<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 WARN: Machine reboot narrowly avoided!<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[6473]: 2010/11/23_16:57:42 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[6462]: 2010/11/23_16:57:42 INFO: Success<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[5604]: 2010/11/23_16:57:42 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[6573]: 2010/11/23_16:57:42 INFO: Success<br style="line-height:17px">heartbeat[5591]: 2010/11/23_16:57:43 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBFIFO process 3362 with signal 15<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBWRITE process 3363 with signal 15<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: killing HBREAD process 3364 with signal 15<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3364 exited. 3 remaining<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3363 exited. 2 remaining<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: Core process 3362 exited. 1 remaining<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:57:45 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[6833]: 2010/11/23_17:00:05 info: Version 2 support: false<br style="line-height:17px">heartbeat[6833]: 2010/11/23_17:00:05 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[6833]: 2010/11/23_17:00:05 info: **************************<br style="line-height:17px">heartbeat[6833]: 2010/11/23_17:00:05 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: Heartbeat generation: 1290373990<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:05 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:06 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[6878]: 2010/11/23_17:00:56 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[6908]: 2010/11/23_17:00:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[6908]: 2010/11/23_17:00:56 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:00:56 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[6952]: 2010/11/23_17:00:56 INFO: Resource is stopped<br style="line-height:17px">heartbeat[6879]: 2010/11/23_17:00:56 info: Local Resource acquisition completed.<br style="line-height:17px">harc[7015]: 2010/11/23_17:00:56 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[7015]: 2010/11/23_17:00:56 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[7036]: 2010/11/23_17:00:56 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[7063]: 2010/11/23_17:00:56 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[7036]: 2010/11/23_17:00:56 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[7161]: 2010/11/23_17:00:56 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[7161]: 2010/11/23_17:00:56 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[7132]: 2010/11/23_17:00:56 INFO: Success<br style="line-height:17px">Filesystem[7280]: 2010/11/23_17:00:56 INFO: Running OK<br style="line-height:17px">ResourceManager[7036]: 2010/11/23_17:00:56 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[7036]: 2010/11/23_17:00:58 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:01:06 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[6834]: 2010/11/23_17:01:06 info: local resource transition completed.<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:15 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:15 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:17 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:17 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:19 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:19 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:21 ERROR: glib: Unable to send bcast [-1] packet(len=217): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:21 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:21 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:21 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:23 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:23 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:25 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:25 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:27 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:27 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:29 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:29 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:31 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:31 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:33 ERROR: glib: Unable to send bcast [-1] packet(len=214): No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:33 ERROR: write_child: write failure on bcast eth1.: No such device<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:33 WARN: Temporarily Suppressing write error messages<br style="line-height:17px">heartbeat[6838]: 2010/11/23_17:01:33 WARN: Is a cable unplugged on bcast eth1?<br style="line-height:17px">heartbeat[3300]: 2010/11/23_17:17:49 info: Version 2 support: false<br style="line-height:17px">heartbeat[3300]: 2010/11/23_17:17:49 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[3300]: 2010/11/23_17:17:49 info: **************************<br style="line-height:17px">heartbeat[3300]: 2010/11/23_17:17:49 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: Heartbeat generation: 1290373991<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:49 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:17:50 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[3980]: 2010/11/23_17:18:40 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[4010]: 2010/11/23_17:18:40 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[4010]: 2010/11/23_17:18:40 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:40 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[4056]: 2010/11/23_17:18:40 INFO: Resource is stopped<br style="line-height:17px">heartbeat[3981]: 2010/11/23_17:18:40 info: Local Resource acquisition completed.<br style="line-height:17px">harc[4117]: 2010/11/23_17:18:40 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[4117]: 2010/11/23_17:18:40 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:40 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[4165]: 2010/11/23_17:18:40 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:40 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[4263]: 2010/11/23_17:18:40 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[4263]: 2010/11/23_17:18:40 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[4234]: 2010/11/23_17:18:40 INFO: Success<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:41 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[4411]: 2010/11/23_17:18:41 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:41 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[4492]: 2010/11/23_17:18:41 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[4481]: 2010/11/23_17:18:41 INFO: Success<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:41 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[4138]: 2010/11/23_17:18:43 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:51 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:18:51 info: local resource transition completed.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:01 info: Heartbeat shutdown in progress. (3301)<br style="line-height:17px">heartbeat[6003]: 2010/11/23_17:22:01 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:01 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:01 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:05 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[6268]: 2010/11/23_17:22:06 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[6268]: 2010/11/23_17:22:06 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[6268]: 2010/11/23_17:22:06 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[6257]: 2010/11/23_17:22:06 INFO: Success<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[6016]: 2010/11/23_17:22:06 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[6416]: 2010/11/23_17:22:06 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[6387]: 2010/11/23_17:22:06 INFO: Success<br style="line-height:17px">heartbeat[6003]: 2010/11/23_17:22:06 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBFIFO process 3307 with signal 15<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBWRITE process 3308 with signal 15<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: killing HBREAD process 3309 with signal 15<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3308 exited. 3 remaining<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3307 exited. 2 remaining<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: Core process 3309 exited. 1 remaining<br style="line-height:17px">heartbeat[3301]: 2010/11/23_17:22:08 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[6810]: 2010/11/23_17:30:44 info: Version 2 support: false<br style="line-height:17px">heartbeat[6810]: 2010/11/23_17:30:44 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[6810]: 2010/11/23_17:30:44 info: **************************<br style="line-height:17px">heartbeat[6810]: 2010/11/23_17:30:44 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: Heartbeat generation: 1290373992<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:44 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:30:45 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[6900]: 2010/11/23_17:31:34 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[6930]: 2010/11/23_17:31:34 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[6930]: 2010/11/23_17:31:34 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:34 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[6974]: 2010/11/23_17:31:35 INFO: Resource is stopped<br style="line-height:17px">heartbeat[6901]: 2010/11/23_17:31:35 info: Local Resource acquisition completed.<br style="line-height:17px">harc[7037]: 2010/11/23_17:31:35 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[7037]: 2010/11/23_17:31:35 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[7058]: 2010/11/23_17:31:35 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[7085]: 2010/11/23_17:31:35 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[7058]: 2010/11/23_17:31:35 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[7183]: 2010/11/23_17:31:35 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[7183]: 2010/11/23_17:31:35 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[7154]: 2010/11/23_17:31:35 INFO: Success<br style="line-height:17px">Filesystem[7302]: 2010/11/23_17:31:35 INFO: Running OK<br style="line-height:17px">ResourceManager[7058]: 2010/11/23_17:31:35 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[7058]: 2010/11/23_17:31:37 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:45 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:31:45 info: local resource transition completed.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:09 info: Heartbeat shutdown in progress. (6811)<br style="line-height:17px">heartbeat[7831]: 2010/11/23_17:46:09 info: Giving up all HA resources.<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:09 info: Releasing resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:09 info: Running /etc/init.d/tomcat6 stop<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:13 info: Running /etc/init.d/postgresql stop<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 stop<br style="line-height:17px">Filesystem[8096]: 2010/11/23_17:46:14 INFO: Running stop for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[8096]: 2010/11/23_17:46:14 INFO: Trying to unmount /replica<br style="line-height:17px">Filesystem[8096]: 2010/11/23_17:46:14 INFO: unmounted /replica successfully<br style="line-height:17px">Filesystem[8085]: 2010/11/23_17:46:14 INFO: Success<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/drbddisk r0 stop<br style="line-height:17px">ResourceManager[7844]: 2010/11/23_17:46:14 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 stop<br style="line-height:17px">IPaddr[8244]: 2010/11/23_17:46:14 INFO: ifconfig eth0:0 down<br style="line-height:17px">IPaddr[8215]: 2010/11/23_17:46:14 INFO: Success<br style="line-height:17px">heartbeat[7831]: 2010/11/23_17:46:14 info: All HA resources relinquished.<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBFIFO process 6814 with signal 15<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBWRITE process 6815 with signal 15<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: killing HBREAD process 6816 with signal 15<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6815 exited. 3 remaining<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6816 exited. 2 remaining<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: Core process 6814 exited. 1 remaining<br style="line-height:17px">heartbeat[6811]: 2010/11/23_17:46:16 info: server1.drbd Heartbeat shutdown complete.<br style="line-height:17px">heartbeat[8389]: 2010/11/23_17:48:37 info: Version 2 support: false<br style="line-height:17px">heartbeat[8389]: 2010/11/23_17:48:37 WARN: Logging daemon is disabled --enabling logging daemon is recommended<br style="line-height:17px">heartbeat[8389]: 2010/11/23_17:48:37 info: **************************<br style="line-height:17px">heartbeat[8389]: 2010/11/23_17:48:37 info: Configuration validated. Starting heartbeat 2.1.3<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: heartbeat: version 2.1.3<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: Heartbeat generation: 1290373993<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_TriggerHandler: Added signal manual handler<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: G_main_add_SignalHandler: Added signal handler for signal 17<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:37 info: Local status now set to: 'up'<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:48:38 info: Link server1.drbd:eth1 up.<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 WARN: node server2.drbd: is dead<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 info: Comm_now_up(): updating status to active<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 info: Local status now set to: 'active'<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 WARN: No STONITH device configured.<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 WARN: Shared disks are not protected.<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 info: Resources being acquired from server2.drbd.<br style="line-height:17px">harc[8472]: 2010/11/23_17:49:27 info: Running /etc/ha.d/rc.d/status status<br style="line-height:17px">mach_down[8502]: 2010/11/23_17:49:27 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired<br style="line-height:17px">mach_down[8502]: 2010/11/23_17:49:27 info: mach_down takeover complete for node server2.drbd.<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 info: mach_down takeover complete.<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:27 info: Initial resource acquisition complete (mach_down)<br style="line-height:17px">IPaddr[8545]: 2010/11/23_17:49:27 INFO: Resource is stopped<br style="line-height:17px">heartbeat[8473]: 2010/11/23_17:49:27 info: Local Resource acquisition completed.<br style="line-height:17px">harc[8609]: 2010/11/23_17:49:27 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<br style="line-height:17px">ip-request-resp[8609]: 2010/11/23_17:49:27 received ip-request-resp IPaddr::10.10.3.56/24/eth0/10.10.3.255 OK yes<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:27 info: Acquiring resource group: server1.drbd IPaddr::10.10.3.56/24/eth0/10.10.3.255 drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 postgresql tomcat6<br style="line-height:17px">IPaddr[8657]: 2010/11/23_17:49:28 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/IPaddr 10.10.3.56/24/eth0/10.10.3.255 start<br style="line-height:17px">IPaddr[8755]: 2010/11/23_17:49:28 INFO: Using calculated netmask for 10.10.3.56: 255.255.255.0<br style="line-height:17px">IPaddr[8755]: 2010/11/23_17:49:28 INFO: eval ifconfig eth0:0 10.10.3.56 netmask 255.255.255.0 broadcast 10.10.3.255<br style="line-height:17px">IPaddr[8726]: 2010/11/23_17:49:28 INFO: Success<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/drbddisk r0 start<br style="line-height:17px">Filesystem[8903]: 2010/11/23_17:49:28 INFO: Resource is stopped<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:28 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext3 start<br style="line-height:17px">Filesystem[8984]: 2010/11/23_17:49:28 INFO: Running start for /dev/drbd0 on /replica<br style="line-height:17px">Filesystem[8973]: 2010/11/23_17:49:28 INFO: Success<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:28 info: Running /etc/init.d/postgresql start<br style="line-height:17px">ResourceManager[8630]: 2010/11/23_17:49:30 info: Running /etc/init.d/tomcat6 start<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:38 info: Local Resource acquisition completed. (none)<br style="line-height:17px">heartbeat[8390]: 2010/11/23_17:49:38 info: local resource transition completed.</span></div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)"><br></span></div><div>secondary node show some like this part when I shutdown the primary node</div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)">hb_standby[5382]: 2010/11/23_16:38:59 Going standby [foreign].<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:39:00 info: server1.drbd wants to go standby [foreign]<br style="line-height:17px">heartbeat[3346]: 2010/11/23_16:39:11 WARN: No reply to standby request. Standby request cancelled.</span></div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)"><br></span></div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)">well always stay in that last line dont do anything and didn't start the resources in the secondary node </span></div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)"><br></span></div><div><span class="ecxApple-style-span" style="font-family:'Segoe UI', Tahoma, Verdana, Arial, sans-serif;color:rgb(42, 42, 42)">The config files are the same that this link but with different ip </span><span class="ecxApple-style-span" style="font-family:'Trebuchet MS', Arial, Helvetica, sans-serif;font-size:12px;color:rgb(102, 102, 102)"><a href="http://jabali.trexma.com/files/Conf-files.rar" rel="nofollow" style="margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;text-decoration:none;color:rgb(71, 60, 139);font-weight:normal;font-size:13px" target="_blank">jabali.trexma.com/files/Conf-files.rar</a> without stonith configuration because that tutorial say that didnt need if I wrong please be patiente with me.</span></div><div><span class="ecxApple-style-span" style="font-family:'Trebuchet MS', Arial, Helvetica, sans-serif;font-size:12px;color:rgb(102, 102, 102)"></span><font style="" color="#5f497a"> </font></div><br><br> </body>
</html>