<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-GB link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Hey guys,<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’ve got a relatively simple drbd + mysql with
heartbeat setup working. In testing this worked well and I tested failover and
failback many times successfully.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>However, now it is in production I’m seeing one
particular issue. It seems that drbd decides to fail over to the secondary, but
heartbeat doesn’t seem to restart mysql. I’m seeing the following
in my syslog at around the time:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>May 4 09:42:56 d1 -- MARK --<o:p></o:p></p>
<p class=MsoNormal>May 4 09:54:28 d1 mysqld[1603]: 100504 9:54:28
[ERROR] /usr/sbin/mysqld: Can't open file: './mydb/T_LOG.frm' (errno: 24)<o:p></o:p></p>
<p class=MsoNormal>May 4 09:54:28 d1 mysqld[1603]: 100504 9:54:28
[ERROR] /usr/sbin/mysqld: Can't open file: './mydb/T_LOG.frm' (errno: 24)<o:p></o:p></p>
<p class=MsoNormal>May 4 09:54:28 d1 mysqld[1603]: 100504 9:54:28
[ERROR] /usr/sbin/mysqld: Can't open file: './mydb/E_SHOW.frm' (errno: 24)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Etc..<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>At the same time in dmesg I get:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>[1044838.725584] drbd1: role( Primary -> Secondary ) <o:p></o:p></p>
<p class=MsoNormal>[1044838.725657] drbd1: Writing meta data super block now.<o:p></o:p></p>
<p class=MsoNormal>[1044838.901794] drbd1: peer( Secondary -> Primary ) <o:p></o:p></p>
<p class=MsoNormal>[1044838.901802] drbd1: Writing meta data super block now.<o:p></o:p></p>
<p class=MsoNormal>[1044860.119442] drbd1: peer( Primary -> Secondary ) <o:p></o:p></p>
<p class=MsoNormal>[1044860.119449] drbd1: Writing meta data super block now.<o:p></o:p></p>
<p class=MsoNormal>[1044860.712560] drbd1: role( Secondary -> Primary ) <o:p></o:p></p>
<p class=MsoNormal>[1044860.712595] drbd1: Writing meta data super block now.<o:p></o:p></p>
<p class=MsoNormal>[1044860.883506] kjournald starting. Commit interval 5
seconds<o:p></o:p></p>
<p class=MsoNormal>[1044860.893665] EXT3 FS on drbd1, internal journal<o:p></o:p></p>
<p class=MsoNormal>[1044860.893672] EXT3-fs: mounted filesystem with ordered
data mode.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Now, all I need to do is restart mysql and all is well.. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Firstly, is there any way of debugging why drbd is switching
over to the secondary and secondly is there any way to determine why heartbeat
isn’t restarting mysql?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Kind regards<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal> -- joe.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Joe Warren-Meeks<o:p></o:p></p>
<p class=MsoNormal>Director Of Systems Development<o:p></o:p></p>
<p class=MsoNormal>ENCORE TICKETS LTD<o:p></o:p></p>
<p class=MsoNormal><span style='color:#999999'>Encore House, 50-51 Bedford Row,
London WC1R 4LR</span><o:p></o:p></p>
<p class=MsoNormal>Direct
line: <span lang=EN-US>+44
(0)20 7492 1506<span style='color:#1F497D'><o:p></o:p></span></span></p>
<p class=MsoNormal>Reservations: +44 (0)20 7492 1500<o:p></o:p></p>
<p class=MsoNormal>Fax:
+44 (0)20 7831 4410<o:p></o:p></p>
<p class=MsoNormal>Email:
<a href="mailto:joe@encoretickets.co.uk"><span style='color:blue'>joe@encoretickets.co.uk</span></a><o:p></o:p></p>
<p class=MsoNormal><span lang=EN-US style='color:black'>web:</span><span
lang=EN-US style='color:navy'>
</span><a href="http://www.encoretickets.co.uk/"><span style='color:blue'>www.encoretickets.co.uk</span></a><o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='color:navy'> <o:p></o:p></span></p>
<p class=MsoNormal align=center style='mso-margin-top-alt:auto;mso-margin-bottom-alt:
auto;text-align:center'><span lang=EN-US style='font-size:7.5pt;color:gray'>Copyright
in this message and any attachments remains with us. It is confidential and may
be legally privileged. If this message is not intended for you it must not be
read, copied or used by you or disclosed to anyone else. Please advise the
sender immediately if you have received this message in error. Although this
message and any attachments are believed to be free of any virus or other
defect that might affect any computer system into which it is received and
opened it is the responsibility of the recipient to ensure that it is virus
free and no responsibility is accepted by Encore Tickets Limited for any loss
or damage in any way arising from its use.<o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>