<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"GE Inspira";
panose-1:2 15 6 3 3 4 0 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Looking at the configuration of the Resource, specifically at the “</span><b><span style="color:#1F497D">on-io-error detach”. What is the best practice configuration for LVM device in order to get the best performace?</span></b><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><b><span style="font-size:10.0pt;font-family:"GE Inspira","sans-serif";color:gray">DR<o:p></o:p></span></b></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Reyes, David (GE Energy Management)
<br>
<b>Sent:</b> Tuesday, July 09, 2013 1:58 PM<br>
<b>To:</b> 'drbd-user@lists.linbit.com'<br>
<b>Subject:</b> DRBD 8.4 LVM setup<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Objective:</b> To set up an inexpensive shared storage environment for Oracle Database Active/Passive failover configuration using DRBD.<o:p></o:p></p>
<p class="MsoNormal"><b>DRBD Release =</b> version: 8.4.3<o:p></o:p></p>
<p class="MsoNormal"><b>Operating System =</b> Red Hat Enterprise Linux Server release 5.9<o:p></o:p></p>
<p class="MsoNormal">DRBD RPMS were compiled for Kernel 2.6.18_348.6.1.el5-8.4.3-2.x86_64<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Two Servers = nfs01 and nfs02<o:p></o:p></p>
<p class="MsoNormal">Each Server with an LVM <o:p></o:p></p>
<p class="MsoNormal"><b><u><o:p><span style="text-decoration:none"> </span></o:p></u></b></p>
<p class="MsoNormal"><b><u> PV VG Fmt Attr PSize PFre<o:p></o:p></u></b></p>
<p class="MsoNormal"> /dev/cciss/c0d1p1 drbd-main lvm2 a-- 136.70G 0<o:p></o:p></p>
<p class="MsoNormal"> /dev/cciss/c0d2p1 drbd-main lvm2 a-- 136.70G 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><u> VG #PV #LV #SN Attr VSize VFree<o:p></o:p></u></b></p>
<p class="MsoNormal"> drbd-main 2 1 0 wz--n- 273.39G 0<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u> LV VG Attr LSize Origin Snap% Move Log Copy% Convert<o:p></o:p></u></b></p>
<p class="MsoNormal"> testlv drbd-main -wi-a- 273.39G<o:p></o:p></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b>Dedicated Network interface for DRDB Traffic (options bond1 mode=active-backup arp_interval=1000 arp_ip_target=10.1.1.2)<o:p></o:p></b></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><b><u><span style="color:#1F497D"><o:p><span style="text-decoration:none"> </span></o:p></span></u></b></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D">On both nodes executed:<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D"><o:p><span style="text-decoration:none"> </span></o:p></span></u></b></p>
<p class="MsoNormal"><span style="color:#1F497D">/etc/drbd.d/global_common.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">global { usage-count no; }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">common {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">syncer { rate 10M; }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">}<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">/etc/drbd.d/main.res<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">resource main {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">protocol C;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">net { allow-two-primaries yes; }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">startup { wfc-timeout 0; degr-wfc-timeout 120; become-primary-on both; }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">disk { on-io-error detach; }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">on nfs01 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">device /dev/drbd0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">disk /dev/drbd-main/testlv;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">meta-disk internal;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">address 10.1.1.1:7788;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">on nfs02 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">device /dev/drbd0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">disk /dev/drbd-main/testlv;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">meta-disk internal;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">address 10.1.1.2:7788;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">}<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">drbdadm create-md main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Writing meta data...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">initializing activity log<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">NOT initializing bitmap<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">New drbd meta data block successfully created.<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">service drbd start<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Starting DRBD resources: [<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> create res: main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> prepare disk: main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> adjust disk: main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> adjust net: main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">outdated-wfc-timeout has to be shorter than degr-wfc-timeout<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">outdated-wfc-timeout implicitly set to degr-wfc-timeout (120s)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">..<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">service drbd status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">drbd driver loaded OK; device status:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">version: 8.4.3 (api:1/proto:86-101)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@nfs01, 2013-06-24 12:46:34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">m:res cs ro ds p mounted fstype<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0:main Connected Secondary/Secondary Inconsistent/Inconsistent C<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#1F497D">ONLY ON NFS01<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D">drbdadm primary --force main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">cat /proc/drbd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">version: 8.4.3 (api:1/proto:86-101)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@nfs01, 2013-06-24 12:46:34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0: cs:SyncSource <b>ro:Primary/Secondary ds:UpToDate/Inconsistent</b> C r-----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> ns:9216 nr:0 dw:0 dr:9216 al:0 bm:0 lo:7 pe:0 ua:7 ap:0 ep:1 wo:f oos:286652844<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> [>....................] sync'ed: 0.1% (279932/279940)M<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> finish: 33:10:38 speed: 2,304 (2,304) K/sec<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D">After a few seconds<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D"><o:p><span style="text-decoration:none"> </span></o:p></span></u></b></p>
<p class="MsoNormal"><span style="color:#1F497D">version: 8.4.3 (api:1/proto:86-101)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@nfs01, 2013-06-24 12:46:34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0: cs:Connected ro:Primary/Secondary ds:<b>Diskless</b>/Inconsistent C r-----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> ns:27648 nr:0 dw:0 dr:28160 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<o:p></o:p></span></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D"><o:p><span style="text-decoration:none"> </span></o:p></span></u></b></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#1F497D">/var/log/messages show:<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:#1F497D"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: read: error=-5 s=53248s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0:
<b>Resync aborted.</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: conn( SyncSource -> Connected ) disk( UpToDate -> Failed )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0:
<b>Local IO failed in drbd_endio_read_sec_final. Detaching...</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: Can not satisfy peer's read request, no local data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: Can not satisfy peer's read request, no local data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0 exit code 0 (0x0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:23 nfs01 kernel: block drbd0: Sending NegRSDReply. sector 53248s.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:28 nfs01 kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:28 nfs01 last message repeated 3 times<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:28 nfs01 kernel: block drbd0: bitmap WRITE of 1 pages took 0 jiffies<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:28 nfs01 kernel: block drbd0: 273 GB (71659115 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 9 13:36:28 nfs01 kernel: block
<b>drbd0: disk( Failed -> Diskless )</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">We are trying to set the environment above for proof of concept and hopefully for customers to start using DRBD as a solution to Shared Storage. If a different configuration or approach should be taken, I am
open to ideas. Any help would be greatly appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><b><span style="font-size:10.0pt;font-family:"GE Inspira","sans-serif";color:gray">DR<o:p></o:p></span></b></p>
</div>
</body>
</html>