<html><body><div style="color:#000; background-color:#fff; font-family:Courier New, courier, monaco, monospace, sans-serif;font-size:10pt"><div><span>I decided to nuke the whole thing and start over:</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>On both nodes, I...</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style:
normal;"><span>1. disabled the cluster</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> sanX:~ # crm node standby</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>2. re-create the md</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>sanX:~ </span># drbdadm create-md r0<br></span></div><div><span> </span><span><span>sanX:~ </span># drbdadm create-md r1</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif;
background-color: transparent; font-style: normal;"><span>3. bring the DRBD resources online</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>sanX:~ </span># drbdadm up r0</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>sanX:~ </span># drbdadm up r1</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style:
normal;"><span>And then, on the node intended to act as the DRBD resource primary, I...</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>4. designated the node as primary [and sync'd the two nodes' resources]<br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>san1:~ </span># drbdadm primary r0 --force</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent;
font-style: normal;"><span> </span><span><span>san2:~ </span># drbdadm primary r1 --force</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;">After the resources completed sync'ing I...</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;">5. took the DRBD resources offline<br><span></span></div><div style="color: rgb(0, 0, 0); font-size:
13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>sanX:~ </span># drbdadm down r0</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> </span><span><span>sanX:~ </span># drbdadm down r1</span></div><span></span><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>6. restarted the cluster</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> sanX:~ # crm node online</span></div><span></span><div style="color: rgb(0, 0, 0); font-size:
13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>7. verified that DRBD was functioning correctly</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> sanX:~ # cat /proc/drbd</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span> version: 8.4.1 (api:1/proto:86-100)<br> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by phil@fat-tyre, 2011-12-20 12:43:15<br> 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:0<br> 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> <br> 1000: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> 1001: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0
lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> 1002: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br> 1003: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>8. Checked the kernel ring buffer...</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>[13107.929867] d-con r0: Starting worker thread (from drbdsetup [27297])<br>[13107.930025] block drbd0: disk( Diskless ->
Attaching ) <br>[13107.935720] d-con r0: Method to ensure write ordering: flush<br>[13107.935728] block drbd0: max BIO size = 1048576<br>[13107.935736] block drbd0: drbd_bm_resize called with capacity == 104868976<br>[13107.936667] block drbd0: resync bitmap: bits=13108622 words=204823 pages=401<br>[13107.936678] block drbd0: size = 50 GB (52434488 KB)<br>[13107.961687] block drbd0: bitmap READ of 401 pages took 6 jiffies<br>[13107.964602] block drbd0: recounting of set bits took additional 1 jiffies<br>[13107.964608] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13107.964619] block drbd0: disk( Attaching -> UpToDate ) <br>[13107.964624] block drbd0: attached to UUIDs 5D13C510E7C43556:0000000000000000:CC7A830506E5F9FA:0000000000000004<br>[13107.970870] d-con r1: Starting worker thread (from drbdsetup [27300])<br>[13107.973936] block drbd1000: disk( Diskless -> Attaching ) <br>[13107.982866] d-con r1: Method to ensure write
ordering: flush<br>[13107.982874] block drbd1000: max BIO size = 1048576<br>[13107.982881] block drbd1000: drbd_bm_resize called with capacity == 104868976<br>[13107.983829] block drbd1000: resync bitmap: bits=13108622 words=204823 pages=401<br>[13107.983841] block drbd1000: size = 50 GB (52434488 KB)<br>[13108.017395] block drbd1000: bitmap READ of 401 pages took 9 jiffies<br>[13108.019262] block drbd1000: recounting of set bits took additional 0 jiffies<br>[13108.019268] block drbd1000: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.019280] block drbd1000: disk( Attaching -> UpToDate ) <br>[13108.019285] block drbd1000: attached to UUIDs 9162525E46599CBB:1B6D97817803DDE5:531ACCA34B5A8A81:0000000000000004<br>[13108.044402] block drbd1: disk( Diskless -> Attaching ) <br>[13108.044863] block drbd1: max BIO size = 1048576<br>[13108.044872] block drbd1: drbd_bm_resize called with capacity == 104868976<br>[13108.045862] block drbd1:
resync bitmap: bits=13108622 words=204823 pages=401<br>[13108.045870] block drbd1: size = 50 GB (52434488 KB)<br>[13108.078330] block drbd1: bitmap READ of 401 pages took 8 jiffies<br>[13108.080379] block drbd1: recounting of set bits took additional 1 jiffies<br>[13108.080385] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.080396] block drbd1: disk( Attaching -> UpToDate ) <br>[13108.080401] block drbd1: attached to UUIDs 3DD43846444979D0:0000000000000000:C2E4C547C3069E48:0000000000000004<br>[13108.153218] block drbd1001: disk( Diskless -> Attaching ) <br>[13108.153641] block drbd1001: max BIO size = 1048576<br>[13108.153653] block drbd1001: drbd_bm_resize called with capacity == 104868976<br>[13108.154769] block drbd1001: resync bitmap: bits=13108622 words=204823 pages=401<br>[13108.154781] block drbd1001: size = 50 GB (52434488 KB)<br>[13108.202463] block drbd1001: bitmap READ of 401 pages took 12
jiffies<br>[13108.204504] block drbd1001: recounting of set bits took additional 1 jiffies<br>[13108.204510] block drbd1001: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.204521] block drbd1001: disk( Attaching -> UpToDate ) <br>[13108.204526] block drbd1001: attached to UUIDs 77EFDEC15B06F199:DF7193C3448F470B:75319BD97ED29A6A:0000000000000004<br>[13108.209437] block drbd2: disk( Diskless -> Attaching ) <br>[13108.209869] block drbd2: max BIO size = 1048576<br>[13108.209879] block drbd2: drbd_bm_resize called with capacity == 52450480<br>[13108.210355] block drbd2: resync bitmap: bits=6556310 words=102443 pages=201<br>[13108.210364] block drbd2: size = 25 GB (26225240 KB)<br>[13108.248940] block drbd2: bitmap READ of 201 pages took 10 jiffies<br>[13108.249882] block drbd2: recounting of set bits took additional 0 jiffies<br>[13108.249887] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.249898] block
drbd2: disk( Attaching -> UpToDate ) <br>[13108.249903] block drbd2: attached to UUIDs 4ED02AF2177212EC:0000000000000000:892841F769F80FD2:0000000000000004<br>[13108.280862] block drbd1002: disk( Diskless -> Attaching ) <br>[13108.284504] block drbd1002: max BIO size = 1048576<br>[13108.284517] block drbd1002: drbd_bm_resize called with capacity == 52450480<br>[13108.284898] block drbd1002: resync bitmap: bits=6556310 words=102443 pages=201<br>[13108.284905] block drbd1002: size = 25 GB (26225240 KB)<br>[13108.310601] block drbd1002: bitmap READ of 201 pages took 6 jiffies<br>[13108.311534] block drbd1002: recounting of set bits took additional 0 jiffies<br>[13108.311538] block drbd1002: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.311549] block drbd1002: disk( Attaching -> UpToDate ) <br>[13108.311554] block drbd1002: attached to UUIDs 6162EF30A167BF53:3874D93F4CBE04F5:CFFD994FAA06C840:0000000000000004<br>[13108.346176] block
drbd3: disk( Diskless -> Attaching ) <br>[13108.346903] block drbd3: max BIO size = 1048576<br>[13108.346914] block drbd3: drbd_bm_resize called with capacity == 52450480<br>[13108.347352] block drbd3: resync bitmap: bits=6556310 words=102443 pages=201<br>[13108.347360] block drbd3: size = 25 GB (26225240 KB)<br>[13108.393579] block drbd3: bitmap READ of 201 pages took 12 jiffies<br>[13108.394521] block drbd3: recounting of set bits took additional 0 jiffies<br>[13108.394526] block drbd3: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.394537] block drbd3: disk( Attaching -> UpToDate ) <br>[13108.394542] block drbd3: attached to UUIDs F424BEA340DA597E:0000000000000000:7534D6E7D2916062:0000000000000004<br>[13108.415953] d-con r0: conn( StandAlone -> Unconnected ) <br>[13108.415992] d-con r0: Starting receiver thread (from drbd_w_r0 [27299])<br>[13108.416137] d-con r0: receiver (re)started<br>[13108.416169] d-con r0: conn(
Unconnected -> WFConnection ) <br>[13108.429798] block drbd1003: disk( Diskless -> Attaching ) <br>[13108.430229] block drbd1003: max BIO size = 1048576<br>[13108.430239] block drbd1003: drbd_bm_resize called with capacity == 52450480<br>[13108.430690] block drbd1003: resync bitmap: bits=6556310 words=102443 pages=201<br>[13108.430700] block drbd1003: size = 25 GB (26225240 KB)<br>[13108.470077] block drbd1003: bitmap READ of 201 pages took 10 jiffies<br>[13108.471604] block drbd1003: recounting of set bits took additional 0 jiffies<br>[13108.471611] block drbd1003: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>[13108.471625] block drbd1003: disk( Attaching -> UpToDate ) <br>[13108.471631] block drbd1003: attached to UUIDs 8FA72E3EFCD7F989:24C98CC08449A9C5:0A1A0D1A2CCE437D:0000000000000004<br>[13108.515667] d-con r1: conn( StandAlone -> Unconnected ) <br>[13108.515759] d-con r1: Starting receiver thread (from drbd_w_r1
[27304])<br>[13108.520305] d-con r1: receiver (re)started<br>[13108.520345] d-con r1: conn( Unconnected -> WFConnection ) <br>[13108.896962] block drbd1000: role( Secondary -> Primary ) <br>[13108.909627] block drbd0: role( Secondary -> Primary ) <br>[13108.919828] block drbd0: new current UUID 9456ACA94896E7CB:5D13C510E7C43556:CC7A830506E5F9FA:0000000000000004<br>[13108.928460] block drbd1001: role( Secondary -> Primary ) <br>[13108.942434] block drbd1: role( Secondary -> Primary ) <br>[13108.953932] block drbd1: new current UUID 61301C50D5932F3B:3DD43846444979D0:C2E4C547C3069E48:0000000000000004<br>[13108.962680] block drbd1002: role( Secondary -> Primary ) <br>[13108.979068] block drbd2: role( Secondary -> Primary ) <br>[13108.994196] block drbd2: new current UUID 7A60A90478413765:4ED02AF2177212EC:892841F769F80FD2:0000000000000004<br>[13109.002821] block drbd1003: role( Secondary -> Primary ) <br>[13109.016954] block drbd3:
role( Secondary -> Primary ) <br>[13109.030957] block drbd3: new current UUID E8E6056FE9F372BB:F424BEA340DA597E:7534D6E7D2916062:0000000000000004</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>However, I still have no idea what caused the failures.</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;">Ideas?
Suggestions?</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>Eric Pretorious</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span>Truckee, CA</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: Courier New,courier,monaco,monospace,sans-serif; background-color: transparent; font-style: normal;"><span><br></span><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; margin-top: 5px; padding-left: 5px;"> <div style="font-family: Courier New, courier,
monaco, monospace, sans-serif; font-size: 10pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Eric <epretorious@yahoo.com><br> <b><span style="font-weight: bold;">To:</span></b> Dan Barker <dbarker@visioncomm.net>; "drbd-user@lists.linbit.com" <drbd-user@lists.linbit.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, January 21, 2013 1:48 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [DRBD-user] Diagnosing a Failed Resource<br> </font> </div> <br>
<div id="yiv1224273097"><div><div style="color:#000;background-color:#fff;font-family:Courier New, courier, monaco, monospace, sans-serif;font-size:10pt;">Dan:<br><br>I attempted to re-connect r1 and then captured the results in the kernel ring buffer:<br><br>[ 7038.868118] d-con r1: conn( StandAlone -> Unconnected ) <br>[ 7038.868170] d-con r1: Starting receiver thread (from drbd_w_r1 [5058])<br>[ 7038.868238] d-con r1: receiver (re)started<br>[ 7038.868269] d-con r1: conn( Unconnected -> WFConnection ) <br>[ 7039.367612] d-con r1: Handshake successful: Agreed network protocol version 100<br>[ 7039.367818] d-con r1: conn( WFConnection -> WFReportParams ) <br>[ 7039.367851] d-con r1: Starting asender thread (from drbd_r_r1 [20387])<br>[ 7039.376119] block drbd1000: drbd_sync_handshake:<br>[ 7039.376127] block drbd1000: self E624A5F197121810:701073FA8F926E0E:B8DFF1CE13CF5A28:B8DEF1CE13CF5A29 bits:0 flags:0<br>[ 7039.376133] block drbd1000: peer
221AECCC2A594D3B:701073FA8F926E0E:B8DFF1CE13CF5A29:B8DEF1CE13CF5A29 bits:0 flags:0<br>[ 7039.376139] block drbd1000: uuid_compare()=100 by rule 90<br>[ 7039.376146] block drbd1000: helper command: /sbin/drbdadm initial-split-brain minor-1000<br>[ 7039.378519] block drbd1000: helper command: /sbin/drbdadm initial-split-brain minor-1000 exit code 0 (0x0)<br>[ 7039.378544] block drbd1000: Split-Brain detected but unresolved, dropping connection!<br>[ 7039.378551] block drbd1000: helper command: /sbin/drbdadm split-brain minor-1000<br>[ 7039.381167] block drbd1000: helper command: /sbin/drbdadm split-brain minor-1000 exit code 0 (0x0)<br>[ 7039.381228] d-con r1: conn( WFReportParams -> Disconnecting ) <br>[ 7039.381237] d-con r1: error receiving ReportState, e: -5 l: 0!<br>[ 7039.381274] d-con r1: asender terminated<br>[ 7039.381283] d-con r1: Terminating asender thread<br>[ 7039.381539] d-con r1: Connection closed<br>[ 7039.381567] d-con r1: conn(
Disconnecting -> StandAlone ) <br>[ 7039.381573] d-con r1: receiver terminated<br>[ 7039.381577] d-con r1: Terminating receiver thread<br><br><div><span>I have no idea what it means, however.</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:Courier New, courier, monaco, monospace, sans-serif;background-color:transparent;font-style:normal;"><br><span></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:Courier New, courier, monaco, monospace, sans-serif;background-color:transparent;font-style:normal;"><span>Eric Pretorious</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:Courier New, courier, monaco, monospace, sans-serif;background-color:transparent;font-style:normal;"><span>Truckee, CA<br></span></div><div><br><blockquote style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;margin-top:5px;padding-left:5px;"> <div style="font-family:Courier New, courier, monaco,
monospace, sans-serif;font-size:10pt;"> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Dan Barker <dbarker@visioncomm.net><br> <b><span style="font-weight:bold;">To:</span></b> "drbd-user@lists.linbit.com" <drbd-user@lists.linbit.com> <br> <b><span style="font-weight:bold;">Sent:</span></b> Monday, January 21, 2013 6:40 AM<br> <b><span style="font-weight:bold;">Subject:</span></b> Re: [DRBD-user] Diagnosing a Failed Resource<br> </font> </div> <br>
<div id="yiv1224273097">
<style><!--
#yiv1224273097
filtered {font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;}
#yiv1224273097 filtered {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}
#yiv1224273097 filtered {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}
#yiv1224273097
p.yiv1224273097MsoNormal, #yiv1224273097 li.yiv1224273097MsoNormal, #yiv1224273097 div.yiv1224273097MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Times New Roman", "serif";}
#yiv1224273097 a:link, #yiv1224273097 span.yiv1224273097MsoHyperlink
{color:blue;text-decoration:underline;}
#yiv1224273097 a:visited, #yiv1224273097 span.yiv1224273097MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}
#yiv1224273097 span.yiv1224273097EmailStyle17
{font-family:"Calibri", "sans-serif";color:#1F497D;}
#yiv1224273097 .yiv1224273097MsoChpDefault
{font-size:10.0pt;}
#yiv1224273097 filtered {margin:1.0in 1.0in 1.0in 1.0in;}
#yiv1224273097 div.yiv1224273097WordSection1
{}
--></style>
<div>
<div class="yiv1224273097WordSection1">
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">The errors in connecting are logged. If you can’t find them, attempt to connect a resource (drbdadm connect r1, for example) to create the errors again, and
then look at the logs for the reason the connection was not established. The “status” will continue to show waiting for connection (WFC) but there will be a reason in the log files. If the logs are unclear, post the relevant portions back here and we’ll help.</span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Something like ‘dmesg | grep drbd’. You may want to do the logs on both drbd servers. You can do the connect command on either.</span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">hth</span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Dan</span></div>
<div class="yiv1224273097MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in;">
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><b><span style="font-size:10.0pt;">From:</span></b><span style="font-size:10.0pt;"> drbd-user-bounces@lists.linbit.com [mailto:drbd-user-bounces@lists.linbit.com]
<b>On Behalf Of </b>Eric<br>
<b>Sent:</b> Monday, January 21, 2013 1:24 AM<br>
<b>To:</b> drbd-user@lists.linbit.com<br>
<b>Subject:</b> [DRBD-user] Diagnosing a Failed Resource</span></div>
</div>
</div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"> </div>
<div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;background:white;"><span style="font-size:10.0pt;color:black;">I've configured corosync+pacemaker to managee a simple two-resource DRBD cluster:</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;background:white;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">> san1:~ # crm configure show | cat -<br>
> node san1 \<br>
> attributes standby="off"<br>
> node san2 \<br>
> attributes standby="off"<br>
> primitive p_DRBD-r0 ocf:linbit:drbd \<br>
> params drbd_resource="r0" \<br>
> op monitor interval="60s"<br>
> primitive p_DRBD-r1 ocf:linbit:drbd \<br>
> params drbd_resource="r1" \<br>
> op monitor interval="60s"<br>
> primitive p_IP-1_253 ocf:heartbeat:IPaddr2 \<br>
> params ip="192.168.1.253" cidr_netmask="24" \<br>
> op monitor interval="30s"<br>
> primitive p_IP-1_254 ocf:heartbeat:IPaddr2 \<br>
> params ip="192.168.1.254" cidr_netmask="24" \<br>
> op monitor interval="30s"<br>
> primitive p_iSCSI-san1 ocf:heartbeat:iSCSITarget \<br>
> params iqn="<a rel="nofollow" target="_blank" href="http://iqn.2012-11.com.example.sa/">iqn.2012-11.com</a>.example.san1:sda" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san1_0 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="<a target="_blank" href="http://iqn.2012-11.com.example.sa/">iqn.2012-11.com</a>.example.san1:sda" lun="0" path="/dev/drbd0" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san1_1 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san1:sda" lun="1" path="/dev/drbd1" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san1_2 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san1:sda" lun="2" path="/dev/drbd2" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san1_3 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san1:sda" lun="3" path="/dev/drbd3" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san2 ocf:heartbeat:iSCSITarget \<br>
> params iqn="iqn.2012-11.com.example.san2:sda" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san2_0 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san2:sda" lun="0" path="/dev/drbd1000" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san2_1 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san2:sda" lun="1" path="/dev/drbd1001" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san2_2 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san2:sda" lun="2" path="/dev/drbd1002" \<br>
> op monitor interval="10s"<br>
> primitive p_iSCSI-san2_3 ocf:heartbeat:iSCSILogicalUnit \<br>
> params target_iqn="iqn.2012-11.com.example.san2:sda" lun="3" path="/dev/drbd1003" \<br>
> op monitor interval="10s"<br>
> group g_iSCSI-san1 p_iSCSI-san1 p_iSCSI-san1_0 p_iSCSI-san1_1 p_iSCSI-san1_2 p_iSCSI-san1_3 p_IP-1_254<br>
> group g_iSCSI-san2 p_iSCSI-san2 p_iSCSI-san2_0 p_iSCSI-san2_1 p_iSCSI-san2_2 p_iSCSI-san2_3 p_IP-1_253<br>
> ms ms_DRBD-r0 p_DRBD-r0 \<br>
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"<br>
> ms ms_DRBD-r1 p_DRBD-r1 \<br>
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"<br>
> location l_iSCSI-san1_and_DRBD-r0 p_IP-1_254 10240: san1<br>
> location l_iSCSI-san2_and_DRBD-r1 p_IP-1_253 10240: san2<br>
> colocation c_iSCSI_with_DRBD-r0 inf: g_iSCSI-san1 ms_DRBD-r0:Master<br>
> colocation c_iSCSI_with_DRBD-r1 inf: g_iSCSI-san2 ms_DRBD-r1:Master<br>
> order o_DRBD-r0_before_iSCSI-san1 inf: ms_DRBD-r0:promote g_iSCSI-san1:start<br>
> order o_DRBD-r1_before_iSCSI-san2 inf: ms_DRBD-r1:promote g_iSCSI-san2:start<br>
> property $id="cib-bootstrap-options" \<br>
> dc-version="1.1.7-77eeb099a504ceda05d648ed161ef8b1582c7daf" \<br>
> cluster-infrastructure="openais" \<br>
> expected-quorum-votes="2" \<br>
> stonith-enabled="false" \<br>
> no-quorum-policy="ignore"</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">The cluster appears to be functioning correctly:</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;background:white;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in;">
<span style="font-size:10.0pt;color:black;">> san1:~ # crm_mon -1<br>
> ============<br>
> Last updated: Sun Jan 20 22:20:17 2013<br>
> Last change: Sun Jan 20 21:59:15 2013 by root via crm_attribute on san1<br>
> Stack: openais<br>
> Current DC: san1 - partition with quorum<br>
> Version: 1.1.7-77eeb099a504ceda05d648ed161ef8b1582c7daf<br>
> 2 Nodes configured, 2 expected votes<br>
> 16 Resources configured.<br>
> ============<br>
> <br>
> Online: [ san1 san2 ]<br>
> <br>
> Master/Slave Set: ms_DRBD-r0 [p_DRBD-r0]<br>
> Masters: [ san1 ]<br>
> Slaves: [ san2 ]<br>
> Resource Group: g_iSCSI-san1<br>
> p_iSCSI-san1 (ocf::heartbeat:iSCSITarget): Started san1<br>
> p_iSCSI-san1_0 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_1 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_2 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_3 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_IP-1_254 (ocf::heartbeat:IPaddr2): Started san1<br>
> Master/Slave Set: ms_DRBD-r1 [p_DRBD-r1]<br>
> Masters: [ san2 ]<br>
> Slaves: [ san1 ]<br>
> Resource Group: g_iSCSI-san2<br>
> p_iSCSI-san2 (ocf::heartbeat:iSCSITarget): Started san2<br>
> p_iSCSI-san2_0 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_1 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_2 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_3 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_IP-1_253 (ocf::heartbeat:IPaddr2): Started san2<br>
<br>
> san2:~ # crm_mon -1<br>
> ============<br>
> Last updated: Sun Jan 20 22:20:17 2013<br>
> Last change: Sun Jan 20 21:59:15 2013 by root via crm_attribute on san1<br>
> Stack: openais<br>
> Current DC: san1 - partition with quorum<br>
> Version: 1.1.7-77eeb099a504ceda05d648ed161ef8b1582c7daf<br>
> 2 Nodes configured, 2 expected votes<br>
> 16 Resources configured.<br>
> ============<br>
> <br>
> Online: [ san1 san2 ]<br>
> <br>
> Master/Slave Set: ms_DRBD-r0 [p_DRBD-r0]<br>
> Masters: [ san1 ]<br>
> Slaves: [ san2 ]<br>
> Resource Group: g_iSCSI-san1<br>
> p_iSCSI-san1 (ocf::heartbeat:iSCSITarget): Started san1<br>
> p_iSCSI-san1_0 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_1 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_2 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_iSCSI-san1_3 (ocf::heartbeat:iSCSILogicalUnit): Started san1<br>
> p_IP-1_254 (ocf::heartbeat:IPaddr2): Started san1<br>
> Master/Slave Set: ms_DRBD-r1 [p_DRBD-r1]<br>
> Masters: [ san2 ]<br>
> Slaves: [ san1 ]<br>
> Resource Group: g_iSCSI-san2<br>
> p_iSCSI-san2 (ocf::heartbeat:iSCSITarget): Started san2<br>
> p_iSCSI-san2_0 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_1 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_2 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_iSCSI-san2_3 (ocf::heartbeat:iSCSILogicalUnit): Started san2<br>
> p_IP-1_253 (ocf::heartbeat:IPaddr2): Started san2</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">However, the two DRBD resources do not appear to be communicating:</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">> san1:~ # cat /proc/drbd
<br>
> version: 8.4.1 (api:1/proto:86-100)<br>
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by phil@fat-tyre, 2011-12-20 12:43:15<br>
> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3259080<br>
> 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 2: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 3: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> <br>
> 1000: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1001: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1002: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1003: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----<br>
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in;">
<span style="font-size:10.0pt;color:black;">> san2:~ # cat /proc/drbd
<br>
> version: 8.4.1 (api:1/proto:86-100)<br>
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by phil@fat-tyre, 2011-12-20 12:43:15<br>
> 0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:140<br>
> 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 2: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 3: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> <br>
> 1000: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1001: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1002: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
> 1003: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----<br>
> ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">How can I begin to troubleshoot this error?</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">Eric Pretorious</span></div>
</div>
<div>
<div class="yiv1224273097MsoNormal" style="margin-left:.5in;"><span style="font-size:10.0pt;color:black;">Truckee, cA</span></div>
</div>
</div>
</div>
</div>
</div><br>_______________________________________________<br>drbd-user mailing list<br><a rel="nofollow" ymailto="mailto:drbd-user@lists.linbit.com" target="_blank" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br><a rel="nofollow" target="_blank" href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br><br><br> </div> </div> </blockquote></div> </div></div></div><br>_______________________________________________<br>drbd-user mailing list<br><a ymailto="mailto:drbd-user@lists.linbit.com" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br><a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br><br><br> </div> </div> </blockquote></div> </div></body></html>