<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>i don't get it synced again.</div><div>they are now both stand alone?</div><div>i can ping them both.</div><div><br></div><div>don't have any options left.</div><div><br></div><div><div>[root@<b>kvmstorage1</b> drbd.d]# cat /proc/drbd </div><div>version: 8.3.12 (api:88/proto:86-96)</div><div>GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by phil@Build64R6, 2012-04-08 09:36:52</div><div> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----</div><div> ns:0 nr:0 dw:412 dr:9926 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:280</div></div><div><br></div><div><div>[root@<b>kvmstorage2</b> drbd.d]# cat /proc/drbd </div><div>version: 8.3.12 (api:88/proto:86-96)</div><div>GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by phil@Build64R6, 2012-04-08 09:36:52</div><div> 0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----</div><div> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:264</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>/var/log/messages on 2 servers</div><div><br></div><div><b>[root@kvmstorage2</b> drbd.d]# service drbd restart</div><div>Stopping all DRBD resources: May 13 15:14:13 kvmstorage2 kernel: block drbd0: disk( UpToDate -> Failed ) </div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: disk( Failed -> Diskless ) </div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: drbd_bm_resize called with capacity == 0</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: worker terminated</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: Terminating worker thread</div><div>May 13 15:14:13 kvmstorage2 kernel: drbd: module cleanup done.</div><div>.</div><div>Starting DRBD resources: May 13 15:14:13 kvmstorage2 kernel: drbd: initialized. Version: 8.3.12 (api:88/proto:86-96)</div><div>May 13 15:14:13 kvmstorage2 kernel: drbd: GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by phil@Build64R6, 2012-04-08 09:36:52</div><div>May 13 15:14:13 kvmstorage2 kernel: drbd: registered as block device major 147</div><div>May 13 15:14:13 kvmstorage2 kernel: drbd: minor_table @ 0xffff88020f7257c0</div><div>[ d(main) May 13 15:14:13 kvmstorage2 kernel: block drbd0: Starting worker thread (from cqueue [1344])</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: disk( Diskless -> Attaching ) </div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: Found 6 transactions (34 active extents) in activity log.</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: Method to ensure write ordering: barrier</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: max BIO size = 131072</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: drbd_bm_resize called with capacity == 6920386232</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: resync bitmap: bits=865048279 words=13516380 pages=26400</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: size = 3300 GB (3460193116 KB)</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: bitmap READ of 26400 pages took 198 jiffies</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: recounting of set bits took additional 90 jiffies</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: 264 KB (66 bits) marked out-of-sync by on disk bit-map.</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: disk( Attaching -> UpToDate ) </div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: attached to UUIDs C12A485E56F51104:9555562D91EACAC2:A615ADBD6A39BD99:A614ADBD6A39BD99</div><div>n(main) May 13 15:14:13 kvmstorage2 kernel: block drbd0: conn( StandAlone -> Unconnected ) </div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: Starting receiver thread (from drbd0_worker [6484])</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: receiver (re)started</div><div>May 13 15:14:13 kvmstorage2 kernel: block drbd0: conn( Unconnected -> WFConnection ) </div><div>]May 13 15:14:14 kvmstorage2 kernel: block drbd0: Handshake successful: Agreed network protocol version 96</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: conn( WFConnection -> WFReportParams ) </div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: Starting asender thread (from drbd0_receiver [6494])</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: data-integrity-alg: <not-used></div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: drbd_sync_handshake:</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: self C12A485E56F51104:9555562D91EACAC2:A615ADBD6A39BD99:A614ADBD6A39BD99 bits:66 flags:0</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: peer E33CEADD1FF28EE1:9555562D91EACAC3:A615ADBD6A39BD98:A614ADBD6A39BD99 bits:70 flags:0</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: uuid_compare()=100 by rule 90</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: meta connection shut down by peer.</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: conn( WFReportParams -> Disconnecting ) </div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: error receiving ReportState, l: 4!</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: asender terminated</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: Terminating asender thread</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: Connection closed</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: conn( Disconnecting -> StandAlone ) </div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: receiver terminated</div><div>May 13 15:14:14 kvmstorage2 kernel: block drbd0: Terminating receiver thread</div><div><br></div><div><br></div><div><br></div><div>second server (primary right now)</div><div><br></div><div><div><b>root@kvmstorage1</b> drbd.d]# service drbd restart</div><div>Stopping all DRBD resources: umount: /datastore: device is busy.</div><div> (In some cases useful info about processes that use</div><div> the device is found by lsof(8) or fuser(1))</div><div>/dev/drbd0: State change failed: (-12) Device is held open by someone</div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: State change failed: Device is held open by someone</div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: state = { cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----- }</div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: wanted = { cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r----- }</div><div>ERROR: Module drbd is in use</div><div>.</div><div>Starting DRBD resources: [ n(main) May 13 15:16:22 kvmstorage1 kernel: block drbd0: conn( StandAlone -> Unconnected ) </div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: Starting receiver thread (from drbd0_worker [1441])</div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: receiver (re)started</div><div>May 13 15:16:22 kvmstorage1 kernel: block drbd0: conn( Unconnected -> WFConnection ) </div><div>]..........</div><div>***************************************************************</div><div> DRBD's startup script waits for the peer node(s) to appear.</div><div> - In case this node was already a degraded cluster before the</div><div> reboot the timeout is 0 seconds. [degr-wfc-timeout]</div><div> - If the peer was available before the reboot the timeout will</div><div> expire after 0 seconds. [wfc-timeout]</div><div> (These values are for resource 'drbd'; 0 sec -> wait forever) </div><div><b><font class="Apple-style-span" color="#811208">(i had to restart drbd on the second node)</font></b></div><div> To abort waiting enter 'yes' [ 54]:May 13 15:17:16 kvmstorage1 kernel: block drbd0: Handshake successful: Agreed network protocol version 96</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: conn( WFConnection -> WFReportParams ) </div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: Starting asender thread (from drbd0_receiver [7458])</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: data-integrity-alg: <not-used></div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: drbd_sync_handshake:</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: self E33CEADD1FF28EE1:9555562D91EACAC3:A615ADBD6A39BD98:A614ADBD6A39BD99 bits:70 flags:0</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: peer C12A485E56F51104:9555562D91EACAC2:A615ADBD6A39BD99:A614ADBD6A39BD99 bits:66 flags:0</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: uuid_compare()=100 by rule 90</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0</div><div><br></div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: conn( WFReportParams -> Disconnecting ) </div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: error receiving ReportState, l: 4!</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: asender terminated</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: Terminating asender thread</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: Connection closed</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: conn( Disconnecting -> StandAlone ) </div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: receiver terminated</div><div>May 13 15:17:16 kvmstorage1 kernel: block drbd0: Terminating receiver thread</div></div><div><br></div><div><br></div></body></html>