Hi all,<div><br></div><div>I am in a very wired situation where I have four DRBD devices setup as primary/primary configuration. I do have a customized upper layer that takes care of the failover related issue(similar to Pacemaker).</div>
<div><br></div><div>In my test environment, <span class="Apple-style-span" style="background-color: rgb(153, 102, 51);">first time</span> after OS and DRBD binaries get installed, I initiate some IOs from one my server, reboot the machine after 4 or 5 mins while IO is still running. During the machine reboots, All IO fails over to the partner server without any problem. When the rebooted machine comes back, split brain happens <span class="Apple-style-span" style="background-color: rgb(204, 153, 51);">only </span>to the disk that has IO going on. But If repeat the above experiment again after split brain been manually resolved, DRBD somehow figures out the sync direction correctly according to my configuration. Can someone help me interpret the syslog and give me a hint on why split brain happens only for the first time but not after its been resolved manually.</div>
<div><br></div><div>Here is my drbd.conf file:</div><div><div><font class="Apple-style-span" color="#006600">common {</font></div><div><font class="Apple-style-span" color="#006600"> protocol C;</font></div><div><font class="Apple-style-span" color="#006600">}</font></div>
<div><font class="Apple-style-span" color="#006600"><br></font></div><div><font class="Apple-style-span" color="#006600">#=#= 1</font></div><div><font class="Apple-style-span" color="#006600">resource drbd1 {</font></div>
<div><font class="Apple-style-span" color="#006600"> on f33 {</font></div><div><font class="Apple-style-span" color="#006600"> device /dev/drbd1;</font></div><div><font class="Apple-style-span" color="#006600"> #meta sd /dev/sdb</font></div>
<div><font class="Apple-style-span" color="#006600"> disk /dev/disk/by-id/scsi-360030480003ae2e0151e54b20c1f82e0;</font></div><div><font class="Apple-style-span" color="#006600"> address <a href="http://192.168.250.1:7790">192.168.250.1:7790</a>;</font></div>
<div><font class="Apple-style-span" color="#006600"> meta-disk internal;</font></div><div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600"> on f34 {</font></div>
<div><font class="Apple-style-span" color="#006600"> device /dev/drbd1;</font></div><div><font class="Apple-style-span" color="#006600"> #meta sd /dev/sdb</font></div><div><font class="Apple-style-span" color="#006600"> disk /dev/disk/by-id/scsi-360030480003ae32015095d1f11bf902b;</font></div>
<div><font class="Apple-style-span" color="#006600"> address <a href="http://192.168.250.2:7790">192.168.250.2:7790</a>;</font></div><div><font class="Apple-style-span" color="#006600"> meta-disk internal;</font></div>
<div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600"> net {</font></div><div><font class="Apple-style-span" color="#006600"> allow-two-primaries;</font></div>
<div><span class="Apple-style-span" style="background-color: rgb(255, 204, 51); "><font class="Apple-style-span" color="#006600"> after-sb-0pri discard-least-changes;</font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 204, 51);"><font class="Apple-style-span" color="#006600"> after-sb-1pri consensus;</font></span></div>
<div><span class="Apple-style-span" style="background-color: rgb(255, 204, 51);"><font class="Apple-style-span" color="#006600"> after-sb-2pri violently-as0p;</font></span></div><div><font class="Apple-style-span" color="#006600"> rr-conflict violently;</font></div>
<div><font class="Apple-style-span" color="#006600"> max-buffers 8000;</font></div><div><font class="Apple-style-span" color="#006600"> max-epoch-size 8000;</font></div><div><font class="Apple-style-span" color="#006600"> unplug-watermark 16;</font></div>
<div><font class="Apple-style-span" color="#006600"> sndbuf-size 0;</font></div><div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600"> syncer {</font></div>
<div><font class="Apple-style-span" color="#006600"> rate 300M;</font></div><div><font class="Apple-style-span" color="#006600"> verify-alg crc32c;</font></div><div><font class="Apple-style-span" color="#006600"> al-extents 3800;</font></div>
<div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600"> startup {</font></div><div><font class="Apple-style-span" color="#006600"> <span class="Apple-style-span" style="background-color: rgb(255, 204, 51);"> become-primary-on both;</span></font></div>
<div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600"> handlers {</font></div><div><font class="Apple-style-span" color="#006600"> before-resync-target "/sbin/before_resync_target.sh";</font></div>
<div><font class="Apple-style-span" color="#006600"> after-resync-target "/sbin/after_resync_target.sh";</font></div><div><font class="Apple-style-span" color="#006600"> }</font></div><div><font class="Apple-style-span" color="#006600">}</font></div>
</div><div><font class="Apple-style-span" color="#006600"># some other drbd resource</font></div><div><br></div><div><div>Apr 11 16:16:07 f33 kernel: block drbd1: Starting worker thread (from cqueue/9 [356])</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: disk( Diskless -> Attaching )</div>
<div>Apr 11 16:16:07 f33 kernel: block drbd1: Found 39 transactions (39 active extents) in activity log.</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: Method to ensure write ordering: barrier</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: max_segment_size ( = BIO size ) = 65536</div>
<div>Apr 11 16:16:07 f33 kernel: block drbd1: drbd_bm_resize called with capacity == 25164984</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: resync bitmap: bits=3145623 words=49151</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: size = 12 GB (12582492 KB)</div>
<div>Apr 11 16:16:07 f33 kernel: block drbd1: recounting of set bits took additional 0 jiffies</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>Apr 11 16:16:07 f33 kernel: block drbd1: Marked additional 156 MB as out-of-sync based on AL.</div>
<div>Apr 11 16:16:07 f33 kernel: block drbd1: disk( Attaching -> UpToDate )</div></div><div><div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( StandAlone -> Unconnected )</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: Starting receiver thread (from drbd1_worker [4269])</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: receiver (re)started</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( Unconnected -> WFConnection )</div></div><div><div>Apr 11 16:16:08 f33 kernel: block drbd1: Handshake successful: Agreed network protocol version 95</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( WFConnection -> WFReportParams )</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: Starting asender thread (from drbd1_receiver [4359])</div></div><div>Apr 11 16:16:08 f33 kernel: block drbd1: data-integrity-alg: <not-used></div>
<div><div>Apr 11 16:16:08 f33 kernel: block drbd1: max_segment_size ( = BIO size ) = 65536</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: drbd_sync_handshake:</div><div><span class="Apple-style-span" style="background-color: rgb(204, 153, 51);">Apr 11 16:16:08 f33 kernel: block drbd1: self 264D92621CE57A74:CBC54463A29032C9:72FCD7718269F032:0000000000000004 bits:39936 flags:0</span></div>
<div><span class="Apple-style-span" style="background-color: rgb(204, 153, 51);">Apr 11 16:16:08 f33 kernel: block drbd1: peer 1CB5F82C62D88D81:CBC54463A29032C9:72FCD7718269F033:0000000000000004 bits:1 flags:0</span></div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: uuid_compare()=100 by rule 90</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1</div></div><div><div>Apr 11 16:16:08 f33 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0)</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: <span class="Apple-style-span" style="background-color: rgb(204, 153, 51);">Split-Brain detected but unresolved, dropping connection!</span></div><div>Apr 11 16:16:08 f33 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: meta connection shut down by peer.</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( WFReportParams -> NetworkFailure )</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: asender terminated</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: Terminating asender thread</div></div><div><div>Apr 11 16:16:08 f33 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 exit code 0 (0x0)</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( NetworkFailure -> Disconnecting )</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: error receiving ReportState, l: 4!</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: Connection closed</div></div><div><div>Apr 11 16:16:08 f33 kernel: block drbd1: conn( Disconnecting -> StandAlone )</div>
<div>Apr 11 16:16:08 f33 kernel: block drbd1: receiver terminated</div><div>Apr 11 16:16:08 f33 kernel: block drbd1: Terminating receiver thread</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>
<br></div><div>Here is the related log from syslog:</div><div><br></div><div> <br><div>Commit yourself to constant self-improvement<br>
</div></div>