<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="GENERATOR" content="MSHTML 8.00.6001.19298">
</head>
<body><!--ppd1000037-->
<div><span class="657382309-28052013"><font face="Calibri">According to the DRBD User Guide, after doing an online verify it is necessary to trigger a resync by disconnecting and reconnecting the resource (</font></span><span class="657382309-28052013"><font face="Calibri"><a href="http://www.drbd.org/users-guide/s-use-online-verify.html">http://www.drbd.org/users-guide/s-use-online-verify.html</a>).</font></span></div>
<div><span class="657382309-28052013"><font face="Calibri"></font></span> </div>
<div><span class="657382309-28052013"><font face="Calibri">However, we conducted a verify yesterday after a RAID array failure event on the standby node and it triggered a resync automatically. Is this normal?</font></span></div>
<div><span class="657382309-28052013"><font face="Calibri"></font></span> </div>
<div><span class="657382309-28052013"><font face="Calibri">During the resync, several troubling log messages were generated, starting at
<font size="2" face="Courier New">23:06:17</font>. It looks like the verify failed to complete and a resync was automatically triggered. Can the resync be trusted, or is another online verify required?</font></span></div>
<div><span class="657382309-28052013"><font face="Calibri"></font></span> </div>
<div><span class="657382309-28052013"><font face="Calibri">-- log snippet from primary (non-failed) node --</font></span></div>
<div><span class="657382309-28052013"><font face="Calibri"></font></span> </div>
<div><span class="657382309-28052013"><font size="2" face="Courier New">May 27 18:32:54 ha10a kernel: block drbd0: conn( Connected -> VerifyS )<br>
May 27 18:32:54 ha10a kernel: block drbd0: Starting Online Verify from sector 0<br>
May 27 18:46:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 19:01:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 19:06:10 ha10a kernel: block drbd0: Out of sync: start=242225192, size=8 (sectors)<br>
May 27 19:06:10 ha10a kernel: block drbd0: Out of sync: start=242225200, size=8 (sectors)<br>
May 27 19:09:31 ha10a kernel: block drbd0: Out of sync: start=267001248, size=8 (sectors)<br>
May 27 19:09:59 ha10a kernel: block drbd0: Out of sync: start=270432752, size=16 (sectors)<br>
May 27 19:12:01 ha10a kernel: block drbd0: Out of sync: start=285172888, size=8 (sectors)<br>
May 27 19:12:01 ha10a kernel: block drbd0: Out of sync: start=285172920, size=8 (sectors)<br>
May 27 19:12:01 ha10a kernel: block drbd0: Out of sync: start=285172744, size=48 (sectors)<br>
May 27 19:12:01 ha10a kernel: block drbd0: Out of sync: start=285172792, size=8 (sectors)<br>
</font></span></div>
<div><span class="657382309-28052013"><font size="2" face="Courier New"><more of these removed></font></span></div>
<div><span class="657382309-28052013"><font size="2" face="Courier New"><font size="3" face="Calibri"></font> </div>
</font></span>
<div><span class="657382309-28052013"><font size="2" face="Courier New">May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295106080, size=8 (sectors)<br>
May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295106192, size=8 (sectors)<br>
May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295106208, size=8 (sectors)<br>
May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295106216, size=16 (sectors)<br>
May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295114664, size=24 (sectors)<br>
May 27 19:13:24 ha10a kernel: block drbd0: Out of sync: start=295115456, size=8 (sectors)<br>
May 27 19:13:26 ha10a kernel: block drbd0: Out of sync: start=295299816, size=24 (sectors)<br>
May 27 19:16:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 19:31:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 19:46:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 20:01:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 20:16:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 20:31:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 20:46:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 21:01:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 21:16:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 21:31:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 21:46:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 22:01:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 22:16:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 22:31:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 22:46:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:01:38 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:05:41 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 6<br>
May 27 23:05:47 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 5<br>
May 27 23:05:53 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 4<br>
May 27 23:05:59 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 3<br>
May 27 23:06:05 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 2<br>
May 27 23:06:11 ha10a kernel: d-con ha01_mysql: [drbd_w_ha01_mys/4088] sock_sendmsg time expired, ko = 1<br>
May 27 23:06:17 ha10a kernel: block drbd0: Remote failed to finish a request within ko-count * timeout<br>
May 27 23:06:17 ha10a kernel: block drbd0: peer( Secondary -> Unknown ) conn( VerifyS -> Timeout ) pdsk( UpToDate -> DUnknown )<br>
May 27 23:06:17 ha10a kernel: block drbd0: Online Verify reached sector 1980019736<br>
May 27 23:06:17 ha10a kernel: d-con ha01_mysql: Terminating drbd_a_ha01_mys<br>
May 27 23:06:23 ha10a kernel: block drbd0: md_sync_timer expired! Worker calls drbd_md_sync().<br>
May 27 23:06:23 ha10a kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
May 27 23:06:23 ha10a kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
May 27 23:06:23 ha10a kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
May 27 23:06:23 ha10a kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
May 27 23:06:23 ha10a kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
May 27 23:06:23 ha10a kernel: block drbd0: new current UUID 9E6336C074323FD7:F8B5F4D0D7E2DF75:A4E3656E5427C691:A4E2656E5427C691<br>
May 27 23:06:23 ha10a kernel: d-con ha01_mysql: helper command: /sbin/drbdadm fence-peer ha01_mysql<br>
May 27 23:06:24 ha10a cibadmin[1176]: notice: crm_log_args: Invoked: cibadmin -C -o constraints -X <rsc_location rsc="ms_drbd0" id="drbd-fence-by-handler-ha01_mysql-ms_drbd0">#012 <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-ha01_mysql-rule-ms_drbd0">#012
<expression attribute="#uname" operation="ne" value="ha10a" id="drbd-fence-by-handler-ha01_mysql-expr-ms_drbd0"/>#012 </rule>#012</rsc_location><br>
May 27 23:06:24 ha10a crm-fence-peer.sh[1132]: INFO peer is reachable, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-ha01_mysql-ms_drbd0'<br>
May 27 23:06:24 ha10a kernel: d-con ha01_mysql: helper command: /sbin/drbdadm fence-peer ha01_mysql exit code 4 (0x400)<br>
May 27 23:06:24 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:06:26 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:06:27 ha10a kernel: d-con ha01_mysql: Starting asender thread (from drbd_r_ha01_mys [4132])<br>
May 27 23:06:27 ha10a kernel: block drbd0: drbd_sync_handshake:<br>
May 27 23:06:27 ha10a kernel: block drbd0: self 9E6336C074323FD7:F8B5F4D0D7E2DF75:A4E3656E5427C691:A4E2656E5427C691 bits:3182 flags:0<br>
May 27 23:06:27 ha10a kernel: block drbd0: peer F8B5F4D0D7E2DF74:0000000000000000:A4E3656E5427C690:A4E2656E5427C691 bits:1581 flags:0<br>
May 27 23:06:27 ha10a kernel: block drbd0: uuid_compare()=1 by rule 70<br>
May 27 23:06:27 ha10a kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )<br>
May 27 23:06:27 ha10a kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 330(1), total 330; compression: 100.0%<br>
May 27 23:06:27 ha10a kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 330(1), total 330; compression: 100.0%<br>
May 27 23:06:27 ha10a kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0<br>
May 27 23:06:27 ha10a kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)<br>
May 27 23:06:27 ha10a kernel: block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )<br>
May 27 23:06:27 ha10a kernel: block drbd0: Began resync as SyncSource (will sync 12728 KB [3182 bits set]).<br>
May 27 23:06:27 ha10a kernel: block drbd0: updated sync UUID 9E6336C074323FD7:F8B6F4D0D7E2DF75:F8B5F4D0D7E2DF75:A4E3656E5427C691<br>
May 27 23:06:27 ha10a kernel: block drbd0: Resync done (total 1 sec; paused 0 sec; 12728 K/sec)<br>
May 27 23:06:27 ha10a kernel: block drbd0: updated UUIDs 9E6336C074323FD7:0000000000000000:F8B6F4D0D7E2DF75:F8B5F4D0D7E2DF75<br>
May 27 23:06:27 ha10a kernel: block drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<br>
May 27 23:06:27 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:06:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:21:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:36:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 27 23:51:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 00:06:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 00:21:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 00:36:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 00:51:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 01:06:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 01:21:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 01:36:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 01:51:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
May 28 02:06:57 ha10a pengine[17218]: notice: unpack_rsc_op: Operation monitor found resource p_drbd0:0 active in master mode on ha10a<br>
[root@ha10a ~]# cat /proc/drbd<br>
version: 8.4.3 (api:1/proto:86-101)<br>
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by </font><a href="mailto:root@ha10a.mycharts.md"><font size="2" face="Courier New">root@ha10a.mycharts.md</font></a><font size="2" face="Courier New">, 2013-04-25 16:27:09<br>
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br>
ns:72109544 nr:168 dw:74947448 dr:1004787613 al:7721 bm:371 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br>
ns:4104 nr:28 dw:40 dr:1073715946 al:1 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
</div>
</font></span>
<div><font face="Calibri"></font> </div>
<p style="MARGIN: 0in 0in 0pt" class="MsoNormal" align="left"><span style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt"><font size="3" face="Calibri">--<br>
</font></span><span style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt"><font size="3" face="Calibri">Eric Robinson<br>
</font></span><span style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt"><font size="3" face="Calibri">Director of Information Technology<br>
</font></span><font size="3"><font face="Calibri"><span style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt">Physician Select Management, LLC<br>
</span><span style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt">775.885.2211 x 111</span></font></font></p>
<div><font face="Calibri"></font> </div>
<div><font face="Calibri"></font> </div>
<div><font face="Calibri"></font> </div>
<BR><BR><FONT color=silver><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana"><FONT size=1>Disclaimer - May 28, 2013</FONT></FONT></SPAN></FONT> <P></P><FONT size=1><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">This email and any files <FONT size=1>transmitted</FONT> with it are confidential and intended solely for <STRONG>'drbd-user@lists.linbit.com'</STRONG></SPAN></FONT><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.</SPAN></FONT></FONT></body>
</html>