<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>I've tried to update the kernel and DRBD 9 to the last available version, but nothing helped with the <span style="white-space: normal;">PingAck issue</span>. So I had to downgrade to DRBD 8.4 which is started to replicate fine except the following messages in the log: </div><div><br></div><pre><font size="1">[139267.930516] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.930529] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><pre><font size="1">[139267.933231] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.933241] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><pre><font size="1">[139267.934064] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.934075] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><pre><font size="1">[139267.934942] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.934950] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><pre><font size="1">[139267.936012] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.936019] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><pre><font size="1">[139267.936818] block drbd0: BAD! enr=34406392 rs_left=-4 rs_failed=0 count=4 cstate=SyncTarget</font></pre><pre><font size="1">[139267.936825] block drbd0: start offset (-2092958208) too large in drbd_bm_e_weight</font></pre><div></div><div><br></div><div>Any ideas how to resolve this problem?</div><div><br></div><div>-----Original Message-----</div><div><b>From</b>: Oleksiy Evin <<a href="mailto:Oleksiy%20Evin%20%3co.evin@onefc.com%3e">o.evin@onefc.com</a>></div><div><b>To</b>: <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a></div><div><b>Subject</b>: PingAck did not arrive in time.</div><div><b>Date</b>: Sun, 16 Jun 2019 22:37:08 +0800</div><div><br></div><!-- text/html --><div>Hi All,</div><div><br></div><div>Can anyone help me on the "PingAck did not arrive in time." repeated error while initial bitmap synchronization? First time it happened after I updated our cluster with the latest centos and drbd updates. I'm using the basic drbd configuration on 527TB LVM volume, replicated on 2 nodes with cross-over 100Gbps connection. The same connection is used for Pacemaker without any problems. I don't see any network adapter errors in logs, no reconnects or packets drop when the error happens for drbd. I've also tried another adapter with 10Gbps direct cable connection and got the same error.</div><div><br></div><pre><font size="1"><b># rpm -q centos-release</b></font></pre><pre><font size="1"><b><br></b></font></pre><pre><font size="1">centos-release-7-6.1810.2.el7.centos.x86_64</font></pre><pre><font size="1"><br></font></pre><pre><font size="1"><b># yum list installed | grep drbd</b></font></pre><pre><font size="1"><b><br></b></font></pre><pre><font size="1">drbd90-utils.x86_64 9.6.0-1.el7.elrepo @elrepo </font></pre><pre><font size="1">kmod-drbd90.x86_64 9.0.16-1.el7_6.elrepo @elrepo </font></pre><pre></pre><div><br></div><pre><font size="1"><b># ifconfig</b></font></pre><pre><font size="1"><br></font></pre><pre><font size="1">ens2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500</font></pre><pre><font size="1"> inet 172.16.1.1 netmask 255.255.255.255 broadcast 172.16.1.1</font></pre><pre><font size="1"> ether b8:83:03:67:3f:d4 txqueuelen 1000 (Ethernet)</font></pre><pre><font size="1"> RX packets 63547 bytes 11147564 (10.6 MiB)</font></pre><pre><font size="1"> RX errors 0 dropped 0 overruns 0 frame 0</font></pre><pre><font size="1"> TX packets 265307 bytes 33045583 (31.5 MiB)</font></pre><pre><font size="1"> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</font></pre><pre><font size="1"><br></font></pre><pre><font size="1"><br></font></pre><pre><font size="1">eno8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500</font></pre><pre><font size="1"> inet 172.20.1.1 netmask 255.255.0.0 broadcast 172.20.255.255</font></pre><pre><font size="1"> ether 20:67:7c:1c:42:c6 txqueuelen 1000 (Ethernet)</font></pre><pre><font size="1"> RX packets 484 bytes 49086 (47.9 KiB)</font></pre><pre><font size="1"> RX errors 0 dropped 0 overruns 0 frame 0</font></pre><pre><font size="1"> TX packets 504 bytes 56974 (55.6 KiB)</font></pre><pre><font size="1"> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</font></pre><pre><font size="1"> device interrupt 116 memory 0xe3000000-e37fffff </font></pre><div><br></div><pre><font size="1"><b># drbdadm dump all</b></font></pre><pre><font size="1"><b><br></b></font></pre><pre><font size="1"># /etc/drbd.conf</font></pre><pre><font size="1">global {</font></pre><pre><font size="1"> usage-count no;</font></pre><pre><font size="1">}</font></pre><pre><font size="1"><br></font></pre><pre><font size="1">common {</font></pre><pre><font size="1"> options {</font></pre><pre><font size="1"> auto-promote yes;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1"> net {</font></pre><pre><font size="1"> protocol C;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1">}</font></pre><pre><font size="1"><br></font></pre><pre><font size="1"># resource r0 on sgpplhan01: not ignored, not stacked</font></pre><pre><font size="1"># defined at /etc/drbd.d/r0.res:1</font></pre><pre><font size="1">resource r0 {</font></pre><pre><font size="1"> volume 0 {</font></pre><pre><font size="1"> device /dev/drbd0 minor 0;</font></pre><pre><font size="1"> disk /dev/storage/data;</font></pre><pre><font size="1"> meta-disk internal;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1"> on sgpplhan01 {</font></pre><pre><font size="1"> node-id 0;</font></pre><pre><font size="1"> address ipv4 172.16.1.1:7788;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1"> on sgpplhan02 {</font></pre><pre><font size="1"> node-id 1;</font></pre><pre><font size="1"> address ipv4 172.16.2.1:7788;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1"> net {</font></pre><pre><font size="1"> after-sb-0pri discard-zero-changes;</font></pre><pre><font size="1"> after-sb-1pri consensus;</font></pre><pre><font size="1"> after-sb-2pri disconnect;</font></pre><pre><font size="1"> }</font></pre><pre><font size="1">}</font></pre><div></div><div><br></div><pre><font size="1"><b># dmesg | grep drbd</b></font></pre><pre><font size="1"><b><br></b></font></pre><pre><font size="1">[37259.335235] drbd r0/0 drbd0 sgpplhan02: drbd_sync_handshake:</font></pre><pre><font size="1">[37259.335245] drbd r0/0 drbd0 sgpplhan02: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:141608532581 flags:24</font></pre><pre><font size="1">[37259.335254] drbd r0/0 drbd0 sgpplhan02: peer B7DA5A657F09CD92:45B54292B9CBC0CF:0000000000000000:0000000000000000 bits:141608532581 flags:20</font></pre><pre><font size="1">[37259.335260] drbd r0/0 drbd0 sgpplhan02: uuid_compare()=-3 by rule 20</font></pre><pre><font size="1">[37259.335265] drbd r0/0 drbd0 sgpplhan02: Writing the whole bitmap, full sync required after drbd_sync_handshake.</font></pre><pre><font size="1">[37265.754528] drbd r0/0 drbd0 sgpplhan02: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )</font></pre><pre><font size="1">[37265.754546] drbd r0/0 drbd0: Resumed AL updates</font></pre><pre><font size="1">[37279.780140] drbd r0 sgpplhan02: PingAck did not arrive in time.</font></pre><pre><font size="1">[37279.781303] drbd r0 sgpplhan02: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )</font></pre><pre><font size="1">[37279.781313] drbd r0/0 drbd0 sgpplhan02: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )</font></pre><pre><font size="1">[37279.781371] drbd r0 sgpplhan02: ack_receiver terminated</font></pre><pre><font size="1">[37279.781376] drbd r0 sgpplhan02: Terminating ack_recv thread</font></pre><pre><font size="1">[37279.833051] drbd r0 sgpplhan02: Connection closed</font></pre><pre><font size="1">[37279.833069] drbd r0 sgpplhan02: conn( NetworkFailure -> Unconnected )</font></pre><pre><font size="1">[37279.833086] drbd r0 sgpplhan02: Restarting receiver thread</font></pre><pre><font size="1">[37279.833098] drbd r0 sgpplhan02: conn( Unconnected -> Connecting )</font></pre><pre><font size="1">[37308.171618] drbd r0 sgpplhan02: Handshake to peer 1 successful: Agreed network protocol version 114</font></pre><pre><font size="1">[37308.171628] drbd r0 sgpplhan02: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.</font></pre><pre><font size="1">[37308.171666] drbd r0 sgpplhan02: Starting ack_recv thread (from drbd_r_r0 [28699])</font></pre><pre><font size="1">[37308.217846] drbd r0: Preparing cluster-wide state change 686534516 (0->1 499/146)</font></pre><pre><font size="1">[37308.218242] drbd r0: State change 686534516: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC</font></pre><pre><font size="1">[37308.218253] drbd r0: Committing cluster-wide state change 686534516 (0ms)</font></pre><pre><font size="1">[37308.218296] drbd r0 sgpplhan02: conn( Connecting -> Connected ) peer( Unknown -> Primary )</font></pre><pre><font size="1">[37308.222753] drbd r0/0 drbd0 sgpplhan02: drbd_sync_handshake:</font></pre><pre><font size="1">[37308.222763] drbd r0/0 drbd0 sgpplhan02: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:141608532581 flags:124</font></pre><pre><font size="1">[37308.222771] drbd r0/0 drbd0 sgpplhan02: peer B7DA5A657F09CD92:45B54292B9CBC0CF:0000000000000000:0000000000000000 bits:141608532581 flags:120</font></pre><pre><font size="1">[37308.222777] drbd r0/0 drbd0 sgpplhan02: uuid_compare()=-3 by rule 20</font></pre><pre><font size="1">[37308.222782] drbd r0/0 drbd0 sgpplhan02: Writing the whole bitmap, full sync required after drbd_sync_handshake.</font></pre><pre><font size="1">[37314.890717] drbd r0/0 drbd0 sgpplhan02: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )</font></pre><pre><font size="1">[37328.669598] drbd r0 sgpplhan02: PingAck did not arrive in time.</font></pre><pre><font size="1">[37328.670759] drbd r0 sgpplhan02: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )</font></pre><pre><font size="1">[37328.670770] drbd r0/0 drbd0 sgpplhan02: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )</font></pre><pre><font size="1">[37328.670823] drbd r0 sgpplhan02: ack_receiver terminated</font></pre><pre><font size="1">[37328.670828] drbd r0 sgpplhan02: Terminating ack_recv thread</font></pre><pre><font size="1">[37328.718096] drbd r0 sgpplhan02: Connection closed</font></pre><pre><font size="1">[37328.718112] drbd r0 sgpplhan02: conn( NetworkFailure -> Unconnected )</font></pre><pre><font size="1">[37328.718127] drbd r0 sgpplhan02: Restarting receiver thread</font></pre><pre><font size="1">[37328.718138] drbd r0 sgpplhan02: conn( Unconnected -> Connecting )</font></pre><pre><font size="1">[37351.755553] drbd r0 sgpplhan02: conn( Connecting -> Disconnecting )</font></pre><pre><font size="1">[37351.794081] drbd r0 sgpplhan02: Connection closed</font></pre><pre><font size="1"><br></font></pre><pre><br></pre><div><br></div><pre></pre><div></div>
</body></html>