<div dir="ltr">I confirmed this as well, details below...<div><br></div><div>---------------------------------------------------</div><div>centos 6.10</div><div>kernel 2.6.32-754.el6.x86_64</div><div>drbd84-utils-9.5.0-1.el6.elrepo.x86_64<br>kmod-drbd84-8.4.11-1.el6_10.elrepo.x86_64<br></div><div>----------------------------------------------------</div><div><br></div><div>----------------------------------------------------</div><div>centos 7.8.2003<br></div><div>kernel 3.10.0-1127.8.2.el7.x86_64</div><div>drbd90-utils-9.12.2-1.el7.elrepo.x86_64<br>kmod-drbd90-9.0.22-2.el7_8.elrepo.x86_64<br></div><div>-----------------------------------------------------</div><div><br></div><div>-----------------------------------------------------</div><div>centos 8.1.1911</div><div>kernel 4.18.0-147.8.1.el8_1.x86_64</div><div>kmod-drbd90-9.0.21-2.el8_1.elrepo.x86_64<br>drbd90-utils-9.10.0-2.el8.elrepo.x86_64<br></div><div>-------------------------------------------------------</div><div><br></div><div>Combinations:</div><div><br></div><div>centos6 <-> centos7 [OK]</div><div>centos7 <-> centos8 [OK]</div><div>centos6 <-> centos8 [NOT_OK] Error reported: "error receiving ReportState, e: -5 l: 0!"</div><div><br></div><div>Migrating from centos6 to centos7 first and then after from centos7 to centos8 seems to be the only option for now.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 25 May 2020 at 06:29, Rupa Schomaker <<a href="mailto:rupa@rupa.com">rupa@rupa.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I have a pair of servers that are running current centos 6 (kernel versions below) which I'm working on upgrading to centos 8. I would like to run for a period of time with one node running centos 6 and one node running centos 8 so that I can failback to centos 6 in the case we have unknown application issues.<div><div><br></div><div>In my testbed, I have a centos 6 server that has a drbd volume that was originally synced with another centos 6 server. I then connected that to a centos 7 server which ran w/out problem, but when I try to connect to a centos 8 server, I get a protocol error.</div><div><br></div><div>Below are configs/logs. Any suggestions?</div><div><br></div><div>node 0: c61<br>uname: Linux centos6-test-1.novalocal 2.6.32-754.29.1.el6.x86_64 #1 SMP Mon Apr 27 15:30:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux<br>kernel: kernel-2.6.32-754.29.1.el6.x86_64<br>kmod: kmod-drbd84-8.4.11-1.el6_10.elrepo.x86_64<br><br>[root@centos6-test-1 schomarx]# drbdadm dump<br># /etc/drbd.conf<br>global {<br> usage-count yes;<br> cmd-timeout-medium 600;<br> cmd-timeout-long 0;<br>}<br><br>common {<br>}<br><br># resource gitlab on centos6-test-1.novalocal: not ignored, not stacked<br># defined at /etc/drbd.d/gitlab.res:1<br>resource gitlab {<br> on centos6-test-1.novalocal {<br> device /dev/drbd0 minor 0;<br> disk /dev/vdb;<br> meta-disk internal;<br> address ipv4 <a href="http://192.168.0.25:7788/" rel="noreferrer" target="_blank">192.168.0.25:7788</a>;<br> }<br> on centos8-test-2.novalocal {<br> device /dev/drbd0 minor 0;<br> disk /dev/vdb;<br> meta-disk internal;<br> address ipv4 <a href="http://192.168.0.22:7788/" rel="noreferrer" target="_blank">192.168.0.22:7788</a>;<br> }<br> net {<br> protocol C;<br> verify-alg sha256;<br> }<br>}<br><br>node1: c82<br>uanme: Linux centos8-test-2.novalocal 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux<br>kernel: kernel-4.18.0-147.8.1.el8_1.x86_64<br>kmod: kmod-25-13.el8.x86_64<br><br>[root@centos8-test-2 schomarx]# drbdadm dump<br># /etc/drbd.conf<br>global {<br> usage-count no;<br>}<br><br>common {<br> net {<br> protocol C;<br> }<br>}<br><br># resource gitlab on centos8-test-2.novalocal: not ignored, not stacked<br># defined at /etc/drbd.d/gitlab.res:1<br>resource gitlab {<br> device /dev/drbd0 minor 0;<br> on centos6-test-1.novalocal {<br> node-id 1;<br> disk /dev/vdb;<br> address ipv4 <a href="http://192.168.0.25:7788/" rel="noreferrer" target="_blank">192.168.0.25:7788</a>;<br> }<br> on centos8-test-2.novalocal {<br> node-id 0;<br> disk /dev/vdc;<br> address ipv4 <a href="http://192.168.0.22:7788/" rel="noreferrer" target="_blank">192.168.0.22:7788</a>;<br> }<br> net {<br> verify-alg sha256;<br> }<br>}<br><br>====<br><br>On centos8-test-2, I have /dev/vdb and /dev/vdc. /dev/vdb is the working backing store that was originally on centos 6, worked on centos 7<br>using drbd90 and then moved to centos 8. /dev/vdc is a brand new metadata 90 backing store. Neither are able to establish a connection<br>without a protocol error.<br><br>I've pasted the logs from each type.<br><br>For /dev/vdb which has data from a prior sync:<br><br>=== centos6-test-1 ===<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe -> Unconnected )<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver terminated<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Restarting receiver thread<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver (re)started<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( Unconnected -> WFConnection )<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Handshake successful: Agreed network protocol version 101<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( WFConnection -> WFReportParams )<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Starting ack_recv thread (from drbd_r_gitlab [10505])<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: self 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004 bits:262127 flags:0<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer 449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0 flags:0<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: uuid_compare()=1 by rule 70<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )<br>May 20 13:23:47 centos6-test-1 kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: sock was shut down by peer<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: peer( Secondary -> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( Consistent -> DUnknown )<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: short read (expected size 16)<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: meta connection shut down by peer.<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Terminating drbd_a_gitlab<br>May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Connection closed<br><br>=== centos8-test-2 ===<br><br>May 20 09:23:52 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )<br>May 20 09:23:52 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Restarting receiver thread<br>May 20 09:23:52 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Unconnected -> Connecting )<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network protocol version 101<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab [6057])<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Connecting -> Connected )<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: My current UUID/flags changed during handshake. Retry connecting.<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: drbd_sync_handshake:<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: self 449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0 flags:120<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: peer 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004 bits:262127 flags:20<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: My current UUID/flags changed during handshake. Retry connecting.<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Connected -> ProtocolError )<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: ack_receiver terminated<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Terminating ack_recv thread<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Aborting remote state change 0 commit not possible<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Restarting sender thread<br>May 20 09:23:53 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Connection closed<br><br>===================<br><br>Now if I try with /dev/vdc which is a fresh invalidated device:<br><br>=== centos6-test-1 ===<br>May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: receiver (re)started<br>May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: conn( Unconnected -> WFConnection )<br>May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Handshake successful: Agreed network protocol version 101<br>May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( WFConnection -> WFReportParams )<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Starting ack_recv thread (from drbd_r_gitlab [10505])<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: self 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004 bits:262127 flags:0<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:4<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: uuid_compare()=2 by rule 30<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: Becoming sync source due to disk states.<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: Writing the whole bitmap, full sync required after drbd_sync_handshake.<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: 1024 MB (262127 bits) marked out-of-sync by on disk bit-map.<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )<br>May 20 13:29:39 centos6-test-1 kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: sock was shut down by peer<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: peer( Secondary -> Unknown ) conn( WFBitMapS -> BrokenPipe )<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: short read (expected size 16)<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: meta connection shut down by peer.<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Terminating drbd_a_gitlab<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Connection closed<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe -> Unconnected )<br>May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: receiver terminated<br><br>=== centos8-test-2 ===<br>May 20 09:29:44 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Restarting receiver thread<br>May 20 09:29:44 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Unconnected -> Connecting )<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network protocol version 101<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab [6471])<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Connecting -> Connected )<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: drbd_sync_handshake:<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:124<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: peer 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004 bits:262127 flags:20<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: My current UUID/flags changed during handshake. Retry connecting.<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0 centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( Connected -> ProtocolError )<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: ack_receiver terminated<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Terminating ack_recv thread<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Aborting remote state change 0 commit not possible<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Restarting sender thread<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: Connection closed<br>May 20 09:29:45 centos8-test-2 kernel: drbd gitlab centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )<br></div><div><br></div><div>-- <br><div dir="ltr">-Rupa</div></div></div></div>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>