[DRBD-user] cross version sync failing
Yannis Milios
yannis.milios at gmail.com
Tue May 26 12:41:38 CEST 2020
I confirmed this as well, details below...
---------------------------------------------------
centos 6.10
kernel 2.6.32-754.el6.x86_64
drbd84-utils-9.5.0-1.el6.elrepo.x86_64
kmod-drbd84-8.4.11-1.el6_10.elrepo.x86_64
----------------------------------------------------
----------------------------------------------------
centos 7.8.2003
kernel 3.10.0-1127.8.2.el7.x86_64
drbd90-utils-9.12.2-1.el7.elrepo.x86_64
kmod-drbd90-9.0.22-2.el7_8.elrepo.x86_64
-----------------------------------------------------
-----------------------------------------------------
centos 8.1.1911
kernel 4.18.0-147.8.1.el8_1.x86_64
kmod-drbd90-9.0.21-2.el8_1.elrepo.x86_64
drbd90-utils-9.10.0-2.el8.elrepo.x86_64
-------------------------------------------------------
Combinations:
centos6 <-> centos7 [OK]
centos7 <-> centos8 [OK]
centos6 <-> centos8 [NOT_OK] Error reported: "error receiving ReportState,
e: -5 l: 0!"
Migrating from centos6 to centos7 first and then after from centos7 to
centos8 seems to be the only option for now.
On Mon, 25 May 2020 at 06:29, Rupa Schomaker <rupa at rupa.com> wrote:
> I have a pair of servers that are running current centos 6 (kernel
> versions below) which I'm working on upgrading to centos 8. I would like
> to run for a period of time with one node running centos 6 and one node
> running centos 8 so that I can failback to centos 6 in the case we have
> unknown application issues.
>
> In my testbed, I have a centos 6 server that has a drbd volume that was
> originally synced with another centos 6 server. I then connected that to a
> centos 7 server which ran w/out problem, but when I try to connect to a
> centos 8 server, I get a protocol error.
>
> Below are configs/logs. Any suggestions?
>
> node 0: c61
> uname: Linux centos6-test-1.novalocal 2.6.32-754.29.1.el6.x86_64 #1 SMP
> Mon Apr 27 15:30:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> kernel: kernel-2.6.32-754.29.1.el6.x86_64
> kmod: kmod-drbd84-8.4.11-1.el6_10.elrepo.x86_64
>
> [root at centos6-test-1 schomarx]# drbdadm dump
> # /etc/drbd.conf
> global {
> usage-count yes;
> cmd-timeout-medium 600;
> cmd-timeout-long 0;
> }
>
> common {
> }
>
> # resource gitlab on centos6-test-1.novalocal: not ignored, not stacked
> # defined at /etc/drbd.d/gitlab.res:1
> resource gitlab {
> on centos6-test-1.novalocal {
> device /dev/drbd0 minor 0;
> disk /dev/vdb;
> meta-disk internal;
> address ipv4 192.168.0.25:7788;
> }
> on centos8-test-2.novalocal {
> device /dev/drbd0 minor 0;
> disk /dev/vdb;
> meta-disk internal;
> address ipv4 192.168.0.22:7788;
> }
> net {
> protocol C;
> verify-alg sha256;
> }
> }
>
> node1: c82
> uanme: Linux centos8-test-2.novalocal 4.18.0-147.8.1.el8_1.x86_64 #1 SMP
> Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> kernel: kernel-4.18.0-147.8.1.el8_1.x86_64
> kmod: kmod-25-13.el8.x86_64
>
> [root at centos8-test-2 schomarx]# drbdadm dump
> # /etc/drbd.conf
> global {
> usage-count no;
> }
>
> common {
> net {
> protocol C;
> }
> }
>
> # resource gitlab on centos8-test-2.novalocal: not ignored, not stacked
> # defined at /etc/drbd.d/gitlab.res:1
> resource gitlab {
> device /dev/drbd0 minor 0;
> on centos6-test-1.novalocal {
> node-id 1;
> disk /dev/vdb;
> address ipv4 192.168.0.25:7788;
> }
> on centos8-test-2.novalocal {
> node-id 0;
> disk /dev/vdc;
> address ipv4 192.168.0.22:7788;
> }
> net {
> verify-alg sha256;
> }
> }
>
> ====
>
> On centos8-test-2, I have /dev/vdb and /dev/vdc. /dev/vdb is the working
> backing store that was originally on centos 6, worked on centos 7
> using drbd90 and then moved to centos 8. /dev/vdc is a brand new metadata
> 90 backing store. Neither are able to establish a connection
> without a protocol error.
>
> I've pasted the logs from each type.
>
> For /dev/vdb which has data from a prior sync:
>
> === centos6-test-1 ===
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe ->
> Unconnected )
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver terminated
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Restarting receiver
> thread
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver (re)started
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( Unconnected ->
> WFConnection )
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Handshake successful:
> Agreed network protocol version 101
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Feature flags enabled
> on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( WFConnection ->
> WFReportParams )
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Starting ack_recv
> thread (from drbd_r_gitlab [10505])
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: self
> 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
> bits:262127 flags:0
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer
> 449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0
> flags:0
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: uuid_compare()=1 by
> rule 70
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer( Unknown ->
> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown ->
> Consistent )
> May 20 13:23:47 centos6-test-1 kernel: block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%May
> 20 13:23:47 centos6-test-1 kernel: drbd gitlab: sock was shut down by peer
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: peer( Secondary ->
> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( Consistent -> DUnknown )
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: short read (expected
> size 16)
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: meta connection shut
> down by peer.
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Terminating
> drbd_a_gitlab
> May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Connection closed
>
> === centos8-test-2 ===
>
> May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )
> May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Restarting receiver thread
> May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Unconnected -> Connecting )
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network
> protocol version 101
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM
> THIN_RESYNC WRITE_SAME WRITE_ZEROES.
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab
> [6057])
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Connecting -> Connected )
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: My current UUID/flags changed during handshake.
> Retry connecting.
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: drbd_sync_handshake:
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: self
> 449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0
> flags:120
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: peer
> 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
> bits:262127 flags:20
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: My current UUID/flags changed during handshake.
> Retry connecting.
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Connected -> ProtocolError )
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: ack_receiver terminated
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Terminating ack_recv thread
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Aborting remote state change 0 commit not possible
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Restarting sender thread
> May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Connection closed
>
> ===================
>
> Now if I try with /dev/vdc which is a fresh invalidated device:
>
> === centos6-test-1 ===
> May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: receiver (re)started
> May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: conn( Unconnected ->
> WFConnection )
> May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Handshake successful:
> Agreed network protocol version 101
> May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Feature flags enabled
> on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( WFConnection ->
> WFReportParams )
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Starting ack_recv
> thread (from drbd_r_gitlab [10505])
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: self
> 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
> bits:262127 flags:0
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer
> 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0
> flags:4
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: uuid_compare()=2 by
> rule 30
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: Becoming sync source
> due to disk states.
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: Writing the whole
> bitmap, full sync required after drbd_sync_handshake.
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: 1024 MB (262127 bits)
> marked out-of-sync by on disk bit-map.
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer( Unknown ->
> Secondary ) conn( WFReportParams -> WFBitMapS )
> May 20 13:29:39 centos6-test-1 kernel: block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: sock was shut down by
> peer
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: peer( Secondary ->
> Unknown ) conn( WFBitMapS -> BrokenPipe )
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: short read (expected
> size 16)
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: meta connection shut
> down by peer.
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Terminating
> drbd_a_gitlab
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Connection closed
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe ->
> Unconnected )
> May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: receiver terminated
>
> === centos8-test-2 ===
> May 20 09:29:44 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Restarting receiver thread
> May 20 09:29:44 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Unconnected -> Connecting )
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network
> protocol version 101
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM
> THIN_RESYNC WRITE_SAME WRITE_ZEROES.
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab
> [6471])
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Connecting -> Connected )
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: drbd_sync_handshake:
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: self
> 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0
> flags:124
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: peer
> 027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
> bits:262127 flags:20
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: My current UUID/flags changed during handshake.
> Retry connecting.
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
> centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( Connected -> ProtocolError )
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: ack_receiver terminated
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Terminating ack_recv thread
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Aborting remote state change 0 commit not possible
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Restarting sender thread
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: Connection closed
> May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
> centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )
>
> --
> -Rupa
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user at lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20200526/c142e225/attachment-0001.htm>
More information about the drbd-user
mailing list