[DRBD-user] cross version sync failing
Rupa Schomaker
rupa at rupa.com
Wed May 20 15:39:30 CEST 2020
I have a pair of servers that are running current centos 6 (kernel versions
below) which I'm working on upgrading to centos 8. I would like to run for
a period of time with one node running centos 6 and one node running centos
8 so that I can failback to centos 6 in the case we have unknown
application issues.
In my testbed, I have a centos 6 server that has a drbd volume that was
originally synced with another centos 6 server. I then connected that to a
centos 7 server which ran w/out problem, but when I try to connect to a
centos 8 server, I get a protocol error.
Below are configs/logs. Any suggestions?
node 0: c61
uname: Linux centos6-test-1.novalocal 2.6.32-754.29.1.el6.x86_64 #1 SMP Mon
Apr 27 15:30:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
kernel: kernel-2.6.32-754.29.1.el6.x86_64
kmod: kmod-drbd84-8.4.11-1.el6_10.elrepo.x86_64
[root at centos6-test-1 schomarx]# drbdadm dump
# /etc/drbd.conf
global {
usage-count yes;
cmd-timeout-medium 600;
cmd-timeout-long 0;
}
common {
}
# resource gitlab on centos6-test-1.novalocal: not ignored, not stacked
# defined at /etc/drbd.d/gitlab.res:1
resource gitlab {
on centos6-test-1.novalocal {
device /dev/drbd0 minor 0;
disk /dev/vdb;
meta-disk internal;
address ipv4 192.168.0.25:7788;
}
on centos8-test-2.novalocal {
device /dev/drbd0 minor 0;
disk /dev/vdb;
meta-disk internal;
address ipv4 192.168.0.22:7788;
}
net {
protocol C;
verify-alg sha256;
}
}
node1: c82
uanme: Linux centos8-test-2.novalocal 4.18.0-147.8.1.el8_1.x86_64 #1 SMP
Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
kernel: kernel-4.18.0-147.8.1.el8_1.x86_64
kmod: kmod-25-13.el8.x86_64
[root at centos8-test-2 schomarx]# drbdadm dump
# /etc/drbd.conf
global {
usage-count no;
}
common {
net {
protocol C;
}
}
# resource gitlab on centos8-test-2.novalocal: not ignored, not stacked
# defined at /etc/drbd.d/gitlab.res:1
resource gitlab {
device /dev/drbd0 minor 0;
on centos6-test-1.novalocal {
node-id 1;
disk /dev/vdb;
address ipv4 192.168.0.25:7788;
}
on centos8-test-2.novalocal {
node-id 0;
disk /dev/vdc;
address ipv4 192.168.0.22:7788;
}
net {
verify-alg sha256;
}
}
====
On centos8-test-2, I have /dev/vdb and /dev/vdc. /dev/vdb is the working
backing store that was originally on centos 6, worked on centos 7
using drbd90 and then moved to centos 8. /dev/vdc is a brand new metadata
90 backing store. Neither are able to establish a connection
without a protocol error.
I've pasted the logs from each type.
For /dev/vdb which has data from a prior sync:
=== centos6-test-1 ===
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe ->
Unconnected )
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver terminated
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Restarting receiver
thread
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: receiver (re)started
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( Unconnected ->
WFConnection )
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Handshake successful:
Agreed network protocol version 101
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Feature flags enabled
on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: conn( WFConnection ->
WFReportParams )
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Starting ack_recv
thread (from drbd_r_gitlab [10505])
May 20 13:23:47 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:
May 20 13:23:47 centos6-test-1 kernel: block drbd0: self
027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
bits:262127 flags:0
May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer
449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0
flags:0
May 20 13:23:47 centos6-test-1 kernel: block drbd0: uuid_compare()=1 by
rule 70
May 20 13:23:47 centos6-test-1 kernel: block drbd0: peer( Unknown ->
Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown ->
Consistent )
May 20 13:23:47 centos6-test-1 kernel: block drbd0: send bitmap stats
[Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%May
20 13:23:47 centos6-test-1 kernel: drbd gitlab: sock was shut down by peer
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: peer( Secondary ->
Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( Consistent -> DUnknown )
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: short read (expected
size 16)
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: meta connection shut
down by peer.
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Terminating
drbd_a_gitlab
May 20 13:23:47 centos6-test-1 kernel: drbd gitlab: Connection closed
=== centos8-test-2 ===
May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )
May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Restarting receiver thread
May 20 09:23:52 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Unconnected -> Connecting )
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network
protocol version 101
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM
THIN_RESYNC WRITE_SAME WRITE_ZEROES.
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab
[6057])
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Connecting -> Connected )
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: My current UUID/flags changed during handshake.
Retry connecting.
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: drbd_sync_handshake:
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: self
449F3AA77FA5A6C6:0000000000000000:35BF4EC8A3C456DA:0000000000000004 bits:0
flags:120
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: peer
027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
bits:262127 flags:20
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: My current UUID/flags changed during handshake.
Retry connecting.
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Connected -> ProtocolError )
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: ack_receiver terminated
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Terminating ack_recv thread
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Aborting remote state change 0 commit not possible
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Restarting sender thread
May 20 09:23:53 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Connection closed
===================
Now if I try with /dev/vdc which is a fresh invalidated device:
=== centos6-test-1 ===
May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: receiver (re)started
May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: conn( Unconnected ->
WFConnection )
May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Handshake successful:
Agreed network protocol version 101
May 20 13:29:38 centos6-test-1 kernel: drbd gitlab: Feature flags enabled
on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( WFConnection ->
WFReportParams )
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Starting ack_recv
thread (from drbd_r_gitlab [10505])
May 20 13:29:39 centos6-test-1 kernel: block drbd0: drbd_sync_handshake:
May 20 13:29:39 centos6-test-1 kernel: block drbd0: self
027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
bits:262127 flags:0
May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer
0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0
flags:4
May 20 13:29:39 centos6-test-1 kernel: block drbd0: uuid_compare()=2 by
rule 30
May 20 13:29:39 centos6-test-1 kernel: block drbd0: Becoming sync source
due to disk states.
May 20 13:29:39 centos6-test-1 kernel: block drbd0: Writing the whole
bitmap, full sync required after drbd_sync_handshake.
May 20 13:29:39 centos6-test-1 kernel: block drbd0: 1024 MB (262127 bits)
marked out-of-sync by on disk bit-map.
May 20 13:29:39 centos6-test-1 kernel: block drbd0: peer( Unknown ->
Secondary ) conn( WFReportParams -> WFBitMapS )
May 20 13:29:39 centos6-test-1 kernel: block drbd0: send bitmap stats
[Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: sock was shut down by
peer
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: peer( Secondary ->
Unknown ) conn( WFBitMapS -> BrokenPipe )
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: short read (expected
size 16)
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: meta connection shut
down by peer.
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: ack_receiver terminated
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Terminating
drbd_a_gitlab
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: Connection closed
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: conn( BrokenPipe ->
Unconnected )
May 20 13:29:39 centos6-test-1 kernel: drbd gitlab: receiver terminated
=== centos8-test-2 ===
May 20 09:29:44 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Restarting receiver thread
May 20 09:29:44 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Unconnected -> Connecting )
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Handshake to peer 1 successful: Agreed network
protocol version 101
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Feature flags enabled on protocol level: 0xf TRIM
THIN_RESYNC WRITE_SAME WRITE_ZEROES.
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Starting ack_recv thread (from drbd_r_gitlab
[6471])
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Connecting -> Connected )
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: drbd_sync_handshake:
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: self
0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0
flags:124
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: peer
027F6E3610640F91:449F3AA77FA5A6C6:35BF4EC8A3C456DA:0000000000000004
bits:262127 flags:20
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: My current UUID/flags changed during handshake.
Retry connecting.
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab/0 drbd0
centos6-test-1.novalocal: uuid_compare()=retry-connect by rule 9
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: error receiving P_STATE, e: -5 l: 0!
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( Connected -> ProtocolError )
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: ack_receiver terminated
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Terminating ack_recv thread
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Aborting remote state change 0 commit not possible
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Restarting sender thread
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: Connection closed
May 20 09:29:45 centos8-test-2 kernel: drbd gitlab
centos6-test-1.novalocal: conn( ProtocolError -> Unconnected )
--
-Rupa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20200520/38e6d2ed/attachment-0001.htm>
More information about the drbd-user
mailing list