[DRBD-user] Content of DRBD volume is invalid during sync after disk replace

[Michal Michaláč]

                 Hello,

 

after replacing backing device of DRBD, content of DRBD volume (not only
backing disk) is invalid on node with inconsistent backing device, until
sync finishes. I think, correct behaviour is to read data from peer's
(consistent) backing device if process running on node with inconsistent
backing device wants to read unsynchronized part of DRBD volume.

 

 

Steps to repeat:

Having 2 nodes (n1 and n2) and 1GB DRBD volume test
(/dev/drbd/by-res/test/0) backed by LVM thin LV /dev/drbdpool/test_00 on
both nodes. 

 

1) Write random data to test volume from one of nodes and generate hash:

n2# head -c 600M < /dev/urandom > /dev/drbd/by-res/test/0

 

n1# md5sum  /dev/drbd/by-res/test/0

8e712153e61d7fe5d74b5aee2805a0e6  /dev/drbd/by-res/test/0

n2# md5sum  /dev/drbd/by-res/test/0

8e712153e61d7fe5d74b5aee2805a0e6  /dev/drbd/by-res/test/0

Both hashes are identical.

 

2) Promote to primary on n2 node and detach disk.

n2# drbdadm primary test

n2# drbdadm detach test

 

n2# md5sum  /dev/drbd/by-res/test/0

8e712153e61d7fe5d74b5aee2805a0e6  /dev/drbd/by-res/test/0

Hash doesn't change - correct.

 

3) Write random data to test volume from (diskless) node n2:

n2# head -c 600M < /dev/urandom > /dev/drbd/by-res/test/0

 

n2# drbdadm secondary test

n1# md5sum  /dev/drbd/by-res/test/0

56cd1b4ca88870357ab156a835435d9e  /dev/drbd/by-res/test/0

n2# md5sum  /dev/drbd/by-res/test/0

56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0

Both hashes has changed but still identical.

 

4) Simulate changed backing device by write new metadata on test volume at
n2:

n2# drbdadm create-md test

 

n2# md5sum  /dev/drbd/by-res/test/0

56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0

Hash does not changed - still correct.

 

5) Attach backing device on n2 to test volume:

drbdadm attach test

 

Full sync started automatically, drdbadm status shows disk on n2 is
inconsistent (correct):

n2# drbdadm status

...

test role:Primary

  disk:Inconsistent

  n1 role:Secondary

    replication:SyncTarget peer-disk:UpToDate done:30.89

...

 

Check consistency of test volume on n2 (node with inconsistent disk):

n2# md5sum  /dev/drbd/by-res/test/0

d3eed24210b1925ed870c8de96765d38  /dev/drbd/by-res/test/0

Again after minute:

n2# md5sum  /dev/drbd/by-res/test/0

28bf148435d78b825296c275b4604605  /dev/drbd/by-res/test/0

 

This is problem - content of test volume on node with incosistent disk is
invalid and changes during sync process! 

 

After sync finishes, test volume will have correct content (on both nodes):

n2# md5sum  /dev/drbd/by-res/test/0

56cd1b4ca88870357ab156a835435d9e  /dev/drbd/by-res/test/0

 

 

If I  skip create-md (step 4), situation is even worse - after attach disk,
DRBD says volume is sychronized(!):

log: Began resync as SyncTarget (will sync 0 KB [0 bits set])

but after verification (drbdadm verify test), there are many out-of-sync
sectors. 

After disconnect/connect volume test, resync not started(!):

log: No resync, but 3840 bits in bitmap!

 

 

If I (on new DRBD volume) just disconnect -> write changes to primary ->
connect, sync works correctly.

 

 

Versions (on both nodes are identical): 

# cat /proc/drbd

version: 9.0.14-1 (api:2/proto:86-113)

GIT-hash: 62f906cf44ef02a30ce0c148fec223b40c51c533 build by root at n2,
2018-07-12 13:18:02

Transports (api:16): tcp (9.0.14-1)

# uname -a

Linux n2 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200)
x86_64 GNU/Linux

# lvm version

  LVM version:     2.02.168(2) (2016-11-30)

  Library version: 1.02.137 (2016-11-30)

  Driver version:  4.37.0

 

 

Is it bug or am I doing something wrong?

 

Thanks

                 Michal Michalac

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20180723/86872f26/attachment-0001.htm>