<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-2">
<meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Prostư text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
mso-fareast-language:EN-US;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Text bubliny Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.ProsttextChar
{mso-style-name:"Prostư text Char";
mso-style-priority:99;
mso-style-link:"Prostư text";
font-family:"Courier New";}
span.StylE-mailovZprvy19
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";}
span.TextbublinyChar
{mso-style-name:"Text bubliny Char";
mso-style-priority:99;
mso-style-link:"Text bubliny";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=SK link=blue vlink=purple><div class=WordSection1><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Hello,<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>after replacing backing device of DRBD, content of DRBD volume (not only backing disk) is invalid on node with inconsistent backing device, until sync finishes. I think, correct behaviour is to read data from peer</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>’s (consistent) backing device if process running on node with inconsistent backing device wants to read unsynchronized part of DRBD volume.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Steps to repeat:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Having 2 nodes (n1 and n2) and 1GB DRBD volume test (/dev/drbd/by-res/test/0) backed by LVM thin LV /dev/drbdpool/test_00 on both nodes. <o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>1) Write random data to test volume from one of nodes and generate hash:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# head -c 600M < /dev/urandom > /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n1# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>8e712153e61d7fe5d74b5aee2805a0e6 /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>8e712153e61d7fe5d74b5aee2805a0e6 /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Both hashes are identical.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>2) Promote to primary on n2 node and detach disk.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# drbdadm primary test<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# drbdadm detach test<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>8e712153e61d7fe5d74b5aee2805a0e6 /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Hash doesn</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>’t change - correct</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>3) Write random data to test volume from (diskless) node n2:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# head -c 600M < /dev/urandom > /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# drbdadm secondary test<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n1# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Both hashes has changed but still identical.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>4) Simulate changed backing device by write new metadata on test volume at n2:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# drbdadm create-md test<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Hash does not changed – still correct.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>5) Attach backing device on n2 to test volume:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>drbdadm attach test<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Full sync started automatically, drdbadm status shows disk on n2 is inconsistent (correct):<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# drbdadm status<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>...<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>test role:Primary<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> disk:Inconsistent<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> n1 role:Secondary<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> replication:SyncTarget peer-disk:UpToDate done:30.89<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>...<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Check consistency of test volume on n2 (node with inconsistent disk):<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>d3eed24210b1925ed870c8de96765d38 /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Again after minute:<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>28bf148435d78b825296c275b4604605 /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>This is problem – content of test volume on node with incosistent disk is invalid and changes during sync process! <o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>After sync finishes, test volume will have correct content (on both nodes):<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>n2# md5sum /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>56cd1b4ca88870357ab156a835435d9e /dev/drbd/by-res/test/0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>If I skip create-md (step 4), situation is even worse – after attach disk, DRBD says volume is sychronized(!):<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>log: Began resync as SyncTarget (will sync 0 KB [0 bits set])<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>but after verification (drbdadm verify test), there are many out-of-sync sectors. <o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>After disconnect/connect volume test, resync not started(!):<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>log: No resync, but 3840 bits in bitmap!<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>If I (on new DRBD volume) just disconnect -> write changes to primary -> connect, sync works correctly.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Versions (on both nodes are identical): <o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'># cat /proc/drbd<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>version: 9.0.14-1 (api:2/proto:86-113)<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>GIT-hash: 62f906cf44ef02a30ce0c148fec223b40c51c533 build by root@n2, 2018-07-12 13:18:02<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Transports (api:16): tcp (9.0.14-1)<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'># uname -a<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Linux n2 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>#</span> <span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>lvm version<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> LVM version: 2.02.168(2) (2016-11-30)<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Library version: 1.02.137 (2016-11-30)<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Driver version: 4.37.0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Is it bug or am I doing something wrong?<o:p></o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Thanks<o:p></o:p></span></p><p class=MsoNormal> Michal Michalac<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>