<html><body><div>Hello,</div><div><br></div><div>OK I will try to explain it a little bit more, sorry for the very long email. This is second try, while the first didn't arrived.</div><div><br></div><blockquote data-email="roland.kammerer@linbit.com">On Thu, Mar 14, 2019 at 01:12:53AM +0100, robinlee@post.cz wrote:
<br>> Hello,
<br>>
<br>> I installed DRBD/Linstor on proxmox with redundancy 1, after adding second
<br>> node to the cluster I would like to expand the resources to another node =
<br>> redundancy 2.
<br>>
<br>> The new resources seems be OK, but the older one, created with redundancy 1
<br>> seems be not mirroing to another side. I just called
<br>>
<br>> linstor resource create <second-node-name> vm-108-disk-1 --storage-pool
<br>> drbdpool
<br>
<br>So far so good, would have done the same.
<br>
</blockquote><p><br></p><p>yes this part is probably simple ;-) So here I have one proxmox machine with working drbd and linstor, no problem so far.</p><p><br></p><p>I installed second node and created proxmox cluster with the first one. Installed linstor-sattelite on the second node and added the second node (from the first one) to the controller:</p><p><br></p><p><span style="font-family:monospace">linstor node create pve-virt2 <someip><br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">and I added second nic for drbd traffic</span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">now I create storage pool on the new node</span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">vgcreate vg-virt2-sata10 /dev/sda4<br></span></p><p><span style="font-family:monospace">lvcreate -L 1.5T -T vg-virt2-sata10/drbdthinpool<br></span></p><p><span style="font-family:monospace">linstor sp c lvmthin pve-virt2 drbdpool vg-virt2-sata10/drbdthinpool<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">now I had two nodes with the same storage</span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">╭───────────────────────────────────────────────────────────────────────╮ <br>┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ <br>╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ <br>┊ pve-virt1 ┊ SATELLITE ┊ 10.0.0.16,X.X.X.16:3366 (PLAIN) ┊ <span style="font-weight:bold;color:rgb(84,255,84)">Online</span> ┊ <br>┊ pve-virt2 ┊ SATELLITE ┊ 10.0.0.66,X.X.X.66:3366 (PLAIN) ┊ <span style="font-weight:bold;color:rgb(84,255,84)">Online</span> ┊ <br>╰───────────────────────────────────────────────────────────────────────╯<br><br></span></p><p><span style="font-family:monospace">╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ <br>┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ SupportsSnapshots ┊ <br>╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ <br>┊ drbdpool ┊ pve-virt1 ┊ LvmThinDriver ┊ drbdpool/drbdthinpool ┊ 1.54 TiB ┊ 3.09 TiB ┊ true ┊ <br>┊ drbdpool ┊ pve-virt2 ┊ LvmThinDriver ┊ vg-virt2-sata10/drbdthinpool ┊ 1.22 TiB ┊ 1.50 TiB ┊ true ┊<br><br></span></p><p> </p><p>it has not the same size on both sides, but this is not problem in this moment. I started to add the replicas like this</p><p><br></p><p><span style="font-family:monospace">linstor resource create pve-virt2 vm-103-disk-1 --storage-pool drbdpool<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">it seemed to work, while on the empty </span><span style="background-color:transparent"><font face="monospace">vg-virt2-sata10/drbdthinpool were created the LV with correct names and even the drbdtop reported some sync.</font></span></p><div><span style="background-color:transparent"><font face="monospace"><br></font></span></div><p><br></p><p> </p><blockquote data-email="roland.kammerer@linbit.com"><br>> now I see that the resource has two volumes, but the second one is actually
<br>> empty, the content wasn't copied there.
<br>
<br>Where and how do you see that?
</blockquote><p><br></p><p>Now it looks like this:</p><p><br></p><p><span style="font-family:monospace">linstor volume-definition list<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">╭─────────────────────────────────────────────────────────────╮ <br>┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ State ┊ <br>╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ <br>┊ vm-101-disk-1 ┊ 0 ┊ 1001 ┊ 20 GiB ┊ <span style="color:rgb(24,178,24)">ok </span> ┊ <br>┊ vm-102-disk-1 ┊ 0 ┊ 1002 ┊ 36 GiB ┊ <span style="color:rgb(24,178,24)">ok </span> ┊ <br>┊ vm-103-disk-1 ┊ 0 ┊ 1003 ┊ 20 GiB ┊ <span style="color:rgb(24,178,24)">ok </span> ┊<br></span><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">linstor resource list<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">╭──────────────────────────────────────────────────────╮ <br>┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ State ┊ <br>╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ <br>┊ vm-101-disk-1 ┊ pve-virt1 ┊ 7001 ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ vm-101-disk-1 ┊ pve-virt2 ┊ 7001 ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ vm-102-disk-1 ┊ pve-virt1 ┊ 7002 ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ vm-102-disk-1 ┊ pve-virt2 ┊ 7002 ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ vm-103-disk-1 ┊ pve-virt1 ┊ 7003 ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ vm-103-disk-1 ┊ pve-virt2 ┊ 7003 ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊<br><br></span></p><p><span style="font-family:monospace">The Unused volume is on the new proxmox node. And this Unused logical volumes are empty, livemigration of the machine leads to immediate kernel panic and fsck (after disconnecting LV on second node) is not able to find any filesystem.</span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">linstor resource list-volumes<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ <br>┊ Node ┊ Resource ┊ StoragePool ┊ VolumeNr ┊ MinorNr ┊ DeviceName ┊ Allocated ┊ InUse ┊ State ┊ <br>╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ <br>┊ pve-virt1 ┊ vm-101-disk-1 ┊ drbdpool ┊ 0 ┊ 1001 ┊ /dev/drbd1001 ┊ 5.68 GiB ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ pve-virt2 ┊ vm-101-disk-1 ┊ drbdpool ┊ 0 ┊ 1001 ┊ /dev/drbd1001 ┊ 2.48 GiB ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ pve-virt1 ┊ vm-102-disk-1 ┊ drbdpool ┊ 0 ┊ 1002 ┊ /dev/drbd1002 ┊ 4.08 GiB ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ pve-virt2 ┊ vm-102-disk-1 ┊ drbdpool ┊ 0 ┊ 1002 ┊ /dev/drbd1002 ┊ 460.90 MiB ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ pve-virt1 ┊ vm-103-disk-1 ┊ drbdpool ┊ 0 ┊ 1003 ┊ /dev/drbd1003 ┊ 5.30 GiB ┊ <span style="font-weight:bold;color:rgb(84,255,84)">InUse </span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊ <br>┊ pve-virt2 ┊ vm-103-disk-1 ┊ drbdpool ┊ 0 ┊ 1003 ┊ /dev/drbd1003 ┊ 1.03 GiB ┊ <span style="color:rgb(24,178,24)">Unused</span> ┊ <span style="color:rgb(24,178,24)">UpToDate</span> ┊<br><br></span></p><p>Here is interesting, that the allocated space isn't the same on both sides.</p><p><br></p><p><br></p><p> </p><blockquote data-email="roland.kammerer@linbit.com"><br>
<br>The Proxmox plugin never creates multi-volume resources. Every disk is a
<br>new, single-volume, DRBD resource. If not, the plugin has a bug.
<br>
</blockquote><p><br></p><p>It wansn't made using proxmox pluing, so there isn't any problem with proxmox plugin. Resp. the firt working volume was made byt plugin, but the second is added manualy.<br></p><p> </p><blockquote data-email="roland.kammerer@linbit.com"><br>If "resource create" created a second volume, which is hard to believe,
<br>LINSTOR has a bug.
<br>
<br>So please post the res file of that resource (in /var/lib/linstor.d/).
<br>And also the "drbdsetup status vm-108-disk-1"
<br>
</blockquote><blockquote data-email="roland.kammerer@linbit.com"></blockquote><p><br></p><p><br></p><p>the resource files are shared here - https://pastebin.com/V7mjSCar</p><p><br></p><p>the status output is</p><p><br></p><p><span style="font-family:monospace">drbdsetup status vm-103-disk-1<br></span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">first node</span></p><p><span style="font-family:monospace">vm-103-disk-1 role:<span style="font-weight:bold;color:rgb(84,255,255)">Primary</span> <br> disk:<span style="font-weight:bold;color:rgb(84,255,84)">UpToDate</span> <br> pve-virt2 role:Secondary <br> peer-disk:<span style="color:rgb(24,178,24)">UpToDate</span><br><br></span></p><p><span style="font-family:monospace">second node</span></p><p><span style="font-family:monospace"><br></span></p><p><span style="font-family:monospace">vm-103-disk-1 role:Secondary <br> disk:<span style="font-weight:bold;color:rgb(84,255,84)">UpToDate</span> <br> pve-virt1 role:<span style="color:rgb(24,178,178)">Primary</span> <br> peer-disk:<span style="color:rgb(24,178,24)">UpToDate</span></span></p><p><br></p><p> </p><p><span style="font-family:monospace">and the last info, I tried to resync the volume and in dmesg I have this, please note, that the amount of data to resync after --discard-my-data is far smaller than the volume itself, see this paste</span><br></p><p><span style="font-family:monospace"><br></span></p><p><font face="monospace">https://pastebin.com/k0rtZGJH</font><br></p><p><font face="monospace"><br></font></p><p>this dmesg is from the second node, where called the <span style="font-family:monospace;background-color:transparent">drbdadm connect --discard-my-data vm-106-disk-1</span></p><p><br></p><p>Thank you for you answer</p><p><br></p><p>Sincerely</p><p><br></p><p>Robin</p><blockquote data-email="roland.kammerer@linbit.com"><br></blockquote></body></html>