[DRBD-user] Proxmox:Linstore, new third node, all volumes diskless, why?
Dr. Volker Jaenisch
volker.jaenisch at inqbus.de
Wed Dec 2 21:08:26 CET 2020
Dear DRBD Users!
I have setup a proxmox 6.2 cluster linstore 1.9, linstore-proxmox with
two nodes Hydra3/4 two months ago. Works like a charm.
Now I added a third node Hydra1 (proxmox 6.3, linstore 1.9,
linstore-proxmox).
[This project is a Proxmox 4->6 migration. I migrated all VMs from
Hydra1/2 (Proxmox4) by hand to our new Hydra3/4 (Proxmox6) servers and
now reintegrate Hydra1/2]
Also no problem. But all the volumes are now and remain in diskless
state on Hydra1?
DRBDTOP v0.2.3-4-g5fa4b14 (kernel: 9.0.25; utils: 9.15.1; host: hydra1)
- Details for vm-118-disk-1
Resource: vm-118-disk-1: (Overall danger score:
0)
│ Local
Disc(*Secondary*):
│ volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)
│
│ Connection to hydra3(*Primary*): Connected(connected to
hydra3)
│ volume
0:
│ UpToDate(normal disk
state)
│
│ Connection to hydra4(Secondary): Connected(connected to
hydra4)
│ volume
0:
│ UpToDate(normal disk state)
The 3-Node cluster is fully operational, creation of VMs and live
migration between all three node works perfectly.
So when I migrate e.g. VM118 from Hydra3 to the new Hydra1 the VMs
switches really fine. But in the end the DRBD-Device is still diskless
on Hydra1
│Resource: vm-118-disk-1: (Overall danger score:
0)
│ Local
Disc(*Primary*):
│ volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)
│
│ Connection to hydra3(*Secondary*): Connected(connected to
hydra3)
│ volume
0:
│ UpToDate(normal disk
state)
│
│ Connection to hydra4(Secondary): Connected(connected to
hydra4)
│ volume
0:
│ UpToDate(normal disk state)
Stupid question: How do I get the volumes also DRBD-replicated onto the
third node? I thought that this would be done in the moment when I start
a migration.
/etc/pve/storage.cfg
drbd: drbdstorage
content images, rootdir
controller 10.1.3.18
resourcegroup ResGrpDRBD
root at hydra1:~# linstor rg list
╭──────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊
╞══════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ ResGrpDRBD ┊ PlaceCount: 2 ┊ 0 ┊ ┊
┊ ┊ StoragePool(s): pool_drbd ┊ ┊ ┊
╰──────────────────────────────────────────────────────────────────╯
root at hydra1:~# linstor volume-definition list
╭───────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ Gross ┊ State ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ 0 ┊ 1000 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-100-disk-2 ┊ 0 ┊ 1001 ┊ 900 GiB ┊ ┊ ok ┊
┊ vm-101-disk-1 ┊ 0 ┊ 1002 ┊ 100 GiB ┊ ┊ ok ┊
┊ vm-102-disk-1 ┊ 0 ┊ 1003 ┊ 16 GiB ┊ ┊ ok ┊
┊ vm-103-disk-1 ┊ 0 ┊ 1004 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-104-disk-1 ┊ 0 ┊ 1005 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-105-disk-1 ┊ 0 ┊ 1006 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-105-disk-2 ┊ 0 ┊ 1007 ┊ 100 GiB ┊ ┊ ok ┊
┊ vm-106-disk-1 ┊ 0 ┊ 1008 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-107-disk-1 ┊ 0 ┊ 1009 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-107-disk-2 ┊ 0 ┊ 1010 ┊ 80 GiB ┊ ┊ ok ┊
┊ vm-108-disk-1 ┊ 0 ┊ 1011 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-109-disk-1 ┊ 0 ┊ 1012 ┊ 100 GiB ┊ ┊ ok ┊
┊ vm-110-disk-1 ┊ 0 ┊ 1013 ┊ 100 GiB ┊ ┊ ok ┊
┊ vm-110-disk-2 ┊ 0 ┊ 1014 ┊ 1.46 TiB ┊ ┊ ok ┊
┊ vm-111-disk-1 ┊ 0 ┊ 1015 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-112-disk-1 ┊ 0 ┊ 1016 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-113-disk-1 ┊ 0 ┊ 1017 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-114-disk-1 ┊ 0 ┊ 1018 ┊ 50 GiB ┊ ┊ ok ┊
┊ vm-115-disk-1 ┊ 0 ┊ 1019 ┊ 22 GiB ┊ ┊ ok ┊
┊ vm-115-disk-2 ┊ 0 ┊ 1020 ┊ 100 GiB ┊ ┊ ok ┊
┊ vm-116-disk-1 ┊ 0 ┊ 1021 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-117-disk-1 ┊ 0 ┊ 1022 ┊ 32 GiB ┊ ┊ ok ┊
┊ vm-118-disk-1 ┊ 0 ┊ 1023 ┊ 32 GiB ┊ ┊ ok ┊
╰───────────────────────────────────────────────────────────────────╯
root at hydra1:~# linstor resource-definition list
╭──────────────────────────────────────────────╮
┊ ResourceName ┊ Port ┊ ResourceGroup ┊ State ┊
╞══════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ 7000 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-100-disk-2 ┊ 7001 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-101-disk-1 ┊ 7002 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-102-disk-1 ┊ 7003 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-103-disk-1 ┊ 7004 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-104-disk-1 ┊ 7005 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-105-disk-1 ┊ 7006 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-105-disk-2 ┊ 7007 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-106-disk-1 ┊ 7008 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-107-disk-1 ┊ 7009 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-107-disk-2 ┊ 7010 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-108-disk-1 ┊ 7011 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-109-disk-1 ┊ 7012 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-110-disk-1 ┊ 7013 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-110-disk-2 ┊ 7014 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-111-disk-1 ┊ 7015 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-112-disk-1 ┊ 7016 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-113-disk-1 ┊ 7017 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-114-disk-1 ┊ 7018 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-115-disk-1 ┊ 7019 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-115-disk-2 ┊ 7020 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-116-disk-1 ┊ 7021 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-117-disk-1 ┊ 7022 ┊ ResGrpDRBD ┊ ok ┊
┊ vm-118-disk-1 ┊ 7023 ┊ ResGrpDRBD ┊ ok ┊
╰──────────────────────────────────────────────╯
root at hydra1:~# linstor resource list
╭─────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊
CreatedOn ┊
╞═════════════════════════════════════════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ hydra3 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04
18:53:37 ┊
┊ vm-100-disk-1 ┊ hydra4 ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04
18:53:37 ┊
┊ vm-100-disk-2 ┊ hydra3 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04
19:49:55 ┊
┊ vm-100-disk-2 ┊ hydra4 ┊ 7001 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04
19:49:55 ┊
┊ vm-101-disk-1 ┊ hydra3 ┊ 7002 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04
20:00:09 ┊
┊ vm-101-disk-1 ┊ hydra4 ┊ 7002 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04
20:00:08 ┊
┊ vm-102-disk-1 ┊ hydra3 ┊ 7003 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04
20:04:28 ┊
┊ vm-102-disk-1 ┊ hydra4 ┊ 7003 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04
20:04:27 ┊
┊ vm-103-disk-1 ┊ hydra3 ┊ 7004 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
17:54:11 ┊
┊ vm-103-disk-1 ┊ hydra4 ┊ 7004 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
17:54:11 ┊
┊ vm-104-disk-1 ┊ hydra3 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:06:05 ┊
┊ vm-104-disk-1 ┊ hydra4 ┊ 7005 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:06:05 ┊
┊ vm-105-disk-1 ┊ hydra3 ┊ 7006 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:09:24 ┊
┊ vm-105-disk-1 ┊ hydra4 ┊ 7006 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:09:22 ┊
┊ vm-105-disk-2 ┊ hydra3 ┊ 7007 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:09:58 ┊
┊ vm-105-disk-2 ┊ hydra4 ┊ 7007 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
21:09:56 ┊
┊ vm-106-disk-1 ┊ hydra3 ┊ 7008 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:12:26 ┊
┊ vm-106-disk-1 ┊ hydra4 ┊ 7008 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:12:26 ┊
┊ vm-107-disk-1 ┊ hydra3 ┊ 7009 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:52:51 ┊
┊ vm-107-disk-1 ┊ hydra4 ┊ 7009 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:52:51 ┊
┊ vm-107-disk-2 ┊ hydra3 ┊ 7010 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:53:27 ┊
┊ vm-107-disk-2 ┊ hydra4 ┊ 7010 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20
22:53:27 ┊
┊ vm-108-disk-1 ┊ hydra3 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
08:18:52 ┊
┊ vm-108-disk-1 ┊ hydra4 ┊ 7011 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
08:18:52 ┊
┊ vm-109-disk-1 ┊ hydra3 ┊ 7012 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
08:20:33 ┊
┊ vm-109-disk-1 ┊ hydra4 ┊ 7012 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
08:20:34 ┊
┊ vm-110-disk-1 ┊ hydra3 ┊ 7013 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
11:17:06 ┊
┊ vm-110-disk-1 ┊ hydra4 ┊ 7013 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
11:17:06 ┊
┊ vm-110-disk-2 ┊ hydra3 ┊ 7014 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
11:17:44 ┊
┊ vm-110-disk-2 ┊ hydra4 ┊ 7014 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
11:17:52 ┊
┊ vm-111-disk-1 ┊ hydra3 ┊ 7015 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
13:14:20 ┊
┊ vm-111-disk-1 ┊ hydra4 ┊ 7015 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
13:14:22 ┊
┊ vm-112-disk-1 ┊ hydra3 ┊ 7016 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24
15:46:42 ┊
┊ vm-112-disk-1 ┊ hydra4 ┊ 7016 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24
15:46:44 ┊
┊ vm-113-disk-1 ┊ hydra3 ┊ 7017 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-26
11:18:34 ┊
┊ vm-113-disk-1 ┊ hydra4 ┊ 7017 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-26
11:18:34 ┊
┊ vm-114-disk-1 ┊ hydra3 ┊ 7018 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-26
22:47:51 ┊
┊ vm-114-disk-1 ┊ hydra4 ┊ 7018 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-26
22:47:51 ┊
┊ vm-115-disk-1 ┊ hydra3 ┊ 7019 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-27
12:47:43 ┊
┊ vm-115-disk-1 ┊ hydra4 ┊ 7019 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-27
12:47:42 ┊
┊ vm-115-disk-2 ┊ hydra3 ┊ 7020 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-27
12:58:01 ┊
┊ vm-115-disk-2 ┊ hydra4 ┊ 7020 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-27
12:58:01 ┊
┊ vm-116-disk-1 ┊ hydra3 ┊ 7021 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-12-02
18:42:30 ┊
┊ vm-116-disk-1 ┊ hydra4 ┊ 7021 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02
18:42:30 ┊
┊ vm-117-disk-1 ┊ hydra3 ┊ 7022 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02
19:31:31 ┊
┊ vm-117-disk-1 ┊ hydra4 ┊ 7022 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02
19:31:31 ┊
┊ vm-118-disk-1 ┊ hydra1 ┊ 7023 ┊ InUse ┊ Ok ┊ Diskless ┊ 2020-12-02
19:40:00 ┊
┊ vm-118-disk-1 ┊ hydra3 ┊ 7023 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02
19:40:04 ┊
┊ vm-118-disk-1 ┊ hydra4 ┊ 7023 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02
19:40:02 ┊
╰─────────────────────────────────────────────────────────────────────────────────╯
root at hydra1:~# linstor storage-pool list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊
TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ hydra1 ┊ DISKLESS ┊ ┊
┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ hydra3 ┊ DISKLESS ┊ ┊
┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ hydra4 ┊ DISKLESS ┊ ┊
┊ ┊ False ┊ Ok ┊
┊ pool_drbd ┊ hydra1 ┊ LVM ┊ vg_drbd ┊ 7.28 TiB
┊ 7.28 TiB ┊ False ┊ Ok ┊
┊ pool_drbd ┊ hydra3 ┊ LVM ┊ vg_drbd ┊ 14.70 TiB
┊ 18.19 TiB ┊ False ┊ Ok ┊
┊ pool_drbd ┊ hydra4 ┊ LVM ┊ vg_drbd ┊ 14.70 TiB
┊ 18.19 TiB ┊ False ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Any pointers appreciated
Volker
--
=========================================================
inqbus Scientific Computing Dr. Volker Jaenisch
Hungerbichlweg 3 +49 (8860) 9222 7 92
86977 Burggen https://inqbus.de
=========================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20201202/f440f7ca/attachment-0001.html>
More information about the drbd-user
mailing list