[DRBD-user] Proxmox:Linstore, new third node, all volumes diskless, why?

Dr. Volker Jaenisch volker.jaenisch at inqbus.de
Wed Dec 2 21:08:26 CET 2020


Dear DRBD Users!

I have setup a proxmox 6.2 cluster linstore 1.9, linstore-proxmox with
two nodes Hydra3/4 two months ago. Works like a charm.

Now I added a third node Hydra1 (proxmox 6.3, linstore 1.9,
linstore-proxmox).

[This project is a Proxmox 4->6 migration. I migrated all VMs from
Hydra1/2 (Proxmox4) by hand to our new Hydra3/4 (Proxmox6) servers and
now reintegrate Hydra1/2]

Also no problem. But all the volumes are now and remain in diskless
state on Hydra1?

DRBDTOP v0.2.3-4-g5fa4b14 (kernel: 9.0.25; utils: 9.15.1; host: hydra1)
- Details for vm-118-disk-1

Resource: vm-118-disk-1: (Overall danger score:
0)                                                                                          

│ Local
Disc(*Secondary*):                                                                                                                                    

│  volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)                                                                 
│                                                                                                                                                                         

│ Connection to hydra3(*Primary*): Connected(connected to
hydra3)                                                             
│  volume
0:                                                                                                                                                        

│   UpToDate(normal disk
state)                                                                                                                          

│                                                                                                                                                                          

│ Connection to hydra4(Secondary): Connected(connected to
hydra4)                                                              
│  volume
0:                                                                                                                                                         

│   UpToDate(normal disk state)         

The 3-Node cluster is fully operational, creation of VMs and live
migration between all three node works perfectly.

So when I migrate e.g. VM118 from Hydra3 to the new Hydra1 the VMs
switches really fine. But in the end the DRBD-Device is still diskless
on Hydra1

│Resource: vm-118-disk-1: (Overall danger score:
0)                                                                                                                                         

│ Local
Disc(*Primary*):                                                                                                                                                                      

│  volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)                                                                                                                           

│                                                                                                                                                                                           

│ Connection to hydra3(*Secondary*): Connected(connected to
hydra3)                                                                                                                           

│  volume
0:                                                                                                                                                                                

│   UpToDate(normal disk
state)                                                                                                                                                             

│                                                                                                                                                                                           

│ Connection to hydra4(Secondary): Connected(connected to
hydra4)                                                                                                                           

│  volume
0:                                                                                                                                                                                

│   UpToDate(normal disk state)

Stupid question: How do I get the volumes also DRBD-replicated onto the
third node? I thought that this would be done in the moment when I start
a migration.

/etc/pve/storage.cfg

drbd: drbdstorage
    content images, rootdir
    controller 10.1.3.18
    resourcegroup ResGrpDRBD

root at hydra1:~# linstor rg list
╭──────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter              ┊ VlmNrs ┊ Description ┊
╞══════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2             ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ ResGrpDRBD    ┊ PlaceCount: 2             ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): pool_drbd ┊        ┊             ┊
╰──────────────────────────────────────────────────────────────────╯

root at hydra1:~# linstor volume-definition list
╭───────────────────────────────────────────────────────────────────╮
┊ ResourceName  ┊ VolumeNr ┊ VolumeMinor ┊ Size     ┊ Gross ┊ State ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ 0        ┊ 1000        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-100-disk-2 ┊ 0        ┊ 1001        ┊ 900 GiB  ┊       ┊ ok    ┊
┊ vm-101-disk-1 ┊ 0        ┊ 1002        ┊ 100 GiB  ┊       ┊ ok    ┊
┊ vm-102-disk-1 ┊ 0        ┊ 1003        ┊ 16 GiB   ┊       ┊ ok    ┊
┊ vm-103-disk-1 ┊ 0        ┊ 1004        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-104-disk-1 ┊ 0        ┊ 1005        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-105-disk-1 ┊ 0        ┊ 1006        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-105-disk-2 ┊ 0        ┊ 1007        ┊ 100 GiB  ┊       ┊ ok    ┊
┊ vm-106-disk-1 ┊ 0        ┊ 1008        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-107-disk-1 ┊ 0        ┊ 1009        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-107-disk-2 ┊ 0        ┊ 1010        ┊ 80 GiB   ┊       ┊ ok    ┊
┊ vm-108-disk-1 ┊ 0        ┊ 1011        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-109-disk-1 ┊ 0        ┊ 1012        ┊ 100 GiB  ┊       ┊ ok    ┊
┊ vm-110-disk-1 ┊ 0        ┊ 1013        ┊ 100 GiB  ┊       ┊ ok    ┊
┊ vm-110-disk-2 ┊ 0        ┊ 1014        ┊ 1.46 TiB ┊       ┊ ok    ┊
┊ vm-111-disk-1 ┊ 0        ┊ 1015        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-112-disk-1 ┊ 0        ┊ 1016        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-113-disk-1 ┊ 0        ┊ 1017        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-114-disk-1 ┊ 0        ┊ 1018        ┊ 50 GiB   ┊       ┊ ok    ┊
┊ vm-115-disk-1 ┊ 0        ┊ 1019        ┊ 22 GiB   ┊       ┊ ok    ┊
┊ vm-115-disk-2 ┊ 0        ┊ 1020        ┊ 100 GiB  ┊       ┊ ok    ┊
┊ vm-116-disk-1 ┊ 0        ┊ 1021        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-117-disk-1 ┊ 0        ┊ 1022        ┊ 32 GiB   ┊       ┊ ok    ┊
┊ vm-118-disk-1 ┊ 0        ┊ 1023        ┊ 32 GiB   ┊       ┊ ok    ┊
╰───────────────────────────────────────────────────────────────────╯

root at hydra1:~# linstor resource-definition list
╭──────────────────────────────────────────────╮
┊ ResourceName  ┊ Port ┊ ResourceGroup ┊ State ┊
╞══════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ 7000 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-100-disk-2 ┊ 7001 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-101-disk-1 ┊ 7002 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-102-disk-1 ┊ 7003 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-103-disk-1 ┊ 7004 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-104-disk-1 ┊ 7005 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-105-disk-1 ┊ 7006 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-105-disk-2 ┊ 7007 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-106-disk-1 ┊ 7008 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-107-disk-1 ┊ 7009 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-107-disk-2 ┊ 7010 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-108-disk-1 ┊ 7011 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-109-disk-1 ┊ 7012 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-110-disk-1 ┊ 7013 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-110-disk-2 ┊ 7014 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-111-disk-1 ┊ 7015 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-112-disk-1 ┊ 7016 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-113-disk-1 ┊ 7017 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-114-disk-1 ┊ 7018 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-115-disk-1 ┊ 7019 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-115-disk-2 ┊ 7020 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-116-disk-1 ┊ 7021 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-117-disk-1 ┊ 7022 ┊ ResGrpDRBD    ┊ ok    ┊
┊ vm-118-disk-1 ┊ 7023 ┊ ResGrpDRBD    ┊ ok    ┊
╰──────────────────────────────────────────────╯

root at hydra1:~# linstor resource list
╭─────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName  ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊    State ┊
CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════╡
┊ vm-100-disk-1 ┊ hydra3 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-04
18:53:37 ┊
┊ vm-100-disk-1 ┊ hydra4 ┊ 7000 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-04
18:53:37 ┊
┊ vm-100-disk-2 ┊ hydra3 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-04
19:49:55 ┊
┊ vm-100-disk-2 ┊ hydra4 ┊ 7001 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-04
19:49:55 ┊
┊ vm-101-disk-1 ┊ hydra3 ┊ 7002 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-04
20:00:09 ┊
┊ vm-101-disk-1 ┊ hydra4 ┊ 7002 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-04
20:00:08 ┊
┊ vm-102-disk-1 ┊ hydra3 ┊ 7003 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-04
20:04:28 ┊
┊ vm-102-disk-1 ┊ hydra4 ┊ 7003 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-04
20:04:27 ┊
┊ vm-103-disk-1 ┊ hydra3 ┊ 7004 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
17:54:11 ┊
┊ vm-103-disk-1 ┊ hydra4 ┊ 7004 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
17:54:11 ┊
┊ vm-104-disk-1 ┊ hydra3 ┊ 7005 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:06:05 ┊
┊ vm-104-disk-1 ┊ hydra4 ┊ 7005 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:06:05 ┊
┊ vm-105-disk-1 ┊ hydra3 ┊ 7006 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:09:24 ┊
┊ vm-105-disk-1 ┊ hydra4 ┊ 7006 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:09:22 ┊
┊ vm-105-disk-2 ┊ hydra3 ┊ 7007 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:09:58 ┊
┊ vm-105-disk-2 ┊ hydra4 ┊ 7007 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
21:09:56 ┊
┊ vm-106-disk-1 ┊ hydra3 ┊ 7008 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:12:26 ┊
┊ vm-106-disk-1 ┊ hydra4 ┊ 7008 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:12:26 ┊
┊ vm-107-disk-1 ┊ hydra3 ┊ 7009 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:52:51 ┊
┊ vm-107-disk-1 ┊ hydra4 ┊ 7009 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:52:51 ┊
┊ vm-107-disk-2 ┊ hydra3 ┊ 7010 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:53:27 ┊
┊ vm-107-disk-2 ┊ hydra4 ┊ 7010 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-20
22:53:27 ┊
┊ vm-108-disk-1 ┊ hydra3 ┊ 7011 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
08:18:52 ┊
┊ vm-108-disk-1 ┊ hydra4 ┊ 7011 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
08:18:52 ┊
┊ vm-109-disk-1 ┊ hydra3 ┊ 7012 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
08:20:33 ┊
┊ vm-109-disk-1 ┊ hydra4 ┊ 7012 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
08:20:34 ┊
┊ vm-110-disk-1 ┊ hydra3 ┊ 7013 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
11:17:06 ┊
┊ vm-110-disk-1 ┊ hydra4 ┊ 7013 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
11:17:06 ┊
┊ vm-110-disk-2 ┊ hydra3 ┊ 7014 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
11:17:44 ┊
┊ vm-110-disk-2 ┊ hydra4 ┊ 7014 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
11:17:52 ┊
┊ vm-111-disk-1 ┊ hydra3 ┊ 7015 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
13:14:20 ┊
┊ vm-111-disk-1 ┊ hydra4 ┊ 7015 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
13:14:22 ┊
┊ vm-112-disk-1 ┊ hydra3 ┊ 7016 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-24
15:46:42 ┊
┊ vm-112-disk-1 ┊ hydra4 ┊ 7016 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-24
15:46:44 ┊
┊ vm-113-disk-1 ┊ hydra3 ┊ 7017 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-26
11:18:34 ┊
┊ vm-113-disk-1 ┊ hydra4 ┊ 7017 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-26
11:18:34 ┊
┊ vm-114-disk-1 ┊ hydra3 ┊ 7018 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-26
22:47:51 ┊
┊ vm-114-disk-1 ┊ hydra4 ┊ 7018 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-26
22:47:51 ┊
┊ vm-115-disk-1 ┊ hydra3 ┊ 7019 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-27
12:47:43 ┊
┊ vm-115-disk-1 ┊ hydra4 ┊ 7019 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-27
12:47:42 ┊
┊ vm-115-disk-2 ┊ hydra3 ┊ 7020 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-11-27
12:58:01 ┊
┊ vm-115-disk-2 ┊ hydra4 ┊ 7020 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-11-27
12:58:01 ┊
┊ vm-116-disk-1 ┊ hydra3 ┊ 7021 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2020-12-02
18:42:30 ┊
┊ vm-116-disk-1 ┊ hydra4 ┊ 7021 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-12-02
18:42:30 ┊
┊ vm-117-disk-1 ┊ hydra3 ┊ 7022 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-12-02
19:31:31 ┊
┊ vm-117-disk-1 ┊ hydra4 ┊ 7022 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-12-02
19:31:31 ┊
┊ vm-118-disk-1 ┊ hydra1 ┊ 7023 ┊ InUse  ┊ Ok    ┊ Diskless ┊ 2020-12-02
19:40:00 ┊
┊ vm-118-disk-1 ┊ hydra3 ┊ 7023 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-12-02
19:40:04 ┊
┊ vm-118-disk-1 ┊ hydra4 ┊ 7023 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2020-12-02
19:40:02 ┊
╰─────────────────────────────────────────────────────────────────────────────────╯

root at hydra1:~# linstor storage-pool list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node   ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊
TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ hydra1 ┊ DISKLESS ┊          ┊             
┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ hydra3 ┊ DISKLESS ┊          ┊             
┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ hydra4 ┊ DISKLESS ┊          ┊             
┊               ┊ False        ┊ Ok    ┊
┊ pool_drbd            ┊ hydra1 ┊ LVM      ┊ vg_drbd  ┊     7.28 TiB
┊      7.28 TiB ┊ False        ┊ Ok    ┊
┊ pool_drbd            ┊ hydra3 ┊ LVM      ┊ vg_drbd  ┊    14.70 TiB
┊     18.19 TiB ┊ False        ┊ Ok    ┊
┊ pool_drbd            ┊ hydra4 ┊ LVM      ┊ vg_drbd  ┊    14.70 TiB
┊     18.19 TiB ┊ False        ┊ Ok    ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Any pointers appreciated

Volker


-- 
=========================================================
   inqbus Scientific Computing    Dr.  Volker Jaenisch
   Hungerbichlweg 3               +49 (8860) 9222 7 92
   86977 Burggen                     https://inqbus.de
=========================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20201202/f440f7ca/attachment-0001.html>


More information about the drbd-user mailing list