[DRBD-user] Issues migrating from drbdmanage to linstor with linstor 0.9.4
Wolfgang Walkowiak
walkowiak at hep.physik.uni-siegen.de
Sun Mar 31 14:29:22 CEST 2019
Hi,
we are experiencing issues while we try to migrate an existing DRBD
cluster from drbdmanage to linstor (Debian stable with proxmox for VMs)
following the instructions at
https://docs.linbit.com/docs/users-guide-9.0/#s-linstor-migrate-from-dm
The issue is that the resource files created when I execute the
migration script (dmmmigrate.sh in my case) created by
monetbase2:~# drbdmanage export-ctrlvol > ctrlvol.json
monetbase2:~# linstor dm-migrate ctrlvol.json dmmmigrate.sh
contain wrong port and wrong VolumeMinor numbers.
However, the numbers are correctly assigned in the lines for the
resource and volume creation inside dmmigrate.sh. This is somewhat
puzzeling.
In order to debug it, I decided to manually execute the contents of
dmmmigrate.sh line by line. (After reverting to drbdmanage and a clean
linstor installation.) I took the volume vm-1802-disk-2
which according to [A] (see below) uses port 7014 and VolumeMinor 125.
First I made sure that 'drbdmanage shutdown -qc' worked and DBUS wouldn't
restart it (on both DRBD nodes). Then I disabled /etc/drbd.d/drbdctrl.res
and /etc/drbd.d/drbdmanage-resources.res on both nodes, started
linstor-satellite on them and the linstor-controller on node monetbase2.
Then, following [B] from dmmmigrate.sh below, on the controller node,
monetbase2, I executed:
monetbase2:~# LS_CONTROLLERS="localhost"
monetbase2:~# export LS_CONTROLLERS
monetbase2:~# linstor node create --node-type Combined munchbase2
10.0.7.13
monetbase2:~# linstor node create --node-type Combined monetbase2
10.0.7.12
monetbase2:~# linstor n l
monetbase2:~# linstor storage-pool-definition create drbdpool
monetbase2:~# linstor storage-pool create lvmthin munchbase2 drbdpool
drbdpool/drbdthinpool
monetbase2:~# linstor storage-pool create lvmthin monetbase2 drbdpool
drbdpool/drbdthinpool
monetbase2:~# linstor sp l
The created storage pool looked fine.
Going on:
monetbase2:~# linstor resource-definition create --port 7014
vm-1802-disk-2
monetbase2:~# linstor resource-definition drbd-options
--allow-two-primaries yes vm-1802-disk-2
monetbase2:~# linstor volume-definition create --vlmnr 0 --minor 125
vm-1802-disk-2 44040192K
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
OverrideVlmId vm-1802-disk-2_00
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
AllowLargerVolumeSize true
monetbase2:~# linstor resource create --node-id 1 --storage-pool drbdpool
munchbase2 vm-1802-disk-2
monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
monetbase2 vm-1802-disk-2
the last command produced an error message:
monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
monetbase2 vm-1802-disk-2
SUCCESS:
Description:
New resource 'vm-1802-disk-2' on node 'monetbase2' registered.
Details:
Resource 'vm-1802-disk-2' on node 'monetbase2' UUID is:
30b98a7f-cb31-4635-89cd-6860b13fa5cc
SUCCESS:
Description:
Volume with number '0' on resource 'vm-1802-disk-2' on node
'monetbase2' successfully registered
Details:
Volume UUID is: 86fc67d7-18f1-41b1-9718-4b15ddf894f9
ERROR:
(Node: 'monetbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
linstor error-reports show 5CA090AF-97F26-000000
ERROR:
(Node: 'munchbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
linstor error-reports show 5CA09099-17F52-000000
Now, looking at e.g. the first error report (see [C] and [D] below for the
full error reports), I see:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup detach --diskless 125
drbdsetup del-minor 125
drbdsetup new-minor vm-1802-disk-2 1000 0
drbdsetup disconnect vm-1802-disk-2 1
drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
ipv4:10.0.7.13:7014
drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000
that a wrong port and a wrong VolumeMinor number are forced.
Looking into /var/lib/linstor.d/vm-1802-disk-2.res (see [E] below),
I see that the wrong asignments are contained in this resource file.
I also see the wrong assignments using linstor now:
monetbase2:/var/lib/linstor.d# linstor vd l -p
+----------------------------------------------------------+
| ResourceName | VolumeNr | VolumeMinor | Size | State |
|----------------------------------------------------------|
| vm-1802-disk-2 | 0 | 1000 | 42 GiB | ok |
+----------------------------------------------------------+
monetbase2:/var/lib/linstor.d# linstor r l -p
+--------------------------------------------------------+
| ResourceName | Node | Port | Usage | State |
|--------------------------------------------------------|
| vm-1802-disk-2 | monetbase2 | 7000 | Unused | Diskless |
| vm-1802-disk-2 | munchbase2 | 7000 | Unused | Diskless |
+--------------------------------------------------------+
Unfortunately, this spoils my attempt to migrate from drbdmanage to
linstor on an active system since I could not figure out,
how to modify the port number and the VolumeMinor using the linstor
client. I'll revert to the drbdmanage setup for now.
However, I would be grateful if someone knowledgable could have a look
and see whether this is a bug or a mistake on my side. A workaround
(like how force these two numbers with a linstor client command)
would be appreciated.
Sorry for the long post. Hopefully I have provided enough details.
Thank you for having a look,
w.w.
[A] -------------------------
munchbase2:~# drbdmanage list-volumes -p -g Port
+----------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | Port | |
State |
|----------------------------------------------------------------------------|
| vm-1801-disk-1 | 0 | 32 GiB | 103 | 7000 | |
ok |
| vm-103-disk-2 | 0 | 1 TiB | 104 | 7001 | |
ok |
| vm-103-disk-3 | 0 | 2 TiB | 105 | 7002 | |
ok |
| vm-103-disk-4 | 0 | 1 TiB | 106 | 7003 | |
ok |
| vm-103-disk-5 | 0 | 500 GiB | 107 | 7004 | |
ok |
| vm-102-disk-2 | 0 | 1 TiB | 108 | 7005 | |
ok |
| vm-102-disk-3 | 0 | 2 TiB | 109 | 7006 | |
ok |
| vm-102-disk-4 | 0 | 1 TiB | 111 | 7007 | |
ok |
| vm-102-disk-5 | 0 | 500 GiB | 112 | 7008 | |
ok |
| vm-102-disk-6 | 0 | 100 GiB | 113 | 7009 | |
ok |
| vm-1802-disk-1 | 0 | 32 GiB | 114 | 7010 | |
ok |
| vm-105-disk-2 | 0 | 200 GiB | 120 | 7011 | |
ok |
| vm-104-disk-1 | 0 | 18 GiB | 119 | 7012 | |
ok |
| vm-105-disk-1 | 0 | 32 GiB | 121 | 7013 | |
ok |
| vm-1802-disk-2 | 0 | 42 GiB | 125 | 7014 | |
ok |
| vm-1801-disk-12 | 0 | 32 GiB | 123 | 7015 | |
ok |
| vm-103-disk-1 | 0 | 50 GiB | 126 | 7016 | |
ok |
| vm-102-disk-1 | 0 | 50 GiB | 127 | 7017 | |
ok |
| vm-106-disk-1 | 0 | 32 GiB | 128 | 7018 | |
ok |
+----------------------------------------------------------------------------+
[B] -------------------------
dmmmigrate.sh:
...
# If the controller is not executed on the local host, set this variable:
LS_CONTROLLERS="localhost"
export LS_CONTROLLERS
...
### Nodes ###
linstor node create --node-type Combined munchbase2 10.0.7.13
linstor node create --node-type Combined monetbase2 10.0.7.12
### Storage ###
linstor storage-pool-definition create drbdpool
linstor storage-pool create lvmthin munchbase2 drbdpool drbdpool/drbdthinpool
linstor storage-pool create lvmthin monetbase2 drbdpool drbdpool/drbdthinpool
...
### Resource: vm-1802-disk-2 ###
linstor resource-definition create --port 7014 vm-1802-disk-2
linstor resource-definition drbd-options --allow-two-primaries yes vm-1802-disk-2
linstor volume-definition create --vlmnr 0 --minor 125 vm-1802-disk-2 44040192K
linstor volume-definition set-property vm-1802-disk-2 0 OverrideVlmId vm-1802-disk-2_00
linstor volume-definition set-property vm-1802-disk-2 0 AllowLargerVolumeSize true
linstor resource create --node-id 1 --storage-pool drbdpool munchbase2 vm-1802-disk-2
linstor resource create --node-id 0 --storage-pool drbdpool monetbase2 vm-1802-disk-2
[C] ------------------
monetbase2:/etc/linstor# linstor error-reports show 5CA090AF-97F26-000000
ERROR REPORT 5CA090AF-97F26-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.9.4
Build ID:
ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time: 2019-03-29T09:40:59+00:00
Error time: 2019-03-31 12:49:17
Node: monetbase2
============================================================
Reported error:
===============
Description:
Failed to adjust DRBD resource vm-1802-disk-2
Category: LinStorException
Class name: ResourceException
Class canonical name:
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at: Method 'adjustDrbd', Source file
'DrbdLayer.java', Line #432
Error message: Failed to adjust DRBD resource
vm-1802-disk-2
Error context:
An error occurred while processing resource 'Node: 'monetbase2', Rsc:
'vm-1802-disk-2''
Call backtrace:
Method Native Class:Line number
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
Caused by:
==========
Description:
Execution of the external command 'drbdadm' failed.
Cause:
The external command exited with error code 1.
Correction:
- Check whether the external program is operating properly.
- Check whether the command line is correct.
Contact a system administrator or a developer if the command line is
no longer valid
for the installed version of the external program.
Additional information:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup detach --diskless 125
drbdsetup del-minor 125
drbdsetup new-minor vm-1802-disk-2 1000 0
drbdsetup disconnect vm-1802-disk-2 1
drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
ipv4:10.0.7.13:7014
drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000
The external command sent the follwing error information:
[ne] vol:0 minor differs: r=125 c=1000
Value of 'shared-secret' differs: r=bXj5Gn1qO2YXgQoX6wLE
c=aPDz2JoXsWxbscd2j70f
vm-1802-disk-2: Failure: (173) Combination of local address(port) and
remote address(port) already in use
Command 'drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000' terminated with exit code 10
Category: LinStorException
Class name: ExtCmdFailedException
Class canonical name:
com.linbit.extproc.ExtCmdFailedException
Generated at: Method 'execute', Source file
'DrbdAdm.java', Line #455
Error message: The external command 'drbdadm' exited
with error code 1
Call backtrace:
Method Native Class:Line number
execute N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
adjust N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
END OF ERROR REPORT.
[D] ------------------
monetbase2:/etc/linstor# linstor error-reports show 5CA09099-17F52-000000
ERROR REPORT 5CA09099-17F52-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.9.4
Build ID:
ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time: 2019-03-29T09:40:59+00:00
Error time: 2019-03-31 12:49:17
Node: munchbase2
============================================================
Reported error:
===============
Description:
Failed to adjust DRBD resource vm-1802-disk-2
Category: LinStorException
Class name: ResourceException
Class canonical name:
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at: Method 'adjustDrbd', Source file
'DrbdLayer.java', Line #432
Error message: Failed to adjust DRBD resource
vm-1802-disk-2
Error context:
An error occurred while processing resource 'Node: 'munchbase2', Rsc:
'vm-1802-disk-2''
Call backtrace:
Method Native Class:Line number
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
Caused by:
==========
Description:
Execution of the external command 'drbdadm' failed.
Cause:
The external command exited with error code 1.
Correction:
- Check whether the external program is operating properly.
- Check whether the command line is correct.
Contact a system administrator or a developer if the command line is
no longer valid
for the installed version of the external program.
Additional information:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup new-peer vm-1802-disk-2 0 --_name=monetbase2
--allow-two-primaries=yes --shared-secret=aPDz2JoXsWxbscd2j70f
--cram-hmac-alg=sha1
drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
ipv4:10.0.7.12:7000
The external command sent the follwing error information:
Value of 'rs-discard-granularity' differs: r=524288 c=65536
vm-1802-disk-2: Failure: (173) Combination of local address(port) and
remote address(port) already in use
Command 'drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
ipv4:10.0.7.12:7000' terminated with exit code 10
Category: LinStorException
Class name: ExtCmdFailedException
Class canonical name:
com.linbit.extproc.ExtCmdFailedException
Generated at: Method 'execute', Source file
'DrbdAdm.java', Line #455
Error message: The external command 'drbdadm' exited
with error code 1
Call backtrace:
Method Native Class:Line number
execute N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
adjust N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
END OF ERROR REPORT.
[E] --------------------------
/var/lib/linstor.d/vm-1802-disk-2.res:
# This file was generated by linstor(0.9.4), do not edit manually.
resource "vm-1802-disk-2"
{
template-file "linstor_common.conf";
net
{
cram-hmac-alg sha1;
shared-secret "TKoHhL4KThL165NGg/ja";
allow-two-primaries yes;
}
on monetbase2
{
volume 0
{
disk /dev/drbdpool/vm-1802-disk-2_00;
disk
{
discard-zeroes-if-aligned yes;
rs-discard-granularity 65536;
}
meta-disk internal;
device minor 1000;
}
node-id 0;
}
on munchbase2
{
volume 0
{
disk /dev/drbd/this/is/not/used;
disk
{
discard-zeroes-if-aligned yes;
rs-discard-granularity 65536;
}
meta-disk internal;
device minor 1000;
}
node-id 1;
}
connection
{
host monetbase2 address ipv4 10.0.7.12:7000;
host munchbase2 address ipv4 10.0.7.13:7000;
}
}
==========================================================2019-03-31==13:56==
Dr. Wolfgang Walkowiak Phone: +49-271-740-3889
Fakultaet IV / Physik Fax : +49-271-740-3886
Emmy Noether Campus
Universitaet Siegen --> Wolfgang.Walkowiak at hep.physik.uni-siegen.de
Walter-Flex-Str. 3 Wolfgang.Walkowiak at gmx.net
57068 Siegen
Germany
=============================================================================
More information about the drbd-user
mailing list